Yes. The tab between the upper and lower sections of each of these outlets have been broken off on the hot (black) side of the receptacles. There are separate wires running to the upper and lower sections. One is always hot and one is switched.
On the outlet you want always hot on both sections, first determine whether the upper or lower section is the switched section. After the power is turned off, disconnect the black wire on the switched part of the outlet. Cap that wire with a wirenut (and tape for backup). Disconnect the black wire from the always hot side of the receptacle. Make two short pigtails of black wire (about 3-4" each), strip both ends of these wires and attach one end of one of these wires to the upper screw and one end of the other wire to the lower screw on the hot side of the receptacle. Twist the other end of these pigtails together, along with the always hot black wire coming into the box.
This is effectively re-establishing the connection that was removed when the tab between the upper an lower sections was removed to make one half switched.
If there is only one white wire going to the receptacle, you don't need to do anything to that wire. However, if there is a separate white wire going to the switched half of the outlet, the tab on that side may also be broken, and then you would need to treat it the same as the black. That is, disconnect the switched white, cap it, create pigtails for the white side and attach them as you did with the black side.
Thanks to DoxyLover for pointing out the possible multiple circuit issue.
Supplement
As pointed out by @Johnny in a comment, an even easier solution is to replace the existing receptacle with a new receptacle that has intact tabs connecting the upper and lower sections (receptacles are very cheap) . Again, you would cap the wires connected to the switched half on the old receptacle, but then you only need to connect the black and white wires from the unswitched side to their respective screws on the new receptacle. The intact tabs will deliver power to both sockets on the receptacle.
First turn off the circuit at the breaker panel. A good practice is to confirm that there is no power on any cables in boxes you are working on using a non-contact tester.
If the switch only has one 2 conductor cable in its box, it is a switch loop. That means the power from the mains comes to the switched fixture box (in your case, the outlet) and only the hot wire is routed to the switch. The other wire in the switch cable is a switched hot (even though it is probably white or white with a black marking) which carries power back to the switched outlet.
What you need to do is to convert that switched hot to a neutral. First you need to determine which wire is always hot. Usually that would be the black wire, but you need to be sure. At the outlet box see which wire from the switch is attached to the black wire from the mains. If it is black, leave it connected. If not, disconnect it and connect the black switch wire to the black mains wire.
Then connect the white switch wire to the white mains wire. You will also need to connect the hot side of the outlet to the black mains wire (you will have already removed one of the switch wires from the hot side of the switched outlet).
Now you have an always hot outlet and a full circuit going to the switch box.
At the switch box, attach the white wire from the outlet box cable (now a neutral) to the white wire going to the light fixture. Attach the black wire from the outlet box cable to the switch (hot) and the black wire from the fixture to the other side of the switch (switched hot).
At the light fixture, black to black, white to white.
We did not mention grounds, but they should all be connected in each box and to the switches and fixtures, using pigtails if needed. These are extra pieces of wire that you attach to the wire bundles to tie the fixtures or switches into the ground lines.
Best Answer
Yes, that's exactly what you should do.
As a safety measure, while you've got the wiring exposed, double check that the hot (black and red) wires are connected to the brass-colored screw on the receptacle, the neutral (white) is connected to the silver-colored screw, and the ground is connected to the green screw.