I would usually go with CLR and think it's worth every cent at that, but for reasons that are beyond the scope of this question, I am expected to justify every purchase necessary, especially chemicals and everything else a good god fearing carpenter might feel a need for but may raise eyebrows.
I found lemon juice does an OK job, basically I want to get all the rust off all the old screws and bolts in a pile of junk, I'm not exactly planning an Apollo mission here so they will do just as good as new.
Best Answer
Citric acid (dry, found at food stores - I get mine at the "asian market" as I have not spotted it at the normal supermarket) is actually a lot cheaper than lemon juice (or vinegar) (when mixed with water to an equivalent strength, or even quite a bit stronger) and fairly good at removing rust without eating the steel after that (I know antique tool users who mostly use it now, rather than electrolysis, which used to be all the rage.)
Rinse, dry quickly (or they rust again immediately) and coat with oil or wax (to slow re-rusting by displacing moisture.) Rinsing with very hot water will help with the drying step to some extent.