Ordinals are used to describe something's order in a sequence. These are the "first, second, third" words you have mentioned. However, numbers by themselves can be used to denote points on a list.
Therefore, we should use ordinals when describing things to do in sequence:
First, we need to till the soil. Second, we'll plant the flower seeds. After that, it's just a matter of waiting.
On the other hand, if someone has multiple, separate points to address, they might verbally number them for emphasis:
Speaker A: What's wrong with you?
Speaker B: Well, for one, my feet are killing me after standing around all day in these heels. Two, I'm starting to get a migrane. Three, I just found out the train is out of service today so I'll have to pay for a ride home.
Ordinals can be used in the second example, too, but it implies that the points are listed in order of most important to least important. Adding -ly to the end just turns the ordinal into its adverb form.
Best Answer
The suggestion in the question is fine. You could also point at the people and say, "You first, then you, then you, ..."
Any difference is practical, rather than linguistic. It's probably easier for most people to remember "I go after Jane" and then just wait for Jane to go, than it is to remember "I'm number 17" and accurately count the number of people who've gone. On the other hand, if anyone before me forgets who they go after and I only know that I go after Jane, the whole thing is going to fall apart.
If you want to use other pronouns, then strictly speaking, the answer depends on whether the people you're pointing at are the subjects or objects of the verb. So, if you're describing the line-up at a talent show, "She sings first, he second, he third, then she, ..."; if you're instructing your assassin, "Kill him first, him second, her third, ..." However, most people would probably use "him/her", even when it's not grammatically proper so, in the first case, "She sings first, him second, him third, then her" sounds a bit more natural, to my ear at least. If the verb is left out of all the list items, you can probably use him/her all the time. For example, "In what order are the contestants singing?" "Her first, him second, ..."