Learn English – get a message home

One of the meanings of serve is to give or present, such as "serve food". In this case the song is clearly not talking about food, however, serve has at least another 2 specific usages, which are also related to the action of giving or presenting. One is in a legal context, where serve means "formally deliver", usually documents such as a court summons. The second comes from tennis where to serve means to start play by hitting the ball towards the other side of the field. In both these cases the serve requires handling of some sort if repercussions are to be avoided (ignoring a summons could lead to trouble and not receiving a serve in racket sports results in losing points)

And if I catch it coming back my way I'm gonna serve it to you.

Whatever "it" is, he is going to give / throw it to you. The "back" is interesting here:

And if I catch it coming my way I'm gonna serve it back to you.

would mean that you are sending something his way and he would send it back, strongly implying that he doesn't want "it" from you (but then the lyrics wouldn't fit with the melody).

As it is written it seems to imply that he sent "it" to a third person (since it is coming back) and if "it" is sent back to him, he will pass "it" on to you.

One of the meanings of the verb ream is

[North American INFORMAL] Rebuke someone fiercely: the agent was reaming him out for walking away from the deal

The gerund form, reaming, is used as a noun to mean a serious rebuke.

Your phrase means to receive a very serious rebuke.

As to its source, etymonline.com give a first use for this meaning as 1950, but does not indicate the derivation. It also notes as a meaning anal sex, dating back to 1942.

There may be a connection between these two uses. In US slang, it is not uncommon to use reference to unwelcome sexual contact to indicate a person was seriously harmed by someone who has power over them, as in

• you got screwed
• he really fucked you over
• they stuck it to you (similarly to schtupp)

To the extent that reaming might be understood to refer to anal sex (in a pejorative way), it may have migrated to mean being abusively dominated and thence to humiliated through criticism.