Pointing a finger at someone may indeed be rude. Pointing a finger to someone is usually not rude at all.
The reason is that these imply different contexts. Pointing at someone is an aggressive gesture often indicating accusation.
The prosecutor pointed at the defendant as she addressed the jury. "Ladies and gentleman," she exclaimed, "the state will show that this is the man who committed the vile murder on the night of October 7th!"
Pointing to often merely indicates selection.
The two captains had the boys line up along the goal line. Each then took turns pointing to one of the boys to call him over onto his team.
This use can subtly change the nuance of an implied action:
"Mrs. Capshaw, can you show us the man you saw entering your neighbor's house the night of the murder?" the prosecutor asked.
Mrs. Capshaw nodded and pointed to the defendant. (selection)
Mrs. Capshaw nodded and pointed at the defendant. (accusation)
As for the answer to your question, it mostly depends on how aggressive you want to phrase your response. A simple request might be:
(Please) Don't point your finger at me.
or, alternately, you can take Canadian Yankee's suggestion of
Get your finger out of my face!
with a range of possible options in-between.
Best Answer
"Referring to" works fine, but you may want to use some variants if you want to talk about many elements in the chart. "In reference to item 7...", "As far as #7 goes...", "With regard to #7...", etc.