X would not matter for Y and X would not matter to Y have slightly different meanings.
X would not matter to Y means Y doesn't care about X.
X would not matter for Y means X doesn't affect Y. Y may still care about X, though.
Pointing a finger at someone may indeed be rude. Pointing a finger to someone is usually not rude at all.
The reason is that these imply different contexts. Pointing at someone is an aggressive gesture often indicating accusation.
The prosecutor pointed at the defendant as she addressed the jury. "Ladies and gentleman," she exclaimed, "the state will show that this is the man who committed the vile murder on the night of October 7th!"
Pointing to often merely indicates selection.
The two captains had the boys line up along the goal line. Each then took turns pointing to one of the boys to call him over onto his team.
This use can subtly change the nuance of an implied action:
"Mrs. Capshaw, can you show us the man you saw entering your neighbor's house the night of the murder?" the prosecutor asked.
Mrs. Capshaw nodded and pointed to the defendant. (selection)
Mrs. Capshaw nodded and pointed at the defendant. (accusation)
As for the answer to your question, it mostly depends on how aggressive you want to phrase your response. A simple request might be:
(Please) Don't point your finger at me.
or, alternately, you can take Canadian Yankee's suggestion of
Get your finger out of my face!
with a range of possible options in-between.
Best Answer
Both of your variants
are understandable to mean someone is not being reasonable.
However, from my experience (this is a disclaimer), a difference might be that
implies levels or degrees or unfairness, which gets used by AmE speakers, whereas when a BrE speaker says
the understanding is that culturally it's binary, since something either is fair or it is not fair.
It can be followed by
So, to me, very not fair, sounds possibly BrE. "Very" also emphasises the "not fairness" aspect.