The concept of the Law-Chaos as appearing in fantasy worlds dichotomy originated with Poul Anderson's Three Hearts and Three Lions, although the underpinnings are much older[1]. Law was what humanity and civilization represented - the imposition of an order on savage, unpredictable wilds - while Chaos was what most non-human things thrived upon: the strange, the weird, the random, and above all, the magical.
Michael Moorcock adapted this, although the interpretation of it grew as his universe expanded. In his earlier works Chaos was essentially antagonistic, even to Elric, who drew his power from it; Law were basically the good guys, albeit ambiguously so (imagine living in a village in France in 1944: Nazis are Chaos, definitely bad guys, but the Allies are Law, and your well-being is for them somewhat secondary to fighting their enemies). Over time, the idea of Balance emerged as an alternative to either side, and although in D&D terms it is 'neutral', in Moorcock's writings it basically became the "good" side and emphasis was put on the reflective similarities of Law and Chaos, at which point they're both basically "evil" (at least once Elric fights a demon and is surprised to learn that it is a demon of Law, not Chaos), but this development didn't occur until well after the D&D system was established.
In the first edition of D&D, Andersonian terms used, but were essentially stand-ins for Good and Evil. PCs were supposed to be Lawful, so Chaotic monsters were the ones you were supposed to fight and Lawful monsters the ones to be friendly with. Supplement I: Greyhawk
had an implicit separation of good from evil, presumably to encourage Chaotic PCs in order to make the newly-introduced Paladin class seem to have more restrictions, but this wasn't made explicit until 2nd edition D&D. When AD&D was first released, D&D went back to the simple Lawful-Neutral-Chaos model and, for the first time, good and evil were fully fleshed out as principles and possible player character alignments.
Roger Zelazny's Chronicles of Amber series also makes use of a Law vs Chaos dichotomy, although it is introduced too late to have a defining influence on D&D, it reflects the characteristics of the two as they appear in AD&D: there are numerous good and bad characters on both sides of the Amber/Courts of Chaos conflict.
With each new product or edition the definitions of the alignments evolved slightly, but it always remained complex and confusing or unsuitable for many people, leaving a lot of variant interpretations and house rules (I, for one, am not satisfied with any printed description of Lawful Evil - that should be where a killer with a sense of honor is found, but the rules never support that interpretation). In the end the philosophical differences between law and chaos were discarded, so that Lawful Good meant "exceptionally good" and Chaotic Evil "exceptional evil"; many people already played as if that were the case.
[1] They can be traced to ancient Greek philosophy, with positive concepts like logos and kosmos (order and reason) opposed to negative ones aporia and khaos (confusion and chaos). Nietzsche, with other German scholars of the 19th century, identified these two modes as Apollonian and Dionysian, the idea being that human nature was a merger of civilized Apollonian tendencies with the wild, animalistic Dionysian ones. Some also read similar ideas into the Epic of Gilgamesh.
Interestingly, the Egyptians had a concept of order but two concepts of chaos; one, represented by the god Seth, is the kind of chaos found in markets and nature, that which sometimes causes some destruction but is also the source of creativity and new growth, and the other represented by the serpent Apep, is the violent, primordial chaos that only destroys. What makes this arrangement interesting is that Seth was the protector of order (in the form of Ra) against Apep.
Alignment is a description of how you have acted in the past, NOT a restriction on how you act in the future
There can be no penalties because alignment is not a straitjacket. This was one thing that 4e absolutely got 100% right (though I prefer the 9 alignments to the 5).
Playing out of alignment may anger others who share your alignment, may draw the attention of people who prefer the alignment that you’re uncharacteristically acting like (e.g. cause Evil creatures to try to corrupt you more), and so on, but these are all roleplaying issues, and that is all they should ever be.
The Dungeons & Dragons alignment system is simplistic and binary (well, nonary, I suppose). It can be a little useful as a shorthand for what side you are on, but trying to use it as anything more is a mistake that leads almost solely to arguments.
There’s a quote I like particularly well here:
Alignment just determines what color your light saber is.
Best Answer
Examples
I can't think of pen-and-paper game that uses a system like this, though even AD&D encouraged DM's to suggest to players to change the alignment if they seem to really be acting outside their alignment. Some of the White Wolf games also track alignment by points, but the path(s), like Humanity, are still normally chosen and laid out ahead of time. Probably the closest is the old Star Wars RPG that carefully tracked dark side points and declared that a character had gone over to the darkside after receiving too many of them.
Video games do this frequently though. Infamous used it, and many Star Wars based games let you choose Light v. Dark entirely based on actions.
Ambiguities and Justifications
But, as others have pointed out, you'll run into the question of what constitutes evil. Look at the examples you gave. In the first case, you have the daughter being seduced. Now, if she was already a part owner of the potion selling shop, then seducing her and asking for a potion isn't necessarily evil, its more neutral...The character had a pleasant night of passion with another character looking for the same and received a gift from his new lover afterwards, that is neither good nor evil. At least not to a liberal Western attitude...In certain more conservative, fundamentalist societies seducing her without marrying her could merit more than one evil point and could lead to one or both of them being executed.
Similarly with the kobolds, it may have been a very kind, generous, and good act to save them...Unless you are using the trope, common in many fantasy works, that kobolds are almost universally evil. In that case, there might be a bounty for killing them. Saving them might then actually be an evil act. Or the middle ground is saving them could be an act of mercy, but an illegal one so they would get both a good point and a chaos point....
Objective Approach
In short, if you want a system like that you may want to make it more objective. There are a number of ways of doing this:
Set up a code that is at least somewhat independent of good or evil. Light v. Dark in Star Wars is deliberately tied to morality to a degree, but only a degree. A dispassionate light jedi can calmly do evil acts (and do in cannon! In The Clone Wars annimated series in particular the Jedi often take actions that at least require an "ends justify the means" atitude if they aren't downright evil). In Vampire, most Vampires with high Humanity will be good, but its more about compassion and control than good per se. So you can wind up with evil vampires with high humanity and good but reckless ones with low humanity.
When you see an action with moral implications, flat out ask the player how they view their action and let that be highly influential, if not the final word. This lets intentions shine through and minimizes debate.
Make it about reputation instead of real alignment. Then its not a matter of "I acted this way, so I am this." Its a matter of "I acted this way, so everyone sees me as this." This takes intention entirely out of it, and also makes it less personal. You avoid questions of what is good and instead are asking about how the populace with their particular culture views the acts. It also avoids arguments about, "What do you mean I'm evil? I was playing good, everything I did was justified!" (Or the reverse, "I'm not good. It just so happened that in every situation the most profitable action also helped other people. I'll stab Joey in the back right now if it would advance my goals, it just hasn't yet!") Of course it does open up questions about witnesses and how everyone knows, but there are various ways of handwaving that.