Yes, it’s “free”
Each person can only do it once per round, but every round if a mage casts within someone’s threatened range, he provokes from that person, who gets the Attack of Opportunity. This does not require a particular action, does not require any kind of preparation, and does not affect initiative.
No, it does not allow the enemy to go first
You cannot make Attacks of Opportunity if you are flat-footed, e.g. if you have not yet acted in the combat. So in the first round (and maybe in the surprise round), the mage can cast freely around those who have not acted yet. After their first turn, they may start taking Attacks of Opportunity.
Combat Reflexes changes these
The Combat Reflexes feat allows a number of Attacks of Opportunity equal to one’s Dex mod, and further allows Attacks of Opportunity while flat-footed.
The rules do allow such a readied Sneak Attack
The wording of that passage in grease is, in my mind, very unclear, but readied-action attacks and attacks of opportunity are at least two cases that clearly do work. They take place during the movement, so they definitely work. Readied actions basically take place whenever the person who readied the action wants it to, so you can definitely ready an action for “as he moves past me” rather than “right before he starts to move” if you like, and clearly the Rogue would like to.
As you note, this is a really difficult maneuver to use effectively. Worse, as the Rogue levels, there’s very little he can do to improve it; stuff like Two-Weapon Fighting et al., iteratives from extra BAB, and the like, aren’t going to work with it. So as he levels up, instead of getting to improve this tactic with his feats and other choices, all he can hope for is extra Sneak Attack dice – which aren’t going to keep up on their own.
Note, though: “he moves and remain at 5ft of the enemy in the grease area, and ready a 5ft step + attack with the same trigger,” doesn’t quite work: he could only ready that action if he had not moved already in the round. If he was already in position and did not need to move into it, then it would work.
I’d strongly consider reverting to the 3.5 rule
While grease was and still is one of the best possible standard actions you can take, this particular facet of the spell was a good thing, in my opinion. It encourages teamwork and is an excellent example of a Wizard helping his teammates shine, rather than stealing all the spotlight himself. If I was to nerf it, rather than removing this aspect, I’d most likely tone down the spell’s double-jeopardy nature, or its effectiveness even when the enemy saves and makes his check.
This particular rule is thus more of a nerf to Rogues (who, after all, could have easily gotten a wand of grease since as a 1st-level spell it’s fairly cheap, and they have UMD in-class) than it is to Wizards (who still have plenty of options). Rogues don’t need a nerf. In general, Pathfinder makes it decidedly more difficult for Rogues to get multiple Sneak Attacks per turn, and I don’t think that improves the game at all. It was, in fact, already too hard for them to do this in 3.5, in my opinion. So I’d strongly consider reverting to the 3.5 rule.
Best Answer
Yes.
Here is why:
A readied action is just a continuation of the player's turn that has been delayed until the occurrence of a certain trigger. Once the trigger requirements have been met (eg. Goblin comes within range of your Fighter,) as long as it is prior to the PCs next turn, the action will be completed.
Now, if an enemy moves through your Rogue's threatened range, or in this case has a Mage cast without defense, as long as the trigger were specified the Rogue would get both his turn's action (the readied action) and the attack of opportunity (with Sneak Attack die only on the readied action, so not a bad ploy at all for some decent damage).
The easiest way to think of it is as the readied action just being a continuation of the player's turn while treating the attack of opportunity like you would in any other situation.
Also note, in the case of casting, the target would need to make a single Concentration check, DC 10+spell level+all damage taken.