If your soldier gets gravely wounded in a mission, there is a chance that the soldier gets shaken. This chance is 20/25/25/30% for the four difficulties.
What is gravely wounded? It means that the recovery time is more than 168 Hours (336 on Legendary) - this is determined at the end of the mission.
How is recovery time determined? Based on the lowest HP percentage at any point in the mission, a number is generated within certain limits. There are four "buckets": up to 20%, 21-50%, 51-75%, 76+%.
Example, your Soldier has 10 HP and takes 6 Damage, which brings him down to 40% health. At the end of the mission, the game looks in the 21-50% bucket to see the range of how long recovery takes and picks a random number from that. If that random number is more than 168 hours (336 on legendary), you're gravely wounded and the game rolls if you're shaken.
Note that the source of damage doesn't play into it, and neither does healing. The game records the lowest your HP ever was. E.g., 10 health, take 6 damage, heal 4 damage, take 3 more damage, heal 4 damage -> Even though you end the mission with 9 HP, the game records that you were down to 4 HP, which is 40% of 10. That percentage is the ONLY thing that counts.
I do note that the spread between best and worst recover time is MASSIVE. For example, a single point of damage (the 76%+ bucket) on Rookie difficulty puts your Soldier out of commission for 5 to 25 days - completely random.
For more technical details, I dug into the source code here.
Bonus Answer: If you recover from being shaken, you get a permanent increase to Will (4 to 13 points) and can't be shaken again - it's like Vaccination.
On Legend, the best way I've found to deal with them is kinda cheesy. I had two Sharpshooters at one point and a Ranger. A ranger with Phantom is a real lifesaver. Never let him get spotted and have him locate the lancer pods with the rest of your squad still in the drop zone. (This wouldn't be much help on the timed missions) You can then pick the buggers off with Squad Sight if your aim is high enough. From experience, even though the squad breaks concealment, if you're far enough away from line of sight, the aggro'ed pod has trouble finding you.
Best Answer
I know this is a very old question and maybe the OP isn't even interested anymore, but I will answer anyway. The debuff is defined by a stat contest between the Lancer's Strength (75 for normal Lancers, 85 for Heavy, 95 for Elite) and the Soldier's Will.
After the Lancer hits the melee attack, he has a 50 + Strength - Will chance to apply a debuff. If he succeeds in this roll there is a second roll to define which debuff. The Weights are defined by the ratio between Strength and Will and the final chances (in %) end up being: