It's rather simple, actually.
Consider a hypothetical scenario where Ashe attacks Garen (who is wearing Thornmail). Ashe has 200 attack damage and 30 MR (Ashe has no Magic Resistance items), and Garen has an effective armor of 100 (Because Ashe has armor penetration runes).
Case 1 - 100 Armor Garen, 30 MR Ashe
Ashe shoots an arrow, which would deal 200 damage. Thornmail uses this base damage to calculate the 30% retaliation. 30% of 200 is 60, so Ashe takes 46.15 damage (30 MR mitigates this to 77% of the base) from the Thornmail and deals 100 damage to Garen (100 armor reduces physical damage by 50%).
Case 2 - 200 Armor Garen, 30 MR Ashe
Garen wises up, and buys another 100 armor, and Ashe shoots him with another arrow. Ashe still takes 46.15 damage (same AD, same MR), but Garen only takes 66.67 damage.
Case 3 - 200 Armor Garen, 0 MR Ashe
But Garen is still not happy, so he buys a "Hypothetical 30 MR penetration item". Thornmail damage treats Garen as the origin, so it uses his magic penetration for the attack (same as proc items). The next time he runs into Ashe, Thornmail means Garen takes 66.67, and Ashe takes the full 60!
So in summary:
Thornmail Damage Return is based on the attacker's effective AD (can potentially be higher than just their AD for On-Hit attacks, like Mystic Shot or Parrrley, which will proc Thornmail), and the attacker's MR (reducible by the defender's magic pen).
The exact formula is pretty easy: Attack Damage * 30% * (1-MR%Reduction)
Yes.
I have not actually tested it, but the League of Legends wiki says that nocturne's ultimate range is 2000 / 2750 / 3500. Additionally, the question you linked to says "If you can outrun it for 5 seconds or get 2000+ units away from it, it'll just explode on the spot."
Assuming that both pieces of information are accurate, it should be possible to use Nocturne's ult to break the 2000 unit limit and escape. In practice, you'd need sufficiently quick reflexes and a target that is far enough away (Jumping to a target 1800 units away would not work.) I think it would be pretty rare in a real game situation for both of those requirements to be satisfied, but it should be possible.
The question of whether it's "worth it" or not will vary by situation. I will note that the cooldown for Nocturne is 160 / 130 / 100, as compared to 120 / 100 / 80 for Nautilus.
Best Answer
The whole purpose of Enchantments is to provide bonus effects, so all the original item's passives will be preserved.
From the wiki page on Enchantments:
As per your example, you could purchase the Runic Echoes Enchantment on a Stalker's Blade and still have Chilling Smite - it is in no way a downgrade.