The mounted combat rules provide a few scenarios for falling off a mount:
If your mount is knocked prone, you can use your reaction to dismount it as it falls and land on your feet. Otherwise, you are dismounted and fall prone in a space within 5 feet it.
The rules for falling state:
A fall from a great height is one of the most common hazards facing an adventurer. At the end of a fall, a creature takes 1d6 bludgeoning damage for every 10 feet it fell, to a maximum of 20d6. The creature lands prone, unless it avoids taking damage from the fall.
In either circumstance of falling off your mount (falling prone, or using your reaction to land on your feet), do you take any damage from falling? Does it depend on the size of your mount?
Very related: It seems obvious that if you are on a flying mount at height, and fall off, you will take some damage – but the answer to the question above will probably determine where you measure the fall distance from in such a scenario (square above mount/top square of mount/bottom square of mount/square below mount).