From Notch's Tumblr:

First of all, let me clarify some things about the “infinite” maps: They’re not infinite, but there’s no hard limit either. It’ll just get buggier and buggier the further out you are. Terrain is generated, saved and loaded, and (kind of) rendered in chunks of 16*16*128 blocks. These chunks have an offset value that is a 32 bit integer roughly in the range negative two billion to positive two billion. If you go outside that range (about 25% of the distance from where you are now to the sun), loading and saving chunks will start overwriting old chunks. At a 16/th of that distance, things that use integers for block positions, such as using items and pathfinding, will start overflowing and acting weird.

Those are the two “hard” limits.

Most other things, like the terrain generation seeds and entity locations use 64 bit doubles for locations, and they do much subtler things. For example, at extreme distances, the player may move slower than near the center of the world, due to rounding errors (the position has a huge mantissa, the movement delta has a tiny, so it gets cut off faster). The terrain generator can also start generating weird structures, such as huge blocks of solid material, but I haven’t seen this lately nor examined exactly what behavior causes it to happen. One major problem at long distances is that the physics starts bugging out, so the player can randomly fall into ground blocks or get stuck while walking along a wall.

Many of these problems can be solved by changing the math into a local model centered around the player so the numbers all have vaguely the same magnitude. For rendering, Minecraft already uses local coordinates within the block and offset the block position relative to the player to give the impression of the player moving. This is mostly due to OpengGL using 32 bit floats for positions, but also because the rounding errors are extremely visible when displayed on a screen.

We’re probably not going to fix these bugs until it becomes common for players to experience them while playing legitimately. My gut feeling is that nobody ever has so far, and nobody will. Walking that far will take a very long time. Besides, the bugs add mystery and charisma to the Far Lands.

As of Beta 1.8 [source]:

In beta 1.8 the strange terrain may be gone, but some of the effects aren't. After X/Z ±30,000,000 the world will start generating fake chunks. Going into the fake chunks will result in falling into the Void, and, subsequently, the player's death. If flying is used, going about 34 blocks out of the limit will cause you to be stuck, being able to use your inventory and look around, but not to move. Also, the clouds will act strangely, moving at abnormally high speeds. Reloading the save should fix the clouds, but the position glitch will still be present. Also, at excessive X/Z positions, particles (rain, water, snow, etc.) fail to appear staggered, instead forming rows of flat "panes". The Far Lands do not lag in 1.8. in 1.0.0 the fire of torches will be in/on a block next to it, instead of on the torch. In 1.0.0 explosion clouds look abnormally big. Piston heads will disappears when the block is activated and will have the same effect as sand does when falling (then disappears). Paintings can be placed over paintings near the far lands. Also, the floating book in the enchantment table seems to behave abnormally when a player is moving near it. Redstone appears either extremely distorted, with stretched graphics, it can appear misplaced, or it can appear completely invisible, but the highlight box still appears indicating where redstone is placed. Cauldrons, when looked from the inside, one of the sides is translucent, and the graphics for that side appear one block away from the cauldron on the opposite side of where the texture is missing from. Highlight box for cake fails to appear correctly. Flames from a monster spawner only appear on one side of the block. The graphic for end portal block fail to appear correctly.

The world will just generate with air blocks replacing non-vanilla blocks, and your non-vanilla items will just disappear. Re-installing the mod will not bring the blocks or items back, unless you do so before loading the world again.

(As a side note, some mods used to render your world unplayable unless the mod was installed again, I'm pretty sure Notch fixed it though, but there might be a few that mess things up.)