As with anything, usage varies, but usually when people say "sandbox" today they mean a campaign that does not have a specific prescribed storyline, but one where the GM sets up a world (or at least a small section of one) and the PCs are free to wander where they will and find adventure where they will. It's about freedom of player choice.
Pure sandbox play is purely simulation driven. A super hardcore sandboxer places a dungeon (or whatever) in the game world and that's where it is, for the PCs to come across or not (and for NPCs to come across before them or not). If a thief is sneaking into a mansion, in a sandbox game he is able to avoid guards and traps, and not have predetermined plot points presented to him regardless of his actions.
Sandbox is not an antonym for adventure module. Some of the early modules, most notably Keep on the Borderlands, were extremely sandboxy, as were many of the early dungeons (Castle Greyhawk, etc.). Here's a place, there's fell monsters and treasure there, go do what you want. More recently, Paizo did a sandbox-style adventure path called Kingmaker for Pathfinder. Sandbox is a different approach from story-driven - a "story of what happened" may emerge from a sandbox session but a preconception of story, or what "the GM wants to happen," is never applied to the game. Adventure paths, being a series of adventures, can try to be sandboxy but generally try to provide enough story to get PCs from one chapter to another, but event timelines and things like that can serve that purpose without being railroads (though people often complain and call things like that railroading, just because they feel pressured to do something).
Railroading, the antonym of sandbox, is simply extreme constraint of choice. Some perceived constraint of choice is always there in any simulated world in that there are always choices that are impossible to physically perform or clearly undesirable, but where you cross the line to railroad is when these things are obviously being imposed by the GM/metagame (usually in the name of "The Story" or "The Plot").
You can be apparently providing a sandbox but using the game world to provide so many restrictions that you are effectively railroaded into a single course of action. A dungeon full of one-way doors that inhibits all teleportation and divination, for example.
Most games are somewhere on the continuum between pure sandbox and railroad, or even move between the two based on need and GM inclination. Many campaigns switch back and forth between railroad and sandbox. Railroading to move the story on when the players lose momentum and sandbox otherwise it a frequent GM tactic that lets the players be free when they want to be but gives them structure when they're feeling lost.
Sandbox gaming can be desirable because it produces a sense of game world reality that enables the player to focus less on the metagame and immerse in their character and the game world. It can be problematic because players can feel like they are spinning their wheels and wasting limited leisure time without more guidance, and because sometimes a preplanned story cam have more "big, interesting" things happen in it plot-wise than a sandbox.
I tend towards sandboxy play, but in my most recent campaign I had players get frustrated and ask for more direct guidance from me on "what they should do" - I am normally reluctant to do that but did so to make them happier. Often players want the illusion of sandbox and unlimited choice, but with the GM pulling strings behind the scenes to keep them headed towards interesting things.
My Background
I have played just about every edition of D&D after the white box, and I play 4E. I'm currently running a 4E campaign that intentionally breaks the Gamist expectations of the rules. I occasionally play in a "Western Marches"-style Darksun 4E campaign that very much adheres to the Gamist expectations of the rules.
Gamism as I define it is a type of play in which the group of players are all interested in being tested by tough challenges and showing what they're made of (as players, not characters). It doesn't preclude creating story or developing a character (personality) or making tough moral decisions or immersing in a fantasy world, but these things take a back seat to the primary goal of play: winning challenges.
Can you play D&D 4E in a non-Gamist way? Yes, obviously. You have to break the reward cycle to do it, though, and the 4E reward cycle is very strong. Breaking the reward cycle means ignoring some of the rules, and if you ignore too many rules, are you still playing 4E?
The Reward Cycle
When I talk about the reward cycle of 4E, I mean the internal currency mechanism that rewards players for a certain kind of play. The basic cycle is this:
Players make some tough choices about their characters. The first thing they do is create a character, but this also includes choices made during play, and choices made between levels.
Players pit their characters against all manner of dangerous challenges. This rewards them (the players) with feedback about how good their choices are.
Players reward each other with feedback about how good they think everyone else's choices were. "Great job, Adam! Your fighter totally saved the lives of everyone in the party because you blocked that narrow door!"
Players earn experience points (XP) and treasure. These are types of game currency that make their characters stronger. Stronger characters can face tougher challenges! This changes play dramatically. This changes the types of decisions players will make during play.
The end of that cycle feeds back into the beginning again. Players take the feedback from the game play, feedback from the group, and the in-game currency, and apply that to new choices. The new choices include new character options when leveling up, and new tactics and strategies "in the dungeon" (or wherever).
Note that my examples are all combat and treasure, but this applies for other parts of the 4E game, like skill challenges.
Also note that I leave the roleplaying stuff out. While this is an important part of play for most people and the rules do talk about roleplay, the rules don't specifically include roleplay in the reward cycle. You earn XP by defeating monsters, by overcoming challenges, and by completing quests. Any house rule that awards XP for good roleplay must award enough XP to overshadow the normal XP awards to have any real impact, too.
Breaking the Reward Cycle
It's hard!
In my Saberpunk campaign, I basically just keep mental track of how many meaningful "scenes" the party finishes and award a new level to the group every ten scenes. Scenes include any kind of interesting conflict that the party faces. Combats are obviously part of that, as well as skill challenges, but some of these scenes are resolved with simple choices and roleplay. I give out treasure at a much slower rate than the book suggests.
So I cut the cord at the currency end of things. However, the leveling is still there. That means the power level still changes and the players are rewarded with shiny new powers and new challenges to face. Those things are a strong pull for players, who inevitably want to see their characters' new powers in action.
My Saberpunk campaign is, as I expected, drifting back to Gamist play. I'd intended a more Sim game built around a cyberpunk mood and setting. That stuff is still there, but it's getting lost. Our sessions are moving back to the "one or two big combats" that I was trying to avoid. The players have actively voiced wanting those combats and I won't deny them the fun they want.
At the same time, 4E doesn't entirely do what I need it to do. The Sim style of play I want begs for some additional character skills (in this case, Perform for the bard, and an Espionage skill for the wizard -- though Insight will do in a pinch). We just hack those in with a house rule. I'm using the Obsidian Skill Challenge system, too, with great success. I use a Blood Points rule to reducing whiffing and make combats less deadly.
This is just my latest attempt in a handful of failed attempts to drift D&D away from Gamist play. Even older versions have a reward cycle much the same as 4E's. My 3.5E campaigns trended the same way, even with significant XP awards for playing to specific, player-written goals/beliefs. While the XP awards were significant enough to make killing stuff less important, the reward of leveling and getting new character abilities was more powerful, drawing players into combats just to see their higher-level character do his or her thing.
The game gives you an asskicking character, and players want to kick some ass.
What is 4E anyway?
How far can you bend (or ignore) the rules of a game until it's no longer that game? There's no one answer to this question, for sure, but my personal feeling is that you should be able to bring in an average 4E player off the street and, without telling him what you're playing, he should recognize the game.
4E is, at its core, about 50 pages of actual "framework" rules. Most of the rest is rules-by-exception stuff: character race and class lists, magic item lists, equipment lists, skill lists, ritual lists, and monster lists. Some of the rest is play advice. The core of the 50 framework pages is the reward cycle. That includes: rules for creating a character (minus race and class definitions), fighting monsters, handling skill challenges, awarding XP, and leveling up.
When you start ignoring the core stuff, you start drifting away from 4E. I replaced the leveling up system with my own system (which happens to be very similar to a variant in the DMG2, though). I replaced the skill challenge system with Obsidian. I tweaked some rules to make combats more fun (like Blood Points, and like letting characters make a skill check as a Move Action instead of a Standard Action).
However, if Joe 4E Player came into my house on alternating Thursday nights, he'd recognize what we were doing as 4E. It might annoy the crap out of him though, since I totally de-emphasize the getting-into-combat stuff, which is the bread and butter of a lot of 4E games.
When I hear people say that they run games where there are never combats and no one rolls any dice, I wonder why they still insist on calling that D&D, let alone D&D 4E. What about it is 4E? I could bring a GURPS Fantasy character, or a D&D 1st Edition character, or a Rifts character into that game and play, right?
If "playing 4E" means using all the rules as written, I'd say that it's nearly impossible to stop playing it in a Gamist way and still have fun. You'd have to build characters, fight monsters, overcome skill challenges, earn XP, level up, and not care about that reward cycle. Let's say you're interested more in reinforcing the cyberpunk-infused-D&D tropes of the Saberpunk world than kicking ass. You're still fighting monsters. You're still earning XP. You level up a few times, and now your character definition includes a bunch of new powers. To use them, you need to level up more. You don't want to die, either, so you start applying the best tactics you have. Maybe you choose powers that make better sense as a build option than a character-development option. You're headed back to Gamism.
But very few people play using all the rules. Say you're like most games and you have a handful of house rules and you ignore some other rules, like I do with Saberpunk. It's still easily recognized by the average 4E player as "4E" and not some other D&D edition or some other RPG. The rules-as-written even tell DMs it's okay to tweak things. So is playing "4E" the same as playing 4E? I don't know. If you have to modify or ignore the rules to get some kind of non-Gamist play out of the system, is it really fair to say you're playing 4E? It's a philosophical point, and I grant that it doesn't have an easy answer.
Can you house-rule 4E so that it supports non-Gamist play and is still recognizable to 4E players? Sure. I recommend starting everyone at a higher level, tossing out XP and leveling altogether, deemphasizing the combats, and focusing on the kind of play you want. But now that you've thrown out the core of what makes 4E a D&D game, why didn't you just use a different ruleset to begin with?
Best Answer
Metagame Rewards: The Different Kinds of Fun
The most useful classification scheme I've found is about the different ways that people find roleplaying games satisfying and rewarding to play. It's best expressed† in the article "Metagame Rewards, or the Different Kinds of Fun":
The article continues with a list describing 16 distinct kinds of fun that players can get from a session of a roleplaying game.
The 16 metagame rewards describe the underlying motivations for players' behaviours – a player motivated by agon is going to try to out-do the other players in a cooperative game and enjoys games where inter-player competition is allowed or encouraged, while a player motivated by catharsis is going to want play deeply-emotional stories that might not have anything to do with "succeeding". Any given player will have a few of these motivations, and the particular combination of metagame rewards a player finds interesting is a kind of gamer "fingerprint".
I found this classification scheme useful in three ways.
Knowing the motives of my players and myself as GM allowed me to diagnose why a particular group wasn't working well together. The problem was that most of them were highly motivated by sociability (the game is a social event) and paida (loose free-wheeling fun), while I was primarily motivated to run the game for the sake of kairosis (fulfilling story/character development) and kenosis (engagement with the fiction). This wasn't a gap I was prepared to bridge at the time, and the players weren't interested in playing the kind of game I wanted to run, so we split.
Knowing the preferences of my players (assuming the group is gelling in the first place), I can tune the style of play and focus of gameplay onto the elements that we all find most fulfilling. If nobody finds kinesis especially fulfilling, then I'd best not run a game that is miniature-heavy, and there's little point in my putting effort into props.
Not only can you classify players this way, you can equally well classify games, making it a very useful tool for matching a game system to a given group. I know now that the group I mentioned above doesn't like Burning Wheel, and figuring out later that there was a mismatch between the metagame rewards of the game and the group largely explained why that campaign failed (BW doesn't do paida well at all, for starters).
I found the metagame rewards taxonomy so useful that I made a metagame rewards survey PDF to print out and have everyone complete, in order to find out where group preferences overlap and what the "orphan" preferences (ones that are highly-rated by only one player) are in the group.
If there are many orphan preferences and few strong overlaps, then I know the group is going to need special effort if I want to keep it together for long. If there are lots of strong overlaps and few orphans, then I can safely ignore the types of gameplay that are outside the core preferences, and I can jump quickly to figuring out an scenario that emphasises that core.
The author of the original post on metagame rewards actually used my PDF survey with his group and blogged about the results, so you can read his own experience of applying this taxonomy to understanding a group:
He looks at how the survey explains why Paranoia fell flat for his group, why a particular Hunter: the Vigil group plays the way it does, and how different games naturally emphasise different metagame rewards via their setting and mechanics.
† I believe the concept of metagame rewards was first articulated by Levi Kornelson of Amagi Games, but that early form of the rewards list seems to have been lost when his site was attacked and brought down some years ago. It was mostly the same if I recall correctly, with a few of the rewards having different names and lacking one or two of the current 16.