How and why has the concept of the "rogue" evolved over D&D's history?
It seems that what the general perception of what people think of with a "rogue" or "thief" has changed drastically over the years. In some eras, they are the sort that is weak – almost defenseless, even – in combat, and do their best to avoid combat while focusing on other elements such as pickpocketing and hiding. In others, they are considered "Combat Strikers" and are actually better at dealing damage than fighters, capable of dealing precision strikes casually in melee, and with their more thief-like skills becoming more of a sidenote (considered "skill-monkey" abilities, as opposed to be their core defining traits). From what I can tell, this seems to be a gradual shift over time.
While this would be nothing special if characters in any system could be customized to be of either style – as is usually claimed by systems – the fact is that most editions of D&D force characters at least partially to one direction over the other. 3rd Edition rogues automatically have Sneak Attack, which is highly damaging, scales well and automatically, and can be triggered merely by flanking (read: ganging up on in normal combat) enemies, cementing them as warrior-rogues. While they can also be, or not be, other types of rogues based on their skill selections (social skills, larceny skills, arcobatic skills), they invariably have strong combat abilities. Pathfinder further enforces this by increasing their hp to d8, something that wouldn't necessarily fit a rogue that avoids fights and doesn't know what it means to even take a hit.
4th Edition rogues are flat-out considered called Strikers and are designed as fragile speedsters or glass cannons. Meanwhile, older versions of D&D appear to treat rogues more like the burglar sort, perhaps based off of Bilbo Baggins (as much of D&D was originally based on LotR), who did little stabbing. Furthermore, some different game systems truly do allow for playing different types (Anima: Beyond Fantasy has three different classes primarily based on being a rogue: Shadow (fighty rogue), Assassin (social rogue), and Thief (stealy rogue)). From what I can tell, however, this type of freedom is rare in D&D proper.
Has the cultural perception of rogues changed over the years? What is the timeline of the evolution of the rogue? How has it changed over different games (i.e. old D&D vs modern D&D) and over the years (80s, 90s, 00s, 10s)?
Best Answer
The Rogue, a bowdlerized Thief, was always an Adventurer / Treasure Hunter
For a swords and sorcery genre fictional or legendary root, you could look at the Grey Mouser (from Fritz Lieber's fiction), Cugel the Clever (Jack Vance), the Thief of Baghdad, or Ali Baba. Thieves guilds became a permanent feature of the game once the class was introduced. If Indiana Jones had a D&D class, he'd be rogue or thief. A reasonable argument has been made that the thief (later rogue) was purpose built to be at home in the dungeon.
Origins
The original published game had neither Thief nor Rogue. Supplement 1 (Greyhawk, 1975) introduced the Thief. With d4 Hit dice (like a Magic User), a Thief could not go toe-to-toe with anyone (unlike the Fighting Man). The Thief got the +dex benefit for ranged weapons, and had the damage multiples for attacks from behind by surprise (+4 to attack, 2x, 3x, etc damage depending on level).
You had to play a thief cleverly to survive, since your HP were low. The key role, the unique addition to the party that this class introduced was what later became the "skill monkey" due to the climb walls, pick pockets, hide, move silently, hear noises, find traps, read languages, and pick locks skills. Use of magic items and scrolls at later levels were a direct Grey Mouser influence. (This later became Use Magic Device(UMD) and is still part of the 5e Thief Archetype). Another boon was that hobbits (later halflings) and other non-human races were not limited by level in the Thief class. (Greyhawk, p. 4). Hobbits were limited to 4th level Fighting Man otherwise, for example.
Supplement 2(Blackmoor) introduced The Assassin, a sub-class of Thieves whose role was specific: hired killer and spy. This was a step toward that glass cannon(non-spell caster). Interestingly, the Monk (a lawful cleric sub-class) got some of the Thief skills and features. The Assassin took the "back stab" of a thief and went one better: this lethal damage dealer struck from behind, the dark, the flank ... anywhere that you weren't looking.
Alignment (which at one time mattered) was restricted: Thieves were chaotic or neutral in the old system, and even in 1e were (by general rule) neither Good nor Lawful (though some DM's relaxed that). Assassins had stricter race and alignment rules in OD&D:
The Thief allowed non-human PC's a way around the level cap from the original game. That boosted the popularity of the class.
AD&D 1e Thieves were a marked improvement
The Thief rose to 1d6 HD in AD&D 1e and the class came into its own. As with Greyhawk, bypassing the non-human level cap was part of its popularity (Thief was a popular multi-class option for non-humans at the tables where I played, and each non-human class got different thief skill bonuses in the PHB (p. 28)). The Thief was a prerequisite class for the game's first effort at a prestige class: the Bard. You had to gain levels in other classes (Thief and Fighting Man) just to play Bard in that edition, if the DM allowed it. This set up the linkage in the next edition for the Bard as a sub-class of Rogue at character creation.
The Assassin sub-class was reclassified as "evil" (which I believe contributed to its removal in AD&D 2e) and racial restrictions were removed.
The final growth spurt came with the Thief Acrobat in the AD&D 1e Unearthed Arcana, which emphasized the 'cat burglar' and acrobatic abilities that later looked like feats.
AD&D 2e: Rogues, Warriors, Wizards, no more Demons; D&D Bowdlerized
After a decade of game growth, the social backlash to D&D influenced the scrub and reorganization that the AD&D game got. The Rogue replaced the Thief as the name of the class, with sub-classes of Thief and Bard. The nasty criminal connotation of "Thief" was de-emphasized. The Assassin sub-class was retired as a PC, although it remained as a hireling or NPC. The sales pitch was that adventurers in high fantasy were more like Bilbo, an "honorable" burglar, or Ali Baba, or "The Thief of Baghdad" of the old movie classics.
Kits arrived for all of the classes, and customization became a feature to a greater extent than previously. The case was made for this adventurer's alignment to be other than evil / dark / chaotic. This change was consistent with how the company (TSR) was trying to overcome negative publicity from the hysterical '80s. (I won't beat that to death, as we have other answers on that topic, but that's a root of why the class is now called Rogue). The Rogue benefited from Skills, both proficient and non-proficient. The Ninja sub-class showed up as an optional Rogue choice, by way of Oriental Adventures. An Assassin kit was added (rather, restored) in the Complete Thief's Handbook (after some of the anti-D&D hysteria had died down ...).
3.x edition.
The Rogue as you now recognize it arrived, combining the skill monkey feature with the hit-and-run skirmisher / glass cannon melee damage dealer. Find a flank and abuse it; use Uncanny Dodge or Evasion to survive combat. The game became more complex, but there were sufficient choices available to make Rogues interesting. The Assassin returned as a prestige class, the Bard graduated to being its own class. You could emphasize damage dealing, social skills, adventuring skills, and of course multi-classing. Racial class limits were gone, but that had never been a problem for Thieves/Rogues in the first place. Granted, as the Tier framework was recognized and popularized, the Rogue didn't fare well (usually found in Tier 3 or 4) but the class was still playable even if it was hardly a match for a martial class in melee without flankers / flanking.
As far as class perception, you could trip over Chaotic Good rogue / thief characters if you went to a guild meeting. A well known rogue in a popular D&D webcomic is described as CG by her creator.
4e Glass Cannon Strikers
4e's tactical scheme found a good niche for the Rogue: damage dealer / striker who was not that well suited for toe-to-toe fighting. This was consistent with the original scheme in OD&D, but had considerably more depth to it. Skills and non-combat adventuring skills / utility remained, as they had been since the introduction of the thief in OD&D.
5e The Bounded Accuracy Rogue
The Rogue is still the skill monkey, is still able to pursue the "use magic item" path, has amped up the spell casting potential via the Arcane Trickster (thanks to 3.x for that prestige class), and can still go for either skirmisher or "pure" assassin with the Assassin archetype. Like the Pathfinder rogue, the HD are d8, so the option to be something like a Striker or Skirmisher remains. The Rogue Thief archetype retains a lot of the feel of the original Thief, but with less crunch and more HP. With the Mastermind archetype (in both XGtE and SCAG) the Rogue class has finally set up a career path for its Guild Masters, though the similarity to Ivy League MBA's is probably a reach. evil grin
One thing that the current edition has done, though, is to make the surprise attack harder to set up. That said, when the surprise attack can be achieved, the Assassin archetype deals significant burst damage, particularly when attacking with advantage. It comes across as less of a Striker and more a "pure" assassin in the mode of the original Assassin from Blackmoor and AD&D 1e due to how surprise rules in combat are now built.
The Adventurer and Treasure Hunter who uses wit and skill
The Rogue is in a pretty good place, being able to fully enjoy all three pillars of a D&D adventure: Exploration, Social Interaction, and Combat. Even though the details of the game have changed, the Rogue is still the skill monkey who is at home in any adventuring scenario: city (wall climbing, sleight of hand, lock picking), dungeon or wilderness (stealth/hiding, climbing, and exploration), at sea, etc. The Rogue has something to offer any adventuring party. That said, in the CharOp discussions that I've been involved in, the Rogue's cousin class (Bard) is very versatile in 5e, can go nuts with skills, and has a substantial advantage in magic use under the Lore Bard archetype as an Arcane Caster. In some ways, the Bard is a hard rival to the Rogue for "skill monkey" on a small team looking for that role.
Full disclosure: I've played more thief / rogue / bard / assassin than all other classes combined since I started this game in 1975, and as such am biased in favor of this class. I consider the Thief, and now the Rogue, to be the ultimate "pure adventurer and treasure hunter." No, that's not your pouch of coins in my hand, why do you ask?