This answer doesn’t attempt to define “viable,” instead focusing on comparing the Pathfinder rogue to the 3.5e version, since the question was asked on the basis of “traumatic experiences” with the 3.5e version.
Also, this answer focuses primarily on the original rogue, not the unchained version, since this question and answer predate Pathfinder Unchained. The mere fact that the rogue needed to be unchained should tell you what you need to know about the original. However, the unchaining was very effective, so that version of the rogue is definitely far stronger than the 3.5e rogue.
Fewer Creatures Are Immune to Sneak Attack
Constructs, plants, and (corporeal) undead are no longer automatically immune to sneak attack. That’s a very big deal, since constructs and undead are such common enemies. Elementals and oozes remain immune, as are the new proteans, but those are often not as common. Incorporeal undead are immune unless you have ghost touch (or similar, presumably).
Notably, the 3.5 rogue could sneak attack these things, but it was kind of a pain to do so, requiring particular wands.
It’s Harder to Trigger Sneak Attack
Recent errata to the Stealth rules finally allow those to be used to get a sneak attack (cf. this answer), or multiple with hide in plain sight (which the rogue can now get from the rogue class, though it’s pretty limited), but aside from that most abilities that a rogue had to trigger multiple sneak attacks per round solo are gone. Most of the time grease doesn’t work (and the cases in which it does work are very ambiguously-described), blinking doesn’t work (and even if it did, Pierce Magical Concealment doesn’t exist any more), alchemical weapons can no longer be used for sneak attacks, and so on.
Fortification still exists
The fortification armor property still means a ton of high-level enemies are going to be immune, or have a chance of ignoring, your sneak attack.
Rogues still cannot stand and fight, and optimal damage is still low
Despite how hard it is to actually get full sneak attack damage on the rogue, since they can’t just stand there flanking someone and dishing it out most of the time, that damage is still lackluster. A typical barbarian or fighter will out-damage a typical rogue every time. This isn’t so bad (after all, the rogue has all those skills to play with, particularly Use Magic Device), but where before the rogue could get his Sneak Attack and remain relevant despite the superiority of the straight-martial types, a lot of the time in Pathfinder he cannot get Sneak Attack and therefore does barely anything at all.
The general consensus among optimizers is the rogue was stealth-nerfed
Sneak attack was improved, but the various subtle ways in which melee was shafted in Pathfinder, combined with the various subtle ways rogues in particular were nerfed (the sneak attack nerfs above combined with the watering down of trapfinding, which is almost pointless with the way so many things have detect magic at-will in Pathfinder), have resulted in a more difficult time for rogues trying to carve out a niche for themselves.
Pathfinder has numerous outright replacements for the rogue
The ninja and the vigilante are more-or-less strictly superior to the rogue, able to do everything the rogue does, either better or along with other things. Plenty of other classes are less direct replacements, but can still stand in for a rogue very well.
The unchained rogue is very strong, however
The above all applies solely to the original rogue; Pathfinder Unchained overhauled the rogue, and that version of the rogue is quite strong. It’s one of the best straight damage-dealers in the game, and it still has lots of skills and trickery. The unchained eldritch scoundrel is very competitive for the title of best non-full-caster class in Pathfinder.
Best Answer
From Pathfinder Core Rulebook, p136 (emphasis mine):
1) Rays (not other ranged spells, just rays) count as ranged weapons for the purposes of feats like
Weapon Focus
and for the purposes of firing into melee. You do take the penalty when firing into melee with a ray, but you can also take thePrecise Shot
feat to remove this penalty.2) By the rules, yes; however, I think a strong case can be made to the DM that you shouldn't take the penalty, because you're not firing into a melee, you're already inside the melee. It's probably not a good way to avoid the penalty, though, since you'll be provoking attacks of opportunity.
3) No, you would not take the penalty if you're the only one in melee with the enemy. The penalty applies when you attack "a target engaged in melee with a friendly character". Madness & clones aside, you are not your own friend or ally, so if it's just you then there are no friendly characters in the melee.