You attack with a flaming long sword non-lethally taking the -4 penalty to hit, and succeed in damaging the target.
The target has no damage reduction, so you definitely (for the sake of argument) deal 1d8 non-lethal damage. And…. any more? Is the 2d6 fire ignored, dealt as non-lethal (pretty ridiculous) or dealt lethally?
The only mention I can see about this kind of interaction is under descriptions for sneak attack which says (text from the rogue class shown here):
…She cannot use a weapon that deals lethal damage to deal nonlethal damage in
a sneak attack, not even with the usual –4 penalty.
Which in itself I find kind of unclear. Is attacking non-lethally against a flatfooted opponent impossible for a rogue? Or does it just mean that if you want to attack non-lethally you cannot add sneak attack dice? (Probably the latter)
To clarify: The core question here is whether magic weapons that add damage also have their extra damage converted to non-lethal damage when used for a non-lethal attack. The section on rogue sneak attacks is merely an example of the apparently minimal rules directly addressing this topic.
Best Answer
A rogue employing a weapon that normally deals lethal damage can only deal lethal sneak attack damage
Normally, with a melee weapon that deals lethal damage, any creature can opt to suffer a –4 penalty on attack rolls with that weapon to deal instead nonlethal damage, but a rogue "cannot use a weapon that deals lethal damage to deal nonlethal damage in a sneak attack, not even with the usual –4 penalty."
Thus a rogue employing a +1 flaming longsword (and suffering a –4 penalty on attack rolls for not being proficient with it) is perfectly capable of dealing (lethal) sneak attack damage with the weapon but is forbidden from and incapable of dealing nonlethal sneak attack damage with the weapon.
The rogue can still opt to suffer a –4 penalty on attack rolls with the +1 flaming longsword to deal with the weapon nonlethal damage, but the rogue would not deal sneak attack damage with the attack. (That is, a rogue "can strike a vital spot for extra damage" to deal sneak attack damage but doesn't have to!)
Were the rogue to opt to deal nonlethal damage with the +1 flaming longsword, all of the weapon's damage would be nonlethal because the rules actually do say, "You can use a melee weapon that deals lethal damage to deal nonlethal damage instead, but you take a –4 penalty on your attack roll," and no exceptions are made for a melee weapon that also deals energy damage!
(A liberal application of doublethink, shouting Magic!, slowly counting to 10, or slamming the
FAQbutton on later posts in this thread (the first question not actually answered in any FAQ) may help overcome the weirdness inherent in the idea of nonlethal fire damage, or just take comfort in the fact that there's sort of a precedent for it.)This answer's author would like to point out that this is an answer to a rules-as-written question, and, as such, things can get a little silly. Ask the GM how this works at his table.
(Note also that a creature's damage reduction needn't be overcome by a flaming weapon's normal damage for the weapon to deal its fire damage. The fire damage—lethal or nonlethal—is applied separately, for example, against any fire resistance the victim may possess.)