I think I put about 40 hours in to Fallout 3 on the PS3 (not the GOTY edition) - finished the main storyline, plus a bunch of side missions and random wandering. The game froze and required a hard reset of the PS3 about 3 or 4 times, and I got stuck on scenery once which required me to go back to my last save. So that's about one reload every 8 hours of play time.
While it's probably the buggiest PS3 game I've played, I wouldn't call it "unplayable", just annoying. I really enjoyed playing it and I think it's worth putting up with the bugs.
That's exactly the situation I was in. Yes, you can use Fallout 3 savegames (boxed) with Fallout 3 GOTY (steam) (if you had any mods installed in your original install, it will tell you "Some features of this savegame are no longer available. These features will be disabled." - my savegames still worked perfectly though)
I had to move the save files to a slightly different location, though. My original saves were stored in
C:\Users\BlueRaja\Documents\My Games\Fallout3\Saves\
(The XP-equivalent of that would be C:\Documents and Settings\BlueRaja\My Documents\My Games\Fallout3\Saves\
)
My new saves were stored in (and thus my old saves had to be moved to):
C:\Users\BlueRaja\Documents\My Games\Fallout3\Saves\BlueRaja\
I would recommend installing and starting the new copy and, if it doesn't recognize your old saves, create a new game and save it real quick to let it create the save-directory, then move your save files to the new directory.
Best Answer
According to Fallout 3's website, It will work fine.
Will my existing Fallout 3 saves be compatible with Fallout 3 Game of the Year Edition?