[RPG] Is it legal to design and publish a game with D&D’s ability scores

OK, so here's game intellectual property 101. There are niche exceptions to all of it, but at a high level it's going to hold for 99% of use cases in the free world.

If you don't already know all the stuff below you should not enter into any commercial enterprise based on someone else's IP without professional legal advice.

1. You are never free to use someone's intellectual property "just because." Unless you have permission/a license to the contrary, use of this sort is well covered by IP (copyright, trademark/trade dress, patent, et al.) law and is illegal. Even if you're not charging for it.

2. In some cases you can use an RPG's rules in your product, if they have a license that says you can. For example, D&D 3e/3.5/PF and original FATE use the Open Gaming License to allow people to use those rules under the conditions of that contract. Some games have "open" licenses like this (you can use them without specifically signing a contract with the company), others have "closed" (you call them up and negotiate). In either case you are bound by the specific terms of the license and/or relevant IP law. (Note that the terms of the license can override rights you'd have under the law. Mayfair was successfully sued by TSR for claiming compatibility with AD&D, not because that claim is illegal per se, but because they'd signed another agreement with TSR that forbade it). Anyway, specific to your case, the OGL is only about the game rules (the things designated as Open Content, e.g. in the SRD), NOT about the fluff and fiction.

3. In more limited cases you can use game rules without a license, as game mechanics can not be copyrighted. Their exact expression can, and various proper-name terms can be trademarked or copyrighted, so you need a decent degree of IP savvy to do this. Patents could block this but as far as I know no RPG rules have been patented.

4. Trade dress law can be used to block use of unique iconography and formatting (like say the Star Wars: Edge of the Empire dice sigils). You also can't format your product to "look like" an existing RPG's product for the same reason.

5. In some cases, more rarely, you can use setting (or adventure, or novel) material, if they offer an open or closed license that says you can. Except for some folks like Eclipse Phase who use a Creative Commons license, an open license is rare and a closed one is expensive and you need to "be someone," not a random fanfic author, to get one. WotC specifically does not have any open licensing for their fluff/fiction under any D&D edition. So ToEE, Greyhawk, etc. are their IP and they're not sharing. You may not use them for your own published works.

6. Anything created based on a copyrighted work is called a 'derivative work' and is covered by the copyright of the initial work. So you may not create and publish a story set in the Temple of Elemental Evil without a specific license from WotC. Even specific characters have been found to be covered by copyright (in addition to trademark and trade dress, which is why you can't use Mickey Mouse). Using "Drizzle the dark elf with two tulwars and a pet black tiger" is probably protected (based on how much everyone pays their lawyers).

7. If you are doing so just as a fanfic or something on your Web site, it is certainly possible that you can do it and they won't do anything about it. WotC has done cease and desists to fan sites but that's usually people making spell cards or otherwise doing things Wizards thinks they should get the money for. However, legally, they certainly could order you to take it down via various means (C&D, DMCA takedown to your ISP, lawsuit). They tend to want people to make content on their websites to spread their product, without ever saying people are allowed to so they retain the legal right to stop whoever they want.

8. If you are trying to profit from a derivative work, however, you can expect to get hammered legally. You won't get that far because publishers etc. won't touch you if you're doing this of course.

9. As for "if I change it enough so it is not ToEE..." There is no hard and fast answer. Transformations of a copyrighted work fall under copyright and it's not legal to do without license, but if you transform it enough that no one knows where it came from (or at least it couldn't be proven in court) it would work. However, having asked this question in a public forum, too late! You are looking at Exhibit A for the plaintiff. Plus, I assume the reason to set your book in the ToEE is to leverage knowledge/popularity of the ToEE, and filing the serial numbers off too hard leaves you with little of that relationship.

TL;DR - Can I Write A Novel About The ToEE?

Sure, you can if you're doing it for fun on your Web site; no never if you are looking to publish it.

Angry GM (theangrygm.com) wrote a comprehensive article on the subject here. You'll learn a lot about Open Game License (OGL), System Reference Document (SRD) and other copyright considerations there. However, be warned that his writing, while full of good advice and wisdom, is liberally sprinkled with rude and vulgar language (I'm borrowing nitsua60's words here).

Basically, you have to choose between two models:

1. Limit yourself to the content given in the SRD (with all its limitations). You're free to create completely new material, but you can't reuse or reproduce anything defined as "Product Identity" per the terms of the Open Gaming License.
2. Sell your work through the Dungeon Masters Guild. Be aware, though, that if you choose this latter model, you won't be allowed to distribute it (even for free) anywhere else, and that includes any copy you would want to give as rewards to your contributors.