To what extent is it permissible to use aspects of a given system (for commercial use)? Say I wanted to have rules for how a skill check works, do I have to be careful that I am not using the same as an existing system? Are names of things such as skills, feats, classes, and spells copyrighted?
[RPG] use existing game mechanics in the own designs
intellectual-propertypublishing
Related Solutions
(usual disclaimer that I'm not a professional lawyer and that you should consult one)
You're quite right to question the PRD as being 100% open content, as there are indeed some PI terms used within it. The PRD's Open Content declaration only specifies that rules included within are open (in similar wording to many printed products that state the same thing), at no point does it actually state the entire text of PRD is Open Content. In addition, PI falls under the implicit terms of the OGL as being protected in section 1(e) of the License. In that way, the PRD cannot be considered as 100% Open Content in the same way as the d20 SRD can.
Your best bet is to pick out those terms, and email Paizo directly to query whether it is okay to use them under the OGL.
One notable example is the Prestige Class "Pathfinder Chronicler", which is a specific term used within Paizo's Golarion setting background and not a generic name such as "Loremaster", "Wizard", or "Fighter", as well as containing the term "Pathfinder" which is protected as PI as the product name itself. In this example, you can create a prestige class in an OGL product using the rules of the Pathfinder Chronicler (or simply reproduce the class itself), but would have to call it something different. To borrow a little from Mxyzplk's answer, you could probably call it "PF Chronicler" ;)
Note that the advice I'm giving is based upon the interpretation of Paizo's OGC declaration, not the intent. It may well be that they intend all terms within the PRD to be Open, but they have not specifically said that in their declaration. Obtaining permission/clarification from Paizo is a simple matter of sending an email, so there's really no reason not to ask them to be 100% sure on the matter. Assuming it is okay because other products have used those terms (and they have) is not good practice - it only means you'll also be in violation if the OGL if they are.
Paizo contact emails can be found at http://paizo.com/paizo/about/contact - my guess is the Licensing Coordinator would be a good place to start asking.
You're somewhat confused, which is understandable since intellectual property rights and D&D is a confusing issue.
OGL
The OGL is a specific license with specific terms. D&D 3e/3.5e was made open for others' use under the OGL and the open portion was published as a SRD, or System Reference Document. Other games derived from the d20 SRD (like Pathfinder and Mutants and Masterminds) and totally unrelated games, like FATE, use the OGL. 4e used a non-open license called the GSL, and previous versions had no available user license except under contract with TSR. You can read the blog post Open Gaming for Dummies to understand these very specific legal terms.
There are some "retroclones" that use the OGL and actually pull the OGL 3.5e information from the d20 SRD into something that looks more like an older version of D&D. See also What content can I reproduce from Pathfinder? that has a similar discussion specific to Pathfinder. Some people use the OGL to put out modules for various other non-OGL D&D versions, but that's actually somewhat complex and you need to understand the rest of what's going on in IP land to know how and why.
Wizards has also released 5e - or at least some of it - under the OGL as well. You can download it and the SRD on their site. Be careful, the SRD only has a subset of the game's content, and it's all that can be used under the OGL.
Intellectual Property
Let's talk about using game IP in other games outside the scope of the OGL. The more general discussion in Can I use existing game mechanics in my own designs? has a lot of relevant details for you here, as well as Is it legal to "use" (reference) copyrighted material in a way that requires ownership of its original publication? You have three main areas of IP concern to contend with - copyright, trademark (including trade dress) and patents.
Copyright Basics
First of all, "fair use" does not apply in this case. At all. It's one of those phrases people like to use on the Internet but don't understand. If you up and use text from another work in your work, and it's not part of the very limited scope of fair use (educational, review, journalistic, etc.) it's illegal.
Secondly, game mechanics cannot be copyrighted. "Copyright does not protect the idea for a game, its name or title, or the method or methods for playing it. Nor does copyright protect any idea, system, method, device, or trademark material involved in developing, merchandising, or playing a game." (United States Copyright Office, "Copyright Registration of Games").
However, "the text matter describing the rules of the game" may be protected by copyright "if it contains a sufficient amount of literary or pictorial expression". For example, a passage describing character creation for the Clans of the Sun and Moon, explaining their society and why they tend to have the skills they do, would probably be protected by copyright. A section merely describing the steps involved in rolling a number of dice or expending a number of points on X attributes would probably not be. (rpglibrary.org) Note that this means even charts and tables can be copyrighted. Everything's copyrighted, there's not a registry or list or anything (well there is, but stuff not on it is still copyrighted).
In general, this is the escape clause some people use to put out supplements/modules that simply have "goblins" and "skeletons" in them which may have "HD: 2+2" or "a DC 12 Perception check" outside the scope of the OGL. If it's "mechanics" and not "expressive" that's legal. But head onward and read about trademark...
Trademark
However, various aspects of the game rules may also be trademarked. Several D&D monsters are, and in general things that would usually be Product Identity in an OGL game can be. Characters, game worlds, etc. Without a more specific license allowing it, you absolutely cannot set your module in the Forgotten Realms and have Elminster fighting a beholder in it. The reason many of the retroclones do use the OGL is so they don't have to fret over whether somewhat unique terms like "Hit Dice" end up getting trademarked or not, since as a term you're explicitly allowed to use it if you are using the OGL.
You will note that many products/publishers, out of fear of trademark won't say they are "compatible with Dungeons & Dragons Fifth Edition" but instead say circuitous things like "for 5e!" Normally, expressing compatibility with something is an OK use of trademark, Hasbro themselves lost a lawsuit on that v. RADGames about a company claiming Monopoly compatibility for an add-on pack. But the OGL giveth and the OGL taketh away, the OGL says you can't claim compatibility in its terms so those using it can't legally say "D&D".
Note that trade dress also means you can't use graphics/art/logos/etc that make your product look "too much like" products from an existing publisher.
You can do trademark searches to look for trademarks, see the discussion in Are the names of the more generic planes copyrighted under the OGL as Product Identity?
Patent
Game mechanics can be patented, since processes can be patented. I do not believe there is a patent on file for RPGs or D&D but I'm not an IP lawyer. You can do patent searches as well. At least one RPG publisher applied for a RPG patent, check it out here (article on it). Just wanted to mention this for completeness.
Dungeon Masters Guild
Speaking of more specific licenses, Wizards has come out with one! Their "Dungeon Masters Guild" program is designed to specifically allow publishing of 5e material with a number of restrictions, primarily that such content may only be distributed through their online storefront and they get a 50% cut of the sales (though you can make it free - although you can't distribute it other than through their site). The upsides are that (a) you can use the Forgotten Realms as a setting and (b) you can use all of5e rules/content, not just the subset provided in the SRD.
Bottom Line for Modules
"I am not an IP lawyer and this is not legal advice blah blah" but in general you have three paths open to you. All require a nontrivial amount of legal understanding, so taking a RPG.SE answer and going forth to publish would be the height of folly.
But the first is to use the OGL, limit yourself only to things found in the SRD, and don't claim D&D compatibility. Most folks do this, for example Frog God as you note. For an adventure, with a little oversight, you are probably fine with just writing general scenario content with pure rules references.
Or, you can not use the OGL, navigate the copyright/trademark waters yourself (and/or hope Hasbro doesn't care), and even claim compatibility. KenzerCo did this with some products; their owner, David Kenzer, is a lawyer so he knew what he was doing and figured he could hold his own. Since modules usually don't contain large amounts of rules content with the exception of stat blocks, the problem space is pretty narrow.
Now, profit vs it's a freebie on the Web doesn't change most of the legal factors, but it does change immensely the likelihood you'll get in trouble over it, so if you're not really looking to publish for sale, go ahead. If you're starting out and looking to publish for sale, it might be a good idea to do that through an established third party company (like FGG) that can help you navigate these waters.
Or, you can use the Dungeon Masters Guild program, if it suits your needs.
Best Answer
I am a game designer with my own game designs already on the market. My primary system at the moment is currently the number one RPG system on Amazon.com. I'm also a writer with over ten books to my name and dozens of shorter stories. I've read a lot of books with regards to copyright law and checked both the Canadian and U.S. laws concerning copyright, trademark, games, and patents. However, I am Canadian so that might be worth noting for slight changes in regulations.
I am not a lawyer and cannot offer legal advice, etc.
Essentially, I agree with @SevenSidedDie's answer. His research mirrors my own. I do have several points I'd like to bring up regarding my own experience in the field of writing and game design:
Less is More: If you have any doubts about whether something is a copyright infringement, it probably is. I'd steer clear of using exact names, turns of phrase, and anything else which makes it clear you're borrowing heavily from another work. However, game mechanics fall under a different set of regulations so you can probably get away with using a d20 or a system called 'skills'.
Don't Copy Names: I could be wrong about this, but I try to steer clear of any specific name wherever possible. I don't name powers in my book feats. I don't call the powers themselves stuff like Cleave and Great Cleave. In fact, I try to use unique powers wherever possible in the first place. My reasoning is that if everything I've created is 'new' a small slip up like using the abbreviation str once in the entire book will seem less like a copyright violation even though technically it might be.
Don't Copy Phrases: Try not to copy anything in any pattern of 3 words or more from any other work without correctly attributing that. This means no stealing tables, exact skill or power lists, standard D&D classes and so forth.
Use Common Sense: Most copyright laws follow fairly common sense grounds of 'reasonable interpretation'. This basically means that if push comes to shove the 'cumulative' effect of your work will be considered. If your book obviously borrows a lot from another source you could be in trouble. On the other hand, if it's clearly 99% original any slight slip ups will probably be taken as that; slip ups. Consider if someone was to copy your game system. How much copying would you want to allow before you considered someone else was stealing your work?
What you Can Do: The 'game' aspects of a game can be taken to some extent. I'd still recommend against taking a system's mechanics wholesale even if it's technically allowed for the number 4 reason above. I.e. yes, but everyone makes small slip ups and this could be what does you in. If you want to use the basic structure of D&D, you can probably change some descriptions and new names for classes/feats/skills and maybe get away with it. As a game designer myself, I'd personally recommend making your 'own' game anyway because it's usually cooler, more fun, and safer against copyright infringement.
Check out the Competition: Many other game designers have already faced your same questions. Posting here is a great move. Another really smart thing to do is check all the other 'new' RPGs out there across the web. You'll soon get a very good feel for what's allowed (most of it) and what's going too far (a few small games out there). Another good rule of thumb, is to make your game 'more' unique than the other games out there which perhaps steal a few too many mechanics. That way, if there's ever trouble you can point out 90% of the other games out there which already do far worse than you.
Personally, if you're serious about game design, I'd recommend building your own system from scratch. In the end it's more fun, more appealing to players, and won't get you in trouble for copyright infringement. Having a few mechanical similarities usually isn't a big problem, apart from that I'd recommend to be creative.
Thanks for listening. Hope this helps!