A little bit of background: We are a new DnD group, living in a country where DnD is obscure, at best (the Philippines). We have no prior background to any kind of RPGs except in video games and MMOs. And we don’t have the benefit of having a mentor DM, as suggested for new groups.
Yesterday, we had our [9th? -I lost count] session as a group in a span of about 5 months and I still get the same questions we had on our first or second session:
“What was my hit die again?”
“How much was my attack roll bonus?”
“My Dex modifier?… 15?”
“A skill check? What do I roll for that?"
You get the idea, it’s frustrating. I could live with that for two or three sessions but now that the majority of the group has already gotten to 3rd-level, it’s just astounding that they still don’t know the basics.
To top it off, most of them still haven’t used their characters to its potential:
We have a 3rd level Bard who has only ever used Bardic Inspiration once in 9 sessions, never remembers to play a Song of Rest except when I remind him to, and even forgot his college.
We have a cleric who never casts spells unless I suggest that he cast one, he just loves to smash things with his mace AND has never used Channel Divinity even though there were multiple opportunities to do so.
We have a paladin who keeps looking at his spellbook each turn to see if he can cast something good, wasting everyone’s time as he contemplates (even though I repeatedly told him to prepare what he wants to do on his turn before every round).
Meanwhile, we have a Battlemaster Fighter who is a god-send; he is our newest member but he keeps index cards of his maneuvers and always knows what his bonuses are. He does 16 damage with his Heavy Crossbow with a goading attack, successfully taunts the big bad Owlbear, and then action surges and shoots again for another 9 damage. The rest of the group just gawks at him when they could have been doing the same with their character’s potential.
How do I, as a new DM myself, encourage or punish my players to do their homework on their characters abilities, features, spells and general game mechanics?
I don't know if the problem is the lack time to read; our games are held at regularly long intervals so there’s plenty of time to invest in reading the PHB. And it’s not like they’re not interested in playing: we bought four copies of the Player’s Handbook over the internet and even paid additional shipping and had to [nudge] a few customs guys to get it here because it is next to impossible to find in this country.
I tried to talk to the group about it and they say that they just forget, then the tone of the conversation turns sour so I steer it at another direction lest there be lasting effects.
1. Get a cheatsheet into each player's hands.
You know that godsend player, the one who always has the notecards? Key thing there: the notecards.
You've spoken to the group, and they got upset, but you know they cared enough to get the books in the first place. It's entirely possible that they do just forget, or maybe they're having a difficult time with the rules and don't know how to articulate it. Even if they read cover to cover (and maybe they already have), there's a lot of information to parse and remember, especially for new players. Even veterans forget rules sometimes! Still, fumbling through the book for basic rules is not time-effective. My solution is to create a cheatsheet. This has helped me as both GM and player.
Figure out your most commonly used rules. Type up these rules and print a copy for each player. Try to condense them as much as you can. Strip them to their bare bones so players can find the information they need at a glance. Also, since you'll be distributing the same page to everyone, concentrate on rules that everyone uses. I'll discuss character-specific rules later on.
It's just more practical for you to make the cheatsheet. If they don't have the rules down yet, they probably don't know which ones are the most important. You likely have a better grasp, and you're also the one who wants more efficiency. The best way to ensure that this happens is to do it yourself, and then the whole group will be on the same page.
For character-specific rules, though, it's best if the players do it. This spreads the effort around. Plus, you can use this as an opportunity to help re-familiarize your players with their mechanics. Sit down with them and help them make index cards or type notes from the book. Point out nifty features and answer any questions.
Assure them that it will make their lives way easier, because it will. They probably don't like book-fumbling either. They'll also be more likely to use their character's features if they're always right in front of them in an easily-digestible format. It's hard to achieve your full potential if you don't know where to find it. The key thing is to emphasize how useful it will be for them (in character and out) if they do this, not how much worse things will be if they don't. Make it about them and their potential fun, not your frustration, to avoid further souring.
It's awesome that the Battlemaster Fighter took the initiative, though. If this is something he enjoys doing, you could enlist his help! I'm this player in my own groups, even when I'm not GMing. I love having a cheat sheet, so if I'm making one anyway, why not share? At the least, use his notes as an example (if he's okay with it).
2. Make use of existing resources.
Cheatsheets you make will be the best-tailored to your needs. That doesn't mean you shouldn't use ready-made tools on top of that. People often share their own tools if they think they're especially useful. If you need something specific, search the web to see if it already exists. There's no guarantee that what you find will be useful -- again, you know your own needs best -- but it never hurts to try. If you hit gold, that's time and effort saved.
For instance, here are some form-fillable initiative cards that provide a quick reference of initiative order and PC skills, traits, and actions; here is a larger, longer version of a similar concept. Here's a site that generates spell cards. ENworld has many sheets available. It only took me a few minutes to locate them by searching "5e spell cards," "5e initiative cards," and "5e character sheets," respectively. You'll be more successful if you already know what you need, but you can still try searches like "5e game resources" and "5e player print outs" if you're looking for tools in general.