For a player

Three or four sets will cover almost everything you could ever want. This will mean you have at least 2d20’s for advantage/disadvantage, at least 3d6’s to roll for abilities and a handful of other dice for rolling damage or hit points. 4d6 is preferable as it means you can drop the lowest each time.

Different classes have different hit die or some add certain dice to each of their attacks at higher levels. Different weapons and spells also use different types of dice for damage, such as a d4 for daggers. Rolling a d6 six times every time you wanted to attack something would get extremely tedious very quickly. The more dice you have on hand, the faster you can get rolls done which keeps the game flowing.

My players most often use d4s or d6s for damage dice - most have a bag each. If they need more dice, such as 8d6 for a spell, I have a stockpile of different dice I can lend to them for the session. My girlfriend has three or four bags of dice (mostly because, being an artist, she likes how the colours compliment each other) which are usually enough for most situations. As she plays a bard though, there are times when she needs to hand out bardic inspiration dice which can mean she doesn’t have enough dice to roll on her own turn. Rather than risk slowing down play each turn, i’d just give her the right amount and type of dice she’d need at the start of the session to distribute for bardic inspiration.

As a DM, I would not expect a player to have every single type of dice and the correct quantity they’d ever need (much like I wouldn't expect every player to have their own minis or their own props or even their own books). Its one of the reasons why I have so many dice to begin with - so they can be distributed out as needed (the other being you get more for your money by buying in bulk rather than in sets of 7s).

For a DM

Many sets of dice are extremely useful, some traps, like those found in Xanathar’s Guide to Everything, require 24d10’s! Also, if you run encounters with large hordes of creatures, rolling a fist full of dice is so much easier and quicker than rolling for each creature individually. Having a large stockpile of dice is also nice as it means every player has their own dice to use.

Personally, I have somewhere in the region of 400+ dice, mostly d6’s and d20’s but theres a fair amount of every dice. I simply bought few bags of 100+ dice which weighed about a pound each, my dice-bag bag weighs 1.5kg (which I find to be amusing as 1.5kg is about the same weight as your average medieval longsword). Of these 400+ dice, I only regularly use about 15 to 20 as a DM, 2d20 for advantage and disadvantage, a handful of d6, a d10 and a percentile for rolling on tables and then a couple of d4s, d8s and d12s for damage die.

I also have some non-standard dice I use, such as a d6 with a skull on the 6 for determining wandering monsters (normally you’d use a d20 but skulls are cool and it fits thematically); a compass die to determine which way the wind blows (useful information for sailing, archery, flying characters or when a scent or poisonous gas is being blow towards the party or another creature) or where a creature, object or location is relative to the party (useful for seeing where creatures are in the wilderness or an ambush, determining where a disarmed weapon goes, where a creature is after randomly teleporting or seeing which way the part has to go to get back on track after getting lost); and d3s for if a homebrew weapon or attack deals less damage than a d4 but more than 1. However, just to be clear, none of these dice are needed, you could use a normal d6 for wandering monsters, a compass dice can be recreated by using a normal d8 and rolling on a table of compass directions and a d3 can be recreated by rolling a d6, dividing the result by 2 and rounding to the nearest whole number. I simply use these dice because I got them one of my 100+ bags after asking for some “unusual dice” and wanted to find a use for them.

It is often far cheaper to buy dice in bulk quantities. I often see a bag of seven dice priced at the same amount, if not more, as a bag of 100. I will inform you that these “pound of dice” or “100+ dice” bags primarily consist of factory 2nds. This simply means that, whilst they are perfectly serviceable, they have minor visual imperfections. From my experience, the vast majority of my dice are perfectly fine and I do not see why they would be considered 2nds, you get the exact same types and styles of dice you may find in stores or online. One issue I found when I bought a 100+ bag was that it was hard to make sets, simply because i wanted them to be organised by colour or style. However, this issue was easily overcome by buying 2 more 100+ bags :P