[RPG] Does a phylactery keep a devil from getting a soul

No

The simple answer is that destroying the phylactery does not destroy the soul contained inside it, because no game rule in D&D 5th edition asserts that it would.

The lore in the Monster Manual, p.203 suggests that the phylactery is not necessary to protect the soul from injury, but rather to prevent it from passing to the afterlife:

A lich is created by an arcane ritual that traps the wizard's soul within a phylactery. Doing so binds the soul to the mortal world, preventing it from traveling to the Outer Planes after death.

Destroying the phylactery doesn't kill the lich, but only makes it possible to kill the lich, since with the phylactery intact, a slain lich simply re-constitutes at their phylactery:

When a lich's body is broken by accident or assault [...] within days , a new body reforms next to the lich's phylactery [...] Because the destruction of its phylactery means the possibility of eternal death, a lich usually keps its phylactery in a hidden, well-guarded location.

What exactly happens to the soul when the phylactery is destroyed is a mystery (see my answer to Can a Lich create a new phylactery?).

Welcome to Hell!

Actual lore and citations below

I want to preface that everything here is what I learned from reading Mordenkainen's Tome of Foes (MToF) and the Monster Manual (MM). I definitely recommend it as the story of the Blood War is absolutely fascinating plus you get to learn about how it could be extremely good or extremely bad for you if a True Neutral archmage like Mordenkainen just shows up one day. Everything here is what's canon for 5e, I can't speak to earlier editions.

tl;dr Every soul starts as a Lemure which is the only type of devil that can die in the Nine Hells and reform. Every other soul becomes a ranked devil of some sort, and with that increased power carries the threat of utter annihilation should they die in the Nine Hells.

Please keep your hands and legs inside the ride at all times! I'm hoping that's an easy rule to follow since you barely have any, because for your first form here in the Nine Hells is going to be a nearly worthless Lemure.

My name is Chad, I'm the horned devil tasked with collecting you. Come along, we're heading to Mammon's Emporium to see just how worthless you really are.

Chad pulls out a scale with a feather on one end and unceremoniously drops you on the other.

Let's see, seems you're about tree-fitty soul bits. That's good news for you! Were you less than three, we just toss you into that bag over there that turns you into a Nupperibo and trust me nobody's going to want to spend any time with you after that.

It seems that you're just on the threshold that none of the other Archdevils want to buy you, so I guess that means you default to Mammon. Say 'Yay Mammon', please!

You say, "Yay Mammon".

Perfect! With that unholy oath, you are damned to serve him for all eternity!

Now typically, most worthless souls like yourself have to earn yourself a reputation in the Blood War before you get promoted. However, I'm thinking to make an exception because I want to turn you into an Imp so that you can spy on the Erinyes, Julia, and see if she's any closer to taking me back.

Now you might think this sort of tactic is gross and weird, but trust me that I've been over the various restraining orders and they are very clear that I need to stay 500' away but there's a glaring omission regarding my spies.

Anyways, hold still this hurts a bunch.

Your form is ripped to pieces and reformed into an Imp.

After several hundred years of various stalkings and the like, you are promoted through the ranks from Imp all the way to Bone Devil as you continue to excel at stalking Julia. Following this promotion, you almost immediately find Julia. Almost like she wanted you to find her.

Finally, he's gone ahead and promoted you right under him. Such a simpleton he doesn't realize this was my plan all along. Listen, were you to make Mammon aware that Chad had inappropriately promoted you from Imp to Bearded Devil 377 years ago without ensuring an appropriate merit rating of 240.2 or greater, he would be subject to demotion. As you are his direct subordinate, the infernal code would obligate Mammon to promote you from a Lesser Devil to Greater Devil and you would take his role.

You do that.

Two days later, Chad receives a letter via courier informing him that you have reported his violation of infernal code 66.33.311.8, subsection 122-5, paragraph 88.333, sentence Jeremy Bearimy, word 5. Almost immediately thereafter comes a form that has been stamped in triplicate which are the forms to have Chad carried away because you are technically correct, the best kind of correct.

In the next minute, you receive your promotion paperwork and key to your new office. Right after, an Imp from HR comes in to explain things.

Mr. New Chad, by decree of Mammon you are hereby promoted to Horned Devil pursuant to infernal code 333.666.999, Standards of Promotion by Default. Congratulations. You are hereby tasked with securing Mammon no less than 3,455 soul bits per lunar cycle. Failure will not be tolerated. If you have any questions, too bad.

You end up rocking this position by shrewdly selling junk souls on the bonds market and bailing out right before they crash screwing over both Levistus and Mephistopheles.

You end up being promoted to an Amnizu serving Mammon directly. Due to your absurdly broad knowledge of the infernal laws and contracts, you become aware that Mammon cannot legally ask for aid if a black cat stands atop a ladder within 400 feet of a the Chili's on 3rd Street on the blood moon while it rains.

So you orchestrate an insanely absurd series of events under his nose to cause this series of events to happen exactly when he's choking on a croissant.

Hear my telepathic pleas for help loyal servant! A demonic croissant has lodged itself in my throat!

And you merely respond, feigning sad resignation:

A thousand pardons my lord, but the law is the law and you know the rules regarding the black cat on the ladder in front of Chili's the and so on.

He dies.

Asmodeus, impressed by your scheming promotes you to Archdevil of that layer.

Two months later, you slip in the tub and die. Your soul destroyed. Maybe next time around you don't run that layer like Mammon and actually spend some money on repairs and safety features. The lack of handrails is certainly ominous, but definitely an OSHA violation.

Because it cannot be definitively proven that you slipped on ice that was put there by Levistus as revenge from your junk soul scheme, he is cleared of all charges.

From MToF, on page 17 there's an entire section on devil ranks and how creatures rise and fall. There's 13 ranks in all, so let's start at the bottom:

Unranked Devils

Lemures Sitting at the bottom, Lemures are created from the souls of evil mortals who more or less ooze out of the River Styx. Even in the Nine Hells, it's hard to kill a Lemure as it has the Hellish Rejuvenation feature which allows it reform after being killed. This is a big deal as it allows Lemures to regularly serve and die as shock troops within the Blood War without the Nine Hells losing too many resources.

In general, the Lemures are a commodity to be exchanged and the manner in which they are exchanges is determined by the Archdevil Mammon, whose forces patrol the River Styx to harvest the Lemures. Mammon then contracts the sale of the Lemures to other Archdevils per the terms of whatever other contracts apply.

Nupperibos Although more powerful than Lemures, these creatures are created from souls that are deemed absolutely worthless because their damnation stemmed from sloth. Thus, even if killed, the Nine Hells does not feel as though they've lost anything.

Lesser Devils

First of all, past this point all devils described fall under the authority of an Archdevil. Meaning, that if an Imp (Rank 2) that serves under Mammon is given an order by a Bone Devil (Rank 7) that serves Levistus, the order will have no weight and be unenforceable. However, were Asmodeus (Rank 13) to give any devil an order, they'd be bound to follow it because Asmodeus commands all devils.

It should be noted that the promotion from a Lemure, presumably includes an oath of loyalty to the Archdevil that is enabling the promotion. Per the MM:

No devil can promote or demote another devil that has not sworn fealty to it, preventing rival archdevils from demoting each other's most powerful servants.

Ranks 2-7 These ranks constitute Imps (2), Spined Devils (3), Bearded Devils and Merregons (4), Barbed Devils (5), Chain Devils (6), and Bone Devils (7).

Per the MM, promotion to any Lesser Devil requires the power of a Greater Devil or an Archdevil. Apparently there's a painful transformation to move the memories from the form of a Lemure into their new form. Promotions to Imps are typically performed based on need, while higher promotions are typically based upon merit. Per MToF, merit based promotions stem from the devils' performance in the Blood War under the scoring category of Glory. Failure and disobedience as a lesser devil can entail punishment via demotion back down to a Lemure, which causes the loss of all memories.

Greater Devils

Greater devils are created from lesser devils and can only be made by Archdevils. It should be noted that this promotion carries substantial risk, because should demotion be necessary down the line the demotion will not remove the Greater Devils memories. Therefore, it may seek revenge.

Ranks 8-11 These ranks consist of Horned Devils and Orthons (8), Erinyes (9), Ice Devils and Narzugons (10), and Aminizus and Pit Fiends (11).

Presumably, promotion to these ranks stems not just from merit as described in MToF under Glory, but likely also include the Souls and Treachery criteria. Devils are expected to harvest souls and the higher the value, the better. An example provided for Souls includes a devil that manages to make a powerful warrior pledge their soul and form as an Ice Devil within the Nine Hells; obviously this soul is of much greater value than any random evil commoner. Thus, the devil that secures it has greater standing an is more likely to be promoted.

Alternately, there's the Treachery option, whereby a devil learns the complex rules of the Nine Hells and the applicable contingencies and causes a situation whereby there's a vacancy above them and they are defaulted. This sort of a trickery is wholly accepted and part of the game. This doesn't necessarily mean that the devil directly kills their superior, thereby destroying their soul (this is likely forbidden by infernal law), but rather they may cause their failure and thus mandate their demotion. Indirectly killing is probably completely permissible.

Archdevil

Not surprisingly, only Asmodeus can promote a Greater Devil to an Archdevil. Generally, this is a pretty rare occurrence and likely only happens via the Treachery option.

Archdevils rule a layer of the Nine Hells and have a full chain of command beneath them. There's no specific lore I could find regarding how this promotion process occurs, presumably it is the same as that for a Greater Devil, however, an Archdevil's form is wholly unique to their personality.

Once promoted to this status, the soul must learn how to be paranoid very quickly because at this point every devil under their command and outside their command is looking for a promotion.