Learn English – the act of breaking a pen nib after signing a death sentence called

An 1844 translation of Wilhelm Meinhold, Mary Schweidler, The Amber Witch (1838) describes the conclusion of a trial for witchcraft that supposedly occurred in 1630 (the book was a piece of fiction but was presented as an old document discovered by the author, in the manner of James Macpherson's discoveries of the works of Ossian). First the judge pronounces the defendant guilty and condemns her to death. The book then continues:

As he spake the last word he brake his wand [of office] in two and threw the pieces before the feet of my innocent lamb [the accused girl], saying to the constable, "Now, do your duty!"

A footnote to a new edition of that translation reports on the meaning of this breaking of the "wand" and then alludes to the custom that the OP asks about:

The staff of office represents the official's power, and he breaks it after pronouncing the death penalty to symbolize the finality of this judgement. In modern times this was replaced by the tradition of the judge breaking the nib of his pen.

A genuine piece of old writing, The Hungarian Rebellion: Or, an Historical Relation of the Late Wicked Practises of Three Counts, Nadasdi, Serini, and Frangepani; ... (1672) has this:

After which [the completion of the first execution], the said Frangepan was brought from the first place, in the fame order, which had been observed with Zerini, as also the reading of the Sentence, breaking of the wand, the declaring the grace for sparing his right Hand, and other things, were just so as had been done to Zerini.

Richard Evans, Rituals of Retribution: Capital Punishment in Germany, 1600–1987 (1996) mentions "the breaking of the wand of office" on three separate occasions as part of the ritual of juridical condemnation in a capital trial.

Chambers's Encyclopædia (1892) includes these comments in its entry for "Lord High Steward" (or rather ("Steward, Lord High"):

Since that time [the reign of Henry IV] there has been no permanent Lord Steward, but the office is temporarily revived when occasion requires, a Lord Steward being appointed under the Great Seal pro hoc vice at a coronation or the trial of a peer. When the proceedings are at an end the Lord Steward terminates his commission by breaking his wand of office.

To similar effect, The Scots Magazine (January 1777) describes the funeral of Alexander Kincaid, Lord Provost of Edinburgh:

As soon as the body was deposited in the grave, the Senior Herald approached to the foot of it, and having received the rod of office from the person who carried it, he pronounced the following words: "Thus it hath pleased Almighty God, to remove from this life to a better, our worthy Chief Magistrate, the Right Honourable ALEXANDER KINCAID, Lord Provost of this city, Representative of the family of Bantaskine." He then broke the rod, and dropped it upon the coffin; and, while the grave-diggers were throwing in the earth, the city-guard fired three vollies over the grave.

Thomas Mortimer, A New History of England, volume 1 (1763) mentions two occasions in 1399 when first the Duke of York and then the Earl of Worcester broke their "staff of office" to betoken his renunciation of fealty to Richard II. This suggests that the ritual breaking of the wand or staff or rod of office might be done as an act of rebellion or (as in the Scottish example) an act of closure.

A much more recent reference to the custom appears in Frontline, volume 15, issues 1–8 (1998):

All of those charged [in the assassination of Rajiv Gandhi] continued to protest their innocence. This process took all of afternoon. At 5-45 p.m., Judge Navaneetham pronounced the sentence. At the end of pronouncing the 26 death sentences, he broke the nib of his pen, in line with judicial custom.

Conclusion

It seems very likely that the ritual breaking of the nib of the pen used to sign a death warrant is an alteration of the much older custom in which the presiding judge would break a wand, staff, or rod of authority upon condemning a defendant to death—and that the breaking of that symbol of office symbolized the irrevocable nature of the decision.

Instances of this ritual appear in the 1600s in Hungary, in Germany at various periods, and in India in modern times. Although the breaking of the staff of office in Britain often signified the termination of a commission, it seems not unlikely that some similar ritual attended the condemnation of prisoners to death, since this would be the simplest way to connect the European tradition of nib breaking mentioned in the notes to The Amber Witch with the Indian use of the same judicial custom.

None of the sources I consulted had a special name for the ritual beyond (for example) "the breaking of the wand." Perhaps the nib-breaking form of the ritual doesn't have a unique name.