Learn English – Any difference between “shouldn’t have done something” and “shouldn’t have to do something”

This could be just my bias, but it seems to me that the more roundabout passive version is more particular than the simple past participle version, referring to a specific circumstance or incident. Although both versions can be understood either as a statement of general truth or as a statement of a specific truth, it's as though the passive version is preceded, by default, with "On that occasion" and the simple past participle is not preceded by a default, but with "Whenever|On that occasion" and needs additional context.

Having been given an order, he obeyed.

Given an order, he obeyed.

Having been shown how to do the task, he completed it.

Shown how to do the task, he completed it.

An infinitive X following to be means to do X at some future time because you are supposed to or scheduled to do X.

I am to go to work tomorrow. (I'm supposed to go to work tomorrow so I plan to do that).

John is to be cleaning and not playing video games. (John is supposed to be playing cleaning. This is likely John's mother telling his friend not to let him play video games.)

Infinitives can take have to specify perfect aspect.

To go to the mountains is something everyone in his family does.

To have gone to the mountains was an important thing to his family. (Implies family or mountains does not exist anymore).

I was to go to work yesterday.

I was to have gone to work yesterday. (Job or need to go doesn't exist anymore).

So the meaning of the second sentence is to have done X at some future time because you were supposed to or scheduled to have done X.

You can't do this:

I am to have gone to work yesterday. (Bad)