Learn English – Difference between “made” and “had made”

An infinitive X following to be means to do X at some future time because you are supposed to or scheduled to do X.

I am to go to work tomorrow. (I'm supposed to go to work tomorrow so I plan to do that).

John is to be cleaning and not playing video games. (John is supposed to be playing cleaning. This is likely John's mother telling his friend not to let him play video games.)

Infinitives can take have to specify perfect aspect.

To go to the mountains is something everyone in his family does.

To have gone to the mountains was an important thing to his family. (Implies family or mountains does not exist anymore).

I was to go to work yesterday.

I was to have gone to work yesterday. (Job or need to go doesn't exist anymore).

So the meaning of the second sentence is to have done X at some future time because you were supposed to or scheduled to have done X.

You can't do this:

I am to have gone to work yesterday. (Bad)

The exact difference is 100% in how the speaker is choosing to frame the event in time. There is no objective difference between them.

In the first, the speaker is choosing to treat the state of sitting as a single event; in the second, as an event extended in time.

The reasons for the speaker's choice cannot be answered without more context. Some examples of reasons they might have are:

If they are then going on to talk about something that happened during the section, they might choose the extended version (where were you sitting). If they are going on to talk about something that happened after the section, they might choose the punctual version (where did you sit).

They might see where were you sitting as more friendly and where did you sit as more formal or officious.

If the other person has just talked about how the room was very full when they entered, where did you sit is much more likely.