Learn English – It was done / It had been done yesterday

The first form is used only when you describe event/state that happened/lasted before another event in the past. The second is just plain past.

"I had thought there was something wrong with the tape recorder until I heard my own voice recorded with studio equipment." (the thought ended when you heard it)

"I thought there was something wrong with the tape recorder."
"No, it was just a bad tape." (there were no other actions that would end your thought in the past)

If you allow me to tweak slightly:

I had gone to work at 6 o'clock.

At 6 o'clock, I had gone to work.

Both these are fine if there is a context allowing them, such as:

Yesterday, I returned home at 10 pm and was fast asleep within ten minutes. My day had been very busy - I had risen before dawn, and done my usual 3-mile jog before returning for a shower and breakfast. I had gone to work at 6 o'clock. ...

But the expression indicates the time at which "I" went to work (left home), not some time by which (as you say) I was at work or some way along my way to work. To avoid confusion, choose by.