Learn English – Is answering “Hope you had a good time!” with “I did!” correct

If you're asking the time, (A) is closer to what a native speaker might say, but it would be more likely to occur as . . . what the time is. However, because the first part of the sentence contains the past tense was, the second part would probably also be in the past tense: . . . what the time was.

If you allow me to tweak slightly:

I had gone to work at 6 o'clock.

At 6 o'clock, I had gone to work.

Both these are fine if there is a context allowing them, such as:

Yesterday, I returned home at 10 pm and was fast asleep within ten minutes. My day had been very busy - I had risen before dawn, and done my usual 3-mile jog before returning for a shower and breakfast. I had gone to work at 6 o'clock. ...

But the expression indicates the time at which "I" went to work (left home), not some time by which (as you say) I was at work or some way along my way to work. To avoid confusion, choose by.