Learn English – It had been a long time since + Past Perfect or Past Simple

Would it be possible to change "had gone" into "went"?

I think it would depend on when it was said—before or after the event.

Said sometime before:

(Two hours ago) She invited me to dinner. But it has been two years [now] since I [only once] went to her house. So I am afraid I may lose my way.

Told sometime after:

(Once/Yesterday) She invited me to dinner. But it was two years since I [only once] had gone to her house. So I lost my way.

OR:

(Once/yesterday) She invited me to dinner. But it had been two years since I [only once] went to her house. So I lost my way.*

*Note: the pattern is either "it had been… since I went" or "it was…since I had gone". Also, I don't think that the pattern "it had been…since I had gone" is incorrect even if it is far less common.

Having said that, I can't but agree with @user3169: No matter how you look at it, the distinctions are rather small and overfine to delve too deeply into.