[RPG] How does attacking from cover work

See? Yes. Notice? Maybe.

The rules on Hiding and stealth give a lot of leeway to a DM, so you are absolutely within your rights to declare that once a creature is looking directly at you, you cease to be hidden from it. But keep in mind a couple of important distinctions:

1. Hiding is different than being unseen

Any creature that is in a heavily obscured area, or invisible, or fighting a blinded enemy is "unseen". But a creature that is hiding is not only unseen, but also unheard, and generally unperceived. They might keep very still, or stand close to similarly colored parts of their surrounding, or compact their body so their silhouette no longer seems humanoid. In short, they are doing more than being out of line of sight.

As evidence of this distinction, consider the following on page 177 of the PHB:

An invisible creature can’t be seen, so it can always try to hide. Signs of its passage might still be noticed, however, and it still has to stay quiet.

To put it another way (and to quote a 2016 Errata in the PHB):

the question isn’t whether a creature can see you when you’re hiding. The question is whether it can see you clearly.

2. You can't Hide from a creature that sees you, but maybe you could remain hidden

The rules are clear that

You can’t hide from a creature that can see you clearly, (PHB, p. 177)

but note that this only rules out becoming hidden: it doesn't necessarily rule out remaining hidden. So as a DM, it will totally be up to you whether or not the Goblin will notice the already hidden rogue when they gain line of sight.

Keep in mind that combat is a chaotic and distracting place. As an example of what that level of chaos can do to perception, note how difficult it is to count how many passes the players in white make in this linked video.

Note, though, that once the hidden rogue attacks anything, they won't be able to hide (from the goblin) again, unless they've moved to some new location that the goblin can't see. The rules are clear that while you could remain hidden while visible, you can't hide.

3. Advantage/Disadvantage (on attacks) is not from being hidden, it is from being unseen

All the points above are meant to indicate that a creature could remain hidden while they are seen: that is, an enemy might not notice them, or might not know where they are. But that does not necessarily mean they will gain advantage on an attack roll against such an enemy.

Any place that the rules suggest a hidden creature will gain advantage on an attack, they justify this as a result of the hidden creature being unseen. Such as:

Master might allow you to stay hidden as you approach a creature that is distracted, allowing you to gain advantage on an attack before you are seen. (PHB, p. 177, bold added)

And:

When you take the Hide action, you make a Dexterity (Stealth) check in an attempt to hide, following the rules in chapter 7 for hiding. If you succeed, you gain certain benefits, as described in the “Unseen Attackers and Targets” section later in this chapter. (PHB, p. 192, bold added)

So the only real benefit a hidden character could get against an enemy that can see it is that the enemy might not notice them. This could cause an enemy to be surprised (first round of combat only) or to be unable to intentionally attack the hidden creature (because they don't know it's there). But a creature hidden in plain sight would not get advantage on attack rolls against a creature that could see them, regardless of whether or not they remain "hidden".

This is absolutely counter-intuitive for me, when you mechanically can't effectively hide behind a 5 foot tree against a non-moving attacker.

What you need to understand is that the mechanics of combat are intended to function as abstractions for what actually happens in combat. Diegetically, the combatants aren't literally standing in place, waiting for all the other combatants to finish their "turn", every six seconds. That's just an abstraction we use to keep combat tactical and strategic.

Or, put another way: if you're a creature that is benefitting from half-cover, the only way you can then make an attack is if you physically move out of the way of that cover. It may not require you to use up movement speed (if, for example, you're peeking around the corner) but regardless, it's not like a potential assailant would specifically wait for you to make your attack and then retreat before trying to make their own attack; in fact, they'd probably specifically use the moment when you pop out of cover to make their attack.

^{Just to be clear, I'm not talking in terms of game mechanics; under most conditions, it's not possible for a creature to make an attack during someone else's turn, without using their Reaction or some other clearly specified game mechanic. I'm purely talking in terms of Narrative Diegesis, where "turns" don't exist and everything that characters do in combat is happening (relatively) simultaneously.}

But of course that makes it more difficult, hence the mechanical advantage granted by cover of having extra AC, making it harder to successfully land a shot. In this context, the +2AC granted by half-cover is the abstraction meant to explain the difficulty of timing a shot to line up with when you pop out of cover.

This also holds for other kinds of cover: Half Cover abstracts for the fact that, due to being just half cover, maybe part of your body is sticking out beyond the cover outside of your control, whereas with 3/4ths cover, maybe this is no longer an issue. Narratively, how you justify these abstractions is ultimately down to the DM, or for flavor purposes, the players themselves.

So it might seem weird that a 5-ft tree is unable to completely block incoming attacks from hostile creatures, but once you follow how combat is abstracted around cover and the use of obstacles in turn-based combat, I think it's more understandable why these mechanics work the way they do.