No, it's not an opportunity attack.
An opportunity attack is a specifically named game mechanic with specific rules set out for it (as found on page 195 of the PHB).
If the third benefit of the Sentinel feat was meant to be an opportunity attack, it would say something like:
[...] you can make an opportunity attack against the attacking creature.
As it's written, it's not counted as an opportunity attack because it doesn't say it is. What the third benefit of the Sentinel feat (PHB, p. 169) does say is that:
[...] you can use your reaction to make a melee weapon attack against the attacking creature.
The targeted enemy can't make opportunity attacks
As you've quoted, the relevant part of the Swashbuckler rogue's Fancy Footwork feature description says (SCAG, p. 135; XGtE, p. 47):
During your turn, if you make a melee attack against a creature, that
creature can’t make opportunity attacks against you for the rest of
your turn.
The only requirement for the feature is that you "make a melee attack" against a creature on your turn; it doesn't specify that the attack has to hit (or miss), so it doesn't. Therefore, if you make a melee attack against a creature on your turn, then that creature can't make opportunity attacks against you for the rest of that turn. That's all there is to it.
Rules designer Jeremy Crawford reiterated this fact in an unofficial tweet from May 2016:
does the Fancy Footwork feature of the Swashbuckler rogue require the melee attack to be successful?
Fancy Footwork works whether or not your attack hits.
This tweet just repeats what's already self-evident from the feature description: you just have to make the attack - you don't have to hit.
As a sidenote, Fancy Footwork doesn't even require an attack roll, since it doesn't mention hitting or missing; it just says you need to make a melee attack against a creature. This means that even trying to grapple or shove a creature is enough to qualify for the benefit of Fancy Footwork, since both shoves and grapples are described as "special melee attacks".
This fact is also reinforced in a Q&A about the interaction of the third benefit of the Mobile feat, which works similarly (as you note): Does a Grapple or Shove count as an attack for the third benefit of the Mobile feat?
Best Answer
Yes, grapples and shoves count as melee attacks for the Mobile feat, according to RAW
The third benefit of the Mobile feat says (PHB, p. 168; emphasis mine):
Thus, it only triggers this effect on a melee attack and does not care if that melee attack results in a hit or not.
Grappling and shoving are melee attacks
The rules for grappling say:
The rules for shoving say:
Rules designer Jeremy Crawford has also made an unofficial ruling to the same effect in this tweet:
So, grappling and shoving are are defined as attacks in the rules and designer clarification has verified that (many times in fact). Thus, they qualify as melee attacks for this part of the mobile feat.
Grapples/Shoves cannot hit, but this technically does not disqualify them
We know that grappling and shoving are attacks. However, they are unusual attacks because they do not use an attack roll.1 And because of that they cannot hit or miss - only succeed or fail (see Does grappling count as a hit?).
This is important because Mobile says that the effect triggers "whether you hit or not".
Despite the fact that they cannot hit or miss, technically the wording still leaves room for grapples/shoves to qualify. If it had said "whether you hit or miss" then you would not be able to use a grapple. However, it does not say that, it says "hit or not" and technically a grapple attack will always not hit because it cannot hit (it can succeed or fail). Thus, they still work via a strict RAW reading of the language.2
1 Because there is some confusion on the matter, the fact that shoving/grappling does not involve an attack roll does not in any way make them not an attack. Most attacks involve an attack roll; grapples and shoves are unusual in the fact that they don't involve an attack roll, but they are still attacks. As Jeremy Crawford once again clarified in an unofficial tweet:
2 This kind of weirdness in the language seems to indicate that the feat was written without consideration for the fact that melee attacks that do not involve attack rolls do exist (they are in fact extremely rare). This seems more likely than an attempt to specifically exclude grapples/shoves. In any case, the language does not actually exclude them so my logic stands regardless of intent.