No. There are no downsides at all (again provided the enemy doesn't have Mage Slayer or some other specific feature) to casting in melee if the spell does not have an attack roll.
For all intents and purposes it's not an attack.
There are several other rules that govern attacks that don't work with spells that don't have attack rolls. For instance, you can't crit with them.
Firstly, yes, if a spell requires an attack, that attack still counts as an attack.
So, what you have to remember is the "specific beats general" rule. This is detailed on page 7 of the PHB. The core of the rule is:
If a specific rule contradicts a general rule, the specific rule wins.
So the general rule here is that familiars can't attack. There are 2 exceptions to this general rule:
A familiar can deliver spells with a range of touch on your behalf, even if the spell requires an attack roll.
A Warlock with the Pact of the Chain can forgo one of their attacks to let their familiar make one.
So when you cast a spell with a range of touch, you can have your familiar do the actual "touching" on your behalf. You cast the spell as normal using whatever actions it requires. The familiar is required to use their reaction, and if the spell requires an attack roll, it makes the attack roll. Since it uses your modifier anyway, it's exactly the same as you making the attack roll except that you don't have to stand next to what you're casting the spell on.
In the case of a Warlock with the Pact of the Chain, the Warlock takes the Attack action as usual, then has the familiar do the actual attack. The errata for the PHB says:
When you
let your familiar attack, it does so with its
reaction.
If the Warlock can make multiple attacks with the Attack action, the familiar can use its reaction to replace one of them with its own attack, then the Warlock makes the rest.
Finally, only spells with a range of touch can be delivered by your familiar. There is no option for having your familiar cast a spell on your behalf, and the option for allowing a familiar to attack instead of yourself specifically says "when you take the Attack action", not just "when you make an attack".
Best Answer
There wouldn't be disadvantage on the attack roll.
If the target is behind the other player, however, the target will benefit from cover. See page 196 of the Player's Handbook for those rules. Likely, the target would gain the +2 AC bonus for half cover.