[RPG] How should I prevent a player from instantly recognizing a magical impostor without making them feel cheated

There really isn't a way to gain a spell book as a gish type class. Wizards don't yet^[1] have a viable melee archtype (though building one wouldn't be too hard, we have some models in the cleric domains).

There are however other ways to handle the problem you are facing.

Since your primary issue is in background type information and characterization, I would handle the majority of this mechanically by writing a custom background. This would give your proficiency in history and arcana (or another skill if you get it from your class), and a book that has arcane writings in it. Discovering the secrets of this book should be related to one of your character's BIFT.

Whether or not the development of the story around the book ends up providing you any future mechanical benefits is kind of up to you and your DM to hash out, but it seems like a very solid starting point for a new character in a game.

_{[1]: This answer was written before the Bladesinger archetype was published.}

Eldritch Sight (PHB 110) allows you

cast detect magic without expending a spell slot

An illusion is magic, so they will detect its presence automatically within 30', and will also be able to use an action to see the aura around the specific source and determine the school of magic, per the Detect Magic spell. Note, the illusion itself won't show an aura, but the trap trigger should, if the warlock is within range of it when it is triggered.

According to Jeremy Crawford (lead rules developer on 5e), when he was asked on Twitter

Would Detect Magic show an aura around an illusion?

His response was

Detect Magic lets you see an aura only around a visible creature or object, not around an illusion. But you can sense the magic.

So the warlock would be able to sense the presence of the magical trigger or illusion source and could detect that it has an illusion school aura if he can spend an action. If the trap is already triggered (assuming the illusions are cast by a trap) then the illusion itself will set off the magic "spidey sense" but wouldn't have an aura. The warlock would have to be within 30' of the trigger/source.

There are a few ways to confound a player using Detect Magic, though you really ought to accept this PCs capability and use it for good story moments, rather than trying to shut it down, which will only frustrate the player.

1. There are so many things radiating with magic that the PC can't distinguish between illusions and just ordinary things that have some residual magic aura without spending a lot of time (actions). Presumably whoever created this haunted house knew of Warlocks and this ability, so they planned accordingly. So give the PC lots of things to investigate, eventually they will get bored and walk into a trap. It is also possible that the use of Nystul's Magic Aura could mask the magical aura of the trigger/source as another school of magic. I would postulate that a warlock who uses Detect Magic frequently would habituate to the magic used by the party (the fighter's magic sword, the thief's magic ring, etc) and could detect a new magic source, lest Detect Magic become fairly useless unless the warlock is alone and divests himself of all his other magic items.

2. Using this magical ability attracts negative attention. Maybe the house responds aggressively to use of magic, the walls close in, things fly at the magic user, etc. This will force the PC to limit use of the ability.

3. Accept it and let the PC detect all the illusions and their source if he is close enough. But remember that things can block Detect Magic, like 1 foot of stone, 1 inch of metal, etc. So chests and such could be behind secret, non-magical doors.

4. The range is also only 30', so things farther than that won't show an aura. An illusion on a high ceiling, or at the end of a long room will require the PC to get close to even tell there is magic involved. Use that to your advantage to lure them into non-magical traps. After a few, they will learn to stay back :)

5. This is also eating up their concentration slot, as the invocation lets them cast the spell without using a slot but doesn't relive them of the need for concentration. So they can't have another concentration spell active. And if something happens that requires them to make a concentration roll and they fail, there could be a brief period of time where an illusion could pop up and the warlock won't be able to sense magic, or determine that it is just an illusion. So you could have a mundane threat that injures the warlock, breaks their concentration, then you trigger an illusion that they do react to. Your warlock may forget to recast the Detect Magic spell or prioritize other actions instead.

6. Mundane creatures with magic items will also trigger the magic sense, but the warlock will have to burn an action to see the source. Have a series of skeletons come at them, the first two just illusions, by the third the warlock may make an assumption that it too is just an illusion and then someone gets hit with the magic sword a real skeleton is carrying! Or vice versa, so the party wastes resources attempting to attack the third illusionary skeleton after fighting the first 2 real ones.

But again, you should embrace this ability, as it is the PCs area to shine and something they obviously care about, since they took that invocation over other ones. Be careful trying to negate it completely. Instead use it to make the character take risks.