[RPG] Can an item resist the Identify spell

Yes, the DM can choose to make the spell not work

The variant rule in p. 136 of the DMG says this:

consider removing the ability to identify the properties of a magic item during a short rest, and require the identify spell, experimentation, or both to reveal what an item does.

Beyond that, the DM is Master of Rules in his game world. (DMG p. 5) There's an extended treatment of how a DM deals with identify as a spoiler in this Q&A.

How does one proceed?

This is likely a home brewed item, and may have additional protection against divination spells on it, such as nondetection.

For the Duration, you hide a target that you touch from Divination magic. The target can be a willing creature or a place or an object no larger than 10 feet in any dimension. The target can’t be targeted by any d⁠ivination magic or perceived through magical Scrying sensors.

While that spell lasts normally only 8 hours, there is nothing stopping the DM from applying that spell to it permanently. Alternately, this book may be a sentient magic item (DMG p. 214-218) that casts nondetection on itself when it is handled, or as a reaction to divination magic.

Given the clear explanations in identify's spell description on what "should happen" you are correct - there is something unique or non-standard about this item. What to do?

1. If you have not already done so, cast detect magic on the item to see what school of magic is the enchantment on it.
   (A nondetection spell should give off an abjuration aura, though @Ruse points out that nondetection would in theory block that spell as well. The fact that detect magic doesn't work could tell you that something like nondetection is protecting this item).

2. Discuss with your DM the making of a Intelligence(History) and or an Intelligence(Arcana) check to see if there is lore or history on that tome that you may have been taught earlier in your character's life.

3. As to experimentation with a book: you may need to read the title page, or something like that, and risk "something happening" to get an idea of what the book does. It looks like your DM is taking an old school approach to making item identification more challenging.

4. You can try to use dispel magic to remove a spell effect like nondetection.

   Choose one creature, object, or magical effect within range. Any spell of 3rd level or lower on the target ends.

If that works, great. If not, the DM may have set it up differently, and you may need something like legend lore (5th level spell) to gain more information about the item.

5. If the DM does that all the time, that's one thing.

6. If this is an exceptional case, this blocking of identify is a sign to you that you are dealing with a very unusual item, perhaps an artifact.

If it's an artifact ...

Proceed with caution.

1. Artifacts can have major detrimental properties. For example, "dilutes magic potions within 10' of it rendering them non magical" is a property an artifact can have.(DMG p. 220) The "identify doesn't work on this item" is a similar (potential) attribute.
2. An artifact can have the minor beneficial property of casting a level 3rd spell (chosen by the DM) as an action. (DMG p. 218). If the item is sentient and an artifact, it would likely be 'attuned to itself' and able to cast nondetection on itself as a reaction to the identify spell - with just a little stretching of the way that option was set up in the DMG. Is is well within a DM's authority to set up, since artifacts are by their nature 'one of a kind' items.

Blessings

There are a variety of blessings (DMG p. 228) that can be bestowed on a PC. A blessing of 'non detection' applied to this item (rather than on a PC) isn't too far of a reach. Homebrewing magical items is as much art as science, and well within the DM's purview.

Another way to frustrate Identify

As Alk pointed out, if someone cast Nystul's Magic Aura on the item for 30 straight days, identify or detect magic can return the result of "not magical" due to how that spell works.