The Easy Ways
These aren't exactly what you want because they're not magic items, but these spells can be put sometimes into wands, sometimes into staffs, or into wondrous items.
- Cast the 3rd-level Sor/Wiz spell sepia snake sigil [conj] (PH 276) on a document that the creature to be imprisoned must read. A failed saving throw puts the creature in stasis for 1d4 +1 day/level. Cast the 2nd-level Sor/Wiz spell rope trick [trans] (PH 273). After finding a way to communicate with an Int 3 magical beast from the lower planes, cast the 1st-level Sor/Wiz spell summon monster I [conj] (PH 285-6), gaining the services of a fiendish monstrous scorpion (Small) (MM 287). Command the scorpion to take the creature affected by the sigil into the trick and retract the rope. Note: A wand of rope trick modified by the feat Extend Spell (PH 94) (2nd-level spell modified to a 3rd-level spell at caster level 20) costs 900 gp per charge and creates a 40-hour trick.
- The 5th-level Sor/Wiz spell soul shackles [necro] (BoVD 104) traps a creature's soul in a tchotchke for questioning later. If the soul's hostile and succeeds on a saving throw when questioned, the soul's freed. The spell is technically broken (the Target is a living creature but the description says the spell's cast on a dead creature), so house rules are necessary to make the spell work. I'm thinking this might be the effect you're looking for without the hassle of making an item.
- Cast on the creature to be imprisoned the 6th-level Sor/Wiz spell smoky confinement [trans] (CM 117). A failed saving throw instantaneously traps the creature in a "tiny receptacle" costing at least 100 gp.
- Cast on the creature to be imprisoned the 7th-level Sor/Wiz spell plane shift [conj] (PH 262), with a failed saving throw sending the creature to a plane with the trait limited magic (DMG 150) that stifles the imprisoned creature's ability to escape the plane. In most campaigns I DM it would be extremely difficult--perhaps even impossible--to find information about such a plane (e.g. the precise kind of "small, forked metal rod" needed as a material component for the spell plane shift to visit the plane via that spell).
- I'd like to recommend the 7th-level Sor/Wiz spell amber sarcophagus [evoc] (BE 90), which is like the spell sepia snake sigil except requiring only a ranged touch attack and a 500 gp material component, but evocations aren't an option.
A number of even higher level options are available, of course, but that list is beyond this abbreviated one's scope as you're looking for the lowest level one available.
The Hard Way
What you're asking for is an extremely high-level--probably epic--magic item. Killing creatures in D&D 3.X is simple; the dead condition is an easily removed status effect that high-level PCs are supposed to casually negate. Capturing things--that is, not killing something but leaving the creature at one's mercy but without options--is a far harder effect to generate. The game resists the very idea because it's not fun; no player wants such power in the DM's hands (because--poof!--there goes the PC forever), and no DM wants such power in the PCs' hands (because--poof!--there goes the plot forever). But it can be done with time and vast wealth.
- Craft a custom magic item that generates a continuous effect like the 8th-level Sor/Wiz spell veil of undeath [necro] (SpC 229). This costs 360,000 gp (base 2,000 gp x8 for spell level x15 for caster level x1.5 for the 10 min./level duration of the veil spell). In addition to the spell's other effects, a creature wearing the item no longer breathes, eats, or sleeps, and the creature's immunity to nonlethal damage makes it so the creature takes no damage from thirst. Note: I could locate no lower-level Sor/Wiz or Drd spell that allows a creature to go without drinking, eating, and--hardest of all--breathing. Finding one would reduce the cost of the custom item appropriately.
- Add to the custom item the effects of antimagic shackles (BE 116) (132,000 gp; 5 lbs.) at a cost of 198,000 gp.
- Add to the custom item the effect of the 5th-level Sor/Wiz spell lobotomize [trans] (Dragon #339 77), picking to affect the skills Escape Artist and Open Locks. This costs 270,000 gp (base 2,000 gp x5 for spell level x9 for caster level x2 for the 1 min./level duration of the lobotomize spell x1.5 for adding the effect to the custom item).
- Add to the custom item the effect of the 1st-level Sor/Wiz spell ray of clumsiness [trans] (SpC 166). This costs 60,000 gp (base 2,000 gp x1 for spell level x10 for caster level x2 for the 1 min./level duration of the ray spell x1.5 for adding the effect to the custom item). I'd argue that this inflicts a -8 penalty to the shackled creature's Dexterity, and but no matter what would helps prevent creatures with high Dexterity scores from simply slipping the shackles.
Then, after crafting and securing the creature, find a place to put the shackled creature. The antimagic field of these shackles will make this difficult. I suggest first using plane shift to locate a secure spot then shackling the creature.
What follows is the suggested end result for a custom magic item incorporating the previous effects.
Shackles of Eternal Confinement (888,000 gp; 5 lbs.)
These adamantine manacles fit any Small to Large creature and, when fastened, give the creature a -8 penalty to its Dexterity while creating an antimagic field to a radius of 5 feet. The DC to slip out of the shackles is 28, but breaking them is nearly impossible, requiring a DC 40 Strength check. Further, a shackled creature can make only untrained Escape Artist skill checks (as if the creature had 0 ranks in the Escape Artist skill) to slip the shackles, and the creature can't make Open Locks skill checks to unfasten them. Finally, a shackled creature is immune to mind-affecting spells and abilities, poison, sleep, paralysis, stunning, disease, death, extra damage from critical hits, nonlethal damage, death from massive damage, ability drain, energy drain, fatigue, exhaustion, damage to physical ability scores, and any effect requiring a Fortitude save unless it is harmless or affects objects. The creature need not breathe, eat, or sleep, and is damaged by cure spells and healed by inflict spells. Strong necromancy; CL 15th; Craft Wondrous Item, antimagic field, lobotomize, ray of clumsiness, veil of undeath; Price 888,000 gp; Weight 5 lbs.
"Wait... What?": Beyond the scope of this question is how any magic item that creates an antimagic field continues functioning when the magic item itself creates an antimagic field. Apparently such items do continue generating their antimagic fields even when the items themselves are in the item-created antimagic field because, well, such items exist in officially published sources sans eratta (e.g. bulwark of antimagic (Dr 118)), and one must assume that activating those items doesn't also immediately deactivate those items. I imagine it's like Magic: The Gathering's White Ward in that the item's own antimagic field doesn't affect the item... um... for some reason (e.g. "A wizard did it"). The above item assumes such a ruling's in place for this item (which is a custom item anyway), making the addition of the shackles of antimagic to the base item possible without having the antimagic field dismissing the item's other effects.
An Assumption: I assume you want the creature conscious so that the capturing creature can interrogate the captured creature or--more likely--gloat over the captured creature's plight. I mean, I would. If that's not a thing--that is, the captured creature need not be conscious--, craft a custom magic item from masterwork manacles (PH 126-7, 128) (50 gp; 2 lbs.) incorporating a continuous effect like the spell feign death [necro] (TB 89) costing 30,050 gp (base 2,000 gp x3 for the spell level x5 for the caster level and +50 gp for the masterwork manacles). The DM must determine how the manacles of the sleeping prisoner (or whatever they'd be called) work--as he must whenever custom items are involved--, but unconscious creatures are always willing, so a downed foe so manacled is pretty much permanently unconscious and doesn't need any of that air, food, or water stuff. Then the capturing creature can simply toss the captured creature into the nearest extradimensional space of choice.
The best way for the wizard to defend a spellbook is the same as the best way for you to protect your precious computer files - have multiple backups.
But, if he hasn't had time to make a copy, if he knows someone is trying to steal the book back, he wouldn't leave the book in the shop overnight. He'd keep it on himself, likely guarded by as many guards as a 7th level wizard can afford to hire on a temporary basis.
If he must leave the book in the shop for plot points, he'd certainly have the book very well protected and secured. He's 7th level, so he knows 4th level spells. He might, for example, use Stone Shape to fabricate a 'doorless' safe in a wall or floor block. And he'd certainly know to keep the book in a lead box to block scrying and detection attempts. He could also use illusions to hide/conceal the book. And decoy books and safes to waste the thieves' time. And of course, a 1st level Alarm spell on the shop would work wonders. He might hire a dozen men at arms who hang out in a neighboring building waiting for the alarm to go off.
You say he can't use anything too damaging in terms of traps. Well, poison gas doesn't cause much physical damage and dissipates after a while. And while you say its illegal for him to create fatal traps, a) he may not care, b) bribes and Charm Person can get the well-to-do out of trouble, and c) dead thieves can't report you to the town guard. If you don't want to do that, you can always fill the shop with a Web.
Beyond defending his shop, since he knows there's likely going to be a break-in, he might have a familiar watch the shop from a distance and follow the thieves back to their home/inn/hideout. And when they aren't looking, he can rob them blind.
Best Answer
The DMG states on p288 under Adding New Abilities:
So you can enchant your spellbook as you like (unless the spell description specifically restricts it). Other items if they take up a body slot incur the 1.5x extra cost. It goes without saying that the DM has the final say on what can and can't be done in their game world.