The RAW doesn't specifically say.

As you note in your research, none of the rules related to hitting objects have any information about what to do if the object is attended.

That said, you can use existing rules to figure out a reasonable alternative. The rules for smashing an object that you link say:

An object's Armor Class is equal to 10 + its size modifier (see Table: Size and Armor Class of Objects) + its Dexterity modifier.

If you try to use the spell rusting grasp on someone's armor:

You may employ rusting grasp in combat with a successful melee touch attack. Rusting grasp used in this way instantaneously destroys 1d6 points of AC gained from metal armor (to the maximum amount of protection the armor offers) through corrosion.

You can similarly touch a weapon with rusting grasp:

Weapons in use by an opponent targeted by the spell are more difficult to grasp. You must succeed on a melee touch attack against the weapon. A metal weapon that is hit is destroyed. Striking at an opponent's weapon provokes an attack of opportunity. Also, you must touch the weapon and not the other way around.

It says that weapons are harder to grasp, but it doesn't actually give any rules for how you should determine the AC of the weapon.

With this in mind, one reasonable ruling would be to apply this rule to attacks against attended objects, but let the object use the attending character's Dex bonus instead of its own, as well as any other modifiers that the character has that apply to touch attacks, like deflection bonuses or a monk's Wisdom to AC. If you try to touch an attended weapon, you provoke an attack of opportunity. This means that it's a little bit harder to hit an object that a character is attending than it is to hit the character, since most objects are smaller than their users.

Another reasonable ruling would be to say that it's a Combat Maneuver, like any other. In this case, I'd say that touching an object that's being attended by another creature would work like Sunder. In this case, you'd simply make a CMB check against your opponent's CMD, provoking an AoO if you don't have Improved Sunder. The drawback of this approach is it means that it's basically impossible for casters to touch an object being attended by a creature except at very low levels, since the CMB of a caster goes up very slowly, but the CMD of most other creatures goes up very quickly.

The desired effect may just be too complicated…

The first spell commonly available to move a foe to another plane is, in fact, the 5th-level cleric spell plane shift [conj] (Player's Handbook 262). Its low level compared to the sorcerer/wizard version means that a high-level cleric can prepare several such spells—enough, anyway, for an infiltration mission during which foes are to be dispatched to other dimensions yet not—deliberately nor technically—outright killed.

Using the spell plane shift in such a fashion still carries with it some risk: if the caster says, "I plane shift the guard to Mechanus," for example, either the spell fails because Mechanus is no more a destination than the Material Plane or the DM randomly determines where on Mechanus the poor guard ends up. This is less of an issue if the cleric does have a destination in mind ("Because everybody loves the City of Brass, right?"), but the subject will still be from 5 to 500 miles away from that destination, which could be fatal if the destination is on, for instance, the Elemental Plane of Fire.

(A unique case is when a creature's sent to the Seven Mounting Heavens of Celestia: folks sent to that plane always end up on the shores of the lake surrounding the base of Mount Celestia, and probably near an inquisitive angel bear. I wish I were kidding.)

The 7th-level cleric spell greater plane shift [conj] (Spell Compendium 159) solves the off-target issue, but the caster must've once visited the destination, and the spell's higher level means devoting even greater resources to multiple castings.

Enough gold solves at least some of volume problem, though. If the caster can modify the spell plane shift et al. with the metamagic feat Reach Spell (Complete Divine 84)—it's level adjustment is usually +2 but ways to mitigate that exist—, the caster can employ an expensive metamagic rod of chaining (Magic Item Compendium 164) and, possibly, disappear all at once a bunch of guards (if they possess not-very-good Willpower saving throw bonuses, anyway).

Keep in mind, though, that spells like plane shift typically don't enable the subject to return to its starting plane. Once the subject's affected, the subject's on its own on that new plane, and that might as well be a death sentence (or at least a sad sentence) if, for instance, the subject's a generic level 4 human warrior or whatever.

Other effects can do similar things. A trap could be set by a high-level caster using the 4th-level Sor/Wiz spell desert diversion [conj] (Sandstorm 113-4) on the infiltrator's own high-level method of entry into the facility, for instance, and such a portal'll briefly zap away investigators to a distant wasteland, but that's sloppy and unreliable without another spell like the 8th-level Sor/Wiz spell sympathy [ench] (PH 292) or something. And the 8th-level Sor/Wiz spell maze [conj] (PH 252) does just about exactly what you want, but, it, too, uses an 8th-level spell to avoid killing folks, making it, frankly, excessive.

…How about something simpler?

Ideally, the infiltrator'd use the 3rd-level psion/wilder power time hop [psychoportation] (Expanded Psionics Handbook 137), but, unfortunately, psionics are unavailable. However, the existence of an effect like time hop raises the possibility of a traditional caster researching a similar original spell (DMG 198). There are, after all, mechanics for such an effect already.

But let's keep in mind the goal: make guards temporarily unaware of actions taking place around them in a nonviolent fashion so that they can't testify as to what the infiltrator did. May I suggest the humble, reliable, 40-year-old 1st-level Sor/Wiz spell sleep [ench] (PH 280)? And it's big brother, the 3rd-level Sor/Wiz spell deep slumber [ench] (PH 217)? Combined with the 4th-level Sor/Wiz spell greater invisibility [illus] (PH 245), the caster should be able to get near enough to use these spells and render the victims insensate so that the caster can go about his task undisturbed, and the victims'll awaken undamaged, hopefully (if the caster's quick enough) after the caster's departure. The metamagic feat Silent Spell (PH 100) and the 2nd-level cleric spell silence [illus] (PH 279) may be of use to the infiltrator, and the object upon which the silence spell's cast could even be [cue dramatic music] a clue!

Such a workaround also eliminates the need for the caster to create a demiplane that erases memories (an unlikely prospect anyway as memory isn't a planar trait that can be specified even with a generous reading of the 9th-level spell genesis).