Magic oils are complicated by being only lightly detailed

I would be remiss in my duties were I not to point out that technically the feat Brew Potion (Player's Handbook 89) does not allow a creature that possesses it to create magic oils. In fact, nothing seems to allow the creation of magic oils! While this is undoubtedly merely an oversight, it's an oversight that's just quirky enough for a really strict DM to rule that magic oils are the product of ancient and lost civilizations, forgotten magicks, vast power beyond the reach of insignificant mortals, or whatever so as to forbid their creation by PCs. Nonetheless, the responses below assume a DM who permits anyway the creation of magical oils using the feat Brew Potion. (I took the liberty of rephrasing the inquiries present in the questions. I hope that's okay.)

• Does the creator of an oil or potion determine if the item is an oil or potion upon the item's creation or upon its use? Upon the item's creation.

  The Dungeon Master's Guide Table 7–17: Potions and Oils (230) lists magic oils only as magic oils and potions as only potions, not providing any examples of hybrid magic oil/potions. (Although some spells like the invisibility and levitate spells can be randomly generated as either a magic oil or a potion, no results are listed as a magic oil and a potion.) This leads this reader to believe that a creator must designate his creation as a magic oil or potion upon the item's creation. (Also see below for further magic oil restrictions.)

• If the spell offers a choice of how its used, does the creator or the user decide if a magic oil or potion's effect will affect an area or be targeted? The creator… but it's more complicated than that.

  This answer needs two parts. First, the Dungeon Master's Guide says

  Potions are like spells cast upon the imbiber. The character taking the potion doesn’t get to make any decisions about the effect—the caster who brewed the potion has already done so. For example, a potion of protection from energy is always designed to protect against a specific energy type chosen by the creator, not the drinker. The drinker of a potion is both the effective target and the caster of the effect…. (229)

  That is, the creator of the magic oil or potion—not the applier or the drinker—makes decisions about the potion or oil's effect (and, obviously, makes those decisions when the item is created). Also, keep in mind that because a potion's consumer is also typically the target of the potion's spell, this makes brewing, for example, a potion of magic missile usually a bad choice except as a prank or trap. (This DM's had the PCs' enemies carry potions of painless death labeled as potions of cure light wounds, for example.)

  Second, according to the feat Brew Potion, the spell that's turned into a potion must be one "that targets one or more creatures," so this DM would mandate that, for example, a potion of dispel magic can only be created so that the potion's effect is a targeted dispel rather than an area dispel. Likewise, this DM would extend this ruling to any spell that could affect an area or a target, mandating that when it's turned into a potion the creator must pick the targeted version of the spell.

• Are oils used like splash weapons? No.

  A magic oil must be smeared upon—not thrown at or splashed on—the subject that's to be affected by it. (Also see below.) The Dungeon Master's Guide says

  [U]sing an oil… is a standard action. The… oil takes effect immediately. Using [an] oil provokes attacks of opportunity. A successful attack (including grappling attacks) against the character forces a Concentration check (as for casting a spell). If the character fails this check, she cannot drink the potion [or, presumably, apply the oil]. An enemy may direct an attack of opportunity against the… oil container rather than against the character. A successful attack of this sort can destroy the container…. [I]ncorporeal creatures cannot use… oils.… Any corporeal creature can use an oil. [I]t takes a full-round action to apply an oil to an unconscious creature. (ibid.)

  There are no other rules for the in-combat application of a magic oil. Apparently, for instance, a magic oil container can't be hurled at a foe in hopes of affecting the foe with its contents, and a creature can't smear a magical oil on another conscious creature, no matter how willing that conscious creature may be. Ask the DM if house rules eliminate either of these restrictions—potions and oils are widely considered sub-par, after all.¹ (Also see here for more about potion-delivery optimization and here for Pathfinder's similar magic-oils-in-combat rules.)

• Can an oil of dispel magic be applied to an object despite the targeting limitations present in the various descriptions of potions? Probably yes.

  If a DM allows a creator to create magic oils at all—which this DM does—this fellow DM recommends the following rulings: A creature that possesses the feat Brew Potion (or that can create potions through different but similar means) can also create a magic oil if the spell that's to become a magic oil has as its target one or more objects. Further, this DM recommends the following ruling: A magic oil can only target an object.

  To be clear, according to the feat Brew Potion, a spell that's to become a potion must be known by the creator and must target one or more creatures, and the potion's creator makes choices as if he cast the spell when the potion's created. Further, the spell can't have a casting time of 1 min. or more (DMG 229), and the spell can't have a Range entry of Personal (286).

  Finally, the Dungeon Master's Guide says, "The person applying an oil is the effective caster, but the object is the target" (229). To this reader that strongly implies that only objects can be affected by magic oils.² This is borne out by the Dungeon Master's Guide's examples of magic oils: it lists as magic oils only spells that can or also target one or more objects, like the spells magic stone, magic weapon, shillelagh, bless weapon, and even darkness. Based on these examples, this DM's ruled that a magic oil can only be applied to an object.

This reader believes that—despite the light details—the reason magic oils even exist is to add versatility to what can be created with the feat Brew Potion. However, this DM doesn't view a magic oil as just a different delivery method from a potion but a necessity when dealing with object-targeting spells.

"So you wanna huck dispel magic grenades?"

So what happens when a creator decides he's going to make an oil or potion of dispel magic? The creator first picks whether to make a magic oil or a potion. Whether the creator opts for a magic oil or a potion, the creator must pick the dispel magic spell's targeted dispel option. If he picks to make a potion, the spell's effect will affect the drinker as if the drinker had been the subject of the spell. If he picks to make an oil, the spell's effect will affect an object on which the oil has been smeared as if the object had been the subject of the spell (i.e. a successful caster level check renders the magic item nonmagical for 1d4 rounds). Using an oil of dispel magic this way in combat against a conscious and mobile foe's gear is probably too complicated: the DM must concoct house rules to do so, or the DM may simply rule that doing so is impossible. ("No," says the DM, "you can't rub the oil of dispel magic onto the balor's sword while it's wielding it. Move on.")

(This reader suspects that Table 7–17 omits an entry of "dispel magic (oil or potion)" because of the complicated nature of the item rather than because such an item can't be created, but this is pure and total speculation.)

However, if this must be the PC's jam, the Dragon #289 Wizards Workshop: Silicon Sorcery column "Gauntlet: Dark Legacy" on Gauntlet Potions says

Gauntlet potions may be either imbibed or thrown as a grenade, and you get a different effect depending on how you use it. If drunk, the potions provide protection from a certain type of energy or creature. If thrown, these potions explode as an area effect spell. Note that breaking the potion vial lets loose the spell, so these can be dangerous items to carry around. Potion spell effects function as though cast by a 10th-level caster. Gauntlet potions have a range increment of 10 feet and follow the rules for throwing [splash weapons.]

Under the normal rules of magic item creation, you cannot create a potion that stores a spell of 4th level or higher. Thus, the creator of these Gauntlet potions must have both the Brew Potion and Craft Wondrous Item feats and use both in the creation of these items. To create potions similar to this, use the following formula: (highest spell’s level × caster level × 50 gp) + (additional spell’s level × caster level × 50 gp × 0.5). The resulting total is the market price of the item. (120)

(More on the video game Gauntlet: Dark Legacy—from which these items were adapted—can be found here.) This article predates the 3.5 revision, making the material subject to minor adjustments by the DM (see DMG 4). Several Gauntlet potions are described in the article, and, as can be seen by the description above, a DM may allow the creation of new ones. This article should be any grenadier's go-to source for potion grenades.³

However, this reader does caution a DM who's considering allowing them into his campaign that these Gauntlet potions are a way to create multiple spell effects with a single activation, so they're potentially unbalanced if the spells within are allowed without restraint.

¹ If a reader's got the idea in his head already, it's definitely possible to read that DMG description of magic oils and assume the DMG is discussing tossing the magic oil at a foe… except that reading's spoiled by that last sentence. That is, if throwing a magic oil at a conscious and mobile creature takes a standard action and has the same efficacy as taking a full-round action to apply a magic oil to an unconscious creature, why not just throw the magic oil at the unconscious creature? While the DMG can make some glaring errors (see below), this reader assumes that the DMG assumes here that a magic oil's user typically takes a standard action to smear a magic oil on himself, sometimes takes a full-round action to smear a magic oil on another creature, and chucks a magic oil so that it affects a foe never.
² As mentioned, the DMG says, "[I]t takes a full-round action to apply an oil to an unconscious creature," and an oil of speak with dead is mentioned as an example. A DM that allows magic oils to affect dead and unconscious creatures must adjudicate the effects of, for example, a potion of darkness when applied to an unconscious creature. An oil of speak with dead, by the way, is impossible according to the DMG's own rules: A potion or oil "can duplicate the effect of a spell… that has a casting time of less than 1 minute" (ibid.), and the spell speak with dead has a casting time of 10 min. Sigh.
³ For syringe-like projectiles that deliver potion effects and that can be thrown, see the prestige class alchemist savant (Magic of Eberron 53–7) and that class's ability to brew magic spellvials.