If a creature doesn't want to be noticed casting spells then, yes, the creature must devote resources to doing exactly that, and that includes Enchantment spells. By eliminating the inexpensive material components with the feat Eschew Materials, somatic components with the feat Silent Spell, and verbal components with the feat Still Spell one can cast a spell nearly undetectably--from onlookers1 (an affected creatures still gets that tingling feeling when it succeeds on a saving throw versus a hostile spell and still has a chance to identify the spell cast on it via a Spellcraft skill check). But, yeah, it's difficult to make casting happen stealthily. And it should be.
Your question focuses on the lack of utility of Enchantment spells when the target knows such a spell's been used on it. I'd be more concerned about the increased utility of Evocation spells were they employed in this nearly-impossible-to-determine-the-source fashion.
The Spell Charm Person
The expectation is that spells charm person et al. actually are cast right out in the open--at an approaching potential enemy. And, if the enemy fails the saving throw he very well might ask, "What did you do?" if the creature is unable to identify the spell as it was cast (and most creatures can't). That's when the caster says, "Just a detect spell; don't worry about it," and, as the enemy is now you're friend, he doesn't.
However, I get the sense that your group might want to use the spell charm person against shopkeepers, city guards, buxom barmaids, and other folks in the middle of town in an effort to get free goods, to encourage a slap on the wrist instead of the gallows, to make them pay more attention to you than the muscleheaded barbarians, or whatever. In those cases, casting a spell can be problematic.
I'd urge casters to role-play these events, using minor magics like the spell prestidigitation to instantly clean the shopkeeper's store, shine the guards' shoes, or conjure a gift for the lady instead of whipping out the charm spells. Harnessing the power of the universe to get a 10% discount on swords, to avoid paying a parking ticket, or to get a lady to pay attention to you2 is... excessive.
Casting in the Campaign
I can't know the kind of campaign you're running, but in my campaigns folks are familiar with magic even though most lack ranks in the skills Spellcraft and Knowledge (arcana). Intelligent creatures in civilized areas view spellcasting the same way many contemporary societies view taking out a pistol and waving it around. You want to clear the bar? Start casting a spell. You want folks to run for cover? Start casting a spell. Spells are worse than firearms--nigh undetectable until used and can level cities.
Polite casters in my campaigns quickly learn to warn onlookers that they're casting a spell, explain what the spell is (making Bluff skill checks if necessary), and provide those who want to the opportunity to leave. Impolite casters get grappled or stabbed by folks claiming self-defense.
The campaign needs to establish guidelines for casting spells in populated areas because, yeah, spells are awesome... if you're the caster. Spells are death via a handful of bat guano, throwing horns, and yelling fireball! to everyone else.
- There doesn't seem to be Pathfinder equivalent of the Dungeons and Dragons, 3rd Edition feat Invisible Spell (Ci 61), but I have no doubt there will be. There is, however, the third-party feat Secret Spell, which, while not making the spell's effects invisible, makes stealthy casting easier.
- Depending on the lady, obviously.
Setting Expectations
Whenever I as GM am dealing with multi-PC/NPC conversations, I first remind my players that I am a single-thread processor and can therefore only handle one conversation at a time, then proceed to deal with them one at a time as appropriate for the configuration of speakers.
1. One-to-Many Conversations
The easiest way to handle this situation is to enforce multiple one-to-many conversations (as opposed to a single many-to-many conversation). This allows multiple PCs who all want to talk to the same NPC to do so as a group; then the PCs who want to talk to another NPC to do so. For example, if John, Jade, and Rose all want to talk to the NPC Jack, I will hold that conversation all at once. Dave, Jane, and Roxy, who want to talk to the NPC Mayor, wait their turn and have their conversation separately after I've finished with the first one.
I also do the reverse: John wants to talk privately to both the Mayor and Jack, so I handle that conversation all in one go, while the other PCs wait their turn for their own conversations.
This is probably the ideal way to handle a bar scene like the one you describe, since bar chatter is typically several one-on-one or one-to-many conversations. While it feels a little artificial, since in-game the conversations would all be taking place at once, it allows you as the GM to focus on each conversation and give the involved PCs your full attention.
2. Many-to-Many Conversations
The best way to handle many-to-many conversations as a GM is to cheat and turn them into one-to-many conversations. When multiple PCs are talking to a group of NPCs, pick one NPC to be the group's "spokesperson", who does most or all of the talking. Occasionally, where plot-relevant, another NPC might interject a few times, but for the most part, this allows a many-to-many scene to actually be a one-to-many conversation.
For example, if the whole party wants to talk to a rival party, I will pick the most talkative of the rivals, and have them do almost all the talking. The rest of the rivals are present, but silent in the background. If I want to illustrate that there's an ideological rift in the rival party, then I might have a second NPC jump in while the spokesperson is talking, but that dialogue is handled as an NPC-to-NPC dialogue. If the party wishes to speak to the second NPC (perhaps because they think she'll be more sympathetic to their arguments), then the first NPC falls silent and the second NPC becomes the spokesperson. Again, this enforces the rule of one-to-many conversations and prevents the GM from being overwhelmed.
3. Multiple Important NPCs
Having multiple important NPCs in a scene means the GM needs to make a choice between making the scene a "cutscene" wherein the NPCs converse with each other, or making the scene a conversation between the NPCs and the PCs. In other words, is the point of the scene to convey a discussion between the NPCs, or is it to allow the PCs to talk to the NPCs?
If it's the former, then the scene can be a brief "cutscene" where the NPCs talk to each other and the players listen. Then, after the NPCs' different viewpoints have been showcased by the cutscene, the PCs can pick one of the NPCs to have a one-to-many conversation with, as described in #1 above.
If it's the latter, then the scene can make use of the spokesperson technique. Even groups with multiple powerful players tend to have a single "speaker", even if that person isn't the leader in any sense of the word. Not everyone is equally talkative, and the most talkative NPC will dominate the conversation with the PCs, leading to (again) a one-to-many conversation with a few interjections by the other NPCs, as described in #2.
As an added bonus, you can use subtle cues in this sort of scenario to showcase group politics. For example, if the spokesperson is respected, the rest of the group falling silent and allowing her to speak for them will indicate this. Alternately, you can note that although one person is doing all the talking, other NPCs are rolling their eyes, shaking their heads, or exchanging meaningful glances behind his back. This indicates that the spokesperson might be the most talkative, but not the most respected, member of the group - and may give the PCs a clue that they need to draw out a different person as the speaker.
Again, the key is to remember that you the GM are a single-threaded processor. No matter what, you cannot effectively have three or four or five conversations as different NPCs with different PCs at once. The best you can do is to cheat in ways that allow you to only ever have one effective conversation going at once.
TL;DR
Remind your players that you are one person and can only have one conversation at a time. Enforce one-to-many conversations by handling one conversation group at a time, and/or by using a spokesman for multiple NPCs.
Best Answer
Treat it as a contest
The players are effectively trying to use a level 0 cantrip to produce the same result as a higher level crowd control spell like Fear or Mass Suggestion. Obviously, this is game breaking.
We had an arcane trickster rogue in our group who tried this kind of thing a lot. The way our DM handled it was to treat it as a contest. The caster is trying to deceive the goblins into thinking that they should not attack the party using the effect of a spell.
The caster casts the cantrip and then the makes a Deception or Persuasion or Intimdation skill check against the goblins' Insight checks.
You could have the goblins roll a single combined check or individually. If you do the latter, any goblins that are not convinced by the cantrip's effect can rally their comrades by saying something like "It's a trick! That's not what Maglubiyet sounds like!" (or teaches, whatever).
Rolling like this gives the goblin pack a high chance of resisting the cantrip's intended effect.
Alternatively, you can split the difference and have some of the goblins sit in one place for a round, quivering in their boots, while the unaffected ones charge forward. The scared goblins buck up the following round and join the braver ones.
Doing it this way allows you to keep the solution "within the game." You don't have to have a meta discussion with your player(s) to tell them that you won't let them do something. That's never fun.
As Kyle points out in his answer: At 10 intelligence and 8 wisdom, Goblins aren't actually "dumb." They're literally of average intelligence with a tinge of obliviousness, and in a world where magic is quite commonplace.
They're probably familiar with many illusion cantrips and Thaumatury, Minor Illusion, and Prestidigitation have a verbal and/or somatic component as part of their casting. If the spellcaster is visible, they'll see or hear him/her casting the spell first. Mighty suspicious, that.
Also, don't forget that, if a character is going to fake the god of the goblins speaking from on high, s/he will need to know the goblin language first!