Focus Fire: Don't spread out damage any more than you have to. It's far more effective for most of the monsters to concentrate on a single target. If you can get a player down, that's that much less damage per round Team Monster is taking, and the party is suddenly on the defensive to try to save the downed player (only particularly nasty monsters should take the opportunity to coup de grace an unconscious player). Which player should you target first? That leads to the next question...
Target the Squishies: Monsters should always focus fire the available target with the least current HP & lowest defenses. That's who they're most likely to successfully take down. Brutes in particular benefit from attacking low-defense targets, since they have high damage but low attack. Note that I said available target, though. If a sticky defender is guarding a doorway, don't try to rush past him to get at the wizard. But if several monsters can get at the wizard...
Weigh Cost/Benefit for Marks: If a monster is marked, you shouldn't have it automatically attack the fighter or always ignore the mark; only ignore the mark when it's worth it to do so. How much damage as a percentage of the max HP are you likely to do to the defender, versus how much are you likely to do to whoever else you could be hitting? If you're probably going to be more effective against someone else, how much HP do you have left and how unpleasant is the defender's mark punishment? Remember that any target that has defenses that are lower than the defender's by more than 2 is easier to hit than the defender even when the monster is marked.
Flank Them: Have a couple lurkers or skirmishers come in behind the party (they were hidden, used secret doors, or whatever). It's very hard for a single defender to stop attackers coming from 2 different directions. Getting a couple melee units in among the party's ranged backline makes it much harder for the party to focus fire, which means monsters last that much longer.
Control, Control, Control: Make good use of controllers and leaders. Unlike damage, control should be targeted at the characters it will be most debilitating on. Immobilize/slow are for melee attackers, blindness & daze should be dropped on the biggest threats, daze shuts down most defender mark punishment (except paladins), and so on. Control has a huge effect on the difficulty of a fight.
Protect Your Squishies: Artillery and ranged controllers, and when possible leaders, should be stationed towards the back. Use soldiers and minion packs to tie down melee players so that they can't get to your ranged units and to keep defenders busy so lurkers, skirmishers, and brutes can slip past to the party's weaker ranged units.
Use the Terrain: Have your ranged attackers hide behind pillars (remember, if you're directly behind a one-square pillar, you have cover but enemy targets don't). Remember that monsters can use each other as cover if necessary. Position your frontline and backline to maximize the amount of difficult or dangerous terrain melee players have to go through to get past the frontline to the backline.
Alpha Strike: Winning initiative helps; if your group simplifies initiative by having all the players go in order then all of Team Monster go, or vice versa depending on which team won initiative, then winning initiative really helps. Monsters should use any encounter or recharge powers they have early & often. It's better to use it at the start of the fight in a slightly less-than-ideal targeting scenario than to save it for later and end up dying without using it. Even if you can't alpha strike (and it's admittedly much harder for Team Monster than it is for the party), it's hugely beneficial to prevent the players from alpha striking. To give an example, with a proper alpha strike the group I play with have often killed or incapacitated (prone + 1 square away for melee enemies, prone & around a corner for ranged enemies, blinded, etc.) over half the monsters in an encounter before any monsters got to act; the record is 7 monsters (no minions) getting a total of 3 attacks before the fight was over (not 3 per monster; Team Monster as a whole got in 3 attacks before the fight was over, including misses).
Retreat & Reinforce: If a group of monsters that are part of a larger faction in the area are getting their butts kicked, have a few of the remaining ones retreat to try to join an allied group a room or two over. This not only makes the next fight harder without technically altering XP budgets, it has the added bonus of reducing the boring mop-up phase.
Encounter Building: Picking the right monsters to use can make a huge difference in an encounter's difficulty, even when your options all have the same total XP budget. Here are some things to remember:
Frontline & Backline: Try to have a mix of frontline monsters that are designed to keep melee players busy and backline monsters that are designed to hit hard. You will especially need a good frontline if most of your "backline" monsters have to stay in melee range (brutes, some skirmishers and lurkers).
Monster Synergy: Pick monsters whose abilities complement each other. If you have a skirmisher/lurker that does bonus damage to slowed foes, find a controller with a nice AoE or save-ends slowing attack. Daze & prone is another nice combo, especially against melee players; immobilize & forced movement is also strong against melee players. If you have several monsters that give a bonus to adjacent allies, go heavy on melee monsters so more of them can benefit.
Terrain Synergy: Are there a lot of hazards players will have to avoid? Bring lots of forced movement to try to shove them into it. Lots of terrain sources of elemental damage? Pick monsters that resist or are immune to that damage, so they can move through it with impunity. Lots of difficult terrain or features that block movement but not line of sight? Go heavy on ranged monsters so you can pincushion the players as they move to engage.
Control Is Good: Control effects, used properly, will generally be more effective than damage if you have a couple high-damage monsters to take advantage of them and you have some blockers to keep your control monsters from dying the instant the players notice them. Generally speaking, a good ratio is 30% blockers, 20% damage, 50% control. This doesn't mean use nothing but controllers, though: remember that lots of non-controller monsters offer decent control effects, especially as you start getting into paragon tier (by epic, almost every non-minion has some sort of control).
Don't Use Pre-MM3 Solos: This one is admittedly a little specific, but it's good advice nonetheless. Before MM3 solos depended entirely on having lots of HP to survive, but this meant a typical solo fight entailed the party stun-locking the solo for a round or two while the strikers went nova on him, typically ending the fight with little or no damage inflicted on the party. I've actually seen a mid-heroic solo killed in the first round by a single player (admittedly he critted with a daily power, action pointed, and then critted with a second daily power, but still). MM3 and later solos tend to have more ways to negate or ignore status effects (especially stun/daze) and more ways to interrupt multi-attack chains, thus giving them considerably more survivability.
Use Higher Level Monsters When Possible: All else being equal, 4 level+1 foes will generally be a harder fight than 5 level-1 foes. Monster and terrain synergy definitely trump this one, though.
That said, not every monster is a tactical genius. Animals in particular will tend to just concentrate on whoever is closest or doing the most damage to them, and will almost always ignore defender marks (though most predators will attempt to start the fight by going for whoever looks the weakest). Dumber and more cowardly enemies (goblins, kobolds) will usually attack in a big pack, and will almost always respect defender marks.
Casting a spell takes a magic-user's entire round. They may not take any other actions, including moving. The number of segments listed for a spell is only used to figure out when during a round the spell completes, which can be important for figuring out whether the spell is interrupted (and spoiled).
For a complete breakdown of the spellcasting process during combat, see page 65 of the DMG.
Learning AD&D for the first time from the original books takes a ridiculous amount of dedication due to the bizarre organisation, the rambling discoursive style, and the way a single rule can be spread across multiple chapters or even across the three core books. I certainly didn't learn the fundaments that way – like most AD&D players, I learned by playing with people who already knew how to play.
To learn directly from the books requires dedicating a lot of time, and using that time to read and re-read them, from cover to cover and in bits and pieces, while putting them to use in an ongoing campaign. A large tolerance for doing it "wrong" for a good while is helpful.
As an alternative, I can recommend reading OSRIC, which is a retro clone of AD&D. Its advantages include being more concise (it's only about 40 pages of rules plus 80 pages of spells), organised more like how we expect rules to be organised today, and free. It makes a good companion to the original books, and can help clarify rules confusion since it's much easier to find everything.
The conciseness of OSRIC is also its downside: it lacks the rambling commentary that puts the rules in context, and which sometimes provide more nuanced guidance that falls somewhere between "background" and "rules". For example, OSRIC makes no mention of spellcasting preventing crouching when a spell has somatic components. I never knew about that rule(?) until I re-read the several different sections in the PHB and DMG about spellcasting while I was looking for a citation for this answer.
Best Answer
There are a few ways to deal with this:
Give the players sufficient warning before stumbling upon the main Trogs encampment. Perhaps a patrol, or guard post, will give the players indications a larger lair is ahead. If they happily skip in then, let the dice fall where they may. Remember, earlier editions of D&D are not as forgiving as later editions....and it's not that hard to roll up another low level party that will (hopefully) be wiser than the first one!
An intelligent group of players, when seeing that a trog lair is ahead, should be smart enough to scout out the lair. They don't have to face it head on; setting traps, ambushes, raiding the lair and then retreating, etc are all viable tactics. Use of fire can be very helpful, as will missile weapons. Hit and run tactics could prevail against a superior yet not very intelligent foe like trogs.
Reduce the amount (obviously). Perhaps this is a splinter tribe off a larger tribe a few miles away, this has the added effect of helping the players prepare and gain experience (and knowledge) before tackling the main tribe. Maybe the tribe has already been attacked by another (now deceased) group of adventurers that did some damage before they perished, knocking the 50 tribe members down to 25 (or whatever number seems sufficient). Add to the atmosphere by having the players find the first adventuring party's bodies and equipment here and there in the lair (perhaps one or two are still alive to assist the party if they can release them from captivity).
Maybe some extras sign on for the assault. The local town council, duke, or priesthood can supply the party with some help because the trog tribe has become a menace. A half dozen men at arms firing flaming arrows, a duo of priests for healing or spell casting (Hold Person does work on Trogs), a mage with Web and/or Stinking Cloud spells, or a combination of all the above. For added intrigue perhaps one of the hired or supplied men has a secret reason for wanting in the Trog lair, be it a treasure he/she has to acquire, or some bit of information they need in the form of a old map or book.