I would do this with roleplaying and whatever social combat rules your system uses. The level of detail is determined by how important you want the trial to be in your game. If you are not using a system with social combat, you can also model it with "chase" or "extended contest" rules, anything where success accrues over time can probably be kit-bashed into doing the trick.
My problem with Jenga is that you're not building something stronger as you progress, you're making something weaker. It just completely goes against the thing you're modeling.
Here's how I would handle it:
- Determine the skill of the lawyers involved. Or the skill of the PC and NPC who will be the primary speakers at the trial.
- Determine the number of successes required to constitute victory.
- Ask what the players want to do in terms of preparation. Typical activities might include:
- Gathering additional evidence
- Tracking down witnesses
- Bribing, intimidating, murdering, and otherwise influencing judges and / or jurors
- Studying the law
- Finding precedents
- Tracking down the real criminal
- Getting someone else to confess
- Convincing frightened people to testify
Each of those activities could be an encounter or string of encounters. In the run-up to the trial, decide what the PCs get for succeeding or failing in these efforts. Smaller benefits could be modeled as skill bonuses, while larger benefits could be "free" successes to apply.
So if most of the PCs go out to convince people who can confirm the accused's alibi to testify, you can play that out. When they succeed give them a bonus to one or more "lawyer" skill rolls for use during the trial. Meanwhile, the (presumably imprisoned) accused PC is trying to track down the real killer in his prison. If he can get the guy to confess, you can grant them some number of free successes, based on how credible the confession is, etc..
In this way, you keep everyone engaged in the pre-trial process, and everyone gets to see their contributions contribute concretely during the trial.
At the start of the skill challenge, if I don't tell the players that
this is a skill challenge, they are likely to mostly propose
role-played actions that are hard to translate into a skill check,
e.g. shouting out "Hold the thief!"
The players can, but don't need to know which skill/ability to roll. Pure RP actions are just fine even if they don't match a skill. You tell them what they can roll when they describe what they're doing, or whether they need to roll at all. Shouting at a crowded marketplace doesn't take much skill, but is unlikely to stop a skilled thief either, so they'll have to follow up with something else. Eventually they figure out they'll need to do better and use their skills. Also, just announcing the beginning of a skill challenge isn't necessarily a bad idea either.
In response to a general "what do you do?" prompt, some players are
more likely to answer than others. Should I impose rules that ensure
that every player gets a turn in the skill challenge and participates?
If this is a problem in your particular group, limiting rolls per player somehow is appropriate. It doesn't have to be either "free rolls" or "turn by turn" - you can decide that the character who succeeded in the last roll is busy carrying out the successful action and can't help with the next roll.
Also don't forget to mix up challenges with different skills; having a Stealth/Thievery/Streetwise -heavy challenge can be done single-handedly by Ron the Rogue if no other rules are imposed.
How do I handle ideas which I like from players, but which don't
really correspond to a skill? Like in the example above, shouting
"Hold the thief!" sounds like a good idea to stop a thief from running
away in a city. But it would translate badly into let's say a
Diplomacy check, especially if the character isn't skilled in that.
If it translates badly, don't translate it. Not everything they do needs to be rolled. Just tell them the action is impossible, or doesn't achieve the desired effect. If the idea is particularly crafty and clever, you can give them a free success. If the idea is particularly bad (tickling the sleeping dragon) you can give them a free failure. It's always up to you.
What if somebody proposes a good idea which corresponds to a skill
that isn't listed in the primary or secondary skills for that skill
challenge?
Let them do it. The primary and secondary skills listed are just suggestions anyway.
what do I do if the players want to cast spells instead of using skills?
It's the GM's call. Some powers have little combat value and yet see use extensively in the RP segments (for example, the Wizard's Cantrip). You can choose to limit the powers your party can use, and it makes sense as it prevents everything from boiling down to combat, but if you do so make sure it's got a sensible reason, like the thief constantly disappearing from view where they're hard to target.
Once the initial spontaneous idea have gotten used up, how do I
prevent the players from simply checking their character sheet for
their best skills and just proposing those?
I don't consider this a huge problem myself. Skill challenges are supposed to be an opportunity for the players to use their best skills. As long as they describe what they're doing, let them go with it. The example in the guide isn't a misuse of the mechanic - for players who don't know the monster manuals and other guides inside-out it's often difficult to remember which knowledge skill (History, Arcana, Religion, Nature...) matches which topic.
Best Answer
Some things to think about:
These will help you design the difficulty of the encounter in a way that matches your objectives - if every character MUST succeed in ALL challenges, with no help, then you're going to want to make them easier than if the party can match challenges to characters and can get away with losing some of them.
Ideas for skill challenges
STR: Wrestling could be a better fit than fighting. I would allow Athletics rolls instead of straight STR checks. You could also cheat on this one by oiling yourself up to make yourself harder to pin.
DEX: Climbing something (Acrobatics) or evading falling objects (Acrobatics again). You could make this one a race and allow for clever planning of a route to influence the outcome.
CON: Enduring hardship such as frigid water, holding your hand over a candle flame, or holding your breath.
INT: Puzzles are hard to run. One I've used before is the "find the lightest stone" puzzle: Player is given 12 apparently identical stones, but one is hollow. How can they find that stone if they're given a balance scale but only allowed 3 measurements?
WIS: Detect lies (Insight) might work. You could link this one with the CHA challenge - the player gets conflicting advice on how to influence the target, and must decide which is the truth and which is a lie, with the result granting either Advantage or Disadvantage on the CHA check if used.
CHA: Persuade the judges that they're fit to be knights. Or deceive them, if they're not really fit.