[RPG] Is the extra damage from Assassin’s Strike maximized on a critical

From the PHB (p. 196):

Critical Hits
When you score a critical hit, you get to roll extra dice for the attack’s damage against the target. Roll all of the attack’s damage dice twice and add them together. Then add any relevant modifiers as normal. To speed up play, you can roll all the damage dice at once.
For example, if you score a critical hit with a dagger, roll 2d4 for the damage, rather than 1d4, and then add your relevant ability modifier. If the attack involves other damage dice, such as from the rogue’s Sneak Attack feature, you roll those dice twice as well.

The example given answers your first question perfectly.

For the spider, the secondary damage is from poison, so is not inherently part of the attack. This can be seen by the fact that the damage doesn't depend on the attack roll, but a separate saving throw.
Critical hits represent hitting a vulnerable area. With poison, it doesn't really matter where you are hit.

Critical hit means, for each damage die you would roll for a non-critical, you roll it twice instead of once.

Death strike means you take the damage you would have dealt if it were not a death strike (which would still be a critical hit), and you double that.

The order of operations is:

1. Decide which damage dice, and how many of them, to roll.
2. Roll them and add them up.
3. Add any relevant damage modifier, e.g. your dexterity modifier or the +10 modifier from the Sharpshooter feat.
4. Multiply that sum by any relevant multiplier, e.g. multiply by 2 for Death Strike.

Therefore:

• Normal rapier damage = 1d8 + dex mod
• Rapier damage + sneak attack = 1d8 + 10d6 + dex mod
• Critical (rapier damage + sneak attack) = 2d8 + 20d6 + dex mod
• Death strike (critical (rapier damage + sneak attack)) = 2 * (2d8 + 20d6 + dex mod)

Assassin is a highly specialized class archetype, and it is a huge investment to put 17 levels into it that you could have put into other classes. The reward for that investment is that the assassin is unmatched in the amount of damage it can inflict on a surprised target. If you ask me, an average of 168 damage (on a failed save) is the least the game could do to reward that investment.