Prepositions – In/On Prepositions for Front or Back-End in Computing

While your sentence

I think we really ought to leave that up to the people we're assigning the label to to judge.

is not grammatically incorrect, it would be considered inelegant by many writers, especially because it involves a duplicate preposition. (It is not uncommon to find "...had had..." in formal contexts and this is not usually frowned upon.) So, you are right your construction is not stylistically preferred. Also, there should not be a comma between the duplicates.

Here are possible revisions in increasing order of elegance:

• I think we really ought to leave that up to the people to which we're assigning the label to judge.

• I think we really ought to leave that judgment up to the people to which we're assigning the label.

• I think we really ought to leave that judgment up to the people whom we're labeling.

• I think we really ought to leave that up to the people we're labeling to judge.

• I think we really ought to leave that judgment up to the people we're labeling.

The 'rule' for prepositions is that they should be placed before their complement, hence the name - pre-position. Failure to do this (and the 'never end a sentence with a preposition' rule...) is called preposition stranding.

Having said that, the 'rule' that prepositions should preceded their complement has no real basis, and was probably made up by John Dryden when he grumbled about Ben Jonson writing "the bodies that those souls were frightened from".

Maybe because of this, preposition stranding in English is very common:

• in Wh- questions, because the preposition complement is the wh-, which has been fronted to make a question - "He talked about what?" > "What did he talk about?"

• in relative clauses, again because the preposition complement gets moved - "I told you about this restaurant" > "This restaurant is the one [that] I told you about"

• in passives/false passives - "He has slept in this bed" > "This bed has been slept in."