Learn English – word meaning a problem that has to be solved in order to work on another problem

You seem to have two different kinds of examples there. In the first example the solution set is ill-defined:

A set is well-defined if any given object either is an element of the set, or is not an element of the set

but it's never clear whether a given number a is an element of the solution set to the equation; whether it is or not depends (as you state) on the value of b.

In your second example, the solution set (that is, the set of solutions to the problem posed) is well-defined, but empty because the conditions are impracticable:

(Definition 2) not practicable : incapable of being performed or accomplished by the means employed or at command

One word that comes to mind is undo, which carries the meaning of rolling back to a point before an action was carried out. To cover the whole process, though, you would still need to use another word like fault-correction.

There is also the more general term, process, which admittedly does not specifically indicate that there was a problem. However, your general situation is covered by this word.

It depends a little on how you plan to use the term.