The best way to write this sentence is to make it simple and direct by eliminating the somewhat awkward-sounding phrase using "both" (in any version): "That's an interesting and challenging problem."
This says the exact same thing and is much more to the point. I spent many years on a newspaper copy desk and would strongly recommend grammar and composition guidelines found in Strunk & White's "The Elements of Style." One of their Elementary Principles of Composition is "Omit needless words."
"..the both parties.." is incorrect. Here, both has already qualified parties and is clear what/who is being referred to. Nevertheless, people use it in an informal way.
Shouldn't it be "both (of) the parties" or "both parties"?
Both your options are correct. "both the parties" is short for "both of the parties" and is completely acceptable. On the other hand, the seems to be extraneous and hence, can be removed altogether, just leaving us with "both parties".
Best Answer
Both are fine, and there's really little difference between them. Other examples:
In some cases one sounds better than the other, but I think this is often what is familiar and preferred rather than related to any grammar rule.