Yes, you nailed it correctly. The comma is what makes the difference. As oerkelens says, It was a pity is an independent sentence and if that is the case, the previous sentence should end before it begins.
This tricky question uses comma and then asks for the word to put. In simple words, the first part of the sentence gives us the gist and the latter is simply adding the emotion or sympathy to it. What we call supplementary relative clause is used here and it is often set off by putting comma and then which. One good way to identify this is removing the latter part of the sentence which was pity. Even after removal, the sentence's meaning is intact and this does not happen in case of integrated relative clauses. That said, removing which part in such examples will make the sentence looking ungrammatical/weird.
OxfordDictionaries gives an example of a restrictive relative clause: The coat that/which Dan had on yesterday was new. Remove which part and it won't look a proper sentence.
So, to answer you in short, the word is which because it introduces supplementary relative clause i.e. additional meaning. The comma before the word confirms it.
Let's review. What you are trying to express is:
John lost his friend. John was crying for his friend.
Of course, such repetition is undesirable, so English has a tool you can use, called a relative-pronoun. Relative pronouns include who, whom, that, and which. Of those choices, who and whom can be used to refer to people. Who is used for subjects; whom is used for objects.
You probably know all that already, but just need advice in applying the rules.
The pronoun you want is the one in the objective case (whom), because John lost his friend. (Using the subjective case pronoun who would mean that the friend lost John, which is not what you want.)
Remember, also, that relative pronouns help you refer to a noun that would otherwise be repeated. Relative pronouns don't stand alone. The correct sentence should be:
John was crying for the friend whom he lost.
That said, even native English speakers often fail to distinguish between subjective and objective cases correctly. Therefore, you have to be careful about parroting patterns that you see and hear, as you may be copying incorrect examples. On the other hand, you are also unlikely to be stigmatized for choosing the wrong case in everyday speech.
Best Answer
"I pity those who lost their money in gambling."
This is correct.
"I know the first one is correct, but I think there is nothing wrong grammatically with the second sentence."
I'm assuming you are a learner of English. You'll not have heard the second sentence used, so a general rule is: Don't use it. Even saying it once is slightly contaminating your understanding with an error you will have to practice to unlearn.
Aside: If as a learner you see the entire English language left to learn as a pile of coal in front of you which you have to shift, it's enormous - and there's in all of us a temptation to find a certain number of general rules so as to reduce this down to a smaller number. I say, resist this urge: not only ie English firstly too irregular for regular rules to help much; but secondly you'll learn the expressions along the way as you practice - and thirdly, Looking for rules is wasting valuable time when you could be speaking reams of English!