PICKVALS have a number of limitations that are often very difficult to work with.
In my experience, a rule that refers to a single value picklist can only refer to the current and previous value PRIORVAL of the picklist in a (valuefield, "textvalue") format. If it's a multi-select picklist, then you can use ISPICVAL(CONTAINS(valuefield, "textvalue1", textvalue2,...) or similar syntax (sorry, but I didn't have time to verify this is the correct way to order this before posting).
If you want the "Does not contain" or Boolean negative", you can use NOT(PICKVAL(valuefield, "textvalue") as an argument in your expression. What I've observed is that if you want if new value AND old value conditions type of validation rule, it's often best to set it up like: AND(ISPICKVAL(valuefield,"textvalue"), PRIORVAL("textvalue")). The editor will expect PRIORVAL to have an ISPICKVAL in front of it, but will only want one ISPICKVAL in the expression if there's two PICKVALS contained in the expression. The help on this is rather vague other than to tell you there are limitations on what you can do with PICKVALS.
Sometimes, it's nearly impossible to create a single validation rule for PICKVALS that address what you're trying to accomplish because they have so many limitations in SF. It can be very trying on one's patience and time-consuming. The syntax is quite different to what we're used to working with in APEX and other languages, or so it seems to me.
The problem looks clear to me as you are facing a challenge with the small point in order of execution of salesforce.
To understand this you need to refer to order of execution in salesforce.
Triggers and Order of Execution
If you go to the link you'll find out in step 4 the custom validations are run once again after the before triggers which fails your logic.
STEP 4: Runs most system validation steps again, such as
verifying that all required fields have a non-null value, and runs any
user-defined validation rules. The only system validation that
Salesforce doesn't run a second time (when the request comes from a
standard UI edit page) is the enforcement of layout-specific rules.
To avoid the problem I would request you to not make the checkbox BypassCloseddateValRule, you created to bypass the validation rule to FALSE in the trigger.
You should do this change in a later step after which custom validations are not called, i.e. in STEP 7 (After Trigger) or STEP 11 (Workflow field Update).
Because custom validation rules will not be called after those steps.
STEP 12: If the record was updated with workflow field updates, fires before
update triggers and after update triggers one more time (and only one
more time), in addition to standard validations. Custom validation
rules and duplicate rules are not run again.
I believe this will solve your problem. Let us know if you face any other challenge.
Thank You.
Best Answer
As mentioned by @Stygon I have used roll-up summary field to get the count and used a validation rule to show message if the count exceeds 3.