[SalesForce] the validation rule for a text field that shouldn’t contain data if the correct picklist value has been selected first

I created this Validation Rule that when a user selects Yes from the, Any_other_liens_filed_against_you__c picklist but tries to save the record without completing the text field, "If_Yes_What_liens_are_field_against_you__c", the error message will prompt the user to go back and complete the text field.

AND( 
    ISBLANK( If_Yes_What_liens_are_field_against_you__c ), 
        ISPICKVAL( Any_other_liens_filed_against_you__c , "Yes") 
)

What I'm trying to solve is, what if the picklist field, Any_other_liens_filed_against_you__c, doesn't need to be selected BUT a user decides to place any random text into the "If_Yes_What_liens_are_field_against_you__c" text field and is allowed to save the record. This will cause confusion later down the road since the picklist and the rouge text info entered won't relate to each other.

I tried modifying the rule to this and other variations, and the rule just let the record save when the picklist value was blank but included text into the text field.

AND( 
    ISBLANK( If_Yes_What_liens_are_field_against_you__c ), 
      ISPICKVAL( Any_other_liens_filed_against_you__c , "") 
)

I supposed I'll need to rewrite my original VR when No is selected from the picklist too?

What should the validation rule be modified to that will no allow a user to enter text into the text field unless the picklist value has been selected as Yes? and if a user selects No, the text field must contain a value.