[SalesForce] Validation Rule: Only one of three fields can be filled in

Use a hierarchical custom setting.

The setting just has two fields.

Location -> Lookup to user/profile
Skip_Validation__c -> Set it to true to skip validation.

Update your validation rules to not fire when Skip validation = true for the custom setting.

&& NOT($Setup.MyCustomSetting__c.Skip_Validation__c)

As the last step, your trigger shuts off validation at the start and enables it at the end.

MyCustomSetting__c tempSkipValidation = 
  MyCustomSetting__c.getInstance();

if(tempSkipValidation == null) {
  tempSkipValidation = new MyCustomSetting__c();
}

tempSkipValidation.Skip_Validation__c = true;
upsert tempSkipValidation;

//WHATEVER TRIGGER NORMALLY DOES HERE

tempSkipValidation.Skip_Validation__c = false;
update tempSkipValidation;

UPDATE

You should be able to accomplish this will a workflow and a rollup summary field (with the assumption that you won't be changing Product Family on existing Products).

If you create a new Text field on your OpportunityLineItem to hold the Family value, you can populate it in a Workflow Field Update. You can make this hidden from the user.

Once you have that populating for your new Opportunity Products, you can use a rollup field to count the number of Opportunity Products that meet the criteria you define (Family = Software).

Then you can base your validation rule on this field being 0 or 1+.

For existing OpportunityLineItems, you can either manually populate the field via a DataLoad or Anonymous Apex.

ORIGINAL ANSWER:

You won't be able to create a validation rule on an Opportunity that references values of an Opportunity Product because it can't tell which Product to validate - You can only reference parent records in a validation rule (so for Opportunity, you could reference Account).

You'll likely need to write a trigger on the OpportunityLineItem that check the Products Family and updates a checkbox (or some field) on the corresponding Opportunity that there is a product that belongs to that family. If you go down this path, you'll want to re-verify that value every time an Opportunity Product is added or deleted (you may have delete the last Product of Family X, or you may have deleted one of two, etc).

Once you have that checkbox field populated on the Opportunity via a trigger, you'd be free to include it in a validation rule.