[SalesForce] Restrict the number of children in lookup relationship

If its a lookup you will need a simple trigger to do this in real time

trigger onParentObjectDelete on CustomObject__c (before delete){
List<Id> idsToQuery = new List<Id>{};
for(CustomObject__c a: Trigger.old){
    idsToQuery.add(a.id);
 }

//query all child records where parent ids were deleted
ChildObject__c[] objsToDelete = [select id from ChildObject__c where ParentId__c IN :idsToQuery];

delete objsToDelete; //perform delete statement
}

This is if you are planning to do a real time delete .Also in this case be sure to check with business what if someone undeletes a record from the bin.May be a trigger to handle this as well

Also if you don't need a real time delete ,a batch process to fetch orphaned childs(ParentId is null) and delete .

There are only technically two types of relationships between objects in Salesforce, master-detail and lookup. Though in reality when coding and in the UI you'll find both have some common behaviours and ways in which they can be used.

So Salesforce does not technically require you to do anything different regardless if your lookup references another object or itself, both are the same field type, lookup. So the difference is more the use case your using the field for than it's type. Note that the Standard object Account uses self references to implement Account hierarchies, parent and child relationship.

This topic in the documentation Relationships Among Objects is a good place to start reading more, and i noticed also covers a cool use case of master-detail relationships relating to many-to-many relationships. Hopefully this sheds a bit more light on the topic for you, but if you need more feel free to explain a little more in your question about your use case.

Extract from the above linked help topic...

Relationships associate objects with other objects. For example, a relationship can link a custom object to standard object in a related list, such as linking a custom object called Bugs to cases to track product defects associated with customer cases. To view the parent and child relationships among standard objects, see the ERD diagrams in Data Model.