Most has two closely-related meanings.
1 the largest in number or amount
2 more than half / almost all of somebody/something
As that OALD link says, the most is often used for the first meaning - but informally, the is often omitted. It's never used with the second meaning.
Suppose The Queen of Hearts were to ask "Who ate my tarts?"...
A1: "The knave ate the most"
A2: "The knave ate most"
If several courtiers ate one tart each, but the knave ate two, then either answer could validly be given, since he ate more than anyone else (but note that A1 is a "very slightly formal" usage).
But if the knave actually ate more than half (but not all) of the tarts, only A2 is correct. Though the Queen might not understand that intended distinction, since the could be omitted anyway.
It's worth making the point that the is never included when most simply means very...
"I am most grateful for your help"
...but interestingly, whereas in...
"There were 50 people there, at [the] most"
... the is normally omitted, it's almost always included if we add very as an intensifier...
"There were 50 people there, at the very most"
In both of the bolded cases, the noun does not refer to particular cases of the things mentioned, but to the classes of those objects in general. This is the OED's second sense of "the":
II. Referring to a term used generically or universally. With a singular n[oun]....
19b. Generally, with the name of anything used as the type of its class; e.g. with the names of musical instruments, tools, etc.
This usage is commonly seen with musical instruments, where the whole class of instruments is referred to.
Bill plays the clarinet.
Could mean that Bill is playing a particular clarinet right now, but more commonly will mean that Bill has the ability to play the class of instruments generally known as "clarinet".
Bill is playing a clarinet.
This definitely means that Bill is in the act of playing a singular clarinet. (Not a particular clarinet that has otherwise been identified, but any given one.)
There were five different instruments on the table. Bill grabbed the clarinet.
In this case, a particular clarinet is being referred to (and its identity has been previously established in context, as one of the five instruments). You could think of the word "clarinet" as having a different meaning in this sentence than in the first: here meaning an instrument of the class, and in the first sentence meaning the class itself.
To use "a beach" in this case would not be incorrect, but I think would be less common in a context like this; the different sense would be the same as the difference between my first two clarinet examples. In this case, the listener (or at least this native AmE listener) would get a vague sense that the speaker is being unnecessarily specific: the context is in broad general terms about "most vacations" "in years gone by", so it makes sense to refer to "the beach" in the sense of "the class of objects known as the beach". "A beach" would almost give the sense that all those people on all those vacations had been going to a particular beach (but from context it would be clear that that was not the case, so not really an error).
On the other hand, I think one would never use "a family" in this case. I think the reason is that everyone has one family (more or less), so it would seem strange to make a non-specific reference: it would give the sense that one might go with someone else's family. It would be fine here to say "one's family", though. (I don't think there's a nuance difference between the two because the class of potential families one might go the beach with is a class of one.)
Best Answer
Either could be correct depending on the context, but "with family" (with no article) is most generally correct. "Family" without an article can refer to any type of relative and to the relatives of any number of people. For example:
Here we are talking about many families of many employees. If we said "the family" in the above context, we would be implying that all the employees are related, which is possible but unlikely.
I would use "the family" only if you are referring to a specific family. For example:
In this case we really do mean one specific family, that of the father and the child.
Note that in this context we can also drop the article without really changing the meaning:
Without the article, we are again referring to family in general. However, we still understand that we are talking about the specific family of the father and child. As such, dropping the article is a safe choice in both situations.