Access dom element from in experience builder

The error message which you are receiving means that at the point when this code is executed, there is no element rendered in the browser that can be found with the id of menuLink. When you manually execute the statement in the browser console, you're doing so after the entire page has been rendered and menuLink is now in the page context. You will probably find that this script is executing before the DOM is fully loaded even though you've got it at the end of the VF markup. (What tags did you use to do this? <apex:includeScript?)

If you're using <apex: id="yourName" tags for these elements now or in the future, the following may be useful.

Salesforce documentation about accessing the Id as it was rendered in the DOM.

Best Practices for Accessing Component IDs

You don't mention where this ui.js file is stored but depending on the answer to that question, this answer may need to be adjusted to pass references rather than using merge field syntax (which only works within the VF page itself).

If you need to get the generated unique id for the node on the page, you should use the {!$Component.yourTagID} merge field syntax to do so. This mechanism will give you the full id attribute of the element in the page as it was rendered in the browser.

If the path to the component cannot be found as you wrote it, your path is likely incorrect or ambiguous in the page context. It will not throw an error when this happens, it will just return an empty string. i.e. var myVar = ''; where you expected var myVar = 'j_id0:j_id2:myThing'; You can verify what's happening by looking at the source in the browser.

For instance:

(function (window, document) {
    var layout   = document.getElementById('{!$Component.layout}');
    var menu     = document.getElementById('{!$Component.menu}');
    var menuLink = document.getElementById('{!$Component.menuLink}');

or something more targeted to a specific path from the <apex:page id="myPage"> to the element itself:

(function (window, document) {
    var layout   = document.getElementById('{!$Component.myPage.myForm.myPageBlock.layout}');
    var menu     = document.getElementById('{!$Component.myPage.myForm.myPageBlock.menu}');
    var menuLink = document.getElementById('{!$Component.myPage.myForm.myPageBlock.menuLink}');

or if you want to pass the DOM id as a function param:

(function (window, document, layoutId, menuId, menuLinkId) {
    var layout   = document.getElementById(layoutId);
    var menu     = document.getElementById(menuId);
    var menuLink = document.getElementById(menuLinkId);

    // check for nulls

    // content trimmed for brevity

}(this, this.document, '{!$Component.layout}', '{!$Component.menu}', '{!$Component.menuLink}'));

In Lightning I would be much more inclined to create this list through an aura:iteration at the component markup level, bound to an array that you can set, rather than appending strings to innerHTML. However if you still want to do it this way...

The innerHTML property is a string, not a reference. You are assigning it to a variable, building it up and then never sending it back. (Unless you have left out part of your code?) So try...

    var family=actionResult.getReturnValue();
    var famLength=family.length;
    var listDOM = document.getElementById("familyList");
    var j= '';
    j=j+'<li id="menu-40-0" class="slds-dropdown__item"> <a href="javascript:void(0);" role="menuitemradio"> <p class="slds-truncate">';
    j=j+'<c:svg ariaHidden="true" class="slds-icon slds-icon--selected slds-icon--x-small slds-icon-text-default slds-m-right--x-small" xlinkHref="/assets/icons/utility-sprite/svg/symbols.svg#check"/>'; 
    j=j+family[0]+'</p> </a> </li>"';
    listDOM.innerHTML += j;

Also if you are not familiar with the shorthand increment/concatenation operator +=, in JavaScript j = j + (expression) can be written more compactly as j += (expression).

Also be aware that inserting c:svg into the markup is probably not going to work because Lightning XML is converted to scripted elements at the server level and not at the browser level.

So what would a more data-based Lightning solution look like? Well in your markup you could have something like:

<aura:attribute name="family" type="String[]" />

<ul>
    <aura:iteration items="{!v.family}" var="item">
        <li class="slds-dropdown__item">
           <a href="javascript:void(0);" role="menuitemradio">
               <p class="slds-truncate">
                   <c:svg ariaHidden="true" class="slds-icon slds-icon--selected slds-icon--x-small slds-icon-text-default slds-m-right--x-small" xlinkHref="/assets/icons/utility-sprite/svg/symbols.svg#check"/>
               {!item}
               </p>
           </a>
        </li>
    </aura:iteration>
</ul>

And then your controller would simply populate the array:

component.set('v.family', actionResult.getReturnValue());

Or merge the new items into an existing array, creating a new one if none is there:

var family = component.get('v.family') || [];
var newFamily = family.concat(actionResult.getReturnValue());
component.set('v.family', newFamily);