[SalesForce] Upload attachment in VisualForce without reloading the whole page

As the error indicates, you simply can't do this. I'd imagine its a security feature, as the inputFile is allowing uploading of a file from the user's computer. You could use an iframe to contain the input file component, which means that other frames would be unaffected, but from the technical point of view it has to be a full refresh of the containing page.

You can use an actionRegion to define a scope for a commandButton or commandLink. This is sufficient to allow inputFile to happily co-exist with reRender attributes. Note that the actionRegion must contain all the information necessary to update the page, because only the components inside the actionRegion is processed by the server. Here's a simple example:

Controller

public with sharing class renderFile {
    public transient Blob fileBody { get; set; }
    public String contentType { get; set; }
    public String fileName { get; set; }
    public string[] items { get; set; }

    public renderFile() {
        items = new string[0];
    }

    public void save() {
        Document d = new Document(FolderId=UserInfo.getUserId(), Name=fileName, Body=fileBody, ContentType=contentType);
        insert d;
        items.add('Uploaded '+d.Name+' as ID '+d.Id);
    }
    public void reRender() {
        items.add('Item: '+(items.size()+1));
    }
}

Page

<apex:page controller="renderFile">
    <apex:form id="form">
        <apex:pageBlock >
            <apex:pageBlockButtons location="bottom">
                <apex:actionRegion id="area51" renderRegionOnly="false">
                    <apex:commandButton value="Something" reRender="form" action="{!reRender}"/>
                </apex:actionRegion>
                <apex:commandButton value="Save" action="{!save}"/>
            </apex:pageBlockButtons>
            <apex:pageBlockSection >
                <apex:repeat value="{!items}" var="value">
                    <apex:outputText >{!value}</apex:outputText>
                </apex:repeat>
                <apex:inputFile contentType="{!contentType}" fileName="{!fileName}" value="{!fileBody}" />
            </apex:pageBlockSection>
        </apex:pageBlock>
    </apex:form>
</apex:page>