[SalesForce] Data Loader Command Line Tool slow

I am trying to automate a data load using the Command Line tools and have everything working smoothly but it is taking 10x longer than if I do it through the Data Loader GUI.

Below is an exert from my process-conf.xml

<bean id="csvUpsertOrderItem"
      class="com.salesforce.dataloader.process.ProcessRunner"
      singleton="false">
    <description>Upsert Transaction Headers into Orders standard object.</description>
    <property name="name" value="csvUpsertOrderItem"/>
    <property name="configOverrideMap">
        <map>
            <entry key="sfdc.debugMessages" value="false"/>
            <entry key="sfdc.endpoint" value="CUSTOM ENDPOINT"/>
            <entry key="sfdc.username" value="USERNAME"/>
            <entry key="sfdc.password" value="ENCRYPTED PASSWORD"/>
            <entry key="process.encryptionKeyFile" value="C:\Program Files (x86)\salesforce.com\Data Loader\bin\key.txt"/>
            <entry key="sfdc.timeoutSecs" value="540"/>
            <entry key="sfdc.loadBatchSize" value="2000"/>
            <entry key="sfdc.entity" value="OrderItem"/>
            <entry key="process.operation" value="upsert"/>
            <entry key="sfdc.useBulkApi" value="true"/>
            <entry key="sfdc.bulkApiSerialMode" value="true"/>
            <entry key="sfdc.externalIdField" value="SlId__c"/>
            <entry key="process.mappingFile" value="C:\Users\User\Google Drive\Automation\OrdersLine21Jan.sdl"/>
            <entry key="process.outputError" value="C:\Users\User\downloads\Logs\errorUpsertOrderItem.csv"/>
            <entry key="process.outputSuccess" value="C:\Users\User\downloads\Logs\successUpsertOrderItem.csv"/>
            <entry key="dataAccess.name" value="C:\Users\User\Google Drive\Automation\JAEG_TransactionDetails.csv" />
            <entry key="dataAccess.type" value="csvRead" />
        </map>
    </property>     </bean>

From my research, it seems to be something to do with either the debug log (I think most likely) or batch size.

I have set the sfdc.debugMessages to 'false' so it is not writing the files but it does seem to write it to the command screen. I feel this could be causing the problem, is there a default log setting? Maybe a process command setting?

In the data loader document http://resources.docs.salesforce.com/200/6/en-us/sfdc/pdf/salesforce_data_loader.pdf it says the max sfdc.loadBatchSize is 200 but through the UI it sets it to 2000 when batch is true. If that does restrict it, that could explain it.

I just cant find anything recent about this problem, anyone had any luck doing this at full pace recently?

Logging.properties file

############################################################

# Default Logging Configuration File

#

# You can use a different file by specifying a filename

# with the java.util.logging.config.file system property.

# For example java -Djava.util.logging.config.file=myfile

############################################################

############################################################

# Global properties

############################################################

# "handlers" specifies a comma separated list of log Handler

# classes. These handlers will be installed during VM startup.

# Note that these classes must be on the system classpath.

# By default we only configure a ConsoleHandler, which will only

# show messages at the INFO and above levels. handlers= java.util.logging.ConsoleHandler

# To also add the FileHandler, use the following line instead.

#handlers= java.util.logging.FileHandler, java.util.logging.ConsoleHandler

# Default global logging level.

# This specifies which kinds of events are logged across

# all loggers. For any given facility this global level

# can be overriden by a facility specific level

# Note that the ConsoleHandler also has a separate level

# setting to limit messages printed to the console. .level= INFO

############################################################

# Handler specific properties.

# Describes specific configuration info for Handlers.

############################################################

# default file output is in user's home directory. java.util.logging.FileHandler.pattern = %h/java%u.log
java.util.logging.FileHandler.limit = 50000
java.util.logging.FileHandler.count = 1
java.util.logging.FileHandler.formatter =
java.util.logging.XMLFormatter

# Limit the message that are printed on the console to INFO and above.

java.util.logging.ConsoleHandler.level = INFO

java.util.logging.ConsoleHandler.formatter =

java.util.logging.SimpleFormatter

Best Answer

In the folder that contains the process-conf file, do you also have a file called log-conf.xml? Because if you don't, I know DataLoader starts logging everything, and creates a huge file...

Do you also have a config file in the folder? If not specified in the bean, that will also set parameters...

I have the command line setup on my machine for various projects - happy to to do a comparison test to check whether it is tool or machine specific - which version are you using?

AMENDED: I added the log-conf.xml and set the priorities to 'OFF'. Not sure why the setting in the process-conf doesnt override this, I am not sure.

Related Topic