Announcement

Collapse
No announcement yet.
X
Collapse
  • Page of 1
  • Filter
  • Time
Clear All
new posts
Previous template Next
    #1

    BatchUploader

    Hi Isomorphic,

    I'm starting to use BatchUploader in current 4.1p (v9.1p_2015-02-07) and hit a short and unmeaningful exception in serverside BU code.

    I'm getting the following error in the client UI:
    SmartGWT Warning window with just the text "charsetName".

    Tomcat server logs are (see screenshots for client-request):
    Code:
    === 2015-02-13 11:11:04,627 [ec-5] INFO  RequestContext - URL: '/lms/lms/sc/IDACall', User-Agent: 'Mozilla/5.0 (Windows NT 6.3; WOW64; rv:26.0) Gecko/20100101 Firefox/26.0': Moz (Gecko) with Accept-Encoding header
=== 2015-02-13 11:11:04,630 [c-31] INFO  RequestContext - URL: '/lms/lms/sc/IDACall', User-Agent: 'Mozilla/5.0 (Windows NT 6.3; WOW64; rv:26.0) Gecko/20100101 Firefox/26.0': Moz (Gecko) with Accept-Encoding header
=== 2015-02-13 11:11:04,630 [ec-5] DEBUG RPCManager - Processing 1 requests.
=== 2015-02-13 11:11:04,630 [ec-5] DEBUG RPCManager - Request #1 (RPCRequest) data: {
    appID:"isc_builtin",
    className:"com.isomorphic.tools.BuiltinRPC",
    methodName:"setAttributes",
    arguments:[
        "session",
        [
            {
                name:"dsName",
                type:"hidden",
                value:"V_LEADUPLOAD"
            },
            {
                name:"delimiter",
                type:"hidden",
                value:","
            },
            {
                name:"quoteString",
                type:"hidden",
                value:"\""
            }
        ],
        null
    ],
    is_ISC_RPC_DMI:true
}
=== 2015-02-13 11:11:04,630 [ec-5] INFO  LMSIDACall - Performing 1 operation(s)
=== 2015-02-13 11:11:04,630 [ec-5] DEBUG RPCDMI - appConfig: isc.Application.create({
    rpcBindings:[
        {
            ID:"builtin",
            className:"com.isomorphic.rpc.BuiltinRPC",
            visibleMethods:[
                {
                    name:"downloadWSDL"
                },
                {
                    name:"downloadClientContent"
                },
                {
                    name:"downloadClientExport"
                },
                {
                    name:"xmlToJS"
                },
                {
                    name:"uploadProgressCheck"
                },
                {
                    name:"saveFile"
                },
                {
                    name:"appendToFile"
                },
                {
                    name:"loadFile"
                },
                {
                    name:"deleteFile"
                },
                {
                    name:"loadSharedXML"
                },
                {
                    name:"saveSharedXML"
                },
                {
                    name:"getAvailableScriptEngines"
                },
                {
                    name:"devConsoleEvalServerScript"
                },
                {
                    name:"evalJava"
                },
                {
                    name:"getLogNames"
                },
                {
                    name:"getLogEntries"
                },
                {
                    name:"clearLogEntries"
                },
                {
                    name:"getLogThresholds"
                },
                {
                    name:"setLogThreshold"
                },
                {
                    name:"setTemporaryLogThreshold"
                },
                {
                    name:"revertTemporaryLogThresholds"
                },
                {
                    name:"getPdfObject"
                },
                {
                    name:"exportImage"
                },
                {
                    name:"areServerTimingsTracked"
                },
                {
                    name:"trackServerTimings"
                }
            ]
        },
        {
            ID:"builtin_tools",
            className:"com.isomorphic.tools.BuiltinRPC",
            visibleMethods:[
                {
                    name:"getDataSourceFromTable"
                },
                {
                    name:"getDataSourceJSONFromTable"
                },
                {
                    name:"getDataSourceFromHibernateMapping"
                },
                {
                    name:"getDataSourceJSONFromHibernateMapping"
                },
                {
                    name:"getTables"
                },
                {
                    name:"getFieldsFromTable"
                },
                {
                    name:"getBeanFields"
                },
                {
                    name:"getHibernateBeans"
                },
                {
                    name:"getDatabaseProductNameAndVersion"
                },
                {
                    name:"getDatabaseTableTypes"
                },
                {
                    name:"setAttributes"
                },
                {
                    name:"clearAttributes"
                },
                {
                    name:"getAttributes"
                },
                {
                    name:"getAttribute"
                },
                {
                    name:"getDataSourceConfigFromJavaClass"
                },
                {
                    args:"cName",
                    language:"groovy",
                    name:"getJavaSource",
                    script:"\n                    if (!com.isomorphic.auth.DevModeAuthFilter.devModeAuthorized(request)) throw new Exception(\"Not Authorized\");                    \n                    //import org.apache.bcel.Repository;\n\n                    try {\n                        return org.apache.bcel.Repository.lookupClass(cName).toString();\n                    } catch (Throwable e) {\n                        return \"Unable to reverse engineer class \"+cName+\": \"+e.getMessage();\n                    }\n                "
                },
                {
                    name:"loadDataSource"
                },
                {
                    name:"dsFromXML"
                },
                {
                    name:"dsConfigFromXML"
                },
                {
                    name:"getDefinedDataSources"
                }
            ]
        },
        {
            ID:"builtin_adminconsole",
            className:"com.isomorphic.tools.AdminConsole",
            visibleMethods:[
                {
                    name:"getDefinedDatabases"
                },
                {
                    name:"testDB"
                },
                {
                    name:"saveDBConfig"
                },
                {
                    name:"setDefaultDB"
                },
                {
                    name:"importDataSources"
                },
                {
                    name:"discoverJNDIDatabases"
                }
            ]
        }
    ]
})

=== 2015-02-13 11:11:04,630 [ec-5] DEBUG RPCDMI - rpc returned data
=== 2015-02-13 11:11:04,630 [ec-5] DEBUG RPCManager - Content type for RPC transaction: text/plain; charset=UTF-8
=== 2015-02-13 11:11:04,630 [ec-5] INFO  Compression - /lms/lms/sc/IDACall: 67 -> 73 bytes
=== 2015-02-13 11:11:04,646 [c-31] DEBUG RPCManager - Processing 1 requests.
=== 2015-02-13 11:11:04,646 [c-31] DEBUG RPCManager - Request #1 (DSRequest) payload: {
    criteria:{
    },
    values:{
        file:"lead2.csv",
        dsName:"V_LEADUPLOAD",
        delimiter:",",
        quoteString:"\"",
        _transaction:null
    },
    operationConfig:{
        dataSource:"batchUpload",
        repo:null,
        operationType:"update",
        textMatchStyle:"exact"
    },
    componentId:"isc_DynamicForm_0",
    appID:"builtinApplication",
    operation:"upload",
    oldValues:{
        dsName:"V_LEADUPLOAD",
        delimiter:",",
        quoteString:"\""
    }
}
=== 2015-02-13 11:11:04,646 [c-31] INFO  LMSIDACall - Performing 1 operation(s)
=== 2015-02-13 11:11:04,646 [c-31] DEBUG DataSourceDMI - Invocation threw exception
java.lang.NullPointerException: charsetName
	at java.io.InputStreamReader.<init>(Unknown Source)
	at com.isomorphic.tools.BatchUpload.batchUpload(BatchUpload.java:124)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
	at java.lang.reflect.Method.invoke(Unknown Source)
	at com.isomorphic.base.Reflection.adaptArgsAndInvoke(Reflection.java:975)
	at com.isomorphic.datasource.DataSourceDMI.execute(DataSourceDMI.java:416)
	at com.isomorphic.datasource.DataSourceDMI.execute(DataSourceDMI.java:64)
	at com.isomorphic.datasource.DSRequest.execute(DSRequest.java:2531)
	at com.isomorphic.servlet.IDACall.handleDSRequest(IDACall.java:215)
	at com.lmscompany.lms.server.LMSIDACall.handleDSRequest(LMSIDACall.java:57)
	at com.isomorphic.servlet.IDACall.processRPCTransaction(IDACall.java:172)
	at com.lmscompany.lms.server.LMSIDACall.processRequest(LMSIDACall.java:39)
	at com.isomorphic.servlet.IDACall.doPost(IDACall.java:73)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:647)
	at com.isomorphic.servlet.BaseServlet.service(BaseServlet.java:152)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:728)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
	at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
	at com.isomorphic.servlet.CompressionFilter.doFilter(CompressionFilter.java:260)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
	at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222)
	at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123)
	at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:611)
	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:100)
	at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:953)
	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:409)
	at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1044)
	at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:607)
	at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:313)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at java.lang.Thread.run(Unknown Source)
=== 2015-02-13 11:11:04,646 [c-31] WARN  RequestContext - dsRequest.execute() failed: 
java.lang.NullPointerException: charsetName
	at java.io.InputStreamReader.<init>(Unknown Source)
	at com.isomorphic.tools.BatchUpload.batchUpload(BatchUpload.java:124)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
	at java.lang.reflect.Method.invoke(Unknown Source)
	at com.isomorphic.base.Reflection.adaptArgsAndInvoke(Reflection.java:975)
	at com.isomorphic.datasource.DataSourceDMI.execute(DataSourceDMI.java:416)
	at com.isomorphic.datasource.DataSourceDMI.execute(DataSourceDMI.java:64)
	at com.isomorphic.datasource.DSRequest.execute(DSRequest.java:2531)
	at com.isomorphic.servlet.IDACall.handleDSRequest(IDACall.java:215)
	at com.lmscompany.lms.server.LMSIDACall.handleDSRequest(LMSIDACall.java:57)
	at com.isomorphic.servlet.IDACall.processRPCTransaction(IDACall.java:172)
	at com.lmscompany.lms.server.LMSIDACall.processRequest(LMSIDACall.java:39)
	at com.isomorphic.servlet.IDACall.doPost(IDACall.java:73)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:647)
	at com.isomorphic.servlet.BaseServlet.service(BaseServlet.java:152)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:728)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
	at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
	at com.isomorphic.servlet.CompressionFilter.doFilter(CompressionFilter.java:260)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
	at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222)
	at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123)
	at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:611)
	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:100)
	at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:953)
	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:409)
	at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1044)
	at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:607)
	at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:313)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at java.lang.Thread.run(Unknown Source)
=== 2015-02-13 11:11:04,646 [c-31] DEBUG RPCManager - Content type for RPC transaction: text/html; charset=UTF-8
=== 2015-02-13 11:11:04,646 [c-31] DEBUG RPCManager - non-DMI response, dropExtraFields: false
=== 2015-02-13 11:11:04,646 [c-31] INFO  Compression - /lms/lms/sc/IDACall: 656 -> 409 bytes
    I used different files as test (included, renamed to .txt for the forum):
    • Your sample supplyItemTest.csv (UTF8 w/o BOM, Unix line endings)
    • Own lead.csv (UTF8 w/o BOM, Dos/Windows line endings) (only "normal"-ANSI characters inside)
    • Own lead.csv (ANSI, Dos/Windows line endings)


    This client code inside a Window:
    Code:
    batchUploader = new BatchUploader();
batchUploader.setWidth(600);
batchUploader.setUploadDataSource(leadUploadDS);

TextItem stringValue = new TextItem("stringValue", "String Value");
IntegerItem numberValue = new IntegerItem();
numberValue.setName("numericValue");
numberValue.setTitle("Numeric Value");

batchUploader.setUploadFormItems(stringValue, numberValue);
addItem(batchUploader);
    No special server code. I copied and sc/DataSourceLoader-loaded batchUpload.ds.xml as suggested in the docs.

    This is my simple testing target .ds.xml:
    Code:
    <DataSource dbName="Oracle" tableName="T_LEAD" ID="V_LEADUPLOAD" serverType="sql"
	serverConstructor="com.lmscompany.lms.server.LMSSQLDataSource">
	<fields>
		<field primaryKey="true" hidden="true" name="ID" type="sequence"/>
		<field hidden="true" name="TENANT_ID" type="integer" canEdit="false" />
		<field foreignKey="V_USER_CREATED_BY.ID" relatedTableAlias="USER_CREATED_BY" name="CREATED_BY" title="Erstellt von" type="creator" />
		<field name="CREATED_AT" title="Erstellt am" type="creatorTimestamp" />

		<field foreignKey="T_CAMPAIGN.ID" name="CAMPAIGN_ID" displayField="CAMPAIGN_NAME" title="Kampagne" type="integer" required="true" />
		<field includeFrom="T_CAMPAIGN.NAME" name="CAMPAIGN_NAME" hidden="true" />
		<field foreignKey="T_LEADTEMPERATURE.ID" name="LEADTEMPERATURE_ID" displayField="LEADTEMPERATURE_NAME" title="Leadtemperature" type="integer"
			required="true" />
		<field includeFrom="T_LEADTEMPERATURE.NAME" name="LEADTEMPERATURE_NAME" hidden="true" />
		<field foreignKey="T_LEADTYPE.ID" name="LEADTYPE_ID" displayField="LEADTYPE_NAME" title="Leadtyp" type="integer" joinType="outer" />
		<field includeFrom="T_LEADTYPE.NAME" name="LEADTYPE_NAME" hidden="true" />
		<field name="POTREV_RESELLER" title="Gesamtprojekt netto (geschätzt)" type="integer">
			<validators>
				<validator type="integerRange" min="1" errorMessage="Bitte geben Sie einen Betrag größer Null ein." />
			</validators>
		</field>
	</fields>
	<serverObject lookupStyle="new" className="com.lmscompany.lms.server.worker.V_LEADUPLOAD" />
	<operationBindings>
		<operationBinding operationType="add" requiresRole="addLead, addLeadByMail">
		</operationBinding>
	</operationBindings>
</DataSource>
    Referenced V_LEADUPLOAD serverObject is just return request.execute():
    Code:
    public class V_LEADUPLOAD {
	// requiresRole="addLead, addLeadByMail" in ds.xml
	public DSResponse add(DSRequest request, HttpServletRequest servletRequest) throws Exception {
		DSResponse response = request.execute();
		return response;
	}
};
    LMSIDACall.java should not be responsible:
    Code:
    package com.lmscompany.lms.server;

import java.io.IOException;
import java.util.Arrays;
import java.util.LinkedList;
import java.util.List;

import javax.servlet.ServletException;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;

import com.isomorphic.datasource.BasicDataSource;
import com.isomorphic.datasource.DSRequest;
import com.isomorphic.datasource.DSResponse;
import com.isomorphic.rpc.RPCManager;
import com.isomorphic.servlet.IDACall;
import com.isomorphic.servlet.RequestContext;
import com.lmscompany.lms.server.util.User;

public class LMSIDACall extends IDACall {
	private static final long serialVersionUID = 4543651445782711736L;

	/*
	 * See http://www.smartclient.com/smartgwtee/javadoc/com/smartgwt/client/docs/serverds/OperationBinding.html#requiresRole
	 */
	@Override
	public void processRequest(HttpServletRequest request, HttpServletResponse response) throws ServletException, IOException {
		Long userId = User.getUserId(request);
		if (userId != null) {
			try {
				RequestContext context = RequestContext.instance(this, request, response);
				RPCManager rpc = new RPCManager(request, response);
				rpc.setUserId(userId.toString());
				// rpc.setAuthenticated(true);
				// rpc.setUserRoles((String) roles);

				// call processRPCTransaction() to iterate through all RPCRequests and
				// DSRequests and execute them
				processRPCTransaction(rpc, context);

			} catch (Throwable e) {
				handleError(response, e);
			}
		} else {
			super.processRequest(request, response);
		}
	}

	@Override
	public DSResponse handleDSRequest(DSRequest dsRequest, RPCManager rpc, RequestContext context) throws Exception {
		String allowedClientOperationTypesProperty = ((BasicDataSource) dsRequest.getDataSource()).getProperty("allowedClientOperationTypes");
		if (allowedClientOperationTypesProperty != null) {
			List<String> allowedClientOperationTypesList = new LinkedList<String>(Arrays.asList(allowedClientOperationTypesProperty.split(" ")));
			if (!allowedClientOperationTypesList.contains(dsRequest.getOperationType()))
				return new DSResponse().setDataSource(dsRequest.getDataSource()).setFailure("OperationType not allowed");
		}
		return super.handleDSRequest(dsRequest, rpc, context);
	}
}
    Do you have an idea what is causing this exception with no good error message? Is it possible that this is a bug? I did not find anything related to "charset" or "encoding" in the BatchUploader-docs. I'll try to experiment with batchUploader.setDataURL() as in the sample now, but I think the error is somewhere before.

    Thank you & Best regards,
    Blama
    Attached Files
    Last edited by Blama; 13 Feb 2015, 02:47.
    Tags: None
    #2
    - Warning message screenshot
    - Last csv-file
    Attached Files

    Comment

      #3
      Hi Isomorphic,

      I'm pretty sure now it is a bug. The URL I gave in batchUploader.setDataURL() does not appear in the logs and is most likely only used for the real ADD, not the UPDATE needed to get the data to the ListGrid-AutoChild of the BatchUploader.

      Is this data flow what's actually happening (my educated guess)?
      1. Select CSV file
      2. Hit upload
      3. Data is transferred to special system DataSource for BatchUpload (batchUpload.ds.xml)
      4. ListGrid-AutoChild of BatchUploader is bound to this DS
      5. Server returns update from CSV data
      6. Client displays data as "normal UPDATE-DSResponse"
      7. Optionally some editing
      8. Hit Commit
      9. ADD-Operation to actual batchUploader.setUploadDataSource() is executed
      10. ListGrid-AutoChild of BatchUploader is destroyed, regardless of success/failure of the different (queued) ADDs


      Best regards,
      Blama

      Comment

        #4
        We're taking a look. One question: Can you show us what the developer console shows in the RPC tab when you issue this request which fails?

        Thanks

        Comment

          #5
          Hi Isomorphic,

          did you see the screenshot included in the 1st post?

          Best regards,
          Blama

          Comment

            #6
            We do now! :)
            This is currently under investigation. We'll let you know when we have more information for you.

            Thanks
            Isomorphic Software

            Comment

              #7
              We have had a fix for this issue on the 5.0 and 5.1 products for a while, but it didn't get committed to 4.1. We have done that now - please try again with tomorrow's nightly build and let us know how you get on.

              Comment

                #8
                Hi Isomorphic,

                I will do. Thanks a lot!

                Comment

                  #9
                  Hi Isomorphic,

                  thanks, your change fixed the bug. I can now see the ListGrid and don't get an error message. I can continue now.

                  One question: As this seems to be pretty basic usage and was unfixed / unnoticed in 4.1p - does this mean nobody is using BatchUploader in 4.1p as of now?

                  Best regards,
                  Blama

                  Comment

                    #10
                    Obviously not given various forums activity (and other privately submitted questions), but no one hit this particular regression, which happened as a result of fixing a different issue.

                    Comment

                    Previous template Next
                    Working...
                    X