Announcement

Collapse
No announcement yet.
X
  • Filter
  • Time
Clear All
new posts

    ListGrid Umlaut Codepage Encoding issues only in exportClientData

    Hi Isomorphic,

    I'm on a recent 4.1p (v9.1p_2015-02-07) and have the following issue on ListGrid.exportClientData():
    All exports have broken umlauts (äöüß) and Euro-signs (see screenshot).

    I'm using
    Code:
    DSRequest requestProperties = new DSRequest() {
    	{
    		setExportAs(ExportFormat.OOXML);
    		setExportFilename(exportFileName);
    	}
    };
    lg.exportClientData(requestProperties);
    The main page is defined with:
    Code:
    <!DOCTYPE html>
    <html>
      <head>
    		<meta http-equiv="content-type" content="text/html; charset=UTF-8">
    		<meta name="gwt:property" content="locale=de">
    ...
    and I'm seeing this behaviour in FF26 as well as GC 40.

    The server request looks like this:
    Code:
    === 2015-02-11 18:09:05,741 [c-15] INFO  RequestContext - URL: '/lms/lms/sc/IDACall', User-Agent: 'Mozilla/5.0 (Windows NT 6.3; WOW64; rv:26.0) Gecko/20100101 Firefox/26.0': Moz (Gecko) with Accept-Encoding header
    === 2015-02-11 18:09:05,749 [c-15] DEBUG RPCManager - Processing 1 requests.
    === 2015-02-11 18:09:05,749 [c-15] DEBUG RPCManager - Request #1 (RPCRequest) data: {
        appID:"isc_builtin",
        className:"builtin",
        methodName:"downloadClientExport",
        arguments:[
            [
                {
                    STATUS_SHORTNAME:"In Erstellung",
                    leadAction:" ",
                    CUST_NAME_CONTACTPERSFULLNAME:"Compäny with A-Umlaut (�ö-two OUmlaut �ü-two UUmlaut)",
                    POTREV_RESELLER:"2.000 â?¬"
                }
            ],
            "ooxml",
            "leads_in_creation.xlsx",
            "download",
            {
                exportAs:"ooxml",
                exportDisplay:"download",
                exportFilename:"leads_in_creation.xlsx",
                exportPath:null,
                exportToClient:null,
                exportToFilesystem:null,
                exportDelimiter:null,
                exportFields:[
                    "STATUS_SHORTNAME",
                    "leadAction",
                    "CUST_NAME_CONTACTPERSFULLNAME",
                    "POTREV_RESELLER"
                ],
                exportHeader:null,
                exportHeaderless:null,
                exportFooter:null,
                exportTitleSeparatorChar:null,
                exportDefaultBGColor:null,
                exportAlternateRowBGColor:null,
                exportRowBGColors:{
                },
                exportColumnBGColors:{
                },
                exportRawValues:false,
                exportCurrencySymbol:"â?¬",
                lineBreakStyle:null,
                exportOtherFields:{
                    STATUS_SHORTNAME:"Leadstatus",
                    leadAction:"Aktionen",
                    CUST_NAME_CONTACTPERSFULLNAME:"Ansprechpartner",
                    POTREV_RESELLER:"geschätztes Auftragsvolumen (Erstellerangabe)"
                },
                exportWrapHeaderTitles:null
            }
        ],
        is_ISC_RPC_DMI:true
    }
    === 2015-02-11 18:09:05,750 [c-15] INFO  LMSIDACall - Performing 1 operation(s)
    === 2015-02-11 18:09:05,752 [c-15] DEBUG RPCDMI - appConfig: isc.Application.create({
        rpcBindings:[
            {
                ID:"builtin",
                className:"com.isomorphic.rpc.BuiltinRPC",
                visibleMethods:[
                    {
                        name:"downloadWSDL"
                    },
                    {
                        name:"downloadClientContent"
                    },
                    {
                        name:"downloadClientExport"
                    },
                    {
                        name:"xmlToJS"
                    },
                    {
                        name:"uploadProgressCheck"
                    },
                    {
                        name:"saveFile"
                    },
                    {
                        name:"appendToFile"
                    },
                    {
                        name:"loadFile"
                    },
                    {
                        name:"deleteFile"
                    },
                    {
                        name:"loadSharedXML"
                    },
                    {
                        name:"saveSharedXML"
                    },
                    {
                        name:"getAvailableScriptEngines"
                    },
                    {
                        name:"devConsoleEvalServerScript"
                    },
                    {
                        name:"evalJava"
                    },
                    {
                        name:"getLogNames"
                    },
                    {
                        name:"getLogEntries"
                    },
                    {
                        name:"clearLogEntries"
                    },
                    {
                        name:"getLogThresholds"
                    },
                    {
                        name:"setLogThreshold"
                    },
                    {
                        name:"setTemporaryLogThreshold"
                    },
                    {
                        name:"revertTemporaryLogThresholds"
                    },
                    {
                        name:"getPdfObject"
                    },
                    {
                        name:"exportImage"
                    },
                    {
                        name:"areServerTimingsTracked"
                    },
                    {
                        name:"trackServerTimings"
                    }
                ]
            },
            {
                ID:"builtin_tools",
                className:"com.isomorphic.tools.BuiltinRPC",
                visibleMethods:[
                    {
                        name:"getDataSourceFromTable"
                    },
                    {
                        name:"getDataSourceJSONFromTable"
                    },
                    {
                        name:"getDataSourceFromHibernateMapping"
                    },
                    {
                        name:"getDataSourceJSONFromHibernateMapping"
                    },
                    {
                        name:"getTables"
                    },
                    {
                        name:"getFieldsFromTable"
                    },
                    {
                        name:"getBeanFields"
                    },
                    {
                        name:"getHibernateBeans"
                    },
                    {
                        name:"getDatabaseProductNameAndVersion"
                    },
                    {
                        name:"getDatabaseTableTypes"
                    },
                    {
                        name:"setAttributes"
                    },
                    {
                        name:"clearAttributes"
                    },
                    {
                        name:"getAttributes"
                    },
                    {
                        name:"getAttribute"
                    },
                    {
                        name:"getDataSourceConfigFromJavaClass"
                    },
                    {
                        args:"cName",
                        language:"groovy",
                        name:"getJavaSource",
                        script:"\n                    if (!com.isomorphic.auth.DevModeAuthFilter.devModeAuthorized(request)) throw new Exception(\"Not Authorized\");                    \n                    //import org.apache.bcel.Repository;\n\n                    try {\n                        return org.apache.bcel.Repository.lookupClass(cName).toString();\n                    } catch (Throwable e) {\n                        return \"Unable to reverse engineer class \"+cName+\": \"+e.getMessage();\n                    }\n                "
                    },
                    {
                        name:"loadDataSource"
                    },
                    {
                        name:"dsFromXML"
                    },
                    {
                        name:"dsConfigFromXML"
                    },
                    {
                        name:"getDefinedDataSources"
                    }
                ]
            },
            {
                ID:"builtin_adminconsole",
                className:"com.isomorphic.tools.AdminConsole",
                visibleMethods:[
                    {
                        name:"getDefinedDatabases"
                    },
                    {
                        name:"testDB"
                    },
                    {
                        name:"saveDBConfig"
                    },
                    {
                        name:"setDefaultDB"
                    },
                    {
                        name:"importDataSources"
                    },
                    {
                        name:"discoverJNDIDatabases"
                    }
                ]
            }
        ]
    })
    
    === 2015-02-11 18:09:05,754 [c-15] INFO  BuiltinRPC - Export data passed to server:
    [
        {
            STATUS_SHORTNAME:"In Erstellung",
            leadAction:" ",
            CUST_NAME_CONTACTPERSFULLNAME:"Compäny with A-Umlaut (�ö-two OUmlaut �ü-two UUmlaut)",
            POTREV_RESELLER:"2.000 â?¬"
        }
    ]
    === 2015-02-11 18:09:05,754 [c-15] INFO  ProxyHttpServletResponse - content-disposition set to: attachment; fileName=leads_in_creation.xlsx - unwrapping response output stream (CompressionFilter)
    === 2015-02-11 18:09:05,754 [c-15] INFO  BuiltinRPC - Generating and streaming ooxml file...
    === 2015-02-11 18:09:05,808 [c-15] DEBUG RPCDMI - rpc returned data
    As you can see, the data arrives already corrupted at the server.

    The client request looks like this (also as screenshot):
    Code:
    {
        "actionURL":"http://localhost:8080/lms/lms/sc/IDACall", 
        "showPrompt":true, 
        "prompt":"Suche Datensätze die den Kriterien entsprechen...", 
        "transport":"xmlHttpRequest", 
        "promptStyle":"cursor", 
        "bypassCache":true, 
        "data":{
            "criteria":{
                "fieldName":"CREATED_BY", 
                "operator":"equals", 
                "value":1083, 
                "_constructor":"AdvancedCriteria"
            }, 
            "operationConfig":{
                "dataSource":"V_LEAD_INCREATION", 
                "repo":null, 
                "operationType":"fetch", 
                "textMatchStyle":"exact"
            }, 
            "startRow":0, 
            "endRow":1000, 
            "sortBy":[
                "CAMPAIGN_NAME"
            ], 
            "componentId":"ListGrid_inCreation", 
            "appID":"builtinApplication", 
            "operation":"V_LEAD_INCREATION_fetch", 
            "oldValues":{
                "fieldName":"CREATED_BY", 
                "operator":"equals", 
                "value":1083, 
                "_constructor":"AdvancedCriteria"
            }
        }
    }
    Is this enough information to find out what is going on? It happens for me on the server (CentOS 6) as well as in dev (German Win8).

    Server startup log, in case that it is important (don't think so) is:
    Code:
    Feb 11, 2015 6:14:12 PM org.apache.catalina.core.AprLifecycleListener init
    INFORMATION: The APR based Apache Tomcat Native library which allows optimal performance in production environments was not found on the java.library.path: C:\Program Files\Java\jre7\bin;C:\WINDOWS\Sun\Java\bin;C:\WINDOWS\system32;C:\WINDOWS;C:\oraclexe\app\oracle\product\11.2.0\client_1;C:\oraclexe\app\oracle\product\11.2.0\server\bin;C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;C:\WINDOWS\System32\WindowsPowerShell\v1.0\;C:\Program Files (x86)\Git\cmd;C:\Program Files\TortoiseGit\bin;.
    Feb 11, 2015 6:14:12 PM org.apache.coyote.AbstractProtocol init
    INFORMATION: Initializing ProtocolHandler ["http-bio-8080"]
    Feb 11, 2015 6:14:12 PM org.apache.coyote.AbstractProtocol init
    INFORMATION: Initializing ProtocolHandler ["ajp-bio-8009"]
    Feb 11, 2015 6:14:12 PM org.apache.catalina.startup.Catalina load
    INFORMATION: Initialization processed in 1394 ms
    Feb 11, 2015 6:14:12 PM org.apache.catalina.core.StandardService startInternal
    INFORMATION: Starting service Catalina
    Feb 11, 2015 6:14:12 PM org.apache.catalina.core.StandardEngine startInternal
    INFORMATION: Starting Servlet Engine: Apache Tomcat/7.0.50
    Feb 11, 2015 6:14:13 PM org.apache.catalina.startup.HostConfig deployWAR
    INFORMATION: Deploying web application archive C:\Program Files\Apache Software Foundation\Tomcat 7.0\webapps\lms.war
    ISC: Configuring log4j from: file:/C:/Program%20Files/Apache%20Software%20Foundation/Tomcat%207.0/webapps/lms/WEB-INF/classes/log4j.isc.config.xml
    === 2015-02-11 18:14:22,615 [op-1] INFO  ISCInit - Isomorphic SmartClient/SmartGWT Framework - Initializing
    === 2015-02-11 18:14:22,620 [op-1] INFO  ConfigLoader - Attempting to load framework.properties from CLASSPATH
    === 2015-02-11 18:14:22,732 [op-1] INFO  ConfigLoader - Successfully loaded framework.properties from CLASSPATH at location: jar:file:/C:/Program%20Files/Apache%20Software%20Foundation/Tomcat%207.0/webapps/lms/WEB-INF/lib/isomorphic_core_rpc.jar!/framework.properties
    === 2015-02-11 18:14:22,732 [op-1] INFO  ConfigLoader - Attempting to load project.properties from CLASSPATH
    === 2015-02-11 18:14:22,732 [op-1] INFO  ConfigLoader - Unable to locate project.properties in CLASSPATH
    === 2015-02-11 18:14:22,737 [op-1] INFO  ConfigLoader - Successfully loaded isc_interfaces.properties from CLASSPATH at location: jar:file:/C:/Program%20Files/Apache%20Software%20Foundation/Tomcat%207.0/webapps/lms/WEB-INF/lib/isomorphic_core_rpc.jar!/isc_interfaces.properties
    === 2015-02-11 18:14:22,737 [op-1] INFO  ConfigLoader - Attempting to load server.properties from CLASSPATH
    === 2015-02-11 18:14:22,742 [op-1] INFO  ConfigLoader - Successfully loaded server.properties from CLASSPATH at location: file:/C:/Program%20Files/Apache%20Software%20Foundation/Tomcat%207.0/webapps/lms/WEB-INF/classes/server.properties
    === 2015-02-11 18:14:22,749 [op-1] INFO  Logger - Logging system started.
    === 2015-02-11 18:14:22,751 [op-1] INFO  ISCInit - Auto-detected webRoot - using: C:\Program Files\Apache Software Foundation\Tomcat 7.0\webapps\lms
    === 2015-02-11 18:14:22,752 [op-1] INFO  ISCInit - Isomorphic SmartClient/SmartGWT Framework (v9.1p_2015-02-07/EVAL Deployment 2015-02-07) - Initialization Complete
    Feb 11, 2015 6:14:22 PM org.apache.catalina.startup.HostConfig deployDirectory
    INFORMATION: Deploying web application directory C:\Program Files\Apache Software Foundation\Tomcat 7.0\webapps\docs
    Feb 11, 2015 6:14:23 PM org.apache.catalina.startup.HostConfig deployDirectory
    INFORMATION: Deploying web application directory C:\Program Files\Apache Software Foundation\Tomcat 7.0\webapps\host-manager
    Feb 11, 2015 6:14:23 PM org.apache.catalina.startup.HostConfig deployDirectory
    INFORMATION: Deploying web application directory C:\Program Files\Apache Software Foundation\Tomcat 7.0\webapps\manager
    Feb 11, 2015 6:14:23 PM org.apache.catalina.startup.HostConfig deployDirectory
    INFORMATION: Deploying web application directory C:\Program Files\Apache Software Foundation\Tomcat 7.0\webapps\ROOT
    Feb 11, 2015 6:14:23 PM org.apache.coyote.AbstractProtocol start
    INFORMATION: Starting ProtocolHandler ["http-bio-8080"]
    Feb 11, 2015 6:14:23 PM org.apache.coyote.AbstractProtocol start
    INFORMATION: Starting ProtocolHandler ["ajp-bio-8009"]
    Feb 11, 2015 6:14:23 PM org.apache.catalina.startup.Catalina start
    INFORMATION: Server startup in 11138 ms
    Thank you & Best regards,
    Blama
    Attached Files

    #2
    As addition:
    The normal use of the application has no problems at all with umlauts etc.
    The ds.xmls I use are UTF-8, if this is important.

    Best regards,
    Blama
    Attached Files

    Comment


      #3
      We are having difficulties reproducing your issue. With exact same code you've shown us, content is encoded correctly in our local environment. Can you provide more info on environment you are using?

      1. Browser type and version
      2. Exact JDK version number
      3. Application Server type and version
      4. The value of Charset.defaultCharset() (java call)

      Comment


        #4
        Hi Isomorphic,

        of course.

        1. Browser type and version
        FF26 Dev mode and Compiled, GC40 Compiled, IE11 Compiled

        2. Exact JDK version number
        Code:
        DEV:
        C:\Program Files\Java\jdk1.7.0_71>java -version
        java version "1.7.0_71"
        Java(TM) SE Runtime Environment (build 1.7.0_71-b14)
        Java HotSpot(TM) 64-Bit Server VM (build 24.71-b01, mixed mode)
        
        PROD:
        root@abcd ~]# java -version
        java version "1.7.0_51"
        Java(TM) SE Runtime Environment (build 1.7.0_51-b13)
        Java HotSpot(TM) 64-Bit Server VM (build 24.51-b03, mixed mode)
        3. Application Server type and version
        Tomcat 7.0.50 running under Win 8.1 64bit (Dev)
        Tomcat 7.0.53 running under CentOS 6 (Prod)
        Startup log Dev:
        Code:
        Feb 17, 2015 1:38:09 PM org.apache.catalina.core.AprLifecycleListener init
        INFORMATION: The APR based Apache Tomcat Native library which allows optimal performance in production environments was not found on the java.library.path: C:\Program Files\Java\jre7\bin;C:\WINDOWS\Sun\Java\bin;C:\WINDOWS\system32;C:\WINDOWS;C:\oraclexe\app\oracle\product\11.2.0\client_1;C:\oraclexe\app\oracle\product\11.2.0\server\bin;C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;C:\WINDOWS\System32\WindowsPowerShell\v1.0\;C:\Program Files (x86)\Git\cmd;C:\Program Files\TortoiseGit\bin;.
        Feb 17, 2015 1:38:10 PM org.apache.coyote.AbstractProtocol init
        INFORMATION: Initializing ProtocolHandler ["http-bio-8080"]
        Feb 17, 2015 1:38:10 PM org.apache.coyote.AbstractProtocol init
        INFORMATION: Initializing ProtocolHandler ["ajp-bio-8009"]
        Feb 17, 2015 1:38:10 PM org.apache.catalina.startup.Catalina load
        INFORMATION: Initialization processed in 1682 ms
        Feb 17, 2015 1:38:10 PM org.apache.catalina.core.StandardService startInternal
        INFORMATION: Starting service Catalina
        Feb 17, 2015 1:38:10 PM org.apache.catalina.core.StandardEngine startInternal
        INFORMATION: Starting Servlet Engine: Apache Tomcat/7.0.50
        Feb 17, 2015 1:38:11 PM org.apache.catalina.startup.HostConfig deployWAR
        INFORMATION: Deploying web application archive C:\Program Files\Apache Software Foundation\Tomcat 7.0\webapps\lms.war
        ISC: Configuring log4j from: file:/C:/Program%20Files/Apache%20Software%20Foundation/Tomcat%207.0/webapps/lms/WEB-INF/classes/log4j.isc.config.xml
        === 2015-02-17 13:38:27,783 [op-1] INFO  ISCInit - Isomorphic SmartClient/SmartGWT Framework - Initializing
        === 2015-02-17 13:38:27,794 [op-1] INFO  ConfigLoader - Attempting to load framework.properties from CLASSPATH
        === 2015-02-17 13:38:28,068 [op-1] INFO  ConfigLoader - Successfully loaded framework.properties from CLASSPATH at location: jar:file:/C:/Program%20Files/Apache%20Software%20Foundation/Tomcat%207.0/webapps/lms/WEB-INF/lib/isomorphic_core_rpc.jar!/framework.properties
        === 2015-02-17 13:38:28,068 [op-1] INFO  ConfigLoader - Attempting to load project.properties from CLASSPATH
        === 2015-02-17 13:38:28,070 [op-1] INFO  ConfigLoader - Unable to locate project.properties in CLASSPATH
        === 2015-02-17 13:38:28,080 [op-1] INFO  ConfigLoader - Successfully loaded isc_interfaces.properties from CLASSPATH at location: jar:file:/C:/Program%20Files/Apache%20Software%20Foundation/Tomcat%207.0/webapps/lms/WEB-INF/lib/isomorphic_core_rpc.jar!/isc_interfaces.properties
        === 2015-02-17 13:38:28,081 [op-1] INFO  ConfigLoader - Attempting to load server.properties from CLASSPATH
        === 2015-02-17 13:38:28,096 [op-1] INFO  ConfigLoader - Successfully loaded server.properties from CLASSPATH at location: file:/C:/Program%20Files/Apache%20Software%20Foundation/Tomcat%207.0/webapps/lms/WEB-INF/classes/server.properties
        === 2015-02-17 13:38:28,112 [op-1] INFO  Logger - Logging system started.
        === 2015-02-17 13:38:28,118 [op-1] INFO  ISCInit - Auto-detected webRoot - using: C:\Program Files\Apache Software Foundation\Tomcat 7.0\webapps\lms
        === 2015-02-17 13:38:28,120 [op-1] INFO  ISCInit - Isomorphic SmartClient/SmartGWT Framework (v9.1p_2015-02-17/EVAL Deployment 2015-02-17) - Initialization Complete
        Feb 17, 2015 1:38:28 PM org.apache.catalina.startup.HostConfig deployDirectory
        INFORMATION: Deploying web application directory C:\Program Files\Apache Software Foundation\Tomcat 7.0\webapps\docs
        Feb 17, 2015 1:38:28 PM org.apache.catalina.startup.HostConfig deployDirectory
        INFORMATION: Deploying web application directory C:\Program Files\Apache Software Foundation\Tomcat 7.0\webapps\host-manager
        Feb 17, 2015 1:38:29 PM org.apache.catalina.startup.HostConfig deployDirectory
        INFORMATION: Deploying web application directory C:\Program Files\Apache Software Foundation\Tomcat 7.0\webapps\manager
        Feb 17, 2015 1:38:30 PM org.apache.catalina.startup.HostConfig deployDirectory
        INFORMATION: Deploying web application directory C:\Program Files\Apache Software Foundation\Tomcat 7.0\webapps\ROOT
        Feb 17, 2015 1:38:30 PM org.apache.coyote.AbstractProtocol start
        INFORMATION: Starting ProtocolHandler ["http-bio-8080"]
        Feb 17, 2015 1:38:30 PM org.apache.coyote.AbstractProtocol start
        INFORMATION: Starting ProtocolHandler ["ajp-bio-8009"]
        Feb 17, 2015 1:38:30 PM org.apache.catalina.startup.Catalina start
        INFORMATION: Server startup in 19892 ms
        4. The value of Charset.defaultCharset() (java call)
        In Dev-Debug, expression java.nio.charset.Charset.defaultCharset().displayName() returns: windows-1252

        I hope this helps.

        Best regards,
        Blama
        Last edited by Blama; 17 Feb 2015, 04:51. Reason: Added prod java version.

        Comment


          #5
          Hi Isomorphic,

          were you able to reproduce the problem?

          Best regards,
          Blama

          Comment


            #6
            Sorry for delayed answer, we were analyzing bunch of similar issues with encoding and no, this is not yet reproduced.

            Could you help us and do the following:

            1. please add encoding=UTF-8 parameter to your log4j config, see below XML example of appender with this parameter. This will ensure that UTF8 strings will be printed to log correctly, without this setting log4j uses default charset which corrupts strings while printing to log. Please make sure you do this to correct appender (you might use different set of appenders), or you may apply this to all of them.
            Code:
            <appender name="STDOUT" class="org.apache.log4j.ConsoleAppender">
                [b]<param name="encoding" value="UTF-8"/>[/b]
                <layout class="org.apache.log4j.PatternLayout">
                <param name="ConversionPattern" value="=== %d [%.4t] %-5p %c{1} - %m%n"/>
                </layout>		
            </appender>
            2. see if data is actually arriving to the server already corrupted, show us your log with utf-8 as encoding for log4j.

            3. could you try change export format to xls like shown below, would this also export wrong characters?
            Code:
            setExportAs(ExportFormat.XLS);
            As a temporary solution, you may change default charset for the whole JVM which is running servlet engine, by providing this argument:
            Code:
            -Dfile.encoding=UTF8
            Thank you for your patience.

            Comment


              #7
              Hi Isomorphic,

              I did as advised (STDOUT explicitly to UTF8, ExportFormat.XLS, Tomcat VM start option), but unfortunately this does not help. The result is the same as before.
              I digged a bit deeper with Firefox F12 tools and it seems that the request sent by FF is still correct:

              Example:
              Developer Console RPC Tab, sub-Tab RPCRequest:
              Code:
              {
                  "actionURL":"http://localhost:8080/lms/lms/sc/IDACall", 
                  "showPrompt":false, 
                  "transport":"hiddenFrame", 
                  "promptStyle":"cursor", 
                  "data":{
                      "appID":"isc_builtin", 
                      "className":"builtin", 
                      "methodName":"downloadClientExport", 
                      "arguments":[
                          [
                              {
                                  "CUST_NAME_CONTACTPERSFULLNAME":"Eurosign:€ (Umlauts:äöü)", 
                                  "STATUS_SHORTNAME":"Abgelehnt (nach Picken)"
                              }
                          ], 
                          "xls", 
                          "leads_in_creation.xls", 
                          "download", 
                          {
                              "exportAs":"xls", 
                              "exportDisplay":"download", 
                              "exportFilename":"leads_in_creation.xls", 
                              "exportPath":null, 
                              "exportToClient":null, 
                              "exportToFilesystem":null, 
                              "exportDelimiter":null, 
                              "exportFields":[
                                  "CUST_NAME_CONTACTPERSFULLNAME"
                              ], 
                              "exportHeader":null, 
                              "exportHeaderless":null, 
                              "exportFooter":null, 
                              "exportTitleSeparatorChar":null, 
                              "exportDefaultBGColor":null, 
                              "exportAlternateRowBGColor":null, 
                              "exportRowBGColors":{
                              }, 
                              "exportColumnBGColors":{
                              }, 
                              "exportRawValues":false, 
                              "exportCurrencySymbol":"€", 
                              "lineBreakStyle":null, 
                              "exportOtherFields":{
                                  "CUST_NAME_CONTACTPERSFULLNAME":"Ansprechpartner"
                              }, 
                              "exportWrapHeaderTitles":null
                          }
                      ], 
                      "is_ISC_RPC_DMI":true
                  }
              }
              Firefox F12 Tools "_transaction" (see exact source for copy&paste in the attached screenshot):
              Code:
              <transaction+xmlns:xsi="http://www.w3.org/2000/10/XMLSchema-instance"+xsi:type="xsd:Object"><transactionNum+xsi:type="xsd:long">8</transactionNum><operations+xsi:type="xsd:List"><elem+xsi:type="xsd:Object"><appID>isc_builtin</appID><className>builtin</className><methodName>downloadClientExport</methodName><arguments+xsi:type="xsd:List"><elem+xsi:type="xsd:List"><elem+xsi:type="xsd:Object"><CUST_NAME_CONTACTPERSFULLNAME>Eurosign:€+(Umlauts:äöü)</CUST_NAME_CONTACTPERSFULLNAME><STATUS_SHORTNAME>Abgelehnt+(nach+Picken)</STATUS_SHORTNAME></elem></elem><elem>xls</elem><elem>leads_in_creation.xls</elem><elem>download</elem><elem+xsi:type="xsd:Object"><exportAs>xls</exportAs><exportDisplay>download</exportDisplay><exportFilename>leads_in_creation.xls</exportFilename><exportPath+xsi:nil="true"/><exportToClient+xsi:nil="true"/><exportToFilesystem+xsi:nil="true"/><exportDelimiter+xsi:nil="true"/><exportFields+xsi:type="xsd:List"><elem>CUST_NAME_CONTACTPERSFULLNAME</elem></exportFields><exportHeader+xsi:nil="true"/><exportHeaderless+xsi:nil="true"/><exportFooter+xsi:nil="true"/><exportTitleSeparatorChar+xsi:nil="true"/><exportDefaultBGColor+xsi:nil="true"/><exportAlternateRowBGColor+xsi:nil="true"/><exportRowBGColors+xsi:type="xsd:Object"></exportRowBGColors><exportColumnBGColors+xsi:type="xsd:Object"></exportColumnBGColors><exportRawValues+xsi:type="xsd:boolean">false</exportRawValues><exportCurrencySymbol>€</exportCurrencySymbol><lineBreakStyle+xsi:nil="true"/><exportOtherFields+xsi:type="xsd:Object"><CUST_NAME_CONTACTPERSFULLNAME>Ansprechpartner</CUST_NAME_CONTACTPERSFULLNAME></exportOtherFields><exportWrapHeaderTitles+xsi:nil="true"/></elem></arguments><is_ISC_RPC_DMI+xsi:type="xsd:boolean">true</is_ISC_RPC_DMI></elem></operations><jscallback>iframe</jscallback></transaction>
              Server log:
              Code:
              === 2015-04-02 17:57:30,965 [ec-2] INFO  BuiltinRPC - Export data passed to server:
              [
                  {
                      CUST_NAME_CONTACTPERSFULLNAME:"Eurosign:€ (Umlauts:äöü)",
                      STATUS_SHORTNAME:"Abgelehnt (nach Picken)"
                  }
              ]
              === 2015-04-02 17:57:30,966 [ec-2] INFO  ProxyHttpServletResponse - content-disposition set to: attachment; fileName=leads_in_creation.xls - unwrapping response output stream (CompressionFilter)
              === 2015-04-02 17:57:30,966 [ec-2] INFO  BuiltinRPC - Generating and streaming xls file...
              === 2015-04-02 17:57:30,975 [ec-2] DEBUG RPCDMI - rpc returned data
              Best regards,
              Blama
              Attached Files

              Comment


                #8
                This was using v10.0p_2015-03-25.

                Comment


                  #9
                  Hi Isomorphic,

                  I compiled BuiltInDS (with an export button) with SNAPSHOT_v10.1d_2015-04-02 and then deployed it on my Tomcat with all the changes you suggested. After changing the data to include Umlauts, I exported it.
                  The interesting result is that the server log is still faulty, but the exported file itself is as expected.

                  I'll search for the reason myself, but does this help? Where would you assume the difference?

                  Best regards,
                  Blama

                  Comment


                    #10
                    Update:

                    The log-outputs are both faulty, but in a different way:
                    Code:
                    BuiltInDS "ä": ä
                    BuiltInDS "€": €
                    (this matches http://www.i18nqa.com/debug/utf8-debug.html)
                    
                    				
                    MyApp     "ä": ä
                    MyApp     "€": €
                    It seems that in my application the data is encoded twice. I also switched BuiltInDS to use my Oracle database with the same driver, so the difference has to lay somewhere else.

                    Best regards,
                    Blama
                    Last edited by Blama; 2 Apr 2015, 11:44.

                    Comment


                      #11
                      Hi Isomorphic,

                      I found the root cause of the behaviour. It is caused by me, but for a reason.
                      My database has a users table with an ID column and a loginname-column (among others, as needed for Tomcat JDBCRealm).

                      All my data tables reference the user table with Foreign Keys to the user table's Primary Key, the ID column. But Tomcat returns the loginname (not the ID) as name for the logged in user.
                      So a SmartGWT generated INSERT for a table with a field list of
                      Code:
                      <field primaryKey="true" hidden="true" name="ID" type="sequence" />
                      <field name="CREATED_BY" type="creator" />
                      ...
                      will look like:
                      Code:
                      INSERT INTO mytable
                          ( ID,
                            CREATED_BY,
                      	  ...)
                          VALUES
                          ( mytable_ID.NextVal,
                            'userloginname',
                      	  ...)
                      This causes an Oracle error as Oracle expects an ID instead of name for the CREATED_BY column.

                      So I came up with this code:
                      Code:
                      public class MyIDACall extends IDACall {
                      	private static final long serialVersionUID = 4543651445782711736L;
                      
                      	@Override
                      	public void processRequest(HttpServletRequest request, HttpServletResponse response) throws ServletException, IOException {
                      		Long userId = User.getUserId(request);
                      		if (userId != null) {
                      			try {
                      				RequestContext context = RequestContext.instance(this, request, response);
                      				RPCManager rpc = new RPCManager(request, response);
                      				rpc.setUserId(userId.toString());
                      				// rpc.setAuthenticated(true);
                      				// rpc.setUserRoles((String) roles);
                      
                      				// call processRPCTransaction() to iterate through all RPCRequests and
                      				// DSRequests and execute them
                      				processRPCTransaction(rpc, context);
                      
                      			} catch (Throwable e) {
                      				handleError(response, e);
                      			}
                      		} else {
                      			super.processRequest(request, response);
                      		}
                      	}
                      which is basically what you suggest here.

                      If I remove the code (=always do a super.processRequest()), the export runs as expected (again, with funny characters in the log, but the exported data is correct).

                      How do I solve this best?

                      Thank you & Best regards,
                      Blama

                      Comment


                        #12
                        This is a known issue (was reported here: http://forums.smartclient.com/showthread.php?t=32283) and is fixed in all builds dated March 24, 2015 and newer.

                        Please try the latest stable build and let us know if you still see any issues.

                        Comment


                          #13
                          Hi Isomorphic,

                          I'm still seeing this in SNAPSHOT_v10.1d_2015-04-02 and v10.0p_2015-03-25. My testing yesterday was based on these versions.

                          Best regards,
                          Blama
                          Last edited by Blama; 3 Apr 2015, 12:42.

                          Comment


                            #14
                            Please also note that user edulid said that his code was working in 4.1p:
                            Originally posted by edulid View Post
                            Did you change something in the UTF-8 encoding in the 5.0p version?
                            I have code which works perfectly in 4.1p but in 5.0p not.
                            For me, umlauts in exports did not work in that version either. Therefore this might be unrelated.

                            Best regards,
                            Blama

                            Comment


                              #15
                              Ah ok - you have, in fact done everything correctly per docs, but the encoding logic has actually has since been moved into the processRequest() method that you are overriding.

                              We will review this and either correct the docs or update the code. In the meantime, please override this method on IDACall instead:

                              Code:
                                  public DSResponse handleDSRequest(DSRequest dsRequest, RPCManager rpc, RequestContext context)
                              That will ensure the proper encoding is applied to the request/response and still enables you to pass a custom userId

                              Comment

                              Working...
                              X