5.1d BatchUploader character encoding problem in Chrome 46

[Blama]

Hi Isomorphic,

I have a problem with my 5.1d BatchUploader (SNAPSHOT_v10.1d_2015-10-16) when I upload a "UTF-8 w/o BOM"-file. All special chars are replaced by "?". This does not happen in IE11 and FF26 (Win 8.1).
I tested the online 5.0p showcase (from May) with following modified supplyItemTest.csv. This worked as expected.

Could you check if this file works for you in the 5.1d BatchUploader in Chrome 46?
I'll try to create a testcase tomorrow, but perhaps you can already see if there is a problem or know if there where changes in this area since May.

Thank you & Best regards
Blama

[Blama]

Hi Isomorphic,

I digged deeper, using SNAPSHOT_v10.1d_2015-10-20/PowerEdition Deployment.

1st observation: The requests leaving the client are correctly encoded (checked in Firefox F12 tools)
2nd observation: It is not browser related (tested IE11, GC46, FF26)
3rd observation: For me, it happens on my server (Apache+Tomcat), but not on my notebook (Tomcat only). I confused that yesterday as I was testing remote with CG and local with FF.

Of course, the deployed code is the same.
I attached you two serverlog files (I already removed the timestamps to diff them better), one local, one remote, for the same request (zipped because of size). I also included the uploaded file.
Search for "ÜÖ" in the good logfile and for the four question marks / boxes (or how ever your editor display UTF-8 encoding errors) found in rows 501/503 in the bad logfile to see where else these strings appear.

Unfortunately I think that the first occurrence in the logs is way to late, so if you need more log data with different settings, I can provide these.
I'll also include my IDACall subclass, as one export error you fixed was triggered by the configuration there.

Best regards
Blama

PS: I do not have any other problems with character encoding. Everything else is working as expected.

[Isomorphic]

Thanks for the update. Is it possible that your Apache is parsing and/or re-encoding the inbound request using a different encoding scheme? Perhaps explicitly setting it to use UTF8 would help?:

http://stackoverflow.com/questions/9...f-8-for-server

Does the Apache+Tomcat configuration work correctly with 5.0p?

[Blama]

Hi Isomorphic,

I had the UTF-8 settings already in place.

But your other hint was very valueable. With 5.0p (v10.0p_2015-10-23/PowerEdition Deployment) it does work in all browsers (FF26, GC46, IE11) local (Tomcat only) and remote(Apache+Tomcat).

So it looks like the error is somehow in the differences between 5.0p and 5.1d. I'll switch back to 5.1d next (2 days newer) and let you know if anything changed.

Best regards
Blama

[Blama]

Hi Isomorphic,

I retested with SNAPSHOT_v10.1d_2015-10-22/PowerEdition and the result is like in post #2.

If I can provide you with any log files I'll happily do so. Do you have an idea what could be causing this?
This one and the one from the configurable upload fields name are currently the most important ones for me.

Best regards
Blama

[Isomorphic]

Having trouble reproducing this. Let's get some more specifics:

- Version of Apache
- Version of Tomcat
- What method of Apache -> Tomcat communication do you have configured? Reverse Proxy? mod_proxy_ajp? mod_jk? Something else?

Ideally, would like to see the relevant Apache config block, and any deviation from defaults that you have for the Tomcat config.

[Blama]

Hi Isomorphic,

I'm using CentOS 6.6 with

• apache-tomcat-7.0.53
• apache httpd 2.2.15-39.el6.centos
• mod_jk/1.2.39

I attached my Tomcat Server.xml and Apache config (sensitive information removed).

If I can try something as well let me know.

Best regards
Blama

[Isomorphic]

So - still not reproducing it - using your exact Apache HTTPD/Tomcat/CentOS versions and all the seemingly relevant config settings. What's in your workers.properties?

[Isomorphic]

Also - long shot - but does the problem go away if you map all resources to the ajp worker via e.g:

<IfModule mod_jk.c>
        JkMount /test/* worker1    
</IfModule>

[Blama]

Hi Isomorphic,

I'll try that and also installing Apache locally as well as soon as possible, hopefully this week,

Best regards
Blama

[Blama]

Hi Isomorphic,

bad news: I installed Apache + mod_jk locally (same setup as on the server) and everything is working like a charm.

• Win 10
• apache-tomcat-8.0.30
• apache httpd 2.2.31
• mod_jk/1.2.40

So also not a httpd / mod_jk / Tomcat issue.


I have an interesting observation, though:

• I have a validator on every field that logs the field value to the console. I use the "Server Logs" from Developer Console then.
  • For the local server, "Server Logs" shows the logged value correctly with Umlauts.
  • Remote, "Server Logs" shows the logged value incorrectly with two squares instead of "ß".
• If I look at the request parameter in Firefox' F12 tools, the data is the same for either server (with a different (expected) boundary value).

Conclusion: This and the wrong remote logging shows that the error is occurring very early in the serverside processing and most likely not related to any Clientside code.


Looking at the requests sent in Firefox F12 tools:
http://localhost:8080/lms/lms/sc/IDA...7&isc_tnum=23:

• shown as DSRequest batchUpload/update<upload> in the Developer Console
• multipart request with the data
• hits processRequest() + handleDSRequest() in my IDACall subclass.

http://localhost:8080/lms/lms/sc/IDA...-17&isc_xhr=1:

• shown as RPC DMI setAttributes in Developer Console
• hits processRequest() only in my IDACall subclass.

I'm thinking that I have to watch the data at as many places as possible to see where it is corrupted and where it is not. I assume that it is related to some Java environment setting, because the code is the same, the input data is the same and the outcome is different. So if I can find where the data gets corrupted it might be easier to find the relevant setting.

Could you tell me how I can find the file contents in the 1st request's handleDSRequest()?

I have this:




I'd like to find the request data here. Somewhere in either dsRequest, rpc or context must the file data (processed or unprocessed) be. It is not directly in dsRequest (see picture).
Could you help me how to find it here (and in processRequest()), also with undocumented methods, as it is just for debugging?

Thank you & Best regards
Blama

[Blama]

Hi Isomorphic,

interesting news:
I added this to my processRequest() as debug measure:

    public void processRequest(HttpServletRequest request, HttpServletResponse response) throws ServletException, IOException {
        int i = request.getContentLength();
        if (i > 3000) {
            String s = IOUtils.toString(((RequestFacade) request).getInputStream());
            Logger logger = new Logger(LMSIDACall.class);
            logger.error(s);
        }

This temporary code breaks SmartGWT afterwards (com.isomorphic.rpc.ClientMustResubmitException), but allows for debug printing to the console and to the Server Log in Developer Console.
It shows that remote, where I have the problem, the content is broken at this early stage in processing. ("ß" -> "??", "ü" -> "??")
Locally, where importing with umlauts works, the Server Log is as expected.

Is there a method even earlier in the processing flow where I could insert such log messages?
Do you have an idea what could be causing this?

Best regards
Blama


PS: The i > 3000 is to only log (and thereby break) the data upload, not normal requests, as I need to bring my normal UI with the BatchUploader up, first.

[Isomorphic]

There is of course doGet/doPost(), standard Servlet APIs.

[Blama]

Hi Isomorphic,

another test: I rolled all the commons-*.jar libs back to the ones that come with 5.0 - no change in result: locally working, remote broken umlauts.

Best regards
Blama

[Blama]

Hi Isomorphic,

I also tried doPost:

    @Override
    public void doPost(HttpServletRequest request, HttpServletResponse response) throws ServletException, IOException {
        int i = request.getContentLength();
        if (i > 3000) {
            String s = IOUtils.toString(((RequestFacade) request).getInputStream());
            Logger logger = new Logger(LMSIDACall.class);
            logger.error("Logging from doPost, test umlauts: \"öß\"");
            logger.error(s);
        }
        super.doPost(request, response);
    };

Also with "test-umlauts" in the log message (see above).
The result is again the same: locally working, remote broken umlauts. The test-logentry is correctly displayed in both, though.

Best regards
Blama