Encoding problem when system encoding different to UTF-8

[helmut]

Hello Isomorphic,

we are currently using Smartgtw power 12.0-p20190920 and we have some display problems with german “umlaute” when the system encoding is different from UTF-8.
With the latest smartgwt versions we have some codepage troubles (browser displays “?” instead of “umlaute”) when customer's didn't include the explicit "-Dfile.encoding=UTF-8" setting for the tomcat.

In the documentation we have found following statement for the DataSourceLoader and the IDACall servlet:
"... This servlet is configured to automatically set character encoding on requests and responses to UTF-8. If you wish to force a different encoding, you can do so by specifying an init-param in your web.xml file, like so ...."
https://www.smartclient.com/smartgwt...rceLoader.html

In our tests explicit specify this setting in the servlets doesn’t change anything.
After some debugging with intellij we assume that the problem lies in the “com.isomorphic.servlet.ServletCompression” class where following statement is used in the method “compressIfPossible”:
context.setOut(new PrintWriter(this.wrapBuf));

The missing encoding parameter for the PrintWriter constructor causes an internal Charset.defaultCharset() call, so the encoding switches to the system encoding.
I believe adding the codepage in the constructor would solve this problem
Are we „barking up the wrong tree” or can you confirm our assumption?

Thank you very much!

[Blama]

Hi helmut, Hi Isomorphic,

if that is the root cause of the problem the OP reports, this might also be the reason for this issue present since/in 5.1d.

Best regards
Blama

[Isomorphic]

Blama, your problem appears to be related to upload encoding, so, wrong direction - this question is related to the compression servlet, which is used for *responses*, not uploads.

Helmut, why are you using a non-UTF8 charset?

[helmut]

Hello Isomorphic,

you are right, nowadays UTF-8 should be used as default system encoding.

But sadly, we have no influence which default system encoding is used by our customers, especially when they install our application into virtualization solutions (openshift, docker etc).
Also, windows don't use UTF-8 as system encoding (windows-1252 for example in German windows), so we have to use explicitly the "-Dfile.encoding=UTF8" argument for every installation to mitigate this problem.

I think it would be a more robust solution if the smartgwt behavior would be as described in the manual, which is different to the current behavior:
"...configured to automatically set character encoding on requests and responses to UTF-8...."

As you know users doesn't read the manual :-) , so it is better if something works out of the box without setting environment variables.

Otherwise in my opinion the docs should be changed.

Thank you very much!

[helmut]

Blama One question: is your problem with the batch uploader still existing in the current 12.0-p20190920 version? . Maybe i can try to reproduce it, if you are interested ..

[Isomorphic]

Ok, the failure to honor a pre-existing charset encoding on the response is definitely a bug - which has now been fixed. Fix will appear in the Sept 29 'p' build. Please give that a shot. Appreciate your deep dive to find the culprit.

[helmut]

Hello Isomorphic,

thank you for the patch.
I have tried version 12.0-p20190929 and it works as expected, so the problem is solved.

Thank you very much!

[Blama]

Blama One question: is your problem with the batch uploader still existing in the current 12.0-p20190920 version? . Maybe i can try to reproduce it, if you are interested ..

Hi helmut,

thank you. I don't know if the issue is still in there. But actually it is fine for me running tomcat with some parameter, as we are hosting the application ourselfes.

Best regards
Blama