Arabic data being fetched through Datasource (ds.xml) is not rendered properly

[sathish.gdis]

Hi team

We are upgrading our application from v10.0p_2015-03-03/PowerEdition Deployment to v11.0p_2017-06-18/Enterprise Deployment.

In this process, we found that all arabic data which is coming through Datasource xml files are coming as "???????". Also we found that the arabic values in fmt tag, js constant and excel export are showing the arabic data properly.

After a deep investigation between excel export and normal listgrid fetch, we found that following line of code is returing the arabic data to ????? .

Class: com.isomorphic.rpc.RPCManager
line: jsTrans.toJS(orderedResponseList, out);

In this line, orderedResponseList is an input and out is an output. Arabic data is proper in the "orderedResponseList" but not in "out".

[Blama]

Hi sathish.gdis,

please see the end of this thread for some encoding change from 5.0/10.0 to 5.1/10.1. Perhaps the java-setting on server start will solve your issue as well.

Best regards
Blama

[sathish.gdis]

Thanks Blama for your inputs.

It worked perfectly. Without this setting it was working fine in 10.0p version.
If possible could you explain why such settings required in latest version.

Thanks once again.

[Blama]

Hi,

great to hear. It took me long to come up with this as root cause as you can see in the other thread. I don't know why the default here changed from 5.0 to 5.1. This would be one for Isomorphic to explain and perhaps also one to include in a migration document / FAQ entry / ...

Best regards
Blama

[sathish.gdis]

Hi Isomorphic

Please explain about the settings of file.encoding in latest version 11.0/6.0

[Isomorphic]

file.encoding is a Java platform setting, not a setting provided by SmartGWT.

We have had a total of about 3 users say that this setting is now required to get i18n to work, whereas it wasn't before. We have never found a test system where this needs to be set, so right now, it remains a mystery.

[sathish.gdis]

Hi Isomorphic,

Hope you could find the resaon for the encoding setting in java server side for i18n to work as you have few users complained about the same.

Waiting for the reason to explain.

[Isomorphic]

As we just explained, we have never found a test system where this setting is required. We've also never found a machine on any of our consulting projects where it was required.

If you're curious as to why your particular environment has this requirement, the best thing is probably to post all the details you can (exact OS version, exact JDK and version, locale of machine, default JVM charset you see when you do not apply this setting, etc) and encourage other users with the same problem to do the same. Then perhaps we will all see that this is a bug in OpenJDK for Windows Server, or whatever it is you all have in common.

[sathish.gdis]

Hi Isomorphic

Please find attached 2 images Writer out variable of RequestContext of smartclient 10 and smartclient 11.

In Smartclient 10, the StreamEncoder is UTF-8.
In Smartclient 11, the Streamencoder is MS1252

Our development environment for smartclient 10 and smartclient 11 is same as given below.
IDE: IBM Rational Developer 9
Application Server: IBM WAS 8.5.5
Machine: Windows 10

[Isomorphic]

Yes, you would expect to see that that PrintWriter has the wrong encoding, given the results you're seeing. However, that PrintWriter is acquired via servletResponse.getWriter()) - we do not directly create it, your servlet engine does.

As we cover in the Internationalization overview (specifically "NOTE: Default Encoding"), we attempt to force the charset to UTF-8 in the IDACall servlet, in order to avoid you having to set either your servlet engine or the JVM to the right encoding. We force the encoding with the standard approach:

        String encoding = getServletConfig().getInitParameter("encoding");
        if (encoding == null) encoding = "UTF-8";
        if (!encoding.toLowerCase().equals("none")) {
            request.setCharacterEncoding(encoding);        
        }

You could break this a couple of ways:
1. if your code, or your servlet engine, sets the encoding before IDACall attempts to
2. if you explicit set an init-param "encoding" and it's not UTF-8, or if your servlet engine provides such an init-param as a default

But this is all standard Java / servlet engine stuff we're explaining - nothing specific to our technology.

So, rather than repeatedly complaining that there is a difference in SmartClient versions, just use the docs and actually look into what's wrong in your environment.

[sathish.gdis]

As you expected, if there is a break in default encoding, it should break in the smartclient 10.0p version as well. But this is not the case.
Please note that we are not creating new project with smartclient 11.0p version. We just using the smartclient 11.0p jars instead of smartclient 10.0p jars with the same source code.
If you beleive that, default encoding is wrong or servlet engine is setting it wrong, then this problem should occur in smartclient 10.0p version as well since we are using the same source code but not happening. Smartclient 10.0p version in our project is working without this issue in all environements development (Windows) and Test/Production (Mainframe).

One more info, please see the two images attached in the last post that PrintWriter is ByteArrayOutputStream in Smartclient 11.0p whereas the the PrintWriter is SRTOutputStream (ServletOutputStream) in Smartclient 10.0p

If I am correct, the PrintWriter which is acquired via servletResponse.getWriter() should be as same as shown image 1 of previous post but the Printwriter in Smartclient 11.0p is not the same as Smartclient 10.0p

[Isomorphic]

Thanks for letting us know your testing methodology is invalid - you must follow the installation instructions rather than just replace .jars, and differences in the default web.xml and server.properties files we ship, among other things, could create the problem you're seeing.

Once you've performed a valid test, if you're still having the problem, again you need to follow the elementary troubleshooting steps we proposed above rather than just posting the same thing over and over here.