appreciate help w file encoding issue

[mathias]

Looked around on other threads, tried most suggestions but haven't managed to get my head around my setup.

I have an app with swedish characters for some titles in the .ds xml files.

I'm running on Tomcat with java_opts set to "-dfile.encoding=UTF-8"

I'm experimenting by opening up a .ds file in Ultraedit, changing stuff and stopping/starting tomcat to see what it looks like in the browser.

Basically, if i save the file as UTF-8 in Ultraedit and have encoding as UTF-8 in the ds. xml ehader, the swedish chars are not displayed properly.

HOWEVER, if i change the xml header to "ISO-8859-1", then save it as UTF-8, it works! This not a good solution, since i don't want to have to re-do that manually every time i change the file in Intellij (intellj saves the file in the encoding of the xml header)

Anybody have any additional ideas on how to sort this? Of course, i would love for it to be displayed properly when i have UTF-8 everywhere...

datasource file example:

<?xml version="1.0" encoding="UTF-8"?>
<DataSource ID="userstatus" serverType="generic">  ***
    <fields>        *******
        <field name="id" type="sequence" hidden="true" primaryKey="true"/>
...

[sjivan]

Have you tried -Dfile.encoding=UTF8

[mathias]

Hey, thanks for your quick response.

Unfortunately, that's what i have. '-d' was a typo on my part.

I just read up on tomcat's http encoding and default seems to be iso-8859-1, but can't see why that should affect anything.

On the other hand, not sure what my next steps should be since to me all is UTF-8... Any more ideas?

[mathias]

Added info:

my bootstrap file, which is a JSP file, has

<%@page contentType="text/html; charset=UTF-8" %>

on top. i added 'едц' to the body, and that gets displayed correctly in the browser.

Also tried with a regular HTML file, works too.

Only the smartgwt ds.-files are giving me headaches... :)

[Isomorphic]

You mentioned Tomcat's default encoding. If your Tomcat instance is configured with something other than UTF-8, change that too. Our servlets don't try to force the servlet engine to use the Java default file-encoding.

[mathias]

Thanks for your reply.

The thing is, i haven't changed anything in my tomcat installation. I just upgraded to 6.0.32 and i'm using it out-of-the-box.


If i need to make an alteration to tomcat's defaullt config just to run smartGWT, i'm a little bit puzzled as to why noone else has this issue if you see what i mean.

I have seen that Tomcats webpage does have an "encoding issues" article describing how to try and switch most stuff into UTF-8, but it feels really strange that everything works besides the smartgwt servlets. I hate it when i don't get it...

[Isomorphic]

Yes, it's really a dumb setting on Tomcat's part in our opinion, and isn't common among other servlet engines. They should be respecting the JVM's default setting. We are in a bad situation where we could try to force the JVM setting instead of the servlet engine setting, but that could be the wrong thing too.

[mathias]

Ah ok cool, so you kind of know that this is a common issue with tomcat and your datasource servlet?

Great stuff. You wouldn't happen to know which setting in tomcat i need to change? If not i guess i'll go through the stuff in that guide. As i mentioned, changing the "-Dfile.encoding" did nothing.

Cheers

[mathias]

Well, i did check the tomcat guidelines, and it basically tells you to set the response encoding to UTF-8 in servlets/jsp's... so not much good there i'm afraid.

If i call the datasourceloader directly from the browser, the data returned is rendered correctly, it's ONLY within the smartgwt components themselves, who load their data from the bootstrap jsp script call to the serlvet i guess, that it gets rendered incorrectly.

I found this thread http://forums.smartclient.com/showthread.php?t=8391, where they hardwired a servlet extension to datasourceloader, explicitly setting the response encoding to UTF-8...

It works for me, but feels pretty hacky.

[Isomorphic]

We find that recommendation bizarre, since it turns one setting (the JVM charset, which Tomcat could be respecting) into 3+ settings (JVM charset, Tomcat default charset, charset for every servlet), but I'll guess we'll have to override Tomcat's default and doc this as a gotcha for people using various asian charsets as a system default, who will now have to disable our forcing it to UTF8...

[mathias]

Alright. FYI, their encoding FAQ is here: http://wiki.apache.org/tomcat/FAQ/CharacterEncoding