[mathias]

Hello,

have had really urgent other stuff to do for a while, but this isn't solved yet... :)

OK, so regarding charset=utf8, i do have it for all communication i can see in Firebug:

Content-Type text/html; charset=utf-8

For the downloading of the specific file though, i.e. when i get the "save as" popup window, nothing shows up in firebug.
Hope this answers your question.

[Isomorphic]

It's surprising that Firebug doesn't show this response, but if so, then you can use Fiddler (or another tool) to check for that critical charset setting.

[mathias]

Hi,

OK, so i managed to install Wireshark and run some analysis. It IS indeed UTF-8 like everything else. Please see attached screenshot for wireshark output when i press my "export" button.

Please allow me to summarize what i've said in this thread, and what i think:


1.
I have problems with the CSV export, because it doesn't open properly in Excel. It looks fine in all other programs that i have tested, such as Ultraedit, TextEdit, etc.


2.
As far as i can see, i currently have characterencoding set to UTF-8 everywhere. I have a custom datasourceloader that sets the encoding
I also have a characterencodingfilter set up to ensure that all requests and responses are UTF-8:

<filter>
        <filter-name>encoding-filter</filter-name>
        <filter-class>org.springframework.web.filter.CharacterEncodingFilter</filter-class>
        <init-param>
            <param-name>encoding</param-name>
            <param-value>UTF-8</param-value>
        </init-param>
        <init-param>
            <param-name>forceEncoding</param-name>
            <param-value>true</param-value>
        </init-param>
    </filter>

3.
From what i can see, the problem with the garbled characters in Excel has to do with the LOUSY way it handles character encoding. Evidences/Indications of this can be found in lots of threads over at stackoverflow, for example here:
http://stackoverflow.com/questions/6002256/is-it-possible-to-force-excel-recognize-utf-8-csv-files-automatically

The problem, then, seems to be that UNLESS a BOM is set in the beginning of the exported file, It will assume encoding based on country and language of the install.

This can be verified by taking the exported file, which again, has UTF-8 in the encoding header, saving it with a BOM in Ultraedit, then opening it up in Excel. THEN it renders OK.


4.
I was thinking that i somehow could encode ONLY exports with ISO-8859 since that's the encoding Excel assumes in Sweden, but then things like euro characters won't work apparently, according to this thread... http://forums.smartclient.com/showthread.php?t=15992



So as a last comment, i think you're wrong that just setting UTF-8 is enough to make exports render OK in Excel. The reason you think this might be because you're in a country where Excel opens the file up with a "better" character encoding than what it does in Nordic countries. Just speculation on my part.

So, from what i can tell, i think you will NEED to set the BOM in order for the files to open up correcyly in Excel...

Please advice.

[mathias]

Forgot the Wireshark screenshot, attached.

[Isomorphic]

The theory that a BOM is required in Nordic countries is interesting, and perhaps you could try adding it and see if it corrects the problem.

[mathias]

Yes, it would be worth a try, but i have zero idea on how to accomplish this...

I'd have to add an UTF-8 BOM (Byte Order Marker) in the beginning of the stream, but as you know, the control flow belongs to the ISC Servlet, I just return a DSResponse with data.
You are the ones generating the CSV data.

If you have any pointers as to how i would go about making this happen, i'd be most happy.

[Isomorphic]

FilterServlets give the ability to inject before or after.

[mathias]

Yes, filters are not unknown to me, i utilize them for a couple of things.


In this case, however, i'm a little bit unsure how to make it work. What's needed is a couple of bytes in the beginning of the CSV-file you return.

I can't just create an response.getOutputStram() and write the BOM to it, it needs to be in front of the csv-bytes if you get my point.

This is what i'm not sure how to realize, i can't really see an option besides you having support for it.

[mathias]

Guys, i desperately need someone to talk to about this...

In addition to the question about how i add BOM to a file when i don't have the assembled CSV-stream you're returning, i've been thinking some more:


One way COULD be to set and return the data in ASCII JUST for the CSV export.

However, since the servlet filter works on URL's, i'm not sure if i can set the httpresponse encoding parameter just for that.

You got any input here?

[Isomorphic]

FilterServlets appear to allow you to add the BOM - Google for samples of adding content before and after normal responses.

But if you can't, then just save the CSV to a file and try delivering it via a servlet where you can add or not add the BOM to see if you can actually get this BOM issue to happen in isolation with correct charset settings.

[ajr]

I know it's old topic, but setting property

exportHeader: "\uFEFF"

seems to be enough to add BOM readable by Excell.

[mathias]

OK, where did you set that property?

[Crack]

It should be passed in requestProperties parameter of DataBoundComponent.exportClientData. That way DataBoundComponent will use it when issuing DSRequest that queries server for your CSV.