CSV export creates improper UTF-8 file

[dag_support]

We are using ListGrid.exportData(dsRequest); to create a CSV file. The file created by this method, in our case, includes UTF-8 characters, and the file can be viewed in some text editors. However, the file itself does not have the UTF-8 BOM marker. Because of this, Excel will not recognize the file as UTF-8, where the UTF-8 characters are not rendered correctly. Adding the BOM to this file will then make it a valid UTF-8 file and it will then render correctly in Excel.

Can the BOM be added on export when the export includes UTF-8?

[Isomorphic]

We haven't seen an issue with Excel having trouble recognizing UTF-8 in CSV files. You didn't mention the platform or version for Excel. Perhaps your Excel is very old, or installed on a version of Windows with very old system libraries.

If you, for some reason, need to work around some faulty version of Excel, you could add a BOM using a Filter Servlet. However, note that BOM use is discouraged by the Unicode standard.

[dag_support]

If I open the exported file in a text editor I can see the Chinese characters just fine. But if I open it in Excel 2013 on Windows 7, the Chinese looks like garbage. If I add the BOM at the beginning of the file, Excel opens it fine. I understand this may be a Microsoft issue, but Excel is by far the primary tool used by many corporate users to view CSV files. I would upload a copy of this file but the uploader will not allow me to do so.

[Isomorphic]

You can email your BOMless file to support@isomorphic.com and we'll see if we get the same problem with our Excel installations.

Be sure to let us know what locale your machine is configured for, perhaps that matters.

[dag_support]

Apologies for the delay, I just got a chance now to send this file. Thanks.

[Isomorphic]

We see the issue with the file you sent. Unfortunately, it doesn't make sense to alter the framework to automatically add a BOM, since other tools don't support BOMs, and at a framework level we don't know the ultimate destination for CSV export data. And of course there's the fact that the official standard recommends against using them.

If you choose to automatically add a BOM, you could do it via a FilterServlet. You could also use Feature Sponsorship to have us add a new setting for export operations to add BOMs for various types of encodings.

Note that, as far as viewing this file in Excel, this StackOverflow answer explains how to get Excel to load the file as UTF-8.