UTF in 5.0p?

[edulid]

Did you change something in the UTF-8 encoding in the 5.0p version?
I have code which works perfectly in 4.1p but in 5.0p not.
I basically send a filename (which can contain the german ä ö ü characters) to a DMI method.
In the DMI Method I have:

Map values = dsRequest.getValues();
String filename = new String(
				((String)values.get("filename")).getBytes(Charsets.UTF_8.name()), Charsets.UTF_8.name());

This is not encoded properly in 5.0p, but in 4.1p it was working. What is different here?
Using 5.0p eval 27.01

[edulid]

Code which was working previously:

TestDMIHandler:

public class TestDMIHandler {

	public void doGetTextfile(DSRequest dsRequest, HttpServletResponse servletResponse) throws Exception {

		Map values = dsRequest.getValues();
		String filename = new String(
				((String)values.get("filename")).getBytes(Charsets.UTF_8.name()), Charsets.UTF_8.name());
		
		servletResponse.setContentType("text/plain; charset="
				+ Charsets.UTF_8.name());
		
		String encodedFilename = CharMatcher.anyOf("+").replaceFrom(
				URLEncoder.encode(filename, Charsets.UTF_8.name()),
				"%20");
		servletResponse.setHeader("Content-Disposition",
				"attachment; filename=\"" + filename + ".txt\"; filename*="
						+ Charsets.UTF_8.name() + "''"
						+ encodedFilename + ".txt");
		

	}

}

simpleTable:

<DataSource ID="simpleTable" serverType="sql" tableName="t_simple_table">

	<fields>
		<field name="f_id" type="sequence" primaryKey="true" />
		<field name="f_text" type="text" />
	</fields>

	<operationBindings>

		<operationBinding operationType="custom" operationId="getTextFile">
			<serverObject className="de.mks_infofabrik.kids.server.dmi.TestDMIHandler"
				methodName="doGetTextfile" />
		</operationBinding>

	</operationBindings>

</DataSource>

EntryPoint:

@Override
	public void onModuleLoad() {

		VLayout layout = new VLayout();

		IButton button = new IButton("click me");
		button.addClickHandler(new ClickHandler() {
			
			@Override
			public void onClick(ClickEvent event) {
				Record data = new Record();
				data.setAttribute("filename", "äöü");
				DSRequest prop = new DSRequest();
				prop.setDownloadResult(true);
				DataSource.get("simpleTable").performCustomOperation("getTextFile",data,null, prop);
			}
		});

		layout.setMembers(button);
		layout.setWidth100();
		layout.setHeight100();

		layout.draw();
	}

[Isomorphic]

This doesn't look like it could ever have worked, since it has no doCustomResponse() call. You wouldn't get a text file, you'd get a response encoded in the usual dataProtocol:"scServer" format.

As far as speculating about a framework change, you are calling Servlet APIs, not SmartGWT APIs - we don't control those behaviors.

[edulid]

The content of the text file is not important, I am trying to set up the filename correctly.

I just tested the code with SmartClient Version: v9.1p_2015-01-27/PowerEdition Deployment (built 2015-01-27) : See screenshot 2.png, which is correct.

Then I tested with SmartClient Version: v10.0p_2015-01-27/EVAL Deployment (expires 2015.03.28_07.42.36) Licensed to: Isomorphic Software (#ISC_EVAL_NIGHTLY): See screenshot 1.png, which is incorrect.

I think the values() are transferred differently in 5.0p and 4.1p, which would explain the different/incorrect encodings.

Since I didn't change anything else, just the smartgwt libraries, I think this cannot be the Servlet APIs.

[Isomorphic]

The problem is, unless you call doCustomResponse, we are going to potentially overwrite your HTTP headers. And that seems to be the only way we could influence this at all, since you are calling Servlet APIs. So again - please actually try doCustomResponse().

[edulid]

I tried, and I get exactly the same result:
The headers are actually arriving, just the encoding of the filename is not correct. As I said, I think the values() are transferred in some different way, that is why the code is working in 4.0p but not in 5.0p.
Have you actually tried my code ? As I said, exactly the same code is working in 4.1p but not in 5.0p.

public class TestDMIHandler {

	public void doGetTextfile(DSRequest dsRequest, HttpServletResponse servletResponse) throws Exception {

		dsRequest.getRPCManager().doCustomResponse();
		
		Map values = dsRequest.getValues();
		String filename = new String(
				((String)values.get("filename")).getBytes(Charsets.UTF_8.name()), Charsets.UTF_8.name());
		
		servletResponse.setContentType("text/plain; charset="
				+ Charsets.UTF_8.name());
		
		String encodedFilename = CharMatcher.anyOf("+").replaceFrom(
				URLEncoder.encode(filename, Charsets.UTF_8.name()),
				"%20");
		servletResponse.setHeader("Content-Disposition",
				"attachment; filename=\"" + filename + ".txt\"; filename*="
						+ Charsets.UTF_8.name() + "''"
						+ encodedFilename + ".txt");
		

	}

}

[edulid]

Isomorphic?
I already tried the doCustomResponse() method you suggested, and, as I said, I get the same results.

Please take a look here:
http://forums.smartclient.com/showthread.php?t=31350
There, using 4.1p, I showed you a fix for UTF-8 in filenames,which worked in DMI methods. Now, in 5.0p, this exact method and code is not working any more.

[Isomorphic]

We are having difficulties reproducing your issue. With exact same code you've shown us, fileName is encoded correctly in our local environment, i.e. it contains expected german characters. Can you provide more info on environment you are using?

1. Browser type and version
2. Exact JDK version number
3. Application Server type and version
4. The value of Charset.defaultCharset() (java call)

[edulid]

Chrome Version 40.0.2214.111 m, but I also observe this with firefox 35.0.1

Java 1.7.0_67

Apache Tomcat/8.0.9

Charset: windows-1252

SmartGWT 5.0p eval 27.01.15

GWT 2.6.1

[Isomorphic]

OK, now this is reproduced in our environment and we are working at fixing it.
Meanwhile you may temporarily change default charset of your JVM to UTF-8, that should solve this for you locally. To do that add following argument to your JVM:

-Dfile.encoding=UTF8

[edulid]

Have you fixed this issue?

[edulid]

Hi Isomorphic,

it would be nice to get an answer on this (and some other from my threads which are still unanswered).

[alius]

Hi,

Try to change this line:

            //data.setAttribute("filename", "äöü");
            data.setAttribute("filename", "\u00E4\u00F6\u00FC");

It is exactly the same letters only in unicode.

[edulid]

This wouldn't work for my case because the letters come from the DB, and I would have to replace all possible characters with their corresponding unicode. For example, á, é, ü, ß, etc. And this would only be a workaround for code that previously worked.

And, as Isomorphic said here http://forums.smartclient.com/showpost.php?p=129209&postcount=10 , this is a bug being fixed. I just want to know if this is already fixed, since the post is already 2 weeks old.

[Isomorphic]

Apologies for late reply, your Feb 19 question was missed somehow. Be sure this issue is not forgotten.

Well, it was said that your issue is reproduced, but it actually wasn't. What was done is we changed default charset in our JVM to match yours (windows-1252) and yes, in logs we started to see "wrong characters", this lead to miss-understanding that issue is reproduced, but it is not true, since filename in browser download dialog was actually correct, i.e. it had all special characters.

So, at the moment it is still unclear what is wrong with this, since in fact this is still not reproduced. Are you sure that SGWT version is the *only* thing you have changed since "it was working in 9.1"?