MIME-types handled by CompressionFilter / MIME-type of DataSourceLoader

[Blama]

Hi Isomorphic,

in http://forums.smartclient.com/showthread.php?t=28437 I set up ISC_FileLoader and served the (compressed) JS files in advance while displaying the login screen.

Preloading the core SC files saves a lot traffic that would otherwise be necessary after the successful login.
So I looked in Firebug whats still coming in for the 1st call of the real application, seeing that about 98% come from the DS loader.

I also have a ServletLogin, coming from your tip at https://isomorphic.atlassian.net/wik...mcat+JDBCRealm.

When I look in the server log for COMPRESSION entries I can see that answers for calls to IDACall are compressed. Calls to my LoginServlet as well as DataSourceLoader are not.

I then looked at http://www.smartclient.com/docs/9.0/...p..compression, section "Compressible mime types and compatibility". It says there that only certain MIME-types are compressed, the rest is ignored by the CompressionFilter. So I changed the MIME-type of my LoginServlet via "response.setContentType("text/xml");". Afterwards the answer of my Servlet is compressed.
Now I'm pretty sure that the CompressionFilter gets the result of DataSourceLoader but ignores it because of its MIME-type.
I did not find a way to tell Tomcat to change the MIME-type of the Servlet afterwards and think that only the servlet can define its contents type.

If that is true, is there a way that you either could change the MIME-type of the servlet (currently application/json;charset=UTF-8) or that one can configure which MIME-types CompressionFilter should compress?

Thanks,
Blama

[Blama]

Hi Isomorphic,

do you have any advice here?
I now configured my system to use Apache httpd+mod_jk+Tomcat and therefore have httpd as possible extra intervention point in order to enable compression.

Right now it seems to me that httpd does not do that for mod_jk-results. But as this is not your problem, I asked a question on this on serverfault.

But if you could give an advice on how to have your CompressionFilter compress the results of the DataSourceLoader-servlet, it would be really great!

Thanks,
Blama

PS: A 2nd call to my cached app would only be VERY IMPRESSIVE 5 http-requests and (because of caching) only 3kb of data transferred, if it were not for DataSourceLoader.
This is 236kb or 23kb when zipped by hand. Even with the 23kb (when not cached) this would be very impressive.

[Blama]

Hi all,

I solved the problem using a ServletFilter. It helped a lot since it - as expected - reduced the transferred data from 236kb -> 23kb.

Here the (simple) code for MimeTypeFilter.java:

import java.io.IOException;
import java.util.Date;

import javax.servlet.Filter;
import javax.servlet.FilterChain;
import javax.servlet.FilterConfig;
import javax.servlet.ServletException;
import javax.servlet.ServletRequest;
import javax.servlet.ServletResponse;
import javax.servlet.http.HttpServletResponse;

public class MimeTypeFilter implements Filter {
	private static final long millis = 24 * 60 * 60 * 1000;

	@Override
	public void destroy() {
	}

	@Override
	public void doFilter(ServletRequest request, ServletResponse response, FilterChain filterChain) throws IOException, ServletException {
		filterChain.doFilter(request, response);
		// Now the DataSourceLoader finished its work. MIME-Type is "application/json;charset=UTF-8".
		// In order for the CompressionFilter to compress the result, we need to change the MIME-Type to a supported value.
		// See http://www.smartclient.com/docs/9.0/a/b/c/go.html#group..compression for a list
		// See also http://grokbase.com/t/tomcat/users/08cy774f3z/httpd-problem-with-mod-headers
		response.setContentType("text/html");
	}

	@Override
	public void init(FilterConfig filterConfig) throws ServletException {
	}
}

web.xml:

	<filter>
		<filter-name>MimeTypeFilter</filter-name>
		<filter-class>com.mycompany.myapp.server.MimeTypeFilter</filter-class>
	</filter>
	<filter-mapping>
		<filter-name>MimeTypeFilter</filter-name>
		<servlet-name>DataSourceLoader</servlet-name>
	</filter-mapping>
...
...
	<!-- Dynamic Compression -->
	<filter>
		<filter-name>CompressionFilter</filter-name>
		<filter-class>com.isomorphic.servlet.CompressionFilter</filter-class>
	</filter>
...
...
	<filter-mapping>
		<filter-name>CompressionFilter</filter-name>
		<servlet-name>DataSourceLoader</servlet-name>
	</filter-mapping>

Please note that the order of the filters might be important here.
After doing this, I found this thread, where a user has a similar problem (modify headers from a mod_jk answer). Using a filter seems to be the only solution in this case.



In order to set the caching headers for the DataSourceLoader-servlet, I change the above doFilter() to:

response.setContentType("text/html");
HttpServletResponse httpResp = (HttpServletResponse) response;
httpResp.addDateHeader("Expires", new Date().getTime() + (millis * 365));

Now one has to make sure that the browser calls the servlet once you expect changed data. As I right now always use the newest nightly, an application-deploy also means a new framework version. Therefore I also include the framework version in my main jsp file call to DataSourceLoader (see here):

<script type="text/javascript" src="myApp/sc/DataSourceLoader<% out.print(version); %>&dataSource=......."></script>

@Isomorphic: Can you confirm that the usage of setContentType("text/html") does not hinder either CompressionFilter or the client-side of the framework, that uses the DataSourceLoader-response from working correctly? For me it works, but it would be nice to have approval. Again, if one could configure the types CompressionFilter works with, this hack wouldn't be necessary.


Best regards,
Blama

[Isomorphic]

We've included mime type used by DataSourceLoader-servlet to compressible mime types. You may download latest (2014-04-09) nightly build and try it out. All calls to DataSourceLoader-servlet will be compressed as well.

Blama, we do not recommend using hack you suggested, since it is not necessary after this fix.

[Blama]

Hello Isomorphic,

I can confirm that the data is served compressed and the MIME-type is application/json;charset=UTF-8 using v9.1p_2014-04-09.

I removed the setContentType(...) from my code and only kept the "Expires"-header.

Thanks a lot & best regards,
Blama

[Blama]

As short note: The list in http://www.smartclient.com/docs/9.1/...p..compression is not updated, yet.

[Blama]

Hi Isomorphic,

I just saw that the docs were updated.
I do have a problem here, though. I want to implement client side DataSourceLoader-result caching like discussed here.
I still use the filter with the

httpResp.addDateHeader("Expires", new Date().getTime() + (millis * 365));

but then have two "Expires" headers in result:

Expires: Thu, 01 Jan 1970 01:00:00 CET
​Expires: Thu, 14 Sep 2017 15:10:54 GMT

This makes the browsers ignore the caching directive.
I'm pretty sure that the 1970-header is not added by my web.xml or my server in general, but somehow by your code (even though I saw that the online server showcase does not show this header).
Is it added by the DataSourceLoader-servlet itself (I'm using v10.1p_2016-08-26)? If so, could you disable it or make adding it at least optional via a query string parameter?

Removing headers is not that easy/impossible in current servlets.

Best regards
Blama

[Blama]

Hi Isomorphic,

I could solve it the hard way in my Apache vhost with this:

    <LocationMatch 'lms/sc/DataSourceLoader'>
        Header unset Expires
        Header set Expires "Thu, 31 Dec 2020 23:59:59 GMT"
    </LocationMatch>

But this does not seem right and does not work locally where I only have Tomcat and no Apache httpd in front of it.

Best regards
Blama

[Isomorphic]

Hello,

Is there a possibility that you missed something and this additional 1970 header is set by your code? Could you double check this please. If you remove your filters and do not set your header, would this mystical 1970 header still be there?

Thank you,
Isomorphic

[Blama]

Hi Isomorphic,

it took me some time, but I found the reason for the behavior . It seems to be standard behavior for protected resources in Tomcat. My web.xml looks now like this:

<web-app xmlns="http://java.sun.com/xml/ns/j2ee" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://java.sun.com/xml/ns/j2ee http://java.sun.com/xml/ns/j2ee/web-app_2_4.xsd"
    version="2.4">
    <login-config>
        <auth-method>FORM</auth-method>
        <realm-name>User Auth</realm-name>
        <form-login-config>
            <form-login-page>/login.jsp</form-login-page>
            <form-error-page>/error.jsp</form-error-page>
        </form-login-config>
    </login-config>
    <security-role>
        <role-name>*</role-name>
    </security-role>

    <security-constraint>
        <web-resource-collection>
            <web-resource-name>LMS Application</web-resource-name>
            <url-pattern>*.html</url-pattern>
            <url-pattern>*.jsp</url-pattern>
            <url-pattern>/lms/sc/IDACall/*</url-pattern>
            [B]<!-- <url-pattern>/lms/sc/DataSourceLoader/*</url-pattern> -->[/B]
            <url-pattern>/lms/sc/screenLoader/*</url-pattern>
            <url-pattern>/ServletLogin</url-pattern>
        </web-resource-collection>

        <auth-constraint>
            <role-name>*</role-name>
        </auth-constraint>
    </security-constraint>

    <welcome-file-list>
        <welcome-file>Lms.jsp</welcome-file>
    </welcome-file-list>

This way the offending header is not set and I can set my "cache 1 year" in a filter as only "Expires"-header.

During research I found these SO-threads: One, Two. So this is not related to your product and I'll have to take care of this myself.

Are there reasons besides DDOS and some information leakage that require DataSourceLoader to be protected?
How do you normally do this in your contract development if it is a SaaS-product reachable from the Internet?

Thank you & Best regards
Blama

[Isomorphic]

Information leakage is the primary reason to protect DataSourceLoader and we do protect it in SaaS deployments where authentication is required to access the app at all. In other deployments it may be left unprotected.

[Blama]

Hi Isomorphic,

ok, I'll do the same then and remove the unnecessary/unwanted Header in Apache httpd.

Thanks for the discussion
Blama