Go Back   SmartClient Forums > Smart GWT Technical Q&A
Wiki Register Search Today's Posts Mark Forums Read

Reply
 
Thread Tools Search this Thread
  #1  
Old 16th Apr 2009, 09:27
dfreire dfreire is offline
Registered Developer
 
Join Date: Mar 2009
Posts: 43
Default How to: SmartGWT on Google App Engine (Java)

Hello fellow developers,

Ok, so I've finally been able to upload my SmartGWT app into GAE.

First of all, if you try to upload your SmartGWT app into GAE without any changes you'll notice an error like:
Code:
400 Bad Request
Max number of files and blobs is 1000.
This happens because there are probably more than 1000 files in your generated smartgwt 'sc' dir.

So far, the proposed solution to this problem was to use another host (like Amazon S3) for the 'sc' static files. You have a thread explaining how to do it here: http://forums.smartclient.com/showthread.php?t=5105

Another approach is to zip the contents of the 'sc' dir and have the needed files unzipped on the fly using a servlet. This was originally proposed by a google developer during the last GAE IRC chat. (This is also a similar solution to the deployment of SmartGWT apps in GAE-Python, as suggested in other threads of this forum.)

So, how to do it:

1. Zip your 'sc' dir and place the produced file in your 'war' dir.
Note that the zip file must be compressed. The uncompressed file will have more than 10 MB and that will also originate an error during upload to GAE.

2. Edit your 'appengine-web.xml' file and exclude all 'sc' files from your app resources. Like this:
Code:
<!-- By default all WAR files are static except WEB-INF  -->
<!-- This overrides the default -->
<static-files>
    <!-- Exclude smartgwt files from the list of static files -->
    <exclude path="/sc/**.*" />
    <exclude path="/myappname/sc/**.*" />
</static-files>

<!-- By default all WAR files are also resource files including WEB-INF -->
<!-- This overrides the default -->
<resource-files>
    <!-- Exclude smartgwt files from the list of resource files -->
    <exclude path="/sc/**.*" />
    <exclude path="/myappname/sc/**.*" />
</resource-files>
Note: you must substitute 'myappname' by your app name.
Another Note: I don't know why, sometimes the request URI for 'sc' begins with '/sc/...' and other times with '/myappname/sc/...'. Maybe someone can explain this. My solution was to exclude both.

3. In your 'web.xml' file, create a new servlet that will catch all requests to 'sc':
Code:
<!-- SmartGWT static resources servlet -->
<servlet>
    <servlet-name>scServlet</servlet-name>
    <servlet-class>org.myapp.server.ScServlet</servlet-class>
</servlet>

<servlet-mapping>
    <servlet-name>scServlet</servlet-name>
    <url-pattern>/sc/*</url-pattern>
</servlet-mapping>

<servlet-mapping>
    <servlet-name>scServlet</servlet-name>
    <url-pattern>/myappname/sc/*</url-pattern>
</servlet-mapping>
4. Implement the Servlet:
Code:
package org.myapp.server;

import java.io.FileInputStream;
import java.io.IOException;
import java.io.OutputStream;
import java.util.logging.Logger;
import java.util.zip.ZipEntry;
import java.util.zip.ZipInputStream;

import javax.servlet.http.HttpServlet;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;

public class ScServlet extends HttpServlet {

	private static final Logger log = Logger.getLogger(ScServlet.class.getName());

	/**
	 * This servlet receives all '/sc/*' requests according to a rule defined in
	 * 'web.xml'. It looks for the necessary resource in 'sc.zip' which should
	 * be present in the application classpath.
	 */
	public void doGet(HttpServletRequest req, HttpServletResponse resp) throws IOException {

		// Sometimes the request URI begins with '/sc' and other times with
		// '/myappname/sc', so we always remove '/myappname' from the request URI
		String requestedURI = req.getRequestURI().replaceFirst("/myappname", "");

		// At this point requestedURI  begins with '/sc/...' but the zip file entries
		// begin with 'sc/...', so we remove the first '/'
		requestedURI = requestedURI.substring(1);

		log.info("Requested URI '" + req.getRequestURI() + "' converted to '" + requestedURI + "'");

		// search for resource
		ZipInputStream in = new ZipInputStream(new FileInputStream("sc.zip"));
		ZipEntry entry;
		while ((entry = in.getNextEntry()) != null) {

			if (requestedURI.equals(entry.getName())) {
				// resource found
				// redirect it to output stream
				OutputStream out = resp.getOutputStream();
				byte[] buf = new byte[1024];
				int len;
				while ((len = in.read(buf)) > 0) {
					out.write(buf, 0, len);
				}
				in.close();
				out.close();
				log.info("Requested '" + requestedURI + "' found in zip file entry: " + entry.getName());
				return;
			}
		}

		log.severe("Requested '" + requestedURI + "' not found in zip file!");
	}

}
5 - Upload your app!

If all goes well you will be able to see your app in GAE.

There are some performance tuning improvements to do. If you have any ideas please share them!

Happy coding...
Reply With Quote
  #2  
Old 16th Apr 2009, 12:34
dfreire dfreire is offline
Registered Developer
 
Join Date: Mar 2009
Posts: 43
Default

An additional warning: please don't use the above solution in your production deployment. I expect that the performance penalty for using it is prohibitive. For one thing, the zip file is compressed, meaning that there is additional time needed to uncompressed it each time a resource is needed. There are several improvements that come to mind, just to name a few:
- split the sc.zip into several uncompressed zip files
- previously store the position of important entries in the zip file
- use an in memory cache to serve the most frequent requests

Please consider the previous post as a simple first step to effectively use SmartGWT on GAE. I'm sure a lot of you can find improvements, please share them!
Reply With Quote
  #3  
Old 17th Apr 2009, 00:07
olinsha olinsha is offline
Registered Developer
 
Join Date: Jan 2009
Posts: 80
Default

thank you for that.

To answer to your question :

"Another Note: I don't know why, sometimes the request URI for 'sc' begins with '/sc/...' and other times with '/myappname/sc/...'. Maybe someone can explain this. My solution was to exclude both."

It depends if you renamed your module in fact (i.e if you use the rename-to attribute in your module). In that case you have to update your isomorphic_dir location to <my_app_name> and so your sc folder will be placed in it.
Else your sc folder will be at the root of your war file

My concern with your solution : as far as I know, the apps deployed on GAE will be placed on clouds, and clouds are not doing great in a performance aspect (I mean that an application deployed on GAE will not have the same performance than the same app deployed on a dedicated java server. So if we add the uncompress performance impact our web apps will be too slow.

I hope someone will be able to share with us a mean to get correct performance with our smartgwt applications deployed on GAE.

Regards
Reply With Quote
  #4  
Old 19th Apr 2009, 20:10
cworley cworley is offline
Registered Developer
 
Join Date: Nov 2008
Posts: 5
Default

Thanks for the instructions. I'm still getting the blob limit error. I followed your instructions, zipping the 'sc' directory in WinZip, modifying the .xml files, and adding the servlet.

Here's the Eclipse console:

Code:
...
Cloned 1100 files.
Cloned 1200 files.
Cloned 1300 files.
Cloned 1400 files.
Cloned 1500 files.
Uploading 1168 files.
Rolling back the update.
Unable to upload:
java.io.IOException: Error posting to URL: http://appengine.google.com/api/appversion/addfile?path=WEB-INF...
400 Bad Request
Max number of files and blobs is 1000.
After watching the deploy process, it looks like Eclipse is adding a 'sc' directory with all of the files to the 'war' folder. It happens when the console reads: "Creating staging directory". I'm not sure why it's doing that. Do you have any suggestions as to what I'm overlooking?

Thanks much.

Last edited by cworley; 19th Apr 2009 at 20:38..
Reply With Quote
  #5  
Old 23rd Apr 2009, 02:12
dfreire dfreire is offline
Registered Developer
 
Join Date: Mar 2009
Posts: 43
Default

Quote:
Originally Posted by cworley
After watching the deploy process, it looks like Eclipse is adding a 'sc' directory with all of the files to the 'war' folder. It happens when the console reads: "Creating staging directory". I'm not sure why it's doing that. Do you have any suggestions as to what I'm overlooking?
Yeah, we are not preventing eclipse from creating the 'sc' directory, but in 'appengine-web.xml' we are preventing the upload.

Can you double check your <static-files> and <resource-files> configurations?

Try playing arround with these configurations and check if that affects the number of files that eclipse tries to upload...

Let us know how it goes... :-)
Reply With Quote
  #6  
Old 28th Apr 2009, 17:16
cworley cworley is offline
Registered Developer
 
Join Date: Nov 2008
Posts: 5
Default

Thanks for the clarification. I'm having trouble running in hosted mode. The ScServlet doesn't seem to find the 'sc.zip' file in the 'war' directory, so it doesn't use css or image files. I updated the '/myappname' reference in the ScServlet.java file, but here are the errors that I get:

Code:
Apr 28, 2009 6:52:55 PM org.myapp.server.ScServlet doGet
SEVERE: Requested 'sc/skins/Enterprise/skin_styles.css' not found in zip file!
Apr 28, 2009 6:53:22 PM org.myapp.server.ScServlet doGet
SEVERE: Requested 'sc/skins/Enterprise/images/blank.gif' not found in zip file!
Apr 28, 2009 6:53:23 PM org.myapp.server.ScServlet doGet
SEVERE: Requested 'sc/skins/Enterprise/images/button/button_start.png' not found in zip file!
...
Reply With Quote
  #7  
Old 6th May 2009, 08:01
dfreire dfreire is offline
Registered Developer
 
Join Date: Mar 2009
Posts: 43
Default

Are you sure the code is looking for the zip file in the right place? I suggest you do some debugging and maybe insert some extra log messages...

Anyway, I must warn you that unzipping the file on every request will turn out to be an expensive operation on GAE, and you might want to cache the extracted zip contents, like this:

Code:
public void doGet(HttpServletRequest req, HttpServletResponse resp)
  throws IOException {

  String key = req.getRequestURI().replaceFirst("/myappname", "");
  try {
     resp.getOutputStream().write(getResourceFromCache(key));
  } catch (CacheException e) {
     resp.getOutputStream().write(getResourceFromZip(key));
  }
}


private byte[] getResourceFromCache(String key)
  throws CacheException, IOException {

  Cache cache = CacheManager.getInstance().getCacheFactory().createCache(Collections.emptyMap());
  byte[] resource = (byte[]) cache.get(key);

  if (resource == null) {
     resource = getResourceFromZip(key);
     cache.put(key, resource);
  }

  return resource;
}
Once the most frequent values are cached, the requests for 'sc' files will run quite rapidly...
Reply With Quote
  #8  
Old 26th May 2009, 01:34
Palantar Palantar is offline
Registered Developer
 
Join Date: May 2009
Posts: 1
Default Still too slow

Well, I ran into the same issues and followed along with this thread. Here is the whole version of the caching servlet:

Code:
package com.blah.server;

import java.io.ByteArrayOutputStream;
import java.io.FileInputStream;
import java.io.FileNotFoundException;
import java.io.IOException;
import java.util.Collections;
import java.util.logging.Logger;
import java.util.zip.ZipEntry;
import java.util.zip.ZipInputStream;

import javax.cache.CacheException;
import javax.cache.CacheManager;
import javax.servlet.http.HttpServlet;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;

public class ScServlet extends HttpServlet {

	private static final Logger log = Logger.getLogger(ScServlet.class.getName());

	/**
	 * This servlet receives all '/sc/*' requests according to a rule defined in
	 * 'web.xml'. It looks for the necessary resource in 'sc.zip' which should
	 * be present in the application classpath.
	 */
	public void doGet(HttpServletRequest req, HttpServletResponse resp) throws IOException {

		// Sometimes the request URI begins with '/sc' and other times with
		// '/myappname/sc', so we always remove '/myappname' from the request URI
		String requestedURI = req.getRequestURI().replaceFirst("/myproject", "");

		// At this point requestedURI  begins with '/sc/...' but the zip file entries
		// begin with 'sc/...', so we remove the first '/'
		requestedURI = requestedURI.substring(1);
		
		requestedURI = requestedURI.replaceAll("//", "/");

		log.info("Requested URI '" + req.getRequestURI() + "' converted to '" + requestedURI + "'");

		getResourceFromZip(resp, requestedURI);
		
		  try {
			     resp.getOutputStream().write(getResourceFromCache(resp,requestedURI));
			  } catch (CacheException e) {
			     resp.getOutputStream().write(getResourceFromZip(resp,requestedURI));
		}
	}
	
	private byte[] getResourceFromCache(HttpServletResponse resp,String key)
	  throws CacheException, IOException {

	  javax.cache.Cache cache = CacheManager.getInstance().getCacheFactory().createCache(Collections.emptyMap());
	  byte[] resource = (byte[]) cache.get(key);

	  if (resource == null) {
	     resource = getResourceFromZip(resp,key);
	     cache.put(key, resource);
	  }

	  return resource;
	}

	private byte[] getResourceFromZip(HttpServletResponse resp,String requestedURI) throws FileNotFoundException, IOException {
		// search for resource
		ZipInputStream in = new ZipInputStream(new FileInputStream("sc.zip"));
		ZipEntry entry;
		while ((entry = in.getNextEntry()) != null) {

			if (requestedURI.equals(entry.getName())) {
				// resource found
				// redirect it to output stream
				ByteArrayOutputStream out = new ByteArrayOutputStream();
				byte[] buf = new byte[1024];
				int len;
				while ((len = in.read(buf)) > 0) {
					out.write(buf, 0, len);
				}
				in.close();
				out.close();
				log.info("Requested '" + requestedURI + "' found in zip file entry: " + entry.getName());
				return out.toByteArray();
			}
		}

		log.severe("Requested '" + requestedURI + "' not found in zip file!");
		return null;
	}

}
It works, but unfortunately when used in real GoogleAppEngine scenarios it is FAR too slow. We/Google needs to come up with a better solution.
Reply With Quote
  #9  
Old 4th Jun 2009, 07:28
keigotanaka keigotanaka is offline
Registered Developer
 
Join Date: Jun 2009
Posts: 2
Default

Quote:
Originally Posted by cworley
Code:
Apr 28, 2009 6:52:55 PM org.myapp.server.ScServlet doGet
SEVERE: Requested 'sc/skins/Enterprise/skin_styles.css' not found in zip file!
Apr 28, 2009 6:53:22 PM org.myapp.server.ScServlet doGet
SEVERE: Requested 'sc/skins/Enterprise/images/blank.gif' not found in zip file!
Apr 28, 2009 6:53:23 PM org.myapp.server.ScServlet doGet
SEVERE: Requested 'sc/skins/Enterprise/images/button/button_start.png' not found in zip file!
...
I got same error above, but it's just my misunderstanding for sc.zip.

Quote:
Originally Posted by dfreire
1. Zip your 'sc' dir and place the produced file in your 'war' dir.
I misunderstood above as that 'sc' dir is from smartgwt.jar..
And, 'sc' dir is from "war/yourappname/sc" on eclipse by right.
Zip the dir and move the created zip file into "war/sc.zip".
And works well.
Reply With Quote
  #10  
Old 4th Jun 2009, 07:49
keigotanaka keigotanaka is offline
Registered Developer
 
Join Date: Jun 2009
Posts: 2
Default

Quote:
Originally Posted by Palantar
It works, but unfortunately when used in real GoogleAppEngine scenarios it is FAR too slow. We/Google needs to come up with a better solution.
Remove below line before try block in doGet method.
Code:
getResourceFromZip(resp, requestedURI);
Reply With Quote
Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search


© 2010,2011 Isomorphic Software. All Rights Reserved