Announcement

Collapse
No announcement yet.
X
  • Filter
  • Time
Clear All
new posts

  • How to: SmartGWT on Google App Engine (Java)

    Hello fellow developers,

    Ok, so I've finally been able to upload my SmartGWT app into GAE.

    First of all, if you try to upload your SmartGWT app into GAE without any changes you'll notice an error like:
    Code:
    400 Bad Request
    Max number of files and blobs is 1000.
    This happens because there are probably more than 1000 files in your generated smartgwt 'sc' dir.

    So far, the proposed solution to this problem was to use another host (like Amazon S3) for the 'sc' static files. You have a thread explaining how to do it here: http://forums.smartclient.com/showthread.php?t=5105

    Another approach is to zip the contents of the 'sc' dir and have the needed files unzipped on the fly using a servlet. This was originally proposed by a google developer during the last GAE IRC chat. (This is also a similar solution to the deployment of SmartGWT apps in GAE-Python, as suggested in other threads of this forum.)

    So, how to do it:

    1. Zip your 'sc' dir and place the produced file in your 'war' dir.
    Note that the zip file must be compressed. The uncompressed file will have more than 10 MB and that will also originate an error during upload to GAE.

    2. Edit your 'appengine-web.xml' file and exclude all 'sc' files from your app resources. Like this:
    Code:
    <!-- By default all WAR files are static except WEB-INF  -->
    <!-- This overrides the default -->
    <static-files>
        <!-- Exclude smartgwt files from the list of static files -->
        <exclude path="/sc/**.*" />
        <exclude path="/myappname/sc/**.*" />
    </static-files>
    
    <!-- By default all WAR files are also resource files including WEB-INF -->
    <!-- This overrides the default -->
    <resource-files>
        <!-- Exclude smartgwt files from the list of resource files -->
        <exclude path="/sc/**.*" />
        <exclude path="/myappname/sc/**.*" />
    </resource-files>
    Note: you must substitute 'myappname' by your app name.
    Another Note: I don't know why, sometimes the request URI for 'sc' begins with '/sc/...' and other times with '/myappname/sc/...'. Maybe someone can explain this. My solution was to exclude both.

    3. In your 'web.xml' file, create a new servlet that will catch all requests to 'sc':
    Code:
    <!-- SmartGWT static resources servlet -->
    <servlet>
        <servlet-name>scServlet</servlet-name>
        <servlet-class>org.myapp.server.ScServlet</servlet-class>
    </servlet>
    
    <servlet-mapping>
        <servlet-name>scServlet</servlet-name>
        <url-pattern>/sc/*</url-pattern>
    </servlet-mapping>
    
    <servlet-mapping>
        <servlet-name>scServlet</servlet-name>
        <url-pattern>/myappname/sc/*</url-pattern>
    </servlet-mapping>
    4. Implement the Servlet:
    Code:
    package org.myapp.server;
    
    import java.io.FileInputStream;
    import java.io.IOException;
    import java.io.OutputStream;
    import java.util.logging.Logger;
    import java.util.zip.ZipEntry;
    import java.util.zip.ZipInputStream;
    
    import javax.servlet.http.HttpServlet;
    import javax.servlet.http.HttpServletRequest;
    import javax.servlet.http.HttpServletResponse;
    
    public class ScServlet extends HttpServlet {
    
    	private static final Logger log = Logger.getLogger(ScServlet.class.getName());
    
    	/**
    	 * This servlet receives all '/sc/*' requests according to a rule defined in
    	 * 'web.xml'. It looks for the necessary resource in 'sc.zip' which should
    	 * be present in the application classpath.
    	 */
    	public void doGet(HttpServletRequest req, HttpServletResponse resp) throws IOException {
    
    		// Sometimes the request URI begins with '/sc' and other times with
    		// '/myappname/sc', so we always remove '/myappname' from the request URI
    		String requestedURI = req.getRequestURI().replaceFirst("/myappname", "");
    
    		// At this point requestedURI  begins with '/sc/...' but the zip file entries
    		// begin with 'sc/...', so we remove the first '/'
    		requestedURI = requestedURI.substring(1);
    
    		log.info("Requested URI '" + req.getRequestURI() + "' converted to '" + requestedURI + "'");
    
    		// search for resource
    		ZipInputStream in = new ZipInputStream(new FileInputStream("sc.zip"));
    		ZipEntry entry;
    		while ((entry = in.getNextEntry()) != null) {
    
    			if (requestedURI.equals(entry.getName())) {
    				// resource found
    				// redirect it to output stream
    				OutputStream out = resp.getOutputStream();
    				byte[] buf = new byte[1024];
    				int len;
    				while ((len = in.read(buf)) > 0) {
    					out.write(buf, 0, len);
    				}
    				in.close();
    				out.close();
    				log.info("Requested '" + requestedURI + "' found in zip file entry: " + entry.getName());
    				return;
    			}
    		}
    
    		log.severe("Requested '" + requestedURI + "' not found in zip file!");
    	}
    
    }
    5 - Upload your app!

    If all goes well you will be able to see your app in GAE.

    There are some performance tuning improvements to do. If you have any ideas please share them!

    Happy coding...

  • #2
    An additional warning: please don't use the above solution in your production deployment. I expect that the performance penalty for using it is prohibitive. For one thing, the zip file is compressed, meaning that there is additional time needed to uncompressed it each time a resource is needed. There are several improvements that come to mind, just to name a few:
    - split the sc.zip into several uncompressed zip files
    - previously store the position of important entries in the zip file
    - use an in memory cache to serve the most frequent requests

    Please consider the previous post as a simple first step to effectively use SmartGWT on GAE. I'm sure a lot of you can find improvements, please share them!

    Comment


    • #3
      thank you for that.

      To answer to your question :

      "Another Note: I don't know why, sometimes the request URI for 'sc' begins with '/sc/...' and other times with '/myappname/sc/...'. Maybe someone can explain this. My solution was to exclude both."

      It depends if you renamed your module in fact (i.e if you use the rename-to attribute in your module). In that case you have to update your isomorphic_dir location to <my_app_name> and so your sc folder will be placed in it.
      Else your sc folder will be at the root of your war file

      My concern with your solution : as far as I know, the apps deployed on GAE will be placed on clouds, and clouds are not doing great in a performance aspect (I mean that an application deployed on GAE will not have the same performance than the same app deployed on a dedicated java server. So if we add the uncompress performance impact our web apps will be too slow.

      I hope someone will be able to share with us a mean to get correct performance with our smartgwt applications deployed on GAE.

      Regards

      Comment


      • #4
        Thanks for the instructions. I'm still getting the blob limit error. I followed your instructions, zipping the 'sc' directory in WinZip, modifying the .xml files, and adding the servlet.

        Here's the Eclipse console:

        Code:
        ...
        Cloned 1100 files.
        Cloned 1200 files.
        Cloned 1300 files.
        Cloned 1400 files.
        Cloned 1500 files.
        Uploading 1168 files.
        Rolling back the update.
        Unable to upload:
        java.io.IOException: Error posting to URL: http://appengine.google.com/api/appversion/addfile?path=WEB-INF...
        400 Bad Request
        Max number of files and blobs is 1000.
        After watching the deploy process, it looks like Eclipse is adding a 'sc' directory with all of the files to the 'war' folder. It happens when the console reads: "Creating staging directory". I'm not sure why it's doing that. Do you have any suggestions as to what I'm overlooking?

        Thanks much.
        Last edited by cworley; 19th Apr 2009, 20:38.

        Comment


        • #5
          Originally posted by cworley
          After watching the deploy process, it looks like Eclipse is adding a 'sc' directory with all of the files to the 'war' folder. It happens when the console reads: "Creating staging directory". I'm not sure why it's doing that. Do you have any suggestions as to what I'm overlooking?
          Yeah, we are not preventing eclipse from creating the 'sc' directory, but in 'appengine-web.xml' we are preventing the upload.

          Can you double check your <static-files> and <resource-files> configurations?

          Try playing arround with these configurations and check if that affects the number of files that eclipse tries to upload...

          Let us know how it goes... :-)

          Comment


          • #6
            Thanks for the clarification. I'm having trouble running in hosted mode. The ScServlet doesn't seem to find the 'sc.zip' file in the 'war' directory, so it doesn't use css or image files. I updated the '/myappname' reference in the ScServlet.java file, but here are the errors that I get:

            Code:
            Apr 28, 2009 6:52:55 PM org.myapp.server.ScServlet doGet
            SEVERE: Requested 'sc/skins/Enterprise/skin_styles.css' not found in zip file!
            Apr 28, 2009 6:53:22 PM org.myapp.server.ScServlet doGet
            SEVERE: Requested 'sc/skins/Enterprise/images/blank.gif' not found in zip file!
            Apr 28, 2009 6:53:23 PM org.myapp.server.ScServlet doGet
            SEVERE: Requested 'sc/skins/Enterprise/images/button/button_start.png' not found in zip file!
            ...

            Comment


            • #7
              Are you sure the code is looking for the zip file in the right place? I suggest you do some debugging and maybe insert some extra log messages...

              Anyway, I must warn you that unzipping the file on every request will turn out to be an expensive operation on GAE, and you might want to cache the extracted zip contents, like this:

              Code:
              public void doGet(HttpServletRequest req, HttpServletResponse resp)
                throws IOException {
              
                String key = req.getRequestURI().replaceFirst("/myappname", "");
                try {
                   resp.getOutputStream().write(getResourceFromCache(key));
                } catch (CacheException e) {
                   resp.getOutputStream().write(getResourceFromZip(key));
                }
              }
              
              
              private byte[] getResourceFromCache(String key)
                throws CacheException, IOException {
              
                Cache cache = CacheManager.getInstance().getCacheFactory().createCache(Collections.emptyMap());
                byte[] resource = (byte[]) cache.get(key);
              
                if (resource == null) {
                   resource = getResourceFromZip(key);
                   cache.put(key, resource);
                }
              
                return resource;
              }
              Once the most frequent values are cached, the requests for 'sc' files will run quite rapidly...

              Comment


              • #8
                Still too slow

                Well, I ran into the same issues and followed along with this thread. Here is the whole version of the caching servlet:

                Code:
                package com.blah.server;
                
                import java.io.ByteArrayOutputStream;
                import java.io.FileInputStream;
                import java.io.FileNotFoundException;
                import java.io.IOException;
                import java.util.Collections;
                import java.util.logging.Logger;
                import java.util.zip.ZipEntry;
                import java.util.zip.ZipInputStream;
                
                import javax.cache.CacheException;
                import javax.cache.CacheManager;
                import javax.servlet.http.HttpServlet;
                import javax.servlet.http.HttpServletRequest;
                import javax.servlet.http.HttpServletResponse;
                
                public class ScServlet extends HttpServlet {
                
                	private static final Logger log = Logger.getLogger(ScServlet.class.getName());
                
                	/**
                	 * This servlet receives all '/sc/*' requests according to a rule defined in
                	 * 'web.xml'. It looks for the necessary resource in 'sc.zip' which should
                	 * be present in the application classpath.
                	 */
                	public void doGet(HttpServletRequest req, HttpServletResponse resp) throws IOException {
                
                		// Sometimes the request URI begins with '/sc' and other times with
                		// '/myappname/sc', so we always remove '/myappname' from the request URI
                		String requestedURI = req.getRequestURI().replaceFirst("/myproject", "");
                
                		// At this point requestedURI  begins with '/sc/...' but the zip file entries
                		// begin with 'sc/...', so we remove the first '/'
                		requestedURI = requestedURI.substring(1);
                		
                		requestedURI = requestedURI.replaceAll("//", "/");
                
                		log.info("Requested URI '" + req.getRequestURI() + "' converted to '" + requestedURI + "'");
                
                		getResourceFromZip(resp, requestedURI);
                		
                		  try {
                			     resp.getOutputStream().write(getResourceFromCache(resp,requestedURI));
                			  } catch (CacheException e) {
                			     resp.getOutputStream().write(getResourceFromZip(resp,requestedURI));
                		}
                	}
                	
                	private byte[] getResourceFromCache(HttpServletResponse resp,String key)
                	  throws CacheException, IOException {
                
                	  javax.cache.Cache cache = CacheManager.getInstance().getCacheFactory().createCache(Collections.emptyMap());
                	  byte[] resource = (byte[]) cache.get(key);
                
                	  if (resource == null) {
                	     resource = getResourceFromZip(resp,key);
                	     cache.put(key, resource);
                	  }
                
                	  return resource;
                	}
                
                	private byte[] getResourceFromZip(HttpServletResponse resp,String requestedURI) throws FileNotFoundException, IOException {
                		// search for resource
                		ZipInputStream in = new ZipInputStream(new FileInputStream("sc.zip"));
                		ZipEntry entry;
                		while ((entry = in.getNextEntry()) != null) {
                
                			if (requestedURI.equals(entry.getName())) {
                				// resource found
                				// redirect it to output stream
                				ByteArrayOutputStream out = new ByteArrayOutputStream();
                				byte[] buf = new byte[1024];
                				int len;
                				while ((len = in.read(buf)) > 0) {
                					out.write(buf, 0, len);
                				}
                				in.close();
                				out.close();
                				log.info("Requested '" + requestedURI + "' found in zip file entry: " + entry.getName());
                				return out.toByteArray();
                			}
                		}
                
                		log.severe("Requested '" + requestedURI + "' not found in zip file!");
                		return null;
                	}
                
                }
                It works, but unfortunately when used in real GoogleAppEngine scenarios it is FAR too slow. We/Google needs to come up with a better solution.

                Comment


                • #9
                  Originally posted by cworley
                  Code:
                  Apr 28, 2009 6:52:55 PM org.myapp.server.ScServlet doGet
                  SEVERE: Requested 'sc/skins/Enterprise/skin_styles.css' not found in zip file!
                  Apr 28, 2009 6:53:22 PM org.myapp.server.ScServlet doGet
                  SEVERE: Requested 'sc/skins/Enterprise/images/blank.gif' not found in zip file!
                  Apr 28, 2009 6:53:23 PM org.myapp.server.ScServlet doGet
                  SEVERE: Requested 'sc/skins/Enterprise/images/button/button_start.png' not found in zip file!
                  ...
                  I got same error above, but it's just my misunderstanding for sc.zip.

                  Originally posted by dfreire
                  1. Zip your 'sc' dir and place the produced file in your 'war' dir.
                  I misunderstood above as that 'sc' dir is from smartgwt.jar..
                  And, 'sc' dir is from "war/yourappname/sc" on eclipse by right.
                  Zip the dir and move the created zip file into "war/sc.zip".
                  And works well.

                  Comment


                  • #10
                    Originally posted by Palantar
                    It works, but unfortunately when used in real GoogleAppEngine scenarios it is FAR too slow. We/Google needs to come up with a better solution.
                    Remove below line before try block in doGet method.
                    Code:
                    getResourceFromZip(resp, requestedURI);

                    Comment


                    • #11
                      Originally posted by keigotanaka
                      I got same error above, but it's just my misunderstanding for sc.zip.

                      Originally posted by dfreire
                      1. Zip your 'sc' dir and place the produced file in your 'war' dir.
                      I misunderstood above as that 'sc' dir is from smartgwt.jar..
                      And, 'sc' dir is from "war/yourappname/sc" on eclipse by right.
                      Zip the dir and move the created zip file into "war/sc.zip".
                      And works well.

                      That's right keigotanaka, thanks for clarifying this. It was exactly what I meant. (Considering you are using 'yourappname' as your 'rename-to' property on your module's XXX.gwt.xml file)
                      Last edited by dfreire; 5th Jun 2009, 03:52.

                      Comment


                      • #12
                        One issue i noticed

                        One issue that I noticed was that after doing the steps outlined in this thread when I viewed my application on appengine all images seemed to be broken. In the myapp.html file (i.e war\myapp.html) I changed the isomorphicDir javascript variable from:

                        <script>
                        var isomorphicDir = "myapp/sc/"
                        </script>

                        to

                        <script>
                        var isomorphicDir = "/sc/"
                        </script>

                        and all images are now working correctly.

                        Thanks for the solution btw.

                        Comment


                        • #13
                          GAE/J increased file limit to 3000

                          SDK 1.2.2 increased file limit to 3000 and seemed resolved this issue.

                          Comment


                          • #14
                            Deployment work fine but its too slow

                            All,

                            I faced the dame issue regarding deployment but was able to resolve the same by following the above solutions. Thanks everyone :)

                            Now issue is about performance. Its too slow giving a very bad user experience.

                            Any help would be really great !

                            Comment


                            • #15
                              ant script to do the zipping

                              Thanks everyone for the excellent workaround. If you're using ant on your project these are the lines you need to do the zipping (I put them on the end of my gwtc target):

                              Code:
                              <!--zip smartGWT dir (see http://forums.smartclient.com/showthread.php?t=5258)-->
                              <zip destfile="${war.dir}/sc.zip" basedir="${war.dir}/myappname/" includes="sc/**"/>
                              <delete dir="${war.dir}/myappname/sc" />

                              Comment

                              Working...
                              X