Announcement

Collapse
No announcement yet.
X
  • Filter
  • Time
Clear All
new posts

    Uploading large files

    Isomorphic SmartClient/SmartGWT Framework (v9.0p_2014-02-13/PowerEdition Deployment 2014-02-13)

    I'm having an issue with using FileItem with large files.

    Here is what the code looks like on the UI Side:

    DynamicForm fileForm = new DynamicForm();
    fileForm.setEncoding( Encoding.MULTIPART );
    fileForm.setDataSource( DataSource.get( FileFormDataSourceFields.DATASOURCE_NAME ) );

    final FileItem fileItem = new FileItem( FileFormDataSourceFields.FILE, "File" );
    fileItem.setMultiple( false );
    fileItem.setRequired( Boolean.TRUE );

    final TextItem displayName = new TextItem( FileFormDataSourceFields.NAME, "Display Name" );
    displayName.setRequired( Boolean.TRUE );

    fileForm.setFields( fileItem, displayName );


    On the serverside, I get access to the list of ISCFileItem objects (in my case there will only be one because it is not a multi-file upload), and I access the byte[] array through the call to get().

    This works with files under 250M, but anything larger, and the server runs out of memory.

    From the log:

    [DEBUG][02/14/2014 11:43:16:904] filled 16384/16384 (org.eclipse.jetty.http.HttpParser)
    [DEBUG][02/14/2014 11:43:16:905] filled 16384/16384 (org.eclipse.jetty.http.HttpParser)
    [DEBUG][02/14/2014 11:43:16:905] filled 16384/16384 (org.eclipse.jetty.http.HttpParser)
    [DEBUG][02/14/2014 11:43:16:905] filled 16384/16384 (org.eclipse.jetty.http.HttpParser)
    [DEBUG][02/14/2014 11:43:16:905] filled 16384/16384 (org.eclipse.jetty.http.HttpParser)
    [DEBUG][02/14/2014 11:43:16:905] filled 16384/16384 (org.eclipse.jetty.http.HttpParser)
    [DEBUG][02/14/2014 11:43:16:905] filled 16384/16384 (org.eclipse.jetty.http.HttpParser)
    [DEBUG][02/14/2014 11:43:16:906] filled 16384/16384 (org.eclipse.jetty.http.HttpParser)
    [DEBUG][02/14/2014 11:43:16:906] filled 16384/16384 (org.eclipse.jetty.http.HttpParser)
    [DEBUG][02/14/2014 11:43:16:906] filled 16384/16384 (org.eclipse.jetty.http.HttpParser)
    [DEBUG][02/14/2014 11:43:16:906] filled 16384/16384 (org.eclipse.jetty.http.HttpParser)
    [DEBUG][02/14/2014 11:43:16:906] filled 16384/16384 (org.eclipse.jetty.http.HttpParser)
    [DEBUG][02/14/2014 11:43:16:906] filled 16384/16384 (org.eclipse.jetty.http.HttpParser)
    [DEBUG][02/14/2014 11:43:16:906] filled 16384/16384 (org.eclipse.jetty.http.HttpParser)
    [DEBUG][02/14/2014 11:43:16:906] filled 16384/16384 (org.eclipse.jetty.http.HttpParser)
    [DEBUG][02/14/2014 11:43:16:906] filled 16384/16384 (org.eclipse.jetty.http.HttpParser)
    [DEBUG][02/14/2014 11:43:16:907] filled 16384/16384 (org.eclipse.jetty.http.HttpParser)
    [DEBUG][02/14/2014 11:43:16:907] filled 16384/16384 (org.eclipse.jetty.http.HttpParser)
    [DEBUG][02/14/2014 11:43:16:907] filled 16384/16384 (org.eclipse.jetty.http.HttpParser)
    [DEBUG][02/14/2014 11:43:16:907] filled 16384/16384 (org.eclipse.jetty.http.HttpParser)
    [DEBUG][02/14/2014 11:43:16:907] filled 16384/16384 (org.eclipse.jetty.http.HttpParser)
    [DEBUG][02/14/2014 11:43:16:907] filled 16384/16384 (org.eclipse.jetty.http.HttpParser)
    [DEBUG][02/14/2014 11:43:16:907] filled 16384/16384 (org.eclipse.jetty.http.HttpParser)
    [DEBUG][02/14/2014 11:43:16:907] filled 16384/16384 (org.eclipse.jetty.http.HttpParser)
    [DEBUG][02/14/2014 11:43:16:907] filled 16384/16384 (org.eclipse.jetty.http.HttpParser)
    [DEBUG][02/14/2014 11:43:16:907] filled 16384/16384 (org.eclipse.jetty.http.HttpParser)
    [DEBUG][02/14/2014 11:43:16:907] filled 16384/16384 (org.eclipse.jetty.http.HttpParser)
    [DEBUG][02/14/2014 11:43:16:907] filled 16384/16384 (org.eclipse.jetty.http.HttpParser)
    [DEBUG][02/14/2014 11:43:16:907] filled 16384/16384 (org.eclipse.jetty.http.HttpParser)
    [DEBUG][02/14/2014 11:43:16:907] filled 16384/16384 (org.eclipse.jetty.http.HttpParser)
    [DEBUG][02/14/2014 11:43:16:907] filled 16384/16384 (org.eclipse.jetty.http.HttpParser)
    [DEBUG][02/14/2014 11:43:16:908] filled 16384/16384 (org.eclipse.jetty.http.HttpParser)
    [DEBUG][02/14/2014 11:43:16:908] filled 16384/16384 (org.eclipse.jetty.http.HttpParser)
    [DEBUG][02/14/2014 11:43:16:908] filled 16384/16384 (org.eclipse.jetty.http.HttpParser)
    [ERROR][02/14/2014 11:43:16:909] org.eclipse.equinox.http.registry.internal.ServletManager$ServletWrapper - Top-level servlet error: (com.isomorphic.servlet.IDACall)
    java.lang.OutOfMemoryError: Java heap space
    at java.util.Arrays.copyOf(Arrays.java:2271)
    at java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:113)
    at java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93)
    at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:122)
    at java.io.FilterOutputStream.write(FilterOutputStream.java:77)
    at com.isomorphic.io.ByteCountingOutputStream.write(ByteCountingOutputStream.java:45)
    at java.io.FilterOutputStream.write(FilterOutputStream.java:125)
    at org.apache.commons.fileupload.util.Streams.copy(Streams.java:101)
    at org.apache.commons.fileupload.util.Streams.copy(Streams.java:64)
    at org.apache.commons.fileupload.FileUploadBase.parseRequest(FileUploadBase.java:362)
    at org.apache.commons.fileupload.FileUploadBase.parseRequest(FileUploadBase.java:310)
    at com.isomorphic.servlet.ISCHttpServletRequest.parseRequest(ISCHttpServletRequest.java:247)
    at com.isomorphic.servlet.ISCHttpServletRequest.parseRequest(ISCHttpServletRequest.java:219)
    at com.isomorphic.servlet.ISCHttpServletRequest.getStringParams(ISCHttpServletRequest.java:119)
    at com.isomorphic.servlet.ISCHttpServletRequest.getParameter(ISCHttpServletRequest.java:300)
    at com.isomorphic.rpc.RPCManager.parseRequest(RPCManager.java:2017)
    at com.isomorphic.rpc.RPCManager.<init>(RPCManager.java:306)
    at com.isomorphic.rpc.RPCManager.<init>(RPCManager.java:286)
    at com.isomorphic.servlet.IDACall.doPost(IDACall.java:73)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:755)
    at com.isomorphic.servlet.BaseServlet.service(BaseServlet.java:152)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:848)
    at org.eclipse.equinox.http.registry.internal.ServletManager$ServletWrapper.service(ServletManager.java:180)
    at org.eclipse.equinox.http.servlet.internal.ServletRegistration.service(ServletRegistration.java:61)
    at org.eclipse.equinox.http.servlet.internal.ProxyServlet.processAlias(ProxyServlet.java:128)
    at org.eclipse.equinox.http.servlet.internal.ProxyServlet.service(ProxyServlet.java:60)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:848)
    at org.eclipse.equinox.http.jetty.internal.HttpServerManager$InternalHttpServiceServlet.service(HttpServerManager.java:384)
    at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:598)
    at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:486)
    at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)



    So, this looks like all of the file is streamed into memory. This causes the server to run out of memory with large files. I'm also worried that even with small files, if enough users are uploading at the same time, this same issue would occur. I can't find an option to tell SmartGWT to stream the file data into a temporary file instead of keeping it all in memory. How can I do this in order to prevent OutOfMemory errors?

    #2
    Currently we don't have a setting that would avoid having the entire file in memory when processed as a standard iscServer request. Feature Sponsorship could be used to add such a feature, but we'd probably do it based on avoiding parsing the entire request whenever possible, rather than temp files (fast, writable filesystems are absent in many deployment scenarios).

    However you can target an ordinary servlet and still use the DataSource layer to store the file to SQL, Hibernate etc without having the file in memory - the sample code below would be the doPost() method for a servlet that you target with dynamicForm.saveData() with an UploadItem present:

    Code:
            // Create a new file upload handler
            ServletFileUpload upload = new ServletFileUpload();
    
            // Parse the request - assumes we only have one upload in the request, and we 
            // aren't interested in anything except the uploaded content
            try {
                FileItemIterator iter = upload.getItemIterator(request);
                if (iter.hasNext()) {
                    FileItemStream item = iter.next();
                    String name = item.getFieldName();
                    DSRequest req = new DSRequest("TestUpload", "add");
                    req.setFieldValue(name, item.openStream());
                    req.execute();
                }
            } catch(Exception e) {
                e.printStackTrace();
            }
    This is using the Commons FileUpload streaming API to grab an InputStream for the uploaded content, and stuff it into the values of a new DSRequest, which is then executed. If you are using a servlet 3.0 container (Tomcat 7, for example) you could use the built-in file upload API instead of Commons FileUpload if you prefer.

    Comment


      #3
      Just to be clear:
      With the servlet, I would need to change the client-side to use an UploadItem instead of a FileItem?

      What would I need to do on the client-side to target the servlet?

      Comment


        #4
        With the code as shown, either an UploadItem or FileItem would work. However, the primary point of FileItem is to allow things like validation errors to be handled in a uniform way even when doing an upload, and that can't be accomplished in this scenario because commons ServletFileUpload is parsing the request, so there's little point in using FileItem over UploadItem here.

        Use actionURL in the DSRequest argument to saveData() to target your servlet.

        Comment


          #5
          Thanks.

          I have a couple of more questions about this mechanism.

          1. How do I get access to the other datasource fields from the request?
          2. How do I fill in the HttpServletResponse so the response goes back to the client as it would with a regular DSRequest/DSResponse?

          Comment


            #6
            That's not possible with thsi approach. The parsing logic that parses the iscServer format is the logic that will pull the entire file stream into memory, so you can't call it.

            What you can do is send any additional fields you want via dsRequest.httpParams, and get them through standard servlet APIs like getParameter().

            Comment


              #7
              Ok, I understand this answer for question #1. I've got it that I can now retrieve the information and pass it through a DSRequest from the servlet.

              Now, in terms of handling a response. How do I send back the response such that when I call saveData with a callback, the callback will get called.

              My request from the client side looks like this:
              final DSRequest requestProperties = new DSRequest();
              requestProperties.setWillHandleError( true );
              requestProperties.setTimeout( AddDevicePackageDlg.FILE_UPLOAD_TIMEOUT );
              requestProperties.setActionURL( AddDevicePackageDlg.this.getAlias() + "/MyFileUpload" );

              this.myFileForm.saveData( new SaveCallback(), requestProperties );

              I want the execute of the SaveCallback (derived from DSCallback) to be called.

              Comment


                #8
                So again, you're outside the normal DSRequest/DSResponse processing cycle, including things like being able to get a normal DSRequest.callback to fire after doing an upload.

                In terms of being able to set up a callback, your situation is essentially similar to the section on "Upload without the Smart GWT Server" from the Uploading Files overview.

                Another option would be to poll the server to discover some consequence of the upload (like a new DB record that represents it), and fire the callback when you see this record appear.

                Comment


                  #9
                  I'm still having an issue with this.

                  I'm using RealTimeMessaging to signal back when it is done, and I get the callback without issue.

                  However, the call to saveData just hangs such that I get an hourglass in the browser.

                  In the SmartClient Developer Console, the "add" operation remains in a Transaction in Progess state. What do I need to send back in the response in order for the saveData call to complete?

                  Comment


                    #10
                    The form is stuck waiting for the expected dataProtocol:"iscServer" response from the server, which will never arrive.

                    The most correct way to resolve this is the previously suggested "Upload without the Smart GWT Server" approach.

                    A moderate hack would be to set showPrompt:false via the DSRequest argument of saveData() (turns off the hourglass) and also set dsRequest.timeout to 0 prevent eventual request timeout. This basically leaves the request as a hung request, which will be mostly harmless - just an entry in the RPC tab that never clears.

                    Comment


                      #11
                      I'm not sure I understand your response. Your suggestion on Feb. 14th was:
                      "However you can target an ordinary servlet and still use the DataSource layer to store the file to SQL, Hibernate etc without having the file in memory - the sample code below would be the doPost() method for a servlet that you target with dynamicForm.saveData() with an UploadItem present"

                      I'm using this sample.

                      Comment


                        #12
                        Yes, we provided you a sample of server code - code which does not involve sending any response to the client. This doesn't conflict in any way with the test rest of the advice here, which is about what client code you might use, and how you could provide a response from the server after you've saved the file.

                        Comment


                          #13
                          Your answer did say to call DynamicForm.saveData().

                          Should I *not* do that?

                          If not, what should I call instead? I tried calling submit() instead, but that ends up taking me completely out of the application after it is processed.

                          Comment


                            #14
                            That's strange, it doesn't seem like we're being particularly ambiguous. After all, we just said:

                            A moderate hack would be to set showPrompt:false via the DSRequest argument of saveData() (turns off the hourglass) and also set dsRequest.timeout to 0 prevent eventual request timeout.
                            To avoid the hack, again use the approach under "Upload without the Smart GWT Server" from the Uploading Files overview. This can involve either saveData() or submitForm() depending on how it's done, and the overview explicitly discusses the issue of how to do a background upload (and hence avoid wiping out the page).

                            Comment


                              #15
                              I would like to avoid the hack.

                              The part I find ambiguous is the part about calling saveData(). Does saveData() not end up making the call as if it is the SmartGWT server? Is there some option to change the behaviour of saveData()? Can you explain "depending on how it is done" -- what options are there from the client side?

                              I can try to do it in an IFRAME as suggested in your "Background upload without the Smart GWT Server" of the overview. That seems to require calling submitForm instead of saveData. Is that not the case?

                              Comment

                              Working...
                              X