Announcement

Collapse
No announcement yet.
X
  • Filter
  • Time
Clear All
new posts

    saveAllEdits() from a listgrid creates a form post > default maxFormContentSize

    Dear isomorphic support:
    I am performing a batch update operation that involves auto-filling a column in a listgrid programmatically. During the saveAllEdits() operation, the form that the framework is trying to post is blowing the 200k default maxFormContentSize setting of Jetty. The grid is being populated with 100 records from a CSV file which contains only about 4K of data. The framework is trying to post a form of 211835 bytes. Is this expected or do I have an issue? It seems like a lot of overhead to simple post a total of 4K of record data. I'm getting form posts around 5 meg in size when at around 2500 records from the same data source.

    Details below:
    Product: SmartGWT 3.0p nightly build Nov 15, 2012
    Browser: IE 9
    Sample Code:
    Code:
    ListGridRecord[] records = grid.getSelectedRecords();
    for (Record record : records) {
       Map<String, Object> recMap = record.toMap();
       int rowNum = grid.getRecordIndex(record);
       recMap.put("test_col", "test value");
       grid.setEditValues(rowNum, recMap);
    }
    grid.saveAllEdits();
    Sample 100 record data set:
    Code:
    1,a,1,1@foo.com,Role 1,1,1,1
    2,b,2,2@foo.com,Role 2,2,2,2
    3,a,3,1@foo.com,Role 3,3,3,3
    4,b,4,2@foo.com,Role 4,4,4,4
    5,a,5,1@foo.com,Role 5,5,5,5
    6,b,6,2@foo.com,Role 6,6,6,6
    7,a,7,1@foo.com,Role 7,7,7,7
    8,b,8,2@foo.com,Role 8,8,8,8
    9,a,9,1@foo.com,Role 9,9,9,9
    10,b,10,2@foo.com,Role 10,10,10,10
    11,a,11,1@foo.com,Role 11,11,11,11
    12,b,12,2@foo.com,Role 12,12,12,12
    13,a,13,1@foo.com,Role 13,13,13,13
    14,b,14,2@foo.com,Role 14,14,14,14
    15,a,15,1@foo.com,Role 15,15,15,15
    16,b,16,2@foo.com,Role 16,16,16,16
    17,a,17,1@foo.com,Role 17,17,17,17
    18,b,18,2@foo.com,Role 18,18,18,18
    19,a,19,1@foo.com...,Role 19,19,19,19
    20,b,20,2@foo.com,Role 20,20,20,20
    21,a,21,1@foo.com,Role 21,21,21,21
    22,b,22,2@foo.com,Role 22,22,22,22
    23,a,23,1@foo.com,Role 23,23,23,23
    24,b,24,2@foo.com,Role 24,24,24,24
    25,a,25,1@foo.com,Role 25,25,25,25
    26,b,26,2@foo.com,Role 26,26,26,26
    27,a,27,1@foo.com,Role 27,27,27,27
    28,b,28,2@foo.com,Role 28,28,28,28
    29,a,29,1@foo.com,Role 29,29,29,29
    30,b,30,2@foo.com,Role 30,30,30,30
    31,a,31,1@foo.com,Role 31,31,31,31
    32,b,32,2@foo.com,Role 32,32,32,32
    33,a,33,1@foo.com,Role 33,33,33,33
    34,b,34,2@foo.com,Role 34,34,34,34
    35,a,35,1@foo.com,Role 35,35,35,35
    36,b,36,2@foo.com,Role 36,36,36,36
    37,a,37,1@foo.com,Role 37,37,37,37
    38,b,38,2@foo.com,Role 38,38,38,38
    39,a,39,1@foo.com,Role 39,39,39,39
    40,b,40,2@foo.com,Role 40,40,40,40
    41,a,41,1@foo.com,Role 41,41,41,41
    42,b,42,2@foo.com,Role 42,42,42,42
    43,a,43,1@foo.com,Role 43,43,43,43
    44,b,44,2@foo.com,Role 44,44,44,44
    45,a,45,1@foo.com,Role 45,45,45,45
    46,b,46,2@foo.com,Role 46,46,46,46
    47,a,47,1@foo.com,Role 47,47,47,47
    48,b,48,2@foo.com,Role 48,48,48,48
    49,a,49,1@foo.com,Role 49,49,49,49
    50,b,50,2@foo.com,Role 50,50,50,50
    51,a,51,1@foo.com,Role 51,51,51,51
    52,b,52,2@foo.com,Role 52,52,52,52
    53,a,53,1@foo.com,Role 53,53,53,53
    54,b,54,2@foo.com,Role 54,54,54,54
    55,a,55,1@foo.com,Role 55,55,55,55
    56,b,56,2@foo.com,Role 56,56,56,56
    57,a,57,1@foo.com,Role 57,57,57,57
    58,b,58,2@foo.com,Role 58,58,58,58
    59,a,59,1@foo.com,Role 59,59,59,59
    60,b,60,2@foo.com,Role 60,60,60,60
    61,a,61,1@foo.com,Role 61,61,61,61
    62,b,62,2@foo.com,Role 62,62,62,62
    63,a,63,1@foo.com,Role 63,63,63,63
    64,b,64,2@foo.com,Role 64,64,64,64
    65,a,65,1@foo.com,Role 65,65,65,65
    66,b,66,2@foo.com,Role 66,66,66,66
    67,a,67,1@foo.com,Role 67,67,67,67
    68,b,68,2@foo.com,Role 68,68,68,68
    69,a,69,1@foo.com,Role 69,69,69,69
    70,b,70,2@foo.com,Role 70,70,70,70
    71,a,71,1@foo.com,Role 71,71,71,71
    72,b,72,2@foo.com,Role 72,72,72,72
    73,a,73,1@foo.com,Role 73,73,73,73
    74,b,74,2@foo.com,Role 74,74,74,74
    75,a,75,1@foo.com,Role 75,75,75,75
    76,b,76,2@foo.com,Role 76,76,76,76
    77,a,77,1@foo.com,Role 77,77,77,77
    78,b,78,2@foo.com,Role 78,78,78,78
    79,a,79,1@foo.com,Role 79,79,79,79
    80,b,80,2@foo.com,Role 80,80,80,80
    81,a,81,1@foo.com,Role 81,81,81,81
    82,b,82,2@foo.com,Role 82,82,82,82
    83,a,83,1@foo.com,Role 83,83,83,83
    84,b,84,2@foo.com,Role 84,84,84,84
    85,a,85,1@foo.com,Role 85,85,85,85
    86,b,86,2@foo.com,Role 86,86,86,86
    87,a,87,1@foo.com,Role 87,87,87,87
    88,b,88,2@foo.com,Role 88,88,88,88
    89,a,89,1@foo.com,Role 89,89,89,89
    90,b,90,2@foo.com,Role 90,90,90,90
    91,a,91,1@foo.com,Role 91,91,91,91
    92,b,92,2@foo.com,Role 92,92,92,92
    93,a,93,1@foo.com,Role 93,93,93,93
    94,b,94,2@foo.com,Role 94,94,94,94
    95,a,95,1@foo.com,Role 95,95,95,95
    96,b,96,2@foo.com,Role 96,96,96,96
    97,a,97,1@foo.com...,Role 97,97,97,97
    98,b,98,2@foo.com,Role 98,98,98,98
    99,a,99,1@foo.com,Role 99,99,99,99
    100,b,100,2@foo.com,Role 100,100,100,100
    Error I am seeing on the console:
    Code:
    === 2012-11-15 15:43:18,664 [l0-3] ERROR IDACall - Top-level servlet error: 
    java.lang.IllegalStateException: Form too large211835>200000
    	at org.mortbay.jetty.Request.extractParameters(Request.java:1404)
    	at org.mortbay.jetty.Request.getParameter(Request.java:749)
    	at com.isomorphic.rpc.RPCManager.initLog(RPCManager.java:339)
    	at com.isomorphic.rpc.RPCManager.<init>(RPCManager.java:294)
    	at com.isomorphic.rpc.RPCManager.<init>(RPCManager.java:281)
    	at com.isomorphic.servlet.IDACall.processRequest(IDACall.java:116)
    	at com.isomorphic.servlet.IDACall.doPost(IDACall.java:73)
    	at javax.servlet.http.HttpServlet.service(HttpServlet.java:637)
    	at com.isomorphic.servlet.BaseServlet.service(BaseServlet.java:152)
    	at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
    	at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:487)
    	at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1097)
    	at com.isomorphic.servlet.CompressionFilter.doFilter(CompressionFilter.java:259)
    	at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1088)
    	at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:360)
    	at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
    	at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181)
    	at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:729)
    	at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:405)
    	at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
    	at org.mortbay.jetty.handler.RequestLogHandler.handle(RequestLogHandler.java:49)
    	at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
    	at org.mortbay.jetty.Server.handle(Server.java:324)
    	at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:505)
    	at org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java:843)
    	at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:647)
    	at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:205)
    	at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:380)
    	at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:395)
    	at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:488)

    #2
    As already covered over email:

    It looks like you are trying to do a multi-record update by simulating the end user editing 1000s of records. This would be extremely inefficient if it works at all - instead, just do a multi-record update directly.

    You can do an "update" DSRequest that affects multiple records by setting allowMultiUpdate:true.

    Comment


      #3
      Dear support:

      Just to understand the response correctly, are you stating that there is a performance limitation with the "mass update" framework? If so, what is the recommended maximum amount of records which can be pended prior to calling saveAllEdits().

      I have another issue that I have not posted yet where at 2500 records, I am getting script hanging warnings from Internet Explorer and it seems that the framework is looping around trying to process the saveAllEdits().

      My code is following the mass update example in the showcase and I wasn't expecting any issues. Is there any examples of an update operation available which involves the use of "allowMultiUpdate".

      Finally, why isn't the "mass update" framework doing a multi-record update on a saveAllEdit() in the first place? Is there a way to have the framework do this instead of refactoring my code to manually do a updateData() call from the listgrid?

      Thanks, T.

      Comment


        #4
        The maximum number of records that will work will differ by browser. As you can see you are already hitting the limits for IE, which are generally the lowest.

        Again, the editValues system is for saving user edits. The "Mass Update" same shows appropriate usage: allowing an end user to modify several records.

        Users do not edit 2500 individual records and then save all at once. What you are really doing is programmatic changes to a whole set of records, so you should represent this operation as a single update that affects multiple records (via allowMultiUpdate).

        No, there is not currently a dedicated sample for allowMultiUpdate, it's a bit too trivial for that: just call setAllowMultiUpdate() and call setCriteria() with whatever criteria you want.

        Comment


          #5
          Dear support:

          The approach that you recommend is not that trivial since my listgrid is databound to my data source.

          If I call updateData from the listgrid, the signature of the method is:

          public void updateData(Record record,
          DSCallback callback,
          DSRequest requestProperties)

          Do I set the record as null and perform setValues() in the DSRequest? Would this even work at all? The updateData method in listgird seems to be tailored to a single record.

          Alternatively, are you suggesting that I bind to my datasource outside of the listgrid and perform the update from there?

          Cheers, T.

          Comment


            #6
            From what you've shared, this update is pretty much unrelated to the grid or to end user editing, it's just a programmatic update. So there doesn't appear to be a reason to go through the grid at all.

            Since this is not a CRUD operation in the usual sense of affecting just one record, the easiest way to do this is probably via DataSource.performCustomOperation() and using the server-side DSRequest.setCriteria() / setAllowMultiUpdate() APIs from a DMI.

            Comment


              #7
              Hi,

              To give a complete picture, we are attempting to implement a bulk live grid editor application. The grid can be loaded with records which are invalid and the end-user is given the ability to fix the errors prior to performing a bulk commit on the record set. One of the useful features we are attempting to implement is an auto-fill column feature which helps the end user correct data in an efficient manner. The auto-fill is only performed on the listgrid and is committed immediately to perminant data store.

              To help understand why the saveAllEdits() is involved, we currently have a listgrid record cache on the server side in-between the client and the permiinant persistant store. The reason why the cache is there is to ensure the end user doesn't lose their edit changes prior to the bulk commit. Since they are dealing with 1000's of record edits prior to the bulk commit, it would be a very bad day for them if the changes are lost due to a browser crash etc. Going with local browser storage is also a concern since to our knowledge of the spec, the storage is not guaranteed to be secure.

              The use case that I am having trouble with looks like this:

              1. Load up the grid with 2500 records
              2. Column A is missing data
              3. Auto-fill the column in the listgrid via range programming techniques
              4. Backup the edit on the server side in the server side cache to ensure that we don't lose the edits for any reason prior to the final commit of all the records.

              The end user can be auto-filling 10's of columns in our application prior to the final commit.

              The backing up of the edit is where I am seeing the performance issues. As you can see the auto-fill operation does have a relation to the grid. Based on what I describe above, do you still recommend that I create a seperate datasource binding outside of the grid to perform the mulit-record update. In addition, since I need the listgrid to be actually updated with the auto-fill data, I would assume the databound listgrid would automatically be updated on the client from the framework. If this is correct, will I see performance issues when the listgrid does the fetch of the updated records (eg. 2500 of them).

              Thanks T.

              Comment


                #8
                do you still recommend that I create a seperate datasource binding outside of the grid to perform the mulit-record update.
                This is an odd way of phrasing it which may indicate that you don't really understand the approach - we would recommend reading the QuickStart Guide about DMIs, and the documentation for the specific APIs we referred to. But, yes, the recommendation remains the same.

                In addition, since I need the listgrid to be actually updated with the auto-fill data, I would assume the databound listgrid would automatically be updated on the client from the framework.
                There's no mechanism that would do this for an update performed with arbitrary criteria. Once the mass update has been committed to the server, you'll need to update the Records using setAttribute (NOTE: *not* setEditValues).

                If this is correct, will I see performance issues when the listgrid does the fetch of the updated records (eg. 2500 of them).
                No reason to expect this, no.

                Comment


                  #9
                  Dear Isomorphic Support:

                  Thanks for the help and the recommended solution worked out.

                  I did have a tough time figuring out how to pass the recordlist of the grid down via the DSRequest or Record parameters from the client side Datasource.performCustomOperation(...) call.

                  There doesn't seem to be any native support for a list of records on the client APIs and when I try to pass the recordList as an attribute value of either the Record or DSRequest, it ends up as empty encoding.

                  I ended up having to encode the listgrid recordlist manually to a JSON string and that got the recordlist from the client to the server.

                  Is there a better way than having to manually serialize the data?

                  Thanks T.

                  Comment


                    #10
                    Yes. Pass the list of primaryKeys as an attribute on the single record, along with the changed fields, since this is common to all records. That's much less data, and far simpler.

                    Comment

                    Working...
                    X