Announcement

Collapse
No announcement yet.
X
  • Filter
  • Time
Clear All
new posts

    DSResponse serialization is taking longer time

    I am using Smart
    GWT version -> v9.0p_2013-07-29/PowerEdition Deployment (built 2013-07-29)
    While analyzing the various IDA calls fired within the application , we came across the call timings captured within “isc_RPCResponseStart”.

    Table below show the time taken (in secs) at different steps of IDA request processing returning ~147K records.

    Code:
    -----------------------------------------------------------
    Processing Step Name	           Time Taken (in Secs)
    -----------------------------------------------------------
    "Server processing"                      44
    "RPCManager processing"	                 9 
    "query" 	                         1.5
    "SQLTransform"	                         7.5
    "DSResponse serialization"	         35
    Total time taken to process the IDA call was around 44 Sec out of which almost 80% (35Secs) is consumed by DSResponse Serialization process to transform147K of records. But if we run the same query for 10K record this process take less than 1 sec.

    We need your help to identify the way to minimize this cost.

    Thanks a lot.
    Last edited by mchoudhary; 25 Jul 2014, 01:08. Reason: table formatting

    #2
    Troubleshooting this kind of issue would require a very large amount of information to be gathered from you, but you have again forgotten to include even your version..

    What we'd recommend is to make use of our Consulting services to determine where time is being spent and what application changes would speed up the process.

    Comment


      #3
      Sorry, I have updated my post with version information as well.

      I am more concerned about the step "DSResponse serialization" that is taking too much time. Please can you explain me what are the processes included in this step? What it does internally?

      Does it convert "JDBC ResultSet" object into "Isomorphic DSResponse" object?

      Thanks again.
      Last edited by mchoudhary; 25 Jul 2014, 00:45.

      Comment


        #4
        Yes, when SQLDataSource is used, serialization does involve, as one step, extracting data from the JDBC ResultSet to populate the DSResponse.

        However again there are many, many factors involved - if you want help tuning your application in this case, this should be done via Consulting, not Support.

        Comment


          #5
          Thanks for your reply.

          I want to know one more thing.

          As I mentioned before:


          Total time taken to process the IDA call was around 44 Sec out of which almost 80% (35Secs) is consumed by DSResponse Serialization process to transform 147K of records.

          But if we run the same query for 10K record this process take less than 1 sec.
          Is it expected time taken by different steps of IDA request processing for that much amount of data?

          I want to process the data within 3-4 seconds. I think I have to reduce the number of records.
          Is there any metrics defined for this IDA request processing steps?

          Thanks again.

          Comment


            #6
            There is no way to provide a "typical" amount of time for serializing 147k records. "DSResponse serialization" is a step which calls APIs provided by your JDBC driver, other libraries in the environment, and potentially your application code. Among other things, this step involves data being streamed from your database to the JDBC driver, so it's performance is affected by the database and network as well.

            So there is no way to comment on what's taking time without doing some actual performance analysis work.

            This is what we keep trying to emphasize - you need someone with some experience with performance analysis to take quite a bit of time looking into this before you will even understand what portion of the time is actually related to SmartGWT. We can provide that analysis service, but we would provide it as a Consulting service.

            Finally, if you are actually delivering this 147k of records to a browser, that's probably not going to work or be very slow. In general, use the load on demand features that are built into SmartGWT to deliver only the needed data to the browser.

            Comment


              #7
              Basically I am drawing 147k+ clusters on Google Map and fetching only specific columns such as CENTROID_X, CENTROID_Y, NAME, ID to display the clusters.

              For better user experience, I am fetching records in chunks(batch) rather than fetching in one request.

              I want your assistance to know, whether am I doing wrong in this approach or not.

              There are 2 approach to achieve it:

              1. Using DSRequest.setStartRow() and DSRequest.setEndRow() to fetch records in pagination way. I have created 147 requests in the batch of 1000 records per request and fired all simultaneously without using RPCManager.startQuery()

              2. Using Oracle's ROWNUM value and making custom operation binding request.

              Code:
                 select * from (select ROWNUM rnum, * from myTable where rnum <= endRow) from temp where rum >= startRow
              Please can you tell me which is the best approach for multiple request in this way? Should I prefer default mechanism.
              Last edited by mchoudhary; 13 Aug 2014, 00:14.

              Comment


                #8
                First, you need to start over and do some basic analysis about the data you are trying to display: displaying ~147k clusters requires nearly 400x400 pixels if every cluster occupies a different pixel. That's an unusable data density from a UI perspective even on a very large screen. So you almost certainly need to start over and think about how this data can actually be utilized by end users.

                Second, as we've tried several times to explain: how well different approaches will perform depends upon dozens of factors of your environment which would require in-depth analysis and there is no way to comment on which approach is better without doing this kind of in-depth analysis.

                If you need this kind of service, we do offer it under Consulting.

                Please do not ask any yet again for such services to be provided as part of Support. Support is here to address product flaws. The fact that you are unsure how to approach this from either a UI or optimization perspective is not a product flaw, and should not be submitted as a Support request.

                Comment


                  #9
                  Hi,

                  Good morning,

                  We need your inputs in below analysis.

                  Issue

                  The main issue we are seeing is that once the data is retrieved from the DB, the DSResponse handler takes a lot of time to return the response object back to the client. This can be seen in the timings returned back as well as through profiling the server side.

                  Demo Application

                  The demo application is simple smart client application with a ListGrid on the client and mysql database at the backend. The application uses a ds.xml file to populate the list grid. To be more realistic to the issue we are seeing in live, all the server side data is return to the client by setDataFetchMode(FetchMode.LOCAL) so that scrolling happens at client side rather than requests back to client side. This is representative to UDT’s implementation where all data is returned to client side for seamless processing of data at client side.

                  The source code has a class DataFiller for populating the DB with representative data. The investigation was carried out with 40K and 80K records. In the investigation the MySQL DB and application both resided on the same machine so there is no network latency.

                  Path to dummy application: https://github.com/schyzad/SmartClientDemo.git

                  Investigation

                  The investigation shows that when using the ds.xml approach to request for data from a datasource, a lot of processing time is spent within the Isomorphic API in preparing the response objects for the client side.

                  The attached image shows that the timing taken on a run with 80K result being processed and delivered to client sided. It can be seen out of 61 sec of processing timing, 32 are taken by RPCManager.send which in turn calls JSTranslater.toJS (all isomorphic functions). 29 seconds are taken by handleDSRequest where 14 seconds are taken by java.sql.Statement.executeQuery and 14 seconds are taken by SQLTransform.toListOfMapsOrBeans (isomorphic function)

                  From the above analysis it can be seen that out of 61 seconds of total server side processing, 46 seconds are taken by operations performed by isomorphic API.

                  Given the above, we believe this is not a consulting task as we can demonstrate the behaviour in a trivial application as well. Let me know if you need more information. Happy to setup a call on this if required.


                  Please find the attachments as well. Due to size constraint here I can't upload smartgwt jprofiler snapshot files.

                  Awaiting for your response...

                  Thanks
                  Attached Files
                  Last edited by mchoudhary; 9 Sep 2014, 22:34. Reason: Specified the path to dummy Application

                  Comment


                    #10
                    The Isomorphic APIs where you think time is being spent are calling other APIs, namely, the JDBC driver, which in turn makes network requests.

                    To put it simply: you do not yet have the skills to perform performance analysis, or to interpret these profiler results, and so you are wasting your time and ours by repeatedly trying to file Support tickets that do not come anywhere close to demonstrating a product issue.

                    Isomorphic can help you to understand how to do performance analysis, via our Consulting program.

                    Comment

                    Working...
                    X