Announcement

**Isomorphic** · 28 Jan 2019, 12:56

Basically it looks like whatever data your DMI is returning is in some very unexpected format, so what data are you returning? You mention you do the export to a file, so are you returning something like just a string?

Also, how are you triggering this on the client-side? Are you calling something like exportData() or exportClientData()?

**finance@opengroup.org** · 31 Jan 2019, 01:56

Many thanks for your response.

My data source is,

<DataSource
ID="OGSYSIndividualItemDMI"
serverType="sql"
tableName="dp_ogsys_individual">
<fields>
<field name="user_id" type="text" title="User Id" length="70"/>
<field name="user_name" type="text" title="User Name" length="70"/>
<field name="salesforce_id" type="text" title="Salesforce Id" length="70"/>
<field name="individual_valid" type="text" title="Individual Valid" length="60"/>
<field name="individual_invalid_reason" type="text" title="Individual Status" length="60"/>
<field name="static_user" type="text" title="Static User" length="80"/>
<field name="email_unsubscribed" type="text" title="Email Opt Out" length="80"/>
<field name="salutation" type="text" title="Salutation" length="70"/>
<field name="first_name" type="text" title="First Name" length="70"/>
<field name="middle_name" type="text" title="Middle Name" length="70"/>
<field name="last_name" type="text" title="Last Name" length="70"/>
<field name="gender" type="text" title="Gender" length="40"/>
<field name="job_title" type="text" title="Job Title" length="60"/>
<field name="other_job_title" type="text" title="Other Job Title" length="60"/>
<field name="primary_organization_name" type="text" title="Primary Organization" length="60"/>
<field name="associated_organizations" type="text" title="Associated Organizations" length="60"/>
<field name="industry_sectors" type="text" title="Industry Sector" length="70"/>
<field name="primary_membership_level" type="text" title="Primary Memberhsip" length="50"/>
<field name="allowed_forums" type="text" title="Allowed Forums" length="50"/>
<field name="associated_membership_levels" type="text" title="Associated Membership Levels" length="60"/>
<field name="department" type="text" title="Department" length="60"/>
<field name="employer" type="text" title="Employer" length="60"/>
<field name="fortune500" type="text" title="Fortune 500" length="20"/>
<field name="preferred_address" type="text" title="Preferred Address" length="200"/>
<field name="address1" type="text" title="Address 1" length="60"/>
<field name="address2" type="text" title="Address 2" length="60"/>
<field name="city" type="text" title="City" length="60"/>
<field name="state" type="text" title="State" length="60"/>
<field name="postal_code" type="text" title="Postal Code" length="60"/>
<field name="country" type="text" title="Country" length="60"/>
<field name="country" type="text" title="Country" length="60"/>
<field name="primary_sales_region" type="text" title="Primary Region" length="60"/>
<field name="secondary_sales_region" type="text" title="Secondary Region" length="60"/>
<field name="geographical_regions" type="text" title="Geographical Regions" length="60"/>
<field name="fax" type="text" title="Fax" length="60"/>
<field name="phone" type="text" title="Phone" length="60"/>
<field name="mobile" type="text" title="Mobile" length="60"/>
<field name="preferred_email" type="text" title="Preferred Email" length="60"/>
<field name="email" type="text" title="Email" length="60"/>
<field name="email_verified" type="text" title="Email Verified" length="60"/>
<field name="email_last_verified_date" type="date" title="Email Last Verified Date" length="60"/>
<field name="email_valid" type="text" title="Email Valid" length="60"/>
<field name="email_invalid_reason" type="text" title="Email Invalid Reason" length="60"/>
<field name="additional_email" type="text" title="Additional Email" length="60"/>
<field name="additional_email_valid" type="text" title="Additional Email Valid" length="60"/>
<field name="additional_email_invalid_reason" type="text" title="Additional Email Invalid Reason" length="60"/>
<field name="notes" type="text" title="Notes" length="60"/>
<field name="created_stamp" type="datetime" title="Individual Created" length="60"/>
<field name="pub_download_titles" type="text" title="Publications Titles Downloaded" length="60"/>
<field name="pub_download_docnos" type="text" title="Publication References Downloaded" length="60"/>
<field name="events" type="text" title="Events Attended" length="60"/>
<field name="event_sessions" type="text" title="Events Sessions Attended" length="200"/>
<field name="certifications" type="text" title="Certifications" length="60"/>
<field name="mailing_lists" type="text" title="Mailing Lists" length="60"/>
<field name="roles" type="text" title="Roles" length="60"/>
<field name="area_of_expertise" type="text" title="Area of Expertise" length="200"/>
<field name="hobbies_and_interests" type="text" title="Hobbies and Interests" length="200"/>
<field name="social_networks" type="text" title="Social Networks" length="200"/>
<field name="sector_of_interests" type="text" title="Sector of Interest" length="100"/>
<field name="third_party_certifications" type="text" title="3rd Party Certifications" length="200"/>
<field name="how_did_you_hear" type="text" title="How did You Hear" length="100"/>
<field name="birth_date" type="date" title="Birth Date" length="60"/>
<field name="marital_status" type="text" title="Marital Status" length="60"/>
<field name="gender" type="text" title="Gender" length="60"/>
<field name="updated_stamp" type="datetime" title="Last Updated" length="60"/>
<field name="plato_site_access" type="text" title="Plato Site Access" length="200"/>
<field name="plato_downloads" type="text" title="Plato Downloads" length="200"/>
<field name="migrate_to_salesforce" type="text" title="Migrate To Salesforce" length="20"/>
</fields>

<operationBindings>
<operationBinding operationType="fetch" operationId="customEmailExport" >
<serverObject className="org.opengroup.gwt.dsat.server.export.CustomEmailExportDMI" lookupStyle="new" />
<serverMethod>customExport</serverMethod>
</operationBinding>
</operationBindings>

<operationBindings>
<operationBinding operationType="fetch" operationId="customServerSideExport" >
<serverObject className="org.opengroup.gwt.dsat.server.export.CustomServerSideExportDMI" lookupStyle="new" />
<serverMethod>customExport</serverMethod>
</operationBinding>
</operationBindings>

</DataSource>

So mainly text, with a couple of dates.

On the client side it is being called by,

SC.say("<p>There are over 1000 records. Therefore this download is being exported to the server</p>" + "<p>You will receive an email with a link to the download once it has finished. Depending on the size of the export, this may take some time.</p>");
DSRequest dsRequestProperties = new DSRequest();
dsRequestProperties.setOperationId("customServerSideExport");
dsRequestProperties.setExportDisplay(ExportDisplay.DOWNLOAD);
grid.exportData(dsRequestProperties);

Then the CustomServerSideExportDMI starts a new asynchronous thread

BigExportThread bet = new BigExportThread(dsRequest, servletRequest);
bet.start();

And the BigExportThread then batches the result and writes the data to a file,

boolean finished = false;
int startRow = 0;
int endRow = 10000;
int rowcount = 1;

while (!finished)
{
this.dsRequest.setEndRow(endRow);
DSResponse response = this.dsRequest.execute();
...
...
...

Its this 'dsRequest.execute' that is causing the null pointer as above!

Many thanks

Andy.

**Isomorphic** · 31 Jan 2019, 03:48

OK, if you are starting a thread, then you must be synchronously returning something - so what is it? That's what we're asking about - that seems to be the unexpected response.

From the stack traces above, it actually looks like both threads (the original exportData() call from a browser, and the thread you launched) crash for different reasons. The original thread started from a browser request seems to be crashing from an unexpected response (and we're still waiting to hear what you're providing as a response).

The other thread appears to be crashing in code related to gathering timing data - have you tried turning this off (see Developer Console documentation)?

**finance@opengroup.org** · 31 Jan 2019, 04:18

Hi, yes, I return an empty DSResponse. This results in an empty file being downloaded, but thats not a problem for us. The user knows that they need to wait for the email with the link to the actual download,

public class CustomServerSideExportDMI
{
public static DSResponse customExport(DSRequest dsRequest,
HttpServletRequest servletRequest)
throws Exception
{
BigExportThread bet = new BigExportThread(dsRequest,servletRequest);
bet.start();
DSResponse response = new DSResponse();
response.setSuccess();
return response;
}

}

So that returns an empty DSResponse, but the original request is passed to the 'BigExportThread', which is the class where I am calling 'dsRequest.execute()', and is returning the null pointer.

Many thanks.

Andy.

**Isomorphic** · 31 Jan 2019, 23:38

This is a funky way to do what you want - if you instead used an ordinary fetch, you wouldn�t have to have users ignore an empty file. Also, forwarding the export dsRequest into a new thread is causing a crash because exports - which are intended to return results immediately to the browser - don�t make sense on a non-browser-request thread.

So we�re looking at avoiding the crash due to the empty response, but you can get around that and also provide a better UE if you just switch things so you�re not using exportData() client-side.

**finance@opengroup.org** · 1 Feb 2019, 01:57

Many thanks for the reply,

The idea of doing this in a new thread was because the export can take quite a wile to perform (maybe 5 minutes), so the user cant be waiting on the browser to return the results. So this needs to be an async method really.

If I used 'an ordinary fetch', where would this be from? You mean don't use a custom export at all, and use the built in download without setting the custom OperationId? I have tried that, and it just takes too long to return, and freezes or crashes the browser.

Or do you mean do it in a different way to that.

Many thanks for you help.

Andy.

**Isomorphic** · 1 Feb 2019, 09:27

It�s perfectly normal to have an export that can�t complete synchronously and must be accomplished by a later download, but when you call exportData(), you are telling the framework that you will be doing an immediate download, but then you are not actually doing that. You can switch to an ordinary fetch to avoid this problem.

There is no need to actually return data from the fetch either, but in this case, you will not have set up both the client and server side of the framework to perform a (useless) file download, as well as set an expectation that you will be returning data that can be turned into a downloadable file, which is what leads to a crash when you in fact return nothing.

**finance@opengroup.org** · 7 Feb 2019, 07:10

Many thanks. I have tried a few things so as to not use exportData, but I'm not having much luck I'm afraid. And I can't find any examples that do something similar.

I've tried with an AsyncCallback actioned from my 'export' button, but I can't get a serialised version of the grid, or its data to pass back to the server, unless I'm missing something.

If I use a direct fetch, then how do I get that data to the server side to process? If you have any examples, or bare bones code then that would be great.

Many thanks.

Andy.

**Isomorphic** · 7 Feb 2019, 13:08

All that exportData() does is send criteria to the server that match the grid, so fetchData() does the same thing.

Are you actually using exportClientData()? We asked about this before, but you never responded.

**finance@opengroup.org** · 8 Feb 2019, 07:45

Great thanks. In ExportData I can set the DSRequest with the 'OperationId' so that it knows which class to run on the server side. For fetchData, there isn't a method that takes in just a DSRequest. I've tried the following,

DSRequest dsRequestProperties = new DSRequest();
dsRequestProperties.setOperationId("customServerSideExport");

//grid.exportData(dsRequestProperties);
grid.fetchData(grid.getCriteria(),
new DSCallback() {
public void execute(DSResponse response,
Object rawData,
DSRequest request)
{
SC.say("Downloading...");
}
},
dsRequestProperties);

But that just seems to refresh the grid client side, and nothing gets back to the server to catch to run my download and save to the filesystem. How do I catch the data on the server side?

As for exportClientData(). No, at the moment I just call - grid.exportData(dsRequestProperties);

I've had a look in the quick start guide, but not found a section on how to get the data server side, so not sure if there is a section in there that I have missed somewhere?

Many thanks

Andy.

**Blama** · 8 Feb 2019, 08:57

Hi finance@opengroup.org,

1st thing you need is to hit a breakpoint in your server code like you already do in your current code.
You need a serverObject (have it already most likely) and an operationBinding like this in your .ds.xml:

Code:

<operationBinding type="fetch" operationId="customServerSideExport" serverMethod="yourOldMethodName">

If thats (basically the criteria) is all information you need serverSide, you are fine now. If you need more information from the client for whatever reason I do this with an additional empty fake field like this, where you can send data to like this for a yes/no. It's also approved by Isomorphic here (the information you were looking for in the QSG).

Code:

<field name="USE_LOOSE_MODE" type="boolean" sqlStorageStrategy="singleCharYN" customSQL="true" customSelectExpression="null" />

In order to tell apart your criteria and your data, I'd use UPDATE instead of FETCH as requestType and send the criteria in criteria and the additional values in values.

Then in the server method just start your additional thread and do a return new DSResponse().setSuccess().

Best regards
Blama

**finance@opengroup.org** · 13 Feb 2019, 08:08

Many thanks,

I now have it coded using 'grid.fetchData()'. However, it still results in the same null pointer exception when calling 'dsRequest.execute();

My datasource has,

<operationBindings>
<operationBinding operationType="fetch" operationId="customServerSideExport">
<serverObject className="org.opengroup.gwt.dsat.server.export.CustomServerSideExportDMI" methodName="customExport"/>
</operationBinding>
</operationBindings>

And my client code is,

grid.setFetchOperation("customServerSideExport");
grid.fetchData();

The CustomServerSideExportDMI class is,

BigExportThread bet = new BigExportThread(dsRequest,servletRequest);
bet.start();
return new DSResponse().setSuccess();

And the BigExportThread is causing the null pointer as before at,

(line 111) DSResponse response = this.dsRequest.execute();

The exception is,

2019-02-13 15:16:33,974 INFO [STDOUT] Export Error : java.lang.NullPointerException
at com.isomorphic.datasource.DSRequest.trackTimings(DSRequest.java:7001)
at com.isomorphic.datasource.DSRequest.recordTimingData(DSRequest.java:7016)
at com.isomorphic.datasource.DSRequest.recordTimingData(DSRequest.java:7013)
at com.isomorphic.datasource.DSRequest.execute(DSRequest.java:2611)
at org.opengroup.gwt.dsat.server.threads.BigExportThread.run(BigExportThread.java:111)

So it looks like its doing the same for fetchData as it is for exportData.

I have also turned off the timing data in the .properties file with,

DSRequest.returnTimingData: false

And I have also disabled the developer console.

So it looks like changing to a fetch did not solve the problem.

If I do the processing NOT in a thread then it works, but of course it freezes the user interface until it completes, which cant happen.

Many thanks

Andy.

**Isomorphic** · 13 Feb 2019, 14:55

Glad you fixed the first part so your users no longer have to download blank files that they ignore :)

The problem now is that lifecycle of the DSRequest object is tied to the servlet request. It's invalid to use a DSRequest created on a servlet thread in another thread - it's resources have already been cleaned up, so basic API calls on it will fail.

So you need to create a new DSRequest() on your separate thread and execute that. See also the Standalone DataSource Usage overview.

**finance@opengroup.org** · 15 Feb 2019, 01:59

Great thanks. What is the best/recomended way to create a clone of the DSRequest? I can't see a clone method, so I tried,

// Clone the request to use in the thread
DSRequest threadRequest = new DSRequest();
threadRequest.setExportFields(dsRequest.getExportFields());
threadRequest.setCriteria(dsRequest.getCriteria());
threadRequest.setDataSource(dsRequest.getDataSource());
threadRequest.setOperationId(dsRequest.getOperationId());

But that doesn't seem to have all that is needed, and I'm getting errors. The latest is,

2019-02-15 09:55:07,176 INFO [STDOUT] Export Error : java.lang.NullPointerException
at com.isomorphic.datasource.DataSource.getOperationBinding(DataSource.java:1364)
at com.isomorphic.datasource.DataSource.getOperationBinding(DataSource.java:1346)
at com.isomorphic.datasource.DataSourceDMI.execute(DataSourceDMI.java:124)
at com.isomorphic.datasource.DataSourceDMI.execute(DataSourceDMI.java:64)
at com.isomorphic.datasource.DSRequest.execute(DSRequest.java:2831)
at org.opengroup.gwt.dsat.server.threads.BigExportThread.run(BigExportThread.java:111)

So if there is a best or recommended way to clone the DSRequest then that would be great.

Many thanks

Andy.

Announcement

Null pointer when calling dsRequest.execute()

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment