SmartGWT 6.0 - DynamicDSGenerator & dsRequest.getHttpServletRequest() == null

[kiipper]

SmartClient Version: v11.0p_2016-04-06/PowerEdition Deployment (built 2016-04-06) - Apache Tomcat/7.0.50

It's a bug, that does not happen on SmartGWT 5.x (PowerEdition)
We use a Custom IDACall:

public void initDynamicDSGenerator() {

DataSource.addDynamicDSGenerator(new DynamicDSGenerator() {
    @Override
    public DataSource getDataSource(String id, DSRequest dsRequest) {
        DataSource ds = null;
        try {
            if (dsRequest != null) {
                HttpSession session =[B] dsRequest.getHttpServletRequest().[/B]getSession();
                String dbUser = (String) session.getAttribute(DB_USER);

                DocumentBuilder builder = DocumentBuilderFactory.newInstance().newDocumentBuilder();
                Document doc = builder.parse(dsRequest.getServletContext().getResourceAsStream("/ds/" + id + ".ds.xml"));
                doc.getDocumentElement().setAttribute("dbName", dbUser + "_db");
                ds = DataSource.fromXML(doc);
            }
        } catch (Exception ex) {
            //ex.printStackTrace();
            Logger.getLogger(getClass().getName()).error(ex.getMessage(), ex);
        }
        return ds;
    }
});

}

When e Update (or Delete) operation is performed, the dsRequest.getHttpServletRequest() returns null.
For Fetch operations it works fine.

Can somebody confirm if it works (or not) in any other SmartGWT version?

[kiipper]

It's working fine on daily build, from 2016-04-27 and beyond.

Edit: Nope, it's not working. We tested on a smargwt 5.x version.

[kiipper]

Isomorphic,
do you guys have any follow up on this?

[Isomorphic]

Try putting together a complete testcase. Make the servlet a separate servlet so we can run it, and try it without any custom servlets on your project, since a very likely cause for this would be incorrect code in overridden servlets. We also need to see how the client request is being sent.

[kiipper]

Fair enough, here it goes.

Client code:

/*
 * Copyright (C) MicroCódigo/Pathfinder Informática- All Rights Reserved
 * Unauthorized copying of this file, via any medium is strictly prohibited
 * Proprietary and confidential
 */
package org.microcodigo.client;

import com.google.gwt.core.client.EntryPoint;
import com.smartgwt.client.core.Function;
import com.smartgwt.client.data.DataSource;
import com.smartgwt.client.widgets.Button;
import com.smartgwt.client.widgets.events.ClickEvent;
import com.smartgwt.client.widgets.events.ClickHandler;
import com.smartgwt.client.widgets.grid.ListGrid;
import com.smartgwt.client.widgets.grid.ListGridRecord;
import com.smartgwt.client.widgets.layout.VLayout;

/**
 * Main entry point.
 *
 * @author kiipper
 */
public class MainEntryPoint implements EntryPoint {

    /**
     * Creates a new instance of MainEntryPoint
     */
    public MainEntryPoint() {
    }

    /**
     * The entry point method, called automatically by loading a module that
     * declares an implementing class as an entry-point
     */
    @Override
    public void onModuleLoad() {
        DataSource.load("table_test", new Function() {

            @Override
            public void execute() {
                final VLayout vLayout = new VLayout();
                final ListGrid lg = new ListGrid(DataSource.get("table_test"));
                final Button addButton = new Button("Add");
                final Button removeButton = new Button("Remove");
                final Button refreshButton = new Button("Refresh");
                lg.setWidth100();
                lg.setHeight100();
                lg.setCanEdit(true);

                addButton.addClickHandler(new ClickHandler() {

                    @Override
                    public void onClick(ClickEvent event) {
                        lg.startEditingNew();
                    }
                });
                removeButton.addClickHandler(new ClickHandler() {

                    @Override
                    public void onClick(ClickEvent event) {
                        ListGridRecord rec = lg.getSelectedRecord();
                        if (rec != null) {
                            lg.removeData(rec);
                        }
                    }
                });
                refreshButton.addClickHandler(new ClickHandler() {

                    @Override
                    public void onClick(ClickEvent event) {
                        lg.invalidateCache();
                        lg.fetchData();
                    }
                });

                vLayout.setWidth100();
                vLayout.setHeight100();
                vLayout.setMembers(lg, addButton, removeButton, refreshButton);
                vLayout.draw();

            }
        });

    }
}

Server Code:

package org.microcodigo.server;

import com.isomorphic.datasource.DSRequest;
import com.isomorphic.datasource.DataSource;
import com.isomorphic.datasource.DynamicDSGenerator;
import com.isomorphic.servlet.IDACall;
import javax.servlet.ServletConfig;
import javax.servlet.ServletException;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpSession;
import javax.xml.parsers.DocumentBuilder;
import javax.xml.parsers.DocumentBuilderFactory;
import org.apache.log4j.Logger;
import org.w3c.dom.Document;

/**
 * @author Kiipper
 */
public class CustomIDACall extends IDACall {

    @Override
    public void init(ServletConfig config) throws ServletException {
        super.init(config);
        initDynamicDSGenerator();
    }

    public void initDynamicDSGenerator() {
        DataSource.addDynamicDSGenerator(new DynamicDSGenerator() {
            @Override
            public DataSource getDataSource(String id, DSRequest dsRequest) {
                DataSource ds = null;
                try {
                    if (dsRequest != null) {

                        HttpServletRequest hsr = dsRequest.getHttpServletRequest();
                        if (hsr == null) {
                            Logger.getLogger(getClass().getName()).error("It is null -> dsRequest.getHttpServletRequest(): " + hsr);
                            //Incoming null pointer exception:
                            HttpSession session = hsr.getSession();
                            String currentUser = hsr.getUserPrincipal().getName();
                        } else {
                            Logger.getLogger(getClass().getName()).error("dsRequest.getHttpServletRequest(): " + hsr);
                        }

                        DocumentBuilder builder = DocumentBuilderFactory.newInstance().newDocumentBuilder();
                        Document doc = builder.parse(dsRequest.getServletContext().getResourceAsStream("/ds/" + id + ".ds.xml"));
                        ds = DataSource.fromXML(doc);
                    }
                } catch (Exception ex) {
                    ex.printStackTrace();
                    Logger.getLogger(getClass().getName()).error(ex.getMessage(), ex);
                }
                return ds;
            }
        });
    }

}

That's my Tomcat web.xml:

<?xml version="1.0" encoding="UTF-8"?>
<web-app version="3.0" xmlns="http://java.sun.com/xml/ns/javaee" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://java.sun.com/xml/ns/javaee http://java.sun.com/xml/ns/javaee/web-app_3_0.xsd">
    <session-config>
        <session-timeout>
            30
        </session-timeout>
    </session-config>
    <welcome-file-list>
        <welcome-file>welcomeGWT.html</welcome-file>
    </welcome-file-list>
    <!--SmartwGET / Isomorphic Server-->
    <jsp-config>
        <!-- Isomorphic JSP tags -->
        <taglib>
            <taglib-uri>isomorphic</taglib-uri>
            <taglib-location>/WEB-INF/iscTaglib.xml</taglib-location>
        </taglib>
    </jsp-config>
    <mime-mapping>
        <extension>manifest</extension>
        <mime-type>text/cache-manifest</mime-type>
    </mime-mapping>
    <!-- RPCManager uses this URL by default for Built-in DataSource operations -->
    <servlet-mapping>
        <servlet-name>IDACall</servlet-name>
        <url-pattern>/org.microcodigo.Main/sc/IDACall/*</url-pattern>
    </servlet-mapping>
    <!-- The IDACall servlet handles all Built-in DataSource operations -->
    <servlet>
        <servlet-name>IDACall</servlet-name>
        <!--<servlet-class>com.isomorphic.servlet.IDACall</servlet-class>-->
        <servlet-class>org.microcodigo.server.CustomIDACall</servlet-class>
    </servlet>
    <!-- The DataSourceLoader servlet returns Javascript representations of the dataSources whose
    ID's are passed to it - it is an alternative to using the <loadDS> JSP tag -->
    <!-- DataSourceLoader requests -->
    <servlet-mapping>
        <servlet-name>DataSourceLoader</servlet-name>
        <url-pattern>/org.microcodigo.Main/sc/DataSourceLoader</url-pattern>
    </servlet-mapping>
    <servlet>
        <servlet-name>DataSourceLoader</servlet-name>
        <servlet-class>com.isomorphic.servlet.DataSourceLoader</servlet-class>
    </servlet>
    <!-- The screenLoader servlet loads screen definitions in javascript notation -->
    <servlet>
        <servlet-name>screenLoader</servlet-name>
        <servlet-class>com.isomorphic.servlet.ScreenLoaderServlet</servlet-class>
    </servlet>
    <!-- The screenLoader servlet loads screen definitions in javascript notation -->
    <servlet-mapping>
        <servlet-name>screenLoader</servlet-name>
        <url-pattern>/org.microcodigo.Main/sc/screenLoader</url-pattern>
    </servlet-mapping>
    <!--INIT Servlet-->
    <servlet>
        <servlet-name>Init</servlet-name>
        <servlet-class>com.isomorphic.base.Init</servlet-class>
        <load-on-startup>1</load-on-startup>
    </servlet>
    <!-- Init: initializes SmartGWT framework.  Note that we invoke the Init logic from 
    both a Listener and a load-on-startup Servlet in this file - see the client-side 
    documentation topic "com.smartgwt.client.docs.ServerInit" for a discussion of this.  -->
    <listener>
        <listener-class>com.isomorphic.base.InitListener</listener-class>
    </listener>
    <!-- The PreCache servlet initializes when the servlet engine starts up and pre-loads 
    data need for all client requests.  This is optional, and improves performance
    of the first few page requests.  PreCache cannot be invoked by a browser, because
    there is no "servlet-mapping" defined for it. -->
    <!-- PreCache is disabled by default because of an issue when debugging
    the project for the first time. However, it can be safely enabled
    when compiling with the GWT compiler prior to deployment. See
    the README.txt for more information. -->
</web-app>

Datasource table_test.ds.xml

<DataSource ID="table_test" serverType="sql" dataSourceVersion="1" useAnsiJoins="true">
    <fields>
        <field name="cod" type="sequence" primaryKey="true" canEdit="false" width="60">
        </field>
        <field name="description" type="text">
        </field>
        
        <!-- 
        CREATE TABLE table_test (
        cod serial NOT NULL PRIMARY KEY,
        description TEXT);
        -->
       
    </fields>
    <allowAdvancedCriteria>true</allowAdvancedCriteria>
</DataSource>

The client request:

{
    dataSource:"table_test", 
    operationType:"remove", 
    componentId:"isc_ListGrid_0", 
    data:{
        cod:5, 
        description:"teste me too"
    }, 
    textMatchStyle:"exact", 
    callback:{
        target:[ListGrid ID:isc_ListGrid_0], 
        methodName:"removeDataComplete"
    }, 
    willHandleError:true, 
    showPrompt:true, 
    oldValues:{
        cod:5, 
        description:"teste me too"
    }, 
    requestId:"table_test$6272", 
    internalClientContext:{
        removeDataCallback:{
        }, 
        editInfo:{
            rowNum:1, 
            values:Obj, 
            editCompletionEvent:"programmatic"
        }
    }, 
    fallbackToEval:false, 
    lastClientEventThreadCode:"MUP0", 
    bypassCache:true, 
    dataProtocol:"getParams"
}

[kiipper]

We use a similar code (check thread: http://forums.smartclient.com/forum/...multiple-users)
Since version 6.0 we can't make it work.

Isomorphic, do you suggest another approach, or a workaround?

[kiipper]

Isomorphic,
it's been a while since we got some feedback.
I still think that it's a 6.0 SmartGWT server bug, since the same code work with version 5.x.
Do the code above could be used to check for the issue?

[Isomorphic]

We are working on this. Your test case is fine, we clearly see the problem.

Regards
Isomorphic Software

[Isomorphic]

This appears to be a usage issue. The test case above registers a DynamicDSGenerator without a prefix - this means it will be asked for *any DataSource the system needs*. This includes lots of different DataSource lookups related to Component XML and other processing, which would not be expected to provide an HttpServletRequest, or in some cases even a DSRequest.

Your code is crashing on one of these internal lookups - if you log the DataSource ID you are being asked for, you'll see that it's not table_test. If you fix your code so that you just return null for other DataSources, then you'll see that the system ends up asking you for table_test a few steps later, and the HttpServletRequest is available.

Note that you'd have the same problem with this DataSource generator in 5.x as well, but it would have shown up only in certain scenarios. Changes made in 6.0 just mean you run into the problem sooner.

As far as the best way to do specifically what you're trying to do (customize all application-defined DataSources by user), you could either:

1. obtain a list of all your DataSources so you can quickly check if you're being asked for one of the DataSources you mean to customize, or use a prefix when naming your application DataSources, so you can either register your generator only for that prefix

OR

2. return null any time there's no HttpServletRequest or DSRequest available, as this obviously isn't a DataSource you mean to customize. Also, cleanly return null (rather than crashing) if you don't find the DS under the project.dataSources directory.

[Isomorphic]

Another question on this - while these lookups are normal, it doesn't appear that they happen unless you've got abnormal settings for "project.dataSources" in server.properties.

What setting are you using for this? We're looking specifically for "ds://fileSourceDataSources" being in the path, which should not be there for a normal application.

[kiipper]

Isomorphic,
the idea of return null when there's no HttpServlet is a good one:

2. return null any time there's no HttpServletRequest or DSRequest available, as this obviously isn't a DataSource you mean to customize. Also, cleanly return null (rather than crashing) if you don't find the DS under the project.dataSources directory.

I understood that the order of the 'lookups' are not the same between 5.x and 6.0, yet the problem persists.
The HttpServletRequest is null when the DataSource Id is 'test_table'.
We changed the DynamicDSGenerator so the bug could be better identified:

public void initDynamicDSGenerator() {
        DataSource.addDynamicDSGenerator(new DynamicDSGenerator() {
            @Override
            public DataSource getDataSource(String id, DSRequest dsRequest) {
                DataSource ds = null;

                Logger.getLogger("DataSourceId: " + id);
                System.out.println("DataSourceId: " + id);

                try {
                    if (dsRequest != null) {

                        HttpServletRequest hsr = dsRequest.getHttpServletRequest();
                        if (hsr == null) {
                            Logger.getLogger(getClass().getName()).error("It is null -> dsRequest.getHttpServletRequest(): " + hsr);
                            System.out.println("It is null -> dsRequest.getHttpServletRequest(): " + hsr);
                            
                            //Incoming null pointer exception:
                            HttpSession session = hsr.getSession();
                            String currentUser = hsr.getUserPrincipal().getName();
                        } else {
                            Logger.getLogger(getClass().getName()).error("dsRequest.getHttpServletRequest(): " + hsr);
                        }

                        DocumentBuilder builder = DocumentBuilderFactory.newInstance().newDocumentBuilder();
                        Document doc = builder.parse(dsRequest.getServletContext().getResourceAsStream("/ds/" + id + ".ds.xml"));
                        ds = DataSource.fromXML(doc);
                    }
                } catch (Exception ex) {
                    ex.printStackTrace();
                    Logger.getLogger(getClass().getName()).error(ex.getMessage(), ex);
                }
                return ds;
            }
        });
    }

After a DELETE performed in the table, the order of the lookups follow like this:

DataSourceId: transaction
DataSourceId: Object
DataSourceId: List
DataSourceId: elem
DataSourceId: table_test
It is null -> dsRequest.getHttpServletRequest(): null

The code already return null for the lookups that don't have a DSRequest, it works just like you said, but when it reaches the 'table_test' a NullPointerException is throw.
Its important to say that the DELETE operation is actually performed in the database and the record is removed as intended.
But, we still need to reach the HttpServletRequest for that operation.

By the way the server.properties used is:

webRoot: __AUTODETECT__
gwtModuleName: org.microcodigo.Main
isomorphicPathRootRelative: $gwtModuleName/sc

sql.defaultDatabase: PostgreSQL
sql.useAnsiJoins: true

project.datasources: $webRoot/ds
project.ui: $webRoot/shared/ui
project.apps: $webRoot/shared/app

RPCManager.enabledBuiltInMethods: *
RPCManager.enableBuiltinRPCs: true

modulesDir: modules/

i18n.resourceBundle.parse.utf-8: true
datasources.pool.enabled: false

sql.defaultDatabase: test
sql.test.driver.databaseName: test
sql.test.driver.user: test
sql.test.driver.password: test
sql.test.driver.portNumber: 5432
sql.test.driver.serverName: 192.168.1.10
sql.test.driver.driverName: postgresql
sql.test.autoJoinTransactions: true
sql.test.driver: org.postgresql.Driver
sql.test.database.type: postgresql
sql.test.interface.credentialsInURL: true
sql.test.driver.networkProtocol: tcp
# sql.test.driver.context: 
sql.test.interface.type: driverManager

Regards

[Isomorphic]

This is fixed now and will be available in nightly builds since June 2 (today). Please let us know how it worked for you.

It would be great if you also tell us in what version exactly this was working, thanks!

[kiipper]

IT WORKS! :)
Thank you.

I can tell, for sure, that those versions are working:
SmartClient Version: v10.0p_2014-11-12/PowerEdition Deployment (built 2014-11-12)
SmartClient Version: v9.1p_2015-06-10/PowerEdition Deployment (built 2015-06-10)

We tried other versions, but I cant remember right now, I'll check if I still have the projects...

[kiipper]

Long time since last bug.
Almost 2 years. :)

I'm using SmartGWT v12.0p_2018-04-05/PowerEdition Deployment (built 2018-04-05)

But our application login is broken, the DynamicDSGenerator.getDataSource(String id, DSRequest dsRequest) is ALWAYS sending null in the dsRequest.

Can you guys confirm this?

[Isomorphic]

No, definitely cannot confirm. Several automated tests and several other users are relying on that parameter, so it is working in general. Please let us know if you can isolate a case where it is not, and that seems to be a framework bug.