5.1 BatchUploader + uploadFieldName + DynamicDSGenerator problem

[Blama]

Hi Isomorphic,

I have a problem with uploadFieldName and BatchUploader.
I have a user configurable matching of fieldName to uploadFieldName that the user has to maintain according to its CSV-headers.
Now I made my upload-ds dynamic and replace the uploadFieldName with the configured value. This does not work, the new value is not used (but it is replaced, see below) and the field therefore not recognized.

Testcase (using v10.1p_2016-03-27/):
DynamicDSGenerator
BuiltInDS.java:

package com.smartgwt.sample.client;

import com.google.gwt.core.client.EntryPoint;
import com.smartgwt.client.core.KeyIdentifier;
import com.smartgwt.client.data.DataSource;
import com.smartgwt.client.util.Page;
import com.smartgwt.client.util.PageKeyHandler;
import com.smartgwt.client.util.SC;
import com.smartgwt.client.widgets.BatchUploader;
import com.smartgwt.client.widgets.IButton;
import com.smartgwt.client.widgets.Window;
import com.smartgwt.client.widgets.events.ClickEvent;
import com.smartgwt.client.widgets.events.ClickHandler;
import com.smartgwt.client.widgets.form.fields.TextItem;
import com.smartgwt.client.widgets.grid.ListGridField;
import com.smartgwt.client.widgets.layout.VLayout;

public class BuiltInDS implements EntryPoint {
    private VLayout mainLayout;
    private IButton recreateBtn;
    private DataSource uploadDS = DataSource.get("employeesUpload");
    private DataSource optionDS = DataSource.get("employees");

    public void onModuleLoad() {
        KeyIdentifier debugKey = new KeyIdentifier();
        debugKey.setCtrlKey(true);
        debugKey.setKeyName("D");

        Page.registerKey(debugKey, new PageKeyHandler() {
            public void execute(String keyName) {
                SC.showConsole();
            }
        });

        mainLayout = new VLayout(20);
        mainLayout.setWidth100();
        mainLayout.setHeight100();

        recreateBtn = new IButton("Recreate");
        recreateBtn.addClickHandler(new ClickHandler() {
            @Override
            public void onClick(ClickEvent event) {
                recreate();
            }
        });
        mainLayout.addMember(recreateBtn);
        recreate();
        mainLayout.draw();
    }

    private void recreate() {
        Window w = new Window();
        w.setWidth("95%");
        w.setHeight("95%");
        w.setMembersMargin(0);
        w.setModalMaskOpacity(70);
        w.setTitle("Import data");
        w.setShowMinimizeButton(false);
        w.setIsModal(true);
        w.setShowModalMask(true);
        w.centerInPage();

        BatchUploader batchUploader = new BatchUploader();
        batchUploader.setWidth100();
        batchUploader.setUploadDataSource(uploadDS);

        TextItem fieldNameToUse = new TextItem("FIELDNAMETOUSE", "Name of the Manager column in the CSV");
        batchUploader.setUploadFormFields(fieldNameToUse);

        ListGridField employeeIdLGF = new ListGridField("EmployeeId");

        ListGridField nameLGF = new ListGridField("Name");

        ListGridField managerLGF = new ListGridField("ReportsTo");
        managerLGF.setOptionDataSource(optionDS);
        managerLGF.setValueField("EmployeeId");
        managerLGF.setDisplayField("Name");

        ListGridField jobLGF = new ListGridField("Job");

        ListGridField genderLGF = new ListGridField("Gender");

        batchUploader.setGridFields(employeeIdLGF, genderLGF, nameLGF, managerLGF, jobLGF);

        w.addItem(batchUploader);
        w.show();
    }
}

employeesUpload.ds.xml:

<DataSource ID="employeesUpload" serverType="sql" tableName="employeeTable" recordName="employee" useAnsiJoins="true">
    <fields>
        <field name="EmployeeId" uploadFieldName="EmployeeId" title="Employee ID" type="integer" primaryKey="true" required="true" />

        <field name="Name" uploadFieldName="Name" title="Name" type="text" length="128" />

        <!-- Removed type="integer" in ReportsTo, as this triggers another error -->
        <field name="ReportsTo" uploadFieldName="ThisIsTheManager" displayField="employeeName" title="Manager" required="true" importStrategy="display"
            foreignKey="employees.EmployeeId" relatedTableAlias="relatedEmployees">
        <!--     <validators>
                <validator type="hasRelatedRecord" errorMessage="Unknown ReportsTo" />
            </validators>  -->
        </field>
        <field name="employeeName" includeFrom="employees.Name" />

        <field name="Job" uploadFieldName="Job" title="Title" type="text" length="128">
            <valueMap>
                <value>Developer</value>
                <value>IT-Infrastructure</value>
            </valueMap>
        </field>

        <field name="Gender" uploadFieldName="Gender" title="Gender" type="text" length="7">
            <valueMap>
                <value>male</value>
                <value>female</value>
            </valueMap>
        </field>
    </fields>
</DataSource>

RegisterDS.java:

package com.smartgwt.sample.server.listener;

import java.io.StringWriter;
import java.io.Writer;

import javax.servlet.ServletContext;
import javax.servlet.ServletException;
import javax.servlet.http.HttpServlet;
import javax.xml.parsers.DocumentBuilder;
import javax.xml.parsers.DocumentBuilderFactory;
import javax.xml.transform.OutputKeys;
import javax.xml.transform.Transformer;
import javax.xml.transform.TransformerFactory;
import javax.xml.transform.dom.DOMSource;
import javax.xml.transform.stream.StreamResult;
import javax.xml.xpath.XPath;
import javax.xml.xpath.XPathConstants;
import javax.xml.xpath.XPathExpressionException;
import javax.xml.xpath.XPathFactory;

import org.w3c.dom.Document;
import org.w3c.dom.Element;
import org.w3c.dom.Node;
import org.w3c.dom.NodeList;

import com.isomorphic.datasource.DSRequest;
import com.isomorphic.datasource.DataSource;
import com.isomorphic.datasource.DynamicDSGenerator;
import com.isomorphic.log.Logger;

public class RegisterDS extends HttpServlet {
    private static final long serialVersionUID = 7849924835392267735L;

    @Override
    public void init() throws ServletException {
        final Logger logger = new Logger(RegisterDS.class);
        final XPathFactory factory = XPathFactory.newInstance();
        DynamicDSGenerator employeesUploadGenerator = new DynamicDSGenerator() {
            @Override
            public DataSource getDataSource(String id, DSRequest dsRequest) {
                String dsRequestConfig = (dsRequest == null) ? null : "DSRequest " + dsRequest.getDataSourceName() + "/"
                        + dsRequest.getOperationId();
                logger.debug("Called RegisterDS for DataSource " + id + " (" + (dsRequestConfig == null ? "dsRequest is null" : dsRequestConfig)
                        + ")");
                DataSource ds = null;
                try {
                    DocumentBuilder builder = DocumentBuilderFactory.newInstance().newDocumentBuilder();
                    ServletContext servletContext = getServletContext();
                    String strAbsolutePath = servletContext.getRealPath("ds");
                    Document doc = builder.parse(strAbsolutePath + "/employeesUpload.ds.xml");

                    // If no dsRequest (e.g. for DataSourceLoader call at start) exit return just the base XML
                    if (dsRequest == null)
                        return DataSource.fromXML(doc);

                    String fieldNameToUse = dsRequest.getValues().get("FIELDNAMETOUSE") == null ? null : dsRequest.getValues().get("FIELDNAMETOUSE")
                            .toString();
                    Node field = getNode(doc, factory, "/DataSource/fields/field[@name='" + "ReportsTo" + "']");
                    Element e = (Element) field;
                    if (fieldNameToUse != null)
                        e.setAttribute("uploadFieldName", fieldNameToUse);
                    ds = DataSource.fromXML(doc);
                    logger.debug(prettyPrint(doc));
                } catch (Exception ex) {
                    ex.printStackTrace();
                }
                return ds;
            }
        };
        DataSource.addDynamicDSGenerator(employeesUploadGenerator, "employeesUpload");
    }

    public static String prettyPrint(Document xml) throws Exception {
        Transformer tf = TransformerFactory.newInstance().newTransformer();
        tf.setOutputProperty(OutputKeys.ENCODING, "UTF-8");
        tf.setOutputProperty(OutputKeys.INDENT, "yes");
        Writer out = new StringWriter();
        tf.transform(new DOMSource(xml), new StreamResult(out));
        return out.toString();
    }

    private static Node getNode(Document doc, XPathFactory factory, String selector) {
        XPath xpath = factory.newXPath();
        NodeList nodes;
        try {
            nodes = (NodeList) xpath.evaluate(selector, doc.getDocumentElement(), XPathConstants.NODESET);
            if (nodes.getLength() > 0)
                return nodes.item(0);
        } catch (XPathExpressionException e) {
            e.printStackTrace();
        }
        return null;
    }
}

web.xml addition:

    <servlet>
        <servlet-name>RegisterDS</servlet-name>
        <servlet-class>com.smartgwt.sample.server.listener.RegisterDS</servlet-class>
        <load-on-startup>1</load-on-startup>
    </servlet>

To reproduce follow these steps:

1. Upload "TestdataImportEmployees.csv" (attached)
2. See that it always works, no matter what you enter in the TextItem
3. Upload "TestdataImportEmployees_Renamed.csv" (attached)
4. See that it never works, no matter what you enter in the TextItem (also not for FOOBAR, the column name in this csv file)

From the result of the DynamicDSGenerator (see Eclipse debug mode screenshot below) I can see that it does what it should do and that the uploadFieldName is indeed exchanged.

This value is just not used in further processing.


Perhaps this is in the same area as your fix here.
This is an important issue for me.

Best regards
Blama

[Isomorphic]

Because DataSources are pooled and reused, you can't return two different DataSource definitions for the same DataSource ID unless you restart the server. Random, undefined behavior will result if you do return different definitions.

[Blama]

Hi Isomorphic,

thanks for the fast answer, I did not know that. Could you suggest the best way to solve this use case?
In my application I don't have the TextItem, but a SelectItem that returns an ID. With that ID I do a fetch in the DSGenerator and reconfigure the uploadFieldsNames.
Should I append the ID or a random String to the DS and register the normal datasource-name as prefix with DSGenerator?

I do think though I'll need the DS with the correct name on the client so that the client side BatchUploader can use the field information it gets from DataSourceLoader at startup in order to display the grid.

But the import-configuration-to-use (will only change the uploadFieldName) is only known right before the "Upload" click that transfers the CSV file, similar to the TextItem in the testcase.

Best regards
Blama

[Isomorphic]

Yes, you've got the general idea - append a random or user-specific suffix and use that same, dynamically generated ID both client and server.

[Blama]

Ok, will do so.
And in the ChangedHandler of the SelectItem I'll do a DataSource.load() everytime, correct? Even if the client-relevant part of the DS will not change, I do need to set the DS of the BatchUploader to the normal .ds.xml-fuel name prefix + the current ID to use, don't I?

Thank you & Best regards,
Blama

[Isomorphic]

Yes, whenever the uploadFieldName changes you will need to provide a fetch DataSource to your BatchUploader, so it's using the right DataSource ID when making requests to the server.

[Blama]

Hi Isomorphic,

thank you very much, it is working now! I construct my DataSource name with prefix + profileId + timestamp of profile-change.
Now it's clear why DataSource.addDynamicDSGenerator() has prefix- and regexp-signatures.

The fact "two different DataSource definitions for the same DataSource ID" was not clear to me reading the relevant docs
(docs.ServerDataSourceImplementation, DataSource.addDynamicDSGenerator(), DynamicDSGenerator).

The very last point in the 1st document would let one think that this could be the case if one pooled also dynamic DataSources.
The docs state that this is not enabled by default, so a hint on this, presumably in the DynamicDSGenerator-docs, would really help.

Please also note that the design for the docs for addDynamicDSGenerator() is broken in 5.1/6.0.

Best regards
Blama

[Blama]

Hi Isomorphic,

I still think that these docs should contain a pointer to "same name means that the generator has to return the same XML".
It is not clear from the current docs and here a different user stumbled upon this.

Best regards
Blama

[Blama]

Hi Isomorphic,

having a look at your post#2 and the docs, it seems there is some mismatch:

Because DataSources are pooled and reused, you can't return two different DataSource definitions for the same DataSource ID unless you restart the server. Random, undefined behavior will result if you do return different definitions.

Docs:

For many applications of DynamicDSGenerator, pooling is inappropriate because the returned DataSource for a given name might be different each time the generator is called. For this reason, pooling of dynamic DataSources is disabled by default.

The name of the pooling setting is mentioned here and it seems to be disabled by default (not included in my server.properties that I modified from an old 4.1p BuiltInDS).

Best regards
Blama

[Isomorphic]

To clarify this, docs are correct. What we were saying in post#2 is that if pooling for dynamic datasources is enabled, then you must return same XML for same ID. Alternatively, if you disable pooling for dynamic datasources, then generators are allowed to return different definitions.

We've added a note on this in serverDataSourceImplementation docs (will be there in the next nightly).

[Blama]

Hi Isomorphic,

thanks for clarifying and adding to the docs.

Best regards
Blama