Regression with binary field data download?

[Blama]

Hi Isomorphic,

do you know of any recent change in binary field handling (Oracle database)?
What worked for me with 5.1d (SNAPSHOT_v10.1d_2015-11-28) does not work with 5.1d (today's nightly, v10.1p_2016-01-10). I don't know when it stopped working.
This is the error I get in the server log:

=== 2016-01-11 06:22:49,290 [ec-1] DEBUG RPCManager - Request #1 (DSRequest) payload: {
    criteria:{
        ID:1,
        download_fieldname:"DATA",
        download_filename:"removed"
    },
    operationConfig:{
        dataSource:"T_ATTACHMENT",
        repo:null,
        operationType:"viewFile",
        textMatchStyle:"exact"
    },
    appID:"builtinApplication",
    operation:"T_ATTACHMENT_viewFile",
    oldValues:{
        ID:1,
        download_fieldname:"DATA",
        download_filename:"removed"
    }
}
=== 2016-01-11 06:22:49,291 [ec-1] INFO  LMSIDACall - Performing 1 operation(s)
=== 2016-01-11 06:22:49,292 [ec-1] WARN  RPCManager - dsResponse.getData().get("DATA") returned null when we were expecting an InputStream, Byte[] or byte[]. Can't continue.

The Datasource is:

<DataSource xmlns="lmscompany/ds" xmlns:fmt="lmscompany/fmt" dbName="Oracle" tableName="T_ATTACHMENT" ID="T_ATTACHMENT" serverType="sql"
    serverConstructor="com.lmscompany.lms.server.LMSSQLDataSource">
    <fmt:bundle basename="com.lmscompany.lms.server.i18n.DSXMLResources-utf8" encoding="utf-8" />
    <fields>
        <field primaryKey="true" hidden="true" name="ID" type="sequence" />
        <field hidden="true" name="TENANT_ID" type="integer" canEdit="false" />

        <field foreignKey="V_USER_CREATED_BY.ID" relatedTableAlias="USER_CREATED_BY" name="CREATED_BY" displayField="CREATED_BY_NAME" type="creator">
            <title><fmt:message key="addedBy" /></title>
        </field>
        <field name="CREATED_BY_NAME" includeFrom="V_USER_CREATED_BY.FULLNAME_INITIALS_GIVENNAME" length="61" hidden="true" />
        <field name="CREATED_BY_NAME_TOOLTIP" includeFrom="V_USER_CREATED_BY.FULLNAME_LEGALENTITYNAME" length="134" hidden="true" />

        <field name="CREATED_AT" type="creatorTimestamp">
            <title><fmt:message key="addedAt" /></title>
        </field>

        <field foreignKey="V_USER_MODIFIED_BY.ID" relatedTableAlias="USER_MODIFIED_BY" name="MODIFIED_BY" displayField="MODIFIED_BY_NAME" type="modifier">
            <title><fmt:message key="modifiedBy" /></title>
        </field>
        <field name="MODIFIED_BY_NAME" includeFrom="V_USER_MODIFIED_BY.FULLNAME_INITIALS_GIVENNAME" length="61" hidden="true" />
        <field name="MODIFIED_BY_NAME_TOOLTIP" includeFrom="V_USER_MODIFIED_BY.FULLNAME_LEGALENTITYNAME" length="134" hidden="true" />

        <field name="MODIFIED_AT" type="modifierTimestamp">
            <title><fmt:message key="modifiedAt" /></title>
        </field>

        <field name="TITLE" type="text" length="100" required="true">
            <title><fmt:message key="title" /></title>
        </field>
        <field name="LEAD_ID" type="integer" hidden="true" />
        <field name="DETAIL_ID" type="integer" hidden="true" />
        <field name="DATA" type="binary" required="true" maxFileSize="2097152">
            <title><fmt:message key="file" /></title>
        </field>
        <field name="DATA_FILENAME" length="255" type="text" hidden="true" required="true" />
        <field name="DATA_FILESIZE" type="integer" required="true">
            <title><fmt:message key="fileSize" /></title>
        </field>
        <field name="DATA_DATE_CREATED" ignore="true" type="datetime" />

        <!-- TENANT_IDs for WHERE-clause generation -->
        <field name="CREATED_BY_TENANT_ID" includeFrom="V_USER_CREATED_BY.TENANT_ID" hidden="true" />
        <field name="MODIFIED_BY_TENANT_ID" includeFrom="V_USER_MODIFIED_BY.TENANT_ID" hidden="true" />
    </fields>
    <serverObject lookupStyle="new" className="com.lmscompany.lms.server.worker.T_ATTACHMENT" />
    <operationBindings>
        <operationBinding operationType="add" />
        <operationBinding operationType="validate" />
        <operationBinding operationType="remove" />
        <operationBinding operationType="fetch" />

        <operationBinding operationType="viewFile" />
        <operationBinding operationType="downloadFile" />
    </operationBindings>
</DataSource>

Does a simple upload / download work for you using Oracle? Upload is working here.

Thank you & Best regards
Blama

[Isomorphic]

We definitely don't see anything in that timeframe that could affect Oracle download, and while we don't currently have an Oracle-specific automated test for downloads, other download automated tests (against other DBs) are passing. We'll check this out as part of some work we're doing on an unrelated, older regression in the download system (related to server-initiated requests).

[Blama]

Hi Isomorphic,

I found the cause of this. I have this in my IDACall subclasses handleDSRequest():

        // Error if the OperationType is unknown
        if (!new LinkedList<String>(
                Arrays.asList(DataSource.OP_ADD, DataSource.OP_CLIENT_EXPORT, DataSource.OP_CUSTOM, DataSource.OP_FETCH, DataSource.OP_GET_FILE,
                        DataSource.OP_GET_FILE_VERSION, DataSource.OP_HAS_FILE, DataSource.OP_HAS_FILE_VERSION, DataSource.OP_LIST_FILES,
                        DataSource.OP_LIST_FILE_VERSIONS, DataSource.OP_REMOVE, DataSource.OP_REMOVE_FILE, DataSource.OP_REMOVE_FILE_VERSION,
                        DataSource.OP_RENAME_FILE, DataSource.OP_SAVE_FILE, DataSource.OP_UPDATE, DataSource.OP_VALIDATE)).contains(operationType)) {
            return new DSResponse().setDataSource(dsRequest.getDataSource()).setFailure(
                    String.format(I18n.getString("unknownOperationTypeIsNotAllowed", dsRequest.getHttpServletRequest()), dsID));
        }

I got the list of Operations from DataSource field summary. This does miss the request types sent for view/download file, namely viewFile and downloadFile.
They are not in the docs and I don't see them in Eclipse either. Adding them directly as Strings to the list fixes my problem. Could you tell me if you have constants for this?
Or perhaps the whole code block is even not necessary if you do the same check already?

Best regards
Blama

[Blama]

FYI: I thought it was related to Oracle only because of the log entry:

dsResponse.getData().get("DATA") returned null when we were expecting an InputStream, Byte[] or byte[]. Can't continue.

But the reason for this row is that the request was not executed because of the return new DSResponse().setDataSource(dsRequest.getDataSource()).setFailure() and the framework tries to fetch the binary data then from the failure-DSResponse().

[Isomorphic]

The existence of these two operation types is covered in the Binary Fields overview. We'll add constants for them as well.

[Isomorphic]

The constants for the viewFile and downloadFile operations have been added. Please try the next nightly build, dated January 14.

Regards
Isomorphic Software

[Blama]

Hi Isomorphic,

thanks for adding, I can see them now.

Best regards
Blama

[Blama]

Hi Isomorphic,

could you also add loadSchema, which is used by BatchUploader?

Thank you & Best regards
Blama

PS: The request to batchUpload.ds.xml is shown in Developer Console's RPC Tab as type RPCRequest and not DSRequest.
Nevertheless it is handled by my IDACall-subclasses handleDSRequest()-method. Is this correct?

[Isomorphic]

We've added the requested constant. Please try the next nightly build, dated February 17.

Regards
Isomorphic Software

[Blama]

Hi Isomorphic,

thanks for adding.

Best regards
Blama