Trouble with binary upload

[bdelia]

Greetings,

We are trying to put together a simple file upload proof of concept with your api and are running into a problem that seems to be related to creating the POJO on the server side during the invocation of the DMI add method. Below I have included our code for the DataSource, Hibernate Mapping File, POJO, and DAO class. I have also included the error message that I see in the console. I am sure I am missing something very simple, but I have poured through your forums and sample code and cannot figure it out. I have tried various values in the type field for the DataSource and Hibernate Mapping file to no avail, currently in the DataSource the image field is set to binary, but we have also tried imageFile as well and get the same error message. Any assistance would be greatly appreciated.

I also uploaded a png file that shows that the POJO in the add method has the *_filename and *_filesize and title fields populated, but the issues seems to be around getting the binary data in the image field.

We are using version 2.4

******************* My DataSource *******************

<DataSource ID="imageLibrary" serverType="sql" tableName="imageLibrary">

    <fields>
    	<!-- The field names in here must match the getter methods in the POJO by Java bean convention, 
    	  therefore, if there is a getter "getTitle", the field name should be "title" with a lower case "t". -->
        <field name="pk" type="sequence" hidden="true" primaryKey="true"/> 
        <field name="title" type="text" /> 
        <field name="image" type="binary"/> 
    </fields> 
    
    <serverObject lookupStyle="spring" bean="imageItemDao"/>

</DataSource>

******************* My Hibernate mapping file *******************

<?xml version="1.0"?>
<!DOCTYPE hibernate-mapping PUBLIC
        "-//Hibernate/Hibernate Mapping DTD 3.0//EN"
        "http://hibernate.sourceforge.net/hibernate-mapping-3.0.dtd">

<hibernate-mapping>
    <class name="com.afileupload.server.ImageItem" table="imageLibrary">
        <id name="pk">
            <generator class="native" />
        </id>
        <property name="title"/>
        <property name="image" type="blob" />
    </class>
</hibernate-mapping>

******************* My POJO *******************

package com.afileupload.server;

import java.io.Serializable;

public class ImageItem implements Serializable {

	// this bean has no business logic.  It simply stores data in these instance variables.
    private Long pk;
    private String title;
    private byte [] image;
    private long image_filesize;
	private String image_filename;
    
	// a zero-argument constructor is not required, but does enable certain convenience
    // features (see the docs for DMI)
    public ImageItem() { }

    public void setPk(Long id) { pk = id; }
    public Long getPk() { return pk; }
    
    public void setTitle(String title) { this.title = title; }
	public String getTitle() { return title; }

	public void setImage(byte [] image) { this.image = image; }
	public byte [] getImage() { return image; }

	public void setImage_filesize(long image_filesize) {
		this.image_filesize = image_filesize;
	}
	public long getImage_filesize() {
		return image_filesize;
	}

	public void setImage_filename(String image_filename) {
		this.image_filename = image_filename;
	}
	public String getImage_filename() {
		return image_filename;
	}
}

******************* My DAO *******************

package com.afileupload.server;

import java.io.File;
import java.io.FileOutputStream;
import java.io.InputStream;
import java.io.OutputStream;
import java.util.List;
import java.util.Map;

import org.hibernate.Criteria;
import org.hibernate.Session;
import org.hibernate.SessionFactory;
import org.hibernate.criterion.Criterion;
import org.hibernate.criterion.MatchMode;
import org.hibernate.criterion.Projections;
import org.hibernate.criterion.Restrictions;

import com.isomorphic.datasource.DSRequest;
import com.isomorphic.datasource.DSResponse;
import com.isomorphic.log.Logger;
import com.isomorphic.util.DataTools;
import com.isomorphic.util.ErrorReport;


public class ImageItemDao {

    Logger log = new Logger(ImageItemDao.class.getName());

    // auto-configured by Spring
    private SessionFactory sessionFactory;
    public void setSessionFactory(SessionFactory sessionFactory) {
        this.sessionFactory = sessionFactory;
    }

    public DSResponse fetch(DSRequest dsRequest)
        throws Exception
    {
        log.info("procesing DMI fetch operation");
        DSResponse dsResponse = new DSResponse();
        System.out.println("fetch");
        

        Session hibernateSession = sessionFactory.getCurrentSession();

        // DataSource protocol: get filter criteria
        String itemName = (String)dsRequest.getFieldValue("itemName");

        // DataSource protocol: get requested row range
        long startRow = (int)dsRequest.getStartRow();
        long endRow = (int)dsRequest.getEndRow();

        Criteria criteria = hibernateSession.createCriteria(ImageItem.class);
        Criterion itemNameRestriction = null;
        if (itemName != null) {
            itemNameRestriction = Restrictions.like("itemName", itemName, MatchMode.ANYWHERE);
            criteria.add(itemNameRestriction);
        }

        // determine total available rows
        // this is used by e.g. the ListGrid to auto-size its scrollbar
        criteria.setProjection(Projections.rowCount());
        Object rowCount = criteria.uniqueResult();
        long totalRows = 0;
        // Later versions of Hibernate return a Long rather than an Integer here, for all
        // those occasions when a fetch returns more than 2.1 billion rows...
        if (rowCount instanceof Integer) {
            totalRows = ((Integer)rowCount).intValue();
        } else if (rowCount instanceof Long) {
            totalRows = ((Long)rowCount).longValue();
        }

        // clamp endRow to available rows and slice out requested range
        endRow = Math.min(endRow, totalRows);

        // rebuilt the criteria minus the rowCount projection
        criteria = hibernateSession.createCriteria(ImageItem.class);
        if (itemName != null) criteria.add(itemNameRestriction);

        // limit number of rows returned to just what the ListGrid asked for
        criteria.setFirstResult((int)startRow);
        criteria.setMaxResults((int)(endRow - startRow));
        List matchingItems = criteria.list();

        // DataSource protocol: return matching item beans
        dsResponse.setData(matchingItems);
        // tell client what rows are being returned, and what's available
        dsResponse.setStartRow(startRow);
        dsResponse.setEndRow(endRow);
        dsResponse.setTotalRows(totalRows);

        return dsResponse;
    }

    public DSResponse add(DSRequest dsRequest, ImageItem item)
        throws Exception
    {
        log.info("procesing DMI add operation");

        DSResponse dsResponse = new DSResponse();

        long startRow = dsRequest.getStartRow();
    	long endRow = dsRequest.getEndRow();
    	long totalRows = 1;
        
    	String fileName = dsRequest.getUploadedFile("image").getFileName();
        File f = new File("C:\\" + fileName);
        
        InputStream is = dsRequest.getUploadedFileStream("image");
        
        OutputStream out = new FileOutputStream(f);
        byte buf[] = new byte[1024];
        int len;
        while((len = is.read(buf)) > 0) {
        	
        	out.write(buf,0,len);	
        }

        out.close();
        is.close();

        // perform validation
        ErrorReport errorReport = dsRequest.getDataSource().validate(DataTools.getProperties(item), false);
        if (errorReport != null) {
           dsResponse.setStatus(DSResponse.STATUS_VALIDATION_ERROR);
           dsResponse.setErrorReport(errorReport);
           System.out.println("Errors: " + DataTools.prettyPrint(errorReport));
           return dsResponse;
        }

        Session hibernateSession = sessionFactory.getCurrentSession();
        hibernateSession.saveOrUpdate(item);
        dsResponse.setData(item);
        
        // tell client what rows are being returned, and what's available
        dsResponse.setStartRow(startRow);
        dsResponse.setEndRow(endRow);
        dsResponse.setTotalRows(totalRows);
        
        return dsResponse;
    }

    public DSResponse update(DSRequest dsRequest, Map newValues)
        throws Exception
    {
        log.info("procesing DMI update operation");

        DSResponse dsResponse = new DSResponse();
        System.out.println("update");

        // perform validation
        ErrorReport errorReport = dsRequest.getDataSource().validate(newValues, false);
        if (errorReport != null) {
           dsResponse.setStatus(DSResponse.STATUS_VALIDATION_ERROR);
           dsResponse.setErrorReport(errorReport);
           System.out.println("Errors: " + DataTools.prettyPrint(errorReport));
           return dsResponse;
        }

        // primary key
        Serializable id = (Serializable)dsRequest.getFieldValue("itemID");

        Session hibernateSession = sessionFactory.getCurrentSession();
        ImageItem item = (ImageItem)hibernateSession.get(ImageItem.class, id);

        log.warn("fetched item: " + DataTools.prettyPrint(item));

        // apply new values to the as-saved bean
        DataTools.setProperties(newValues, item);

        log.warn("Saving record: " + DataTools.prettyPrint(item));

        // persist
        hibernateSession.saveOrUpdate(item);
        dsResponse.setData(item);
        return dsResponse;
    }

    public ImageItem remove(ImageItem item)
        throws Exception
    {
        log.info("procesing DMI remove operation");
        System.out.println("remove");

        Session hibernateSession = sessionFactory.getCurrentSession();
        hibernateSession.delete(item);

        return item;
    }
}

Error:

=== 2011-03-01 08:28:06,012 [l0-0] INFO  RequestContext - URL: '/afileupload/sc/IDACall', User-Agent: 'Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.2.13) Gecko/20101203 Firefox/3.6.13': Moz (Gecko) with Accept-Encoding header
=== 2011-03-01 08:28:06,083 [l0-0] DEBUG XML - Parsed XML from (in memory stream): 0ms
=== 2011-03-01 08:28:06,083 [l0-0] DEBUG XML - Parsed XML from C:\Projects\Product_Development_And_Engineering\POC\AFileUpload\war\afileupload\sc\system\schema\List.ds.xml: 0ms
=== 2011-03-01 08:28:06,093 [l0-0] DEBUG RPCManager - Processing 1 requests.
=== 2011-03-01 08:28:06,103 [l0-0] DEBUG RPCManager - Request #1 (DSRequest) payload: {
    values:{
        image:"Desert.jpg",
        image_filename:"Desert.jpg",
        image_filesize:845941,
        image_date_created:new Date(1298996886103)
    },
    operationConfig:{
        dataSource:"imageLibrary",
        operationType:"add"
    },
    componentId:"isc_DynamicForm_0",
    appID:"builtinApplication",
    operation:"imageLibrary_add",
    oldValues:{
    },
    criteria:{
    }
}
=== 2011-03-01 08:28:06,103 [l0-0] INFO  IDACall - Performing 1 operation(s)
=== 2011-03-01 08:28:06,113 [l0-0] WARN  Validation - No such type 'blob', not processing field value at /imageLibrary/image
=== 2011-03-01 08:28:06,113 [l0-0] INFO  ServerObject - DMI on Spring bean: imageItemDao
=== 2011-03-01 08:28:06,123 [l0-0] WARN  DataTools - Impossible to convert to target type [B - it is not a concrete class
java.lang.IllegalArgumentException: Can't convert value of type java.lang.String to target type [B
	at com.isomorphic.util.DataTools.convertType(DataTools.java:2826)
	at com.isomorphic.util.DataTools.createSetterArgument(DataTools.java:2718)
	at com.isomorphic.util.DataTools.coerceProperty(DataTools.java:2656)
	at com.isomorphic.util.DataTools.setProperties(DataTools.java:2441)
	at com.isomorphic.util.DataTools.setProperties(DataTools.java:2384)
	at com.isomorphic.datasource.DataSource.setProperties(DataSource.java:1190)
	at com.isomorphic.base.Reflection.adaptValue(Reflection.java:1397)
	at com.isomorphic.base.Reflection.adaptArgsAndInvoke(Reflection.java:856)
	at com.isomorphic.datasource.DataSourceDMI.execute(DataSourceDMI.java:586)
	at com.isomorphic.datasource.DataSourceDMI.execute(DataSourceDMI.java:64)
	at com.isomorphic.datasource.DSRequest.execute(DSRequest.java:1440)
	at com.isomorphic.servlet.IDACall.handleDSRequest(IDACall.java:173)
	at com.isomorphic.servlet.IDACall.processRPCTransaction(IDACall.java:130)
	at com.isomorphic.servlet.IDACall.processRequest(IDACall.java:95)
	at com.isomorphic.servlet.IDACall.doPost(IDACall.java:54)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:637)
	at com.isomorphic.servlet.BaseServlet.service(BaseServlet.java:152)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
	at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:487)
	at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:362)
	at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
	at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181)
	at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:729)
	at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:405)
	at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
	at org.mortbay.jetty.handler.RequestLogHandler.handle(RequestLogHandler.java:49)
	at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
	at org.mortbay.jetty.Server.handle(Server.java:324)
	at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:505)
	at org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java:843)
	at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:647)
	at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:211)
	at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:380)
	at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:395)
	at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:488)
=== 2011-03-01 08:28:06,284 [l0-0] DEBUG RPCManager - Content type for RPC transaction: text/html; charset=UTF-8
=== 2011-03-01 08:28:06,284 [l0-0] DEBUG RPCManager - DMI response, dropExtraFields: true

[Isomorphic]

Several things wrong here..

First, you've written far more code than you needed to. Your DAO file contains a re-implementation of functionality built into the SQL and Hibernate connectors. You could currently delete *all* of your Java code, your <serverObject> declaration, and your Hibernate mapping to fix this problem, since the SQLDataSource supports storing files to tables with no server code required at all (as shown in this sample).

Note also that by hand-writing this logic you've actually severely degraded the built-in functionality of the system - you've removed support for AdvancedCriteria, transactions, and various other features which are built-in.

As far as what's actually going wrong in this sample, basically the auto-construction of your ImageItem implied by having it as an argument to the "add" method won't work because of the inbound file. This is an interesting edge case and we'll go ahead and make it work in the future, however what would fix it for now:
1. remove the argument of type ImageItem, and create one in the method instead
2. grab the InputStream of the file and apply to the ImageItem's byte[] property manually
3. use dsRequest.setFieldValue("image", null) to clear out the inbound value for this field (which is a String placeholder)
4. apply the remaining values to your bean using DataSource.setProperties()

Just to reiterate, none of these steps are necessary - the right path forward is to delete everything you've written except the .ds.xml file.

[bdelia]

Thanks for the very quick response!!

We implemented it this way mostly because we wanted to intercept the file and also write it to the server file system as well as to the DB. Is there a recommended way of both storing to the DB as well as intercepting the request and like with DMI?

In the mean time I will try your suggestions.

Thanks again for your help and any additional thoughts.

[Isomorphic]

In that case, the only thing you need is just that one bit of logic in a DMI. You can route all CRUD operations to one Java method using the "methodName" attribute of <serverObject> (like this).

In that DMI method, grab the file, store it to disk, then just return dsRequest.execute() to cause the default storage operation to continue. This is discussed in more detail in the QuickStart chapter on the Server Framework.

You don't need any other code - no need for an ImageItem bean, Hibernate cfg, etc - just the one Java DMI method.

[bdelia]

Well that certainly seems much more elegant than our solution ;-)

One more question though. How would you inject possible error conditions in that DMI method so that you could cause the default .execute to not fire and use either your built in error handling, like for form validation, or override it and handle ourselves?

Thanks again for such speedy responses, I am very impressed!!

[Isomorphic]

That's also in the QuickStart Guide - you'll want to read the whole thing, but specifically look at all the samples under the section "Adding DMI Business Logic" in the Server Framework chapter.

See also these two samples which show you how you can add a validator that uses DMI to call your Java validation logic, and how to add a validator that uses a Velocity expression right in the .ds.xml file to perform validation.

[bdelia]

Worked thanks for your help and guidance!!

[Isomorphic]

We're curious what the mistake was? That was a bit of a stumper of an error message (it was claiming mismatched columns and values but they clearly matched).

[bdelia]

It was a bit weird to figure out at first, but then I tracked it down. I am currently just using the in memory hibernate sql db as per most of your examples. Originally my table was created by the hibernate file associated with my POJO. I had since refactored my project to use your recommended patterns and took out those files. But doing so left the original table in the db. That table did not really match the sql insert that was going on. I have since recreated with the proper columns, my original did not have image_filesize, image_filename, and image_create_date. Now it seems to work. I just had to create a compatible table in the db.

Thanks again for your help.

[Isomorphic]

Ah, thanks for the clarification, that makes sense now.