Announcement

**Isomorphic** · 23 Sep 2013, 11:18

BatchUploader has two phases of committing - one for initial upload of data, before the data is shown to the user, and one for final commit of data to the DataSource.

At which point do you want to reject the upload? Do you want the end user to be able to correct the dates by editing the grid?

**henkiemaritz** · 24 Sep 2013, 23:36

I want to validate as soon as possible. So, the first phase should be fine.
I do NOT want to give the user opportunity to correct dates. He is importing a bank statement and an error means that he is importing duplicate entries.

**Isomorphic** · 25 Sep 2013, 09:44

It should be enough for your batchUploader.uploadDataSource to simply return a DSResponse with an error result for any individual attempted "add" operation that has bad data.

But you mentioned you'd read all the docs - so had you already tried this and had an issue with it? If so, we'll need more information - server log, the code you used, etc.

**henkiemaritz** · 27 Sep 2013, 05:03

Thank you. I got the DMI validation to work. My issue was that I used a new DSRequest without setting the RPCManager on the new DSRequest.

So, the validation works fine, it rollbock the entire CSV even if the last record fails the validation.

My issue is, how do I get the message to the user.

Right now, it works as follows:
BatchUploader -> DataSource -> DMIClass

The DMI class through an exception if the validation fails. I also tried adding an error to the DSResponse:
DSResponse error = new DSResponse();
error.addError("field","The date is wrong.");
return error;

I only get a message that says "Records added".
Here's my code:
isc.BatchUploader.create({
ID: "buAccountsReconImportCSV",
height: 400,
width: 1200,
required: true,
partialCommit: "prevent",
partialCommitError: "Error!!",
uploadDataSource: "account_bank_csv_upload",
gridProperties: {
width: 1200,
height: 400,
canEdit: false,
showHeaderContextMenu: false
}
});

Here's the log:
=== 2013-09-27 14:55:30,119 [ec-4] DEBUG RPCManager - Request #1 (DSRequest) payload: {
criteria:{
},
values:{
file:"C:\\fakepath\\testStatement.csv",
dsName:"account_bank_csv_upload",
delimiter:",",
quoteString:"\""
},
operationConfig:{
dataSource:"batchUpload",
operationType:"update"
},
componentId:"isc_DynamicForm_0",
appID:"builtinApplication",
operation:"upload",
oldValues:{
dsName:"account_bank_csv_upload",
delimiter:",",
quoteString:"\""
}
}
=== 2013-09-27 14:55:30,122 [ec-4] INFO IDACall - Performing 1 operation(s)
=== 2013-09-27 14:55:30,122 [ec-4] DEBUG DeclarativeSecurity - Processing security checks for DataSource null, field null
=== 2013-09-27 14:55:30,122 [ec-3] DEBUG RPCDMI - rpc returned data
=== 2013-09-27 14:55:30,128 [ec-4] DEBUG DeclarativeSecurity - DataSource batchUpload is not in the pre-checked list, processing...
=== 2013-09-27 14:55:30,129 [ec-3] DEBUG RPCManager - Content type for RPC transaction: text/plain; charset=UTF-8
=== 2013-09-27 14:55:30,135 [ec-3] INFO Compression - /war/version/9.0/isomorphic/IDACall: 67 -> 73 bytes
=== 2013-09-27 14:55:30,145 [ec-4] DEBUG BatchUpload - Rows: [
{
line_d:"20130423",
line_descr:"IBANK BETALING NA VEREFFEN ABSA BANK Betlhem toer hemde ",
line_amount:"-700"
},
{
line_d:"20130423",
line_descr:"ACB KREDIET VEREFFEN PIETER SCHOEMAN ",
line_amount:"420"
},
{
line_d:"20130902",
line_descr:"ACB KREDIET VEREFFEN 12625 DE KOCK ",
line_amount:"300"
}
]
=== 2013-09-27 14:55:30,146 [ec-4] INFO DSResponse - DSResponse: Map with 1 keys
=== 2013-09-27 14:55:30,147 [ec-4] DEBUG RPCManager - Content type for RPC transaction: text/html; charset=UTF-8
=== 2013-09-27 14:55:30,147 [ec-4] DEBUG RPCManager - DMI response, dropExtraFields: false
=== 2013-09-27 14:55:30,152 [ec-4] INFO Compression - /war/version/9.0/isomorphic/IDACall: 961 -> 548 bytes
=== 2013-09-27 14:55:31,649 [ec-5] INFO RequestContext - URL: '/war/version/9.0/isomorphic/IDACall', User-Agent: 'Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/29.0.1547.76 Safari/537.36': Safari with Accept-Encoding header
=== 2013-09-27 14:55:31,650 [ec-5] DEBUG IDACall - Header Name:Value pair: host:localhost:8888
=== 2013-09-27 14:55:31,650 [ec-5] DEBUG IDACall - Header Name:Value pair: connection:keep-alive
=== 2013-09-27 14:55:31,651 [ec-5] DEBUG IDACall - Header Name:Value pair: content-length:2850
=== 2013-09-27 14:55:31,653 [ec-5] DEBUG IDACall - Header Name:Value pair: origin:http://localhost:8888
=== 2013-09-27 14:55:31,654 [ec-5] DEBUG IDACall - Header Name:Value pair: user-agent:Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/29.0.1547.76 Safari/537.36
=== 2013-09-27 14:55:31,656 [ec-5] DEBUG IDACall - Header Name:Value pair: content-type:application/x-www-form-urlencoded; charset=UTF-8
=== 2013-09-27 14:55:31,657 [ec-5] DEBUG IDACall - Header Name:Value pair: accept:*/*
=== 2013-09-27 14:55:31,658 [ec-5] DEBUG IDACall - Header Name:Value pair: referer:http://localhost:8888/war/secure/index.jsp
=== 2013-09-27 14:55:31,659 [ec-5] DEBUG IDACall - Header Name:Value pair: accept-encoding:gzip,deflate,sdch
=== 2013-09-27 14:55:31,661 [ec-5] DEBUG IDACall - Header Name:Value pair: accept-language:en-US,en;q=0.8
=== 2013-09-27 14:55:31,661 [ec-5] DEBUG IDACall - Header Name:Value pair: cookie:JSESSIONID=AE08866211F8EEB0416EAFA51DD5D197; isc_cState=ready
=== 2013-09-27 14:55:31,663 [ec-5] DEBUG IDACall - session exists: AE08866211F8EEB0416EAFA51DD5D197
=== 2013-09-27 14:55:31,664 [ec-5] DEBUG IDACall - remote user: null
=== 2013-09-27 14:55:31,671 [ec-5] DEBUG XML - Parsed XML from (in memory stream): 2ms
=== 2013-09-27 14:55:31,673 [ec-5] DEBUG RPCManager - Processing 3 requests.
=== 2013-09-27 14:55:31,675 [ec-5] DEBUG RPCManager - Request #1 (DSRequest) payload: {
values:{
line_d:"20130423",
line_descr:"IBANK BETALING NA VEREFFEN ABSA BANK Betlhem toer hemde ",
line_amount:"-700"
},
operationConfig:{
dataSource:"account_bank_csv_upload",
operationType:"add"
},
appID:"builtinApplication",
operation:"account_bank_csv_upload_add",
oldValues:{
line_d:"20130423",
line_descr:"IBANK BETALING NA VEREFFEN ABSA BANK Betlhem toer hemde ",
line_amount:"-700"
},
criteria:{
}
}
=== 2013-09-27 14:55:31,677 [ec-5] DEBUG RPCManager - Request #2 (DSRequest) payload: {
values:{
line_d:"20130423",
line_descr:"ACB KREDIET VEREFFEN PIETER SCHOEMAN ",
line_amount:"420"
},
operationConfig:{
dataSource:"account_bank_csv_upload",
operationType:"add"
},
appID:"builtinApplication",
operation:"account_bank_csv_upload_add",
oldValues:{
line_d:"20130423",
line_descr:"ACB KREDIET VEREFFEN PIETER SCHOEMAN ",
line_amount:"420"
},
criteria:{
}
}
=== 2013-09-27 14:55:31,679 [ec-5] DEBUG RPCManager - Request #3 (DSRequest) payload: {
values:{
line_d:"20130902",
line_descr:"ACB KREDIET VEREFFEN 12625 DE KOCK ",
line_amount:"300"
},
operationConfig:{
dataSource:"account_bank_csv_upload",
operationType:"add"
},
appID:"builtinApplication",
operation:"account_bank_csv_upload_add",
oldValues:{
line_d:"20130902",
line_descr:"ACB KREDIET VEREFFEN 12625 DE KOCK ",
line_amount:"300"
},
criteria:{
}
}
=== 2013-09-27 14:55:31,681 [ec-5] INFO IDACall - Performing 3 operation(s)
=== 2013-09-27 14:55:31,684 [ec-5] DEBUG DeclarativeSecurity - Processing security checks for DataSource null, field null
=== 2013-09-27 14:55:31,684 [ec-5] DEBUG DeclarativeSecurity - DataSource account_bank_csv_upload is not in the pre-checked list, processing...
=== 2013-09-27 14:55:31,699 [ec-5] DEBUG DeclarativeSecurity - Processing security checks for DataSource null, field null
=== 2013-09-27 14:55:31,700 [ec-5] DEBUG DeclarativeSecurity - Processing security checks for DataSource null, field null
=== 2013-09-27 14:55:31,700 [ec-5] DEBUG AppBase - [builtinApplication.null] No userTypes defined, allowing anyone access to all operations for this application
=== 2013-09-27 14:55:31,701 [ec-5] DEBUG AppBase - [builtinApplication.null] No public zero-argument method named '_null' found, performing generic datasource operation
=== 2013-09-27 14:55:31,703 [ec-5] INFO SQLDataSource - [builtinApplication.null] Performing add operation with
criteria: {company_id:"2",line_d:new Date(1366668000000),status_cd:"Unallocated",line_descr:"IBANK BETALING NA VEREFFEN ABSA BANK Betlhem toer hemde ",line_amount:700.0} values: {company_id:2,line_d:new Date(1366668000000),status_cd:"Unallocated",line_descr:"IBANK BETALING NA VEREFFE
N ABSA BANK Betlhem toer hemde ",line_amount:700.0}
=== 2013-09-27 14:55:31,706 [ec-5] DEBUG SQLValuesClause - [builtinApplication.null] Sequences: {account_bank_id=__default, account_journal_id=__default}
=== 2013-09-27 14:55:31,713 [ec-5] DEBUG PoolableSQLConnectionFactory - [builtinApplication.null] Executing pingTest 'select 1 from dual' on connection 1573984250
=== 2013-09-27 14:55:31,714 [ec-5] DEBUG SQLConnectionManager - [builtinApplication.null] Borrowed connection '1573984250'
=== 2013-09-27 14:55:31,715 [ec-5] DEBUG SQLTransaction - [builtinApplication.null] Started new Mysql transaction "1573984250"
=== 2013-09-27 14:55:31,715 [ec-5] DEBUG SQLDriver - [builtinApplication.null] About to execute SQL update in 'Mysql' using connection'1573984250'
=== 2013-09-27 14:55:31,717 [ec-5] INFO SQLDriver - [builtinApplication.null] Executing SQL update on 'Mysql': INSERT INTO account_bank (company_id, line_amount, line_d, line_descr, status_cd) VALUES (2, 700.0, '2013-04-23', 'IBANK BETALING NA VEREFFEN ABSA BANK Betlhem toer hemde ', 'Unallocated
')
=== 2013-09-27 14:55:31,731 [ec-5] DEBUG SQLDataSource - [builtinApplication.null] add operation affected 1 rows
=== 2013-09-27 14:55:31,731 [ec-5] DEBUG MysqlDriver - [builtinApplication.null] fetchLastRow data - primaryKeysPresent: {}sequencesNotPresent: [account_bank_id]
=== 2013-09-27 14:55:31,732 [ec-5] DEBUG SQLDriver - [builtinApplication.null] About to execute SQL query in 'Mysql' using connection '1573984250'
=== 2013-09-27 14:55:31,732 [ec-5] INFO SQLDriver - [builtinApplication.null] Executing SQL query on 'Mysql': SELECT LAST_INSERT_ID()
=== 2013-09-27 14:55:31,734 [ec-5] INFO SQLDataSource - [builtinApplication.null] primaryKeys: {account_bank_id=259}
=== 2013-09-27 14:55:31,735 [ec-5] DEBUG DeclarativeSecurity - [builtinApplication.null] Processing security checks for DataSource null, field null
=== 2013-09-27 14:55:31,735 [ec-5] DEBUG DeclarativeSecurity - [builtinApplication.null] DataSource account_bank is not in the pre-checked list, processing...
=== 2013-09-27 14:55:31,738 [ec-5] DEBUG DeclarativeSecurity - [builtinApplication.null] Processing security checks for DataSource null, field null
=== 2013-09-27 14:55:31,738 [ec-5] DEBUG DeclarativeSecurity - [builtinApplication.null] DataSource account_bank is not in the pre-checked list, processing...
=== 2013-09-27 14:55:31,739 [ec-5] DEBUG AppBase - [builtinApplication.null, builtinApplication.null] No userTypes defined, allowing anyone access to all operations for this application
=== 2013-09-27 14:55:31,740 [ec-5] DEBUG AppBase - [builtinApplication.null, builtinApplication.null] No public zero-argument method named '_null' found, performing generic datasource operation
=== 2013-09-27 14:55:31,742 [ec-5] INFO SQLDataSource - [builtinApplication.null, builtinApplication.null] Performing fetch operation with
criteria: {account_bank_id:259} values: {account_bank_id:259}
=== 2013-09-27 14:55:31,743 [ec-5] INFO SQLDataSource - [builtinApplication.null, builtinApplication.null] derived query: SELECT $defaultSelectClause FROM $defaultTableClause WHERE $defaultWhereClause
=== 2013-09-27 14:55:31,744 [ec-5] INFO SQLDataSource - [builtinApplication.null, builtinApplication.null] 80: Executing SQL query on 'Mysql': SELECT account_bank.account_bank_id, account_bank.account_journal_id, account_bank.company_account_number, account_bank.company_id, account_bank.line_amount
, account_bank.line_d, account_bank.line_descr, account_bank.status_cd FROM account_bank WHERE (account_bank.account_bank_id=259)
=== 2013-09-27 14:55:31,745 [ec-5] DEBUG SQLDriver - [builtinApplication.null, builtinApplication.null] About to execute SQL query in 'Mysql' using connection '1573984250'
=== 2013-09-27 14:55:31,747 [ec-5] INFO SQLDriver - [builtinApplication.null, builtinApplication.null] Executing SQL query on 'Mysql': SELECT account_bank.account_bank_id, account_bank.account_journal_id, account_bank.company_account_number, account_bank.company_id, account_bank.line_amount, accoun
t_bank.line_d, account_bank.line_descr, account_bank.status_cd FROM account_bank WHERE (account_bank.account_bank_id=259)
=== 2013-09-27 14:55:31,755 [ec-5] INFO DSResponse - [builtinApplication.null, builtinApplication.null] DSResponse: List with 1 items
=== 2013-09-27 14:55:31,755 [ec-5] INFO DSResponse - [builtinApplication.null] DSResponse: List with 1 items
=== 2013-09-27 14:55:31,756 [ec-5] DEBUG DeclarativeSecurity - Processing security checks for DataSource null, field null
=== 2013-09-27 14:55:31,756 [ec-5] DEBUG DeclarativeSecurity - DataSource account_bank_csv_upload is not in the pre-checked list, processing...
=== 2013-09-27 14:55:31,757 [ec-5] DEBUG DeclarativeSecurity - Processing security checks for DataSource null, field null
=== 2013-09-27 14:55:31,759 [ec-5] DEBUG DeclarativeSecurity - Processing security checks for DataSource null, field null
=== 2013-09-27 14:55:31,759 [ec-5] DEBUG AppBase - [builtinApplication.null] No userTypes defined, allowing anyone access to all operations for this application
=== 2013-09-27 14:55:31,761 [ec-5] DEBUG AppBase - [builtinApplication.null] No public zero-argument method named '_null' found, performing generic datasource operation
=== 2013-09-27 14:55:31,762 [ec-5] INFO SQLDataSource - [builtinApplication.null] Performing add operation with
criteria: {company_id:"2",line_d:new Date(1366668000000),status_cd:"Unallocated",line_descr:"ACB KREDIET VEREFFEN PIETER SCHOEMAN ",line_amount:-420.0} values: {company_id:2,line_d:new Date(1366668000000),status_cd:"Unallocated",line_descr:"ACB KREDIET VEREFFEN PIETER SCHOEMA
N ",line_amount:-420.0}
=== 2013-09-27 14:55:31,764 [ec-5] DEBUG SQLValuesClause - [builtinApplication.null] Sequences: {account_bank_id=__default, account_journal_id=__default}
=== 2013-09-27 14:55:31,764 [ec-5] DEBUG SQLDriver - [builtinApplication.null] About to execute SQL update in 'Mysql' using connection'1573984250'
=== 2013-09-27 14:55:31,765 [ec-5] INFO SQLDriver - [builtinApplication.null] Executing SQL update on 'Mysql': INSERT INTO account_bank (company_id, line_amount, line_d, line_descr, status_cd) VALUES (2, -420.0, '2013-04-23', 'ACB KREDIET VEREFFEN PIETER SCHOEMAN ', 'Unallocated')
=== 2013-09-27 14:55:31,766 [ec-5] DEBUG SQLDataSource - [builtinApplication.null] add operation affected 1 rows
=== 2013-09-27 14:55:31,767 [ec-5] DEBUG MysqlDriver - [builtinApplication.null] fetchLastRow data - primaryKeysPresent: {}sequencesNotPresent: [account_bank_id]
=== 2013-09-27 14:55:31,768 [ec-5] DEBUG SQLDriver - [builtinApplication.null] About to execute SQL query in 'Mysql' using connection '1573984250'
=== 2013-09-27 14:55:31,769 [ec-5] INFO SQLDriver - [builtinApplication.null] Executing SQL query on 'Mysql': SELECT LAST_INSERT_ID()
=== 2013-09-27 14:55:31,771 [ec-5] INFO SQLDataSource - [builtinApplication.null] primaryKeys: {account_bank_id=260}
=== 2013-09-27 14:55:31,772 [ec-5] DEBUG DeclarativeSecurity - [builtinApplication.null] Processing security checks for DataSource null, field null
=== 2013-09-27 14:55:31,772 [ec-5] DEBUG DeclarativeSecurity - [builtinApplication.null] DataSource account_bank is not in the pre-checked list, processing...
=== 2013-09-27 14:55:31,774 [ec-5] DEBUG DeclarativeSecurity - [builtinApplication.null] Processing security checks for DataSource null, field null
=== 2013-09-27 14:55:31,775 [ec-5] DEBUG DeclarativeSecurity - [builtinApplication.null] DataSource account_bank is not in the pre-checked list, processing...
=== 2013-09-27 14:55:31,776 [ec-5] DEBUG AppBase - [builtinApplication.null, builtinApplication.null] No userTypes defined, allowing anyone access to all operations for this application
=== 2013-09-27 14:55:31,776 [ec-5] DEBUG AppBase - [builtinApplication.null, builtinApplication.null] No public zero-argument method named '_null' found, performing generic datasource operation
=== 2013-09-27 14:55:31,779 [ec-5] INFO SQLDataSource - [builtinApplication.null, builtinApplication.null] Performing fetch operation with
criteria: {account_bank_id:260} values: {account_bank_id:260}
=== 2013-09-27 14:55:31,779 [ec-5] INFO SQLDataSource - [builtinApplication.null, builtinApplication.null] derived query: SELECT $defaultSelectClause FROM $defaultTableClause WHERE $defaultWhereClause
=== 2013-09-27 14:55:31,781 [ec-5] INFO SQLDataSource - [builtinApplication.null, builtinApplication.null] 80: Executing SQL query on 'Mysql': SELECT account_bank.account_bank_id, account_bank.account_journal_id, account_bank.company_account_number, account_bank.company_id, account_bank.line_amount
, account_bank.line_d, account_bank.line_descr, account_bank.status_cd FROM account_bank WHERE (account_bank.account_bank_id=260)
=== 2013-09-27 14:55:31,781 [ec-5] DEBUG SQLDriver - [builtinApplication.null, builtinApplication.null] About to execute SQL query in 'Mysql' using connection '1573984250'
=== 2013-09-27 14:55:31,783 [ec-5] INFO SQLDriver - [builtinApplication.null, builtinApplication.null] Executing SQL query on 'Mysql': SELECT account_bank.account_bank_id, account_bank.account_journal_id, account_bank.company_account_number, account_bank.company_id, account_bank.line_amount, accoun
t_bank.line_d, account_bank.line_descr, account_bank.status_cd FROM account_bank WHERE (account_bank.account_bank_id=260)
=== 2013-09-27 14:55:31,785 [ec-5] INFO DSResponse - [builtinApplication.null, builtinApplication.null] DSResponse: List with 1 items
=== 2013-09-27 14:55:31,785 [ec-5] INFO DSResponse - [builtinApplication.null] DSResponse: List with 1 items
=== 2013-09-27 14:55:31,786 [ec-5] DEBUG DeclarativeSecurity - Processing security checks for DataSource null, field null
=== 2013-09-27 14:55:31,787 [ec-5] DEBUG DeclarativeSecurity - DataSource account_bank_csv_upload is not in the pre-checked list, processing...
=== 2013-09-27 14:55:31,790 [ec-5] DEBUG RPCManager - Content type for RPC transaction: text/plain; charset=UTF-8
=== 2013-09-27 14:55:31,791 [ec-5] DEBUG SQLTransaction - Rolling back Mysql transaction "1573984250"
=== 2013-09-27 14:55:31,835 [ec-5] DEBUG RPCManager - DMI response, dropExtraFields: true
=== 2013-09-27 14:55:31,835 [ec-5] DEBUG RPCManager - DMI response, dropExtraFields: true
=== 2013-09-27 14:55:31,835 [ec-5] DEBUG RPCManager - DMI response, dropExtraFields: true
=== 2013-09-27 14:55:31,837 [ec-5] DEBUG SQLTransaction - Returning transactional connection for Mysql with hashcode "1573984250"
=== 2013-09-27 14:55:31,837 [ec-5] DEBUG SQLTransaction - Ending Mysql transaction "1573984250"
=== 2013-09-27 14:55:31,838 [ec-5] DEBUG SQLConnectionManager - About to close PoolableConnection with hashcode "1573984250"
=== 2013-09-27 14:55:31,838 [ec-5] DEBUG PoolableSQLConnectionFactory - Executing pingTest 'select 1 from dual' on connection 1573984250
=== 2013-09-27 14:55:31,841 [ec-5] DEBUG SQLTransaction - Returning transactional connection for Mysql (connection is null)
=== 2013-09-27 14:55:31,841 [ec-5] INFO Compression - /war/version/9.0/isomorphic/IDACall: 649 -> 309 bytes

**henkiemaritz** · 12 Jan 2014, 00:59

Hi guys
This is getting an issue now. I did provide more detail. Can we re-look at this please?
Tx

**Isomorphic** · 14 Jan 2014, 12:36

We show no issue with the process of returning an error via the uploadDataSource, and, while you posted some code for returning an error, your log does not show an error being returned, which suggests you are not actually running this code (or it's misplaced).

If you think there's a framework issue here, a good approach would be modify the Batch Uploader sample to try to reproduce the problem. If you are able to reproduce the problem, you can just show the modifications you made, and then we'd have something we can actually run to see the problem.

**henkiemaritz** · 4 Feb 2014, 17:10

I'm trying to bed this thing down. I actually need to validate the data before the add operations are done to the uploadDataSource.
I see the following logs in tomcat:

=== 2014-02-05 03:04:38,056 [ec-6] DEBUG DeclarativeSecurity - DataSource batchUpload is not in the pre-checked list, processing...
=== 2014-02-05 03:04:38,119 [ec-9] DEBUG RPCDMI - rpc returned data
=== 2014-02-05 03:04:38,125 [ec-6] DEBUG BatchUpload - Rows: [
{
line_d:"2014/02/03",
line_descr:"EFT Example 1",
line_amount:"500.00"
},
{
line_d:"2014/02/01",
line_descr:"EFT Example 2",
line_amount:"200.00"
},
{
line_d:"2014/02/01",
line_descr:"Expense Example",
line_amount:"-2602.00"
}
]
=== 2014-02-05 03:04:38,135 [ec-9] DEBUG RPCManager - Content type for RPC transaction: text/plain; charset=UTF-8
=== 2014-02-05 03:04:38,138 [ec-6] INFO DSResponse - DSResponse: Map with 1 keys
=== 2014-02-05 03:04:38,140 [ec-9] INFO Compression - /war/version/9.0/isomorphic/IDACall: 67 -> 73 bytes
=== 2014-02-05 03:04:38,147 [ec-6] DEBUG RPCManager - Content type for RPC transaction: text/html; charset=UTF-8
=== 2014-02-05 03:04:38,152 [ec-6] DEBUG RPCManager - DMI response, dropExtraFields: false

Question, how can I intercept the first phase of data import? My issue is, I need to trim some fields. Some text fields have leading spaces which cause it not to be imported.

**henkiemaritz** · 4 Feb 2014, 17:18

Sorry, let me just re state my actual issue:

I have a CSV like this:
Date,Amount,Balance,Description
2014/02/03,500.00,10500.00, EFT Example 1,more,text
2014/02/01,200.00,10700.00, EFT Example 2
2014/02/01,-2602.00,8098.00, Expense Example

This CSV have 2 issues:
1. ONLY Line one have two extra fields: more, text
2. Column Description has a leading space

Both these issues cause the CSV to not load at all. It does not even populate the initial grid or give an error.

Here's my data source again:
<DataSource
dbName="Mysql"
tableName="account_bank"
ID="account_bank_csv_upload"
dataSourceVersion="1"
serverType="sql"
>
<fields>
<field sqlType="varchar" sqlLength="10" name="line_d" title="Date" type="text"></field>
<field sqlType="varchar" sqlLength="100" name="line_descr" title="Description" length="100" type="text"></field>
<field sqlType="decimal" sqlLength="24" name="line_amount" title="Amount" type="float"></field>
</fields>

<serverObject className="za.co.adminonline.dmi.CSVImportDMI"/>

</DataSource>

Please help - this is causing serious customer issues.

**Isomorphic** · 4 Feb 2014, 17:27

If you want to preprocess the uploaded CSV to correct it in a way that can't be achieved by validation, take a look at this thread.

Note that a "regexp" validator with a "transformTo" property could be used to remove the leading space. But there's no validator that could correct a structural error such as those extra columns - for that you do need to use the technique described in the other thread.

**henkiemaritz** · 5 Feb 2014, 02:03

Thank you - this is really helpful.
Please help me with the syntax for the leading space issue. I've tried changing the datasource as follows:

Code:

<DataSource
dbName="Mysql"
tableName="account_bank"
ID="account_bank_csv_upload"
dataSourceVersion="1"
serverType="sql"
        >
<fields>
    <field sqlType="varchar" sqlLength="10" name="line_d" title="Date" type="text">
    </field>
    <field sqlType="varchar" sqlLength="100" name="line_descr" title="Description" length="100" type="text">
        <validators>
            <validator type="mask" mask="^ *[a-zA-Z0-9 ]*" transformTo="$0" />
        </validators>
    </field>
    <field sqlType="decimal" sqlLength="24" name="line_amount" title="Amount" type="float">
        <validators>
            <validator type="mask" mask="^ *[0-9. ]*" transformTo="$0" />
        </validators>
    </field>
</fields>

<serverObject className="za.co.adminonline.dmi.CSVImportDMI"/>

</DataSource>

This is not working well. I don't quite understand the transformTo. What do you include in there?

**Isomorphic** · 5 Feb 2014, 09:16

Syntax is the same as for JavaScript String.replace() - use parens ("(" and ")") to surround the parts of the regular expression you want to keep, then use $1, $2 etc to refer to the match for the first parenthesized expression, second parenthesized expression, and so on.

**henkiemaritz** · 7 Feb 2014, 06:48

Thank you, this works well - but if the heading has a leading space, it does not work. Any options to fix that?

**Isomorphic** · 7 Feb 2014, 09:44

For that, you need the technique explained in post #10 in this thread.

**henkiemaritz** · 11 Feb 2014, 06:34

Thank you - one more question to bed down Batch Upload validation.
In my DMI of the upload data source I do validations before sending it to the database. How do I send an error back to the BatchUploader?

If I throw an exception in the DMI code. This works since the upload as a whole is rolled back - but I can't get a proper message to the users. This is confusing them heavily.

See the following logs - when I through an exception:

Code:

=== 2014-02-11 16:25:39,375 [ec-3] DEBUG RPCManager - Request #1 (DSRequest) payload: {
    values:{
        line_d:"201411/12431234-12324",
        line_descr:"Format 9 ABSA BANK Afrigting",
        line_amount:"8000"
    },
    operationConfig:{
        dataSource:"account_bank_csv_upload",
        operationType:"add"
    },
    appID:"builtinApplication",
    operation:"account_bank_csv_upload_add",
    oldValues:{
        line_d:"201411/12431234-12324",
        line_descr:"Format 9 ABSA BANK Afrigting",
        line_amount:"8000"
    },
    criteria:{
    }
}
=== 2014-02-11 16:25:39,386 [ec-3] INFO  IDACall - Performing 1 operation(s)
=== 2014-02-11 16:25:39,390 [ec-3] DEBUG DeclarativeSecurity - Processing security checks for DataSource null, field null
=== 2014-02-11 16:25:39,405 [ec-3] DEBUG DeclarativeSecurity - DataSource account_bank_csv_upload is not in the pre-checked list, processing...
=== 2014-02-11 16:25:39,416 [ec-3] DEBUG DataSourceDMI - Invocation threw exception
java.lang.Exception: There is a date with an invalid format in the import file: 201411/12431234-12324
        at za.co.adminonline.dmi.CSVImportDMI.add(CSVImportDMI.java:48)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
        at java.lang.reflect.Method.invoke(Unknown Source)
        at com.isomorphic.base.Reflection.adaptArgsAndInvoke(Reflection.java:973)
        at com.isomorphic.datasource.DataSourceDMI.execute(DataSourceDMI.java:416)
        at com.isomorphic.datasource.DataSourceDMI.execute(DataSourceDMI.java:64)
        at com.isomorphic.datasource.DSRequest.execute(DSRequest.java:2427)
        at com.isomorphic.servlet.IDACall.handleDSRequest(IDACall.java:215)
        at com.isomorphic.servlet.IDACall.processRPCTransaction(IDACall.java:172)
        at com.isomorphic.servlet.IDACall.processRequest(IDACall.java:137)
        at com.isomorphic.servlet.IDACall.doPost(IDACall.java:73)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:647)
        at com.isomorphic.servlet.BaseServlet.service(BaseServlet.java:152)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:728)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
        at za.co.adminonline.servlet.SecurityCheck.doFilter(SecurityCheck.java:77)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
        at com.isomorphic.servlet.CompressionFilter.doFilter(CompressionFilter.java:259)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123)
        at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:472)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:99)
        at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:936)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:407)
        at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1004)
        at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:589)
        at org.apache.tomcat.util.net.AprEndpoint$SocketProcessor.run(AprEndpoint.java:1822)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
        at java.lang.Thread.run(Unknown Source)
=== 2014-02-11 16:25:39,438 [ec-3] WARN  RequestContext - dsRequest.execute() failed:
java.lang.Exception: There is a date with an invalid format in the import file: 201411/12431234-12324
        at za.co.adminonline.dmi.CSVImportDMI.add(CSVImportDMI.java:48)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
        at java.lang.reflect.Method.invoke(Unknown Source)
        at com.isomorphic.base.Reflection.adaptArgsAndInvoke(Reflection.java:973)
        at com.isomorphic.datasource.DataSourceDMI.execute(DataSourceDMI.java:416)
        at com.isomorphic.datasource.DataSourceDMI.execute(DataSourceDMI.java:64)
        at com.isomorphic.datasource.DSRequest.execute(DSRequest.java:2427)
        at com.isomorphic.servlet.IDACall.handleDSRequest(IDACall.java:215)
        at com.isomorphic.servlet.IDACall.processRPCTransaction(IDACall.java:172)
        at com.isomorphic.servlet.IDACall.processRequest(IDACall.java:137)
        at com.isomorphic.servlet.IDACall.doPost(IDACall.java:73)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:647)
        at com.isomorphic.servlet.BaseServlet.service(BaseServlet.java:152)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:728)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
        at za.co.adminonline.servlet.SecurityCheck.doFilter(SecurityCheck.java:77)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
        at com.isomorphic.servlet.CompressionFilter.doFilter(CompressionFilter.java:259)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123)
        at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:472)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:99)
        at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:936)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:407)
        at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1004)
        at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:589)
        at org.apache.tomcat.util.net.AprEndpoint$SocketProcessor.run(AprEndpoint.java:1822)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
        at java.lang.Thread.run(Unknown Source)
=== 2014-02-11 16:25:39,480 [ec-3] DEBUG RPCManager - Content type for RPC transaction: text/plain; charset=UTF-8
=== 2014-02-11 16:25:39,481 [ec-3] DEBUG RPCManager - non-DMI response, dropExtraFields: false
=== 2014-02-11 16:25:39,485 [ec-3] INFO  Compression - /war/version/9.0/isomorphic/IDACall: 201 -> 171 bytes

See the following logs - when I DO NOT through an exception but addError to the response returned:

Code:

=== 2014-02-11 16:20:34,900 [ec-5] DEBUG RPCManager - Request #1 (DSRequest) payload: {
    values:{
        line_d:"201411/12431234-12324",
        line_descr:"Format 9 ABSA BANK Afrigting",
        line_amount:"8000"
    },
    operationConfig:{
        dataSource:"account_bank_csv_upload",
        operationType:"add"
    },
    appID:"builtinApplication",
    operation:"account_bank_csv_upload_add",
    oldValues:{
        line_d:"201411/12431234-12324",
        line_descr:"Format 9 ABSA BANK Afrigting",
        line_amount:"8000"
    },
    criteria:{
    }
}
=== 2014-02-11 16:20:34,900 [ec-5] INFO  IDACall - Performing 1 operation(s)
=== 2014-02-11 16:20:34,901 [ec-5] DEBUG DeclarativeSecurity - Processing security checks for DataSource null, field null
=== 2014-02-11 16:20:34,901 [ec-5] DEBUG DeclarativeSecurity - DataSource account_bank_csv_upload is not in the pre-checked list, processing...
=== 2014-02-11 16:20:34,906 [ec-5] DEBUG RPCManager - Content type for RPC transaction: text/plain; charset=UTF-8
=== 2014-02-11 16:20:34,906 [ec-5] DEBUG RPCManager - DMI response, dropExtraFields: true
=== 2014-02-11 16:20:34,907 [ec-5] INFO  Compression - /war/version/9.0/isomorphic/IDACall: 259 -> 213 bytes

What must I do to display a message to the user?

Announcement

How to add import CSV validation

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment