Nested XML structures and databinding

[Isomorphic]

Posted on behalf of tgochenour:
---------------
Now I'm running up against the wall regarding what to do with XML that contain nested collections, as I don't see how to model this with the DataSource. Unlike the table fetches, which are flat, the edit fetch contains a more complex structure. When rendering a view of the data with the HTMLFlow pane, I can't iterate through these collections. I haven't yet tried to render a edit form with nested tables to see if I can maintain the XML structure. I'll try this next.

[Isomorphic]

Hi tgochenour,

You can declare schema for nested structures by just declaring a DataSource for the substructures and declaring a field with type:"subStructureDSID". This will result in a nested record structure in your DSResponse, which you could provide to a ListGrid via setData().

I'm unclear on what you're up to with the HTMLFlow. Are you doing some custom HTML rendering of the data you've loaded?

[tgochenour]

Hmmmm, no reference to subStructureDSID in the Quick Start Guide or the Developer's reference. I just read all about WSDL support but don't see an example of nested collections utilized here. Every example has simple flat data.

I also just read the docs on loadXMLSchema(). I think what I'll do next is build a test schema with nested data, and load it to see what gets emitted by the system.

Any examples you want to send my way would be appreciated.

Regarding HTMLFlow: One of my use cases has the data formatted as a document, so I thought perhaps I could use HTMLFlow to render the document, using the ${} tag replacements. This document has a collection of photos, so I would need to iterate through this collection to render that portion of the document.

[Isomorphic]

1) nested structures and schema: Sorry I was unclear, "subStructureDSID" was intended to mean the ID of another DataSource. So:

isc.DataSource.create({
    ID:"Person", 
    fields : [
        { name: "address", type:"Address" },
        ...
    ]
});

isc.DataSource.create({
    ID:"Address",
    fields : [ ... ]
})

If you use loadXMLSchema() you'll find it generates DataSources that have relationships set up in this way. There is an loadXMLSchema here although it does not show nested structures as above.

2) HTMLFlow / HTML templating

HTMLFlow is a good approach for that if the presentation you want to render doesn't look anything like a DynamicForm or DetailViewer (even suitably skinned). Since you can put any JavaScript inside the ${} braces, so for the collection of photos, you could call a function to generate that portion.

To minimize the amount of non-declarative code, you could use a second, undrawn HTMLFlow that has it's contents set to an HTML template to be used for each photo, and call getContents() to have the template evaluated repeatedly.

[tgochenour]

The link doesn't work (http://wrath:16011) but yeah, I saw that example from http://www.smartclient.com/#_Data.Integration_XML_XML.Schema.Import and it's what gave me the idea about testing out loadXMLSchema().

I generated this XSD of an event document with nested "contact" elements. The dynamicForm that was generated displays the contact field as a generic text box. I suppose its too much to ask for an editable table to be presented in this case.

<?xml version="1.0"?>
<xs:schema attributeFormDefault="unqualified" elementFormDefault="qualified" xmlns:xs="http://www.w3.org/2001/XMLSchema">
  <xs:element name="events">
    <xs:complexType>
      <xs:sequence>
        <xs:element maxOccurs="unbounded" name="event">
          <xs:complexType>
            <xs:sequence>
              <xs:element name="id" type="xs:string" />
              <xs:element name="eventDescTxt" type="xs:string" />
              <xs:element name="severity">
                <xs:complexType>
                  <xs:simpleContent>
                    <xs:extension base="xs:string">
                      <xs:attribute name="id" type="xs:unsignedByte" use="required" />
                    </xs:extension>
                  </xs:simpleContent>
                </xs:complexType>
              </xs:element>
              <xs:element name="startDate" type="xs:date" />
              <xs:element name="endDate" type="xs:date" />
              <xs:element name="restrictions" type="xs:string" />
              <xs:element name="status">
                <xs:complexType>
                  <xs:simpleContent>
                    <xs:extension base="xs:string">
                      <xs:attribute name="id" type="xs:unsignedByte" use="required" />
                    </xs:extension>
                  </xs:simpleContent>
                </xs:complexType>
              </xs:element>
              <xs:element minOccurs="0" maxOccurs="unbounded" name="contact">
                <xs:complexType>
                  <xs:sequence>
                    <xs:element name="id" type="xs:unsignedByte" />
                    <xs:element name="first_name" type="xs:string" />
                    <xs:element name="last_name" type="xs:string" />
                    <xs:element name="phone" type="xs:string" />
                    <xs:element name="email" type="xs:string" />
                  </xs:sequence>
                </xs:complexType>
              </xs:element>
            </xs:sequence>
          </xs:complexType>
        </xs:element>
      </xs:sequence>
    </xs:complexType>
  </xs:element>
</xs:schema>

[tgochenour]

I'm stuck

I'm not sure I can render complex XML data models using SmartClient forms.

The use case is such: I'm receiving an XML document from the server. I need to edit the document and send it back. The document has nested collections of subelements. I'd like to automate (generate) the presentation based on the structure of the XML document itself. I'd like to then modify (decorate) this default layout by adding labels and such.

With SmartClient, I've been able to instantiate the two DataSources (event and contact) found in an XSchema definition with this code

isc.XMLTools.loadXMLSchema("/lite/pages/projects.xsd", function(schemaset)
{
	var ds;
	ds = schemaset.getSchema("event");

});

I've instantiated a dynamicForm with this code

pane =  isc.DynamicForm.create({autoFocus: true, dataSource: "event", showEdges:true, padding:8, backgroundColor:"white" });

The result is a form listing 8 fields, the last field, "contact", should be interpreted as a nested collection. I'd like an editable grid to be displayed as a default in this case. I'd then like a way to extract and save this definition so I can modify titles and such.

Finally, I'd like for the DynamicForm and it's nested grids to update a shared valuesManager object so that I can xmlSerialize() the valuesManager and recreate the XML document with the nested elements in place.

I could develop this myself within the SmartClient architecture if only I had the JS client code that wasn't obfuscated. Does the enterprise license include readable source code?

Really what I want is an XForms processing engine. It's too bad none of them are mature enough yet.

I have yet to find an Ajax framework that handles editing hierarchical XML data easily. All the form implementations assume flat table structures.

SmartClient has the best Grid implementation I've found so far. If only I could solve this editing requirement I'd be good to go.

[tgochenour]

fetchData

I'm using a debugger to inspect the return value from fetchData of this XML file:

?xml version="1.0" encoding="utf-8"?>
<event>
  <id>PW123456</id>
  <eventDescTxt>I70-228-Between empire and eisenhower tunnel</eventDescTxt>
  <severity id="1">Minimal</severity>
  <startDate>2006-11-14</startDate>
  <endDate>2006-11-15</endDate>
  <restrictions>The work desription goes here</restrictions>
  <status id="1">new</status>
  <contact>
    <id>1</id>
    <first_name>Todd</first_name>
    <last_name>Gochenour</last_name>
    <phone>555-555-5555</phone>
    <email>gochenourt@cdot.gov</email>
  </contact>
  <contact>
   <id>2</id>
    <first_name>James</first_name>
    <last_name>Gosling</last_name>
    <phone>444-444-4444</phone>
    <email>gosling@sun.com</email>  
  </contact>
  <contact>
    <id>3</id>
    <first_name>Jack</first_name>
    <last_name>Spratt</last_name>
    <phone>333-333-3333</phone>
    <email>jack@spratt.com</email>
  </contact>
</event>

and the data[0] object has the "event" fields parsed OK. It has a "contact" array with three items. These three items are strings "\n ".

The DataSource is defined as:

isc.XMLTools.loadXMLSchema("/lite/pages/event.xsd", function(schemaset)
{
	var ds;
	ds = schemaset.getSchema("event");
	ds.dataURL = "/lite/pages/event.xml";
});

The event.xsd file is:

<?xml version="1.0"?>
<xs:schema attributeFormDefault="unqualified" elementFormDefault="qualified" xmlns:xs="http://www.w3.org/2001/XMLSchema">
        <xs:element name="event">
          <xs:complexType>
            <xs:sequence>
              <xs:element name="id" type="xs:string" />
              <xs:element name="eventDescTxt" type="xs:string" />
              <xs:element name="severity">
                <xs:complexType>
                  <xs:simpleContent>
                    <xs:extension base="xs:string">
                      <xs:attribute name="id" type="xs:unsignedByte" use="required" />
                    </xs:extension>
                  </xs:simpleContent>
                </xs:complexType>
              </xs:element>
              <xs:element name="startDate" type="xs:date" />
              <xs:element name="endDate" type="xs:date" />
              <xs:element name="restrictions" type="xs:string" />
              <xs:element name="status">
                <xs:complexType>
                  <xs:simpleContent>
                    <xs:extension base="xs:string">
                      <xs:attribute name="id" type="xs:unsignedByte" use="required" />
                    </xs:extension>
                  </xs:simpleContent>
                </xs:complexType>
              </xs:element>
              <xs:element minOccurs="0" maxOccurs="unbounded" name="contact">
                <xs:complexType>
                  <xs:sequence>
                    <xs:element name="id" type="xs:unsignedByte" />
                    <xs:element name="first_name" type="xs:string" />
                    <xs:element name="last_name" type="xs:string" />
                    <xs:element name="phone" type="xs:string" />
                    <xs:element name="email" type="xs:string" />
                  </xs:sequence>
                </xs:complexType>
              </xs:element>
            </xs:sequence>
          </xs:complexType>
        </xs:element>
</xs:schema>

The fetch logic is:

     DataSource.getDataSource("event").fetchData({id:"PW123456"},
    function(data)
    {
         isc.Log.logDebug(data);
    });

So there's a "contact" DataSource created by the loadXMLSchema(). When I load this sample XML file, what happens to my contact data?

[Isomorphic]

Hi tgochenour,

This is straightforward to develop with SmartClient. Bind a DynamicForm to the "event" DataSource, as you have, then bind an editable ListGrid to the "contact" DataSource.

Compose them together with a VLayout and they will appear as one logical form to the user. If you wish to label the grid "contacts", you could combine an isc.Label with the ListGrid in a nested HLayout, or, for an alternative presentation, use a SectionStack around both DynamicForm and ListGrid.

To load data, call dataSource.fetchData() on the "event" DataSource, provide the single row of dsRequest.data as values to the form via dynamicForm.setValues(). Within this single row, under the property "contact", you'll find the multiple rows of existing contact data, which you can provide to the grid via ListGrid.setData().

On saving, grab the data-as-saved from the grid via getData(), and combine it with form.getValues(), then call xmlSerialize() on the "event" DataSource with this data.

We are moving in the direction of obviating this glue code by offering higher-level compound editing components (eg take a look at Examples > Experimental > Relational Editing in the SDK).

On your other questions/comments:
1) XForms: even with the glue code required above, SmartClient provides much more automation of XML editing than is currently contemplated in XForms - correct me if I'm wrong.

2) Source code: enterprise licenses of SmartClient don't include source code except by special arrangement at the very largest scales. However, I think you'll agree source code is the last thing that would help with the above problem, just a deeper understanding of the available APIs and perhaps more documentation targetted at your use case.

[tgochenour]

Source Code

When my debugger breaks on an error and reports "_52 has no properties",
how exactly am I suppose to debug this?

It is because there is no documentation describing how to do what I'm trying to do that source code is necessary. I'd ask a lot less questions if I had a way of discovering the truth for myself.

What XForms provides is preservation of the XML document as the data model. It doesn't convert the model to JSON or DataSource objects or anything else. That XForms implementations are slow as molasses is a problem.

I've done what you suggested.

I have this code:

isc.XMLTools.loadXMLSchema("/lite/pages/event.xsd", function(schemaset)
{
	var ds;
	ds = schemaset.getSchema("event");
	ds.dataURL = "/lite/pages/event.xml";
});
var pane =  isc.DynamicForm.create({ID:"pane", autoFocus: true, dataSource: "event" });
var contactsTable = isc.ListGrid.create({ID:"contactsTable", dataSource: "contact" } );
var vlayout = isc.VLayout.create({members:[pane,contactsTable], showEdges:true, padding:8, backgroundColor:"white" });
DataSource.getDataSource("event").fetchData({ID:"PW123456"},function(data)
{
	pane.editRecord(data.data[0]);
	contactsTable.setData(data.data[0].contact);
});

The result is a dynamicForm with appropriate content and a grid with appropriate column labels and three rows of blank data.

The debugger tells me the contact array in data[0] has three elements, all strings with "\n ".

[Isomorphic]

This is a bug, now fixed. You'll need to apply this patch: http://forums.smartclient.com/showthread.php?t=91

You're probably going to run into a further problem with what appears to be an enumeration declaration for the "severity" and "status" fields in your xsd. SmartClient will not create valueMaps for this type of definition, but it will if you use a restriction as shown in this example: http://www.smartclient.com/#_Data.In....Schema.Import (see the supplyItem.xsd tab, definition of the "units" field).

Unless you have something generating these schemas already, the XML schema approach to databinding may not be buying you much here - have you considered using the our DataSource format directly?

With regards to debugging: we have a whole topic on it here: http://www.smartclient.com/docs/5.5....oup..debugging. One of the things mentioned there is that we produce stack traces to the location of the problem (IE only).

More generally though, diving into the source is almost always the slowest possible approach, even for expert developers, simply because SmartClient is a very large codebase, with lots of browser-specific workarounds and context to absorb. And even if for a given problem you can quickly find the issue in the vendor code, you're still going to want to simply report it to them because changing their code means you take on the burden of applying that patch to subsequent releases - this is true even of open source projects, unless you happen to have commit rights to their codebase.

Also, everything we've provided so far is publicly documented. We're always working on putting together additional examples that'll make it easy to start with a working skeleton and I understand that in some cases without that it's hard to find the relevant information. That's what these forums are for :)

[tgochenour]

It works.

That did the trick. Thanks for your patience with me.

I'm using Microsoft's XSD Inference tool to generate the XSD files. I guess I could build an XSL template to generate DataSource files instead. I would prefer to adhere to standards.

Regarding debugging. I read the topic you referenced. It primarily covers logging, this is a technique I'm already familiar with. It recommends the use of a javascript debugger, I already use Venkman with FF and Microsoft's Script Editor for IE. I also now use your admin console, which I find very useful. I am doing everything outlined in this document.

I would never initiate my own patch on vendor software. This isn't what I'm trying to do. I can see the possibility of subclassing ISC objects and in this case would want to know what is being overridden to make sure I'm not missing anything.

But the real issue is understanding problems when they arise. When the error reads "_52 has no properties" how am I to debug this? (Turns out I used "contacts" for the grid's datasource, not "contact"). When I inspect an object and all I see are mangled attributes, how am I to understand what I am looking at to determine how best to implement my code?

Whether coding open source or Java or whatnot, I've found answers quicker by reading/tracing the code rather than parsing the prose of documentation. Documentation is always out of date and incomplete.

I understand the issue of intellectual property and preserving IP value. To release the source would requires a bit of trust that customers won't abuse their licensing agreement.

I like SmartClient very much. Its fast and clean. I appreciate your quick response to issues posted here. My only problem is spending thousands of dollars for a framework that doesn't provide source code. Perhaps I need to just get over it.

[Isomorphic]

Glad to hear things are working for you.

I understand the frustration with lack of code. I think most capable people get annoyed when they want to get to the root of the problem, but can't. What we try to do is provide enough documentation and debugging tools that you can easily determine when the problem is in your code vs SmartClient doing something wrong and to simply throw the problem over the fence to us in the latter case.

I agree that there is a class of bugs that have to do more with configuration of a component and tracing the code path can make you realize that you were e.g. misusing the APIs. That's actually why I mentioned the IE stack trace facility - because even though you will see local variables obfuscated to e.g. _52, you'll still see the entire call stack, complete with local variable values that were passed into each function, e.g:

    ListGrid.draw(_1 => "foo")
    Canvas.init()

I just made that up, but you can see how the values do give some context on what's going on.

Subclassing is definitely supported and in fact encouraged. We actually do flag methods that are overridden from the base class in the documentation to let you know that if you don't call the superclass implementation you may miss out on some functionality.

Thanks for the kudos on the product, we really appreciate it. The primary problem with source licenses is that if the code leaks, the cat's out of the bag and it's likely to be difficult to figure out who did it and even harder to collect any kind of meaningful damages.

We intend to keep answering your questions quickly and competently - and if you have any suggestions on how we can improve our debugging facilities to make it easier for you to determine when to throw a problem over the fence to us - short of exposing the code - we'd love to hear it.

Thanks!

[tgochenour]

OK, Over the fence

So what do I do with these log messages?

10:05:33.140:INFO:Log:initialized
10:05:36.375:INFO:Log:isc.Page is loaded
10:05:36.562:WARN:parseXML:Error parsing XML: XML Parsing Error: no element found
Location: http://localhost:8080/lite/pages/workspace.jsp
Line Number 1, Column 1:
^
10:05:36.578:WARN:Log:TypeError: _5 has no properties

[Isomorphic]

That's odd - typically we report the offending line as well. If you do this in IE, do you get a stack trace? For example, here's a Firefox error on an invalid XML document (from your previous example):

10:32:11.997:WARN:parseXML:Error parsing XML: XML Parsing Error: mismatched tag. Expected: </event>.
Location: http://localhost:8080/foo.isc
Line Number 5, Column 28:  severity id="1">Minimal</severity>
---------------------------^
10:32:12.013:WARN:Log:TypeError: data.data[0] has no properties

And here's the corresponding IE error report:

10:31:45.700:WARN:parseXML:Error parsing XML: 
Reason: End tag 'severity' does not match the start tag 'event'.
Line number: 5, character: 28
Line contents:   severity id="1">Minimal</severity>
10:31:45.716:WARN:Log:Error:
	''null' is null or not an object'
	in http://localhost:8080/foo.isc
	at line 67
    XMLTools.$343(Obj, "<?xml version="1.0" encoding="utf-8"?>\n<..."[882], Obj, Obj, Obj, Obj, Obj, Obj, undef)

You might want to change the RPCManager logging threshold to DEBUG to see what the last data returned by the server was as well - you can do this using the "Logging Preferences" pulldown in the "Results" tab of the Developer Console.

If there's any other context you can provide, that should help us get to the bottom of this. Thanks!

[tgochenour]

RPCManager Log set to 'debug'

Nice! In the 1 week I've had this product under review I haven't played with the log settings yet. This is good to know.

It appears the issue is with fetching the event.xsd file. The response returns immediately and empty, but then the schema file is received later in the stream.

13:27:39.094:INFO:RPCManager:sendQueue[3]: 1 RPCRequest(s); transport: xmlHttpRequest; target: /lite/pages/event.xsd
13:27:39.375:DEBUG:RPCManager:XMLHttpRequest GET from /lite/pages/event.xsd with fields: {} full URL string: /lite/pages/event.xsd
13:27:39.656:DEBUG:RPCManager:Result string: ""
13:27:39.953:INFO:RPCManager:rpcResponse(unstructured) results -->""<--
13:27:40.234:WARN:parseXML:Error parsing XML: XML Parsing Error: no element found
Location: http://localhost:8080/lite/pages/workspace.jsp
Line Number 1, Column 1:
^
13:27:40.515:WARN:Log:TypeError: _5 has no properties
13:27:40.797:INFO:RPCManager:transaction 3 arrived after 1422ms

...and a little later...

13:27:42.859:DEBUG:RPCManager:Result string: "<?xml version="1.0"?>
<xs:schema attributeFormDefault="unqualified" elementFormDefault="qualified" xmlns:xs="http://www.w3.org/2001/XMLSchema">
  <xs:element name="event">
    <xs:complexType>
      <xs:sequence>
        <xs:element name="eventDescTxt" type="xs:string" />
        <xs:element name="severity">
          <xs:complexType>
...

And now that I look at it posted here, I no longer think the event.xsd file is related. It's complaining about something in the workspace.jsp. It's contents are:

<isomorphic:loadDS ID="projects"/>
<isomorphic:XML>
<%@include file="workspace.xml" %>
</isomorphic:XML>

The workspace.xml file loads and parses OK. It's not valid XML, by the way, as it has more than one root element.

And if I keep refreshing, I get different results. This time:

13:44:16.122:WARN:parseXML:Error parsing XML: 
Reason: XML document must have a top level element.
Line number: 0, character: 0
Line contents: 
13:44:16.387:WARN:Log:Error:
	''null' is null or not an object'
	in http://localhost:8080/lite/pages/workspace.jsp
	at line 66
    XMLTools.$343(Obj, "", Obj, Obj, Obj, Obj, Obj, Obj, undef)
    Class.fireCallback(Obj, "operationResult,data,context,transaction..."[95], Array[9]) on [Class RPCManager]
    RPCManager.fireReplyCallback(Obj, Obj, Obj, "")
    RPCManager.fireReplyCallbacks(Obj, Obj)
    RPCManager.performOperationReply()
    RPCManager.$36v(1)
    Class.fireCallback(Obj, undef, Array[1], [Class RPCManager], true) on [Class Timer]
    Timer.$ib("$if11")
    anonymous()
        "isc.Timer.$ib('$if11')"