How best to map complex json for binding to multiple widgets

[jay.l.fisher]

I'd like to use the SOLR search engine with my SmartGWT app. Interaction with SOLR is through a standard HTTP GET with a variety of search options specified as query params. The results can be returned as JSON. Here's an example request URL;

http://balboa:8085/apache-solr-1.4.0/core0/select/?indent=on&q=red&wt=json&facet=true&facet.mincount=1&start=3&rows=3&facet.query=price:[*+TO+100]&facet.query=price:[100+TO+*]

The response can look like this.

{
 "responseHeader":{
  "status":0,
  "QTime":0,
  "params":{
	"facet":"true",
	"facet.mincount":"1",
	"indent":"on",
	"start":"3",
	"facet.query":["price:[* TO 100]",
	 "price:[100 TO *]"],
	"q":"red",
	"wt":"json",
	"rows":"3"}},
 "response":{"numFound":126,"start":3,"docs":[
	{
	 "name":"JUMPSUITS                     ",
	 "id":"1520004",
	 "price":[30.0,30.0,30.0,30.0],
	 "color":[
	  "Red                 ",
	  "Red                 ",
	  "Red                 ",
	  "Red                 "],
	 "sku":[
	  "1010297",
	  "1010305",
	  "1010313",
	  "1010347"],
	 "size":[
	  "No Size   ",
	  "S     3   ",
	  "S     4   ",
	  "Girls 9   "]},
	{
	 "name":"BELL PEP. ORN./RED            ",
	 "id":"413005 ",
	 "price":[4.0],
	 "color":[
	  "Red                 "],
	 "sku":[
	  "413005"],
	 "size":[
	  ".         "]},
	{
	 "name":"ORNAMENT,RED POMEGRA          ",
	 "id":"486076 ",
	 "price":[4.5],
	 "color":[
	  "Red Pomegranate     "],
	 "sku":[
	  "486076"],
	 "size":[
	  ".         "]}]
 },
 "facet_counts":{
  "facet_queries":{
	"price:[* TO 100]":123,
	"price:[100 TO *]":8},
  "facet_fields":{},
  "facet_dates":{}}}

What is the best way to map the response data so I can bind it to various grids or other UI widgets. For example, I'd like to bind "response:docs" to one grid and "facet_counts:facet_queries" to another.

Also, notice that each "record" in response:docs has a couple of normal fields (name, id) but also others that are arrays (color, sku, size). I'd like to treat the arrays like a nested datasource so that they could expand into a nested grid in the main "response:docs" grid. An example would be much appreciated.

[Isomorphic]

Since you've got a kind of "compound response", you want to use the RPCManager to directly issue an HTTP request with a query string according to whatever criteria you want. You can use JSOHelper to eval() the result, then break it out, possibly using XMLTools.selectObjects() to select parts that apply to (for XPath-like ability to query a JSON structure, eg, "/response/docs").

It looks like the objects returns are basically a list of objects expressed as parallel Arrays, with some common properties expressed as singular values. There's no particular shortcut here other than iterating through this structure and forming the Records as you want them, and it looks like you'd need to do so anyway to get rid of whitespace padding in the values.

Having finally obtained a List of Records, you're ready to call setData().

[jay.l.fisher]

Thanks for the quick response. I've tried to make the call with RPCManager, but I'm getting a transport error with HTTP code 0. Here's my code.

RPCRequest rpcReq = new RPCRequest();
rpcReq.setUseSimpleHttp(true);
rpcReq.setHttpMethod("GET");
rpcReq.setActionURL("http://balboa:8085/apache-solr-1.4.0/core0/select/?indent=on&wt=json&facet=true&facet.field=price&facet.mincount=1&rows=15&q=" 
		+ event.getForm().getItem("searchTerms").getDisplayValue());
RPCManager.sendRequest(rpcReq, new RPCCallback(){
	public void execute(RPCResponse response, Object rawData,
			RPCRequest request) {
	}
});

And this is all that shows up in the dev console for the RPC request.

{
    actionURL:"http://balboa:8085/apache-solr-1.4.0/core0/select/?indent=on&wt=json&facet=true&facet.field=price&facet.mincount=1&rows=15&q=red", 
    showPrompt:false, 
    transport:"xmlHttpRequest", 
    useSimpleHttp:true, 
    promptStyle:"dialog", 
    httpMethod:"GET"
}

I can cut and paste the URL into a browser and get the expected response so I know the url string is correct.

[Isomorphic]

That looks like a cross-site call. If it isn't, remove the http://balboa:8085 part of it. If it is a cross-site call, that's only allowed with transport:"scriptInclude" and the server is not returning a correct response for this transport (it would need to include JavaScript code to call the specified callback function).

[jay.l.fisher]

I'm using SmartGWT EE. Can I use HTTPProxy to make the call and forward the returned JSON back to the client?

[Isomorphic]

Yes, setUseHttpProxy(true) on the request.

[jay.l.fisher]

I get a transport error HTTP code 0 when I try this. Is that the right way to do it?

RPCRequest rpcReq = new RPCRequest();
rpcReq.setUseSimpleHttp(true);
rpcReq.setHttpMethod("GET");
rpcReq.setActionURL("http://balboa:8085/apache-solr-1.4.0/core0/select/?indent=on&wt=json&facet=true&facet.field=price&facet.mincount=1&rows=15&q=" 
		+ event.getForm().getItem("searchTerms").getDisplayValue());
RPCManager.setUseHttpProxy(true);
RPCManager.sendRequest(rpcReq, new RPCCallback(){
	public void execute(RPCResponse response, Object rawData,
			RPCRequest request) {
	}
});

[Isomorphic]

HTTP Code 0 is a bogus error typically only returned by IE when it's in an odd state, such as thinking that you are in "offline mode" because your connection went down.

Try another browser, and use the RPC tab of the Developer Console to verify a request was actually issued.

[jay.l.fisher]

I'm using Safari, but I also tried Firefox and get the same error. The RPC tab of the Developer Console shows this in the Decoded Request tab.

{
    actionURL:"http://balboa:8085/apache-solr-1.4.0/core0/select/?indent=on&wt=json&facet=true&facet.field=price&facet.mincount=1&rows=15&q=red", 
    showPrompt:false, 
    transport:"xmlHttpRequest", 
    useSimpleHttp:true, 
    promptStyle:"dialog", 
    httpMethod:"GET"
}

The Raw tab shows nothing for both the Request and Response.

I also tried without setting useSimpleHttp and still get HTTP Code 0.

[Isomorphic]

Are you seeing a request going to the server at all (check Firebug and the server logs). If so, what happens server side - the attempt to connect to the ultimate target is logged in detail.

If you're not seeing a request leave the browser at all, please post complete code for how you're making the request.

[jay.l.fisher]

The Firebug console shows the GET with the correct URL, but nothing shows up on the server console. The complete code is what I posted earlier.

RPCRequest rpcReq = new RPCRequest();
rpcReq.setUseSimpleHttp(true);
rpcReq.setHttpMethod("GET");
rpcReq.setActionURL("http://balboa:8085/apache-solr-1.4.0/core0/select/?indent=on&wt=json&facet=true&facet.field=price&facet.mincount=1&rows=15&q=" 
		+ event.getForm().getItem("searchTerms").getDisplayValue());
RPCManager.setUseHttpProxy(true);
RPCManager.sendRequest(rpcReq, new RPCCallback(){
	public void execute(RPCResponse response, Object rawData,
			RPCRequest request) {
	}
});

[Isomorphic]

Hi Jay,

This revealed a missing SGWT API - you'd need to be able to call the SmartClient API RPCManager.sendProxied() to cause the proxy to kick in correctly. You could access this via JSNI, but what might be a better approach is actually to go ahead and do this via a DataSource with dataFormat:"json".

In this case you would call DataSource.fetchData() and, in the callback, each Record that has been derived from the JSON response will have nested JavaScript structures accessible via the various getAttributeAs*() methods and via JSOHelper APIs. Make sense?

[jay.l.fisher]

I actually switched to using a custom servlet to be the proxy. It gives me a cleaner interface on the client side. It's working well in hosted mode, but I'm having a problem when trying to deploy it to Tomcat. Here is what I have in my web.xml

    <servlet>
        <servlet-name>SearchServlet</servlet-name>
        <servlet-class>com.islandpacific.gui.server.search.ProductSearch</servlet-class>
    </servlet>
    <servlet-mapping>
        <servlet-name>SearchServlet</servlet-name>
        <url-pattern>/sc/productSearch/*</url-pattern>
    </servlet-mapping>

In my client code I'm using RPCManager.sendRequest() but I can't seem to find the right way to construct the actionURL on my RPCRequest for Tomcat to find the servlet. In hosted mode setActionURL("/sc/productSearch?q=xyz") works fine. But when deployed to Tomcat I get a 404 error. I've also tried it without the leading /

[jay.l.fisher]

I realize this doesn't have anything to do with SmartGWT at this point, but any tips you can pass on would help. It appears that Tomcat is recognizing the mapping of url to servlet class. I wanted to change the name of the package and servlet and that gave me an opportunity to recheck the configuration. Here is the setting I have in web.xml now.

    <servlet>
        <servlet-name>ProductSearch</servlet-name>
        <servlet-class>com.islandpacific.gui.server.search.ProductSearch</servlet-class>
    </servlet>
    <servlet-mapping>
        <servlet-name>ProductSearch</servlet-name>
        <url-pattern>/sc/productSearch</url-pattern>
    </servlet-mapping>

The RPCRequest shows up in the SmartClient developer console with a URL of "sc/productSearch?q=chair" for example, yet the error says "Servlet ProductSearch is not available" so that web.xml mapping seems to be working. I've double-checked that the ProductSearch class is deployed correctly to WEB-INF/com/islandpacific/gui/server/search/ProductSearch.class. What else would cause Tomcat to say the servlet is not available?

[Isomorphic]

Look for something like a crashing static initializer, sometimes reported poorly by Java as though the class were missing rather than that it failed to load.