Realtime messaging with Apache is breaking.

[subashk]

We have implemented the Realtime messaging module in one of our applications which works fine when we access it directly through the tomcat, but whereas if we deploy the application behind apache and access through HTTPS it breaks.

The scenario we see is, repeated requests of the one below, looks like not able to keep the connection open.

https://myserver.com/myapp/isomorphic/messaging?ts=1278509744675


type=connect
connectionID=isc_HiddenFrame_9
subscribedChannels={
"Relay.ModelUpdate.ExampleModelIndex.TE_EX1001-48911900-6":{
callback:"ExampleView.processMRUpdate(data)"
}
}
jscallback=parent.isc.Comm.hiddenFrameReply(undefined,results)
_transaction=<transaction xmlns:xsi="http://www.w3.org/2000/10/XMLSchema-instance" xsi:type="xsd:Object"><type>connect</type><connectionID>isc_HiddenFrame_9</connectionID><subscribedChannels>{&#x000D; &quot;Relay.ModelUpdate.ExampleModelIndex.TE_EX1001-48911900-6&quot;:{
callback:&quot;ExampleView.processMRUpdate(data)&quot;
}
}</subscribedChannels><jscallback>parent.isc.Comm.hiddenFrameReply(undefined,results)</jscallback></transaction>
protocolVersion=1.0
__iframeTarget__=isc_HiddenFrame_9



What might cause this issue? Appreciate your response asap.

Thanks.

[Isomorphic]

That's not a lot of information to go on, but some things to try are:

1. Increase messaging.flushBufferSize to 129000 to check if buffering is the issue

2. try Tomcat's HTTPS connector to isolate whether the problem is at the Apache level

3. compare the HTTP response headers in the working and non-working cases

[subashk]

Increase messaging.flushBufferSize to 129000 to check if buffering is the issue

this worked

But how did you get this # (129000)?

What if the message is less or more than that? what will happen

Thanks.

[Isomorphic]

What's going on here is that something in between the SC Server and the browser is buffering data and refusing to actually send anything to the browser until a certain amount of data accumulates.

The size of the buffer is somewhere between the default of 8096 and the 129000 we gave you, which we picked just because it's larger than any plausible buffer. You can figure out the minimum buffer size by adding 8096 repeatedly until Messaging starts working.

Realize that what this setting does is cause the Messaging system to add extra data to messages to force the buffer to flush. This isn't very efficient, so you might want to identify what component is buffering (probably Apache) and find a setting to disable buffering or reduce the size of the buffer.

[subashk]

Even after setting to the suggested buffer size, we still see the connection gets disconnected and tries again, which resulting in too many requests in the 2 minute interval.

Is this real-time messaging ever implemented behind Apache or is there specific setting has to be modified?

Any help is greatly appreciated.

[Isomorphic]

Apache implementations are very common and should not require special settings.

We have not suggested a buffer size - do you mean 129000? That was to check if buffering was the problem. Again, try to figure out the actual buffer size and/or try to figure out what component is buffering.

[subashk]

Yes, we found out that this was the optimal setting - 89056 and Apache is buffering

# RealtimeMessaging configuration
# how often do we send keepalives to the client (ms)
messaging.keepaliveInterval: 3000
# how long the client waits after the keepaliveInterval before re-establishing
# the connection (ms)
messaging.keepaliveReestablishDelay: 1000
# how long the client waits for the connect handshake to complete before
# retrying
messaging.connectTimeout: 30000
# connection time to live - the maximum amount of time a persistent connection
# is allowed to stay open before being re-established (ms)
messaging.connectionTTL: 120000
# total response size to pad out to in order to defeat intervening
# bufferring by proxies (bytes)
messaging.flushBufferSize: 89056

But how to make apache stop buffering is what we would like to know if someone already implemented.

[subashk]

7.0RC3 Power Edition

Some of our clients are experiencing issue with Smartclient Realtime Messaging.

The connection is not successful, here is the one set of retry, it keeps trying every 4 seconds.

15:14:19.828:TMR5:DEBUG:HiddenFrame:isc_HiddenFrame_1:Destroyed isc_HiddenFrame_1
15:14:19.828:TMR5:DEBUG:Messaging:connect failed, retrying in: 4000ms
15:14:19.860:TMR6:INFO:RPCManager:Sending data via frame isc_HiddenFrame_2 to server at URL: https://www.mydomain.com/myapp/isomorphic/messaging?ts=1315422859860
15:14:19.891:TMR6:INFO:draws:isc_HiddenFrame_2$ig:draw(): drawing Canvas
Canvas.draw(_1=>undef)
Canvas.init(_1=>Obj{ID:isc_HiddenFrame_2$ig}, _2=>undef, _3=>undef, _4=>undef, _5=>undef, _6=>undef, _7=>undef, _8=>undef, _9=>undef, _10=>undef, _11=>undef, _12=>undef, _13=>undef)
Class.completeCreation(_1=>Obj{ID:isc_HiddenFrame_2$ig}, _2=>undef, _3=>undef, _4=>undef, _5=>undef, _6=>undef, _7=>undef, _8=>undef, _9=>undef, _10=>undef, _11=>undef, _12=>undef, _13=>undef)
Class.create(_1=>Obj{ID:isc_HiddenFrame_2$ig}, _2=>undef, _3=>undef, _4=>undef, _5=>undef, _6=>undef, _7=>undef, _8=>undef, _9=>undef, _10=>undef, _11=>undef, _12=>undef, _13=>undef)
HiddenFrame.sendData()
HiddenFrame.sendForm(_1=>"Submitting to:<B>https://www.mydomain...."[945], _2=>"AUTO_FORM_2", _3=>Obj{type:connect})
Comm.sendHiddenFrame(_1=>Obj)
Messaging.$z1()
callback(undefined=>undef)
"
isc.Messaging.$z1()
"
Class.fireCallback(_1=>"isc.Messaging.$z1()", _2=>undef, _3=>Array[0], _4=>Obj{length:4}, _5=>true) on [Class Timer]
Timer.$in(_1=>"$ir17")
anonymous()
"
isc.Timer.$in('$ir17')
"

15:14:19.891:TMR6:DEBUG:drawing:isc_HiddenFrame_2$ig:inserting HTML at top level
15:14:19.891:TMR6:INFO:sizing:isc_HiddenFrame_2$ig:Specified size: 1x1, drawn scroll size: 395x734, border: 0x0, margin: 0x0, reason: draw
15:14:19.907:TMR6:DEBUG:sizing:isc_HiddenFrame_2$ig:handle sizes: {scrollLeft: 0,
scrollTop: 0,
scrollWidth: 395,
scrollHeight: 734,
clientWidth: undef,
clientHeight: undef,
offsetWidth: 395,
offsetHeight: 734,
styleLeft: "0px",
styleTop: "-9999px",
styleWidth: "1px",
styleHeight: "1px",
styleClip: ""}
15:14:19.907:TMR6:INFO:HiddenFrame:isc_HiddenFrame_2:Form submitted to server


******** server.properties ********

# RealtimeMessaging configuration
# how often do we send keepalives to the client (ms)
messaging.keepaliveInterval: 30000
# how long the client waits after the keepaliveInterval before re-establishing
# the connection (ms)
messaging.keepaliveReestablishDelay: 30000
# how long the client waits for the connect handshake to complete before
# retrying
messaging.connectTimeout: 20000
# connection time to live - the maximum amount of time a persistent connection
# is allowed to stay open before being re-established (ms)
messaging.connectionTTL: 120000
# total response size to pad out to in order to defeat intervening
# bufferring by proxies (bytes)
messaging.flushBufferSize:4096

What does the above error stack says, this is happening only to few users.

Thanks.

[Isomorphic]

There's no error in those logs, just INFO level logging that you seem to have enabled manually. You should troubleshoot this like any other failure to connect - is the user's network working in general, are any of your servers either down or returning errors, etc.

[subashk]

All the servers are up and we see no errors.

Rest of the pages/content are loading from the same server works fine.

How to debug this more? anything specific we should look for in the SC Console.

[Isomorphic]

It's not a SmartClient-level error. The browser is failing to connect. Again, you should troubleshoot this like any network connection failure - if you've just checked *now* that the server is up, that doesn't mean that it wasn't intermittently returning errors whenever this user experienced this problem. So use standard network diagnostics techniques to figure this out.

[subashk]

We see from the logs that the request to open the realtime message connection is processed by the server but we guess because the response didnt reach the client within 4 seconds it tries to reconnect due to slow network.

How to override this 4 second property to waiting for a response in server.properties?

[Isomorphic]

If it takes more than 4 seconds to establish a connection it's not clear what good real-time messaging is doing you!

However this timeout is controlled by Messaging.connectTimeout and is time in millseconds. Change it like so:

// 10 seconds
isc.Message.addClassProperties({ connectTimeout:10000 })

[ewilliams]

We had found a similar issue with our app when switching to https. Found the problem to be that the https Connector in server.xml was configured to use compression. I verified that even on non-https connections ( i.e. http ) if "compression=on" was defined in the Connector element of the server.xml the real-time messaging didn't work.

You mentioned only some of your clients.. possibly only the ones using https? and your https Connector is defined to use compression while non secure connections don't?

[bbruyn]

We're seeing a problem similar to the OP's, only we don't see any disconnect - we just never seem to receive a response. We have one server that sits behind a lighttpd instance. Go directly to Tomcat on 8080 and everything works. Try to go through 80 and it does not.

We see the same behavior on another environment where we have an F5 in front. Its buffer size is set at 4096. Set the flushBuffer property explicitly to 8192 as directed but no change.

The isc console doesn't seem to offer any clues - all you see is that it keeps opening new connections. I'm sure this is our problem and not Isomorphic's, but I was hoping maybe you'd seen something like this before?