Announcement

Collapse
No announcement yet.
X
  • Filter
  • Time
Clear All
new posts

    Real Time Messaging using JMS questions (JBoss/Hornetq)

    SmartClient Version: v9.0p_2013-12-19/PowerEdition Deployment (built 2013-12-19)
    Firefox 17.0.9
    JBoss EAP 6.1 with Hornetq
    Using the RTM JMSMessageDispatcher

    server.properties configuration:

    Code:
    messaging.dispatcherImplementer: com.isomorphic.messaging.JMSMessageDispatcher
    messaging.jms.context: _container_
    messaging.jms.jndiPrefix: 
    messaging.jms.topicConnectionFactory = ConnectionFactory
    Note that in JNDI, the JMS ConnectionFactory is located at "java:/ConnectionFactory". This is NOT the JCA (pooled) connection factory. (I tried that, but that results in errors in JBoss about using the JCA connection factory in the wrong context, probably when the MessagingServlet tried to call javax.jms.Session.setMessageListener()).

    (BTW, what does the messaging.jms.context property do? I could not find documentation for it.)

    A JMS Queue is defined in JBoss named "/webapp/events".

    I am using RTM to send events to a SmartGWT web application. In particular, a message driven bean listening on a JMS queue for externally generated events, is converting and dispatching those events using ISCMessageDispatcher..send(). More precisely:

    Code:
    public void onMessage(Message message) {
        try {
            // Process incoming JMS message to produce an "event" to send to SmartGWT client
           Map event = ....
            ISCDispatcher.instance().send("/webapp/events", event);
        } catch(Exception e) {
            //...
        }
    }
    The SmartGWT client application is calling:

    Code:
    Messaging.subscribe("/webapp/events", ..)
    The client is receiving the messages as expected. (And I must say, RTM and Data Sources makes it possible to do some really cool stuff!)

    The first problem are the following warnings in the JBoss log file:
    Code:
    15:06:15,097 WARN  [org.hornetq.core.client] (Finalizer) HQ212016: I am closing a core ClientSession you left open. Please make sure you close all ClientSessions explicitly before letting them go out of scope! 126,844,628: java.lang.Exception
    	at org.hornetq.core.client.impl.DelegatingSession.<init>(DelegatingSession.java:91)
    	at org.hornetq.core.client.impl.ClientSessionFactoryImpl.createSessionInternal(ClientSessionFactoryImpl.java:912)
    	at org.hornetq.core.client.impl.ClientSessionFactoryImpl.createSession(ClientSessionFactoryImpl.java:317)
    	at org.hornetq.jms.client.HornetQConnection.authorize(HornetQConnection.java:648)
    	at org.hornetq.jms.client.HornetQConnectionFactory.createConnectionInternal(HornetQConnectionFactory.java:676)
    	at org.hornetq.jms.client.HornetQConnectionFactory.createTopicConnection(HornetQConnectionFactory.java:131)
    	at org.hornetq.jms.client.HornetQConnectionFactory.createTopicConnection(HornetQConnectionFactory.java:126)
    	at com.isomorphic.messaging.JMSMessageDispatcher.reconnect(JMSMessageDispatcher.java:79)
    	at com.isomorphic.messaging.JMSMessageDispatcher.ensureConnected(JMSMessageDispatcher.java:98)
    	at com.isomorphic.messaging.JMSMessageDispatcher.deliver(JMSMessageDispatcher.java:199)
    	at com.isomorphic.messaging.ISCMessageDispatcher.send(ISCMessageDispatcher.java:236)
            at xxxx.webapp.server.messaging.RequestCreatedListener.onMessage(RequestCreatedListener.java:58)
    ...
    (I've omitted the rest of the stack trace which just leads up the the call to onMessage()). Hornetq keeps track of when a Connection is obtained from the ConnectionFactory, and if that Connection is GC'd (finalize) without being closed, it you get this warning. If a lot of messages are processed in succession (say several thousand) the log is full of these warnings.

    From this it appears that ISCMessageDispatcher.instance() is creating a new instance of ISCMessageDispatcher each time, each instance opening a new JMS Connection, and that it is not closing the Connection. At first I assumed that ISCMessageDispatcher.instance() was returning a (thread safe) singleton, but these warnings suggest otherwise. So my questions are:

    - does ISCMessageDispatcher.instance() return a new instance with each invocation? If so, is that instance thread safe?
    - Does each ISCMessageDispatcher instance open a new JMS Connection using the connection factory?
    - How do you close the underlying JMS Connection that the ISCMessageDispatcher instance is using ? There does not seem to be a close() method, etc.

    My second question is regarding com.isomorphic.messaging.MessagingServlet. If I try to stop the JBoss server while there are active client sessions (active subscriptions), the server will hang, presumably because the MessagingServlet still has an open JMS Connection and Session, listening on the "/webapp/events" Queue. If I close the client sessions, then this hang does not occur. What is the recommended way of handling this?

    Thanks.

    #2
    Did you already see this wiki page showing full settings for JBoss/Hornetq? These warnings were not occurring when we used this setup procedure.

    Comment


      #3
      Thanks for the quick response. I was actually following another example on the Wiki "How to propagate data source changes to all clients by using the Real-Time Messaging Module" (example code real-time-grid-updates.zip). In that code, it uses ISCMessageDispatcher.send() to send the messages. In the Stock Quote example that you reference, the messages are sent using the JMS API - ISCMessageDispatcher.send() is not even called. That would probably explain why you did not see the messages because you are not calling ISCMessageDispatcher.send(). I will make some changes to my code to use JMS directly - in particular, using the JCA pooled ConnectionFactory as you are supposed to use inside a container, and see if that resolves the first issue. I will report back with results next week.

      Comment


        #4
        After I switched from using ISCMessageDispatcher.send() to sending the JMS messages directly (and properly closing the Connection after use) the warnings went away. You may want to check ISCMessageDispatcher, because if it is not calling javax.jms.Connection.close() then that is generally considered a bad practice (see the javadocs for Connection) and JBoss/Hornetq is going to warn you if you do not (see http://docs.jboss.org/hornetq/2.4.0....ction-ttl.html).

        I would still like to know what "messaging.jms.context" property in server.properties is used for. I cannot find any documentation on it, and I have seen a couple of different values in the examples, such as "_container_" and "messaging". Can you point me to the docs for that?

        Now the second problem is if a SmartGWT client is active and has an active subscription when JBoss is shutdown, then the following error appears in the JBoss log:
        Code:
        17:56:32,538 WARN  [org.hornetq.core.client] (Thread-23 (HornetQ-client-global-threads-61579593)) HQ212037: Connection failure has been detected: HQ119015: The connection was disconnected because of server shutdown [code=DISCONNECTED]
        The server then hangs for about one and a half minutes and then finally halts with the errors at the end of this post. If I close all client sessions, then this does not happen and JBoss shuts down cleanly. I am guessing that this is a life cycle issue with com.isomorphic.messaging.MessagingServlet, but since I don't have the source code to it, I cannot reproduce it in a way that I can ask for help from JBoss...... This is not going to fly in production as it will almost be certain that there will be users logged into the application and the sys admins are not going to like a hang and errors in the log when they shut down the app server. Suggestions?

        =========


        Code:
        17:58:26,266 ERROR [org.jboss.as.txn] (http-/0.0.0.0:8443-6) JBAS010151: Unable to get transaction state: java.lang.IllegalStateException
        	at org.jboss.msc.value.InjectedValue.getValue(InjectedValue.java:47)
        	at org.jboss.as.txn.deployment.TransactionRollbackSetupAction.checkTransactionStatus(TransactionRollbackSetupAction.java:112)
        	at org.jboss.as.txn.deployment.TransactionRollbackSetupAction.teardown(TransactionRollbackSetupAction.java:66)
        	at org.jboss.as.web.ThreadSetupBindingListener.unbind(ThreadSetupBindingListener.java:61) [jboss-as-web-7.2.0.Final-redhat-8.jar:7.2.0.Final-redhat-8]
        	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:184) [jbossweb-7.2.0.Final-redhat-1.jar:7.2.0.Final-redhat-1]
        	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:97) [jbossweb-7.2.0.Final-redhat-1.jar:7.2.0.Final-redhat-1]
        	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:102) [jbossweb-7.2.0.Final-redhat-1.jar:7.2.0.Final-redhat-1]
        	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:336) [jbossweb-7.2.0.Final-redhat-1.jar:7.2.0.Final-redhat-1]
        	at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:856) [jbossweb-7.2.0.Final-redhat-1.jar:7.2.0.Final-redhat-1]
        	at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:653) [jbossweb-7.2.0.Final-redhat-1.jar:7.2.0.Final-redhat-1]
        	at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:920) [jbossweb-7.2.0.Final-redhat-1.jar:7.2.0.Final-redhat-1]
        	at java.lang.Thread.run(Thread.java:722) [rt.jar:1.7.0_17]
        
        17:58:26,269 ERROR [org.apache.catalina.connector] (http-/0.0.0.0:8443-6) JBWEB001018: An exception or error occurred in the container during the request processing: java.lang.RuntimeException: java.lang.IllegalStateException
        	at org.jboss.as.web.ThreadSetupBindingListener.unbind(ThreadSetupBindingListener.java:67) [jboss-as-web-7.2.0.Final-redhat-8.jar:7.2.0.Final-redhat-8]
        	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:184) [jbossweb-7.2.0.Final-redhat-1.jar:7.2.0.Final-redhat-1]
        	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:97) [jbossweb-7.2.0.Final-redhat-1.jar:7.2.0.Final-redhat-1]
        	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:102) [jbossweb-7.2.0.Final-redhat-1.jar:7.2.0.Final-redhat-1]
        	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:336) [jbossweb-7.2.0.Final-redhat-1.jar:7.2.0.Final-redhat-1]
        	at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:856) [jbossweb-7.2.0.Final-redhat-1.jar:7.2.0.Final-redhat-1]
        	at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:653) [jbossweb-7.2.0.Final-redhat-1.jar:7.2.0.Final-redhat-1]
        	at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:920) [jbossweb-7.2.0.Final-redhat-1.jar:7.2.0.Final-redhat-1]
        	at java.lang.Thread.run(Thread.java:722) [rt.jar:1.7.0_17]
        Caused by: java.lang.IllegalStateException
        	at org.jboss.msc.value.InjectedValue.getValue(InjectedValue.java:47)
        	at org.jboss.as.connector.deployers.ra.processors.CachedConnectionManagerSetupProcessor$CachedConnectionManagerSetupAction.teardown(CachedConnectionManagerSetupProcessor.java:105)
        	at org.jboss.as.web.ThreadSetupBindingListener.unbind(ThreadSetupBindingListener.java:61) [jboss-as-web-7.2.0.Final-redhat-8.jar:7.2.0.Final-redhat-8]
        	... 8 more
        
        17:58:26,341 INFO  [org.apache.coyote.http11] (MSC service thread 1-14) JBWEB003075: Coyote HTTP/1.1 pausing on: http-/0.0.0.0:8443
        17:58:26,341 INFO  [org.apache.coyote.http11] (MSC service thread 1-14) JBWEB003077: Coyote HTTP/1.1 stopping on : http-/0.0.0.0:8443
        Last edited by JonFields; 9 Feb 2014, 15:11. Reason: Clarification

        Comment


          #5
          ISCMessageDispatcher is for single-JVM usage only - basically for setting up test environments without needing to have JMS. So switching to direct JMS usage or using JMSMessageDispatcher is the right approach for you.

          We'll add some details on messaging.jms.context. Basically it provides the prefix for the JNDI string we use to locate JMS support in your container.

          We'll check into the shutdown issue for JBoss - can you clarify, presumably you are not killing the entire JBoss process, you are actually doing something like a restart from the JBoss admin console?

          Comment


            #6
            Thanks. Some clarification on the JMS Dispatcher might be helpful in the docs.

            As for the shutdown, I will see if I can reproduce using one of your examples and provide code. Yes, I am initiating a graceful JBoss shutdown (e.g. through jboss-cli.sh which sends a shutdown request to the container).

            I am probably going to use RTM to initiate a graceful logout of clients anyway (display message "application is down" and then close the session), before a shutdown, but I'd still like to get the hang addressed because you cannot always count on that happening (sys admin just shuts entire server down)

            Comment


              #7
              We are trying to solve the issue with errors when the server is shutting down. We are able to reproduce the issue, and we have added some validations so that when this event occurs, the opened connections are closed automatically. Unfortunately, this is not solving this issue.
              In order to identify where the problem is, could you tell us what your shutdown code is doing?, or Could you post your shutdown code to test it?

              Comment

              Working...
              X