Announcement

**claudiobosticco** · 27 Oct 2017, 07:53

SmartClient Version: v11.1p_2017-10-18/Enterprise Deployment (built 2017-10-18)

Hello, any news on this issue? Any hints or something I may try?

Actually I've got a little update: I've migrated to a more recent version of activeMQ broker (standalone install), 5.14.5 version.
Also I'm using the same version of activemq-all.jar in the two tomcat instances of the cluster.
Also I've updated the configuration to use org.apache.activemq.pool.PooledConnectionFactory instead of org.apache.activemq.ActiveMQConnectionFactory.
I've verified that realtime messaging in my application is working.

When I put ?isc_remoteDebug=true and press <enter> in one browser I see this in the logs of the node which serves its requests:

Code:

2017-10-27 17:41:10,721 DEBUG [claudio.bosticco@juventus.com 797] JMSMessageDispatcher Received message from JMS (Topic) 
2017-10-27 17:41:10,722 DEBUG [claudio.bosticco@juventus.com 797] JMSMessageDispatcher Decoded jmsMessage for channel: isc_DebugMaster data: {payload={methodName=targetAvailable, args=[{GUID=700F1FCC-EE04-409F-8051-1325EF684567, receiveChannel=700F1FCC-EE04-409F-8051-1325EF684567, discoverableOnChannel=isc_DebugTarget, generatedDate=Fri Oct 27 17:41:10 CEST 2017, userAgent=Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/62.0.3202.75 Safari/537.36, platform=MacIntel, vendor=Google Inc., visibleMethods=[], documentTitle=Gestione Terzo Livello Palchi, URL=https://custom-apps-dev.juventus.com/Legend/?isc_remoteDebug=true, formFactor=Desktop, browserVersion=62, browserMinorVersion=62}]}, sequence=1, originChannel=700F1FCC-EE04-409F-8051-1325EF684567, expectsReply=false} 
2017-10-27 17:41:10,722 DEBUG [claudio.bosticco@juventus.com 797] ISCMessageDispatcher sending message to channel: isc_DebugMaster

Then, when I open the dev console, I see this in the logs of the other node:

Code:

2017-10-27 17:41:22,604 DEBUG [claudio.bosticco@juventus.com 797] JMSMessageDispatcher Received message from JMS (Topic) 
2017-10-27 17:41:22,604 DEBUG [claudio.bosticco@juventus.com 797] JMSMessageDispatcher Decoded jmsMessage for channel: isc_DebugTarget data: {payload={methodName=discover, timeout=null, args=[]}, sequence=1, originChannel=BADAD207-8F81-454B-8ECC-2F83B77FC7C6, expectsReply=true} 
2017-10-27 17:41:22,604 DEBUG [claudio.bosticco@juventus.com 797] ISCMessageDispatcher sending message to channel: isc_DebugTarget

But still, when I open the 'remote' select, it's empty.

To me the logs seem correct, or am I missing something?

**Isomorphic** · 27 Oct 2017, 14:34

Apologies for delay - we have been working on fixing this. This actually works in our dev environment, but does appear to be broken in the SDK...tracking this down - it is definitely a supported configuration that should work.

**Isomorphic** · 27 Oct 2017, 15:01

Possibly this is the problem?: http://activemq.apache.org/objectmessage.html

I'm seeing this in my logs:

Code:

javax.jms.JMSException: Failed to build body from content. Serializable class not available to broker. Reason: java.lang.ClassNotFoundException: Forbidden class org.apache.commons.collections4.map.LinkedMap! This class is not trusted to be serialized as ObjectMessage payload. Please take a look at http://activemq.apache.org/objectmessage.html for more information on how to configure trusted classes.
    at org.apache.activemq.util.JMSExceptionSupport.create(JMSExceptionSupport.java:36)
    at org.apache.activemq.command.ActiveMQObjectMessage.getObject(ActiveMQObjectMessage.java:208)
    at com.isomorphic.messaging.JMSMessageDispatcher.iscMessage(JMSMessageDispatcher.java:463)
    at com.isomorphic.messaging.JMSMessageDispatcher$2.onMessage(JMSMessageDispatcher.java:271)
    at org.apache.activemq.ActiveMQMessageConsumer.dispatch(ActiveMQMessageConsumer.java:1401)
    at org.apache.activemq.ActiveMQSessionExecutor.dispatch(ActiveMQSessionExecutor.java:131)
    at org.apache.activemq.ActiveMQSessionExecutor.iterate(ActiveMQSessionExecutor.java:202)
    at org.apache.activemq.thread.PooledTaskRunner.runTask(PooledTaskRunner.java:133)
    at org.apache.activemq.thread.PooledTaskRunner$1.run(PooledTaskRunner.java:48)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1167)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641)
    at java.base/java.lang.Thread.run(Thread.java:844)
Caused by: java.lang.ClassNotFoundException: Forbidden class org.apache.commons.collections4.map.LinkedMap! This class is not trusted to be serialized as ObjectMessage payload. Please take a look at http://activemq.apache.org/objectmessage.html for more information on how to configure trusted classes.
    at org.apache.activemq.util.ClassLoadingAwareObjectInputStream.checkSecurity(ClassLoadingAwareObjectInputStream.java:112)
    at org.apache.activemq.util.ClassLoadingAwareObjectInputStream.resolveClass(ClassLoadingAwareObjectInputStream.java:57)
    at java.base/java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1850)
    at java.base/java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1737)
    at java.base/java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2024)
    at java.base/java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1559)
    at java.base/java.io.ObjectInputStream.readObject(ObjectInputStream.java:423)
    at org.apache.activemq.command.ActiveMQObjectMessage.getObject(ActiveMQObjectMessage.java:206)
    ... 10 more

**claudiobosticco** · 27 Oct 2017, 23:29

after your other suggestion https://forums.smartclient.com/forum...031#post250031
now I'm starting tomcat instances with this option:
-Dorg.apache.activemq.SERIALIZABLE_PACKAGES=*
but the remote debug behaviour hasn't changed

**claudiobosticco** · 30 Oct 2017, 02:56

Hello, I've also just noticed that if I open the developer console from the browser page where I've enabled remote debugging (with isc_remoteDebug=true), then if (in this same developer console) I open the 'remote' select, here I see listed the same browser from which I've opened the developer console. I don't know if it's normal...

When I over the listed browser, the developer console starts logging:

Code:

11:45:07.262:DEBUG:Messaging:keepalive on conn: isc_HiddenFrame_2
11:45:10.873:DEBUG:Messaging:ignoring duplicate messageID: ID:srvubuntu8-32793-1509118509641-1:3:6:1:2
11:45:10.878:DEBUG:Messaging:ignoring duplicate messageID: ID:srvubuntu8-32793-1509118509641-1:3:6:1:3
11:45:10.880:DEBUG:Messaging:ignoring duplicate messageID: ID:srvubuntu8-32793-1509118509641-1:3:6:1:4
11:45:10.881:DEBUG:Messaging:ignoring duplicate messageID: ID:srvubuntu8-32793-1509118509641-1:3:6:1:5
11:45:10.884:DEBUG:Messaging:ignoring duplicate messageID: ID:srvubuntu8-32793-1509118509641-1:3:6:1:6
11:45:10.886:DEBUG:Messaging:ignoring duplicate messageID: ID:srvubuntu8-32793-1509118509641-1:3:6:1:7
11:45:10.892:DEBUG:Messaging:ignoring duplicate messageID: ID:srvubuntu8-32793-1509118509641-1:3:6:1:8
11:45:11.876:DEBUG:Messaging:ignoring duplicate messageID: ID:srvubuntu8-32793-1509118509641-1:3:6:1:9
11:45:11.881:DEBUG:Messaging:ignoring duplicate messageID: ID:srvubuntu8-32793-1509118509641-1:3:6:1:10

I don't know if this helps...

**Isomorphic** · 31 Oct 2017, 06:26

Ok, I'm quite sure now that the serlialization "security" feature is the actual problem. In order to fix the problem, you not only need to start Tomcat, but also the broker with this option:

Code:

-Dorg.apache.activemq.SERIALIZABLE_PACKAGES=*

I'm going to see if we can remove this constraint by eliminating the use of the non-jvm-native org.apache.commons.collections4.map.LinkedMap, but this will require some testing to make sure we don't break anything else. We will also improve the JMS logging and error handling a bit to make it easier to track this sort of thing down in the future.

As an aside, the RabbitMQ works fine out of the box. And ActiveMQ may not have a future: http://activemq.apache.org/new-features-in-60.html.

**Isomorphic** · 1 Nov 2017, 07:34

On further testing, I take it back - the special serialization flag only needs to be enabled on the Tomcat JVM. Can you please double check that that option is definitely enabled in your JVM invocation?

On a related note, we will release an update to at least 11.1 in the next few days to enable better debugging and possibly include a workaround to requiring this flag for ActiveMQ - depending on feasibility and testing outcomes.

**claudiobosticco** · 3 Nov 2017, 08:41

Hello, yes, both my tomcat instances are started with that option (see the text in bold):

Code:

tomcat@srvubuntu8:~$ ps aux | grep java

tomcat   22744 37.2 44.9 6206144 1819608 ?     Sl   17:06   8:37 /usr/bin/java -Djava.util.logging.config.file=/opt/tomcat/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djava.awt.headless=true -Djava.security.egd=file:/dev/./urandom -Djdk.tls.ephemeralDHKeySize=2048 -Djava.protocol.handler.pkgs=org.apache.catalina.webresources -Xms512M -Xmx3072M -server -XX:+UseParallelGC -XX:PermSize=512m -XX:MaxPermSize=512m -Xdebug -Xrunjdwp:transport=dt_socket,address=8000,server=y,suspend=n [B]-Dorg.apache.activemq.SERIALIZABLE_PACKAGES=*[/B] -classpath /opt/tomcat/bin/bootstrap.jar:/opt/tomcat/bin/tomcat-juli.jar -Dcatalina.base=/opt/tomcat -Dcatalina.home=/opt/tomcat -Djava.io.tmpdir=/opt/tomcat/temp org.apache.catalina.startup.Bootstrap start

But thanks for the heads up about activeMQ. Actually I don't want to set up a new environment with a jms broker which has no future.

So, I've installed RabbitMQ (and disabled activeMQ). After putting these jars in the tomcat /lib:

amp-client-4.3.0.jar
rabbitmq-jms-1.7.0.jar
geronimo-jms_1.1_spec-1.1.1.jar
slf4j-api-1.6.1.jar

and configuring the ConnectionFactory like this:

Code:

<Resource name="jms/ConnectionFactory"
                     type="javax.jms.ConnectionFactory"
                     factory="com.rabbitmq.jms.admin.RMQObjectFactory"
                    username="foo"
                    password="bar"
                    virtualHost="/"
                    host="myBrokerHost"/>

and my channel:

Code:

    <Resource  name="jms/updateOrdineChannel"
                   type="javax.jms.Queue"
                  factory="com.rabbitmq.jms.admin.RMQObjectFactory"
                  destinationName="updateOrdineChannel"
                  amqp="true"
                  amqpQueueName="updateOrdineChannel"/>

I've got a situation where the messaging and the remote debug seem to work but only when the browsers are served by the same tomcat instance.
Also, note that the messaging on the updateOrdineChannel works even without the above channel definition. Is this expected?

I'm still using v11.1p_2017-10-18 build. Have you already released a build with better debugging? I hope will help me to configure everything correctly.

EDIT: see post #26 for working channel config.

**Isomorphic** · 3 Nov 2017, 09:12

Yeah, We are reproducing this issue here as well. There is in fact a problem with the way the messages our routed. There will be a fix to this issue as well as the duplicate message deliver to the browser (which you mentioned earlier). The dups are weeded out by some defensive logic on the client, but it's inefficient. There are a few other inefficiencies that are being addressed as part of this work as well that should substantially speed up JMS configurations with many clients.

Will update this once we have a definitive build to point you to, but work is almost complete. Expect it in the next couple of days.

**Isomorphic** · 3 Nov 2017, 09:13

Just to be clear, the routing issue is just for the remote debugging logic which uses temporary topics. JMS delivery to durable queues and topics is fine (which is why your use of Messaging in general works, but the Developer Console remote debugging is broken).

**claudiobosticco** · 6 Nov 2017, 02:09

Hello, actually there was a typo in my config, instead the realtime messaging works in cluster with this configuration for the channel:

Code:

  
<Resource
              name="jms/myChannel"
              type="javax.jms.Topic"
              factory="com.rabbitmq.jms.admin.RMQObjectFactory"
              destinationName="myChannel"/>

**Isomorphic** · 6 Nov 2017, 10:17

So, that's definitely possible with some brokers in certain configurations, but there was definitely a bug with Remote debugging specifically in clustered configs. Last night's (Nov 5) build addresses this. If you have some time to try it out, that would be great. That build also contains additional JMS improvements to help with server load with a high number of clients.

Either way - glad you have a working solution - thanks for providing all this context to help us troubleshoot!

**claudiobosticco** · 7 Nov 2017, 08:30

SmartClient Version: v11.1p_2017-11-05/Enterprise Deployment (built 2017-11-05)

Hello, I can confirm that the remote debug is working, thank you very much.

Regarding the "normal" messaging, I noticed that when I call

Code:

isc.Messaging.send("jms/updateOrdineChannel", [{foo: "bar"}])

in the tomcat logs (of the node serving this request) I see:

Code:

2017-11-07 17:25:56,582 DEBUG [claudio.bosticco@juventus.com 798] RPCManager Processing 1 requests.
2017-11-07 17:25:56,582 DEBUG [claudio.bosticco@juventus.com 798] RPCManager Request #1 (RPCRequest) data: {
    appID:"isc_builtin",
    className:"builtin",
    methodName:"messagingSend",
    arguments:[
        {
            type:"send",
            sendToChannels:[
                "jms/updateOrdineChannel"
            ],
            subscribedChannels:{
                "91A94D4E-2FDE-41A1-96A1-6AC77B79431F":{
                },
                isc_DebugTarget:{
                },
                "jms/removeOrdineChannel":{
                    subscriptionCallback:null
                },
                "jms/updateOrdineChannel":{
                    subscriptionCallback:null
                },
                "jms/addOrdineChannel":{
                    subscriptionCallback:null
                }
            },
            data:[
                {
                    foo:"bar"
                }
            ]
        }
    ],
    is_ISC_RPC_DMI:true
}
2017-11-07 17:25:56,583 INFO  [claudio.bosticco@juventus.com 798] IDACall Performing 1 operation(s)
2017-11-07 17:25:56,584 DEBUG [claudio.bosticco@juventus.com 798] MessagingConnectionHandler Delivering data: [
    {
        foo:"bar"
    }
] to: com.isomorphic.messaging.JMSMessageDispatcher, channels: [
    "jms/updateOrdineChannel"
]
2017-11-07 17:25:56,587 DEBUG [claudio.bosticco@juventus.com 798] RPCDMI rpc returned RPCResponse
2017-11-07 17:25:56,587 DEBUG [claudio.bosticco@juventus.com 798] JMSMessageDispatcher Decoded jmsMessage for channel: jms/updateOrdineChannel data[
    {
        foo:"bar"
    }
]
2017-11-07 17:25:56,587 DEBUG [claudio.bosticco@juventus.com 798] RPCManager Content type for RPC transaction: text/plain; charset=UTF-8
2017-11-07 17:25:56,588 DEBUG [claudio.bosticco@juventus.com ] MessagingConnectionHandler Sending message: ID:debbd9ca-f6ae-4cee-bf6e-be08b04bf23b:[
    {
        foo:"bar"
    }
]

while in the logs of the other tomcat I see:

Code:

2017-11-07 17:25:56,585 DEBUG [ ] JMSMessageDispatcher Decoded jmsMessage for channel: jms/updateOrdineChannel data[
    {
        foo:"bar"
    }
]
2017-11-07 17:25:56,585 DEBUG [ ] JMSMessageDispatcher Decoded jmsMessage for channel: jms/updateOrdineChannel data[
    {
        foo:"bar"
    }
]

Do these mean that there are still some duplicated messages?

**claudiobosticco** · 7 Nov 2017, 09:06

Another doubt.
Note that I'm no JMS nor RabbitMQ expert.
In the Exchanges tab of the rabbitMQ management console, if I click on jms.durable.topic, I find what I think are the jmsMessages sent.
Please see the first 5 in the attachment.

These topics are durable (I can't even Unbind them), is it normal? After some testing, that list has 1300+ rows.

Attached Files

**Isomorphic** · 7 Nov 2017, 13:14

Regarding the question about duplicates: what you're seeing is normal and does not indicate duplicates. Each JVM (Tomcat instance in this case) gets on JMS connection to the broker. For a broadcast on any given channel, all connections will get a copy of the message (so both of your Tomcat instances log the fact that they received the message at DEBUG threshold - this logging wasn't present previously). But as you can see only the first tomcat instance actually delivered the message to the client (browser) - this is indicated by the other logs.

The client-side logging that notes a duplicate is still in place, so you'd see it in the client side logs if that occurred. Note however that some duplicates are normal because due to firewalls and such, we have a connectionTTL setting (defaults to 2 minutes) which forces the browser to re-establish a connection. During re-establishment, the original connection is maintained to ensure continuity of message delivery and within this window a message may be delivered both on the original (closing) connection and on the newly established connection. This, in addition to simple defensive programming, is why we have a dup check client-side.

Regarding the question on durable topics. Yes, this is normal. We generate a unique ID (that long Routing Key) for both the DeveloperConsole browser page and the page being debugged. This is turned into a durable topic of the same name. Because these IDs are pre-generated, we can't use a pre-configured durable topic (without a bunch of reworking of how the Remote Debugging code works). We can't use a temporary topic either because by spec, those will only deliver to the JMS connection on which they were created. This last bit is why you saw the issue with Remote Debugging only working properly when both browser instance were connected to the same Tomcat instance. Because we used to use temporary topics, but by definition two separate JVMs must have two different JMS connections, so the routing would fail for non-preconfigured durable topics/queues.

My understanding is that JMS topics/queues that are dynamically created via the JMS session.createTopic() API (which is what we do unless a preconfigured Topic is available at the specified channelName) should be automatically deleted when the last consumer and producer disconnect. However, this won't happen if no messages are ever sent to the topic (which will happen if you have remote debugging enabled but never select that browser to be debugged). I believe they should go away when you restart the broker. And keep in mind that this is a developer tool - so this wouldn't be an issue in a production setting, because presumably there you use pre-configured durable topics/queues for communication and these ad-hoc channels wouldn't exist.

Announcement

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment