Announcement

Collapse
No announcement yet.
X
  • Filter
  • Time
Clear All
new posts

    12.0p Strange DataSourceLoader exception

    Hi Isomorphic,

    out of nothing we started to get this error today (v12.0p_2020-03-11). I assume it's some race condition or something similar, because we did not change anything and it was only happening for one customer. Does this stack trace mean something to you?
    I had a look at the .ds.xml and the only apparent thing is that it uses many (~70) fields with value map (2 of them with fmt tags (<value ID="W"><fmt:message key="won" /></value>)).

    Two stacktraces, always appearing together:
    Code:
    javax.servlet.ServletException: DataSource 'T_STATUS_CHANGE' failed to load due to an exception on the server: Instantiation of class: java.util.LinkedHashMap$LinkedKeySet threw an InstantiationException - most likely cause is the class represents an abstract class, an interface, an array class, a primitive type, or void; or the class has no zero-argument constructor. See the server-side log for additional details. at com.isomorphic.servlet.DataSourceLoader.processRequest(DataSourceLoader.java:295) at com.isomorphic.servlet.DataSourceLoader.doGet(DataSourceLoader.java:110) at javax.servlet.http.HttpServlet.service(HttpServlet.java:634) at com.isomorphic.servlet.BaseServlet.service(BaseServlet.java:176) at javax.servlet.http.HttpServlet.service(HttpServlet.java:741) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at com.lmscompany.lms.server.MimeTypeFilter.doFilter(MimeTypeFilter.java:26) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at com.lmscompany.lms.server.RequestLoggingFilter.doFilter(RequestLoggingFilter.java:41) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at com.lmscompany.lms.server.ExceptionLoggingFilter.doFilter(ExceptionLoggingFilter.java:28) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at com.isomorphic.servlet.CompressionFilter._doFilter(CompressionFilter.java:260) at com.isomorphic.servlet.BaseFilter.doFilter(BaseFilter.java:93) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at org.apache.catalina.filters.HttpHeaderSecurityFilter.doFilter(HttpHeaderSecurityFilter.java:126) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at org.apache.catalina.filters.HttpHeaderSecurityFilter.doFilter(HttpHeaderSecurityFilter.java:126) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:202) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96) at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:666) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:139) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92) at org.apache.catalina.valves.rewrite.RewriteValve.invoke(RewriteValve.java:555) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74) at org.apache.catalina.ha.tcp.ReplicationValve.invoke(ReplicationValve.java:330) at org.apache.catalina.valves.RemoteIpValve.invoke(RemoteIpValve.java:747) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343) at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:367) at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65) at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:868) at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1639) at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) at java.lang.Thread.run(Thread.java:748)
    Code:
    java.lang.InstantiationException: Instantiation of class: java.util.LinkedHashMap$LinkedKeySet threw an InstantiationException - most likely cause is the class represents an abstract class, an interface, an array class, a primitive type, or void; or the class has no zero-argument constructor. at com.isomorphic.base.Reflection.newInstance(Reflection.java:230) at com.isomorphic.base.Reflection.newInstance(Reflection.java:193) at com.isomorphic.util.DataTools.deepClone(DataTools.java:490) at com.isomorphic.util.DataTools.deepClone(DataTools.java:486) at com.isomorphic.util.DataTools.deepClone(DataTools.java:493) at com.isomorphic.util.DataTools.deepClone(DataTools.java:486) at com.isomorphic.util.DataTools.deepClone(DataTools.java:486) at com.isomorphic.util.DataTools.deepClone(DataTools.java:486) at com.isomorphic.datasource.DataSource._cloneConfigForSecurityAnnotations(DataSource.java:1762) at com.isomorphic.servlet.DataSourceLoader.processRequest(DataSourceLoader.java:232) at com.isomorphic.servlet.DataSourceLoader.doGet(DataSourceLoader.java:110) at javax.servlet.http.HttpServlet.service(HttpServlet.java:634) at com.isomorphic.servlet.BaseServlet.service(BaseServlet.java:176) at javax.servlet.http.HttpServlet.service(HttpServlet.java:741) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at com.lmscompany.lms.server.MimeTypeFilter.doFilter(MimeTypeFilter.java:26) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at com.lmscompany.lms.server.RequestLoggingFilter.doFilter(RequestLoggingFilter.java:41) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at com.lmscompany.lms.server.ExceptionLoggingFilter.doFilter(ExceptionLoggingFilter.java:28) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at com.isomorphic.servlet.CompressionFilter._doFilter(CompressionFilter.java:260) at com.isomorphic.servlet.BaseFilter.doFilter(BaseFilter.java:93) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at org.apache.catalina.filters.HttpHeaderSecurityFilter.doFilter(HttpHeaderSecurityFilter.java:126) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at org.apache.catalina.filters.HttpHeaderSecurityFilter.doFilter(HttpHeaderSecurityFilter.java:126) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:202) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96) at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:666) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:139) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92) at org.apache.catalina.valves.rewrite.RewriteValve.invoke(RewriteValve.java:555) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74) at org.apache.catalina.ha.tcp.ReplicationValve.invoke(ReplicationValve.java:330) at org.apache.catalina.valves.RemoteIpValve.invoke(RemoteIpValve.java:747) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343) at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:367) at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65) at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:868) at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1639) at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) at java.lang.Thread.run(Thread.java:748)
    Thank you & Best regards
    Blama
    Last edited by Blama; 29th Apr 2020, 05:00.

    #2
    Sorry for the broken linefeeds, they appear OK in edit-mode.

    Code:
    Test1
    Test2
    Test3

    Comment


      #3
      Trace 1:
      Code:
      java.lang.InstantiationException: Instantiation of class: java.util.LinkedHashMap$LinkedKeySet threw an InstantiationException - most likely cause is the class represents an abstract class, an interface, an array class, a primitive type, or void; or the class has no zero-argument constructor. at com.isomorphic.base.Reflection.newInstance(Reflection.java:230) at com.isomorphic.base.Reflection.newInstance(Reflection.java:193) at com.isomorphic.util.DataTools.deepClone(DataTools.java:490) at com.isomorphic.util.DataTools.deepClone(DataTools.java:486) at com.isomorphic.util.DataTools.deepClone(DataTools.java:493) at com.isomorphic.util.DataTools.deepClone(DataTools.java:486) at com.isomorphic.util.DataTools.deepClone(DataTools.java:486) at com.isomorphic.util.DataTools.deepClone(DataTools.java:486) at com.isomorphic.datasource.DataSource._cloneConfigForSecurityAnnotations(DataSource.java:1762) at com.isomorphic.servlet.DataSourceLoader.processRequest(DataSourceLoader.java:232) at com.isomorphic.servlet.DataSourceLoader.doGet(DataSourceLoader.java:110) at javax.servlet.http.HttpServlet.service(HttpServlet.java:634) at com.isomorphic.servlet.BaseServlet.service(BaseServlet.java:176) at javax.servlet.http.HttpServlet.service(HttpServlet.java:741) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at com.lmscompany.lms.server.MimeTypeFilter.doFilter(MimeTypeFilter.java:26) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at com.lmscompany.lms.server.RequestLoggingFilter.doFilter(RequestLoggingFilter.java:41) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at com.lmscompany.lms.server.ExceptionLoggingFilter.doFilter(ExceptionLoggingFilter.java:28) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at com.isomorphic.servlet.CompressionFilter._doFilter(CompressionFilter.java:260) at com.isomorphic.servlet.BaseFilter.doFilter(BaseFilter.java:93) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at org.apache.catalina.filters.HttpHeaderSecurityFilter.doFilter(HttpHeaderSecurityFilter.java:126) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at org.apache.catalina.filters.HttpHeaderSecurityFilter.doFilter(HttpHeaderSecurityFilter.java:126) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:202) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96) at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:666) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:139) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92) at org.apache.catalina.valves.rewrite.RewriteValve.invoke(RewriteValve.java:555) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74) at org.apache.catalina.ha.tcp.ReplicationValve.invoke(ReplicationValve.java:330) at org.apache.catalina.valves.RemoteIpValve.invoke(RemoteIpValve.java:747) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343) at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:367) at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65) at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:868) at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1639) at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) at java.lang.Thread.run(Thread.java:748)

      Comment


        #4
        Trace 2 shortened:
        Code:
        javax.servlet.ServletException: DataSource 'T_STATUS_CHANGE' failed to load due to an exception on the server:
        Instantiation of class: java.util.LinkedHashMap$LinkedKeySet threw an InstantiationException - most likely cause is the class represents an abstract class, an interface, an array class, a primitive type, or void; or the class has no zero-argument constructor.
        See the server-side log for additional details.
            at com.isomorphic.servlet.DataSourceLoader.processRequest(DataSourceLoader.java:295)
            at com.isomorphic.servlet.DataSourceLoader.doGet(DataSourceLoader.java:110)
            at javax.servlet.http.HttpServlet.service(HttpServlet.java:634)
            at com.isomorphic.servlet.BaseServlet.service(BaseServlet.java:176)
            at javax.servlet.http.HttpServlet.service(HttpServlet.java:741)
            at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)
            at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
            at com.lmscompany.lms.server.MimeTypeFilter.doFilter(MimeTypeFilter.java:26)
            at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
            at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
            at com.lmscompany.lms.server.RequestLoggingFilter.doFilter(RequestLoggingFilter.java:41)
            at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
            at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
            at com.lmscompany.lms.server.ExceptionLoggingFilter.doFilter(ExceptionLoggingFilter.java:28)
            at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
            at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
        Trace 1 shortened:
        Code:
        java.lang.InstantiationException: Instantiation of class: java.util.LinkedHashMap$LinkedKeySet threw an InstantiationException - most likely cause is the class represents an abstract class, an interface, an array class, a primitive type, or void; or the class has no zero-argument constructor.
            at com.isomorphic.base.Reflection.newInstance(Reflection.java:230)
            at com.isomorphic.base.Reflection.newInstance(Reflection.java:193)
            at com.isomorphic.util.DataTools.deepClone(DataTools.java:490)
            at com.isomorphic.util.DataTools.deepClone(DataTools.java:486)
            at com.isomorphic.util.DataTools.deepClone(DataTools.java:493)
            at com.isomorphic.util.DataTools.deepClone(DataTools.java:486)
            at com.isomorphic.util.DataTools.deepClone(DataTools.java:486)
            at com.isomorphic.util.DataTools.deepClone(DataTools.java:486)
            at com.isomorphic.datasource.DataSource._cloneConfigForSecurityAnnotations(DataSource.java:1762)
            at com.isomorphic.servlet.DataSourceLoader.processRequest(DataSourceLoader.java:232)
            at com.isomorphic.servlet.DataSourceLoader.doGet(DataSourceLoader.java:110)
            at javax.servlet.http.HttpServlet.service(HttpServlet.java:634)
        Last edited by Blama; 29th Apr 2020, 05:05.

        Comment


          #5
          This doesn't look like a concurrency exception, as this is code that is going through and copying a data structure while not modifying it.

          This actually looks more like the JVM getting internally confused about types. This kind of thing can happen if you do something that causes on-the-fly application reloads, where the JVM is trying to garbage collect actual Java class definitions and then re-load them. If you are reloading your application without restarting the JVM, you might find that this happens only after a reload.

          Finally, you should look at the surrounding logs to see which DataSource was being processed (or if you don't have enough logging to tell this, turn it on for next time). If this error recurs for a single DataSource, there could be something unique the .ds.xml usage in that case, maybe something like custom properties and subelements.

          Comment


            #6
            Hi Isomorphic,

            yes, I find it also very strange - looks to me like a JVM thing as well.

            W.r.t. the .ds.xml: I wrote in #1 what I find special about this DataSource: The only apparent thing is that it uses many (~70) fields with value map (2 of them with fmt tags (<value ID="W"><fmt:message key="won" /></value>)).

            The application wasn't restarted IMHO, everything mysterious.

            If you say that there is DataTools.java:486 3 times in a row in the trace is correct (perhaps because of recursion(?)), then everything is fine for me. This was a one time thing so far here.

            Thank you & Best regards
            Blama

            Comment


              #7
              Yes, that's just recursion, that part looks normal.

              Have you seen this error more than once, and is it always with the same DataSource?

              Comment


                #8
                Hi Isomorphic,

                I saw it only once in production. I think I also saw it locally before (which could involve stop/start-cycles), but not 100% sure.
                I'll report back here if it happens again.

                Thank you & Best regards
                Blama

                Comment


                  #9
                  Hi Isomorphic,

                  this problem happened again today in our production system. Any new ideas about it?
                  SmartClient Version: v12.0p_2020-07-24/PowerEdition Deployment (built 2020-07-24)

                  Best,
                  Pavo

                  Comment


                    #10
                    We don't have any new data other than that it happened again, so no. Same speculations as before.

                    Comment


                      #11
                      Hi Isomorphic,

                      it happend once again just now, again with the same DataSource and on a yesterday restarted Tomcat (we don't do application stop/starts in a running Tomcat in production). The application itself was not much used (it's still being prepared for the customer).
                      There seems to be something special about the DataSource indeed.
                      The next time it happens I'll take the VM out of load balancing and try to remote debug the JVM instead of just restarting it.

                      Best regards
                      Blama
                      Last edited by Blama; 11th Aug 2020, 02:30.

                      Comment

                      Working...
                      X