Announcement

Collapse
No announcement yet.
X
  • Filter
  • Time
Clear All
new posts

    Appender SmartClientLog java.util.NoSuchElementException


    Hi Isomorphic,

    We are running into this exception in the com.isomorphic.log.RevolvingMemoryAppenderImpl.

    v14.1p_2025-04-04

    Regards,
    Jeremey

    ERROR An exception occurred processing Appender SmartClientLog java.util.NoSuchElementException: queue is empty
    at org.apache.commons.collections4.queue.CircularFifoQueue.remove(CircularFifoQueue.java:320)
    at org.apache.commons.collections4.queue.CircularFifoQueue.add(CircularFifoQueue.java:243)
    at com.isomorphic.log.RevolvingMemoryAppenderImpl.append(RevolvingMemoryAppenderImpl.java:129)
    at org.apache.logging.log4j.core.config.AppenderControl.tryCallAppender(AppenderControl.java:160)
    at org.apache.logging.log4j.core.config.AppenderControl.callAppender0(AppenderControl.java:133)
    at org.apache.logging.log4j.core.config.AppenderControl.callAppenderPreventRecursion(AppenderControl.java:124)
    at org.apache.logging.log4j.core.config.AppenderControl.callAppender(AppenderControl.java:88)
    at org.apache.logging.log4j.core.config.LoggerConfig.callAppenders(LoggerConfig.java:714)
    at org.apache.logging.log4j.core.config.LoggerConfig.processLogEvent(LoggerConfig.java:672)
    at org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:648)
    at org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:584)
    at org.apache.logging.log4j.core.config.AwaitCompletionReliabilityStrategy.log(AwaitCompletionReliabilityStrategy.java:92)
    at org.apache.logging.log4j.core.Logger.log(Logger.java:187)
    at org.apache.logging.log4j.spi.AbstractLogger.tryLogMessage(AbstractLogger.java:2904)
    at org.apache.logging.log4j.spi.AbstractLogger.logMessageTrackRecursion(AbstractLogger.java:2857)
    at org.apache.logging.log4j.spi.AbstractLogger.logMessageSafely(AbstractLogger.java:2839)
    at org.apache.logging.log4j.spi.AbstractLogger.logMessage(AbstractLogger.java:2618)
    at org.apache.logging.log4j.spi.AbstractLogger.logIfEnabled(AbstractLogger.java:2565)
    at org.apache.logging.log4j.spi.AbstractLogger.info(AbstractLogger.java:1568)
    at com.isomorphic.log.Log4j2Logger.info(Log4j2Logger.java:207)
    at com.isomorphic.log.Logger.info(Logger.java:499)
    at com.isomorphic.servlet.RequestContext.init(RequestContext.java:350)
    at com.isomorphic.servlet.RequestContext.instance(RequestContext.java:263)
    at com.isomorphic.servlet.RequestContext.instance(RequestContext.java:199)
    at com.isomorphic.servlet.BaseServlet.service(BaseServlet.java:166)
    at jakarta.servlet.http.HttpServlet.service(HttpServlet.java:658)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
    ***
    at java.base/java.lang.Thread.run(Thread.java:1583)

    #2
    There is indeed a theoretical concurrency issue here, which could cause a single log message to be lost. What's strange is:

    1. we have extensive stress tests to make sure we have no concurrency issues

    2. no one has ever reported this before on our forums or other places (like StackOverflow)

    We've gone ahead and fixed this for older versions with the narrowest possible synchronized check - available in tomorrow's builds.

    For the latest version (15.0) we're doing a more elaborate rework to make sure there can't even be a microsecond of thread wait even with massive logs

    If you don't mind, can you:

    1. confirm that for you, this has been a very rare, intermittent error, or perhaps that it resulted from a monster stress test?

    2. let us know if there's anything special about your environment - experimental or unusual JVM, log interceptors or DataDog-like system, something like that?

    3. confirm the error disappears after our fix - it would be great if you could follow up once you've had this deployed for a while, to let us know this issue is definitely gone

    Comment


      #3
      Thank you

      1. Yes, rare, we are not hitting this internally in our own environments.
      2. Unfortunately, I cannot share much here (due to #1), only that there appeared to be a burst of invalid requests. I cannot be sure what these requests where or if they were indeed related to the exception.
      3. It may be a while before we can confirm, and the underlying condition causing the issue may no longer be present.

      If by chance we do find out more, we will let you know.

      Comment

      Working...
      X