Resource exhaustion with Informix in Production

[siegersallee]

I apologize for how long this post is, but it's a big problem, and I we have already spent 10's of hours on it, so it might take some explaining. Please understand, and read this carefully, to avoid needless back-and-forth. Thank you.

We finally have a customer using our product with a lot of users in production, and we have a problem.

First:

SmartGwt: 5.0-p20150816
GWT 2.7.0
Java 1.8
Tomcat 7
CentOS 6.6

The Informix DB engine, after some days, dies from resource exhaustion, with a number of FD(file descriptor)-type resources above 85,000. Unfortunately, these "resources" can be a lot of things, like DB connections, cursors, result sets, prepared statements, temp tables. Basically, anything in the engine that an application has a "handle" to.

Here are some facts:

1. The number of resources grows steadily, by maybe 15k - 20k per day, seems to depend on user activity.
2. When Tomcat is re-started, the number resets (to <2k). Nothing else seems to decrease it much.
3. On a test system with nothing else going on, when I click on almost anything in our app, it goes up by 1-3.
4. Closing a window (which should, as far as we know, cause no DB activity) often causes it to go down by 3 or 4.
5. Closing a "major" window might make it go down by 10-20, but not as much as it went up when opening it.

Right now, we're logging in and re-starting Tomcat every few nights on the customer's system. Obviously, that has to stop.

It seems the only way for the number of resources in the DB server to change when a window in the browser is opened or closed would be through DataSources. Our code doesn't actually do anything when a window is closed.

We have spent a lot of time (>40 hours) ruling out everything else we can think of. All evidence seems to point to DataSources. We don't have custom code using transactions on DS's in our app; we mostly just connect them to UI widgets, and only to fetch data. (We have to go through a Web Service to change anything.)

We have not set any ConnectionPool properties in smartgwt.properties (does it still use DBCP? The past post I saw to mention that was a long time ago).

From everything we can see, it sure looks like DataSources are creating something and not freeing it.

I've looked at some posts that seem related, but don't seem to give me what I need to resolve this problem:

http://forums.smartclient.com/forum/...-side-requests
http://forums.smartclient.com/forum/...ooling-problem
http://forums.smartclient.com/forum/...-us-an-example

Also, I have looked at the doc for freeAllResources() and ConnectionPools, without figuring it out.

Can you verify that you have tested ConnectionPooling with Informix, and suggest what my next steps should be?

[Isomorphic]

When you say you are "closing a window" are you talking about closing a client-side SmartGWT Window component?

If so, does this cause server contact? It normally would not, it's an action that happens entirely in the browser. And if so, there is no way it could possibly affect your database.

Now if the browser is running on the same machine as Tomcat and the Informix server, the browser itself might somehow be leaking file descriptors - but you haven't yet mentioned anything about what browser(s) are in use.

[siegersallee]

Good questions, and thank you for the prompt response. Here, "closing a window", as you guessed, does mean closing a SmartGWT Window component. And the browser is Firefox running on Windoze, with the server running on Linux. That's what was so puzzling to me, too: closing a window in the browser somehow seemed to affect resource usage in the DB server, at the other end of the wire.

That's what got me thinking "well, the DS's have a Javascript component, maybe that's affected when the Window is closed."

Because we don't have any code that runs when the window closes, that we wrote.

But, even without the "resources change when a window is closed" issue, can you give me any suggestions to make this work better in production? They're not even hitting it that hard, yet: about 10 users. We should have 100 users a month from now, which (if it's linear) would kill the DB server in about 3 hours.

[Isomorphic]

First, as an aside, if this is the output of >40 hours investigation, you desperately desperately need to add an Isomorphic consultant to your team. These questions we're asking now are the first ones that our consultant would have asked and you wouldn't have spent a week on this.

Back to the problem, again, is there any server contact when the Window is closed? Use the browser's developer tools (press F12) to be sure.