Friday, June 5, 2009

HttpConnections errors in WebLogic 10.0.1

Recently, I was working on migrating a J2EE app from WebLogic8.1 SP5 to WebLogic10.0.1.

There are so many changes:
  1. JDK1.5
  2. The way JMS is implemented by WebLogic
  3. The Work Managers and self tuning that WebLogic does...
I was expecting the J2EE app to work the same with minor changes to the codebase. While this was true for the most part, one particular "feature" caught me - HttpConnections...

Not sure how this is managed by WebLogic in 8.1 but in WLS10, I was getting errors. The http post messages started to throw errors:

<Jun 4, 2009 11:51:34 PM GMT> <Error> <WebLogicServer> <BEA-000337> <<< PPMEJB.java A.01.28x1 >> has been busy for "667" seconds working on the request "weblogic.work.ExecuteRequestAdapter@1669674", which is more than the configured time (StuckThreadMaxTime) of "600" seconds. Stack trace:
java.net.SocketInputStream.socketRead0(Native Method)
java.net.SocketInputStream.read(SocketInputStream.java:134)
java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
java.io.BufferedInputStream.read(BufferedInputStream.java:23

.....

And, my app logs:

2009-05-29T20:27:40.848 21319886:[STUCK] ExecuteThread: '3' for queue: 'weblogic.kernel.Default (self-tuning)'.GSOHub_AM|<< TransportEJB.java A.00.17 >>|DEBUG|999999||<< TransportEJB.java A.00.17 >>Exception.java.io.EOFException: Response contained no data

The self tuning done by WLS10 wasn't helping as I wasn't sure which is causing the HttpConnections problem. So, I turned it off by setting
use81-style-execute-queues true in the config.xml.

After much testing, research and code changes, I found out that the socket connections are not managed as I expected. Somehow, the http connections are not timing out or the sockets are not properly reused ...

There is some problems with the way sockets are managed in HttpConnections.

The following system properties are available to manage httpconnections correctly:
  • -Dsun.net.client.defaultConnectTimeout=5000
  • -Dsun.net.client.defaultReadTimeout=5000
  • -Dhttp.keepAlive=false
sun.net.client.defaultConnectTimeout: specifies the timeout (in milliseconds) to establish the connection to the host.

sun.net.client.defaultReadTimeout: specifies the timeout (in milliseconds) when reading from an input stream when a connection is established to a resource.

http.keepAlive: Indicates if keep alive (persistent) connections should be supported. Persistent connections improve performance by allowing the underlying socket connection be reused for multiple http requests.

As a quick fix, I decided to try http.keepAlive and set it to false. This fixed the HttpConnections problem but it may introduce a performance hit. I may use the other 2 instead -connection or read timeouts.

Also, if you need to read the response body, you will have to read the error stream if there is a IOException when the inputstream is read.

some references:
  1. Networking Properties
  2. Persistent Connections
Hope this is useful.

Daniel Rathinam
டேனியல் ரத்தினம்