54070 - The SASServer7 JBoss web application server takes a long time to start

Problem Note 54070: The SASServer7 JBoss web application server takes a long time to start

The SASServer7 instance of the JBoss web application server can take 2 hours or more to start. When this problem occurs, there are no errors in the server.log file. The server seems to stop responding and does not log any trace information in server.log for almost 1.5 hours.

If you initiate a thread dump, you will see the following thread activity:

"main" prio=10 tid=0x00007fdb5c1d0800 nid=0xc68 runnable [0x00007fdb48d52000]
   java.lang.Thread.State: RUNNABLE
	at org.hsqldb.Expression.getValue(Unknown Source)
	at org.hsqldb.Expression.testInCondition(Unknown Source)
	at org.hsqldb.Expression.test(Unknown Source)
	at org.hsqldb.Expression.testCondition(Unknown Source)
	at org.hsqldb.Select.buildResult(Unknown Source)
	at org.hsqldb.Select.getSingleResult(Unknown Source)
	at org.hsqldb.Select.getResult(Unknown Source)
	at org.hsqldb.CompiledStatementExecutor.executeSelectStatement(Unknown Source)
	at org.hsqldb.CompiledStatementExecutor.executeImpl(Unknown Source)
	at org.hsqldb.CompiledStatementExecutor.execute(Unknown Source)
	at org.hsqldb.Session.sqlExecuteCompiledNoPreChecks(Unknown Source)
	at org.hsqldb.Session.sqlExecute(Unknown Source)
	at org.hsqldb.Session.sqlExecute(Unknown Source)
	at org.hsqldb.Session.execute(Unknown Source)
	- locked <0x00000006f6e0fc50> (a org.hsqldb.Database)
	at org.hsqldb.jdbc.jdbcPreparedStatement.executeQuery(Unknown Source)
	at org.jboss.resource.adapter.jdbc.CachedPreparedStatement.executeQuery(CachedPreparedStatement.java:91)
	at org.jboss.resource.adapter.jdbc.WrappedPreparedStatement.executeQuery(WrappedPreparedStatement.java:342)
	at org.jboss.messaging.core.impl.JDBCPersistenceManager$1GetMessageListTX.doTransaction(JDBCPersistenceManager.java:561)
	at org.jboss.messaging.core.impl.JDBCPersistenceManager$1GetMessageListTX.doTransaction(JDBCPersistenceManager.java:497)
	at org.jboss.messaging.core.impl.JDBCSupport$JDBCTxRunner2.execute(JDBCSupport.java:474)
	at org.jboss.messaging.core.impl.JDBCSupport$JDBCTxRunner2.executeWithRetry(JDBCSupport.java:512)
	at org.jboss.messaging.core.impl.JDBCPersistenceManager.getMessages(JDBCPersistenceManager.java:614)
	at org.jboss.messaging.core.impl.PagingChannelSupport.processReferences(PagingChannelSupport.java:565)
	at org.jboss.messaging.core.impl.PagingChannelSupport.doLoad(PagingChannelSupport.java:498)
	at org.jboss.messaging.core.impl.PagingChannelSupport.load(PagingChannelSupport.java:211)
	- locked <0x00000006e0cc5628> (a java.lang.Object)
	at org.jboss.jms.server.destination.QueueService.startService(QueueService.java:105)
	- locked <0x00000006f44046d8> (a org.jboss.jms.server.destination.QueueService)
	at org.jboss.system.ServiceMBeanSupport.jbossInternalStart(ServiceMBeanSupport.java:376)
	at org.jboss.system.ServiceMBeanSupport.jbossInternalLifecycle(ServiceMBeanSupport.java:322)
	at sun.reflect.GeneratedMethodAccessor80.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:157)
	at org.jboss.mx.server.Invocation.dispatch(Invocation.java:96)
	at org.jboss.mx.interceptor.AbstractInterceptor.invoke(AbstractInterceptor.java:138)
	at org.jboss.mx.server.Invocation.invoke(Invocation.java:90)
	at org.jboss.mx.interceptor.ModelMBeanOperationInterceptor.invoke(ModelMBeanOperationInterceptor.java:140)
	at org.jboss.mx.server.Invocation.invoke(Invocation.java:90)
	at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
	at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:668)
	at org.jboss.system.microcontainer.ServiceProxy.invoke(ServiceProxy.java:189)
	at $Proxy38.start(Unknown Source)
	at org.jboss.system.microcontainer.StartStopLifecycleAction.installAction(StartStopLifecycleAction.java:42)
	at org.jboss.system.microcontainer.StartStopLifecycleAction.installAction(StartStopLifecycleAction.java:37)
	at org.jboss.dependency.plugins.action.SimpleControllerContextAction.simpleInstallAction(SimpleControllerContextAction.java:62)

In addition, the SASServer7 log file (server.log) indicates the lag in logging activity:

2014-08-25 04:21:14,471 INFO  [org.jboss.resource.connectionmanager.ConnectionFactoryBindingService] (main) Bound ConnectionManager 'jboss.jca:service=DataSourceBinding,name=DefaultDS' to JNDI name 'java:DefaultDS'
2014-08-25 06:02:20,524 INFO  [org.jboss.resource.connectionmanager.ConnectionFactoryBindingService] (main) Bound ConnectionManager 'jboss.jca:service=ConnectionFactoryBinding,name=JmsXA' to JNDI name 'java:JmsXA'

To resolve this problem, follow these steps on the middle-tier machine:

Stop the JBoss SASServer7 instance.
Make a backup copy of the Hypersonic data directory JBOSS-home-directory\server\default\data\hypersonic.
Delete all of the files under the original Hypersonic directory JBOSS-home-directory\server\default\data\hypersonic.
Restart the SASServer7 instance.

Operating System and Release Information

Product Family	Product	System	Product Release		SAS Release
			Reported	Fixed*	Reported	Fixed*
SAS System	JBoss Application Server	Linux for x64	5.1.0		9.3 TS1M2

* For software releases that are not yet generally available, the Fixed Release is the software release in which the problem is planned to be fixed.

Type:	Problem Note
Priority:	medium

Date Modified:	2014-09-12 13:05:16
Date Created:	2014-09-08 14:37:22