[#MB-617] Broker hangs during destroy() after getting Out Of Memory when executing NetworkQueueSystemUsage using MeshKeeper

FUSE Message Broker

Broker hangs during destroy() after getting Out Of Memory when executing NetworkQueueSystemUsage using MeshKeeper

Created: 23/Dec/09 06:47 PM Updated: 23/Dec/09 06:50 PM

Component/s:

broker

Affects Version/s:

5.3.0.5-fuse

Fix Version/s:

None

File Attachments:	1. multiple-broker1-thread-dumps-before-process-waitfor-hang.txt (87 kB)
Environment:	MeshKeeper. NetworkQueueSystemUsage executing from within Eclipse. MeshKeeper control server and launch agent executing on unix (soniclx18). Issue is reproducible when executing against vm-fuseubtXX machines

Description

« Hide

A BrokerManager.kill() (Active MQ Broker destroy()) hangs after the broker OOMs.

To reproduce start a MeshKeeper Control Server and Lauch Agent on a remote host (preferably Linux/Unix). Execute NetworkQueueSystemUsage using the Control Server and Launch Agent. The Broker will hang during the kill():

13:27:33 [REMOTE] TEST BROKER2: 09:30:14 INFO DemandForwardingBridge: localhost bridge to localhost stopped
Messages Sent before producer blocked: 25582 (MSGCOUNT is 25583)
13:27:46 [REMOTE] TEST BROKER1: Exception in thread "ActiveMQ Transport Server Thread Handler: tcp://0.0.0.0:61616" java.lang.OutOfMemoryError: Java heap space
Killing Brokers
TEST CLIENT: Wire format negotiation timeout: peer did not send his wire format.
TEST CLIENT: Socket closed
13:30:47 [REMOTE] TEST BROKER1: Exception in thread "ActiveMQ Scheduler"

Java thread dump of broker1 attached.

All

Comments

Change History

Sort Order:

[ Permlink | « Hide ]

Mike Cavedon added a comment - 23/Dec/09 06:50 PM

Modified the test by adding in a sleep prior to BrokerManager.kill(). Performed several "kill -QUIT"s against the broker launched broker process.

Once the hang occurred, there is no output to be found associated with the "kill -QUIT"s.