[#MB-834] using durable subs messages stay in KahaDB indefinitely if broker is restarted

FUSE Message Broker

using durable subs messages stay in KahaDB indefinitely if broker is restarted

Created: 04/Mar/11 05:45 PM Updated: 07/Mar/11 03:19 PM

Component/s:

None

Affects Version/s:

5.4.2-fuse-01-00

Fix Version/s:

5.4.2-fuse-02-00

Environment:

« Hide

http://repo.fusesource.com/nexus/content/repositories/snapshots/org/apache/activemq/apache-activemq/5.4.2-fuse-SNAPSHOT/apache-activemq-5.4.2-fuse-20110225.134226-7-bin.tar.gz

Description

« Hide

Test case from DEV-2707 comment http://fusesource.com/issues/browse/DEV-2707?focusedCommentId=64476&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#action_64476 demonstrates that after running for a while, with restarts every 3 minutes, KahaDB log files build up and seem to never be reclaimed. For instance, log files look like:

drwxr-xr-x  23 user  staff      782 Mar  3 14:23 .
drwxr-xr-x   4 user  staff      136 Mar  3 13:22 ..
-rw-r--r--   1 user  staff  5243078 Mar  3 13:25 db-4.log
-rw-r--r--   1 user  staff  5243010 Mar  3 13:26 db-5.log
-rw-r--r--   1 user  staff  5243644 Mar  3 13:50 db-38.log
-rw-r--r--   1 user  staff  5243478 Mar  3 13:51 db-39.log
-rw-r--r--   1 user  staff  5243003 Mar  3 13:57 db-48.log
-rw-r--r--   1 user  staff  5242976 Mar  3 13:59 db-51.log
-rw-r--r--   1 user  staff  5243047 Mar  3 14:12 db-74.log
-rw-r--r--   1 user  staff  5242897 Mar  3 14:13 db-75.log
-rw-r--r--   1 user  staff  5242950 Mar  3 14:15 db-78.log
-rw-r--r--   1 user  staff  5242998 Mar  3 14:15 db-79.log
-rw-r--r--   1 user  staff  5242924 Mar  3 14:16 db-80.log
-rw-r--r--   1 user  staff  5242965 Mar  3 14:18 db-83.log
-rw-r--r--   1 user  staff  5243749 Mar  3 14:18 db-84.log
-rw-r--r--   1 user  staff  5243027 Mar  3 14:19 db-85.log
-rw-r--r--   1 user  staff  5242999 Mar  3 14:20 db-86.log
-rw-r--r--   1 user  staff  5242904 Mar  3 14:22 db-89.log
-rw-r--r--   1 user  staff  5243047 Mar  3 14:23 db-90.log
-rw-r--r--   1 user  staff  4718592 Mar  3 14:23 db-91.log
-rw-r--r--   1 user  staff  6303744 Mar  3 14:23 db.data
-rw-r--r--   1 user  staff      553 Mar  3 14:23 db.free
-rw-r--r--   1 user  staff  3299608 Mar  3 14:23 db.redo

whereas when there are no restarts, the log files eventually do get deleted:

drwxr-xr-x  9 user  staff      306 Mar  3 15:30 .
drwxr-xr-x  4 user  staff      136 Mar  3 15:04 ..
-rw-r--r--  1 user  staff  5243016 Mar  3 15:29 db-74.log
-rw-r--r--  1 user  staff  5243353 Mar  3 15:29 db-75.log
-rw-r--r--  1 user  staff  5243785 Mar  3 15:30 db-76.log
-rw-r--r--  1 user  staff  4718592 Mar  3 15:30 db-77.log
-rw-r--r--  1 user  staff  8957952 Mar  3 15:30 db.data
-rw-r--r--  1 user  staff  3303712 Mar  3 15:30 db.redo
-rw-r--r--  1 user  staff        0 Mar  3 15:04 lock

Description

drwxr-xr-x  23 user  staff      782 Mar  3 14:23 .
drwxr-xr-x   4 user  staff      136 Mar  3 13:22 ..
-rw-r--r--   1 user  staff  5243078 Mar  3 13:25 db-4.log
-rw-r--r--   1 user  staff  5243010 Mar  3 13:26 db-5.log
-rw-r--r--   1 user  staff  5243644 Mar  3 13:50 db-38.log
-rw-r--r--   1 user  staff  5243478 Mar  3 13:51 db-39.log
-rw-r--r--   1 user  staff  5243003 Mar  3 13:57 db-48.log
-rw-r--r--   1 user  staff  5242976 Mar  3 13:59 db-51.log
-rw-r--r--   1 user  staff  5243047 Mar  3 14:12 db-74.log
-rw-r--r--   1 user  staff  5242897 Mar  3 14:13 db-75.log
-rw-r--r--   1 user  staff  5242950 Mar  3 14:15 db-78.log
-rw-r--r--   1 user  staff  5242998 Mar  3 14:15 db-79.log
-rw-r--r--   1 user  staff  5242924 Mar  3 14:16 db-80.log
-rw-r--r--   1 user  staff  5242965 Mar  3 14:18 db-83.log
-rw-r--r--   1 user  staff  5243749 Mar  3 14:18 db-84.log
-rw-r--r--   1 user  staff  5243027 Mar  3 14:19 db-85.log
-rw-r--r--   1 user  staff  5242999 Mar  3 14:20 db-86.log
-rw-r--r--   1 user  staff  5242904 Mar  3 14:22 db-89.log
-rw-r--r--   1 user  staff  5243047 Mar  3 14:23 db-90.log
-rw-r--r--   1 user  staff  4718592 Mar  3 14:23 db-91.log
-rw-r--r--   1 user  staff  6303744 Mar  3 14:23 db.data
-rw-r--r--   1 user  staff      553 Mar  3 14:23 db.free
-rw-r--r--   1 user  staff  3299608 Mar  3 14:23 db.redo

whereas when there are no restarts, the log files eventually do get deleted:

drwxr-xr-x  9 user  staff      306 Mar  3 15:30 .
drwxr-xr-x  4 user  staff      136 Mar  3 15:04 ..
-rw-r--r--  1 user  staff  5243016 Mar  3 15:29 db-74.log
-rw-r--r--  1 user  staff  5243353 Mar  3 15:29 db-75.log
-rw-r--r--  1 user  staff  5243785 Mar  3 15:30 db-76.log
-rw-r--r--  1 user  staff  4718592 Mar  3 15:30 db-77.log
-rw-r--r--  1 user  staff  8957952 Mar  3 15:30 db.data
-rw-r--r--  1 user  staff  3303712 Mar  3 15:30 db.redo
-rw-r--r--  1 user  staff        0 Mar  3 15:04 lock

Show »

All

Comments

Change History

Sort Order:

[ Permlink | « Hide ]

Gary Tully added a comment - 07/Mar/11 03:00 PM

again, thanks for the test. I found the culprit, the durable unsubscribe processing was not correctly persisted so after a restart, stale state was in use which blocked journal data file cleanup.
Issue: https://issues.apache.org/jira/browse/AMQ-3206

[ Permlink | « Hide ]

Gary Tully added a comment - 07/Mar/11 03:19 PM

fix on 5.4.x-fuse, will be in the next release and in tonights snapshot.