[#MB-873] KahaDB error: "Could not locate data file"

FUSE Message Broker

KahaDB error: "Could not locate data file"

Created: 27/Apr/11 04:31 PM Updated: 10/May/11 09:34 AM

Component/s:

broker

Affects Version/s:

5.4.2-fuse-02-00

Fix Version/s:

5.5.0-fuse-00-00

External Issue URL:

https://issues.apache.org/jira/browse/AMQ-3310

Description

« Hide

Hi,

This bug is related to AMQ-3120. That bug dealt with the KahaDB store itself, however, there seems to be a companion issue when processing non-persistent messages:

2011-04-11 13:18:00,914 [Queue:loadq-3  ] ERROR Journal                        - Looking for key 332 but not found in fileMap: {275=db-275.log number = 275 ,
 length = 33589360, 274=db-274.log number = 274 , [...]
length = 33590116, 241=db-241.log number = 241 , length = 33590228}
2011-04-11 13:18:00,916 [Queue:loadq-3  ] ERROR Queue                          - Failed to page in more queue messages 
java.lang.RuntimeException: java.io.IOException: Could not locate data file /var/cache/activemq/data/lxbra2107.cern.ch/tmp_storage/db-332.log
        at org.apache.activemq.broker.region.cursors.FilePendingMessageCursor$DiskIterator.<init>(FilePendingMessageCursor.java:486)
        at org.apache.activemq.broker.region.cursors.FilePendingMessageCursor.reset(FilePendingMessageCursor.java:129)
        at org.apache.activemq.broker.region.Queue.doPageInForDispatch(Queue.java:1712)
        at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1932)
        at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1440)
        at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:104)
        at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:42)
Caused by: java.io.IOException: Could not locate data file /var/cache/activemq/data/lxbra2107.cern.ch/tmp_storage/db-332.log
        at org.apache.kahadb.journal.Journal.getDataFile(Journal.java:350)
        at org.apache.kahadb.journal.Journal.read(Journal.java:597)
        at org.apache.activemq.store.kahadb.plist.PListStore.getPayload(PListStore.java:337)
        at org.apache.activemq.store.kahadb.plist.PList.getFirst(PList.java:280)
        at org.apache.activemq.broker.region.cursors.FilePendingMessageCursor$DiskIterator.<init>(FilePendingMessageCursor.java:482)
        ... 6 more

I haven't been able to reproduce this issue, however, I spoke to Gary who fixed the other issue and thinks there is more work to do in this area. He also saw a very similar issue in one of the unit tests yesterday so we're logging a jira. Will continue to try to come up with a test case.

Description

Hi, This bug is related to AMQ-3120. That bug dealt with the KahaDB store itself, however, there seems to be a companion issue when processing non-persistent messages:

2011-04-11 13:18:00,914 [Queue:loadq-3  ] ERROR Journal                        - Looking for key 332 but not found in fileMap: {275=db-275.log number = 275 ,
 length = 33589360, 274=db-274.log number = 274 , [...]
length = 33590116, 241=db-241.log number = 241 , length = 33590228}
2011-04-11 13:18:00,916 [Queue:loadq-3  ] ERROR Queue                          - Failed to page in more queue messages 
java.lang.RuntimeException: java.io.IOException: Could not locate data file /var/cache/activemq/data/lxbra2107.cern.ch/tmp_storage/db-332.log
        at org.apache.activemq.broker.region.cursors.FilePendingMessageCursor$DiskIterator.<init>(FilePendingMessageCursor.java:486)
        at org.apache.activemq.broker.region.cursors.FilePendingMessageCursor.reset(FilePendingMessageCursor.java:129)
        at org.apache.activemq.broker.region.Queue.doPageInForDispatch(Queue.java:1712)
        at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1932)
        at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1440)
        at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:104)
        at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:42)
Caused by: java.io.IOException: Could not locate data file /var/cache/activemq/data/lxbra2107.cern.ch/tmp_storage/db-332.log
        at org.apache.kahadb.journal.Journal.getDataFile(Journal.java:350)
        at org.apache.kahadb.journal.Journal.read(Journal.java:597)
        at org.apache.activemq.store.kahadb.plist.PListStore.getPayload(PListStore.java:337)
        at org.apache.activemq.store.kahadb.plist.PList.getFirst(PList.java:280)
        at org.apache.activemq.broker.region.cursors.FilePendingMessageCursor$DiskIterator.<init>(FilePendingMessageCursor.java:482)
        ... 6 more

Show »

All

Comments

Change History

Sort Order:

[ Permlink | « Hide ]

Gary Tully added a comment - 09/May/11 04:15 PM

resolved on trunk, https://issues.apache.org/jira/browse/AMQ-3310 issue is unique to the temp store, the PListStore. data file cleanup needed to be re-implemented to deal with async journal location allocation, reference counts were getting lost with the old schema.

[ Permlink | « Hide ]

Susan Javurek added a comment - 10/May/11 08:44 AM

Thanks Gary, that's great. Do you know when it might be in a SNAPSHOT that we can test with from fusesource? I looked at the latest
5.5 versions still seems to be from april 21st, although some files are from the 10th.

All the best,
Susan

[ Permlink | « Hide ]

Gary Tully added a comment - 10/May/11 09:34 AM

new snapshot available with this fix. http://ci.fusesource.com/hudson/job/activemq-5.5.x-fuse-deploy/25/