I think there might be a misuse of message groups in this example. Here's what happens with this test case:

When you set message group on a message, the consumer that consumes the first message from that group is "associated" with the group. From that point on all messages from that group will be sent to that particular consumer. Now, the consumer also has a selector on it, so that will further filter messages sent to it.

If the group count is small (1 to 10) it can cause that the first consumer get "assigned" to all groups because of the prefetch size.

With the large group count (>1000) groups will be shared between two consumers, but again, only portion of those messages will be actually consumed.

What is the purpose for message groups in this scenario, maybe we can find some alternative way to achieve requested behavior.

Regards
Dejan

The message header PartitionID is directly correlated to the store broker. The message header StoreID is directly correlated to a store consumer / store queue. The JMSXGroupID is set to the store id as we must guarantee that messages are delivered in publish order to a store without any dependence on message content.

The message groups are partitioned into clean subsets - e.g. stores (message groups) 1-1000 are in Partition 0; stores 1001-200 are in Partition 1; ... There will ultimately be ~10,000 message groups / consumers spread over 10 non-overlapping partition sets.

I started down combining selectors and groups based on earlier tests (and at the suggestion of Rob Davies where message groups with competing consumers wasn't delivering the required throughput.

thanks for the info. I created a pure-amq test case and there is definitely something wrong with this behavior. I hope I can provide a fix soon.

Regards
Dejan

I played a bit more with your test case and got it working. Here's what's happening:

1. First of all, in order to have message groups with selectors working fine, we have to "cluster" selectors among partition id. Otherwise we can end up with the following situation: we send a message with group id 5 to partition 0 which will assign consumer to group id 5. If you later send a message with group id 5 and partition 1, consumer for partition 1 will not be able to consume the message, since it is assigned to the other consumer (because of the group id). I modified your MessageCreator to set group ids such as 0-5 and 1-5 so we can distinguish groups for these two consumers.

2. This will solve the problem for small number of message groups, but with large number of message groups the will block again. The problem is that by default, message group to consumer association is kept in map that uses the hash value of the group to save some memory. This works fine with small number of groups, but in our use case (large number of similary named groups), there is a big chance these hash values will overlap and some groups will be left unassigned. To fix this you should use another type of message group map implementation. You can set it with the following policy entry:

<amq:messageGroupMapFactory>
    <amq:simpleMessageGroupMapFactory/>
</amq:messageGroupMapFactory>

with this modification, you're test should be running fine even with large number of message groups.

I attached a test case with described modifications (mb515.zip). Please let me know if it works for you.

I've tried Dejan's (Thanks Dejan) suggestion and it seems to work for me.

Pedro