Forum Home » Fuse Distributions » Fuse ESB

Thread: Problem setting up a clustered ESB

 
This question is answered.


Permlink Replies: 6 - Last Post: Sep 26, 2011 5:39 AM Last Post By: dhoyt
dhoyt

Posts: 32
Registered: 03/03/11
Problem setting up a clustered ESB
Posted: Sep 19, 2011 4:45 PM
 
  Click to reply to this thread Reply
Hello all,

I'm trying to setup clustering of Servicemix (apache-servicemix-4.3.1-fuse-02-05 to be exact). After doing a bit of research, it appears that a "network of brokers" is the best way to accomplish this. I have tried to follow several tutorials online, but I can't seem to get the a consumer on myHost2 to listen on a topic that is being populated by a producer on myHost1 (I am using two very simple camel routes that are publishing and subscribing to an ActiveMQ topic). I can telnet to each server from the other on port 61616, so I know the ports are open, but I am not seeing any communication between them.

As far as I can tell, the only file that needs to be changed is activemq-broker.xml (included below). The only changes I made to the activemq-broker.xml file was to change brokerName (they are different for each instance), added the networkConnectors node, and changed localhost to 0.0.0.0 so it could be reached from other servers. Is there anything else that needs to be done? Any help would be greatly appreciated.

<blueprint xmlns="http://www.osgi.org/xmlns/blueprint/v1.0.0"
xmlns:cm="http://aries.apache.org/blueprint/xmlns/blueprint-cm/v1.0.0"
xmlns:ext="http://aries.apache.org/blueprint/xmlns/blueprint-ext/v1.0.0"
xmlns:amq="http://activemq.apache.org/schema/core">
<ext:property-placeholder />

<broker xmlns="http://activemq.apache.org/schema/core" brokerName="atlas-myHost2-master" dataDirectory="${karaf.data}/activemq/default" useShutdownHook="false">
<destinationPolicy>
<policyMap>
<policyEntries>
<policyEntry topic=">" producerFlowControl="true" memoryLimit="1mb">
<pendingSubscriberPolicy>
<vmCursor />
</pendingSubscriberPolicy>
</policyEntry>
<policyEntry queue=">" producerFlowControl="true" memoryLimit="1mb">
</policyEntry>
</policyEntries>
</policyMap>
</destinationPolicy>

<managementContext>
<managementContext createConnector="false"/>
</managementContext>

<persistenceAdapter>
<kahaDB directory="${karaf.data}/activemq/default/kahadb"/>
</persistenceAdapter>

<networkConnectors>
<networkConnector name="master_myHost2_to_myHost1" uri="static://(tcp://myHost1:61616)"/>
</networkConnectors>

<transportConnectors>
<transportConnector name="openwire" uri="tcp://0.0.0.0:61616"/>
<transportConnector name="stomp" uri="stomp://0.0.0.0:61613"/>
</transportConnectors>
</broker>

<bean id="activemqConnectionFactory" class="org.apache.activemq.ActiveMQConnectionFactory">
<property name="brokerURL" value="tcp://0.0.0.0:61616" />
</bean>

<bean id="pooledConnectionFactory" class="org.apache.activemq.pool.PooledConnectionFactory">
<property name="maxConnections" value="8" />
<property name="connectionFactory" ref="activemqConnectionFactory" />
</bean>

<bean id="resourceManager" class="org.apache.activemq.pool.ActiveMQResourceManager" init-method="recoverResource">
<property name="transactionManager" ref="transactionManager" />
<property name="connectionFactory" ref="activemqConnectionFactory" />
<property name="resourceName" value="activemq.default" />
</bean>

<reference id="transactionManager" interface="javax.transaction.TransactionManager" />

<service ref="pooledConnectionFactory" interface="javax.jms.ConnectionFactory">
<service-properties>
<entry key="name" value="localhost"/>
</service-properties>
</service>
</blueprint>

iocanel

Posts: 153
Registered: 03/16/10
Re: Problem setting up a clustered ESB
Posted: Sep 21, 2011 8:12 AM   in response to: dhoyt in response to: dhoyt
 
  Click to reply to this thread Reply
From what I make from you configuration file you are adding a network connector that is connecting to itself, while it should be pointing to the other broker.

so if for instance you have on broker A the following config:

<transportConnector name="openwire" uri="tcp://0.0.0.0:61616"/>

and on broker B:

<transportConnector name="openwire" uri="tcp://0.0.0.0:61617"/>


Then the network connector of broker A should point to 61617.

I hope this helps!

dhoyt

Posts: 32
Registered: 03/03/11
Re: Problem setting up a clustered ESB
Posted: Sep 21, 2011 10:24 PM   in response to: iocanel in response to: iocanel
 
  Click to reply to this thread Reply
I made that change, but now the broker won't come up. Looking at the log, I see the error listed below. It seems that no matter where I put the networkConnectors element, I run into problems with the following element. Any idea what is causing this? Is the networkConnectors element no longer supported?

org.osgi.service.blueprint.container.ComponentDefinitionException: Unable to validate xml
at org.apache.aries.blueprint.container.Parser.validate(Parser.java:288)7:org.apache.aries.blueprint:0.2.0.incubating
at org.apache.aries.blueprint.container.BlueprintContainerImpl.doRun(BlueprintContainerImpl.java:261)7:org.apache.aries.blueprint:0.2.0.incubating
at org.apache.aries.blueprint.container.BlueprintContainerImpl.run(BlueprintContainerImpl.java:213)7:org.apache.aries.blueprint:0.2.0.incubating
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441):1.6.0_20
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303):1.6.0_20
at java.util.concurrent.FutureTask.run(FutureTask.java:138):1.6.0_20
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:98):1.6.0_20
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:207):1.6.0_20
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886):1.6.0_20
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908):1.6.0_20
at java.lang.Thread.run(Thread.java:619):1.6.0_20
Caused by: org.xml.sax.SAXParseException: cvc-complex-type.2.4.a: Invalid content was found starting with element 'destinationPolicy'. One of '{"http://activemq.apache.org/schema/core":persistenceAdapter, "http://activemq.apache.org/schema/core":persistenceFactory, "http://activemq.apache.org/schema/core":persistenceTaskRunnerFactory, "http://activemq.apache.org/schema/core":plugins, "http://activemq.apache.org/schema/core":producerSystemUsage, "http://activemq.apache.org/schema/core":proxyConnectors, "http://activemq.apache.org/schema/core":regionBroker, "http://activemq.apache.org/schema/core":services, "http://activemq.apache.org/schema/core":shutdownHooks, "http://activemq.apache.org/schema/core":sslContext, "http://activemq.apache.org/schema/core":systemUsage, "http://activemq.apache.org/schema/core":taskRunnerFactory, "http://activemq.apache.org/schema/core":tempDataStore, "http://activemq.apache.org/schema/core":transportConnectorURIs, "http://activemq.apache.org/schema/core":transportConnectors, WC}' is expected.
at com.sun.org.apache.xerces.internal.util.ErrorHandlerWrapper.createSAXParseException(ErrorHandlerWrapper.java:233)[:]


<blueprint xmlns="http://www.osgi.org/xmlns/blueprint/v1.0.0"
xmlns:cm="http://aries.apache.org/blueprint/xmlns/blueprint-cm/v1.0.0"
xmlns:ext="http://aries.apache.org/blueprint/xmlns/blueprint-ext/v1.0.0"
xmlns:amq="http://activemq.apache.org/schema/core">

<!-- Allows us to use system properties as variables in this configuration file -->
<ext:property-placeholder />

<broker xmlns="http://activemq.apache.org/schema/core" brokerName="atlas-erosdapp02-master" dataDirectory="${karaf.data}/activemq/Atlas" useShutdownHook="false">

<networkConnectors>
<networkConnector uri="static://(tcp://erosdapp03:61616)"/>
</networkConnectors>

<destinationPolicy>
<policyMap>
<policyEntries>
<policyEntry topic=">" producerFlowControl="true" memoryLimit="1mb">
<pendingSubscriberPolicy>
<vmCursor />
</pendingSubscriberPolicy>
</policyEntry>
<policyEntry queue=">" producerFlowControl="true" memoryLimit="1mb">
<!-- Use VM cursor for better latency
For more information, see:

http://activemq.apache.org/message-cursors.html

<pendingQueuePolicy>
<vmQueueCursor/>
</pendingQueuePolicy>
-->
</policyEntry>
</policyEntries>
</policyMap>
</destinationPolicy>

<!-- Use the following to configure how ActiveMQ is exposed in JMX -->
<managementContext>
<managementContext createConnector="false"/>
</managementContext>

<persistenceAdapter>
<kahaDB directory="${karaf.data}/activemq/Atlas/kahadb"/>
</persistenceAdapter>

<!-- The transport connectors ActiveMQ will listen to -->
<transportConnectors>
<transportConnector name="openwire" uri="tcp://myhost2:61616"/>
</transportConnectors>
</broker>

<bean id="activemqConnectionFactory" class="org.apache.activemq.ActiveMQConnectionFactory">
<property name="brokerURL" value="tcp://0.0.0.0:61616" />
</bean>

<bean id="pooledConnectionFactory" class="org.apache.activemq.pool.PooledConnectionFactory">
<property name="maxConnections" value="8" />
<property name="connectionFactory" ref="activemqConnectionFactory" />
</bean>

<bean id="resourceManager" class="org.apache.activemq.pool.ActiveMQResourceManager" init-method="recoverResource">
<property name="transactionManager" ref="transactionManager" />
<property name="connectionFactory" ref="activemqConnectionFactory" />
<property name="resourceName" value="activemq.Atlas" />
</bean>

<reference id="transactionManager" interface="javax.transaction.TransactionManager" />

<service ref="pooledConnectionFactory" interface="javax.jms.ConnectionFactory">
<service-properties>
<entry key="name" value="localhost"/>
</service-properties>
</service>
</blueprint>

davsclaus

Posts: 1,893
Registered: 10/14/08
Re: Problem setting up a clustered ESB
Posted: Sep 22, 2011 7:02 AM   in response to: dhoyt in response to: dhoyt
Correct
  Click to reply to this thread Reply
The XML tags in the XML file is sorted A..Z. So make sure you position the tags in correct order.
dhoyt

Posts: 32
Registered: 03/03/11
Re: Problem setting up a clustered ESB
Posted: Sep 23, 2011 3:00 AM   in response to: davsclaus in response to: davsclaus
 
  Click to reply to this thread Reply
That did it! My XML tags were not in alphabetical order. Below is the config (for myHost1) that worked for me. Thanks for all the help!

<blueprint xmlns="http://www.osgi.org/xmlns/blueprint/v1.0.0"
xmlns:cm="http://aries.apache.org/blueprint/xmlns/blueprint-cm/v1.0.0"
xmlns:ext="http://aries.apache.org/blueprint/xmlns/blueprint-ext/v1.0.0"
xmlns:amq="http://activemq.apache.org/schema/core">

<!-- Allows us to use system properties as variables in this configuration file -->
<ext:property-placeholder />

<broker xmlns="http://activemq.apache.org/schema/core" brokerName="broker-myHost1" dataDirectory="${karaf.data}/activemq/default" useShutdownHook="false">
<destinationPolicy>
<policyMap>
<policyEntries>
<policyEntry topic=">" producerFlowControl="true" memoryLimit="1mb">
<pendingSubscriberPolicy>
<vmCursor />
</pendingSubscriberPolicy>
</policyEntry>
<policyEntry queue=">" producerFlowControl="true" memoryLimit="1mb">
</policyEntry>
</policyEntries>
</policyMap>
</destinationPolicy>

<!-- Use the following to configure how ActiveMQ is exposed in JMX -->
<managementContext>
<managementContext createConnector="false"/>
</managementContext>

<networkConnectors>
<networkConnector uri="static://(tcp://myHost2:61616)"/>
</networkConnectors>

<persistenceAdapter>
<kahaDB directory="${karaf.data}/activemq/default/kahadb"/>
</persistenceAdapter>

<!-- The transport connectors ActiveMQ will listen to -->
<transportConnectors>
<transportConnector name="openwire" uri="tcp://0.0.0.0:61616"/>
</transportConnectors>
</broker>

<bean id="activemqConnectionFactory" class="org.apache.activemq.ActiveMQConnectionFactory">
<property name="brokerURL" value="tcp://localhost:61616" />
</bean>

<bean id="pooledConnectionFactory" class="org.apache.activemq.pool.PooledConnectionFactory">
<property name="maxConnections" value="8" />
<property name="connectionFactory" ref="activemqConnectionFactory" />
</bean>

<bean id="resourceManager" class="org.apache.activemq.pool.ActiveMQResourceManager" init-method="recoverResource">
<property name="connectionFactory" ref="activemqConnectionFactory" />
<property name="resourceName" value="activemq.default" />
</bean>

<reference id="transactionManager" interface="javax.transaction.TransactionManager" />

<service ref="pooledConnectionFactory" interface="javax.jms.ConnectionFactory">
<service-properties>
<entry key="name" value="localhost"/>
</service-properties>
</service>
</blueprint>
michal.warecki

Posts: 49
Registered: 01/23/11
Re: Problem setting up a clustered ESB
Posted: Sep 24, 2011 3:34 PM   in response to: dhoyt in response to: dhoyt
 
  Click to reply to this thread Reply
I'm little bit confused about this "ESB Clustering".
Is it really ESB clustering or just ActiveMQ clustering? What if I deploy jaxws web service instead of queue? Will it redirect message to slave Servicemix instance? I think no.
Shouldn't we create ActiveMQ cluster and separate database locking based Servicemix cluster (http://fusesource.com/docs/esb/4.4.1/esb_runtime/ESBRuntimeFailoverJdbc.html)?
dhoyt

Posts: 32
Registered: 03/03/11
Re: Problem setting up a clustered ESB
Posted: Sep 26, 2011 5:39 AM   in response to: michal.warecki in response to: michal.warecki
 
  Click to reply to this thread Reply
I am doing that as well, but I required an active-active cluster. To accomplish that, I have a network of brokers where each node has a slave. This gives me both clustering and HA. As far as I have seen, clustering of ActiveMQ is what is required to cluster ServiceMix.