Resolving Durable Duplicate Client ID Issues

Once I completed the previous blog post: Spring and WebLogic Durable Subscriber, the configurations were deployed to a multiple domain cluster (with some Managed Servers running on a separate physical machines). Errors started to occur within the cluster based on the use of the SingleConnectionFactory configuration. What appeared to be happening was when the subscribe war was deployed out to WebLogic Server, the 1st Managed Server started and created a durable session with the configured clientId. The problem was when the war started on the 2nd Managed Server, it attempted to create a durable session with that same configured clientId and we started receiving errors such as this:

[JMSClientExceptions:055083]Cannot set clientid to sampleSubscribeService. The client ID is already set to sampleSubscribeService
2012-12-28 12:40:12,604 THREAD:[143] WARN ID:[] org.springframework.jms.listener.DefaultMessageListenerContainer – Could not refresh JMS Connection for destination ‘jmsServer1@com/foo/dest/SampleTopic’ – retrying in 3600000 ms. Cause: [JMSClientExceptions:055083]Cannot set clientid to sampleSubscribeService. The client ID is already set to sampleSubscribeService

The following configuration existed (as shown in the other blog) for the subscribe service:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
<bean id="sampleDurableTopicConnectionFactory"
    class="org.springframework.jms.connection.SingleConnectionFactory"
    p:targetConnectionFactory-ref="sampleMessagingConnectionFactory"
    p:clientId="sampleSubscribeService"/>

<bean id="sampleSubscribeServiceListenerContainer"  
    class="org.springframework.jms.listener.DefaultMessageListenerContainer"
    p:connectionFactory-ref="sampleDurableTopicConnectionFactory"
    p:destinationName="jmsServer1@com/foo/dest/SampleTopic"
    p:destinationResolver-ref="sampleDestinationResolver"
    p:messageListener-ref="sampleSubscribeServiceMessageListener"
    p:pubSubDomain="true"
    p:subscriptionDurable="true"
    p:durableSubscriptionName="sampleSubscribeService"
    p:transactionManager-ref="GlobalDataTransactionManager"/>

This included the configuration for the WebLogic Server ConnectionFactory:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
<connection-factory name="SampleConnectionFactory">
    <default-targeting-enabled>true</default-targeting-enabled>
    <jndi-name>com/foo/SampleConnectionFactory</jndi-name>
    <client-params>
        <client-id-policy>Restricted</client-id-policy>
        <subscription-sharing-policy>Exclusive</subscription-sharing-policy>
        <messages-maximum>15</messages-maximum>
    </client-params>
    <transaction-params>
        <xa-connection-factory-enabled>true</xa-connection-factory-enabled>
    </transaction-params>
    <security-params>
        <attach-jmsx-user-id>false</attach-jmsx-user-id>
    </security-params>
</connection-factory>

So in order to resolve this issue, there were a few necessary changes. The first change dealt with moving the clientId configuration out of Spring (which required the SingleConnectionFactory) and moving the clientId configuration into the WebLogic Server ConnectionFactory. It also required the modification of how the ConnectionFactory was configured with clientId properties. The configuration needed to change to add the clientId, allow the clientId to be duplicated, and change the subscription sharing policy. The JMS resource configuration changed the the following.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
<connection-factory name="SampleConnectionFactory">
    <default-targeting-enabled>true</default-targeting-enabled>
    <jndi-name>com/foo/SampleConnectionFactory</jndi-name>
    <client-params>
                <client-id>sampleClientId</client-id>
            <client-id-policy>Unrestricted</client-id-policy>
            <subscription-sharing-policy>Sharable</subscription-sharing-policy>
                <messages-maximum>15</messages-maximum>
    </client-params>
    <transaction-params>
        <xa-connection-factory-enabled>true</xa-connection-factory-enabled>
    </transaction-params>
    <security-params>
        <attach-jmsx-user-id>false</attach-jmsx-user-id>
    </security-params>
</connection-factory>

Also changed the forwarding-policy for Topics from Replicated to Partitioned.

There was one last item that needed to be configured because of the use of UDTs in WebLogic Server as opposed to default Topics. UDTs do not inherently support durable subscribes unless the WAR subscribes to an individual member of the cluster. This requires the WAR to subscribe to a specific jmsServer instance. The problem is when there is a multiple machine cluster (jmsServer1, jmsServer2, etc) and a WAR is deployed to the cluster, how do we specify the individual jmsServer instance in the Spring configuration. This approach would be solved by supplying a system level property in the domain scripts of WebLogic Server (there is a start script per Managed Server) and so each Managed Server script would export a system property of the name of the jmsServer.

1
2
3
4
5
JMS_SERVER_NAME=jmsServer1
export JMS_SERVER_NAME

EXTRA_JAVA_PROPERTIES="$ $ $ $ -DjmsServer.Name=$ -Dweblogic.debug.DebugDeploy -Dweblogic.debug.MasterDeployer -Dweblogic.debug.ApplicationContainer -Dcom.sun.management.jmxremote.authenticate=false"
export EXTRA_JAVA_PROPERTIES

Then, the Spring configuration for the DefaultMessageListenerContainer could simply inject that system property into the JMS destinationName property, as such:

1
2
3
4
5
6
7
8
9
10
<bean id="sampleSubscribeServiceListenerContainer"  
    class="org.springframework.jms.listener.DefaultMessageListenerContainer"
    p:connectionFactory-ref="sampleMessagingConnectionFactory"
    p:destinationName="${jmsServer.Name:jmsServer1}@com/foo/dest/SampleTopic"
    p:destinationResolver-ref="sampleDestinationResolver"
    p:messageListener-ref="sampleSubscribeServiceMessageListener"
    p:pubSubDomain="true"
    p:subscriptionDurable="true"
    p:durableSubscriptionName="sampleSubscribeService"
    p:transactionManager-ref="GlobalDataTransactionManager"/>

This resolved the duplicate clientId issues, allowed the WARs to work in a cluster against UDTs, and also correctly allowed the UDTs to load balance incoming messages to each instance in the cluster.

Share and Enjoy

WebLogic Password Decryption

In a previous blog post (WebLogic Encryption and Domains), I talked about how we can store the Salt files inside a Maven Project and be able to generate encrypted passwords that will be used on the server.

We can use that same project, and the code from this resource to allow the same project to decrypt passwords. This has been necessary when a password appears to get out of sync with another environment, and we can decrypt the password to verify what it was set to.

http://jagadesh4java.blogspot.com/2012/07/weblogic-password-decrypt.html

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
import weblogic.security.internal.SerializedSystemIni;
import weblogic.security.internal.ServerAuthenticate;
import weblogic.security.internal.encryption.ClearOrEncryptedService;
import weblogic.security.internal.encryption.EncryptionService;

public class Decrypt {
    static EncryptionService es = null;
    static ClearOrEncryptedService ces = null;

    public static void main(String args[]) {
        String s = null;
        if (args.length == 0) {
            s = ServerAuthenticate.promptValue("Password to decrypt: ", false);
        } else if (args.length == 1) {
            s = args[0];
        } else {
            System.err.println("Usage: java Decrypt [ password ]");
        }
        es = SerializedSystemIni.getExistingEncryptionService();
        if (es == null) {
            System.err.println("Unable to initialize encryption service");
            return;
        }
        ces = new ClearOrEncryptedService(es);
        if (s != null) {
            System.out.println("\nDecrypted Password is:" + ces.decrypt(s));
        }
    }
}

We can then add a profile for decrypt and add one for encrypt (which will be activated by default):

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
<profiles>
   <profile>
       <id>encrypt</id>
       <activation>
           <activeByDefault>true</activeByDefault>
        </activation>
       <properties>
           <security.name>Encryption</security.name>
           <security.class>weblogic.security.Encrypt</security.class>
       </properties>
   </profile>
   <profile>
       <id>decrypt</id>
       <properties>
           <security.name>Decryption</security.name>
           <security.class>com.foo.Decrypt</security.class>
       </properties>
   </profile>
</profiles>

And now we can just run a command of “mvn exec:exec -Pdecrypt” and we will be prompted to enter an encrypted password. I decrypted password will be returned to the console.

Share and Enjoy

Spring and WebLogic Durable Subscriber

Based on a previous post, it was evident that continuing with the use of DefaultMessageListenerContainer with Topics, that messages can be lost. In order to resolve this issue, it is necessary to utilize Spring’s configuration (along with WebLogic requirements) for configuring a durable subscriber. This is needed in order to guarantee that when the DefaultMessageListenerContainer disconnects based on the receiveTimeout, that messages are not lost. Durable subscriber will also guarantee that as multiple subscribers to the Topic are connected/disconnected, that they will still be able process messages received while they were disconnected.

To have a good understanding of durable and just general JMS concepts, a good older article to read is Guaranteed Messaging with JMS.

And before I get too much in the details, the definitions defined in this snippet (Persistent vs Durable JMS messages), helps define the difference between durable and persistent – just to make sure I don’t get the two confused:

A ” durable message ” is a message where the JMS server will hold on to a message if the subscriber is temporarily unavaliable. So the durability is defined by the relationship between a “Topc Subscriber” and the “JMS Server”. Durability is applicable only to publish/Subscribe paradigm. For this to happen subscribers need to register themselves with a unique ” client id “.

A ” persistent message ” is a message that defines the relationship between a “Message Producer” and the “JMS Server”. This can be established for both point-to-point and publish/subscribe. This has to do with the guaranteed once only delivery of the message by persisting the message after it has been recieved from the message producer.

The ActiveMQ documentation gives us a little more information as well(How do durable queues and topics work):

Durable queues keep messages around persistently for any suitable consumer to consume them. Durable queues do not need to concern themselves with which consumer is going to consume the messages at some point in the future. There is just one copy of a message that any consumer in the future can consume.

Durable topics however are different as they must logically persist an instance of each suitable message for every durable consumer – since each durable consumer gets their own copy of the message.

So I need to take the following configuration that I had for a subscribe service against a Topic and make it durable.

1
2
3
4
5
6
7
8
<bean id="sampleSubscribeServiceListenerContainer"  
    class="org.springframework.jms.listener.DefaultMessageListenerContainer"
    p:connectionFactory-ref="sampleMessagingConnectionFactory"
    p:destinationName="com/foo/dest/SampleTopic"
    p:destinationResolver-ref="sampleDestinationResolver"
    p:messageListener-ref="sampleSubscribeServiceMessageListener"
    p:pubSubDomain="true"
    p:transactionManager-ref="GlobalDataTransactionManager"/>

The interesting part is that quite a bit of time can be taken on Google trying to find information on how to do this and the Spring documentation does not really seem to describe what is necessary for durable subscription.

Here are a few resources with some details as to creating durable subscriber services via Spring:

Persistent JMS Topics using ActiveMQ and Spring

So based on this article and a few more resources, it is apparent that I need to add some additional pieces of configuration:

1
2
3
4
5
6
7
8
9
10
11
<bean id="sampleSubscribeServiceListenerContainer"  
    class="org.springframework.jms.listener.DefaultMessageListenerContainer"
    p:connectionFactory-ref="sampleMessagingConnectionFactory"
    p:destinationName="com/foo/dest/SampleTopic"
    p:destinationResolver-ref="sampleDestinationResolver"
    p:messageListener-ref="sampleSubscribeServiceMessageListener"
    p:pubSubDomain="true"
    p:subscriptionDurable="true"
    p:durableSubscriptionName="sampleSubscribeService"
    p:clientId="sampleSubscribeService"
    p:transactionManager-ref="GlobalDataTransactionManager"/>

So I add three pieces of configuration (subscriptionDurable, durableSubscriptionName, and clientId), and then I deploy this to my server. Interestingly enough, the deployment works but the application fails on startup with the following error:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
weblogic.jms.common.IllegalStateException: [JMSClientExceptions:055032]An attempt was made to create a named consumer (sampleSubscribeService) on a connection with no clientID
    at weblogic.jms.client.JMSSession.throwWhenInvalidSubscriberName(JMSSession.java:2770) ~[weblogic.jar:10.3.5.0]
    at weblogic.jms.client.JMSSession.setupConsumer(JMSSession.java:2723) ~[weblogic.jar:10.3.5.0]
    at weblogic.jms.client.JMSSession.createConsumer(JMSSession.java:2691) ~[weblogic.jar:10.3.5.0]
    at weblogic.jms.client.JMSSession.createDurableSubscriber(JMSSession.java:2487) ~[weblogic.jar:10.3.5.0]
    at weblogic.jms.client.WLSessionImpl.createDurableSubscriber(WLSessionImpl.java:1233) ~[weblogic.jar:10.3.5.0]
    at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.createConsumer(AbstractPollingMessageListenerContainer.java:493) ~[spring-full.jar:3.0.5]
    at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.createListenerConsumer(AbstractPollingMessageListenerContainer.java:223) ~[spring-full.jar:3.0.5]
    at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.doReceiveAndExecute(AbstractPollingMessageListenerContainer.java:307) ~[spring-full.jar:3.0.5]
    at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.receiveAndExecute(AbstractPollingMessageListenerContainer.java:243) ~[spring-full.jar:3.0.5]
    at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.invokeListener(DefaultMessageListenerContainer.java:1059) [spring-full.jar:3.0.5]
    at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.executeOngoingLoop(DefaultMessageListenerContainer.java:1050) [spring-full.jar:3.0.5]
    at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run(DefaultMessageListenerContainer.java:947) [spring-full.jar:3.0.5]
    at java.lang.Thread.run(Thread.java:662) [na:1.6.0_24]

So even though I defined a clientId within the configuration as is available through the Javadoc on DefaultMessageListenerContainer, I get hit with a clientID issue. If we go back to the ActiveMQ discussion we mentioned earlier, it gives some more information with regards to the importance of a clientID:

So for durable topic subscription, the JMS provider needs to be able to identify S when it shuts down and later on in the future reconnects, so it can know what messages to send to it while it was not running. JMS specification dictates that the identification of S is done by a combination of the clientID and the durable subscriber name. This is so that the JMS connection S uses can have many different durable subscriptions on different topics or on the same topic with different selectors – yet the JMS provider can know which message for which subscription to keep around for it.

So setting the clientID on a JMS connection is vital (along with using a sensible durable consumer name) for durable topic subscription.

So based upon this information, it appears that I need to have a clientId registered on the ConnectionFactory and not technically on the DefaultMessageListenerContainer (even though the API seems to allow it, it is not working based on the configuration I provided). So after quite a bit of searching, I found an article that I can’t seem to locate…but it points me towards SingleConnectionFactory which would allow me to wrap the ConnectionFactory that I am using and supply a clientId on that ConnectionFactory. Then I just utilize this new bean id (for the SingleConnectionFactory) in the configuration for my DefaultMessageListenerContainer.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
<bean id="sampleDurableTopicConnectionFactory"
    class="org.springframework.jms.connection.SingleConnectionFactory"
    p:targetConnectionFactory-ref="sampleMessagingConnectionFactory"
    p:clientId="sampleSubscribeService"/>

<bean id="sampleSubscribeServiceListenerContainer"  
    class="org.springframework.jms.listener.DefaultMessageListenerContainer"
    p:connectionFactory-ref="sampleDurableTopicConnectionFactory"
    p:destinationName="com/foo/dest/SampleTopic"
    p:destinationResolver-ref="sampleDestinationResolver"
    p:messageListener-ref="sampleSubscribeServiceMessageListener"
    p:pubSubDomain="true"
    p:subscriptionDurable="true"
    p:durableSubscriptionName="sampleSubscribeService"
    p:transactionManager-ref="GlobalDataTransactionManager"/>

So I passed in our JMS ConnectionFactory to the configuration of the SingleConnectionFactory, and moved the clientId configuration to it as well. Then I pass in the sampleDurableTopicConnectionFactory as the connectionFactory within the configuration for my DefaultMessageListenerContainer. I deploy this new web application and to my surprise, it still doesn’t work. I get the following error (which is the same error I get if I connect to a Uniform Distributed Topic (WLS) with HermesJMS:

1
2
3
4
5
6
7
8
weblogic.jms.common.JMSException: [JMSClientExceptions:055030]This topic does not support durable subscriptions
    at weblogic.jms.client.JMSSession.createDurableSubscriber(JMSSession.java:2482)
    at weblogic.jms.client.WLSessionImpl.createDurableSubscriber(WLSessionImpl.java:1233)
    at hermes.impl.TopicBrowser.getEnumeration(TopicBrowser.java:220)
    at hermes.browser.tasks.BrowseDestinationTask.invoke(BrowseDestinationTask.java:146)
    at hermes.browser.tasks.TaskSupport.run(TaskSupport.java:175)
    at hermes.browser.tasks.ThreadPool.run(ThreadPool.java:170)
    at java.lang.Thread.run(Unknown Source)

Now I mentioned this in a previous blog post, but with WebLogic Uniform Distributed Topics, you have to do special configuration within a Subscribe Service in order to use Durable Subscriber since WebLogic does not allow a Durable Subscriber against a Uniform Distributed Topic (unless you point to one of the specific members…by including a single JMS Server in the JNDI url…like jmsServer1@/com/foo/SampleBasicTopic). The reason being is explained in the WebLogic Server Documentation:

Note: Durable subscribers (DurableTopicSubscriber) cannot be created for distributed topics. However, you can still create a durable subscription on distributed topic member and the other topic members will forward the messages to the topic member that has the durable subscription.

So in order to make this work, I simply need to modify the destinationName to include the JMS Server name:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
<bean id="sampleDurableTopicConnectionFactory"
    class="org.springframework.jms.connection.SingleConnectionFactory"
    p:targetConnectionFactory-ref="sampleMessagingConnectionFactory"
    p:clientId="sampleSubscribeService"/>

<bean id="sampleSubscribeServiceListenerContainer"  
    class="org.springframework.jms.listener.DefaultMessageListenerContainer"
    p:connectionFactory-ref="sampleDurableTopicConnectionFactory"
    p:destinationName="jmsServer1@com/foo/dest/SampleTopic"
    p:destinationResolver-ref="sampleDestinationResolver"
    p:messageListener-ref="sampleSubscribeServiceMessageListener"
    p:pubSubDomain="true"
    p:subscriptionDurable="true"
    p:durableSubscriptionName="sampleSubscribeService"
    p:transactionManager-ref="GlobalDataTransactionManager"/>

I rebuild my war and deploy, and luckily this time, it works. I now have a durable subscriber. The interesting part is that as I start clicking around WebLogic console and look at the configured ConnectionFactory, I notice a Client Id configuration.

With this newly discovered piece of information, I find some more information that clarifies the role of the Client Id in Developing Advanced Pub/Sub Applications.

A subscription key is used to uniquely identify a subscription. For non-durable subscriptions, the key is composed of the Client ID and Client ID Policy. For durable subscriptions, the key is composed of the Client ID, Client ID Policy, and Subscription Name.

So this clarifies that I should just be able to set the Client Id on the ConnectionFactory within WebLogic and remove the configuration of the SingleConnectionFactory from my Spring configuration. So I went ahead and modified my ConnectionFactory by adding into the Client Id the value “sampleSubscribeService”, and then removed the SingleConnectionFactory configuration from my Spring Context file, rebuild/redeployed my web application, and my subscribe service works with this configuration as well. I can even go into the WebLogic Server console, click on the my JMS Resource (the Topic), go to Monitoring, and I should see a “Durable Subscribers” tab with the subscription name and client id that I defined.

Share and Enjoy