Spring DefaultMessageListenerContainer and Loss of Topic Messages

As I start developing JMS based services, I need to determine how I will architecture consumers and what types of performance best practices should be considered. First I took the time to read through the Spring Documentation and Spring Recipes book, but then read through these next two posts:

Using Spring to Send JMS Messages
Using Spring to Receive JMS Messages

There are some excellent details in this article and after evaluation of the multiple implementation types for Spring based consumers, I decided to use DefaultMessageListenerContainer with the intregration of MessageListener. I also have needs for my consumers to have transactionality, in some cases I want the transaction to participate in a JTA and in other cases I want some more control over rollbacks and message handling by implementing JMS based transactions. Searching for some general understanding of JMS Performance Best Practices also brings to light the needs around certain configuration options. I took a look at articles like “JMS Performance like Best practices to improve performance in JMS“, which will talk about understanding when to and not to use persistent messages.

It seems like JMS has a plethora of configurations that really are specific to the unique case of each implementation of a JMS consumer, but for now, I will build DefaultMessageListenerContainers with most of the default capabilities regardless if I am listening to Queues or Topics.

Below is an example of a default DefaultMessageListenerContainer for a Queue

1
2
3
4
5
6
7
8
9
10
<bean id="sampleSubscribeServiceListenerContainer"  
    class="org.springframework.jms.listener.DefaultMessageListenerContainer"
    p:connectionFactory-ref="sampleMessagingConnectionFactory"
    p:destinationName="com/foo/dest/SampleQueue"
    p:destinationResolver-ref="sampleDestinationResolver"
    p:messageListener-ref="sampleSubscribeServiceMessageListener"
    p:transactionManager-ref="GlobalDataTransactionManager"/>
or
    <!-- Use instead of transactionManager-ref for JMS Transactions-->
    p:sessionTransacted="true"/>

And here is an example of a DefaultMessageListenerContainer for a Topic

1
2
3
4
5
6
7
8
9
10
11
<bean id="sampleSubscribeServiceListenerContainer"  
    class="org.springframework.jms.listener.DefaultMessageListenerContainer"
    p:connectionFactory-ref="sampleMessagingConnectionFactory"
    p:destinationName="com/foo/dest/SampleTopic"
    p:destinationResolver-ref="sampleDestinationResolver"
    p:messageListener-ref="sampleSubscribeServiceMessageListener"
    p:pubSubDomain="true"
    p:transactionManager-ref="GlobalDataTransactionManager"/>
or
    <!-- Use instead of transactionManager-ref for JMS Transactions-->
    p:sessionTransacted="true"/>

Everything was working perfectly until the amount of messages that need to be consumed increased and seemed to overwhelm the consumers of Topics, because messages were starting to get lost. For instance, I could either create a JMS Client or use HermesJMS to send a message to a Topic and then watch the server logs (as well as attach to the Topic via HermesJMS). I could essentially send 10 messages through and HermesJMS would receive all 10 and the consumer on the server would only show 9. The servers/apps were never restarted, so initially the concern wasn’t with regards to whether the messages were persistent or not. But once I searched for “defaultmessagelistenercontainer lose message topic”, I started receiving more details about the architecture of the different types of Spring containers.

Many discussions or forum postings simply told the developer that the consumer wasn’t working because it was not durable, but did that mean I couldn’t have a Topic consumer that didn’t have to be durable?

http://stackoverflow.com/questions/10851064/losing-messages-from-a-topic-with-defaultmessagelistenercontainer

Then I found a JIRA entry related to exactly the issue that was occurring:

https://jira.springsource.org/browse/SPR-7883

This gave a little more detail, but the true clarity was with the following forum post:

http://forum.springsource.org/showthread.php?33037-What-s-the-best-practice-for-using-JMS-in-spring

While DefaultMessageListenerContainer does use a pull approach behind the scenes, it is still considered a recommendable approach in many environments. After all, any kind of listening approach always boils down to some kind of receive loop on the underlying socket connection; it’s only really about the level at which it happens.

DefaultMessageListenerContainer’s advantage is the level of control that it gives, in particular with respect to thread management and transaction demarcation. DMLC is the only listener container that does not impose the thread management onto the JMS provider, that is, does not use/block JMS provider threads. It is also able to gracefully recover from JMS provider failure, such as connection loss. Furthermore, as noted, it is the only variant that supports external transaction managers, in particular for XA transactions.

DefaultMessageListenerContainer is also the only listener container variant that is compatible with both managed JMS and non-managed JMS, i.e. JMS in a J2EE environment as well as standalone JMS usage, since it sticks with standard JMS API that is compatible with all of J2EE’s JMS restrictions. SimpleMessageListenerContainer on the other hand uses “Session.setMessageListener”, not supported in a J2EE environment.

The only issue in a J2EE environment is thread management: On WebLogic and WebSphere, we recommend to specify a CommonJ WorkManagerTaskExecutor, which makes DMLC delegate to a server-managed thread pool, resulting in fully integrated thread management. Alternatively, simply stick with the default SimpleAsyncTaskExecutor, which works fine as well in many cases. (It will work just as well as Quartz does…)

Regarding transaction management: If you specify a “transactionManager” on DefaultMessageListenerContainer, it will wrap its receive loop in a transaction. This is typically used with JtaTransactionManager, where no transactional resources are actually bound: It’s only really marking the thread (and the listener’s JMS Session) as “XA-active” and waiting for something to happen. This should be pretty efficient with any decent JTA provider.

As a consequence, DefaultMessageListenerContainer is the variant to use in a J2EE environment, and also a primary choice for native JMS usage. It is often used with native JMS providers such as Tibco, talking to an external broker process (even when running in a J2EE server), typically without XA transactions. Alternatively, consider using SimpleMessageListenerContainer, but only for native JMS usage without XA, and only if your JMS provider gracefully handles thread management and connection recovery.

FWIW, there is a further approach for JMS listening, for native JMS usage only but with the provider pushing incoming messages – and with full XA support: the JCA 1.5 message endpoint mechanism. It requires a JMS 1.1 provider that ships a JCA 1.5 connector (e.g. ActiveMQ), running in a local JCA container within the application. This is the approach that the Jencks project (http://jencks.org) takes, as a third-party add-on to Spring: worth considering if XA transactions are needed, as alternative to DMLC.

-Juergen Hoeller

Bingo, now it all makes sense. So the core issue is not really the durable/non-durable, it is around the type of consumer listener (one that polls or one that pushes). And since I was using the default settings and DefaultMessageListenerContainer, the thread for the DefaultMessageListenerContainer would wake up and stay alive for a second and if it did not find a message, it would close and be recycled. So this happens every single second because of the default of the “receiveTimeout”. When this dropping of the connection happens, there is a potential for a message to be lost because at that split second or so, there is no consumer subscribed to the Topic (which is why the durable fixes this issue).

But this now exposes an interesting concept. In the case that an XA Transaction is being used, the XA AOP Pointcuts have been designed without a timeout to prevent run away threads. One of the suggestions in the discussion is to put a negative value in the “receiveTimeout”, but the problem is that this would potentially result in XA Timeouts anytime the thread didn’t find a message within the allotted XA timeout configuration. So to alleviate this problem, I will set the “receiveTimeout” to the XA AOP timeout (which for these consumers is 180 seconds). This will not fix the loss of messages, because at the end of the 180 seconds when the connection is recycled, I could still lose messages. But this will save the resources of reconnecting every single second waiting for a message. The only thing I need to be concerned is how do I configure the “receiveTimeout” for a Topic versus a Queue. Topics should essentially only have one concurrent consumer (otherwise they could potentially receive a message multiple times), but a Queue can have multiple concurrent consumers (to assist with performance). I just now need to be aware that if I set the “receiveTimeout” to 180 seconds and the concurrentConsumers=”5″ and happen to have 5 different DefaultMessageListenerContainers, then I potentially have a minimum of 25 threads waiting around for 180 seconds. So I just need to make sure I understand how many potential threads could be opened across my domain and any potential resource issues.

So in order to complete the configuration of a Topic consumer so that it does not lose messages, I will need to add configuration to define what is necessary to make it durable. This will be covered in another blog post.

crazy mass cutting stack

Share and Enjoy