Messaging based asynchronous processing is a key component of any complexed software especially in transactional environment. There are a number of solutions that provide high performance and reliable messaging in Java space such as ActiveMQ, FUSE broker, JBossMQ, SonicMQ, Weblogic, Websphere, Fiorano, etc. These providers support JMS specification, which provides abstraction for queues, message providers and message consumers. In this blog, I will go over some recipes for recovering messages from dead letter queue when using ActiveMQ.
What is Dead Letter Queue
Generally, when a consumer fails to process a message within a transaction or does not send acknowledgement back to the broker, the message is put back to the queue. The message is then delivered upto certain number of times based on configuration and finally the message is put to dead letter queue when that limit is exceeded. The ActiveMQ documentation recommends following settings for defining dead letter queues:
<broker...> <destinationPolicy> <policyMap> <policyEntries> <!-- Set the following policy on all queues using the '>' wildcard --> <policyEntry queue=">"> <deadLetterStrategy> <individualDeadLetterStrategy queuePrefix="DLQ." useQueueForQueueMessages="true" /> </deadLetterStrategy> </policyEntry> </policyEntries> </policyMap> </destinationPolicy> ... </broker>and you can control redelivery policy as follows:
RedeliveryPolicy policy = connection.getRedeliveryPolicy();policy.setInitialRedeliveryDelay(500);policy.setBackOffMultiplier(2);policy.setUseExponentialBackOff(true);policy.setMaximumRedeliveries(2);It is important that you create dlq per queue, otherwise ActiveMQ puts them into a single dead letter queue.
Handle QueueViewMBean
ActiveMQ provides QueueViewMBean to invoke administration APIs on the queues. The easiest way to get this handle is to use BrokerFacadeSupport class, which is extended by RemoteJMXBrokerFacade and LocalBrokerFacade. You can use RemoteJMXBrokerFacade if you are connecting to remote ActiveMQ server, e.g. here is Spring configuration for setting it up:
<bean id="brokerQuery" class="org.apache.activemq.web.RemoteJMXBrokerFacade" autowire="constructor" destroy-method="shutdown"> <property name="configuration"> <bean class="org.apache.activemq.web.config.SystemPropertiesConfiguration"/> </property> <property name="brokerName"> <null/> </property></bean>Alternatively, you can use LocalBrokerFacade if you are running embedded ActiveMQ server, e.g. below is Spring configuration for it:
<bean id="brokerQuery" class="org.apache.activemq.web.LocalBrokerFacade" autowire="constructor" scope="prototype"/>Getting number of messages from the queue
Once you got handle to QueueViewMBean, you can use following API to find the number of messages in the queue:
public long getQueueSize(final String dest) { try { return brokerQuery.getQueue(dest).getQueueSize(); } catch (Exception e) { throw new RuntimeException(e); }}Copying Messages using JMS APIs
The JMS specification provides APIs to browse queue in read mode and then you can send the messages to another queue, e.g.
import org.apache.activemq.web.BrokerFacadeSupport;import org.springframework.beans.factory.annotation.Autowired;import org.springframework.jms.core.JmsTemplate;import org.springframework.jms.core.MessageCreator;import javax.jms.*;import java.util.Enumeration;public class DlqReprocessor { private JmsTemplate jmsTemplate; BrokerFacadeSupport brokerQuery; ConnectionFactory connectionFactory; ("unchecked") void redeliverDLQUsingJms( final String brokerName, final String from, final String to) { Connection connection = null; Session session = null; try { connection = connectionFactory.createConnection(); connection.start(); session = connection.createSession(false, Session.AUTO_ACKNOWLEDGE); Queue dlq = session.createQueue(from); QueueBrowser browser = session.createBrowser(dlq); Enumeration<Message> e = browser.getEnumeration(); while (e.hasMoreElements()) { Message message = e.nextElement(); final String messageBody = ((TextMessage) message).getText(); jmsTemplate.send(to, new MessageCreator() { public Message createMessage(final Session session) throws JMSException { return session.createTextMessage(messageBody); } }) } } catch (Exception ex) { throw new RuntimeException(ex); } finally { try { session.close(); } catch (Exception ex) { } try { connection.close(); } catch (Exception ex) { } } }}The downside of above approach is that it leaves the original messages in the dead letter queue.
Copying Messages using Spring’s JmsTemplate APIs
You can effectively do the same thing with JmsTemplate provided by Spring with a bit less code, e.g.
void redeliverDLQUsingJmsTemplateBrowse( final String from, final String to) { try { jmsTemplate.browse(from, new BrowserCallback() { ("unchecked") public Object doInJms(Session session, QueueBrowser browser) throws JMSException { Enumeration<Message> e = browser.getEnumeration(); while (e.hasMoreElements()) { Message message = e.nextElement(); final String messageBody = ((TextMessage) message).getText(); jmsTemplate.send(to, new MessageCreator() { public Message createMessage(final Session session) throws JMSException { return session.createTextMessage(messageBody); } }); } return null; } }); } catch (Exception e) { throw new RuntimeException(e); } }Moving Messages using receive/send APIs
As I mentioned, the above approaches leave messages in the DLQ, which may not be what you want. Thus, another simple approach would be to consume messages from the dead letter queue and send it to another,e.g.
public void redeliverDLQUsingJmsTemplateReceive( final String from, final String to) { try { jmsTemplate.setReceiveTimeout(100); Message message = null; while ((message = jmsTemplate.receive(from)) != null) { final String messageBody = ((TextMessage) message).getText(); jmsTemplate.send(to, new MessageCreator() { public Message createMessage(final Session session) throws JMSException { return session.createTextMessage(messageBody); } }); } } catch (Exception e) { throw new RuntimeException(e); } }Moving Messages using ActiveMQ’s API
Finally, the best approach I found waas to use ActiveMQ’s APIs to move messags, e.g.
public void redeliverDLQUsingJMX( final String brokerName, final String from, final String to) { try { final QueueViewMBean queue = brokerQuery.getQueue(from); for (int i = 0; i < 10 && queue.getQueueSize() > 0; i++) { CompositeData[] compdatalist = queue.browse(); for (CompositeData cdata : compdatalist) { String messageID = (String) cdata.get("JMSMessageID"); queue.moveMessageTo(messageID, to); } } } catch (Exception e) { throw new RuntimeException(e); } }I have been using this approach and have found to be reliable for reprocessing dead letter queue, though these techniques an also be used for general queues. I am sure there are tons of alternatives including using full-fledged enterprise service bus route. Let me know if you have interesting solutions to this problem.