Smarter Email appender for Log4j with support of duplicate-removal, summary-report and JMX
Wednesday, March 17th, 2010I have been using SMTPAppender for a while to notify developers when something breaks on the production site and for most part it works well. However, due to some misconfiguration or service crash it can result in large number of emails. I was struck by similar problem at work when my email box suddently got tons of emails from the production site. So I decided to write a bit intelligent email appender. My goals for the appender were:
- Throttle emails based on some configured time
- Remove duplicate emails
- Support JMX for dynamic configuration
- Provide summary report with count of errors and their timings
I created FilteredSMTPAppender class that extends SMTPAppender. The FilteredSMTPAppender defines a nested class Stats for keeping track of errors. For each unique exception, it creates an instance of Stats, that stores the first and last occurrence of this exception as well as count. The Stats class uses hash of stack trace to identify unique exceptions, however it ignores first line, which often stores some dynamic information. FilteredSMTPAppender registers iteslf as MBean so that it can be configured at runtime. It overrides append method to capture the event and overrides checkEntryConditions to add filtering. It also changes the layout so that the summary count of error messages are added to the footer of email message.
The FilteredSMTPAppender uses a number of helper classes such as ServiceJMXBeanImpl for MBean definition, LRUSortedList to keep fixed cache of exceptions. Here is listing of LRUSortedList and ServiceJMXBeanImpl.
Listing of FilteredSMTPAppender.java
1 package com.plexobject.log; 2 3 import java.beans.PropertyChangeEvent; 4 import java.beans.PropertyChangeListener; 5 import java.util.Comparator; 6 import java.util.Date; 7 8 import javax.mail.MessagingException; 9 10 import org.apache.commons.lang.builder.EqualsBuilder; 11 import org.apache.commons.lang.time.FastDateFormat; 12 13 import org.apache.log4j.Layout; 14 import org.apache.log4j.net.SMTPAppender; 15 import org.apache.log4j.spi.LoggingEvent; 16 17 import com.plexobject.jmx.JMXRegistrar; 18 import com.plexobject.jmx.impl.ServiceJMXBeanImpl; 19 import com.plexobject.metrics.Metric; 20 import com.plexobject.metrics.Timer; 21 import com.plexobject.util.Configuration; 22 import com.plexobject.util.LRUSortedList; 23 24 public class FilteredSMTPAppender extends SMTPAppender { 25 26 private static final String SMTP_FILTER_MIN_DUPLICATE_INTERVAL_SECS = "smtp.filter.min.duplicate.interval.secs"; 27 private static final int MAX_STATS = Configuration.getInstance().getInteger("smtp.filter.max", 100); 28 private static int MIN_DUPLICATE_EMAILS_INTERVAL = Configuration.getInstance().getInteger(SMTP_FILTER_MIN_DUPLICATE_INTERVAL_SECS, 29 60); // 1 minute 30 private static final Date STARTED = new Date(); 31 private static final FastDateFormat DATE_FMT = FastDateFormat.getInstance("MM/dd/yy HH:mm"); 32 33 final static class Stats implements Comparable<Stats> { 34 35 final int checksum; 36 final long firstSeen; 37 long lastSeen; 38 long lastSent; 39 int numSeen; 40 int numEmails; 41 42 Stats(LoggingEvent event) { 43 StringBuilder sb = new StringBuilder(); 44 String[] trace = event.getThrowableStrRep(); 45 for (int i = 1; i < trace.length && i < 20; i++) { // top 20 lines 46 // of trace 47 sb.append(trace[i].trim()); 48 } 49 this.checksum = sb.toString().hashCode(); 50 firstSeen = lastSeen = System.currentTimeMillis(); 51 numSeen = 1; 52 } 53 54 boolean check() { 55 long current = System.currentTimeMillis(); 56 long elapsed = current - lastSent; 57 58 numSeen++; 59 lastSeen = current; 60 61 if (elapsed > MIN_DUPLICATE_EMAILS_INTERVAL * 1000) { 62 lastSent = current; 63 numEmails++; 64 return true; 65 } else { 66 return false; 67 } 68 } 69 70 @Override 71 public boolean equals(Object object) { 72 if (!(object instanceof Stats)) { 73 return false; 74 } 75 Stats rhs = (Stats) object; 76 return new EqualsBuilder().append(this.checksum, rhs.checksum).isEquals(); 77 78 } 79 80 @Override 81 public int hashCode() { 82 return checksum; 83 } 84 85 @Override 86 public String toString() { 87 return " (" + checksum + ") occurred " + numSeen + " times, " + numEmails + " # of emails, first @" + DATE_FMT.format(new Date(firstSeen)) + ", last @" + DATE_FMT.format(new Date(lastSeen)) + " since server started @" + DATE_FMT.format(STARTED); 88 } 89 90 @Override 91 public int compareTo(Stats other) { 92 return checksum - other.checksum; 93 } 94 } 95 96 final static class StatsCmp implements Comparator<Stats> { 97 98 @Override 99 public int compare(Stats first, Stats second) { 100 return first.checksum - second.checksum; 101 } 102 } 103 private static final LRUSortedList<Stats> STATS_LIST = new LRUSortedList<Stats>( 104 MAX_STATS, new StatsCmp()); 105 private LoggingEvent event; 106 private ServiceJMXBeanImpl mbean; 107 private Layout layout; 108 109 public FilteredSMTPAppender() { 110 mbean = JMXRegistrar.getInstance().register(getClass()); 111 mbean.addPropertyChangeListener(new PropertyChangeListener() { 112 113 @Override 114 public void propertyChange(PropertyChangeEvent event) { 115 try { 116 if (event != null && SMTP_FILTER_MIN_DUPLICATE_INTERVAL_SECS.equalsIgnoreCase(event.getPropertyName())) { 117 MIN_DUPLICATE_EMAILS_INTERVAL = Integer.parseInt((String) event.getNewValue()); 118 } 119 } catch (Exception e) { 120 e.printStackTrace(); 121 } 122 } 123 }); 124 125 } 126 127 public void append(LoggingEvent event) { 128 this.event = event; 129 if (layout == null) { 130 layout = getLayout(); 131 } 132 super.append(event); 133 } 134 135 protected boolean checkEntryConditions() { 136 final Timer timer = Metric.newTimer(getClass().getSimpleName() + ".checkEntryConditions"); 137 try { 138 boolean check = true; 139 if (event != null) { 140 Stats newStats = new Stats(event); 141 Stats stats = STATS_LIST.get(newStats); 142 if (stats == null) { 143 stats = newStats; 144 STATS_LIST.add(stats); 145 } else { 146 check = stats.check(); 147 } 148 if (check) { 149 setMessageFooter(stats); 150 } 151 } 152 return check && super.checkEntryConditions(); 153 } finally { 154 timer.stop(); 155 } 156 } 157 158 private void setMessageFooter(Stats stats) { 159 String message = event.getMessage().toString(); 160 161 final String footer = "\n\n-------------------------\n" + message + " - " + stats; 162 163 if (layout != null) { 164 setLayout(new Layout() { 165 166 @Override 167 public void activateOptions() { 168 layout.activateOptions(); 169 170 } 171 172 @Override 173 public String format(LoggingEvent evt) { 174 return layout.format(evt); 175 } 176 177 @Override 178 public String getFooter() { 179 return footer; 180 } 181 182 @Override 183 public boolean ignoresThrowable() { 184 return layout.ignoresThrowable(); 185 } 186 }); 187 } 188 } 189 } 190 191
Listing of ServiceJMXBeanImpl.java
1 package com.plexobject.util; 2 3 import java.util.ArrayList; 4 import java.util.Collection; 5 import java.util.Collections; 6 import java.util.Comparator; 7 import java.util.Iterator; 8 import java.util.List; 9 import java.util.ListIterator; 10 11 import org.apache.log4j.Logger; 12 13 14 public class LRUSortedList<T> implements List<T> { 15 private static final Logger LOGGER = Logger.getLogger(LRUSortedList.class); 16 private final int max; 17 private final Comparator<T> comparator; 18 19 private final List<Pair<Long, T>> list = new ArrayList<Pair<Long, T>>(); 20 private final List<Pair<Long, Integer>> timestamps = new ArrayList<Pair<Long, Integer>>(); 21 22 // comparator to sort by timestamp 23 private static final Comparator<Pair<Long, Integer>> CMP = new Comparator<Pair<Long, Integer>>() { 24 @Override 25 public int compare(Pair<Long, Integer> first, Pair<Long, Integer> second) { 26 if (first.getFirst() < second.getFirst()) { 27 return -1; 28 } else if (first.getFirst() > second.getFirst()) { 29 return 1; 30 } else { 31 return 0; 32 } 33 } 34 }; 35 36 public LRUSortedList(int max, Comparator<T> comparator) { 37 this.max = max; 38 this.comparator = comparator; 39 } 40 41 @Override 42 public boolean add(T e) { 43 if (list.size() > max) { 44 removeOldest(); 45 } 46 // add object 47 long timestamp = System.nanoTime(); 48 int insertionIdx = Collections.binarySearch(this, e, comparator); 49 if (insertionIdx < 0) {// not found 50 insertionIdx = (-insertionIdx) - 1; 51 list.add(insertionIdx, new Pair<Long, T>(timestamp, e)); 52 } else { 53 // found 54 list.set(insertionIdx, new Pair<Long, T>(timestamp, e)); 55 } 56 57 // as timestamps are sorted, we just remove the oldest (first) 58 if (timestamps.size() > max) { 59 timestamps.remove(0); 60 } 61 // update timestamp 62 Pair<Long, Integer> t = new Pair<Long, Integer>(timestamp, insertionIdx); 63 timestamps.add(t); 64 return true; 65 } 66 67 @Override 68 public void add(int index, T element) { 69 throw new UnsupportedOperationException( 70 "can't add element at arbitrary index, must use add to keep sorted order"); 71 } 72 73 @Override 74 public boolean addAll(Collection<? extends T> c) { 75 for (T e : c) { 76 add(e); 77 } 78 return c.size() > 0; 79 } 80 81 @Override 82 public boolean addAll(int index, Collection<? extends T> c) { 83 throw new UnsupportedOperationException( 84 "can't add element at arbitrary index, must use addAll to keep sorted order"); 85 } 86 87 @Override 88 public void clear() { 89 list.clear(); 90 } 91 92 @SuppressWarnings("unchecked") 93 @Override 94 public boolean contains(Object e) { 95 if (e == null) { 96 return false; 97 } 98 try { 99 return Collections.binarySearch(this, (T) e, comparator) >= 0; 100 } catch (ClassCastException ex) { 101 LOGGER.error("Unexpected type for contains " 102 + e.getClass().getName() + ": " + e); 103 return false; 104 } 105 } 106 107 @Override 108 public boolean containsAll(Collection<?> c) { 109 for (Object e : c) { 110 if (!contains(e)) { 111 return false; 112 } 113 } 114 return true; 115 } 116 117 @Override 118 public T get(int index) { 119 Pair<Long, T> e = list.get(index); 120 return e != null ? e.getSecond() : null; 121 } 122 123 public T get(Object e) { 124 int ndx = indexOf(e); 125 if (ndx >= 0) { 126 return get(ndx); 127 } 128 return null; 129 } 130 131 @SuppressWarnings("unchecked") 132 @Override 133 public int indexOf(Object e) { 134 try { 135 return Collections.binarySearch(this, (T) e, comparator); 136 } catch (ClassCastException ex) { 137 LOGGER.error("Unexpected type for get " + e.getClass().getName() 138 + ": " + e); 139 return -1; 140 } 141 } 142 143 @Override 144 public boolean isEmpty() { 145 return list.isEmpty(); 146 } 147 148 @Override 149 public Iterator<T> iterator() { 150 final Iterator<Pair<Long, T>> it = list.iterator(); 151 return new Iterator<T>() { 152 153 @Override 154 public boolean hasNext() { 155 return it.hasNext(); 156 } 157 158 @Override 159 public T next() { 160 Pair<Long, T> e = it.next(); 161 return e.getSecond(); 162 } 163 164 @Override 165 public void remove() { 166 it.remove(); 167 } 168 }; 169 } 170 171 @Override 172 public int lastIndexOf(Object o) { 173 for (int i = list.size() - 1; i >= 0; i--) { 174 T e = get(i); 175 if (e.equals(o)) { 176 return i; 177 } 178 } 179 return -1; 180 } 181 182 @Override 183 public ListIterator<T> listIterator() { 184 final ListIterator<Pair<Long, T>> it = list.listIterator(); 185 return buildListIterator(it); 186 } 187 188 @Override 189 public ListIterator<T> listIterator(int index) { 190 final ListIterator<Pair<Long, T>> it = list.listIterator(index); 191 return buildListIterator(it); 192 } 193 194 @SuppressWarnings("unchecked") 195 @Override 196 public boolean remove(Object e) { 197 try { 198 int ndx = Collections.binarySearch(this, (T) e, comparator); 199 if (ndx >= 0) { 200 remove(ndx); 201 return true; 202 } else { 203 return false; 204 } 205 206 } catch (ClassCastException ex) { 207 LOGGER.error("Unexpected type for remove " + e.getClass().getName() 208 + ": " + e); 209 return false; 210 } 211 } 212 213 @Override 214 public T remove(int index) { 215 Pair<Long, T> e = list.remove(index); 216 Pair<Long, Integer> t = new Pair<Long, Integer>(e.getFirst(), 0); 217 218 int insertionIdx = Collections.binarySearch(timestamps, t, CMP); 219 if (insertionIdx >= 0) { 220 timestamps.remove(insertionIdx); 221 } 222 return e != null ? e.getSecond() : null; 223 } 224 225 @Override 226 public boolean removeAll(Collection<?> c) { 227 boolean all = true; 228 for (Object e : c) { 229 all = all && remove(e); 230 } 231 return all; 232 } 233 234 @Override 235 public boolean retainAll(Collection<?> c) { 236 boolean changed = false; 237 Iterator<?> it = c.iterator(); 238 while (it.hasNext()) { 239 Object e = it.next(); 240 if (!contains(e)) { 241 it.remove(); 242 changed = true; 243 } 244 } 245 return changed; 246 } 247 248 @Override 249 public T set(int index, T element) { 250 throw new UnsupportedOperationException(); 251 } 252 253 @Override 254 public int size() { 255 return list.size(); 256 } 257 258 @Override 259 public List<T> subList(int fromIndex, int toIndex) { 260 List<T> tlist = new ArrayList<T>(); 261 List<Pair<Long, T>> plist = list.subList(fromIndex, toIndex); 262 for (Pair<Long, T> e : plist) { 263 tlist.add(e.getSecond()); 264 } 265 return tlist; 266 } 267 268 @Override 269 public Object[] toArray() { 270 return subList(0, list.size()).toArray(); 271 } 272 273 @SuppressWarnings("hiding") 274 @Override 275 public <T> T[] toArray(T[] a) { 276 return subList(0, list.size()).toArray(a); 277 } 278 279 @Override 280 public String toString() { 281 StringBuilder sb = new StringBuilder(); 282 Iterator<T> it = iterator(); 283 while (it.hasNext()) { 284 sb.append(it.next() + ", "); 285 } 286 return sb.toString(); 287 } 288 289 private void removeOldest() { 290 timestamps.remove(timestamps.size() - 1); 291 } 292 293 private ListIterator<T> buildListIterator( 294 final ListIterator<Pair<Long, T>> it) { 295 return new ListIterator<T>() { 296 297 @Override 298 public void add(T e) { 299 it.add(new Pair<Long, T>(System.nanoTime(), e)); 300 } 301 302 @Override 303 public boolean hasNext() { 304 return it.hasNext(); 305 306 } 307 308 @Override 309 public boolean hasPrevious() { 310 return it.hasPrevious(); 311 312 } 313 314 @Override 315 public T next() { 316 Pair<Long, T> e = it.next(); 317 return e.getSecond(); 318 } 319 320 @Override 321 public int nextIndex() { 322 return it.nextIndex(); 323 324 } 325 326 @Override 327 public T previous() { 328 Pair<Long, T> e = it.previous(); 329 return e.getSecond(); 330 } 331 332 @Override 333 public int previousIndex() { 334 return it.previousIndex(); 335 336 } 337 338 @Override 339 public void remove() { 340 it.remove(); 341 342 } 343 344 @Override 345 public void set(T e) { 346 it.set(new Pair<Long, T>(System.nanoTime(), e)); 347 348 } 349 }; 350 } 351 352 } 353 354
Listing of LRUSortedList.java
1 package com.plexobject.jmx.impl; 2 3 import java.beans.PropertyChangeListener; 4 import java.beans.PropertyChangeSupport; 5 import java.util.Map; 6 import java.util.concurrent.ConcurrentHashMap; 7 import java.util.concurrent.atomic.AtomicLong; 8 9 import javax.management.AttributeChangeNotification; 10 import javax.management.MBeanNotificationInfo; 11 import javax.management.Notification; 12 import javax.management.NotificationBroadcasterSupport; 13 import javax.management.NotificationListener; 14 15 import org.apache.commons.lang.builder.EqualsBuilder; 16 import org.apache.commons.lang.builder.HashCodeBuilder; 17 import org.apache.commons.lang.builder.ToStringBuilder; 18 import org.apache.log4j.Logger; 19 20 import com.plexobject.jmx.ServiceJMXBean; 21 import com.plexobject.metrics.Metric; 22 import com.plexobject.util.TimeUtils; 23 24 public class ServiceJMXBeanImpl extends NotificationBroadcasterSupport 25 implements ServiceJMXBean, NotificationListener { 26 private static final Logger LOGGER = Logger 27 .getLogger(ServiceJMXBeanImpl.class); 28 private Map<String, String> properties = new ConcurrentHashMap<String, String>(); 29 private final PropertyChangeSupport pcs = new PropertyChangeSupport(this); 30 31 private final String serviceName; 32 private AtomicLong totalErrors; 33 private AtomicLong totalRequests; 34 35 private AtomicLong sequenceNumber; 36 private String state; 37 38 public ServiceJMXBeanImpl(final String serviceName) { 39 this.serviceName = serviceName; 40 this.totalErrors = new AtomicLong(); 41 this.totalRequests = new AtomicLong(); 42 this.sequenceNumber = new AtomicLong(); 43 } 44 45 @Override 46 public double getAverageElapsedTimeInNanoSecs() { 47 return Metric.getMetric(getServiceName()) 48 .getAverageDurationInNanoSecs(); 49 } 50 51 public String getProperty(final String name) { 52 return properties.get(name); 53 } 54 55 public void setProperty(final String name, final String value) { 56 final String oldValue = properties.put(name, value); 57 final Notification notification = new AttributeChangeNotification(this, 58 sequenceNumber.incrementAndGet(), TimeUtils 59 .getCurrentTimeMillis(), name + " changed", name, 60 "String", oldValue, value); 61 sendNotification(notification); 62 handleNotification(notification, null); 63 } 64 65 @Override 66 public String getServiceName() { 67 return serviceName; 68 } 69 70 @Override 71 public long getTotalDurationInNanoSecs() { 72 return Metric.getMetric(getServiceName()).getTotalDurationInNanoSecs(); 73 } 74 75 @Override 76 public long getTotalErrors() { 77 return totalErrors.get(); 78 } 79 80 public void incrementError() { 81 final long oldErrors = totalErrors.getAndIncrement(); 82 final Notification notification = new AttributeChangeNotification(this, 83 sequenceNumber.incrementAndGet(), TimeUtils 84 .getCurrentTimeMillis(), "Errors changed", "Errors", 85 "long", oldErrors, oldErrors + 1); 86 sendNotification(notification); 87 } 88 89 @Override 90 public long getTotalRequests() { 91 return totalRequests.get(); 92 } 93 94 public void incrementRequests() { 95 final long oldRequests = totalRequests.getAndIncrement(); 96 final Notification notification = new AttributeChangeNotification(this, 97 sequenceNumber.incrementAndGet(), TimeUtils 98 .getCurrentTimeMillis(), "Requests changed", 99 "Requests", "long", oldRequests, oldRequests + 1); 100 sendNotification(notification); 101 } 102 103 @Override 104 public MBeanNotificationInfo[] getNotificationInfo() { 105 String[] types = new String[] { AttributeChangeNotification.ATTRIBUTE_CHANGE }; 106 String name = AttributeChangeNotification.class.getName(); 107 String description = "An attribute of this MBean has changed"; 108 MBeanNotificationInfo info = new MBeanNotificationInfo(types, name, 109 description); 110 111 return new MBeanNotificationInfo[] { info }; 112 } 113 114 @Override 115 public String getState() { 116 return state; 117 } 118 119 /** 120 * @param state 121 * the state to set 122 */ 123 public void setState(String state) { 124 this.state = state; 125 } 126 127 /** 128 * @see java.lang.Object#equals(Object) 129 */ 130 @Override 131 public boolean equals(Object object) { 132 if (!(object instanceof ServiceJMXBeanImpl)) { 133 return false; 134 } 135 ServiceJMXBeanImpl rhs = (ServiceJMXBeanImpl) object; 136 return new EqualsBuilder().append(this.serviceName, rhs.serviceName) 137 .isEquals(); 138 } 139 140 /** 141 * @see java.lang.Object#hashCode() 142 */ 143 @Override 144 public int hashCode() { 145 return new HashCodeBuilder(786529047, 1924536713).append( 146 this.serviceName).toHashCode(); 147 } 148 149 /** 150 * @see java.lang.Object#toString() 151 */ 152 @Override 153 public String toString() { 154 return new ToStringBuilder(this) 155 .append("serviceName", this.serviceName).append("totalErrors", 156 this.totalErrors).append("totalRequests", 157 this.totalRequests).append("totalRequests", 158 this.totalRequests).append("state", this.state).append( 159 "properties", this.properties).toString(); 160 } 161 162 public void addPropertyChangeListener(PropertyChangeListener pcl) { 163 pcs.addPropertyChangeListener(pcl); 164 } 165 166 public void removePropertyChangeListener(PropertyChangeListener pcl) { 167 pcs.removePropertyChangeListener(pcl); 168 169 } 170 171 @Override 172 public void handleNotification(Notification notification, Object handback) { 173 LOGGER.info("Received notification: ClassName: " 174 + notification.getClass().getName() + ", Source: " 175 + notification.getSource() + ", Type: " 176 + notification.getType() + ", tMessage: " 177 + notification.getMessage()); 178 if (notification instanceof AttributeChangeNotification) { 179 AttributeChangeNotification acn = (AttributeChangeNotification) notification; 180 pcs.firePropertyChange(acn.getAttributeName(), acn.getOldValue(), 181 acn.getNewValue()); 182 183 } 184 } 185 } 186 187
Testing
Finally, here is how you can test this filter:
1 package com.plexobject; 2 3 import java.net.InetAddress; 4 import java.util.Date; 5 6 import org.apache.log4j.Logger; 7 import org.apache.log4j.PatternLayout; 8 import org.apache.log4j.net.SMTPAppender; 9 10 import com.plexobject.log.FilteredSMTPAppender; 11 12 public class Main { 13 private static final Logger LOGGER = Logger.getLogger(Main.class); 14 public static void main(String[] args) { 15 SMTPAppender appender = new FilteredSMTPAppender(); 16 try { 17 appender.setTo("bhatti@xxx.com"); 18 appender.setFrom("bhatti@xxx.com"); 19 appender.setSMTPHost("smtp.xxx.net"); 20 appender.setLocationInfo(true); 21 appender.setSubject("Error from " + InetAddress.getLocalHost()); 22 23 appender.setLayout(new PatternLayout()); 24 appender.activateOptions(); 25 LOGGER.addAppender(appender); 26 } catch (Exception e) { 27 LOGGER.error("Failed to register smtp appender", e); 28 } 29 while (true) { 30 try { 31 throw new Exception("throwing exception at " + new Date()); 32 } catch (Exception e) { 33 LOGGER.error("Logging error at " + new Date(), e); 34 } 35 try { 36 Thread.sleep(1000); 37 } catch (InterruptedException e) { 38 Thread.interrupted(); 39 } 40 } 41 } 42 } 43 44
Above code simulates error generation every second, but it sends email based on the throttling level defined in the configuration. Obviously you can use log4j properties file to define all this configuration, e.g.
<!– Send email when error happens –>
<appender name=”APP-EMAIL” class=”com.plexobject.log.FilteredSMTPAppender”>
<param name=”BufferSize” value=”256″ />
<param name=”SMTPHost” value=”smtp.xxx.net” />
<param name=”From” value=”bhatti@xxx.com” />
<param name=”To” value=”bhatti@xxx.com” />
<param name=”Subject” value=”Production Error” />
<layout class=”org.apache.log4j.PatternLayout”>
<param name=”ConversionPattern”
value=”[%d{ISO8601}]%n%n%-5p%n%n%c%n%n%m%n%n” />
</layout>
<filter class=”org.apache.log4j.varia.StringMatchFilter”>
<param name=”StringToMatch” value=”My Error”/>
<param name=”AcceptOnMatch” value=”false” />
</filter>
</appender>
Summary
I am skipping other classes, but you can download entire code from FilteredSMTPAppender.zip. This solution seems to be working from me but feel free to share your experience with similar problems.
