This is a guest post by Albin Sunnanbo, introducing a burst filter preventing log4net from filling up a support mailbox.

One Friday afternoon, just before leaving for the weekend, I glanced at our support mail inbox.
The unread counter was literally spinning. Three mails a second.

To make a long story short we had two problems.

1. A lost db connection due to a DNS failure.
2. A mail logger that tried to nuke Outlook into oblivion. A mail logger that took a while to stop because we could not reach our server due to the aforementioned DNS failure.

This incident taught us two lessons. Write down the IP addresses of all your servers and throttle your logging.
That project mentioned above used Enterprise Library Logging block.
When starting a new project I decided to try log4net for logging. I also decided to not repeat previous mistakes.

Kentor.Log4NetExtensions.BurstFilter

When starting with log4net I wanted to implement log throttling in the logging framework to be able to have different throttling levels for log files and emails.
I found no suitable solution for log4net, but log4j has a BurstFilter. I borrowed the idea and implemented a similar filter for log4net.

The code is available at github.com/KentorIT/Log4NetExtensions. And as the log4net.BurstFilter NuGet package.

<log4net>
    <appender name="SMTPAppender" type="log4net.Appender.SmtpAppender,log4net">
      <bufferSize value="1"/>
      <authentication value="Basic" />
      <to value="to@example.com" />
      <from value="from@example.com" />
      <subject value="Automatmail: system X is totally broken, please fix" />
      <smtpHost value="smtp.example.com" />
      <username value="username"/>
      <password value="password"/>
      <lossy value="false" />
      <threshold value="ERROR"/>
      <filter type="Kentor.Log4NetExtensions.BurstFilter,Kentor.Log4NetExtensions.BurstFilter">
        <!-- Allow two mails a minute on average, with bursts up to 20 mails -->
        <BurstLength value="00:10:00"/>
        <BurstSize value="20"/>
      </filter>
      <layout type="log4net.Layout.PatternLayout">
        <conversionPattern value="%property{log4net:HostName}%newline%date user:%aspnet-request{AUTH_USER} %-5level %logger %newline %message%newline%exception%newline%newline"/>
      </layout>
    </appender>
    <root>
      <level value="ALL" />
      <appender-ref ref="SMTPAppender" />
    </root>
</log4net>

Some Notes About Mail Alerts

Over the years I have found email alerts a very effective way to deal with major production incidents in line of business applications. We usually send mails from the production environment directly to our support mailbox. This way we get the fastest possible attention to production incidents. We have been able to diagnose and fix many incidents before any end users have contacted our support.

I usually send emails for all log messages with level ERROR or higher. Sometimes I “upgrade” an event to ERROR because I want to monitor if it happens even if it is strictly not an ERROR.
When setting up mail logging in an existing project I review the used logging levels used to make sure all “interesting” events have ERROR level or higher and all informative logging have lower levels.