Load Balancing with Postfix [and SpamAssassin]
Michael L Torrie
torriem at chem.byu.edu
Fri Jan 20 09:32:35 MST 2006
On Fri, 2006-01-20 at 06:49 -0700, Josh Coates wrote
> i'm not an email-original-designer-intention-historian, so please correct me
> if i'm mistaken.
I can't say if or where you are mistaken about the history of and the
RFCs surrounding e-mail, but it appears e-mail was designed to be "best
effort." That is all. There is no QoS for e-mail. E-mail was
certainly designed with the fact that the internet was (and still is)
very fragile and routes and servers can go down with very little notice
at any time, at least temporarily. If we want to make poor analogies,
e-mail is probably more like the post of an earlier era, where mail gets
through most of the time, but sometimes pieces are lost permanently and
there are no means of tracing the messages, and finding out what
happened to them.
This best effort design is a very important fact, because, unlike other
internet services, this allows me to take an entire e-mail server down
at any time for emergency maintenance without worrying about lost e-
mails or disruption of service. Mail messages still get enqueued at
various servers along the way. Pretty ingenious really. For various
reasons, scheduled and unscheduled, I've actually had a large e-mail
system off for an entire night. Once the server is started again, e-
mail that was blocked during this time starts coming in within a few
minutes (ie the delay is not long at all). By some metrics, this would
indicate that e-mail is reliable, but certainly it was never designed
with a built-in QoS mechanism.
E-mail may evolve into something reliable and with strict and rigid QoS
(2nd-day guaranteed overnight airborne delivery for example), but there
are some fundamental freeloader problems that will have to be solved
before it will (would you be willing to pay to send e-mail?).
In the meantime, spam will either be dealt with in a variety of ever-
expanding and creative means, or e-mail will have to be abandoned
entirely (there are those that have already given up on it).
Technological measure have advanced pretty rapidly, allowing e-mail to
still be useful today, but what about 10 years from now? Maybe by then
bandwidth and CPU power will be so cheap that we'll have artificially
intelligent e-mail readers that can read all our mail for us and just
let us see the ones it knows we're interested in (throwing away all
those forwards from that random family friend that mails you along with
2000 other e-mail addresses all in the "To:" field).
The problem with dpsam or spamassassin alone today is that it doesn't
scale (ie linearly) in terms of resources required. It might if you
throw exponentially increasing amounts of hardware at the problem, but
that's not always economical. It seems kind of silly to require a small
super-computing cluster of 16-way 64-bit AMD blade servers to handle
what is really just document storage and transfer. Besides all that,
the bandwidth wasted with spam e-mail (which spamassassin cannot
address) is staggering. BYU estimates well over half the incoming e-
mail (before filtering) is spam and is summarily filtered silently after
being received. This spam filtering load regularly introduces multiple
hour delays (up to 12 hours on rare days) into the receiving of e-mail.
I don't believe this is ineptness on BYU's part (yes you did hear me
right). It's an illustration of the mess that e-mail has become. Now
if we are going to talk about delays that cause problems, I'd say that
would be one of them, moreso than a 20-minute occasional message delay
due to the gray list.
I'm not pushing gray listing as a be-all solution. I am just trying to
point out the fallacies (can't resist an argument even if I don't really
care that much about my point of view) in dismissing it out of hand by
making unrealistic statements about the delay or disruption it would
cause. Based on the experiences I've been hearing about from people
who've actually implemented it, rather than just think about all the
problems it might cause (even at some larger installations), it is a
successful strategy that doesn't have a large impact compared to the
problems it solves.
More information about the PLUG