Unsolicited Bulk Verification

During my time as an administrator of email security and spam filtering, I have come across many different techniques and solutions for blocking spam. The ones I am familiar with are highlighted here: Anti-Spam Techniques. Some are more thorough, some are more accurate. I have thought about ways to combine the less accurate but thorough techniques into a single result of higher accuracy - and I came up with UBV.

UBV stands for Unsolicited Bulk Verification.

Let's take a look at the two parts that make this work: Bulk and Untrusted Sources.

Bulk is obvious, it comes in the form of newsletters and mass mailings. A lot of people get the same email. But how do we tell if it's legitimate or not? A lot of vendors block the bulk and whitelist the ones you want, while some vendors meticulously review bulk emails and err on the side of least false positives. The most commonly used database for bulk identification is the DCC - Distributed Checksum Clearinghouse. But just comparing an email to the DCC will not tell us if the email was legitimate or not.

Now the Untrusted Sources. This includes the use of Open Relay RBLs such as ORDB and SORBS. They are databases that list all known mail servers that allow anonymous relay of email (with or without the host's permission). Most of the time the host doesn't even know they are a relay! And that fact is why using these databases for lookups is risky in a corporate environment, when email communications are time sensitive and any failure of service can be an embarassment to the firm. This is why I have provided the Optimized RBL List. Other untrusted sources would include email coming from an IP without a registered dns record, or perhaps an RBL based on country location or even an entire ISP.

So what do we do now? We have two very effective ways of stopping spam, but they also come with high risk...

UBV is a logical AND of these two techniques. First determine where the email has originated, and if it's an untrusted source then flag it. Now that we have the email, look it up in the DCC or any other bulk database (Ciphertrust calls their's SLS). If it's bulk coming from an untrusted source, now we have verified that it is unsolicited (or a marketing company has an incompetent IT department).

Now we can safely delete the bulk of the bulk. But here is the magic - if a company innocently left their mail server open to spammers, only that sudden rush of viagara ads will get stopped. The legitimate company email will still make it to their destination unscathed!

I hope someone gives this a try and finds it helpful.