The Anti Spam Challenge – Minimizing False Positives

by: Paul Judge, CTO, CipherTrust, Inc.
Email is the quintessential business communication tool, so when it doesn’t work like it’s supposed to, business suffers. Anti spam software is designed to protect your inbox from unwanted messages, but unless your system is properly trained even the best software misses the mark and flags legitimate messages as spam. These messages are referred to as “false positives.” While consumer and ISP anti spam products focus on blocking messages and even consider some false positives acceptable, businesses require anti spam solutions that treat their messages as very valuable. Failing to receive critical messages in a timely fashion can do irreparable damage to customer and partner relationships and cause important orders to be missed, so eliminating false positives while maintaining high anti spam accuracy is paramount to any enterprise anti spam solution. What causes false positives? Different anti spam solutions utilize different methods of detecting and blocking spam. Anti spam software typically uses content filtering or Bayesian Logic, an advanced content filtering method, to score each email, looking for certain tell-tale signs of spammer habits such as frequently used terms like “Viagra” or “click here.” Other anti spam solutions reference blacklists and whitelists to determine whether the sender has shown spammer tendencies in the past. A false positive can occur when a legitimate sender raises enough red flags, either by using too many “spam terms” or sending from an IP address that has been used by spammers in the past. Minimizing False Positives Although it takes a person only a moment to process a message and identify it as spam, it is difficult to automate that human process because no single message characteristic consistently identifies spam. In fact, there are hundreds of different message characteristics that may indicate an email is spam, and an effective anti spam solution must be capable of employing multiple spam detection techniques to effectively cover all bases. A comprehensive anti spam approach involves examining both message content and sender history in tandem. By using a reputation system to evaluate senders based on their past behavior, a more accurate picture of their intentions and legitimacy can be discerned, and a solution’s false positive rate can be further lowered. Has the sender engaged in spamming, virus distribution or phishing attacks in the past? If not, the likelihood of their message getting past the email gateway just went up, and the chances of a false positive declined accordingly. If they have, an effective reputation system knows and flags the message. Self-Optimization In order to be most effective, anti spam solutions must learn based on a recipient’s preferences. While most of us prefer not to receive emails containing the term Viagra, some medical organizations might need to receive these emails in order to process patient data. In order to best learn your organizational preferences, anti spam solutions should put filtered emails into a quarantine that allows users to review and make decisions as to whether a particular message is spam. Making this quarantine available to the end-user lowers the administration costs and increases the accuracy of the anti spam system. Each time a user makes a decision about whether a particular email is or is not spam, the system becomes more personalized and intelligent about filtering email for that individual in the future. Over time, users find that they rarely need to review their quarantines anymore because the system has learned how to identify messages that are important to that user. Don’t throw the baby out with the bathwater In conclusion, it is imperative that false positives be kept to an absolute minimum for business users. Although consumers may have more patience with incorrectly blocked email, businesses cannot afford these types of problems. An effective, accurate anti spam solution aggregates multiple spam detection technologies, combining the benefits of each individual technique to stop spam while minimizing false positives. It also puts suspected spam into a quarantine that is available to end-users, and learns how to better identify spam in the future.