VIPUL'S RAZOR

aims to give greatest network coverage, 2. C. Potter et al, “DTI Information About the author
careful tuning is usually necessary to Security Breaches Survey 2006”, April
ensure that alerts do not overwhelm 2006, www.pwc.com/extweb/pwcpub- Tom Rowan is the lead security consult-
administrators. Once implemented, lications.nsf/docid/ ant for the Magirus UK security divi-
IPS provides a value contribution to a 7FA80D2B30A116D7802570B9005 sion. He works with partners to enable
strong network security stance. C3D16 them to provide best of breed security
3. http://en.wikipedia.org/wiki/ solutions to their clients. He is a ‘hands
References Anton_van_Leeuwenhoek on’ technical consultant with over
4. R. Fielding et al, “RFC 1918.” eleven years of security experience. He
1. N. Stanley, “ArcSight and Insider Internet EFC.STD.FYI/BCP archives. is equally happy installing a firewall
(or Inside?) Threat Management”. June 1996. www.faqs.org/rfcs/rfc1918. or penetration testing a client network.
October 2006 http://www.arcsight. html In his spare time, Tom is also a RAF
com/articles/Bloor-Research-Insider- 5. http://en.wikipedia.org/wiki/ Reserve Officer with the Air Training
or-Inside-Threat.pdf Aesop’s_Fables Corps.

The mechanics of Vipul’s Razor

technology
Jamie De Guerre, chief technology officer, Cloudmark tremendously through the rapid adoption
of Cloudmark’s commercial solutions by
Much has been said about the wisdom of crowds. The idea that many people service providers, enterprises, and consum-
can achieve results more effectively than individuals has gained credence, espe- ers. The network now encompasses over
cially as the internet has bought those people together and allowed them to co- 180 million sources in 163 countries,
operate in innovative ways. These techniques apply to everything from online including highly sophisticated reporters
encyclopedias to citizen journalism – and even anti-spam technologies. such as service provider abuse teams and
systems administrators in addition to
Vipul’s Razor, developed by Vipul Ved generate lightweight fingerprints that trusted honeypots and end users. The size
Prakash in 1998 as an open source prod- accurately identify messaging abuses and and geographic scope of the network is a
uct, was the world’s first collaborative their variants. In addition, additional algo- key factor in the collaborative process that
spam filtration network for messaging rithms to combat new classes of threats leads to higher accuracy.
security. Collaborative human intelligence can be easily integrated without the need Vipul’s innovation has proven that
‘identifies’ a message as spam, and an for changing the system architecture. individual users can accurately distinguish
automated technology verifies and pre- between spam and legitimate email early
vents its proliferation1. “Collaborative human in the lifecycle of a threat. It also began
Collaborative spam filtering works intelligence ‘identifies’ a the building of a large pool or commu-
by allowing members of a community nity of self-organising email readers who
message as spam, and an
to identify and vote on messages. The all receive the same unwanted messages
automated technology verifies
reputation of the members is continually and decide as a group by individually
rated and can be coupled with an auto- and prevents its proliferation.” nominating messages as spam or not
mated system of message fingerprinting. spam. Vipul’s Razor further proved that
The result is a system that can be used this community approach, with its col-
to detect spam, email-borne viruses, and Developing a global laborative decision making, dramatically
phishing threats.
Vipul’s Razor provides a framework for
network of trusted users improved blocking accuracy.
Most importantly, this process of col-
changes to fingerprinting algorithms and Vipul’s Razor began the process of devel- lective decision making reduces the
adjustments to security protocols, ena- oping a global network of trusted users training and learning curve of the spam
bling easy system adaptation to evolving now characterised by years of institutional filter, thereby reducing message misclas-
or new threats. Unlike rules or heuristic- anti-abuse learning. The network of trust- sifications and their associated costs. One
based schemes that require continuous ed users, which initially numbered in the of Ved Prakash’s first principles in Vipul’s
updates and entail heavy processing, tens of thousands with the Razor com- Razor was the preservation of legitimate
fingerprinting algorithms automatically munity, has expanded communications.

15
September 2007 Network Security
 VIPUL'S RAZOR

“As a community – by properly training Razor added the attributes of reputation is marked spam or not spam. In addition,
a spam filter – reduces the false positive and validation to the community of users the system provides for the rapid correc-
and critical false positive rate, the costs who report suspect messages. tion of any misclassified message.
incurred to the sender of a legitimate mes- The reputation or trust level of users is The trust rating is also dynamic. It is
sage misclassified as spam is reduced, as tracked and rated in terms of how often difficult to gain trust, but easy to lose it.
well as the cost incurred to the recipient the larger trusted community agrees with The system continually re-evaluates trust
who needs the message…. Reducing train- the classification given by a user on a mes- values of individual reporters based on
ing and accuracy costs, particularly accu- sage as spam or not spam. False reports of their past assessments of messages. The
racy as it pertains to false positives, is the spam by those wishing to subvert the sys- trust level or value is computed based on
primary motivation for creating collabora- tem are not acted upon because proven or the user’s history of reports and whether or
tive spam filtering architectures, such as trusted users have not corroborated them. not the assertions made by the user match
Vipul’s Razor and its progeny.” 1 The system’s trust component is the classifications that the system landed
Today, given the worldwide nature of two-fold: on for a particular fingerprint. “The circu-
spam origination, trust networks, just lar assignment effectively turns the classi-
like Vipul’s Razor and Cloudmark, also • The confidence the community has in fier into a stable, closed loop system.”1
must be able to precisely detect spam in the classification of the fingerprint.
multiple languages. Reporters across the • The trust the system places in the
globe evaluate messages in native lan- decisions made by members of the Using collaborative
guages and in real time. In addition, the community.
language-agnostic nature of fingerprinting filtering and finger
algorithms obviate the need for endless A confidence level is assigned to each printing algorithms
language dictionaries as used by Bayesian fingerprint of any suspect message based
spam filters or sweeping rules that result in on the reputation or trust level of the Whereas spam attempts to convincing
legitimate emails being blocked in error. individuals reporting the fingerprint. someone to take a specific action, a classic
The confidence level in the message clas- computer virus seeks to alter the execution
Validating trusted users sification is reported back to the filtering of computer software. Today, virus creators
software, and the message is filtered or transmit viruses via email with the goal
Any network of messaging users suf- allowed through based on the commu- of getting the recipient to open the virus-
fers from the problem of discernment nity’s agreement. The system updates infected attachments.
– distinguishing messages as spam or not confidence levels in real time with every Unlike Vipul’s collaborative community
spam, viruses or not viruses, phishing or report. The Cloudmark back-end system, approach, conventional anti-spam and
not phishing. To improve discernment rather than the feedback of any individu- conventional anti-virus systems use the
and thus collaborative accuracy, Vipul’s al reporter, determines whether a message knowledge of select experts who develop
new rules, heuristics, or signatures to com-
bat evolving threats. The software gener-
ates a unique signature for each new file
it encounters, which it then attempts to
match against a database of existing signa-
tures. If the database is updated frequently
and the signature is precise enough to
avoid generating matches with legitimate
software, then viruses are blocked before
they can do damage2.

“Collaborative filtering has

many advantages, such as the
size of the community and its
collaborative training, high rates
of accuracy, and scalability and
suitability for large environments”

Since the development of Vipul’s Razor,

collaborative filtering has proved to work
well for both spam and viruses and is
Figure 1: The process flow for the Vipul’s Razor-based Cloudmark trust evaluation system.
emerging as a solution suited to a new

16
Network Security September 2007
 VIPUL'S RAZOR

landscape where solutions must adapt con- signatures. Fingerprinting algorithms rap- collaborative reporting of spam togeth-
stantly to new threats. Phishing, malware, idly generate fingerprints that can iden- er with advanced message fingerprint-
and other messaging threats are recognis- tify messaging abuse and its variants in all ing algorithms. By combining human
able by individuals in the community. languages and formats. The fingerprints intelligence and automated technology,
Rather than waiting for an expert to are also generated quickly, meaning that Vipul’s Razor revolutionised messaging
detect and extract a virus feature or write a they don’t need to be as tolerant to muta- protects against the growing burden of
signature, collaborative filtering techniques tion as virus signatures do. spam and other email abuse that threat-
use community consensus and quickly According to Prakesh, “Cloudmark ens to disable the internet.
and efficiently flag what is spam and what developed a specific fingerprinting scheme
is not spam, and then automatically filter for x86 binaries that extracts instructions References
the messages associated with fingerprints. from the code section of the binary while
This approach has many advantages, such skipping non-critical instructions. This 1. Vipul Ved Prakash and Adam J.
as the size of the community and its col- fingerprinting scheme disassembles an exe- O’Donnell. A Reputation-Based
laborative training, high rates of accuracy, cutable and extracts potentially invariant Approach for Efficient Filtration of Spam.
and scalability and suitability for large envi- sections of the code.”2 In contrast to alter- Cloudmark Research Center. Cloudmark,
ronments, over other common anti-spam native approaches using time-consuming Inc. April 2007. Cloudmark, Inc.
methodologies: for example, network-layer analysis and the creation of new rules up <www.cloudmark.com/serviceproviders/
analysis, heuristics, and machine learning. front, this new fingerprinting algorithm research/cloudmark_technology/>
IP blacklisting, mail delivery rate limit- automates the manual reverse-engineer- 2. Adam J. O’Donnell and Vipul
ing, and several other techniques depend ing process common to other anti-virus Ved Prakash. Applying Collaborative
upon traffic analysis at the network layer solutions and speeds the response proc- Anti-Spam Techniques to the Anti-
alone. IP blacklisting works by blocking ess with suspect emails evaluated and Virus Problem. Cloudmark Research
all incoming traffic from mail servers that reported by trusted users in all languages Center. Cloudmark, Inc. June 2006.
are known to send spam or that have the worldwide. Cloudmark, Inc. <www.cloudmark.
potential to send spam due to incorrect Cloudmark message fingerprints also com/serviceproviders/research/cloud-
configuration. The problem is that block- are developed with two core design prin- mark_technology/>
ing all incoming mails from a server can ciples in mind: (1) high multiplicity and
increase the rate of false positives. (2) low cross-class collisions. High multi- About the author
Even when heuristics – human rules to plicity refers to cases where many distinct
identify certain behavioural differences threat messages are covered by a single As Cloudmark’s Chief Technology Officer,
between legitimate mail and spam – are fingerprint. In such cases, spam and virus Jamie is responsible for technical strat-
automatically updated and refined by emails that have undergone polymorphic egy and roadmap. Jamie also manages
machines based upon training sets, they changes in content, code, and message Cloudmark’s technology services, sales
may prove inadequate due to infrequent formats can be detected using the same engineering and ISP support teams.
training and the demands of large-scale fingerprint as the original spam message. As a core member of the design team,
environments. It can be difficult to train Thus, mutated threats can be detected Jamie wrote the first design specifications for
these events with sufficient frequency, for in real time, without even the need for Cloudmark Server Edition and multiple
example, and they also often start with a reporting by any member of the commu- versions of Cloudmark Authority. Jamie was
basic set of assumptions used as the foun- nity. Low cross-class collision minimises helped rapidly grow Cloudmark’s Global
dation for the ongoing training. According false positives by ensuring that two classes Threat Network with the invention of the
to Prakesh, “most real-world deployments of mail, which are discrete, do not share Cloudmark Network Feedback System ena-
of statistical text classification are aug- the same fingerprint. bling automatic incorporation of feedback
mented with orthogonal classifiers, such from all subscribers within a service provider’s
as blacklisting, to derive acceptable spam Conclusion network.
detection performance.” 2 Jamie previously worked at Microsoft
Fingerprinting methodologies and Vipul’s Razor led the battle against on the .NET Compact Framework,
collaborative filtering also have speed spam and future threats with two where he first began working with
and accuracy advantages over anti-virus unique contributions: reputation-based service providers.

17
September 2007 Network Security