A Bayesian spam filter does not know what spam is. It knows which words turned up in the mail you called spam, which turned up in the mail you did not, and it does arithmetic on the difference. The arithmetic is small enough to do by hand, so let us do it.
The spam filter in the Y2K Lab, from my Information Retrieval coursework, works on this principle: show it a handful of spam and normal mail, and it builds its own sense of which words give a message away. The training set below is made up for this post, and deliberately tiny so the numbers stay readable.
Six emails and a tally
Three spam, and three normal messages, which filter people call ham:
- Spam: “free money now”, “claim free gift now”, “overdue invoice now”
- Ham: “invoice for march attached”, “free for lunch”, “call me now”
Each pile happens to hold ten words. Counted, they are all the filter knows:
- Spam (10 words): now 3, free 2, money 1, claim 1, gift 1, overdue 1, invoice 1
- Ham (10 words): for 2, invoice 1, march 1, attached 1, free 1, lunch 1, call 1, me 1, now 1
Divide a count by its pile’s total and you have the probability of that word, given the pile. In spam, “now” is 3 in 10. In ham, 1 in 10. That gap is the evidence.
Scoring a new message
A new email arrives: “free invoice now”. For each pile, the filter multiplies the probabilities of these words, then multiplies by how common that kind of mail was in training. Three of the six were spam, so both start at one half:
- Spam: 2/10 × 1/10 × 3/10 × 1/2 = 0.003
- Ham: 1/10 × 1/10 × 1/10 × 1/2 = 0.0005
Divide each score by their total and they become probabilities: 0.003 out of 0.0035 is about 86 percent spam. “Invoice” appears once in each pile, so it pulls neither way. The verdict comes from “now”, three times as common in spam, and “free”, twice as common.
That is Bayes’ rule: start from how common each kind of mail is, update by how well each kind explains the words, and compare. The naive part is the multiplication, which assumes every word turns up independently of the others. Hold that thought.
The word it has never seen
Now score “free lunch now”. “Lunch” never appeared in spam, so its spam probability is 0 in 10, and anything multiplied by zero is zero. One harmless word has made the message impossible as spam, however loudly the rest of it shouts.
The standard fix is add-one (Laplace) smoothing: add one to every count, and the vocabulary size to every total. Our six emails use 13 different words, so each total becomes 23:
- Spam: free 3/23, lunch 1/23, now 4/23. The tops multiply to 12.
- Ham: free 2/23, lunch 2/23, now 2/23. The tops multiply to 8.
Everything else is equal on both sides, so it is 12 against 8: 60 percent spam, a reasonable maybe. Smoothing also softens the first example from about 86 percent to 75, which is the right instinct when the evidence is six emails. With thousands, the extra one barely moves anything.
Numbers too small to hold
A real email has hundreds of words. If each has a probability around one in a thousand, three hundred of them multiplied together make 10 to the power of minus 900. A JavaScript number cannot hold anything below about 10 to the minus 324, so both piles score exactly zero and the filter cannot tell them apart.
The fix is logarithms. The log of a product is the sum of the logs, so you add ordinary numbers like minus 2 instead of multiplying tiny ones, and the higher total still wins. For “free lunch now”, with natural logs and the equal halves left out:
- Spam: ln(3/23) + ln(1/23) + ln(4/23) = -2.037 - 3.135 - 1.749 = about -6.92
- Ham: 3 × ln(2/23) = 3 × -2.442 = about -7.33
Spam is ahead by about 0.41, and e raised to 0.41 is about 1.5: the same 12-to-8 ratio. Same answer, for an email of any length.
The lie that works
Words in email are not independent. “Claim” and “gift” travel together; so do “invoice” and “attached”. A filter that treats them as independent counts the same evidence twice, so the confidence it reports is exaggerated. A fairly spammy message can come out looking certain.
The surprise is that the decision is often still right. Double counting tends to push the score further the way the evidence already points, so the winning side survives even when the percentage is silly. Read the output as a vote, not a measurement.
Where it stands today
Bayesian filtering became the standard defence against junk mail in the early 2000s, helped along by Paul Graham’s 2002 essay A Plan for Spam. Spammers adapted. They padded messages with innocent words, misspelled the giveaway ones so the counts never matched, and moved the text into images.
Mail providers still classify text, but the words are now one signal among many, next to sender authentication, the reputation of servers and links, how other recipients treated similar mail, and models that read a message as language rather than a bag of words.
Naive Bayes is not retired. It trains in one pass of counting, needs very little data, and can explain any decision by pointing at the words that moved it. For a first text classifier, sorting support tickets or tagging documents, it is an honest baseline: if a large model cannot beat a page of counting, it is not earning its cost. What naive Bayes cannot do is understand. To it, “free” and “not free” are nearly the same message, and a sentence that means the opposite of its words is judged by its words.
