Welcome to

DOGSBYTE.COM


  My Guide To Anti-Spam Strategies
Mailwasher is a product of Firetrust, a commercial package that offers multiple strategies for the effective control of SPAM.


Press the Play button for some soothing music





*

Friends

*

Blacklist

*

Filters

*

Learning

*

Blacklist
Servers

*

FirstAlert!

*

SpamCop

*

Friends

*

Blacklist

*

Filters

*

Learning

*

Blacklist
Servers

*

FirstAlert!

*

SpamCop

*

Friends

*

Blacklist

*

Filters

*

Learning

*

Blacklist
Servers

*

FirstAlert!

*

SpamCop

*

Friends

*

Blacklist

*

Filters

*

Learning

*

Blacklist
Servers

*

FirstAlert!

*

SpamCop

*
Friends

*

Blacklist

*

Filters

*

Learning

*

Blacklist
Servers

*

Were you looking for a guide to BLScanPlus Classic?, see the ClassicTutorial, or for BLScanPlusPro?, see the ProTutorial.



My Guide

I have been asked many times about the anti-spam strategies available using Mailwasher, what I use, what my setup is and so forth. So I have prepared this page with a dump of my thoughts on the subject.

It's a statement of the obvious, but spam can be depressing. Seeing hundreds of spam messages load up when you expect one solitary email is frustrating. Since I will never buy anything from a spam site, they are wasting their time sending me invitations to buy cheap software, make a million in a week, buy tranquilizers, buy pills that will enlarge my bodily organs etc. The crazy thing is that if the spammers opened a fully functional internet shop, selling products at competitive prices, I would probably visit it and might even buy something! It is the 'pressure sell' culture of spam which drives me away from ever making any purchases.

To get rid of unwanted messages there are many products on the market, my particular favourite is Mailwasher. This can be customized to suit just about everybody's needs, it features a range of strategies for cleaning out an inbox. A key issue though is how those strategies are used. Like most things in life, if something is used incorrectly, the benefits are lost and it could even be dangerous.

With any anti-spam strategy, one of the biggest concerns is 'false positives' where legitimate email is identified as spam and subject to whatever rules or options are being employed to deal with it. The most successful strategy for me has proven to be the use of Mailwasher with heavy usage of blacklisting for domains and some use of filters. Having collected many spam domains, my Mailwasher blacklist alone is now capturing most of my spam, with no false positives, my Mailwasher filters catch most of the rest. So below I have reviewed the strategies available in Mailwasher and stated my personal opinion on each.



My Guide


The friends list is very important and worth protecting, since your contacts contained in this list are allowed through your anti-spam defences.

Many people have suggested that you only need a friends list and that any other email should be treated as spam. This is referred to as the 'perfect whitelist', this is a great idea but impractical however, unless you have complete control over your friends! The problem with this approach is that you may lose important email. This is because it is very difficult to keep a whitelist current, you can lose email for a number of reasons;
  • Your friends may transfer to a new ISP.
  • Your friends may email you from some other account (their work address for instance).
  • You need to instantly add a new friend to the list before they send their first email to you.
  • You need to remember to manage the friends list and add entries for any new contacts that you supply your email address to.
The net result of any of the above is that email from your friends does not make it to your browser window.

A second type of problem is that an email message has a header section which carries the source, destination, date etc. This header can be modified (or forged) with another email address. Header forging allows spammers to send emails that appear to come from other people, maybe even your friends. So even if you adopt the perfect whitelist strategy, it does not mean that you will be spam free. So clearly something extra is needed to filter your incoming mail.



My Guide


The blacklist allows you to specify email addresses from which you want to reject any incoming email. When an incoming email meets the criteria of one of your blacklist entries, you choose what to do with it. The email can be automatically deleted and hidden for instance, this means that you will never 'see' an email from a blacklisted address ever again. Blacklisting does not stop incoming emails, it just provides a means to push spam into the trash.

The blacklist (and indeed the whitelist) can also be specified using wildcard characters, so rather than entering a long list of email addresses, it is possible to specify a single entry that blacklists an entire domain (e.g. *@spam.com). Care is needed to ensure that you do not inadvertently blacklist a domain from which you may receive legitimate mail, the solution here is to either add that particular address into your friends list or to make the blacklist entry more specific.

Spammers are wise to blacklisting and regularly change their address. From analyzing the spam I receive, I find that on average most spammers will change their username every two weeks. This type of one-time or short-term address usage lulls many people into abandoning their blacklist (a victory for the spammer!). However, I also found that spammers do not change their domain with the same tenacity (not at all in most cases). So by collecting all those one-time use email addresses in the blacklist, a tool (like BLScan for instance) will convert them into domain wildcards and will mask out huge amounts of spam going forward. I only see perhaps two or three new domains appearing a week.

The nice thing about blacklisting is that you are in total control of what email makes it to your browser window. I particularly like this circumstance since I have many accounts and I am always concerned about losing email by having an overly aggressive spam filter.



My Guide


Filters may in time become the most effective defence. Filters are similar to small 'programs' where a set of conditional statements can be captured and incoming email subject to each one in turn. For instance, in words a filter could be something like 'if the email comes from an address with "sales" in it, and the title contains the phrase "instant win" and the message body contains the words "winner" or "congratulations" then blacklist the address and delete the email'. So with some ingenuity, filters can be very powerful.

The downside here is that writing filters is firstly, not easy for people who do not have some basic computing knowledge and secondly, filters are vulnerable to false positives where a filter is not specific enough and mistakenly blocks a valid message that happens to meet that criteria. The latter can be minimized with more accurate filters, but these then tend to get a lot more complicated. Although I can write fairly complex filters, I always try to make filters as simple as possible. As mentioned before, given that I am always concerned about losing genuine email, I scan through 'filtered' email to check whether any valid email has been inadvertently filtered. So filters are very good, but you need a lot of them, you need to maintain/refine them and you need to check for false positives.




My Guide


Blacklist servers such as FirstAlert!, SpamCop and many others are intended to be accumulators of spam (IP) addresses. By comparing the origin of incoming email versus one or more of these databases, spam can be identified. The databases often feature a mechanism where users can report spam such that the database stays current and gets more comprehensive over time.

These servers work well, but in my opinion do not work quite as intended. My concern is that these databases have a flaw, many users are reporting email from professional/responsible companies as being spam, rather than opting out of their mailing list or just deleting the messages (with a filter or blacklist). This means that many of the blacklist server databases will contain and identify email from respectable companies, organisations and sometimes innocent individuals as originating from a spam source. I have seen this happen many times with mail order companies in particular and I have even had some private email addresses from friends using major ISP's being marked as spam.

I regularly use a very well known computer mail order company to buy my computer stuff, any emails from that company are always marked as spam by the blacklist servers. If you are unfortunate enough to set yourself up a new email address on a new ISP, but subsequently discover that you are now sending through an IP address previously used for spamming, there are services available to get innocent IP addresses cleared from blacklist servers.

So although spam databases are a great idea, I find that they result in too many false positives and I need to keep checking whether they have denied me any valid email. I would still recommend their use, provided you are willing to check through what they identify or perhaps write some filters to override their conclusion when necessary.



My Guide


Bouncing spam is an emotive subject within the anti-spam community! My advice is to only bounce email that comes from people you know, that have become pests. A bounce message says to that person 'your wasting your time sending me email because I'm not reading it'. This does not work with spammers though for several reasons;
  • A huge amount of spam is sent from automated programs called 'zombies'. These programs are deposited onto computers, usually in publicly accessible places such as schools, colleges, libraries, internet cafes etc. A zombie can also be deposited on a computer by malicious software. With a zombie, there is noone physically sat at a machine sending spam to you and thus noone will actually receive any bounce messages that you send. 
  • Spammers are aware of bouncing and the receipt of a bounce message will indicate to a spammer that they have hit on a live account, you will thus get even more spam.
  • Email header forging allows spammers to send emails that appear to come from someone else, so when you bounce your message it will go to the owner of the forged address, not the spammer. One day your address will probably be forged and when you receive hundreds of bounce messages for an email that you did not send, then you will agree with me that bouncing is not a good idea!










My Guide


So what do I do?

Well firstly, I do not think there is a general solution that suits everybody. The amount and type of spam that people receive varies enormously and it seems logical that an anti-spam strategy should be customized to suit a persons circumstances. So I can discuss my setup and the solution that works well for me, but this may not suit everybody. What I would suggest however is that new users try out some of the strategies, see how effective each is and also how effective various combinations are. As your confidence builds, you can gradually spend less and less time hunting for false positives. Combating spam is something of a learning process and Mailwasher is a great tool to help facilitate that process.

So to answer the question, my setup is;
  • Friends List
    • I list my friends, relatives, contacts and associates. I keep this list under control and make sure I know exactly who each and every entry is. I have some wildcards for my work domain, a few shops, organizations, clubs etc.
  • Black List.
    • I have accumulated a lot of entries in my blacklist but also in turn captured a lot of spam domains. Using BLScan, I replace the individual addresses for a given domain with a wildcard to block out that entire domain. I run BLScan once a week. I use the date code option in BLScan but set my wildcards to 'never expire' in Mailwasher, this way I can see how recently my wildcards have been used (in the BLScanS file). I set the Mailwasher expiry to 30 days for non-wildcarded blacklist entries. I also blacklist entire countries by having wildcards for the two letter country code appended to domains, a full list of potential country wildcards is here. (Beware of blacklisting Tuvalu since it's country code is "tv", the same as many television related sites!). In my BLScan folder, I also place a shortcut to the Mailwasher application files so I can easily move files around.
  • Filters.
    • I have about 55 filters, most of which were based on the set produced by Gary Partain. I have added extra filters for specific types of spam and to automatically delete emails with forged headers.
  • FirstAlert!
    • I use this, with 'delete' and not on automatic.
  • Blacklist Servers.
    • I use these, with 'delete' and not on automatic. I use ORDB (relays.ordb.org), SORBS (dnsbl.sorbs.net), Spamhaus (sbl-xbl.spamhaus.org) and Distributed Boycott (list.dsbl.org). I have also used Multihop (multihop.dsbl.org) and CombinedNJ (combined.njabl.org) although the latter two were a bit slow.
  • SpamCop.
    • I use this (bl.spamcop.net), with 'delete' and not on automatic.
  • Learning.
    • I use this and have Mailwasher mark items to be deleted/blacklisted, but not to auto-delete or mark as friends.
  • Bouncing.
    • Definitely switched off and disabled!
  • Restore
    • I use the option to recover deleted email. So even if I do get a bit over enthusiastic with my spam defences, I can pull messages back out of the trash.
As you can see, I rely heavily on my blacklist as my first line of defence and my filters second. My blacklist now soaks up around 85% of my spam which is auto deleted. I never check through it since I know exactly what is in my blacklist and I do not have to worry about false positives. My filters catch the rest, I do check through filtered emails on the screen before hitting "Process Mail" and deleting/blacklisting them. Click here to see some sample screen shots from my Mailwasher statistics, showing how effective my blacklist and filters are. Since I am paranoid about deleting legitimate email, I wrote Totter to scan my trash file and I use this as a tool to hunt down any false positives.

Can I have your blacklist and filters?

I've been asked many times to distribute my blacklist and filter files. As regards the blacklist file, there could be a jeopardy here, in that I may have blacklisted domains that wind up blocking your legitimate email. If the blacklist file were short then no problem, you could trawl through it and check I do not block anything that is important to you, but it is not short!

If I publish my filter file, spammers can see it too and can simply work around the filters, but since my filters only look at the last few percent, what the hell, here is an edited version of my filters.

Having had the pleasure of many discussions with Mailwasher and BLScan users, it occurred in conversation many times that perhaps a solution would be to build a repository of spam domains, perhaps with different 'strengths' and making these lists available as blacklist files. This may have some merit. To facilitate this, some software tools will be needed for which BLScan and BLComp now have the required functionality. A utility called BLConv has been produced. So check back to this domain in the future to see if this idea gets off the ground.

I've read that some people do not use the blacklist?

After experimenting with Mailwasher's anti-spam strategies, users may find that some of the anti-spam strategies are not effective in their circumstance. Some users may find that the blacklist is ineffective, some may find that filters are ineffective, similarly others may find the blacklist servers to be ineffective, some users may find Mailwasher to be entirely ineffective! Everyone is different.

As discussed earlier, users of Mailwasher need to find what combination of anti-spam strategies works best for their particular circumstance. It's human nature, but once a user finds the magic 'recipe' that clears out their own inbox, with few false positives, it's a Eureka moment. But there is a tendency or common perception that this same recipe will then work for everybody else, it rarely does. Since the quantity and type of spam varies tremendously from one person to another and maybe even from one email account to another, one users magic setup can be devastatingly ineffective when somebody else adopts it. In my circumstance the blacklist is very effective. There may well be circumstances where the blacklist is not the most appropriate defence, as a general guideline I would suggest that;
  • If you receive large quantities of general spam, the blacklist works very well and you can get very few false positives (since you control exactly what is blotted out). This is my circumstance and I get spam about every subject imaginable.
  • If you receive large amounts of spam from one or two sources, you can blacklist those two sources and get by with a very small blacklist.
  • If you get large amounts of spam which are all 'themed' to one or two particular services, products or subjects, then using some filters may be more appropriate, to pick out commonly occurring words consistent with that theme.
Out of all of the anti-spam strategies available in Mailwasher, the blacklist is probably the simplest to use, certainly one of the most reliable in terms of very low false positives and is also very easy to use. It simply says "I do not want email from that address in my inbox". Unlike filters (which can be error prone) and blacklist servers (which can be very error prone), there is not much that can go wrong. There are a couple of criticisms of the blacklist feature which surface from time to time but which don't really hold water;
  • Spammers keep changing their address so there is no point in blacklisting the originating address of spam
    • The first part is certainly true in that spammers do frequently change their address. [I'm repeating myself here but here we go again...] Over a 12 month period I stored thousands of incoming spam addresses. I looked at how many times each address occurred (using BLScan). I then put this data into categories and sorted it into three groups;
      A: Unique username with unique domain name (an address I had never seen before and which only occurred once)
      B: Different username but with a previously used domain name (a spammer using a different username but on the same domain)
      C: Same username with same domain (an address from which I have received spam before)
The percentages of the total were A=4%, B=87% and C=9%. On average spammers appear to change their username once every two weeks, but not the domain name. So by collecting all those apparently useless one-time addresses, saving them into the blacklist, wildcarding them with BLScan, I now blot out somewhere between 87% (group B above) to 96% (group B plus group C above) of my incoming spam with no false positives! This is supported by some hard facts and data, here are some of my statistics. You might ask why spammers do not keep changing their domains?, well the simple answer is that this has become more difficult, they have to pay to do it and also because it is traceable.

So spammers do keep changing their address, but do not change their domain with the same tenacity. This is the whole point of the wildcarding facility within Mailwasher. I only see perhaps one or maybe two new domains appearing per week in my incoming spam. So I would actually recommend the opposite, i.e. do store those one-time addresses and do not set your blacklist expiry too low, because you can then capture a lot of spam domains and then subsequently wildcard them out (using BLScan).

To be fair, I do agree with the point if users do not actively use the wildcarding facility (or where users do not use a tool like BLScan).
  • Spammers make up their email address so there is no point in blacklisting email
    • This is a logical statement, but when carried a few steps further forward does not entirely make sense. It is certainly true that some spammers use header forging (spoofing) to hide the originating email address. Again, the quantity of this type of incoming email varies from one person to another. To gather some hard facts and data, I wrote a modified version of Totter to search for header forging, to look for inconsistent IP addresses, to identify evidence of spoofing etc. I counted up that <3% of my incoming spam is forged.
If spammers keep making up new addresses, my previous statistics would have shown a figure far bigger than the 4% I extracted for category A (unique username and unique domain).
  • If they use an invented email address on a domain I already blacklist then great, I blot it out anyway.
  • If the spammer keeps using the same forged email address then great, I can blacklist it.
  • If the spammer keeps inventing new usernames on the same domain, then great, I can wildcard the domain.
  • If the spammer forges a truly one-time address on a fictitious domain then I can blacklist it and my blacklist expiry setting will over time delete that address.
  • If the spammer uses an address of someone who happens to be a friend, then I can use some nifty filters to detect this.
  • If the invented address is on a domain from which I get legitimate email, then I do not wildcard the domain, I just leave the individual address in the blacklist.
  • Also, whether the origin address is forged or not, the email is still unwanted spam!
So whichever way you look at it, whether the address is forged or not makes absolutely no difference.

So the only time I do not get a spam 'hit' is when spammers forge an address on a domain I do not want to blacklist. So in that situation I store the address, but I do not wildcard the domain (I put it into the BLScan ignore list). I then use a set of filters to weed out the remaining percentage (group A). But filtered emails can be a pain since they need to be checked for false positives and effective filters are hard to write/maintain. So if you rely soley on filters, you can spend a lot of time scanning through the summary screen every time you check your email. I guess I would inevitably say that Totter can help with this.

So some lucky people probably can get away with a very short blacklist or maybe just a very small set of filters, but that will not work for everyone and users need to establish for themselves what works for their own circumstance.

Having said all that, I do think it's great that people share their experiences, successes and failures at combating spam. To be clear, the only irritation I have is that occasionally a user will assume that everyone gets exactly the same type and amount of spam that they do and the whole world should copy their specific setup. I have a friend who recently bought a new car and he spent weeks spouting the virtues of that car, suggesting that everyone should buy one, everyone elses car was junk in comparison etc. It really is a nice car, but (a) I cant afford it, (b) it drinks gas like a drain, (c) I can't get five cases in the trunk and (d) I need four doors! So it's not the right car for me. This type of evangelizing often makes new users follow the evangelizer like sheep and can also give new users a false expectation of success (instead of encouraging experimentation or advice on the pro's and con's of Mailwasher's facilities). I sometimes get a little concerned that many users who often vigorously make claims about how they are right and everyone else is wrong never seem to substantiate their claims with any hard facts and data, which makes me think they might be spammers trying to persuade people to drop their most effective anti-spam defences!


Any other tips?

A couple of problems that come up time after time are problems restoring email and filtering out forged emails.
  • When restoring an email it is resent to your inbox. When it arrives, your spam defence will recognise it and will exercise whatever options you are using. So if it matches something where you autodelete, the restored email immediately gets deleted and dropped back into the trash! Use the "Restored" filter in my filters file and this will no longer be a problem..
  • Forged emails that appear to be sent by yourself can be disturbing first time around. They are not as dangerous/damaging/harmful as they may first appear. A good filter to get around this is also in my filters file and is called "Forged". This basically checks whether your name appears in both the To list and the From list. Since you may have your own email addresses in your friends list, this filter takes precedence over your friends list. Using this filter does mean that if you ever send a message to your own email address from that same email address, perhaps to test your email, you need to switch that filter off first.
General points about email addresses;
  • It's too late for most of us, but if you're new to the internet or perhaps just about to change ISP's, pick your email address carefully. It's always nice to have an email address which is the same as your name, but such addresses are easily spammed. There is a very high probability that a name such as "asmith" will be present on every single ISP, as well as a "bsmith", "csmith" etc. There are actually not that many names in the world!, so spammers will send out spam speculatively to likely email addresses. You're better off going for an address which is a little unique, so something like asmith_2004, or i_am_a_smith, mail4smitty etc.
  • Many ISP's will allow you to have several email aliasses, so use one for a while for unimportant stuff, if you start getting spammed, delete it and use another. Save one address for your close friends, maybe use another for shopping (always keep a note of shops where you've used it), one for forums etc. If you register on a site for something, always make a note of the site so that you know where to go and change your contact details at some later date.



My Guide


In terms of current technology, I would dearly love to be able to combine different strategies in Mailwasher. For instance, to be able to write filters such as 'if the domain is not blacklisted and no filters match and two out of three blacklist servers show the email to be spam, as well as the learning function then if the address is not forged, mark the email as highly likely to be genuine but do not add it to the friends list'. This would be an awesome feature.

The future of anti-spam software probably lies in developing more complex and 'intelligent' filters. This has the drawback of the higher risk of false positives but this can probably be minimized with more available functions and a lot of ingenuity.

The ultimate fix for spam though, lies in legislation and improved internet infrastructure such as email identification/non-repudiation, so be sure to support any projects that are addressing this. Finally, in my opinion the spam problem could have been avoided a long time ago. For the future, the ideal solution is not better anti-spam products!, this may seem counter intuitive but philosophically we are faced with a simple question; If a river is being poisoned, do we solve this by filtering the water with ever more complex filters?, or should we find the source and stop the poison getting into it in the first place?

Back to top



Visit Firetrust Website
Google
Visit Mailwasher Forum