Welcome to

DOGSBYTE.COM


  BLConv © 2004
BLConv is a utility for CONVerting wildcard domains in a Mailwasher blacklist file into a set of filters
Mailwasher is a product of Firetrust, a commercial package that offers multiple strategies for the effective control of SPAM.


Press the Play button for some soothing music

BLConv

*

Convert

your

blacklist

to

filters

and

share

it

with

others

*

BLConv is an option within BLScanPlus.

Over time I have accumulated many domain names from which I have received nuisance email. I have been asked many times whether I would be prepared to publish my blacklist file and make its content available for general consumption. My immediate thoughts about 'blacklist sharing' between users were positive and at popular request I implemented an exchange file output (BLScanX) to BLScan. I have remained concerned however about the simple problem that a blacklist supplier may have blacklisted domains from which a blacklist recipient may receive legitimate email. Not only would the suppliers blacklist filter out a lot of nuisance email, but it may also filter out the recipients genuine email too! Having given this some thought, I have come up with some solutions and a new utility.

This page documents the BLConv utility and it makes no difference whether you're using the Classic or Pro version of BLScanPlus unless otherwise specified.


BLConv

  Sharing blacklist entries between users is a potentially valuable anti-spam strategy that does not suffer from some of the issues associated with Blacklist Servers discussed in my guide. As already mentioned above there is a problem which I can break down into four parts;
  • RISK TO LEGITIMATE EMAIL: It is a risky activity since a suppliers blacklist may have entries that wind up blocking the recipients legitimate email. This is only a serious problem if you use the Mailwasher 'hide' and 'autodelete' functions as these will conceal any incoming email that aligns with one of your blacklist entries.
  • NO ASSESSMENT CRITERIA: Potential recipients could of course check through a blacklist/spam/pest domain list and evaluate the risks, but with a long list (or numerous lists) this is a very arduous process.
  • NO UNDO: Adding a blacklist into your own blacklist is one way process, over time the original file content and any blacklist file content added to it will become merged and difficult to separate/remove at a later date (although there is a way of doing this documented later).
  • NO GRANULAR CONTROL: Mailwasher has some options that allow control over what happens to email from sources that match the blacklist, but these apply to the entire file. There is no ability to define 'grades' or 'classes' of blacklist entries.
These roadblocks look tough to knock down and I thought about this subject for quite some time without any real progress. While out walking the dogs one day, the fog cleared, the sun burst through the trees (it just needed the accompaniment of an angelic harmony) and suddenly a light bulb illuminated over my head. I was not standing under a street light (joke), I had one of those creative moments and an idea surfaced.



BLConv


Having gone into 'deep thought' mode on this, I concluded the best way forward is to convert a suppliers blacklist into a Mailwasher filter. So rather than exchanging Blacklist files, it is more sensible to collect blacklist entries but then exchange them as filters. Adding a filter alleviates the four problem areas I described earlier. The main advantage is that filters;
  • can be associated with the originator
  • can be enabled/disabled
  • can be deleted
  • can have different settings for different filters
  • can be examined for effectiveness since Mailwasher already records statistics for each filter
In terms of what content should be in the filter, the information most worthy of exchange is wildcarded domains. Users of BLScan will over time accumulate many blacklist entries of the form *@domain.com. These wildcarded domains are the most valuable to exchange since they are the most 'noisy'. Individual email addresses are typically one-time-only usage and have little value (but collecting them is still a good thing to do, since if they occur frequently, you can then blacklist the entire originating domain).

To this end I have completed a third utility called BLConv, which will read a blacklist file or address list, separate out any wildcarded domains (like *@domain.com) and create filters that can be added to your Mailwasher 'filters.txt' file. Neat huh?

What's even better is that I recently added the BLScanX exchange file output to BLScan, so running that file through BLConv will also remove any entries with country specific domain codes! I love it when unrelated things suddenly dovetail together.



BLConv


Before I go further I want to step back to my previous comments about blacklist content. It is important to note that the presence of a given domain in someones blacklist (or filters) does not automatically imply that domain is a disreputable spam source. It would be more accurate to state that my blacklist entries for instance, are sources from which I have received large quantities of unwanted/nuisance email that I cannot opt-out of receiving. Many entries in the blacklist file may be respectable companies, but unfortunately they send out excessive quantities of email and many do not offer an opt-out mechanism.

So if a supplier makes their blacklist entries available for general usage, users should be aware that these entries probably do not represent a list of spam domains. The content of my own blacklist I would describe as "domains from which I have received significant quantities of unwanted or nuisance emails, that have been sent to my private email addresses without my consent and that have interfered with my reasonable enjoyment of internet services".

If you happen to see your name/company/ISP in someone elses blacklist, the filters produced by BLConv only 'mark' the incoming email as being present in the suppliers blacklist, the filters do not by default conceal email from domains in that list. It is entirely at the recipients discretion what they subsequently do with that email and how they employ the lists.



BLConv

BLConv simply reads in a Blacklist file (or sequential list of blacklist entries), separates out the entries within it that are of the form *@domain.com and creates one or more filters. Note that because of the maximum number of terms in a Mailwasher regular expression function, BLConv may generate several mutually exclusive filters for the same input blacklist.

BLConv generates two output files;


BLConvL

 		This is the log file and carries the output filters together with a fully expanded list of the filters for reference. Because of the format of Mailwasher filters, the filters generated are very difficult to read. This log file expands out the filters to make them much easier to read. This file is for reference only.



BLConvF

 		This is the filter output file and carries only the output filters. These are the filters that can be cut/paste into your filter file. They are in the correct format and can easily be appended to your existing filters file.



BLConv has no options as such, it only requires the input file name and the output filter name.
  • The input file name can be a Blacklist file or a sequential list. If it is a blacklist file, only the blacklist section is read and only entries with wildcarded domains are passed into the output filter (i.e. entries of the form *@domain.com). I would actually suggest using the BLScanX file as the input. The BLConv reader is actually the same software routines used in BLComp, so the 'sequential file' formats discussed on the BLComp page are valid here too.
  • The filter file is the name that will be associated with the filter and will be visible in Mailwasher. If BLConv generates multiple filters, the same name will be used and the filters numbered.









BLConv


To add the filters generated to your filters file, this must be done manually.

Go to the Mailwasher application area, the path to this is shown at the bottom of the Mailwasher Help->About dialog box. In that folder is a file called "filters(.txt)" which contains the information that you see presented in the "My Filters" dialog. The format of this file is fairly obvious, but care is needed to ensure that you maintain the file format. If you accidentally introduce any blank lines, or  delete parts of another filter, Mailwasher will give an error on startup. So always backup or retain a copy of your current filters file before you start editing, you can then always revert to the previous version should you get into difficulty.
  • Simply paste in the BLConv output (from the BLConvF file) into the filter file, then go into Mailwasher Spam Tools->My Filters and set up your preferred options for the filters.
  • In case the filters carry entries that overlap with your friends list/blacklist, it is wise to set the precedence of the filters versus the friends list/blacklist (i.e. if an email address is matched by both the friends list/blacklist and a filter, precedence decides which one wins).
  • You should also decide on and set the priority order of any new filters.
On a cautionary note, only update the filters file when Mailwasher is not running, otherwise on closing Mailwasher it will overwrite your newly edited file!

Once you have appended these into you Mailwasher filters.txt file, you will also be able to examine the Mailwasher statistics to see how much of your incoming email matches up to each set of filters.



BLConv


As an example I ran the 'starter' blacklist produced by Gary Partain and my own blacklist files through BLConv. For my own blacklist file, I actually used the exchange file that I generated from BLScan (the BLScanX file).

The filters generated for the (three) different runs were obviously generated in different output files, but I have appended them together into one file here to illustrate what BLConv can do.

Note: The filters generated in this demonstration are an example, I would not recommend you actually use these filters because they might filter out too much of your incoming email (especially if you switch on hide and autodelete!). Although you could paste them into your filter file to see what they look like and make sure they work. I will update this example from time to time. These example filters thus represent the domains present in both Gary's starter blacklist and my blacklist as of 02-Oct-04.



BLConv


I mentioned earlier about 'subtracting' one blacklist from another. This might be needed if you perhaps over enthusiastically incorporated somebody elses blacklist and found that it was too aggressive and your inbox is cleared out completely!

Although my solution does not require this, I wanted to document it as a useful facility to be aware of. So at some point let's assume you take your blacklist file (containing your original list, let's call it A) and append to it a new list (let's call it B). If you later decide that list B is too aggressive and that you wished you had never added it, then it is extremely difficult to 'undo' the addition and remove it. This is because the blacklist file does not remain in the same order, it constantly gets sorted and modified as you add new entries. You could of course revert to a backup copy of your original list, but this would not contain any new entries added since the time of that backup.

The solution is to use the subtract function of BLComp, you must however have either retained a copy of list B, have it on backup or be able to reconstruct it. Using BLComp, you compare your current blacklist file versus list B. BLComp generates a list of the domains that are unique to either file and those unique to your current blacklist are your original list (A) plus any new entries that you have added! How cool is that?



BLConv
BLConv
is
included
with
BLScanPlus

The version history is given below;
  • 1.0
    • First release

  • Integrated into both BLScanPlus Classic 2.0 and BLScanPlus Pro 3.0

Back to top



Visit Firetrust Website
Google
Visit Mailwasher Forum