My email hosting provider allows users to configure anti-spam settings on the incoming server. For each hosted domain you can enable or disable three different anti-spam systems. I enable SpamAssassin filtering and set the threshold to 5.0 which stops most spam but results in very few false positives. I don't enable the other two because they are somewhat contoversial. One is greylisting and the other is "sender verify" aka callback verification. The fact that my provider is doing callback verification for at least some of its customers seems to have resulted it being listed at Backscatterer.org at least once. When I pointed it out they argued that doing callback verification results in a load reduction on their servers even though most spammers now use valid from addresses to get through this checkpoint. That may be the case, but this sort of probing puts an extra load on other mail servers and it can even be used to carry out a DDoS attack.
I have add "set text_flowed" to my .muttrc which means that mutt now adds "format=flowed" to the Content-Type header in my outgoing emails. I have also modified my .vimrc file so that when it gets called by mutt to edit an email I am composing it automagically adds a single space at the end of lines when they wrap, thus indicating that the line is part of a paragraph which can be reflowed by agents that read it. It is a bit difficult to explain this briefly, so I just refer you to RFC 2646, "The Text/Plain Format Parameter".
My experimentation with format=flowed came out of a Usenet discussion about how to delimit long URLs in Usenet messages so that client software can render them clickable even when they don't fit on a single line. Information on this can be found in Appendix C. "Delimiting a URI in Context" of RFC 3986, "Uniform Resource Identifier (URI): Generic Syntax".
MarkMail is a nice searchable archive of (currently) 1,069 active mailing lists, accumulating over 7,000 messages per day.
Since December 8th over 80% of emails sent to me have been rejected as spam by the Exim filter I have on my mail server.
There is a filter called t-prot which can be used with any sane MUA to protect against TOFU, a term explained in the man page:
TOFU is an abbreviation which mixes German and English words; it expands to "text oben, full-quote unten" which means "text above - full quote below" and describes the style of so many users who let their mailer or newsreader quote everything of the previous message and just add some text at the top; obviously they think that quoted text must not be changed at all.
I don't currently use t-prot with mutt (the MUA for which it was originally written) but I am considering it. The above mentioned web page has some useful mutt tips.
The ASCII Ribbon Campaign is "Against all HTML e-mail and proprietary attachments".
I must take a look at Inbox Zero:
These are posts from a special 43 Folders series looking at the skills, tools, and attitude needed to empty your email inbox - and then keep it that way.
John Gruber writes about the iPhone's lack of support for proprietary Microsoft Exchange email servers. Hopefully this may result in companies activating IMAP functionality on their Exchange servers, or abandoning them altogether for proper standards based email systems.
The BBC is reporting on IETF acceptance of a draft standard called DomainKeys Identified Mail (DKIM) which it is claimed will help in the battle against spam and phishing attacks.
My contact form recently came under attack by spammers. The bots must have got clever enough to fill in all fields and that was sufficient for my CGI script to let it through. I have responded by adding a fourth field that has to be filled in with a specific value that will not be obvious to bots but which humans will have no trouble with. It has stopped the spam for now, I wonder how long before I have to add my next countermeasure?
I have just added a couple of useful lines to my .muttrc file. This came about as a result of a thread in comp.sys.mac.system which was originally to do with the advantages of Entourage over the Mac OS X email client, imaginatively called Mail (or sometimes Mail.app). I have never used Entourage but I sometimes use Mail just because I think I should have basic knowledge about the default GUI email client on my system. This thread then morphed into one about different methods of forwarding an email. Basically there are two; "inline" and as an attachment. I will describe the way mutt implements them.
When you forward inline, mutt creates a new message ready for you to edit. This message contains the orignal message text in the body, surrounded by some delimiting lines. Typically you would then add your own message at the top and send it as usual to the new recipient. The recipient would receive the email from you, view it, and see your text followed by the forwarded message. That is how I have always done it but it has not always seemed to be satisfactory. First of all there are times when I might want to remove some of the original message for reasons of privacy or brevity and intersperse my own comments. This is tricky because there is no easy way for the new recipient to distinguish my comments from the original text. The solution was to set forward_quote in my .muttrc, which results in the original text being indented with the same string used for quoting in replies. The second problem is that the recipient has no way to automatically recover the orignal email in exactly the form it was sent to me, which can lead to problems. For example, If the original email was multipart/signed they will be unable to verify the signature
So the other way to forward is as an attachment. Mutt implements this by constructing a MIME attachment of type message/rfc822 containing the original message (including any attachments) and attaching it to a new blank email for you to edit. You can then write your own message to the new recipient and send as normal. This way they recieve an exact copy of the original message which they can (given a suitable MUA such as mutt) process independently of your message. This seems to be the logical way of doing things if you want to forward the original email in its entirety.
So I have also added "set mime_forward = ask-yes" to my .muttrc file. Now when I forward an email it will be sent as an attachment by default, but I can revert to forwarding as text just by hitting "n" instead of RETURN when it asks me, giving me the best of both worlds. As I see it, the most appropriate will almost always be pretty obvious in any situation.
Apparently the Mac OS X Mail application does not provide this as a forwarding option, and when it receives such a forward it dos not make the attachment available as an attachment but attempts to display it inline.
It looks like Yahoo! have done something silly - no surprise there then:-) I administer a Mailman list and over the last few days all the subscribers with yahoo.co.uk email addresses have had their subscriptions disabled due to excessive or fatal bounces. Apparently a Yahoo! SMTP server has been rejecting list emails with a "554 Message not allowed" error. Specifically it is saying "UP Email not accepted for policy reasons" but Yahoo! does not seem to be providing any further information.
I have had to implement some extra anti-Spam measures to tackle a recent upsurge in spam. It seems like everyone has noticed it - I would estimate that it started about two months ago. A lot of the new spam is related to (probably illegal) stock-pumping schemes. For example, today I have been getting about one email per hour trying to get me to invest in a company called Cana Petroleum. Their shares opened today at 4.50 and then mysteriously shot up to 10.10 before dropping back to 5.50 - go figure.
Someone recently forwarded me a strange email and asked me to explain it. It seems like it was notification of a delay sending an email due to the receiving SMTP server implementing Greylisting.
I am still handling all my email using mutt on a remote Debian box via ssh but I recently got mutt running on my iBook, with msmtp to do the actual sending. It is working fine and I will be migrating my config files and folders over from the remote box soon. This will allow me to start using GnuPG for signing and decrypting (currently I only use it for verifying and crypting because it would be foolish to put my private key on the remote box).
I will also have to start doing something about filtering out spam. It has been at a managable volume for years, but over the past few months it has started getting silly. I get all the usual bollox. Loads of emails just have some random text so I don't know what they are even about (payload is an attachment which I never open). Then there are the ads for online pharmacies, counterfeit designer watches etc, plus a recent surge in stock manipulation schemes such as one for Texhoma Energy.
In contrast to yesterday's entry about horribly broken email software, today I am reporting on something which looks potentially very cool. Mutt next generation, or mutt-ng as it is known, was forked from mutt with the goal of incorporating many of the cool mutt patches and creating an MUA which sucks even less than mutt. It has been around for a while but I only just became aware of it. From reading the development blog it seems that they started a complete rewrite of the code towards the end of last year, but there has been nothing posted since January this year.
I just read an entry on the lessig blog called Code, realised about DidTheyReadIt, a "service" which claims:
When you use didtheyreadit, every e-mail that you send
is invisibly tracked without alerting the recipient.
But when they read your message, you will immediately
receive the following information:
1) When, exactly, your email was opened.
2) How long your email remained opened.
3) Where, geographically, your email was viewed.
Their site by no means gives the full picture of how it works though, and in fact it only works if you send HTML email. I never send HTML email because it is a bad thing but admittedly many people do. I read all my email using mutt which displays the message body as plain text, so if the HTML is in the body the tracking won't work and furthermore I will be able to clearly see the presence of the web bug alerting me to the fact that the sender used DidTheyReadIt. If the HTML is in an attachment I might not bother opening it, but if I do it will be with my default viewer, which happens to be Links. So again the tracking won't work because Links will not attempt to load images.
For recipients that do view HTML email with a browser that automatically fetches images, the tracking does not really reveal where the email was viewed. The server attempts to geolocate the source IP address of the HTTP GET request that it receives, but geolocation based on IP is not particularly accurate or reliable - see for example Dr Svantesson's Geo-identification page.
Now to the interesting bit! How can they claim to know how long an email remained opened? Well it seems when the recipient's box does the HTTP GET for the web bug, it tries to retrieve the image but the DidYouReadIt server holds the TCP connection open by sending down a constant stream of data - Eeek! Check it out in this 2004 O'Reilly Network posting.
Analysis of the contents of millions of e-mails has revealed that less than 4% is legitimate traffic.
Black Cat Networks (my email service provider) today made an announcement about their policy of not responding to challenges posed by challenge response email systems. For those interested in their reasoning they referred to Challenge-Response Anti-Spam Systems Considered Harmful.
I have recently tidied up my .muttrc file, making use of the fact that the "lists" and "subscribe" commands now take regular expressions (when I first used them they would only match initial substrings). However, it seems I will need to be prepared to make another change at some point because the developers are discussing a new systematic variables naming scheme. For details check this wiki page (I voted for the change).
Pascal Hakim comments on bounce handling in list managers.
I have just created a page about the black hole called hotmail.
SAUCE (Software Against Unsolicited Commercial Email) is an SMTP server that sits between the Internet and your actual mail software. It was originally written to help in the fight against spam, but it also helps encourage good configuration and administration in general. I saw it mentioned somewhere recently but it doesn't seem to be under active development (latest beta is from 2004-01-15).
AOL has adopted a system called CertifiedEmail provided by Goodmail Systems. Some say that this sets a bad precedent and there is a Dear AOL campaign which aims to persuade AOL to reconsider. Goodmail Systems have responded with a Get the Facts rebuttal.
Despite Goodmail's protestations I get the impression that most CertifiedEmail is going to be marketing and sales stuff which I would not want to see anyway. If I was an AOL customer (more chance of Hell freezing over) I would be inclined to assume that all CertifiedEmail was spam - but then what if my bank started using CertifiedEmail to send out valid customer emails?
I manage a few mailing lists at riseup.net and in their latest newsletter they announced that they would be implementing Greylisting as an additional spam defence layer (it has now been enabled).
When I get an email with a Microsoft Word Document attached I sometimes assume that it must be spam and just delete the email without investigating any further. Other times it is obvious that it is not spam and that I should read the attachment, which should not be a problem since I have mutt configured to pipe such attachments through antiword. However, as often as not the offending document has been attached as application/octet-stream rather than application/msword and therefore can't be displayed. The workaround is to use Ctrl-E in mutt to "edit attachment content type" before attempting to view it.
The Internet Mail Consortium claim that their website has information about all the Internet mail standards and technologies. That is a bold claim, but they certainly seem to have plenty of information there.
I have recently been getting quite a lot of bounces for messages I never sent. Obviously someone is spamming with one of my email addresses in the From: header and there is not a lot I can do about it. I was thinking about creating an SPF record for my primary domain but when I started looking into it I found that many people consider SPF to be harmful. I will give a brief explanation of how it works and then a link to a page which explains why it is harmful.
The basic idea of SPF ("sender permitted from" a.k.a. "sender policy framework") is that it provides a mechanism for MTAs to determine whether an SMTP server is permitted to send email for a domain. It works by adding a DNS record for a domain which lists all the permitted SMTP servers. For example, every email I send out from a zenatode.org.uk address is sent from smokey.blackcatnetworks.co.uk so I could create an SPF record for zenatode.org.uk which contained only that server. If a server received an email from a different server which purported to be from a zenatode.org.uk address then it would be able to determine that it was bogus. Sounds like a good idea? Well check out Jonathan de Boyne Pollard's SPF is Harmful page and then see what you think.
To cut down on potential spam sources I never put my email address on the web. Instead I give the URL of my website - and that has a contact form. I have noticed at least one instance of an automated attack on my form mail script (looking to use it as a relay - which is not possible since I hard coded everything in the script). However, today for the first time I received a contact form spam! I suppose it could have been sent by a human but I suspect it was a bot. It will be interesting to see whether this type of spam becomes more frequent - I might have to modify my script. Incidentally, the spam was an effort to get me to visit www.666myth.co.nr with an invite to "Please check it out and have fun". I checked it out, but it is a pointless waste of time - almost but not quite entirely unlike fun. The spam claimed to be from "Titus" (webmaster of the useless waste of web space), the machine that actually made the contact is 216.239.94.84.
I recently sent an email to someone who it turned out is a knowspam.net user. I had never seen a system like this before but it worked well and seems like quite a good idea. If you read all your email from a POP email account and have a spam problem it might be worth considering.
I just had to log in to Yahoo! Groups and unbounce my address. It was again caused by an invalid header in a message sent to one of my subscribed groups, this time apparently generated by Opera 5.02 and of the form:
To: a <b@c>, d@e <f@g>, <h@i>
Exim correctly threw a 550 Syntax error but Yahoo! is wrong to assume that this implies anything about whether or not my email address is bouncing I am now firmly of the opinion that their bounce handling is seriously broken.
Further to yesterday's post about Yahoo! incorrectly flagging my email address as bouncing, it is apparently a known problem. Others agree that their system is broken, but I do not know whether anyone has tried to get them to fix it. I might try if/when I get some spare time.
I have just had another occurrence of a really annoying problem involving the abominations known as Microsoft Outlook Express and Yahoo! To set the scene, I am subscribed to a number of mailing lists that are run under Yahoo! Groups (of course I would prefer if they weren't but it is not up to me). Anyway, I have configured things so that the email gets sent to me at an address handled by Exim running on a Debian box. The problem occurs when someone using Outlook Express sends an email to one of my subscribed groups. For some reason (basically that it is broken) OE creates an invalid header:
To: <Undisclosed-Recipient:;>
The message, complete with mangled header, gets sent out to all subscribers including me, Exim throws a 550 Syntax error and rejects the message and Yahoo!, assuming my email address no longer works, flags it as "bouncing". The net result is that I stop getting email from any of my subscribed groups until I "unbounce" my address. This is a PITA, and it is not clear that the fault lies entirely with Outlook Express. Yes, it is generating a syntactically incorrect header, but such a header could easily be generated using other methods, and it should not be that easy to screw someone. Although one could argue that Exim is being overly strict, it seems to me that the real problem is with Yahoo!
I have tried configuring mutt to use GnuPG for signing/verifying and encrypting/decrypting email but I don't currently use this facility on a regular basis for a variety of reasons. First of all I don't really have anything to hide, and no reason to think that anyone would want to write something and falsely attribute it to me. Then even if I wanted to sign/encrypt stuff, very few of my regular contacts actually have PGP keys. Finally, I run mutt in a shell account on a box that is not under my control, and although I have no reason to distrust the administrators of that box I am reluctant to leave a valuable private key on it. I might revisit the issue in the future but for now I am not using PGP.
Although sending and receiving email is probably considered by many to be one of the most basic uses of the Internet, it is actually a remarkably complex process - as Lars Wirzenius said, "all programs that interact with e-mail are broken in one way or another".
Currently I handle all my personal email on a Debian box called smokey at Black Cat Networks where I also have my own domain registered. Their name servers have an MX record for my domain which points to smokey, so when someone sends an email to any username at my domain it should result in an SMTP connection to smokey, which is handled by the Exim Mail Transfer Agent (MTA). Exim is configured to accept email for my domain and to look in my home directory for dot forward files which I use to specify how mail for a given username is handled.
So what about reading my email? Although there are other alternatives I always read my email by making an ssh connection to smokey and running a Mail User Agent (MUA) called mutt, which seems to be the preferred choice of geeks everywhere. Mutt was originally written by Michael Elkins who said of it "All mail clients suck. This one just sucks less."
www.zenatode.org.uk Ian Gregory 2010