I am now using smtp1.bethere.co.uk as my outgoing SMTP server and I don't seem to have any problems sending email to hotmail accounts. I don't plan on doing any more hotmail testing so I have just terminated my test account.
Someone has suggested that phx.gbl be included in the Wikipedia entry about pseudo-top-level domains, but it needs someone to come forward with a reliable source of information - right now it is just speculation.
Just noting the URL of the Microsoft MSN Postmaster website where you can find Microsoft's answers to some frequently asked questions.
Since my last entry I have changed my outgoing SMTP server (when Black Cat Networks closed down their ADSL service and transferred customers to Nitrex). My new outgoing server is mail.nitrex.net but I have not done any testing to see what Microsoft thinks about it.
A few weeks ago I attended an event in London where I asked someone about the phx.gbl domain. I didn't fully understand his reply but he implied that it is some sort of default internal domain name for a piece of Microsoft enterprise software.
I have managed to make a little more progress on my own. If I issue the command "dig @ns1.msft.net gbl" then under the AUTHORITY SECTION I get:
gbl. 3600 IN NS cpipsdnsp01.phx.gbl.
A google search for cpipsdnsp01 only brings up 14 results, the first of which is for this posting about root server traffic on the ICANN/GNSO GNSO (Generic Names Supporting Organisation) email list, which points out that cpipsdnsp01.phx.gbl claims to be authoritative for both .gbl and .local.
Awareness of this issue is gradually spreading. Someone called Ian Bell noticed the problem and posted about it on FoRK, where John Capo of Tuffmail.com responded by describing the results of some tests he had done - "Recent test show its worse than I could have ever imagined".
I wasn't expecting the new Black Cat Networks spam forwarding policy to have resulted in any improvement so soon but I tried sending my standard pair of test messages today and this time one of them did get through. Then I realised that both might have been marked down by the Hotmail muncher because it may consider them to be "overly simple" and they were both repeats of previous messages. So I started from scratch by writing a brand new email in mutt, and sent it to my Hotmail test account without using any special tricks to try to get it past the filter. It got through OK but clearly Hotmail has a bizarre concept of what spam looks like because although I tried to make it as unspammy as possible (realistic message with no likely spam words etc) it still got delivered to the junk folder. Still, it does seem like it is at least giving email from Black Cat Networks a fighting chance now.
Faced with Hotmail's failure to acknowledge that their system is broken, Black Cat Networks have implemented a technical solution to prevent likely spam being sent to Hotmail addresses from their SMTP server. See the BCN spam forwarding page.
Someone contacted me with further information about Hotmail which has been posted on the mozillaZine forum (info originally from "Cassini" and posted to forum by Daifne on 2006-07-05). This is interesting and confirms a few of my findings:
Cassini appears to have derived some Baysian coefficients which also indicate that HTML is preferred and plain text deprecated.
This is probably affecting millions of Hotmail users and yet there seems to be little awareness of the problem. It is like having a delivery person who just burns certain letters on a whim rather than depositing them in your mail box.
More strangeness to report. Hotmail has reverted to black hole mode albeit with at least one slight difference.
Yesterday I tried to send an email to someone with a hotmail.com address, and to get an indication of whether they were likely to have received it I Bcc'd it to my own Hotmail test account. Well I almost immediately got a delivery failure for my account due to "mailbox unavailable". I tried logging in to my Hotmail account and realised that it had been suspended because I had not logged in for over 30 days - fair enough. I reactivated the account and sent a test message to it, and lo and behold the message vanished without trace. This tells us something. It seems that the message is not vaporised before checking whether the recipient mailbox is available.
OK, so it seems the old problem is back. The next check was to send my standard pair of test messages which are identical except for the subject line - with one being "test 888" and the other being "Re: test 888". On 2006-04-29 when I tried them the former vanished and the latter got through, on 2006-05-08 both got through. Yesterday both vanished!
So it seems that the problem is back, and that the trick of prepending "Re: " to the subject of a new email no longer works. When I sent an email from my reactivated Hotmail account and then replied to it, the reply got through.
There is a bit more information about the phx.gbl domain in an article at artific.com called On The Importance of Reverse DNS.
Meanwhile, I heard that other Black Cat users have also been reporting problems sending to Hotmail users where the email gets silently dropped. Black Cat have been on to Hotmail and their independent content filtering provider Symantec but have not reached any useful conclusions as yet.
Someone contacted me a few days ago after reading this page. He has noticed that MSN messenger relays messages via machines on the mysterious phx.gbl domain and is as puzzled as me about it.
I did some more testing today and a test email which was consistently vanishing before is now getting through.
I have not had time to work on this since my last entry but I thought I would just do a quick test to see if Hotmail is still acting like a black hole. I composed a pair of new messages, one with "Subject: test 888" and the other with "Subject: Re: test 888". The former vanished and the latter ended up in the junk folder. This is just what I expected based on my previous tests, so yes, Hotmail is still broken.
In the early hours this morning I posted to this thread on comp.mail.mutt (the same one I mentioned in yesterday's entry) and have now had a response. The suggestion was to:
Try sticking copies of the References: line into entirely new emails constructed the same way as the ones that went missing.
In fact I had already tried that but I didn't get very far because mutt seemed to prevent me from doing so - but that is a mutt issue not a Hotmail one. So what I tried next was to choose an email that I had received from a non-Hotmail account and reply to it - but while composing the reply I changed the To: header so that it would go to my Hotmail account. This email got through fine, which told me that Hotmail does not mind what is in the References: and In-Reply-To: headers as long as they exist.
At this point I posted again in the thread on comp.mail.mutt but since then I have made more progress. I composed two new messages in mutt and sent them to my Hotmail account. The only difference was in the Subject: header - one contained "test xxx" and the other contained "Re: HI". Guess what happened? The first one vanished and the second got through (albeit to the junk folder). This raised an obvious question. Is it simply the presence or absence of "Re: " in the Subject: header that determines whether the message gets vaporised? To test this I replied to the email I had sent myself from the Hotmail account but edited the Subject: header and changed it to "test xxx" - it got through. So now I have two ways of persuading Hotmail not to throw away email that I send to my account:
Of course both of these are horrible kludges and the fact that I have to resort to such shenanigans is a clear indication that Hotmail is seriously broken. I wonder how many people are being affected by this?
Over the past few months I have had a few emails apparently vanish without trace and the only common factor is that they were emails I sent to Hotmail accounts. In each case I received no indication that the email had failed to be delivered and I only realised something was wrong when I failed to receive a reply. In one case I asked the recipient (who happened to be my father) to double check, look in their junk folder etc, but to no avail. I knew that some of my messages were reaching the troublesome addresses but I only recently spotted a pattern - replies were getting through but not new messages. At this point I should probably describe my email setup which is basically:
Debian 3.1
Exim version 3.36 #1 built 08-Apr-2005 20:01:27
Mutt 1.5.9i (2005-03-13)
A couple of weeks ago I read a thread on comp.mail.mutt titled "mutt not sending to Hotmail addresses but sendmail will..." Now this was not exactly the same problem but it was similar in that email sent to Hotmail accounts using mutt was sometimes vanishing without trace (read the thread in question for details).
To do any further testing I would have to bite the bullet and sign up for a Hotmail account. The account I created was @hotmail.co.uk and the big question was whether or not this account would exhibit the same symptoms. I started off by sending three test emails to the new account and 24 hours later none of them had shown up - no bounce messages, nothing in the inbox or the junk folder. So then I sent an email to myself from the test account and replied to it - the reply arrived in the test account almost immediately. Finally I sent another test message and 12 hours later there is no sign of it.
The only significant difference between the email that got through and one that didn't is that the successful one had two extra headers:
References: <BAY15-F228DD799B24C92118E639484BF0@phx.gbl>
In-Reply-To: <BAY15-F228DD799B24C92118E639484BF0@phx.gbl>
At this point I am at a bit of a loss as to what to try next so I will probably post something in comp.mail.mutt and see what comes back.
In the meantime I was intrigued by that phx.gbl domain in the Hotmail Message-ID because gbl does not appear to be a valid TLD. A little Googling gave me some more clues, check this out:
$ whois 64.4.8.116|grep OrgName
OrgName: MS Hotmail
$ host 64.4.8.116
116.8.4.64.in-addr.arpa domain name pointer by1sch4041907.phx.gbl.
$ host by1sch4041907.phx.gbl
Host by1sch4041907.phx.gbl not found: 3(NXDOMAIN)
Just what the fsck is Microsoft up to?
www.zenatode.org.uk Ian Gregory 2010