Status

We are cautiously optimistic that the Bell Canada mail issue has finally been resolved.

We are still running tests and keeping an eye out for reports and errors.

Stay tuned for a more detailed follow-up after we get a post-mortem of the situation.

Short version and something I'd like to put to rest right away: This wasn't "a Bell cock-up", it wasn't a Bell traffic shaping issue, and, in the words of one customer, this was not "typical Bell *mumble*".

Everything we experienced working this issue proved the exact opposite: Bell tech staff and NOC-ops are on-the-ball, responsive, clueful and they know what they're doing. Our hats are off to them because without their persistence which culminated in a breakthrough last night with one of their guys and one of our guys sitting on the phone, running packet captures and comparing tcpdumps until close to midnight, we would still be chasing our tails on this.

We are profoundly sorry for the heartache and pain this has caused some of our users. I will be making myself available for anybody who needs to make explanations downstream, to their clients, to their customers, or to their boss. Just shoot me an email or call me at extension 225 and we'll take it from there.

After several calm interludes where we think the problem has gone away, it starts back up again and did so again today.

To recap:

People attempting to send email FROM network points inside Bell Canada, Bell Nexxia or other Bell ips (like from an ISP downstream from Bell), are finding that sporadically and randomly their email is being attempted to be delivered to the A record for a domain instead of the MX record, which can cause bounces if the two are different and the former doesn't normally handle email.

We are well aware of the pain this is causing affected customers and while we're now running off of a DNS configuration we have been using for the better part of 10 years (i.e. no anycast at the moment) and we are only seeing this from Bell customers we have to emphasize that this issue is right now, and has been for close to two weeks: the number one absolute top priority within this company

We're working on this literally around the clock.

Here are some things you can try to lessen the pain if you are affected:

• If your A record is different from your MX and you have the ability to do so, have your A record relay mail for your domains over to the MX. That way when Bell (or anybody else) mistakenly delivers email to your A record host, instead of bouncing, it'll get to where it's actually supposed to go.

  
  

• If you are on Bell and email you are originating is encountering these problems, try setting your DNS resolver to: 67.69.184.199

  
  

• If you are on Bell, try setting your outbound SMTP handler to smtp10.bellnexxia.net

  
  

• If you are an easyDNS customer and your outbound mail via Bell is affected, you can use our easySMTP outbound mail service for free until this is resolved. Contact support to have it activated for your account.
  

Again, this is the number one priority here, we have many many mutual customers with Bell and it we are fully cognizant of the need to get this figured out.

We seem to have traced the recent Bell network mail delivery anomalies to unexpected behavior from the NS1 and NS2 anycast deployments.

For the moment, we have reverted NS1 and NS2 back to non-anycast nameservers.

We'll bring the anycast strands back online ASAP on new hostnames and commence an incremental cutover at that time.

Last friday night Bell NOC staff informed us that based on our reports they had found an issue with their DNS resolvers and would be committing a patch overnight friday. This appeared to rectify things until this morning we began receiving reports once again that email originating on the Bell Canada, Bellnexxia network is sporadically being delivered to the domain name's A record instead of the designated MX record.

We have been working closely with Bell NOC staff on this issue which is currently priority one here. Bell staff have been very responsive and forthcoming, we are happy to report. It's just that something weird is going on and we're all trying to figure out what that is.

We are only receiving reports of this behavior from users within the Bell Canada/Bellnexxia network at this time.

If this is having an impact on you, please forward as much information as you can to our support staff at support@easydns.com. We need mail headers, what local DNS resolvers you are configured to use, and if you can replicate the issue from a command line and email us the results we will be very much obliged.

Thanks to all for their co-operation and patience in this matter.

We are pleased to announce that NS2.EASYDNS.COM has now been deployed as an anycast strand with an initial four nodes as follows:

• Miami
  
• Phoenix
  
• London, UK
  
• Hong Kong
  

The new IP address for NS2 is 72.52.2.1

This compliments our earlier deployment of NS1.EASYDNS.COM as an anycast strand with nodes in

• Chicago
  
• San Jose
  
• Amsterdam
  
• Tokyo
  

DNS Anycast is where multiple servers around the world/internet respond to the same IP address, which results in faster query times as query packets are routed to the "closest" node to the querying server, in network topology terms. It also adds additional robustness for network anomolies and helps diffuse DDoS attacks.

You may notice a pattern here, we will be replacing the remaining "stand alone" nameserver nodes with a 3rd anycast strand within the next few months. After that point we will be delegating all domains to our three anycast strands instead of 6 "stand alone" nameservers. The new architecture is far more robust, actually increases the number of nameservers resolving your domains and improves response times.

We are happy to report that NS1.EASYDNS.COM has been redeployed as a four-node anycast cluster with the following nodes:

• Chicago, Illinois
  
• San Jose, California
  
• Amsterdam, Netherlands
  
• Tokyo, Japan
  

What does this mean? For starters, the IP address of NS1.EASYDNS.COM is now 66.225.199.10, the old NS1 (in the Q9 datacenter in Toronto) is still answering queries and will be amalgamated with the other Q9 nameserver node: REMOTE2.EASYDNS.COM

Next, anycast provides a couple of benefits which further add to the reliability and performance of the easyDNS nameserver cluster. All nodes in the strand respond to the same IP address, the node which is "closest" (in network topology terms) should respond to the querying servers, adding speed to initial DNS resolution and providing a further layer of redundancy during network outages and associated phenomenon (like DDoS attacks).

With the deployment of NS1 as an anycast strand, combined with the nameservers deployed on the Prolexic anti-DDoS cleanpipe as well the NS6.EASYDNS.NET round-robin cluster, brings our nameserver cluster to nearly 20 nameservers deployed over 6 nodes in Canada, USA, the UK, The Netherlands and Japan (stand by for Hong Kong any day now) with at least dozen being active at any given time.

A number of factors have aligned to lead us to believe the worst is behind us for the DDoS attack on URL forwarding. The good news is some upgrades that were planned for a date to be named later basically happened overnight, so we've massively upgraded capacity on the URL forwarding pool and installed some new web application level firewalls on the forwarder webservers themselves.

All of the URL forwards we are testing now are loading, sometimes normally, sometimes slower.
We expect forwarding performance to steadily improve as the day goes on as the stale attack traffic "drops off".

We have a couple confirmations that Comcast is no longer filtering/discarding/losing email containing links created using our easyURL.net shortening/redirection service.

No word from Comcast on what happened.