Entries by Mark Jeftovic

We are cautiously optimistic that the Bell Canada mail issue has finally been resolved.

We are still running tests and keeping an eye out for reports and errors.

Stay tuned for a more detailed follow-up after we get a post-mortem of the situation.

Short version and something I'd like to put to rest right away: This wasn't "a Bell cock-up", it wasn't a Bell traffic shaping issue, and, in the words of one customer, this was not "typical Bell *mumble*".

Everything we experienced working this issue proved the exact opposite: Bell tech staff and NOC-ops are on-the-ball, responsive, clueful and they know what they're doing. Our hats are off to them because without their persistence which culminated in a breakthrough last night with one of their guys and one of our guys sitting on the phone, running packet captures and comparing tcpdumps until close to midnight, we would still be chasing our tails on this.

We are profoundly sorry for the heartache and pain this has caused some of our users. I will be making myself available for anybody who needs to make explanations downstream, to their clients, to their customers, or to their boss. Just shoot me an email or call me at extension 225 and we'll take it from there.

After several calm interludes where we think the problem has gone away, it starts back up again and did so again today.

To recap:

People attempting to send email FROM network points inside Bell Canada, Bell Nexxia or other Bell ips (like from an ISP downstream from Bell), are finding that sporadically and randomly their email is being attempted to be delivered to the A record for a domain instead of the MX record, which can cause bounces if the two are different and the former doesn't normally handle email.

We are well aware of the pain this is causing affected customers and while we're now running off of a DNS configuration we have been using for the better part of 10 years (i.e. no anycast at the moment) and we are only seeing this from Bell customers we have to emphasize that this issue is right now, and has been for close to two weeks: the number one absolute top priority within this company

We're working on this literally around the clock.

Here are some things you can try to lessen the pain if you are affected:

• If your A record is different from your MX and you have the ability to do so, have your A record relay mail for your domains over to the MX. That way when Bell (or anybody else) mistakenly delivers email to your A record host, instead of bouncing, it'll get to where it's actually supposed to go.

  
  

• If you are on Bell and email you are originating is encountering these problems, try setting your DNS resolver to: 67.69.184.199

  
  

• If you are on Bell, try setting your outbound SMTP handler to smtp10.bellnexxia.net

  
  

• If you are an easyDNS customer and your outbound mail via Bell is affected, you can use our easySMTP outbound mail service for free until this is resolved. Contact support to have it activated for your account.
  

Again, this is the number one priority here, we have many many mutual customers with Bell and it we are fully cognizant of the need to get this figured out.

We seem to have traced the recent Bell network mail delivery anomalies to unexpected behavior from the NS1 and NS2 anycast deployments.

For the moment, we have reverted NS1 and NS2 back to non-anycast nameservers.

We'll bring the anycast strands back online ASAP on new hostnames and commence an incremental cutover at that time.

Last friday night Bell NOC staff informed us that based on our reports they had found an issue with their DNS resolvers and would be committing a patch overnight friday. This appeared to rectify things until this morning we began receiving reports once again that email originating on the Bell Canada, Bellnexxia network is sporadically being delivered to the domain name's A record instead of the designated MX record.

We have been working closely with Bell NOC staff on this issue which is currently priority one here. Bell staff have been very responsive and forthcoming, we are happy to report. It's just that something weird is going on and we're all trying to figure out what that is.

We are only receiving reports of this behavior from users within the Bell Canada/Bellnexxia network at this time.

If this is having an impact on you, please forward as much information as you can to our support staff at support@easydns.com. We need mail headers, what local DNS resolvers you are configured to use, and if you can replicate the issue from a command line and email us the results we will be very much obliged.

Thanks to all for their co-operation and patience in this matter.

We are pleased to announce that NS2.EASYDNS.COM has now been deployed as an anycast strand with an initial four nodes as follows:

• Miami
  
• Phoenix
  
• London, UK
  
• Hong Kong
  

The new IP address for NS2 is 72.52.2.1

This compliments our earlier deployment of NS1.EASYDNS.COM as an anycast strand with nodes in

• Chicago
  
• San Jose
  
• Amsterdam
  
• Tokyo
  

DNS Anycast is where multiple servers around the world/internet respond to the same IP address, which results in faster query times as query packets are routed to the "closest" node to the querying server, in network topology terms. It also adds additional robustness for network anomolies and helps diffuse DDoS attacks.

You may notice a pattern here, we will be replacing the remaining "stand alone" nameserver nodes with a 3rd anycast strand within the next few months. After that point we will be delegating all domains to our three anycast strands instead of 6 "stand alone" nameservers. The new architecture is far more robust, actually increases the number of nameservers resolving your domains and improves response times.

Trivial but handy: I found myself having to email out some Fedex tracking ID's today, so I thought what would make it easy would be a way to create a redirect to the Fedex tracking page for that ID without having to visit a URL shortener site to create the redirect.

That's the core idea behind the "URL Widgets" or "Redirect Widgets" of easyURL, which are described here We also have them setup for Amazon products, domain lookups (surprise), Wikipedia pages and RFC's.

Many Ayromlou does it again, publishing another step-by-step tutorial, complete with screen shots on how to use your own domain name on easyDNS with Google Apps..

We are happy to report that NS1.EASYDNS.COM has been redeployed as a four-node anycast cluster with the following nodes:

• Chicago, Illinois
  
• San Jose, California
  
• Amsterdam, Netherlands
  
• Tokyo, Japan
  

What does this mean? For starters, the IP address of NS1.EASYDNS.COM is now 66.225.199.10, the old NS1 (in the Q9 datacenter in Toronto) is still answering queries and will be amalgamated with the other Q9 nameserver node: REMOTE2.EASYDNS.COM

Next, anycast provides a couple of benefits which further add to the reliability and performance of the easyDNS nameserver cluster. All nodes in the strand respond to the same IP address, the node which is "closest" (in network topology terms) should respond to the querying servers, adding speed to initial DNS resolution and providing a further layer of redundancy during network outages and associated phenomenon (like DDoS attacks).

With the deployment of NS1 as an anycast strand, combined with the nameservers deployed on the Prolexic anti-DDoS cleanpipe as well the NS6.EASYDNS.NET round-robin cluster, brings our nameserver cluster to nearly 20 nameservers deployed over 6 nodes in Canada, USA, the UK, The Netherlands and Japan (stand by for Hong Kong any day now) with at least dozen being active at any given time.

A number of factors have aligned to lead us to believe the worst is behind us for the DDoS attack on URL forwarding. The good news is some upgrades that were planned for a date to be named later basically happened overnight, so we've massively upgraded capacity on the URL forwarding pool and installed some new web application level firewalls on the forwarder webservers themselves.

All of the URL forwards we are testing now are loading, sometimes normally, sometimes slower.
We expect forwarding performance to steadily improve as the day goes on as the stale attack traffic "drops off".

We have a couple confirmations that Comcast is no longer filtering/discarding/losing email containing links created using our easyURL.net shortening/redirection service.

No word from Comcast on what happened.

We received a report today that Comcast seems to be discarding any email that contained a URL shortened with our easyURL shortening service:

"Just a note to let you know that the major cable provider/ISP Comcast is currently blocking all emails which include any "easyurl.net" link in them. No matter what I do if the URL contains an easyURL link Comcast doesn't deliver the message and (even worse) does not bounce the message back to me. Comcast simply discards the message. I've tested this with multiple emails with various forms of text in the body. Regardless of other content if there is an EasyURL link in it the email does not get delivered."

We have subsequently confirmed that any email containing the string "easyurl.net" does not arrive at comcast email destinations, and the sender does not receive a bounce message or any other notification on non-delivery.

About easyURL.net

easyURL is a URL shortening service launched in 2005 (and last year we added a social bookmarking component that supports openid.)

We know that spammers like to employ links created via URL shortening services to cloak their tracks and confuse trace attempts, so we have tried to make easyURL as useless as possible to the spammer.

• Our robots.txt file tells all robots and search engine spiders not to follow any URL under the easyURL.net domain
  
• We do not create redirects to any URL listed in the sURBL blacklist
  
• We do not create redirects to any URL listed in the black.uribl.org blacklist
  
• We perform keyword analysis for spam strings on the remote destination before we create the redirect
  
• We maintain a database of other URL shortening services and will not create a redirect to another URL shortener (spammers like to chain them together)
  

As you may gather, we don't like spam and you get zero linkpop from using easyURL. It is designed to be used as intended: a convenient way to shorten URLs.

For Comcast Users

You may want to contact Comcast and ask them if they are indeed, uhm, filtering your email based on content and discarding messages without telling you or notifying the sender. If so, that may be of concern to you.

You can try using easyURL with an alternate domain, we acquired a couple URL redirectors last year and are in the process of re-activating those names for new URL redirects. We will also register a few new names and make those available.

A modest proposal to search engines

As I thought of this predicament the idea occurred to me that URL redirectors could add a TXT record to their DNS which which would assert a "rel=nofollow" to all links under that domain a spider may come across in its crawls. Something like


easyurl.net IN TXT "html attrib rel=nofollow"


Or something like that. The key is to make URL redirects as useless to spammers as possible and one element of that is to completely remove any SEO lift that could be gotten via URL shorteners.