easyDNS Blog - Status

Bell Canada mail issue seems to be solved

nospam@example.com (Mark Jeftovic) — Tue, 29 Apr 2008 18:46:56 -0400

We are cautiously optimistic that the Bell Canada mail issue has finally been resolved.

We are still running tests and keeping an eye out for reports and errors.

Stay tuned for a more detailed follow-up after we get a post-mortem of the situation.

Short version and something I'd like to put to rest right away: This wasn't "a Bell cock-up", it wasn't a Bell traffic shaping issue, and, in the words of one customer, this was not "typical Bell *mumble*".

Everything we experienced working this issue proved the exact opposite: Bell tech staff and NOC-ops are on-the-ball, responsive, clueful and they know what they're doing. Our hats are off to them because without their persistence which culminated in a breakthrough last night with one of their guys and one of our guys sitting on the phone, running packet captures and comparing tcpdumps until close to midnight, we would still be chasing our tails on this.

We are profoundly sorry for the heartache and pain this has caused some of our users. I will be making myself available for anybody who needs to make explanations downstream, to their clients, to their customers, or to their boss. Just shoot me an email or call me at extension 225 and we'll take it from there.

Bell Canada mail problems to easyDNS domains

nospam@example.com (Mark Jeftovic) — Mon, 28 Apr 2008 16:53:04 -0400

After several calm interludes where we think the problem has gone away, it starts back up again and did so again today.

To recap:

People attempting to send email FROM network points inside Bell Canada, Bell Nexxia or other Bell ips (like from an ISP downstream from Bell), are finding that sporadically and randomly their email is being attempted to be delivered to the A record for a domain instead of the MX record, which can cause bounces if the two are different and the former doesn't normally handle email.

We are well aware of the pain this is causing affected customers and while we're now running off of a DNS configuration we have been using for the better part of 10 years (i.e. no anycast at the moment) and we are only seeing this from Bell customers we have to emphasize that this issue is right now, and has been for close to two weeks: the number one absolute top priority within this company

We're working on this literally around the clock.

Here are some things you can try to lessen the pain if you are affected:

If your A record is different from your MX and you have the ability to do so, have your A record relay mail for your domains over to the MX. That way when Bell (or anybody else) mistakenly delivers email to your A record host, instead of bouncing, it'll get to where it's actually supposed to go.
If you are on Bell and email you are originating is encountering these problems, try setting your DNS resolver to: 67.69.184.199
If you are on Bell, try setting your outbound SMTP handler to smtp10.bellnexxia.net
If you are an easyDNS customer and your outbound mail via Bell is affected, you can use our easySMTP outbound mail service for free until this is resolved. Contact support to have it activated for your account.

Again, this is the number one priority here, we have many many mutual customers with Bell and it we are fully cognizant of the need to get this figured out.

NS1 and NS2 rolled back off of anycast

nospam@example.com (Mark Jeftovic) — Thu, 24 Apr 2008 14:14:00 -0400

We seem to have traced the recent Bell network mail delivery anomalies to unexpected behavior from the NS1 and NS2 anycast deployments.

For the moment, we have reverted NS1 and NS2 back to non-anycast nameservers.

We'll bring the anycast strands back online ASAP on new hostnames and commence an incremental cutover at that time.

Bell Canada / Bellnexxia mail delivery problems return

nospam@example.com (Mark Jeftovic) — Wed, 23 Apr 2008 12:26:48 -0400

Last friday night Bell NOC staff informed us that based on our reports they had found an issue with their DNS resolvers and would be committing a patch overnight friday. This appeared to rectify things until this morning we began receiving reports once again that email originating on the Bell Canada, Bellnexxia network is sporadically being delivered to the domain name's A record instead of the designated MX record.

We have been working closely with Bell NOC staff on this issue which is currently priority one here. Bell staff have been very responsive and forthcoming, we are happy to report. It's just that something weird is going on and we're all trying to figure out what that is.

We are only receiving reports of this behavior from users within the Bell Canada/Bellnexxia network at this time.

If this is having an impact on you, please forward as much information as you can to our support staff at support@easydns.com. We need mail headers, what local DNS resolvers you are configured to use, and if you can replicate the issue from a command line and email us the results we will be very much obliged.

Thanks to all for their co-operation and patience in this matter.

NS2 now deployed as a second anycast strand

nospam@example.com (Mark Jeftovic) — Wed, 16 Apr 2008 15:00:22 -0400

We are pleased to announce that NS2.EASYDNS.COM has now been deployed as an anycast strand with an initial four nodes as follows:

Miami
Phoenix
London, UK
Hong Kong

The new IP address for NS2 is 72.52.2.1

This compliments our earlier deployment of NS1.EASYDNS.COM as an anycast strand with nodes in

Chicago
San Jose
Amsterdam
Tokyo

DNS Anycast is where multiple servers around the world/internet respond to the same IP address, which results in faster query times as query packets are routed to the "closest" node to the querying server, in network topology terms. It also adds additional robustness for network anomolies and helps diffuse DDoS attacks.

You may notice a pattern here, we will be replacing the remaining "stand alone" nameserver nodes with a 3rd anycast strand within the next few months. After that point we will be delegating all domains to our three anycast strands instead of 6 "stand alone" nameservers. The new architecture is far more robust, actually increases the number of nameservers resolving your domains and improves response times.

remote2.easydns.com - Scheduled maintenance

nospam@example.com (easyDNS Support) — Fri, 11 Apr 2008 16:47:52 -0400

To increase the responsiveness of "remote2.easydns.com", we have scheduled some emergency maintenance for the nameserver tomorrow on Saturday, April 12th.

Downtime is expected to be minimal, and the nameserver should return to normal operation shortly afterwards.

We apologise for any inconvenience.

Update: The maintenance will take place between 6:00am and 6:30am EST tomorrow morning (once again, Saturday April 12th).

NS1 deployed as anycast strand

nospam@example.com (Mark Jeftovic) — Tue, 19 Feb 2008 11:23:03 -0500

We are happy to report that NS1.EASYDNS.COM has been redeployed as a four-node anycast cluster with the following nodes:

Chicago, Illinois
San Jose, California
Amsterdam, Netherlands
Tokyo, Japan

What does this mean? For starters, the IP address of NS1.EASYDNS.COM is now 66.225.199.10, the old NS1 (in the Q9 datacenter in Toronto) is still answering queries and will be amalgamated with the other Q9 nameserver node: REMOTE2.EASYDNS.COM

Next, anycast provides a couple of benefits which further add to the reliability and performance of the easyDNS nameserver cluster. All nodes in the strand respond to the same IP address, the node which is "closest" (in network topology terms) should respond to the querying servers, adding speed to initial DNS resolution and providing a further layer of redundancy during network outages and associated phenomenon (like DDoS attacks).

With the deployment of NS1 as an anycast strand, combined with the nameservers deployed on the Prolexic anti-DDoS cleanpipe as well the NS6.EASYDNS.NET round-robin cluster, brings our nameserver cluster to nearly 20 nameservers deployed over 6 nodes in Canada, USA, the UK, The Netherlands and Japan (stand by for Hong Kong any day now) with at least dozen being active at any given time.

URL Forwarding DDoS status (worst is over we think)

nospam@example.com (Mark Jeftovic) — Sat, 09 Feb 2008 10:44:16 -0500

A number of factors have aligned to lead us to believe the worst is behind us for the DDoS attack on URL forwarding. The good news is some upgrades that were planned for a date to be named later basically happened overnight, so we've massively upgraded capacity on the URL forwarding pool and installed some new web application level firewalls on the forwarder webservers themselves.

All of the URL forwards we are testing now are loading, sometimes normally, sometimes slower.
We expect forwarding performance to steadily improve as the day goes on as the stale attack traffic "drops off".

URL Forwarding

nospam@example.com (easyDNS Support) — Fri, 08 Feb 2008 11:19:37 -0500

We are currently experiencing a DOS attack from Eastern Europe and as such our URL forwarding service is experiencing intermittent connectivity issues. Our systems and development staff are currently working on the issue and we hope for a speedy resolution.

thank you
easyDNS

Primary Web Forwarder Trouble

nospam@example.com (easyDNS Support) — Thu, 07 Feb 2008 20:33:38 -0500

Greetings,

Between approximately 6:00pm and 8:00pm we had intermittent network trouble with our primary web forwarder. We have corrected the problem and are watching the situation closely. During the trouble the web forwarder would have had difficulty serving web pages, with most requests being dropped (although some may still have completed).

Thank you for your patience.

Update: We had another disruption this morning between 5:00am and 6:00am which would have intermittently impacted web forwarding. We have normalized the situation and are continuing to monitor.

Telephone Support

nospam@example.com (easyDNS Support) — Fri, 01 Feb 2008 16:10:37 -0500

Due to the severity of todays winter storm, telephone support will
be available until 5:00PM EST only.

We do apologise for the inconvenience.

thank you
easyDNS

Update: Comcast no longer filtering on easyURL links

nospam@example.com (Mark Jeftovic) — Sun, 27 Jan 2008 16:27:30 -0500

We have a couple confirmations that Comcast is no longer filtering/discarding/losing email containing links created using our easyURL.net shortening/redirection service.

No word from Comcast on what happened.

Comcast silently discarding email containing URLs shortened via easyURL

nospam@example.com (Mark Jeftovic) — Sat, 26 Jan 2008 19:52:54 -0500

We received a report today that Comcast seems to be discarding any email that contained a URL shortened with our easyURL shortening service:

"Just a note to let you know that the major cable provider/ISP Comcast is currently blocking all emails which include any "easyurl.net" link in them. No matter what I do if the URL contains an easyURL link Comcast doesn't deliver the message and (even worse) does not bounce the message back to me. Comcast simply discards the message. I've tested this with multiple emails with various forms of text in the body. Regardless of other content if there is an EasyURL link in it the email does not get delivered."

We have subsequently confirmed that any email containing the string "easyurl.net" does not arrive at comcast email destinations, and the sender does not receive a bounce message or any other notification on non-delivery.

About easyURL.net

easyURL is a URL shortening service launched in 2005 (and last year we added a social bookmarking component that supports openid.)

We know that spammers like to employ links created via URL shortening services to cloak their tracks and confuse trace attempts, so we have tried to make easyURL as useless as possible to the spammer.

Our robots.txt file tells all robots and search engine spiders not to follow any URL under the easyURL.net domain
We do not create redirects to any URL listed in the sURBL blacklist
We do not create redirects to any URL listed in the black.uribl.org blacklist
We perform keyword analysis for spam strings on the remote destination before we create the redirect
We maintain a database of other URL shortening services and will not create a redirect to another URL shortener (spammers like to chain them together)

As you may gather, we don't like spam and you get zero linkpop from using easyURL. It is designed to be used as intended: a convenient way to shorten URLs.

For Comcast Users

You may want to contact Comcast and ask them if they are indeed, uhm, filtering your email based on content and discarding messages without telling you or notifying the sender. If so, that may be of concern to you.

You can try using easyURL with an alternate domain, we acquired a couple URL redirectors last year and are in the process of re-activating those names for new URL redirects. We will also register a few new names and make those available.

A modest proposal to search engines

As I thought of this predicament the idea occurred to me that URL redirectors could add a TXT record to their DNS which which would assert a "rel=nofollow" to all links under that domain a spider may come across in its crawls. Something like



easyurl.net IN TXT "html attrib rel=nofollow"

Or something like that. The key is to make URL redirects as useless to spammers as possible and one element of that is to completely remove any SEO lift that could be gotten via URL shorteners.

New Years Eve, New Years Day Support

nospam@example.com (easyDNS Support) — Mon, 31 Dec 2007 11:51:31 -0500

Telephone support will be offered until 3:00 PM EST, December 31st, 2007
and will be unavailable on New Years Day, January 1st, 2008.

thank you,
the easyDNS team.

Propagation speeds back to normal

nospam@example.com (Mark Jeftovic) — Thu, 13 Dec 2007 08:10:27 -0500

Normal speeds for DNS update propagation were restored at approximately 3:30am this morning, sorry for not posting the resolution to this earlier. Apologies to all affected. Suffice it to say, the next version of the easyDNS platform (which we are launching in early '08) uses a different DNS backend methodology at the master server which will avoid bottleneck situations of the type encountered.