easyDNS Blog

Bell Canada mail issue seems to be solved

nospam@example.com (Mark Jeftovic) — 2008-04-29T22:46:56Z

We are cautiously optimistic that the Bell Canada mail issue has finally been resolved.

We are still running tests and keeping an eye out for reports and errors.

Stay tuned for a more detailed follow-up after we get a post-mortem of the situation.

Short version and something I'd like to put to rest right away: This wasn't "a Bell cock-up", it wasn't a Bell traffic shaping issue, and, in the words of one customer, this was not "typical Bell *mumble*".

Everything we experienced working this issue proved the exact opposite: Bell tech staff and NOC-ops are on-the-ball, responsive, clueful and they know what they're doing. Our hats are off to them because without their persistence which culminated in a breakthrough last night with one of their guys and one of our guys sitting on the phone, running packet captures and comparing tcpdumps until close to midnight, we would still be chasing our tails on this.

We are profoundly sorry for the heartache and pain this has caused some of our users. I will be making myself available for anybody who needs to make explanations downstream, to their clients, to their customers, or to their boss. Just shoot me an email or call me at extension 225 and we'll take it from there.

Bell Canada mail problems to easyDNS domains

nospam@example.com (Mark Jeftovic) — 2008-04-28T20:53:04Z

After several calm interludes where we think the problem has gone away, it starts back up again and did so again today.

To recap:

People attempting to send email FROM network points inside Bell Canada, Bell Nexxia or other Bell ips (like from an ISP downstream from Bell), are finding that sporadically and randomly their email is being attempted to be delivered to the A record for a domain instead of the MX record, which can cause bounces if the two are different and the former doesn't normally handle email.

We are well aware of the pain this is causing affected customers and while we're now running off of a DNS configuration we have been using for the better part of 10 years (i.e. no anycast at the moment) and we are only seeing this from Bell customers we have to emphasize that this issue is right now, and has been for close to two weeks: the number one absolute top priority within this company

We're working on this literally around the clock.

Here are some things you can try to lessen the pain if you are affected:

If your A record is different from your MX and you have the ability to do so, have your A record relay mail for your domains over to the MX. That way when Bell (or anybody else) mistakenly delivers email to your A record host, instead of bouncing, it'll get to where it's actually supposed to go.
If you are on Bell and email you are originating is encountering these problems, try setting your DNS resolver to: 67.69.184.199
If you are on Bell, try setting your outbound SMTP handler to smtp10.bellnexxia.net
If you are an easyDNS customer and your outbound mail via Bell is affected, you can use our easySMTP outbound mail service for free until this is resolved. Contact support to have it activated for your account.

Again, this is the number one priority here, we have many many mutual customers with Bell and it we are fully cognizant of the need to get this figured out.

NS1 and NS2 rolled back off of anycast

nospam@example.com (Mark Jeftovic) — 2008-04-24T18:14:00Z

We seem to have traced the recent Bell network mail delivery anomalies to unexpected behavior from the NS1 and NS2 anycast deployments.

For the moment, we have reverted NS1 and NS2 back to non-anycast nameservers.

We'll bring the anycast strands back online ASAP on new hostnames and commence an incremental cutover at that time.

Bell Canada / Bellnexxia mail delivery problems return

nospam@example.com (Mark Jeftovic) — 2008-04-23T16:26:48Z

Last friday night Bell NOC staff informed us that based on our reports they had found an issue with their DNS resolvers and would be committing a patch overnight friday. This appeared to rectify things until this morning we began receiving reports once again that email originating on the Bell Canada, Bellnexxia network is sporadically being delivered to the domain name's A record instead of the designated MX record.

We have been working closely with Bell NOC staff on this issue which is currently priority one here. Bell staff have been very responsive and forthcoming, we are happy to report. It's just that something weird is going on and we're all trying to figure out what that is.

We are only receiving reports of this behavior from users within the Bell Canada/Bellnexxia network at this time.

If this is having an impact on you, please forward as much information as you can to our support staff at support@easydns.com. We need mail headers, what local DNS resolvers you are configured to use, and if you can replicate the issue from a command line and email us the results we will be very much obliged.

Thanks to all for their co-operation and patience in this matter.

NS2 now deployed as a second anycast strand

nospam@example.com (Mark Jeftovic) — 2008-04-16T19:00:22Z

We are pleased to announce that NS2.EASYDNS.COM has now been deployed as an anycast strand with an initial four nodes as follows:

Miami
Phoenix
London, UK
Hong Kong

The new IP address for NS2 is 72.52.2.1

This compliments our earlier deployment of NS1.EASYDNS.COM as an anycast strand with nodes in

Chicago
San Jose
Amsterdam
Tokyo

DNS Anycast is where multiple servers around the world/internet respond to the same IP address, which results in faster query times as query packets are routed to the "closest" node to the querying server, in network topology terms. It also adds additional robustness for network anomolies and helps diffuse DDoS attacks.

You may notice a pattern here, we will be replacing the remaining "stand alone" nameserver nodes with a 3rd anycast strand within the next few months. After that point we will be delegating all domains to our three anycast strands instead of 6 "stand alone" nameservers. The new architecture is far more robust, actually increases the number of nameservers resolving your domains and improves response times.

remote2.easydns.com - Scheduled maintenance

nospam@example.com (easyDNS Support) — 2008-04-11T20:47:52Z

To increase the responsiveness of "remote2.easydns.com", we have scheduled some emergency maintenance for the nameserver tomorrow on Saturday, April 12th.

Downtime is expected to be minimal, and the nameserver should return to normal operation shortly afterwards.

We apologise for any inconvenience.

Update: The maintenance will take place between 6:00am and 6:30am EST tomorrow morning (once again, Saturday April 12th).

Please note: ORDB anti-spam list no longer operational...

nospam@example.com (easyDNS Support) — 2008-03-26T18:46:17Z

A number of our customers who maintain their own mailservers have called reporting issues with the delivery of their email in the last 24 hours. If you are experiencing something similar, please ensure that you are not using the ORDB anti-spam list.

The ORDB anti-spam list was shut down in December 2006, and in an effort to fully deactivate the list, ORDB is now sending out false positives. This means that if your mailserver relies on the ORDB anti-spam list, your mailserver is more than likely rejecting ALL EMAIL that is being relayed to it.

Please ensure you remove your mailserver's dependence on ORDB, as this will correct this specific issue.

Discussion about this recent development with ORDB can be found at the following URL upon Slashdot:

http://it.slashdot.org/article.pl?sid=08/03/25/2124224

Webforwarding Outage.

nospam@example.com (easyDNS Support) — 2008-03-12T16:51:17Z

There was a brief 15 minute outage to our web forwarding service (between 12:50EST and 13:05EST).
We experienced a hardware failure which caused the service interuption. The issue has been resolved and webforwarding is back
online.

Please do not hesitate to contact support should you have any questions or concerns.

easyURL adds "FEDEX" tracking widget

nospam@example.com (Mark Jeftovic) — 2008-03-06T20:08:39Z

Trivial but handy: I found myself having to email out some Fedex tracking ID's today, so I thought what would make it easy would be a way to create a redirect to the Fedex tracking page for that ID without having to visit a URL shortener site to create the redirect.

That's the core idea behind the "URL Widgets" or "Redirect Widgets" of easyURL, which are described here We also have them setup for Amazon products, domain lookups (surprise), Wikipedia pages and RFC's.

How to use your own domain name with Google Apps

nospam@example.com (Mark Jeftovic) — 2008-03-02T23:40:09Z

Many Ayromlou does it again, publishing another step-by-step tutorial, complete with screen shots on how to use your own domain name on easyDNS with Google Apps..

Steps involved in a .COM, .NET or .ORG domain transfer

nospam@example.com (DomainHELP) — 2008-02-25T03:19:55Z

(These steps in transferring a .COM, .NET or .ORG domain to a new registrar also apply to .BIZ and .INFO or any domain within a Top Level Domain that runs on EPP)

In these explanations "losing registrar" means the domain registrar the domain is currently using before the domain transfer, the "gaining registrar" is the domain registrar you are transferring the domain to.

Here are the steps:

NS1 deployed as anycast strand

nospam@example.com (Mark Jeftovic) — 2008-02-19T16:23:03Z

We are happy to report that NS1.EASYDNS.COM has been redeployed as a four-node anycast cluster with the following nodes:

Chicago, Illinois
San Jose, California
Amsterdam, Netherlands
Tokyo, Japan

What does this mean? For starters, the IP address of NS1.EASYDNS.COM is now 66.225.199.10, the old NS1 (in the Q9 datacenter in Toronto) is still answering queries and will be amalgamated with the other Q9 nameserver node: REMOTE2.EASYDNS.COM

Next, anycast provides a couple of benefits which further add to the reliability and performance of the easyDNS nameserver cluster. All nodes in the strand respond to the same IP address, the node which is "closest" (in network topology terms) should respond to the querying servers, adding speed to initial DNS resolution and providing a further layer of redundancy during network outages and associated phenomenon (like DDoS attacks).

With the deployment of NS1 as an anycast strand, combined with the nameservers deployed on the Prolexic anti-DDoS cleanpipe as well the NS6.EASYDNS.NET round-robin cluster, brings our nameserver cluster to nearly 20 nameservers deployed over 6 nodes in Canada, USA, the UK, The Netherlands and Japan (stand by for Hong Kong any day now) with at least dozen being active at any given time.

URL Forwarding DDoS status (worst is over we think)

nospam@example.com (Mark Jeftovic) — 2008-02-09T15:44:16Z

A number of factors have aligned to lead us to believe the worst is behind us for the DDoS attack on URL forwarding. The good news is some upgrades that were planned for a date to be named later basically happened overnight, so we've massively upgraded capacity on the URL forwarding pool and installed some new web application level firewalls on the forwarder webservers themselves.

All of the URL forwards we are testing now are loading, sometimes normally, sometimes slower.
We expect forwarding performance to steadily improve as the day goes on as the stale attack traffic "drops off".

URL Forwarding

nospam@example.com (easyDNS Support) — 2008-02-08T16:19:37Z

We are currently experiencing a DOS attack from Eastern Europe and as such our URL forwarding service is experiencing intermittent connectivity issues. Our systems and development staff are currently working on the issue and we hope for a speedy resolution.

thank you
easyDNS

Primary Web Forwarder Trouble

nospam@example.com (easyDNS Support) — 2008-02-08T01:33:38Z

Greetings,

Between approximately 6:00pm and 8:00pm we had intermittent network trouble with our primary web forwarder. We have corrected the problem and are watching the situation closely. During the trouble the web forwarder would have had difficulty serving web pages, with most requests being dropped (although some may still have completed).

Thank you for your patience.

Update: We had another disruption this morning between 5:00am and 6:00am which would have intermittently impacted web forwarding. We have normalized the situation and are continuing to monitor.