Around 12:30 PM on 1/11/2012 a customer called me to inform that their website wasn't loading and emails were bouncing.

After a very brief investigation, the root cause was traced to MyDomain.com.  At the bottom of this blog post is the email I sent to customers covering the issue.

Update:  At 3:20 PM the @MyDomainDotCom twitter account announced that the problem was resolved, and after the approximately 30 minute expected turnaround for DNS propagation, everything is back up and running.  That turnaround time may vary for different ISPs around the world as this corrected information must propagate around.

Update:  At 4:01 PM they finally explained the problem, albeit very briefly, along with a statement that it was resolved at 2:30 PM, which doesn't really seem correct given the 30 minute TTL which would have meant things would have started working again around 3PM but that was definitely not the case.

Final thought:  Overall I'm fairly unhappy about MyDomain's handling of the situation.  No blog, no explanation of the problem on Twitter or otherwise, beyond a vague acknowledgement that there was in fact some problem they were working to resolve - the problem likely affected thousands of businesses and Twitter was blowing up at the incident.  While overall MyDomain's track record for actual technology is great, going forward, I intend to use Mydomain less as an additional unnecessary point of potential failure, opting instead to prefer the built-in DNS management services at whatever domain registrar is being used for a domain.  I will not be moving all of my MyDomain domains elsewhere, however, as I do not feel that's warranted, unless this problem happens again with equally poor customer communication on their part.

I hope this email to our customers serves as an example of how to handle customer communication properly. (This was sent in two blocks to two separate customer groups at 2:10 and 2:30 PM as the explanations varied slightly for each, sent just as quickly as it could be written and properly explained.)

Hello,

You are receiving this message because a site you have hosted with us may be affected by a problem and not working correctly.

Symptom/Summary

1. One or more of your sites may be showing a junk sedoparking landing page, email may not be working, or AyaNova may not be launching.
2. 2. This problem is even affecting haveabyte.com’s own website and email. 
   (This fact is affecting some AyaNova installations which do not use a dedicated domain).
3. The problem is at MyDomain.com, not HaveAByte, and their technicians are working to resolve it.  ETA is 2-3 hours (give or take one hour), maybe sooner.
4. MyDomain is a major ISP and this problem is likely affecting thousands of domains, thousands of businesses.
5. I hope that by the time you actually read this, the problem will be fixed.

If you require more information, keep reading.

---

What is NOT happening

1. Your domain hasn’t expired
2. No data is lost
3. The HaveAByte web/email servers are not down
4. The problem is mostly affecting top-level domains (i.e., haveabyte.com will not load our website, but shop.haveabyte.com does indeed load our shop.  However, this does not apply to the “www.” prefix which is indeed affected, as well as “mail.” which is required for email to work.  Some subdomains, which several AyaNova accounts use, use a subdomain/prefix like this, and therefore may or may not be affected.)

What IS happening

1. Domains point to our server by way of a DNS record which says “this domain = this ip address”
2. Those records are managed through our MyDomain.com account (a service like Godaddy where we buy domains and manage DNS even for domains not purchased there.)
3. Today, suddenly and without warning and not in response to any action we took, those records started returning incorrect data.
4. This means that anything dependent on the domain name pointing to the correct hosting server is not going to work

We have been using MyDomain for nearly a decade and can count the number of problems on one hand. This is the first time ever seeing a problem of this scale.

Whatever caused this has nothing to do with any action on our part.  This happened suddenly and without warning and not in response to any incorrect configuration change made by HaveAByte to our account there.  It “just happened”.

What Is being done / How long will this last?

I am on hold with support as I write this email I just spoke with their customer support.

1. They’re aware of the problem
2. They’re working to fix it
3. 3. ETA is 2 – 2.5 hours to repair the problem, after which the TTL of 30 minutes goes into effect for a total possible turnaround of 3 hours
4. That is not a guaranteed turnaround, just an estimate, and they may need some wiggle room (how much I don’t know – maybe an hour, maybe two, so there is some chance it could drag on for around 5 hours, but I believe that is very unlikely)
5. They are posting updates to their twitter feed but the information there is light at best, currently just acknowledging the problem.

It is important to understand that any DNS change, intentional or not, that affects a domain can sometimes take up to 24 hours to repair – DNS records are stored in a central location (mydomain.com) and the rest of the internet gets copies of this on their own schedule, more or less.  They consider the records to be “good” because they came from an authoritative source.  Once  errors are corrected at the authoritative source, they cannot be “pushed” out to all the DNS servers on the internet – all that can be done is to simply wait for the rest of the internet’s DNS servers to come back and check for an update.  How often they are instructed to come back and check is based on MyDomain’s TTL (“time to live” which is attached to every domain they provide data for.)  MyDomain’s TTL is 30 minutes, meaning, it tells other DNS servers getting copies of its authoritative information to only trust that information for 30 minutes, and recommends they come back and check to see if anything has changed, though this may or may not be honored by all DNS systems on the Internet – but it generally is.  When it’s not honored, it can take up to 24-48 hours, but my experience with the DNS server community has been only of ever-shrinking TTLs and more aggressive updating for more real-time information.

They told me that an unknown problem (or at least a problem that this support agent did not personally know) caused many domains to have “default” DNS settings added to them inadvertently and they’re working to resolve it.

What about email that is bouncing for senders trying to reach me while it’s down?

Generally one of two things happen in this situation:

1. The email bounces immediately because the sender’s mailserver believes nothing is there or that there is some other kind of irrecoverable problem, it does not retry, but in this situation, it informs the sender so that they can take action
2. Their mail server holds the message and retries every so often for up to a day or two, and only bounces it back to the sender if the full retry window elapses with no positive result.

In either situation the sender is informed of a problem, and if they’re not informed, it’s because the server is still trying to send it out and there’s a chance you’ll get it later once your server comes back up.

If your business is highly dependent on things like automated notifications from an external ecommerce system where bounces wouldn’t be handled by a human, please be aware that you could miss some notifications and nobody would ever know unless you logged into the ecom system and looked directly at activity that occurred during this period.

How Can This Be Prevented In The Future?

Short of moving all DNS to another service, which is theoretically susceptible to whatever caused this, there isn’t a way to really prevent this. DNS is the glue that holds the Internet together, and when something goes wrong, there isn’t much recourse.  No matter who is running the DNS, problems can happen.  All we can do is work to resolve it as quickly as possible and try to learn as much as we can from the situation.

Not all HaveAByte customers are managed with MyDomain, in fact, these days that number is quite small and shrinking.  This is because lately we have been not using MyDomain for DNS on domains purchased through sites like Godaddy or Network Solutions in order to keep the domain registration and DNS management in the same place, since we found we need to manage those registration accounts anyway, meaning it’s quicker to make any changes the customer needs when we just leave DNS with the domain registration and not use an external management service.

If your domain was not purchased through MyDomain and you would like to de-couple your domain from them, let me know and I will reconfigure that for you at no cost.

I hope this information is considered thorough and complete and answers any questions you may have.

Feel free to call or email me at Knepfler@gmail.com if you have any other questions.

Erik Knepfler
714-369-5849
HaveAByte | Wegoz | Beautiful Engineering
@haveabyte | @eknepfler | @wegozcom
Facebook | LinkedIn | G+ | Skype | Carrier Pigeon

 

Big Brother for your website!

HaveAByte was recently was granted technical responsibilities for advance-tapes.com / advancecollege.org - a website with legal case summaries going back to 1992, developed by the late Professor Steven R. Finz and recently taken under the wing of Pincus Legal Education, a client of HaveAByte.  I wouldn't be surprised if the website goes back nearly that far as well.  It's a pretty amazing website for those interested in these legal case summaries, there are thousands of cases posted.  If you saved every page on the site to your PC, there would be over 6000 files.  I also truly love the design - I'm a real minimalist.  :)

Of course, in this modern age, we use databases to store smaller bits of information, use shared include files (to share content across pages), and then would build the case summary pages dynamically with code pulling the cases from a database.  This entire site could potentially be around 50 actual source pages, and easy to update.  However, amazingly, it is actually 6000 unique HTML files due to how far back it dates.  Looking at the site, however, you'd never know this - the menu footer and various other common elements are virtually identical across all 6000+ pages, as it was very well constructed.

It's a bit like going back in time to when everything was simply an HTML page and edited manually, but when you add the modern expectations that clients have which have evolved over time, it presents an interesting challenge to update.

Enter TextCrawler developed by Digital Volcano in the UK.

This amazing, free little application has made maintaining this site a lot easier.  From their website:

TextCrawler is a fantastic tool for anyone who works with text files. This cool utility enables you to instantly find and replace words and phrases across multiple files and folders.  It utilises a powerful Regular Expression engine to enable you to create sophisticated searches, perform batch operations, extract text from files and more. It is fast and easy to use, and as powerful as you need it to be.

Some features:

• • Find and Replace across files
  • Fast searching, even on large files.
  • Simple to use interface
  • Flexible search parameters
  • Text Extractor - rip text into a new file
  • Search and replace using Regular Expressions. Create sophisticated searches.
  • Regular Expression test tool
  • Regular Expression library - Save your searches.
  • Create backup files
  • Highlighted search results
  • Export Results
  • Batch find and replace operations

TextCrawler is freeware, but any donations towards future development are gratefully received. If in the future a pay-for version of Textcrawler is released all contributors will receive a full licence!  (I just donated 10 GBP.)

My favorite part of this application is the real-time preview of exactly what your replacement is going to replace.  In the case of regular expressions, it's easy sometimes to accidentally write a "greedy" expression that replaces more than it should, which in this case would be devastating, forcing a restoration from backup.  Essentially this application has allowed me to fulfill the requests of the client nearly as well as if the site was constructed on a modern database or CMS platform.  I look forward to rebuilding this site using modern techniques when and if it makes sense to do so, but for now, I'm glad I have this little application to help ease the pain a bit.  The best things in life are indeed free!

(This blog entry is about PrestaShop, an e-commerce web application for which HaveAByte provides services.  For more information about our offering, please visit our PrestaShop page.)

About a year ago I built shop.haveabyte.com using PrestaShop 1.1.0.5.  The experience was fairly good and at the end I wound up recommending Prestashop to a lot of people.  I had a lot of trouble with the Paypal interface, but I got it to work well enough to allow me to launch the site.  I also do a lot of Magento maintenance and personally I like the back-end interface to PrestaShop much better for smaller projects where you just want to sell a few items, even though PrestaShop is also well designed in some areas to scale very well.

One of our customers, Julie Fogg of ActivePort, who uses our hosted AyaNova solution, and for whom we also built a very nice Joomla website (http://www.activeport.com), has referred a lot of business to HaveAByte.  So, when she came to me saying she wanted to build an ecommerce website for her daughter and wanted it done in just a few days, I jumped at the chance.  It's great to have real-world practice on web applications like this.  It's great to have a real site to be working on when experimenting with software like this - you're just not quite as driven to solve problems when you make an "ABC Test Shop" and just start poking at it randomly.  It was also a great opportunity to repay the favors of all the referrals from Julie!

So her daughter, Olivia, came up with BananaFoot.net, Julie registered it and I hooked it into the server.  I downloaded and installed PrestaShop 1.2.0.5 and set file permissions.  Just like that, less than 30 minutes later, we're up and running.

Julie bought a nice template from http://www.prestashopthemes.net/pres-4 and it took just a few minutes to install.  The template had just 3 original PhotoShop PSDs for the main logo and homepage images which didn't take too long to tweak with an image of Olivia and modify the text a bit.  And just like that, within an hour or so, we have this:

The top menu and footer were pretty easy to edit, just some .TPL files full of HTML in the theme folders.  Some of the pages are powered by PrestaShop's CMS (Content Management System) feature, which provides a simple in-browser editor for making pages such as About Us, Olivia's Photo Gallery, and so on.

The Back Office interface makes adding items and categories a snap.  After just a few minutes of instruction over Skype, Julie was able to take over this part of it while I worked on some of the other back-end configuration details.

The Shipping setup is a little odd, but once you figure it out, it suddenly makes sense.  You create carriers and price ranges, then map them together, then price the carriers.  It feels a bit convoluted and as if you're working backwards but once it's set up it works well enough.

The sample data included provided tons of countries and zones, and if you plan to ship internationally, it's a great setup.  When you only want to ship in the US, it takes a little time to disable all of those other countries properly.  Changing the store defaults from French to English (PrestaShop is a french-borne application) and changing the currency from Euro to Dollar is simple enough.

Now that everything was ready to go, it's time to sell!  There's a modules page where you enable and disable various modules, and the Payment Methods you wish to support are all modules.  I disabled all of them except Paypal.  The Paypal module's configuration has only a small handful of options, mainly the Paypal address and whether you want to run it in Sandbox mode or not.

This is where things get a little tricky.

After placing an order via Paypal, it returns the user to their account Order History screen, but says there are no orders.  Searching around on the forums seems to indicate that this problem has appeared off and on for many users and no clear explanation is readily available yet.  I've followed all of the instructions in both the Paypal module's help screen as well as the wiki and tried many forum suggestions.

The basic issue is that whether IPN is off or on, the Paypal purchase ends on a page with a form and button to return to BananaFoot.net.  Inspection of the form shows that it's sending a ton of information back to Prestashop by way of the order-confirmation.php file in the root folder.  Unfortunately, when this information reaches this page, order-confirmation.php (or some other dependent sub-script) is throwing away the information, not writing anything to the orders table, and then dumping the user to the history screen - which of course then says that they have not placed any orders.  The order is recieved and processed OK via Paypal, and even with IPN on, Paypal confirms that the IPN data is being sent properly.  But no order history is tracked on Presta's side.

Amazingly, I went back to the HaveAByte Shop and tested it there - same problem, to my surprise.

For now I've decided to create a simple CMS page explaining that the order has been recieved, and redirect the user there from order-confirmation.php.  An ugly hack but it will have to do since the goal is to be online ASAP.

Fortunately, within a day or two of posting to the forums and PrestaShop bug tracker, this page was revealed to me

http://www.prestashop.com/forums/viewthread/17934/P15/third_party_modules/paypal_2_dot_1_beta_module

I will be trying this soon and will update this blog entry with more information as it develops!  If you're looking to set up a PrestaShop using Paypal as the primary payment method, I'd encourage you to study up on these issues.  I haven't tried the Google Checkout module yet but my sense is that it does not have these problems, since most fixes I've seen attempted by the community lie in the Paypal module, though it could be sharing techniques with the Google Checkout module in this area.  Hard to say really.  I'd simply suggest you understand that you may need to place a lot of orders to get it all working just right (having a test item and test shipping both at $0.01 or using a Paypal Sandbox account helps [if you can get one - Paypal has been having issues lately creating sandbox accounts]).