Network Issue Takeaways

One of the biggest takeaways from this week’s network issue was our communication problem. Everything that could have possible gone wrong seemed to and our customer outreach was poor. We will improve and here is a list of some things we are implementing asap as a result. This is not the final list but a work-in-progress.

Our office phone number was not failing over. That will be fixed. In the meantime, if there were to ever be an emergency such as this you can reach me (Brad Seiler) at 402-500-0007. That is my Google Voice number that should find me.

The @boberdoo.com email server happened to be on the affected IP range. We will be separating our email server either to Gmail or another network.

Our website was moved into Amazon about a year ago so we could communicate with our clients if our network was down. Unfortunately, the people that know how to post a message at the top of our website were not available so our notices were only posted on Twitter and Facebook. Other people will be trained in editing the website so this problem will not reoccur.

We will notify clients sooner and more often even if the problem appears to be something that will be fixed shortly. This was a problem that was constantly, "a few minutes," from being fixed. The problem was not concrete like a broken router or hard drive that needed to get replaced and would take X amount of time before we were back up. In the future we will provide more frequent updates, even if the update is that we do not have any new information yet.

We have collected emergency contact information from many clients in the past but we would like to verify this information and make sure it is in our system. Several of us will be keeping hard copies of this on our phones and will be texting and emailing until our thumbs bleed if this were to happen again and our automated methods for updating clients fail.

Please fill out the following information with your emergency contact information and an optional backup person who will also receive emails/SMS messages from us if there is an emergency in the future. Please fill out this form as soon as possible – https://www.boberdoo.com/emergency-contacts/

In terms of infrastructure, we have been using roughly the same host and infrastructure for many years now (8+). We have been working on moving boberdoo clients into Amazon but it is not something that can be rushed and we will not move current clients until we are very comfortable. New clients will eventually go there and then current clients will be migrated.

Amazon, or any other cloud provider, is not a panacea to all problems such as those seen this week. More scalable? Yes. More ability to setup redundancies? Yes. Worse support if very bad things happen? Definitely. Less control and ability to fix things yourself when there are problems? Definitely. It is a tradeoff. We think it does make sense and we will get there but only when we know it is better than where we are today.

We have been in business since 2000. That is a pretty long time. Rarely have we seen problems like this but they do happen. We promise we will do what we can to prevent them and to make sure our communication with you is better if they do.

Brad Seiler

boberdoo.com