Server Outage

[CaptPatrick]

The problem with our site outages yesterday have been fixed...

The problem wasn't the server itself, but the fiber optics connection supplied by Verizon. Internet Connection, (our web server), assures me that Verizon has fixed the problem and we are operating at optimal efficiency now.

****************

From: "Brent Stock - Internet Connection" <bstock>
To: "recipients not shown:"
Subject: Regarding the September 21st service outage


To our valued customers:


Today at 12:05 and 5:10 PM EST, we experienced outages at the Verizon Central Office (CO). The CO is the first hop just down stream from us on the fiber optic connection. Our network, and our Internet Backbone Provider's network were both working perfectly, but with testing from all sides it was found that the problem was indeed at the CO. The initial indication was that it was an intermittent hardware issue on their circuit.


I am not going to gloss over the fact that we also went through the very same although shorter experience some months back. Back then Verizon assured everyone on all sides of the fiber connection that the issue was found and resolved (and we then assured you). I can't say for sure, but after 17 years of working with connectivity and providing hosting services (and also from them providing us the logging)... today sure felt like it was a recurrence of the same issue.


You may have noticed the connection bouncing up and down for about 15 minutes after it was restored this evening. That was from Verizon performing higher level stress testing of the connectivity. In this there was found to be a bad connection at the CO. Below is an email from Cavalier explaining this in more detail.


We have strenuously been working very closely with everyone to determine the cause, and to ensure there is a permanent solution for today's outages. After further testing tonight by us and Cavalier, it is determined that our network operations are in optimal condition.


We will continue to pursue this until we are 100% confident there is certainty of the resolution. We want to guarantee that such outages do not occur in the future so that we can maintain our reputation of reliability.


Thank-you for your patience!! We fully understand this was a huge inconvenience, and we very much take that seriously!


Please feel free to contact either myself or Support if you would like to learn more or discuss this further!


Once again... Thank You So Much!!


Brent Stock


President
Internet Connection, Inc.


----


Subject: RE: Easton Outage
From: xxxxxxxxxxxxxxxx
Date: Tue, 21 Sep 2010 23:37:55 -0400
To: xxxxxxxxxxxxxxxx ,<brent>
CC: "Jolley, Eugina A." <eajolley>


Brent,


As a follow up to our conversations, this is what was achieved while testing the circuit:


Prior to the dispatch of our tech to the Verizon Easton Central Office, Cavalier had not observed any problems in our network regarding this circuit.


Being that this was an ongoing trouble in your network, we dispatched our technician regardless. Upon his arrival at the Central Office (CO), he confirmed that the Cavalier network was in good working condition. This was achieved by checking the equipment, cables and the wiring ports (DSX jacks) in our network and the one leading to Verizon.


Verizon on the other hand, after their initial testing, observed some errors in their network. They narrowed it down to a defective DSX jack (wiring port) and the associated cable. Both the wiring port and the cable have been replaced. These actions we believe should ensure the stability of the circuit.


Since the work was completed, the circuit remains clean. Both Cavalier and Verizon will leave the associated tickets open for at least 48 hours to ensure the integrity of the circuit.


I will call in tomorrow evening to provide your tech an update on the performance monitor results from both networks. Please feel free to follow up with us if you need the results earlier than that.


We sincerely apologize for the inconvenience this outage may have caused you and your esteemed customers.


Thank You,

Network Operations
Cavalier Telephone

[Brewster Minton]

I thought it was me.

[CaptPatrick]

The quickest way to verify that your Internet connection is working on your computer is to visit a few very high profile sites with dedicated servers, such as yahoo.com, microsoft.com, google.com, msnbc.com, etc.

If you access those, but not a small site like ours, then it's a pretty good indicator that your computer isn't the problem.

Probably 85% - 90% of the websites on the Net use shared servers. That's an Internet server that hosts more than one client, or web address, on the same server. The fewer clients, the more costly the server space costs. A dedicated server, (one server hosting a single client), will be astronomically expensive. And still not immune to problems like we just experienced, being a malfunction with Verizon's fiber optics.

We're on a shared server, but with only a few other clients. Affordable, fairly fast, reasonably low server load at any given time, but far more likely to get caught in an occasional outage than the giants like Google.

You can take it to the bank though, that any time I see a server glitch on our site I'm up the host's ass like a tick. Fortunately, that's not often...