Beginning midday on 11/06/2017, widespread internet outages were reported throughout the United States, primarily among Comcast and Level 3 internet customers. These networks were gradually restored and stabilized, but the impact hit so many service holders, that we’re providing next-steps for those affected.
[11/08/2017] We will be updating this post as more information becomes available.
Level 3 (now known as Century Link), which provides partial support to other internet networks across the country, experienced a disruption on Monday, due to a configuration error (if you really want to go down a rabbit hole, you can read more about it here).
On Nov. 6, our network experienced a disruption affecting some IP customers due to a configuration error. All are restored.
— Level 3 Network Ops (@Level3NOC) November 6, 2017
The primary issues have been resolved within Level 3, but the outage has rippled through a number of other providers and web services, creating other issues and service disruptions.
What is the impact of the outage?
The outage, affecting Comcast and Level 3 internet customers, is hitting users in the form of sporadic disconnects, high latency, timeouts, and all around sluggish connections.
The outage is large enough that a number of cloud applications and web services were affected, and be behaving strangely in the hours & days following this event.
Even if you don’t receive internet service from these providers, you may be experiencing issues with other web-based services. Due to the widespread nature of the outage, many users are experiencing it in different ways, including the loss of services such as:
- Google services (Drive, Hangouts, Maps)
- Microsoft services (Azure, Sharepoint, Office 365)
- Quickbooks
- Skype
- Slack
- Yahoo Mail
Cloud Backup Services
For many of these services, there isn’t much you can do, besides wait for services to become available. However, if your environment relies on any pre-scheduled tasks and processes, those tasks may have been disrupted! This could include cloud storage and backup services.
If you rely on a web-based or cloud-based backup service, we highly recommend you check to ensure that any jobs or backups were not interrupted during this outage, because they may need to be rerun!
Who is affected?
The outage has affected a large population of Comcast and Level 3 service holders throughout the country. Comcast Enterprise (fiber) clients do not seem to be affected.
The following smaller service providers are also experiencing issues related to this outage:
- RCN
- Charter
- MetroPCS
- Time Warner Cable
You can find an in-depth list of affected services here.
Where is the outage occurring?
Outages as of 3 PM EST, 11/06/17
Thanks to DownDetector.com for their outage maps. View their site for updated customer reports on the status of these internet providers:
Comcast Map
Level 3 Map
When did the outage begin (and when will it be restored)?
Beginning around 12 PM EST, outages were being reported throughout the nation. As of 3 PM EST, customers are gradually seeing services return to normal, but many are still affected, and the outage may have affected your environment in ways that you haven’t yet detected, by extension of any cloud and web-based services that you’re using.
This post will be updated as more services are restored.
What should I do next…
…if I’m experiencing internet service trouble?
With regards to internet service, the best thing you can do is keep tabs on the outage, by checking status using DownDetector, or by following your providers on Twitter:
@Comcast
@Level3NOC
If this type of outage is commonplace for you, you might consider implementing a fail-over internet service—a number of our clients use these to avoid downtime, even if their primary provider has an outage.
…if I’m experiencing web app trouble?
There isn’t much you can do, besides wait for fixes to occur on the provider’s end. However, if your environment relies on web apps for any pre-scheduled tasks and processes, those tasks may have been disrupted.
If you run internal web apps on your own server, or depend on API integrations with other web platforms, you should inspect those processes to ensure everything is interacting the way it should be.
…if I rely web-based and cloud backup services?
If you rely on a web-based or cloud-based backup service, we highly recommend you check to ensure that any jobs or backups were not interrupted during this outage, because they may need to be rerun! This may seem obvious, but we also recommend that you have alerts in place, so that you’re immediately notified when backup processes succeed or fail.
