FC: Microsoft websites blacked out -- but what happened?

jf noonan jfn1 at msc.com
Thu Jan 25 07:15:05 PST 2001


On Wed, 24 Jan 2001, Marco Anglesio wrote:
>
> Btw, I don't think that those machines are necessarily close
> together. The IPs are, but the IPs probably translate to
> dispersed locations inside the MSFT firewall. Or so I'd
> expect. But perhaps I expect too much.

That would be a pretty unusual thing to do with your *public* resolvers, to firewall/NAT them. It's quite unclear to me what the point would be.

If you can believe M$'s explanation of the problem, then a single router misconfiguration caused this problem. That is because all their DNS servers were on the same network, which is an incredibly broken design that violates all common engineering practice and several RFC's.


>From MiniSquish:

http://www.microsoft.com/info/siteaccess.htm

Microsoft Explains Site Access Issues

On Tuesday evening and Wednesday, many Microsoft customers had difficulty accessing the company's Web sites. The cause has been determined, and the issue is resolved.

At 6:30 p.m. Tuesday (PST), a Microsoft technician made a configuration change to the routers on the edge of Microsoft's Domain Name Server network. The DNS servers are used to connect domain names with numeric IP addresses (e.g. 207.46.230.219) of the various servers and networks that make up Microsoft's Web presence.

The mistaken configuration change limited communication between DNS servers on the Internet and Microsoft's DNS servers. This limited communication caused many of Microsoft's sites to be unreachable (although they were actually still operational) to a large number of customers throughout last night and today.

This was an operational error, and not the result of any issue with Microsoft or third-party products nor the security of our networks. Microsoft regrets any inconvenience caused to customers due to this issue.

At approximately 5 p.m. Wednesday (PST), Microsoft removed the changes to the router configuration and immediately saw a massive improvement in the DNS network.

All sites are currently available to customers. Again, Microsoft apologizes for the inconvenience.



More information about the lbo-talk mailing list