jain.com — data centers, connectivity and security, analyzed daily

Tag: Uptime

  • Namecheap’s 15-Hour Outage Shows Why Denser AI Halls Can’t Rely on One Chiller Plant

    Namecheap’s 15-Hour Outage Shows Why Denser AI Halls Can’t Rely on One Chiller Plant

    TL;DR · 30-second read

    The Short Version

    A storm damaged the cooling system at a data center in Phoenix. A data center is a warehouse-sized building full of computers that must be kept cold to keep working. Without enough cooling, the web company Namecheap went dark for about 15 hours.

    Websites, email and even Namecheap’s own customer help line went offline. Fixing the cooling took hours. Getting everything else running again took much longer.

    This matters more as computers built for artificial intelligence, which run far hotter, fill new buildings. Backup cooling inside one building may not be enough.

    Namecheap, a leading domain registrar and web host, suffered an outage of roughly 15 hours on August 14 after what it called “a failure of cooling systems” at its data center in Phoenix. TechRadar reported that the company’s hosting plans, its EasyWP managed WordPress service, DNS (the internet’s address book, which points domain names at servers) and email were all offline, along with customer accounts and Namecheap’s own support desk. The facility’s operator, PhoenixNAP, said heavy overnight storms had caused damage that led to “elevated white space temperatures” in the data halls.

    Two of the data center’s four chillers were back online around 12pm EDT. Core databases returned at 5:30pm EDT, Namecheap.com and live chat about 40 minutes later, and full service at 3:50am EDT. Chief executive Hillan Klein apologised, saying “we failed you today,” and committed to publishing a post-mortem.

    Executive Summary

    A storm damaged the cooling plant of a Phoenix data center, and one of the best-known names in domain registration and small-business hosting went down for most of a day. Customers lost websites, email and name resolution. They also lost the usual way to get help, because Namecheap’s help desk and live chat went down with everything else.

    The incident matters beyond Namecheap’s customers because it isolates a failure mode the industry usually discusses in terms of power: the thermal plant. The building had four chillers, the industrial refrigeration units that produce the cold water used to pull heat out of server rooms. That redundancy did not prevent an outage when a single external event, a storm, affected the cooling system. Even after partial cooling returned, restoring the service stack took many more hours.

    That lesson gets sharper as operators pack far hotter AI hardware into new halls. When heat density rises, the time between losing cooling and losing servers shrinks. Resilience that stops at the building’s own chiller plant leaves less margin than it once did.

    Four Chillers, One Storm

    Data center cooling is normally designed with spare capacity. The common “N+1” approach means one more chiller than the load requires, so a single unit can fail or be serviced without consequence. Namecheap’s Phoenix facility had four chillers. Bringing two of them back online was the milestone that began recovery, which indicates the plant had been running well short of what the halls needed. Neither company had said, as of August 15, exactly how many units failed or which part of the cooling system the storm damaged.

    This is the core distinction between component redundancy and site resilience. Spare chillers protect against one machine breaking. They protect far less well against a common-mode event, meaning a single cause that hits several pieces of equipment at once. Storm damage to shared infrastructure is exactly that kind of event. PhoenixNAP’s own explanation traced the problem to weather rather than to an isolated equipment fault, so the redundancy on paper did not match the resilience in practice.

    Cooling Came Back Hours Before the Service Did

    The published timeline is the most instructive part of this incident. Partial chiller capacity returned around noon. Core databases did not come back until 5:30pm, and the public website and live chat returned about 40 minutes after that. Full service was not restored until 3:50am. Most of the outage therefore happened after the thermal problem had begun to ease.

    That gap reflects how hosting platforms behave after a thermal event. When a room overheats, servers throttle, shut themselves down, or are powered off by operators to protect the hardware. Bringing them back is not a single switch. Storage has to be verified, databases checked for consistency, and dependent services restarted in the right order. For anyone writing recovery objectives, the lesson is that restoring the cooling plant starts the recovery clock. It does not stop it.

    Why AI Halls Can’t Rely on One Chiller Plant

    Namecheap’s outage involved conventional web hosting, not AI hardware. The mechanism still applies directly to AI buildout. A room’s temperature rises after cooling fails at a rate set by how much heat the equipment keeps producing. Conventional hosting racks give operators some buffer. AI training and inference racks, built around dense GPU servers, concentrate far more heat in the same floor space, so the window for an orderly shutdown or failover gets shorter.

    Liquid cooling, now standard in many AI designs, moves heat off chips more efficiently than air. It still hands that heat to the facility’s central plant: chillers, dry coolers or cooling towers. A storm that damages that plant affects a liquid-cooled AI hall as surely as it affected Namecheap’s hosting floor, and the higher density leaves less time to react. The practical conclusion for operators and tenants is that resilience needs to extend beyond one plant. That can mean heat-rejection equipment that is physically separated and weather-hardened, or the ability to move workloads to another site. Buyers of AI capacity have reason to ask colocation providers how their cooling redundancy performs against shared causes, not only against a single failed unit.

    When the Help Desk Shares Fate With the Outage

    The outage also took down Namecheap’s support channels, leaving staff unable to answer live chat or email. The company fell back on its status page and on Klein’s posts on X. Namecheap did keep customers informed with detailed status updates throughout. The episode still shows why tools for talking to customers during a crisis are best hosted somewhere other than the infrastructure they report on.

    Klein’s commitment to examine “where our existing safeguards and protocols failed” is the right question. The test will be whether the post-mortem addresses architecture as well as the storm, specifically whether services like DNS and email had anywhere else to run when one building lost its cooling.

    Background

    Namecheap is one of the most widely used domain registrars, the companies that sell and manage website addresses. It also sells web hosting, managed WordPress through its EasyWP service, email and DNS, mostly to individuals and small businesses. Because many customers buy their domain, website and email from Namecheap together, one infrastructure failure can take down a business’s whole online presence at once.

    Data centers spend a large share of their engineering effort on cooling, because nearly all the electricity servers use becomes heat. Chilled-water plants are the workhorse of larger facilities and are normally built with spare units. Heat density per rack is rising sharply as AI hardware spreads, so the reliability of this thermal plant is becoming as central to uptime as power supply and backup generators.

    Sources

    Source: TechRadar: ‘We failed you today’: Namecheap down for several hours after a data center cooling failure, leaving customers furious, covering the roughly 15-hour Namecheap outage caused by a storm-related cooling failure at its Phoenix data center.

  • AI Workloads Shift Data Center Focus From Uptime to Resilience

    AI Workloads Shift Data Center Focus From Uptime to Resilience

    An analysis published by Data Center Frontier on May 22, 2026 argues that the rise of AI workloads is reshaping how data center operators define and manage risk, moving the conversation beyond the long-standing focus on uptime toward a broader notion of resilience that spans power, cooling, network, and workload recovery.

    Executive Summary

    The piece reframes a debate that has quietly been building for several years. For decades, the data center industry benchmarked itself on uptime — the percentage of time facilities remained available, typically measured against Uptime Institute tier definitions. AI training and inference workloads, with their concentrated power draw, thermal density, and tightly coupled cluster behavior, expose the limits of that single metric.

    Why it matters: buyers of colocation and cloud capacity have historically negotiated on service-level agreements built around availability. If the operative risk is now cluster-level disruption, cooling excursions, or grid interaction rather than isolated component failure, the contracts, insurance, and design standards that underpin the industry will need to evolve alongside the hardware.

    Uptime Was Built for a Different Workload

    The uptime-first mindset was calibrated for enterprise and early cloud workloads: many independent servers, stateless front ends, and applications that tolerated the loss of a node without disrupting the service. A five-nines facility (99.999 percent availability, roughly five minutes of downtime a year) was a defensible proxy for customer experience because software above it was designed to route around small failures.

    AI training clusters behave differently. A single training job may span thousands of GPUs (graphics processing units, the specialized chips that do the heavy math for AI models) synchronized on every step. A brief power event, a cooling excursion, or a network partition can force a checkpoint restart that costs hours of compute and, at current GPU rental rates, meaningful money. Availability at the facility level says little about whether the job actually finishes.

    Resilience Is a Wider Surface

    Resilience, as the source frames it, is a superset of uptime. It includes how quickly a site can ride through a grid disturbance, whether liquid cooling loops degrade gracefully under partial failure, how the network fabric behaves when a spine switch drops, and how workloads are checkpointed so that a disruption does not erase a day of training. Each of those is a distinct engineering discipline, and each has its own vendors, standards, and blind spots.

    That widening surface also expands who bears the risk. Uptime SLAs put the operator on the hook for a narrow, well-defined failure mode. Resilience, by contrast, is a shared problem: the utility, the operator, the cooling vendor, the network provider, and the customer’s own software all shape whether a workload survives a bad afternoon. Contract structures have not caught up.

    What Changes for Buyers and Operators

    For operators, the practical implication is that design margins that looked conservative in a CPU-era facility can look thin under AI density. Rack power draws that used to sit in the 5 to 15 kilowatt range are now routinely quoted in the tens to over a hundred kilowatts per rack for GPU deployments, which stresses power distribution, cooling headroom, and the assumptions baked into concurrent maintainability. Retrofitting a legacy hall is not always cheaper than greenfield.

    For buyers, the negotiation should widen. Beyond the availability guarantee, questions worth asking include how the site responds to grid frequency events, how cooling redundancy is validated under load rather than at commissioning, what the network’s failure domains look like, and whether the operator can produce evidence — not just design documents — of resilience under stress. None of this makes uptime irrelevant; it just makes uptime insufficient.

    Background

    The data center industry has organized itself for decades around the Uptime Institute’s tier system, which rates facilities from Tier I to Tier IV based on redundancy and concurrent maintainability. That framework, alongside vendor SLAs measured in nines of availability, became the common vocabulary for negotiating colocation and cloud contracts.

    The rapid buildout of AI training and inference capacity from roughly 2023 onward has introduced rack densities, power profiles, and workload behaviors that the tier framework was not designed around. Industry publications including Data Center Frontier have been tracking the resulting rethink of design standards, power procurement, and cooling architecture.

    Source: From Uptime to Resilience: AI Infrastructure Changes the Data Center Risk Equation — Data Center Frontier analysis on how AI workloads are reshaping data center risk management.