Power & Electrical Infrastructure Design

Philosophy:

Data centers must balance redundancy, efficiency, and scalability in their power design. USDCS focuses on high-density power distribution, future-proof expansion, and compliance with NEC and NFPA 70E.

Our Approach:

  1. Power Capacity Planning:

    • Load calculations based on AI/HPC workloads.

    • Designs for Tier III and Tier IV redundancy models.

    • Analysis of UPS, PDUs, and generator sizing.

  2. Grid Interconnection & Utility Negotiations:

    • USDCS assists in securing capacity agreements with regional transmission operators (RTOs) and independent system operators (ISOs).

  3. High-Efficiency Distribution Design:

    • Evaluating medium-voltage (MV) vs. low-voltage (LV) architecture.

    • Implementing 480V and 415V power delivery for optimized efficiency.

  4. Redundancy & Resilience Planning:

    • 2N and N+1 architectures for critical loads.

    • Battery energy storage systems (BESS) for grid independence.

Real-World Experience:

  • USDCS designed a 2.4MW electrical system for a high-density AI compute facility, utilizing liquid-cooled GPU clusters with an advanced busway power distribution system.

Electrical substation with transformers and switchgear

Designing the power system for a data center is a critical task, as it must ensure reliability, scalability, and efficiency while meeting the stringent uptime requirements (typically aiming for 99.999% availability, also known as “five nines” uptime). Below is an outline of a best practice power design for data centers:

1. Power Supply Sources

  • Dual Utility Feeds: Ideally, a data center should be fed by two separate utility power lines from different substations. This ensures that if one line fails, the other will continue to provide power. These utility feeds should be configured in a way that ensures reliability, ideally through two independent transformers in separate locations.

  • Backup Generators: Uninterruptible power is crucial. Diesel generators (or other fuel types, depending on location and application) are used for long-term backup power. Generators should be capable of running the entire data center's critical load and have enough capacity to run for several days if necessary. Typically, N+1 or 2N redundancy is employed for the generator systems (meaning one or two spare generators to ensure reliability).

2. Uninterruptible Power Supply (UPS) Systems

  • UPS Configuration: UPS systems provide short-term backup power to protect against power fluctuations, such as sags or surges, and to bridge the gap until the backup generators kick in. The best practice is to use modular UPS systems that offer flexibility in terms of scalability and redundancy.

    • N+1 Redundancy: Ensures that the system has one additional unit to cover for failure. For example, if the data center requires 4 UPS units, having 5 ensures that one unit can always be in reserve.

    • Battery Technology: The battery banks are typically designed using lead-acid or lithium-ion batteries. Lithium-ion is gaining popularity due to its longer lifespan and reduced maintenance requirements, though it comes at a higher upfront cost.

3. Power Distribution Units (PDUs)

  • Rack PDUs: These distribute power to servers and other IT equipment within racks. They come with features like metering, remote monitoring, and surge protection.

  • Redundancy: PDUs should be configured in an N+1 or 2N redundancy scheme to ensure continuous operation in the event of a failure. The PDUs should be connected to different UPS units to further enhance redundancy.

4. Power Distribution at the Rack Level

  • Dual Power Supplies to Racks: Each rack should have at least two independent power sources, connected to different PDUs or UPS units. This ensures that if one source fails, the other can take over without causing downtime.

  • Power Segmentation: Segmentation of power within the data center helps isolate faults. By dividing the data center into zones, power issues in one section won’t impact others.

5. Electrical Distribution System (Mains Distribution)

  • Switchgear: High-voltage switchgear and transformers should be used to step down the power from utility voltage to the levels required by the data center. Redundant transformers with N+1 or 2N configurations are typically employed to ensure uninterrupted service.

  • Busbars: Busbars distribute electrical power throughout the facility and are essential for connecting transformers to switchgear, and switchgear to PDUs and other parts of the facility.

6. Cooling & Power Integration

  • Load Balancing: Efficient cooling systems (like CRAC/CRAH units) need to be in balance with power needs. As servers consume power, they generate heat, which must be managed effectively. Power systems should support the cooling infrastructure, which may include HVAC systems, in-row cooling, and liquid cooling.

7. Power Monitoring and Management

  • SCADA (Supervisory Control and Data Acquisition) Systems: A comprehensive monitoring system for tracking power usage, generator status, battery levels, UPS performance, and the overall health of the electrical infrastructure.

  • Energy Management Software: The system should offer real-time reporting and analytics, which help in identifying inefficiencies, tracking energy usage, and ensuring compliance with industry standards.

  • Power Usage Effectiveness (PUE): Monitoring and aiming to reduce PUE (a measure of how efficiently a data center uses energy) is essential for sustainability and operational cost savings.

8. Safety Features

  • Surge Protection: All electrical components must be fitted with surge protection to guard against spikes and electrical surges that could damage sensitive equipment.

  • Grounding and Bonding: Ensuring the entire electrical system is properly grounded is vital for both operational safety and equipment protection.

  • Fire Suppression: Electrical rooms and critical areas should have dedicated fire suppression systems, such as clean agent systems, to protect against potential fires caused by electrical faults.

9. Design for Scalability

  • Modular Design: Power infrastructure should be designed to scale easily as the data center grows. This includes modular UPS systems, easily expandable generator capacity, and flexible power distribution systems.

  • Future-Proofing: Plan for future increases in power demands due to technological upgrades (e.g., more energy-hungry processors) and the potential shift towards renewable energy sources.

10. Consideration for Renewable Energy

  • Integration of Renewable Energy: Depending on location, integrating solar, wind, or other renewable energy sources can help reduce operating costs and provide sustainability benefits. A hybrid solution where the data center can use both renewable energy and the grid power can be an effective strategy.

In conclusion, a successful power design for a data center involves ensuring reliability, scalability, and efficiency through redundancy, proper power management, integration of backup power systems, and safety measures. The goal is to ensure the data center runs smoothly without any power disruptions, meeting the demanding uptime requirements expected by modern businesses.