Validating Data Center Performance Using Cadence Reality DC Design

Data centers have become critical for efficiently processing massive data sets and artificial intelligence (AI) workloads. According to a recent report by McKinsey and Company, hyper scalers, co-location companies, and enterprises in the US are expected to witness a 10% annual growth in data center power consumption until 2030. However, a change in the management of data centers is expected in the coming years that prioritizes controlled water and energy usage. This shift is driven by the early signs of climate change becoming increasingly evident across the globe. Ignoring these signs in favor of increased compute and storage power can pose a significant threat. Therefore, experts must recognize the need for sustainable data center practices capable of efficiently managing resources while mitigating the environmental impact.

Four Tier Ratings for Data Center Performance

Data center facilities are rated based on their level of performance and reliability. Understanding the technical details of each tier can provide insights into the facility's infrastructure, such as its power and cooling systems, redundancy, and security measures. The Uptime Institute, an independent organization, ranks data centers internationally based on service cost, uptime guarantee, and fault tolerance. Data centers are ranked from Tier 1 (lowest) to Tier 4 (highest) based on their infrastructure and redundancy capabilities.

Tier 1 data centers have only a single source for power and cooling and don't have any backup in case of a failure. They are expected to have an uptime of 99.671% in a year.

Tier 2 data centers have some redundant backup systems but still have only a single distribution path for power and cooling. They are expected to have an uptime of 99.741% per year.

Tier 3 data centers have multiple distribution paths and redundant backup systems for power and cooling, with an N+1 availability, where ‘N’ is the capacity to support the entire IT load, and ‘+1’ is an additional backup component. They are expected to have an uptime of 99.982% per year.

Finally, Tier 4 data centers are fault-tolerant and have a backup for every component, with a 2N+1 redundancy. This model ensures twice the operational capacity and additional backup in case of a failure while a secondary system operates. They are expected to have an uptime of 99.995% per year.

Validating Data Center Operational Design With CFD

Validation through virtual design and testing involves subjecting the design to various risk conditions and parameters. This process helps in identifying the highest risks associated with the design. Additionally, performance-based design is achieved through computational fluid dynamics (CFD) analysis, which allows for predictive analysis to ensure the operational design is close to real-world scenario.

Benefits of CFD in Data Center Design 

3D analysis: Airflow and heat transfer design validation

Virtual design: Testing the design before implementation

Performance-based analysis: Provides insight into the airflow (more cold air, starvation, bypass, recirculation, cold, and hot air mixing)

What-if scenarios: Predictive analysis avoids the risk of failure, i.e., optimized and better designs

Design effectiveness: Can be analyzed, design changes can be implemented

Cadence Reality DC Design is the industry's leading CFD tool, built for data center design and operations. Its 3D representation of the physical data center, combined with a cutting-edge CFD (computational fluid dynamics) solver, lets you safely simulate the impact of change on a data center's resilience, physical capacity, and cooling efficiency.

Larsen and Toubro (L&T) utilized the latest Cadence Reality DC Design 16.3 release for a recent project, which led to outstanding results. Below is a summary of L&T's simulation scenarios and design evaluation of a data center test case:

Figure 1. Data center geometry – plan view (left), 3D view (right).

For a data center, the CFD model comprises the room geometry, including the server cabinet layout and design data. Simulation scenarios are defined to evaluate the data center for steady-state operation, normal mode of operation, and failure mode. Transient analysis is carried out to observe the effects of utility power failure and chiller malfunction. The evaluation of cooling includes all components, such as power distribution unit (PDU) containments, cabinets, columns, beams, power cables, and data cables. Service level agreement (SLA) and temperature sensors are positioned at the cabinet to determine these sites' temperature values accurately.

Simulation Scenarios

Data server hall simulation scenarios for the following operational modes are considered:

• Normal Steady State Operation: at 100% loads with uniform distribution of load throughout the data hall and with functional cooling units.
• Failure Mode Operation: where the designated number of cooling equipment is offline in the worst-case scenario while operating on the same load as normal mode.
• Transient Mode Operation:

Scenario I: Chiller Power Failure and Cooling units and pumps with UPS at 100% IT load with uniform load distribution throughout the data hall

Scenario II: Chiller Power Failure and Cooling units and pump without UPS at 100% IT load with uniform load distribution throughout the data hall.

Design Evaluation

1. Normal Mode

Figure 2. Evaluating the operational design of a data center in normal mode, i.e., the peak inlet temperature of the cabinet- i) with control and with leakages, ii) with control and without leakages, iii) without control and with leakages, iv) without control and without leakages

Leakages in IT cabinets can cause the hot air from the hot aisle to recirculate back into the cabinet inlet through small gaps, leading to an increase in the inlet air temperature of the cabinets. Although all IT cabinets are within the recommended range of peak inlet air temperature (18-27°C), the presence of leakages can cause the American Society of Heating, Refrigerating and Air-Conditioning Engineers (ASHRAE) compliance to not be met for cases with control logic due to recirculation of hot air. However, the SLA sensor readings at 0.9m and 1.5m off the floor in front of each cabinet are less than 27°C, which meets the design requirements for cases without control logic.

2. Failure Mode

Figure 3. Evaluating the operational design of a data center in failure mode, i.e., the peak inlet temperature of the cabinet - i) with control and with leakages, ii) with control and without leakages, iii) without control and with leakages, iv) without control and without leakages

The simulation results of the operational design of the data center in failure mode showcased that the peak inlet temperature of the cabinet is almost the same for cases with and without controls. The leakages significantly impacted the performance during the failure mode. In an ideal case of no leakages, the simulation result showed that all IT cabinets except a few have peak inlet air temperature in the range of 18 -27⁰C, which is within the ASHRAE recommended range.

Also, SLA sensor readings at 0.9m and 1.5 m off the floor in front of each cabinet are less than 27⁰C, which meets the design requirement. In a practical case of leakages, simulation results showed that more than half of IT cabinets had peak inlet air temperature greater than 27⁰C, which does not comply with ASHRAE's recommended temperature range. Also, SLA sensor readings at 0.9m and 1.5m off the floor in front of each cabinet are more than 27⁰C, which does not meet the design requirement.

3. Transient Mode

Figure 4. Evaluating the operational design of a data center in transient mode, i.e., the peak inlet temperature of the cabinet - i) with control, with leakages, and with UPS ii) with control, with leakages, and without UPS iii) without control, with leakages, and with UPS, iv) without control, with leakages, and without UPS.

The presence of support infrastructure such as UPS significantly impacts the performance of data center cooling design. With UPS, the SLA breach time is zero for all cases, making it a critical component of the data center design. In comparison, breach time is significantly less for cases without control during transient mode when compared to cases with control. In the worst-case scenario, a breach time of 295 seconds with control is observed, considering all practical leakages and without UPS, above the threshold value of 120 seconds.

However, without control, the worst-case scenario breach time is 90 seconds, below the threshold value of 120 seconds, and hence acceptable as per the design requirements. Therefore, it is recommended to override the control during chiller failure and allow fans and cooling coil to run at rated capacity.

By implementing CFD technology, it was possible to forecast the performance of data center cooling systems precisely. The approach guarantees a comprehensive design study and accurate performance prediction for different cases in data center projects, instilling confidence in their reliable and efficient operation. In summary, using Cadence Reality DC Design for CFD modeling and simulation represents an innovative and highly effective technique for evaluating data center cooling design.

To learn more about managing data center performance, watch the CadenceLIVE India 2023 on-demand video 'Use of 6SigmaRoom for Design of a Large Data Centre' by L&T.