Thermal Analysis of Protium X1

19 Aug 2020

Thermal Analysis of Protium X1

There's a phrase in software development "eat your own dogfood". In fact, there's even an ugly verb "dogfooding". This means using your own software for real. If, say, you are developing a source-code management system (although why would you, Linus Torvalds already did that for you with git) then you use your own software to manage your source code. Cadence has a hardware business as well as EDA tools: the Palladium Z1 and Protium X1 platforms. As far as I know, we eat our own dogfood and only use our own tools to design these boxes (except for FPGA P&R since that comes from the vendor).

At the recent CadenceLIVE, Isaac Lee and Karthick Gopalakrishnan presented System-Level Thermal Analysis of Protium Platform Using Celsius Thermal Solver. They started with a guide to Celsius, but I'm going to skip that since I've covered it in:

Protium X1 is a modern, blade-based hardware and debug software for early firmware and software development, high-performance hardware regression, and full-system validation. It goes into an enterprise rack and is intended to be installed in the data center. I've written about Protium before, too, in:

Inside the chassis, there are "blades", which have an enclosure and contain three boards. Depending on how big a system you purchase, you get some number of these. Each blade is a 17.5in x 33.5in x 5.2in air-cooled chassis that dissipates 900W (not the actual number, but gives you an idea we're not talking about 25W nor 10KW). Each blade contains three boards, one with 2 FPGAs, one with 4 FPGAs, and an interface board. If it's not obvious, now you know why each blade can be used by up to six users (if the designs are small enough to fit in a single FPGA). The main source of heat are the FPGAs, of course, so they have big heatsinks.

Analysis with Celsius Thermal Solver

The goal of the analysis described in the presentation was to predict component operating temperatures of the major heat-dissipating components of the blade from a steady-state thermal analysis, and optimize the design if required. The diagram above shows how the blades go in Protium. There are three different kinds of boards: one with two FPGAs for prototyping, one with four FPGAs, and an interface board. There are heatsinks on the FPGAs and two fans in the center of the enclosure that draw air in from the front and force it out the rear.

The analysis was done by:

MCAD and ECAD model data used to build a computational fluid dynamics CFD model of the enclosure, PCBs, and components in the Celsius Thermal Solver (to model the airflow)
The heat source was the FPGAs and the heat sink was the (assumed) 32°C air temperature.
Allegro technology was used to open the PCB files and for each layer calculated the amount of metal and the conductivity.
For Connect-X5, we had a detailed thermal model from the supplier.
We modeled the FPGA heat sink assembly and its thermal resistance (see the image)
The two fans were initially modeled with fan curves and an RPM rate of 3000 and later with Celsius doing wind-tunnel modeling

The analysis by the Celsius Thermal Solver took two hours to simulate, based on 12 million elements. There are only a few dozen components but we had a very detailed model of the heatsinks. That two hours is both for meshing and solving. The meshing is entirely automatic, with no manual guidance required.

The final step was to compare the results from the analysis to the operational limits. The worst was the interface board that had three components with negative margin, the worst being -26ºC (negative is bad), and two more borderline at -1ºC.

Trade Study

A tradeoff study was done as to how to make the temperature within its specification.

Two things were considered:

adding air baffles near the 3" axial fan since the problem was occurring between the two fans
adding a third fan

Adding the air baffle got to -5ºC (negative is bad remember)

Adding the third fan meant all the components met their specs (see the pictures above)

So in summary:

Celsius Thermal Solver is a complete (Chip, package-PCB, and System) electro-thermal co-simulation tool with accuracy (FEA-CFD integration)
Protium X1 blade is an air-cooled chassis system that was analyzed in the Celsius Thermal Solver to predict the component operating temperatures
- Three components did not meet the rated operating temperatures with the worst temperature margin of -26°C
In the trade study, two additional cases (air baffles and third axial fan) were run to improve the thermal performance
- The worst temperature margin component’s temperature decreased by 21°C and 35°C for the air baffle case and third axial fan case, respectively

Q&A

A few additional points came out of the Q&A:

Radiation? Yes, the Celsius Thermal Solver can handle it (but didn't do it in this study)

Transient? This was steady-state analysis, but the Celsius Thermal Solver can do transient analysis

Bult-in models? We have some, like material libraries. We don't have fan models but support them.

Breakfast Bytes Blogs