0
Research Papers

Experimentally Validated Computational Fluid Dynamics Model for a Data Center With Cold Aisle Containment

[+] Author and Article Information
Sami A. Alkharabsheh

Department of Mechanical Engineering,
Binghamton University,
State University of New York,
Binghamton, NY 13902
e-mail: salkhar1@binghamton.edu

Bahgat G. Sammakia

Department of Mechanical Engineering,
Binghamton University,
State University of New York,
Binghamton, NY 13902

Saurabh K. Shrivastava

PANDUIT Corporation,
Panduit Dr.,
Tinley Park, IL 60487

1Corresponding author.

Contributed by the Electronic and Photonic Packaging Division of ASME for publication in the JOURNAL OF ELECTRONIC PACKAGING. Manuscript received February 11, 2014; final manuscript received December 5, 2014; published online January 21, 2015. Assoc. Editor: Madhusudan Iyengar.

J. Electron. Packag 137(2), 021010 (Jun 01, 2015) (9 pages) Paper No: EP-14-1016; doi: 10.1115/1.4029344 History: Received February 11, 2014; Revised December 05, 2014; Online January 21, 2015

This paper presents the results of an experimentally validated computational fluid dynamics (CFD) model for a data center with fully implemented fan curves on both the servers and the computer room air conditioner (CRAC). Both open and contained cold aisle systems are considered in this study. This work is divided into sections for the baseline system (prior to installing containment) calibration and the fully contained cold aisle system calibration and leakage characterization. In the open system, the fan curve of the CRAC unit is extracted from the manufacturer data, while the fan curve of the load banks is obtained through experimental measurements. The experimental results are found to be in good agreement with the average model predictions. In the fully contained cold aisle system, a detailed containment CFD model is developed based on experimental measurements. The model is validated by comparing the flow rate through the perforated floor tiles and the rack inlet temperatures with the experimental measurements. The CFD results are found to be in good agreement with the experimental data with an average relative error between the measured and computed flow rate of approximately 6.7%. Temperature measurements are used to calibrate the sources of leakage in the containment and rack mounting rails. The temperature measurements and the CFD results agree well with an average difference of less than 1 °C. This study provides important modeling guidelines for data centers. In order to predict the performance of contained cold aisle systems flow distribution, it is crucial that physics based models of fan curves, server internal resistances, detailed rack models, and other design details are all accurate and experimentally verified.

FIGURES IN THIS ARTICLE
<>
Copyright © 2015 by ASME
Your Session has timed out. Please sign back in to continue.

References

U.S. Environmental Protection Agency (EPA), 2007, “Report to Congress on Server and Data Center Energy Efficiency: Public Law 109-43,” U.S. Environmental Protection Agency, Washington, D.C.
Choi, J., Kim, Y., Sivasubramaniam, A., Srebric, J., Wang, Q., and Lee, J., 2008, “A CFD-Based Tool for Studying Temperature in Rack-Mounted Servers Computers,” IEEE Trans.57(8), pp. 1129–1142. [CrossRef]
Iyengar, M., Schmidt, R., Hamann, H., and VanGilder, J., 2007, “Comparison Between Numerical and Experimental Temperature Distributions in a Small Data Center Test Cell,” ASME Paper No. IPACK2007-33508. [CrossRef]
Shrivastava, S., Iyengar, M., Sammakia, B., Schmidt, R., and VanGilder, J., 2006, “Experimental-Numerical Comparison for a High-Density Data Center: Hot Spot Fluxes in Excess of 500 W/ft2,” Intersociety Conference on Thermal and Thermomechanical Phenomena in Electronics Systems (ITHERM), San Diego, CA, May 30–June 2, pp. 402–411.
Kang, S., Schmidt, R., Kelkar, K. M., Radmehr, A., and Patankar, S. V., 2001, “A Methodology for the Design of Perforated Tiles in Raised Floor Data Centers Using Computational Flow Analysis,” IEEE Trans. Compon., Packag. Manuf. Technol., 24(2), pp. 177–183. [CrossRef]
Shrivastava, S., Sammakia, B., Schmidt, R., and Iyengar, M., 2005, “Comparative Analysis of Different Data Center Airflow Management Configurations,” ASME Paper No. IPACK2005-73234. [CrossRef]
Abdelmaksoud, W., 2012, “Experimental and Numerical Investigations of the Thermal Environment in Air-Cooled Data Centers,” Ph.D. thesis, Syracuse University, New York.
Abdelmaksoud, W. A., Dang, T. Q., Khalifa, H. E., and Schmidt, R. R., 2013, “Improved Computational Fluid Dynamics Model for Open-Aisle Air-Cooled Data Center Simulations,” ASME J. Electron. Packag., 135(3), p. 030901. [CrossRef]
Bhopte, S., Sammakia, B., Schmidt, R., Iyengar, M., and Agonafer, D., 2006, “Effect of Under Floor Blockages on Data Center Performance,” Intersociety Conference on Thermal and Thermomechanical Phenomena in Electronics Systems (ITHERM), San Diego, CA, May 30–June 2, pp. 426–433.
Patankar, S. V., 2010, “Airflow and Cooling in a Data Center,” ASME J. Heat Transfer, 132(7), p. 073001. [CrossRef]
Samadiani, E., Rambo, J., and Joshi, Y., 2012, “Numerical Modeling of Perforated Tile Flow Distribution in a Raised-Floor Data Center,” ASME J. Electron. Packag., 132(2), p. 021002. [CrossRef]
Demetriou, D. W., and Khalifa, H. E., 2012, “Optimization of Enclosed Aisle Data Centers Using Bypass Recirculation,” ASME J. Electron. Packag., 134(2), p. 020904. [CrossRef]
Zhou, R., Wang, Z., Bash, C. E., and McReynolds, A., 2011, “Modeling and Control for Cooling Management of Data Centers With Hot Aisle Containment,” ASME Paper No. IMECE2011-62506. [CrossRef]
Shrivastava, S. K., Calder, A. R., and Ibrahim, M., 2012, “Quantitative Comparison of Air Containment Systems,” Intersociety Conference on Thermal and Thermomechanical Phenomena in Electronics Systems (ITHERM), San Diego, CA, May 30–June 1, pp. 68–77.
Muralidharan, B., Ibrahim, M., Shrivistava, S., Alkharabsheh, S., and Sammakia, B., 2013, “Impact of Cold Aisle Containment on Thermal Performance of Data Center,” ASME Paper No. IPACK2013-73201. [CrossRef]
Arghode, V. K., Sundaralingam, V., Joshi, Y., and Phelps, W., 2013, “Thermal Characteristics of Open and Contained Data Center Cold Aisle,” ASME J. Heat Transfer, 135(6), p. 061901. [CrossRef]
Arghode, V. K., and Joshi, Y., 2014, “Room Level Modeling of Air Flow in a Contained Data Center Aisle,” ASME J. Electron. Packag., 136(1), p. 011011. [CrossRef]
Arghode, V. K., Sundaralingam, V., Joshi, Y., and Phelps, W., 2015, “Experimental Characterization of Various Cold Aisle Containment Configurations for Data Centers,” ASME J. Electron. Packag., 137(1), p. 011007. [CrossRef]
Alkharabsheh, S. A., Sammakia, B. G., Shrivastava, S. K., and Schmidt, R., 2013, “Utilizing Practical Fan Curves in CFD Modeling of a Data Center,” Semiconductor Thermal Measurement and Management Symposium (SEMI-THERM), San Jose, CA, Mar. 17–21, pp. 211–215. [CrossRef]
Alkharabsheh, S. A., Sammakia, B. G., Shrivastava, S. K., Ellsworth, M., David, M., and Schmidt, R., 2013 “A Numerical Steady State and Dynamic Study in a Data Center Using Calibrated Fan Curves for CRACs and Servers,” ASME Paper No. IPACK2013-73217. [CrossRef]
Alkharabsheh, S. A., Sammakia, B. G., Shrivastava, S. K., and Schmidt, R., 2014, “Dynamic Models of Servers Racks and CRAHs in a Room Level CFD Model of a Data Center,” Inter Society Conference on Thermal Phenomena (ITHERM), Orlando, FL, May 27–30.
Mentor, 2012, FloTHERM V9.3 User Reference Guide, Mentor Graphics Corp., Wilsonville, OR.
Alkharabsheh, S. A., Muralidharan, B., Ibrahim, M., Shrivastava, S., and Sammakia, B., 2013, “Open and Contained Cold Aisle Experimentally Validated CFD Model Implementing CRAC and Server Fan Curves for a Data Center Testing Laboratory,” ASME Paper No. IPACK2013-73214. [CrossRef]
Radmehr, A., Schmidt, R., Karki, K., and Patankar, S., 2005, “Distributed Leakage Flow in Raised-Floor Data Centers,” ASME Paper No. IPACK2005-73273. [CrossRef]
Zhang, X., VanGilder, J., Iyengar, M., and Schmidt, R., 2008, “Effect of Rack Modeling Detail on the Numerical Results of a Data Center Test Cell,” Inter Society Conference on Thermal Phenomena (ITHERM), Orlando, FL, May 28–31, pp. 1183–1190.
Patel, C., Bash, C., and Belady, C., 2001, “Computational Fluid Dynamics Modeling of High Compute Density Data Centers to Assure System Inlet Air Specifications,” ASME of the Pacific Rim Technical Conference and Exposition of Packaging and Integration of Electronic and Photonic Systems (InterPACK), Kauai, Hawaii, July 8–13, ASME Paper No. IPACK2001-15622.
Patankar, S. V., 1980, Numerical Heat Transfer and Fluid Flow, Taylor and Francis Publishing Corporation, New York.
Mentor, 2012, FloTHERM V9.3, Mentor Graphics Corp., Wilsonville, OR.
Herrlin, M. K., 2005, “Rack Cooling Effectiveness in Data Centers and Telecom Central Offices: The Rack Cooling Index (RCI),” ASHRAE Trans., 111(2), pp. 725–731.
Herrlin, M. K., and Belady, C., 2006, “Gravity-Aassisted Air Mixing in Data Centers and How It Affects the Rack Cooling Effectiveness,” Inter Society Conference on Thermal Phenomena (ITHERM), San Diego, CA, May 30–June 2, p. 438.
Rambo, J., and Joshi, Y., 2007, “Modeling of Data Center Airflow and Heat Transfer: State of the Art and Future Trends,” Distributed Parallel Databases, 21(2–3), pp. 193–225. [CrossRef]
Cruz, E., Joshi, Y., Iyengar, M., and Schmidt, R., 2009, “Comparison of Numerical Modeling to Experimental Data in a Small Data Center Test Cell,” ASME Paper No. InterPACK2009-89306. [CrossRef]
Demetriou, D., and Khalifa, H. E., 2011, “Evaluation of a Data Center Recirculation Non-Uniformity Metric Using Computational Fluid Dynamics,” ASME Paper No. IPACK2011-52005. [CrossRef]
Alkharsbsheh, S. A., Sammakia, B. G., and Murray, B. T., 2014, “Experimental Characterization of Pressure Drop in a Server Rack,” Inter Society Conference on Thermal Phenomena (ITHERM), Orlando, FL, May 31–June 4, pp. 547–556.
Fried, E., and Idelchik, I. E., 1989, Flow Resistance: A Design Guide for Engineers, Hemisphere, New York.

Figures

Grahic Jump Location
Fig. 1

Modular data center: (a) 3D view and (b) top view

Grahic Jump Location
Fig. 2

Detailed experimental measurements per tile at 100% CRAC fan speed. This data are used for airflow calibration in the numerical model.

Grahic Jump Location
Fig. 3

Calibrated CRAC operating point and fan characteristics curve. The internal resistance simulating the internal components of a CRAC is modeled based on the manufacturer data.

Grahic Jump Location
Fig. 4

(a) Server simulator (load bank) and (b) measured fan curves used in the numerical model. The experimental measurements take into account the internal resistance of the load bank without the need to simulate the internal resistance in the numerical model.

Grahic Jump Location
Fig. 5

Schematic of (a) detailed CAC model and (b) detailed rack model

Grahic Jump Location
Fig. 6

Total CRAC flow rate at different load banks internal resistance. It is noteworthy that for open systems increasing the internal resistance has little impact on the flow rate, but for contained systems, a significant drop in flow rate is observed as the resistance is increased.

Grahic Jump Location
Fig. 7

Experimental measurements per tile at 60% CRAC fans speed for CAC system. This data are used for CAC system validation.

Grahic Jump Location
Fig. 8

Comparison between the measured inlet temperatures and the detailed model. The average temperature difference between the simulations and the measurements is 0.99 °C.

Grahic Jump Location
Fig. 9

The inlet temperatures at different elevations of the rack using simulations. The rise in inlet temperature as a function of elevation in the cold aisle is due to cold air escaping the cold aisle at low elevations, and hot air penetrating at the upper portion of the cold aisle. Due to fixed temperature at the lower part of the racks, a broken y-axis is used between 0.8 and 1.8 m.

Grahic Jump Location
Fig. 10

Comparison between the measured inlet temperatures and the simplified model. The average temperature difference between the simulations and the measurements is 1.32 °C.

Grahic Jump Location
Fig. 11

The effect of varying the containment leakage on the total static pressure and total flow rate. After certain leakage area ratio, the CAC system behaves in a similar manner as an uncontained cold aisle system.

Grahic Jump Location
Fig. 12

Effect of the leakage variation on the inlet temperature. At small leakage area ratios, the increase in temperature is compensated by the increase in the cooling flow rate. At high leakage area ratios, the cooling flow rate becomes steady and the hot air leakage increases.

Grahic Jump Location
Fig. 13

The effect of leakage on the flow rate of rack 7/9 RU. The flow rate increases due to the decrease in the static pressure, however, this does not indicate better cooling because of the mixing with the leaking hot air at high leakage area ratios.

Grahic Jump Location
Fig. 14

The effect of leakage area ratio on the airflow rate through leakage. The airflow that leaks from the leakage locations increases as the leakage area ratio increases. While the leakage airflow rate is constant above a certain ratio, the leakage airflow rate through doors and ceiling varies which explains the change in temperature.

Tables

Errata

Discussions

Some tools below are only available to our subscribers or users with an online account.

Related Content

Customize your page view by dragging and repositioning the boxes below.

Related Journal Articles
Related eBook Content
Topic Collections

Sorry! You do not have access to this content. For assistance or to subscribe, please contact us:

  • TELEPHONE: 1-800-843-2763 (Toll-free in the USA)
  • EMAIL: asmedigitalcollection@asme.org
Sign In