0
Review Article

A Brief Overview of Recent Developments in Thermal Management in Data Centers

[+] Author and Article Information
Sami Alkharabsheh

Mechanical Engineering Department,
Binghamton University,
State University of New York,
Binghamton, NY 13902
e-mail: salkhar1@binghamton.edu

John Fernandes, Betsegaw Gebrehiwot, Dereje Agonafer

Mechanical and Aerospace
Engineering Department,
University of Texas at Arlington,
Arlington, TX 76019

Kanad Ghose

Computer Science Department,
Binghamton University,
State University of New York,
Binghamton, NY 13902

Alfonso Ortega

Department of Mechanical Engineering,
Villanova University,
Villanova, PA 19085

Yogendra Joshi

The George W. Woodruff
School of Mechanical Engineering,
Georgia Institute of Technology,
Atlanta, GA 30332

Bahgat Sammakia

Mechanical Engineering Department,
Binghamton University,
State University of New York,
Binghamton, NY 13902

Contributed by the Electronic and Photonic Packaging Division of ASME for publication in the JOURNAL OF ELECTRONIC PACKAGING. Manuscript received December 23, 2014; final manuscript received August 16, 2015; published online September 10, 2015. Editor: Y. C. Lee.

J. Electron. Packag 137(4), 040801 (Sep 10, 2015) (19 pages) Paper No: EP-14-1117; doi: 10.1115/1.4031326 History: Received December 23, 2014; Revised August 16, 2015

Data centers are mission critical facilities that typically contain thousands of data processing equipment, such as servers, switches, and routers. In recent years, there has been a boom in data center usage, leading their energy consumption to grow by about 10% a year continuously. The heat generated in these data centers must be removed so as to prevent high temperatures from degrading their reliability, which would cost additional energy. Therefore, precise and reliable thermal management of the data center environment is critical. This paper focuses on recent advancements in data center modeling and energy optimization. A number of currently available and developmental thermal management technology in data centers are broadly reviewed. Computational fluid dynamics (CFD) for raised-floor data centers, experimental measurements, containment systems, economizer cooling, hybrid cooling, and device level cooling are all thoroughly reviewed. The paper concludes with a summary and presents areas of potential future research, which are based on the holistic integration of workload prediction and allocation, and thermal management using smart control systems.

Copyright © 2015 by ASME
Your Session has timed out. Please sign back in to continue.

References

U.S. Environmental Protection Agency (EPA), 2007, “ Report to Congress on Server and Data Center Energy Efficiency, Public Law 109-43,” U.S. Environmental Protection Agency, Washington, DC.
Columbus, L., 2012, “ Predicting Enterprise Cloud Computing Growth,” Forbes, accessed January 31, 2014, http://www.forbes.com/sites/louiscolumbus/2013/09/04/predicting-enterprise-cloud-computing-growth/
Koomey, J. , 2011, Growth in Data Center Electricity Use 2005 to 2010, Analytics Press, Oakland, CA.
Venkatraman, A. , 2013, “ Global Census Shows Datacenter Power Demand Grew 63% in 2012,” DatacenterDynamics (DCD) Intelligence, London.
Abramovitz, B., 2013, “ Industry Research Monitor Data Centers,” General Electric Capital, Norwalk, CT.
McNevin, A., 2014, “ 15% Growth Forecast for North America Colocation Market 2014,” DatacenterDynamics, London.
Stansberry, M. , and Kudritzki, J. , 2013, “ Uptime Institute 2012 Data Center Industry Survey,” Uptime Institute, New York.
Wikipedia, 2014, “Mission Critical,” accessed January 31, 2014, http://en.wikipedia.org/wiki/Mission_critical
“Mission Critical Facilities Management Principals of Design, Operations, and Maintenance,” 2012, Last accessed January 31, 2014, http://www.construction.org/clientuploads/resource_center/facilities_management/MissionCriticalFacilities.pdf
Ponemon Institute, 2010, “ National Survey on Data Center Outages,” Ponemon Institute, Traverse City, MI.
Zuo, Z. J. , Hoover, L. R. , and Phillips, A. L. , 2002, An Integrated Thermal Architecture for Thermal Management of High Power Electronics, Millpress, Rotterdam, The Netherlands.
Salim, M. , and Tozer, R. , 2010, “ Data Centers' Energy Auditing and Benchmarking: Progress Update,” ASHRAE Trans., 116(1), pp. 109–117.
Patankar, S. , and Karki, K. , 2004, “ Distribution of Cooling Airflow in a Raised-Floor Data Center,” ASHRAE Trans., 110(2), pp. 629–634.
Sharma, R. K. , Bash, C. E. , and Patel, C. D. , 2002, “ Dimensionless Parameters for Evaluation of Thermal Design and Performance of Large-Scale Data Centers,” AIAA Paper No. 2002-3091.
Muralidharan, B. , Shrivastava, S. , Ibrahim, M. , Alkharabsheh, S. A. , and Sammakia, B. G. , 2013, “ Impact of Cold Aisle Containment on Thermal Performance of Data Center,” ASME Paper No. IPACK2013-73201.
Joshi, Y. , and Kumar, P. , 2012, Energy Efficient Thermal Management of Data Centers, Springer, New York.
Rambo, J. , and Joshi, Y. , 2007, “ Modeling of Data Center Airflow and Heat Transfer: Stat of the Art and Future Trends ,” Distrib. Parallel Databases, 21(2–3), pp. 193–225. [CrossRef]
Rambo, J. , and Joshi, Y. , 2006, “ Reduced-Order Modeling of Multiscale Turbulent Convection: Application to Data Center Thermal Management,” Ph.D. disseration, Georgia Institute of Technology, Atlanta, GA.
Rambo, J. , and Joshi, Y. , 2005, “ Reduced Order Modeling of Steady Turbulent Flows,” ASME Paper No. HT2005-72143.
Somani, A. , and Joshi, Y. , 2009, “ Data Center Cooling Optimization: Ambient Intelligence Based Load Management (AILM),” ASME Paper No. HT2009-88228.
Samadiani, E. , 2009, “ Energy Efficient Thermal Management of Data Centers Via Open Multi-Scale Design,” Ph.D. disseration, Georgia Institute of Technology, Atlanta, GA.
Ghosh, R. , and Joshi, Y. , 2013, “ Error Estimation in POD-Based Dynamic Reduced-Order Thermal Modeling of Data Centers,” Int. J. Heat Mass Transfer, 57(2), pp. 698–707. [CrossRef]
Belady, C. , Kelkar, K. , and Patankar, S. , 1999, “ Improving Productivity of Electronic Packaging With Flow Network Modeling (FNM),” Electron. Cool., 5(1), pp. 36–40.
Radmehr, A. , Kelkar, K. , Kelly, P. , Patankar, S. , and Kang, S. , 1999, “ Analysis of the Effect of Bypass on Performance of Heat Sinks Using Flow Network Modeling (FNM),” 15th Annual IEEE Semiconductor Thermal Measurement and Management Systems (SEMI-THERM), San Diego, CA, Mar. 9–11, pp. 42–47.
Steinbrecher, R. , Radmehr, A. , Kelkar, K. , and Patankar, S. , “ Use of Flow Network Modeling (FNM) for the Design of Air-Cooled Servers,” Innovative Research Inc., Minneapolis, MN, http://inres.com/assets/files/macroflow/MF08-Air-Cooled-Server.pdf
Innovative Research, 2003, MacroFlow, Innovative Research, Plymouth, MN.
Kelkar, K. , and Patankar, S. , “ Analysis and Design of Liquid-Cooling Systems Using Flow Network Modeling (FNM),” ASME Paper No. IPACK2003-35233.
Cross, H. , 1936, “ Analysis of Flow in Networks of Conduits or Conductors,” University of Illinois Bulletin, University of Illinois at Urbana-Champaign, Urbana, IL, Report No. 286.
Fernandes, J. , Ghalambor, S. , Docca, A. , Aldham, C. , Agonafer, D. , Chenelly, E. , Chan, B. , and Ellsworth, M. , 2013, “ Combining Computational Fluid Dynamics (CFD) and Flow Network Modeling (FNM) for Design of a Multi-Chip Module (MCM) Cold Plate,” ASME Paper No. IPACK2013-73294.
Radmehr, A. , and Patankar, S. , 2004, “ A Flow Network Analysis of a Liquid Cooling System That Incorporates Microchannel Heat Sinks,” IEEE Intersociety Conference on Thermal and Thermomechanical Phenomena in Electronic Systems (ITHERM '04), Las Vegas, NV, June 1–4, pp. 714–721.
Ellsworth, M. , 2014, “ Flow Network Analysis of the IBM Power 775 Supercomputer Water Cooling System,” IEEE Intersociety Conference on Thermal and Thermomechanical Phenomena in Electronic Systems (ITHERM), Orlando, FL, May 27–30, pp. 715–722.
Patel, C. D. , Bash, C. E. , Belady, C. , Stahl, L. , and Sullivan, D. , 2001, “ Computational Fluid Dynamics Modeling of High Compute Density Data Centers to Assure System Inlet Air Specifications,” Pacific Rim Technical Conference and Exposition of Packaging and Integration of Electronic and Photonic Systems (IPACK), Kauai, HI, July 8–13, ASME Paper No. IPACK2001-15622.
Schmidt, R. R. , Karki, K. C. , Kelkar, K. M. , Radmehr, A. , and Patnkar, S. V. , 2001, “ Measurements and Predictions of the Flow Distribution Through Perforated Tiles in Raised Floor Data Centers,” Pacific Rim Technical Conference and Exposition of Packaging and Integration of Electronic and Photonic Systems (IPACK), Kauai, HI, July 8–13, ASME Paper No. IPACK2001-15728.
Kang, S. , Schmidt, R. , Kelkar, K. M. , Radmehr, A. , and Patankar, S. V. , 2001, “ A Methodology for the Design of Perforated Tiles in Raised Floor Data Centers Using Computational Flow Analysis,” IEEE Trans. Compon. Packag. Technol., 24(2), pp. 177–183. [CrossRef]
Karki, K. , Patankar, S. , and Radmehr, A. , 2003, “ Techniques for Controlling Airflow Distribution in Raised-Floor Data Centers,” ASME Paper No. IPACK2003-35282.
VanGilder, J. , and Schmidt, R. , 2005, “ Airflow Uniformity Through Perforated Tiles in a Raised-Floor Data Center,” ASME Paper No. IPACK2005-73375.
Radmehr, A. , Schmidt, R. , Karki, K. , and Patankar, S. , 2005, “ Distributed Leakage Flow in Raised-Floor Data Centers,” ASME Paper No. IPACK2005-73273.
Abdelmaksoud, W. A. , Khalifa, H. E. , Dang, T. Q. , Elhadidi, B. , Schmidt, R. R. , and Iyengar, M. , 2010, “ Experimental and Computational Study of Perforated Floor Tile in Data Centers,” 12th IEEE Intersociety Conference on Thermal and Thermomechanical Phenomena in Electronics Systems (ITHERM), Las Vegas, NV, June 2–5.
Arghode, V. K. , Kumar, P. , Joshi, Y. , Weiss, T. , and Meyer, G. , 2013, “ Rack Level Modeling of Air Flow Through Perforated Tile in a Data Center,” ASME J. Electron. Packag., 135(3), p. 030902. [CrossRef]
Arghode, V. , and Joshi, Y. , 2013, “ Modeling Strategies for Air Flow Through Perforated Tiles in a Data Center,” IEEE Trans. Compon. Packag. Technol., 3(5), pp. 800–810. [CrossRef]
Abdelmaksoud, W. , Dang, T. , Khalifa, H. E. , Schmidt, R. , and Iyengar, M. , 2012, “ Perforated Tile Models for Improving Data Center CFD Simulation,” Intersociety Conference on Thermal and Thermomechanical Phenomena in Electronics Systems (ITHERM), San Diego, CA, May 30–June 1, pp. 60–67.
Shrivastava, S. K. , Iyengar, M. , Sammakia, B. G. , Schmidt, R. , and Vangilder, J. W. , 2006, “ Experimental-Numerical Comparison for a High-Density Data Center: Hot Spot Fluxes in Excess of 500 W/ft2 ,” IEEE Intersociety Conference on Thermal and Thermomechanical Phenomena in Electronics Systems (ITHERM '06), San Diego, CA, May 30–June 2, pp. 402–411.
Tan, S. P. , Toh, K. C. , and Wong, Y. W. , 2007, “ Server-Rack Air Flow and Heat Transfer Interactions in Data Centers,” ASME Paper No. IPACK2007-33672.
Zhang, X. S. , VanGilder, J. W. , Iyengar, M. , and Schmidt, R. R. , 2008, “ Effect of Rack Modeling Detail on the Numerical Results of a Data Center Test Cell,” IEEE Intersociety Conference on Thermal and Thermomechanical Phenomena in Electronics Systems (ITHERM 2008), Lake Buena Vista, FL, May 28–31, pp. 1183–1190.
Zhai, J. Z. , Hermansen, K. A. , and Al-Saadi, S. , 2012, “ The Development of Simplified Rack Boundary Conditions for Numerical Data Center Models,” ASHRAE Trans., 118(2), pp. 436–449.
North, T. , 2011, “ Understanding How Cabinet Door Perforation Impacts Airflow,” BICSI News, Sept./Oct., pp. 36–42.
Schmidt, R. , Chu, R. , Ellsworth, M. , Iyengar, M. , Porter, D. , Kamath, V. , and Lehman, B. , 2005, “ Maintaining Datacom Rack Inlet Air Temperatures With Water Cooled Heat Exchanger,” ASME Paper No. IPACK2005-73468.
Coxe, K. , 2009, “ Rack Infrastructure Effects on Thermal Performance of a Server,” Dell Enterprise Thermal Engineering, White Paper.
Rubenstein, B. , 2008, “ Cable Management Arm Airflow Impedance Study,” IEEE Intersociety Conference on Thermal and Thermomechanical Phenomena in Electronics Systems (ITHERM 2008), Orlando, FL, May 28–31, pp. 577–582.
Alkharabsheh, S. A. , Sammakia, B. G. , and Murray, B. T. , 2014, “ Experimental Characterization of Pressure Drop in a Server Rack,” IEEE Inter Society Conference on Thermal Phenomena (ITHERM), Orlando, FL, May 27–30, pp. 547–556.
Radmehr, A. , Karki, K. C. , and Patankar, S. V. , 2007, “ Analysis of Airflow Distribution Across a Front-to-Rear Server Rack,” ASME Paper No. IPACK2007-33574.
Ghosh, R. , Sundaralingam, V. , and Joshi, Y. , 2012, “ Effect of Rack Server Population on Temperatures in Data Centers,” Intersociety Conference on Thermal and Thermomechanical Phenomena in Electronics Systems (ITHERM), San Diego, CA, May 30–June 1, pp. 30–37.
Almoli, A. , Thompson, A. , Kapur, N. , Summers, J. , Thompson, H. , and Hannah, G. , 2012, “ Computational Fluid Dynamic Investigation of Liquid Rack Cooling in Data Centres,” Appl. Energy, 89(1), pp. 150–155. [CrossRef]
Samadiani, E. , Rambo, J. , and Joshi, Y. , 2010, “ Numerical Modeling of Perforated Tile Flow Distribution in a Raised-Floor Data Center,” ASME J. Electron. Packag., 132(2), p. 021002. [CrossRef]
Patankar, S. V. , 2010, “ Airflow and Cooling in a Data Center,” ASME J. Heat Transfer, 132(7), p. 073001. [CrossRef]
Ibrahim, M. , Bhopte, S. , Sammakia, S. , Murray, B. , Iyengar, M. , and Schmidt, R. , 2010 “ Effect of Thermal Characteristics of Electronic Enclosures on Dynamic Data Center Performance,” ASME Paper No. IMECE2010-40914.
Alkharabsheh, S. , Sammakia, B. , Shrivastava, S. , and Schmidt, R. , 2014, “ Dynamic Models for Server Rack and CRAH in a Room Level CFD Model of a Data Center,” IEEE Intersociety Conference on Thermal and Thermomechanical Phenomena in Electronics Systems (ITHERM), Orlando, FL, May 27–30, pp. 1338–1345.
Schmidt, R. , 2001, “ Effect of Data Center Characteristics on Data Processing Equipment Inlet Temperatures,” ASME Paper No. IPACK2001-15870.
Schmidt, R. , and Cruz, E. , 2002, “ Raised Floor Computer Data Center: Effect on Rack Inlet Temperatures of Chilled Air Exiting Both the Hot and Cold Aisles,” IEEE Inter Society Conference on Thermal and Thermomechanical Phenomena in Electronic Systems (ITHERM 2002), San Diego, CA, June 1, pp. 580–594.
Schmidt, R. , and Cruz, E. , 2002, “ Raised Floor Computer Data Center: Effect on Rack Inlet Temperatures When High Powered Racks are Situated Amongst Lower Powered Racks,” ASME Paper No. IMECE2002-39652.
Schmidt, R. , and Cruz, E. , 2003, “ Raised Floor Computer Data Center: Effect on Rack Inlet Temperatures When Adjacent Racks are Removed,” ASME Paper No. IPACK2003-35240.
Schmidt, R. , and Cruz, E. , 2003, “ Raised Floor Computer Data Center: Effect of Rack Inlet Temperatures When Rack Flowrates are Reduced,” ASME Paper No. IPACK2003-35241.
Patel, C. D. , Sharma, R. , Bash, C. E. , and Beitelmal, A. , 2002, “ Thermal Considerations in Cooling Large Scale High Compute Density Data Centers,” IEEE Intersociety Conference on Thermal and Thermomechanical Phenomena in Electronic Systems (ITHERM 2002), San Diego, CA, San Diego, CA, June 1, pp. 767–776.
Schmidt, R. , and Cruz, E. , 2004, “ Cluster of High-Powered Racks Within a Raised-Floor Computer Data Center: Effect of Perforated Tile Flow Distribution on Rack Inlet Air Temperatures,” ASME J. Electron. Packag., 126(4), pp. 510–519. [CrossRef]
Schmidt, R. , Cruz, E. , and Iyengar, M. , 2005, “ Challenges of Data Center Thermal Management,” IBM J. Res. Dev., 49(4.5), pp. 709–723. [CrossRef]
Bhopte, S. , Agonafer, D. , Schmidt, R. , and Sammakia, B. , 2006, “ Optimization of Data Center Room Layout to Minimize Rack Inlet Air Temperature,” ASME J. Electron. Packag. 128(4), pp. 380–387. [CrossRef]
Bhopte, S. , Sammakia, B. , Schmidt, R. , Iyenger, M. , and Agonafer, D. , 2006, “ Effect of Under Floor Blockages on Data Center Performance,” Intersociety Conference on Thermal and Thermomechanical Phenomena in Electronics Systems (ITHERM '06), San Diego, CA, May 30–June 2, pp. 426–433.
Hannaford, P. , 2006, “ Ten Cooling Solutions to Support High-Density Server Deployment,” White Paper, American Power Conversion, West Kingston, RI, Report No. WP-42 v5.
Greenberg, S. , Mills, E. , Tschudi, B. , Rumsey, P. , and Myatt, B. , 2006, “ Best Practices for Data Centers: Lessons Learned From Benchmarking 22 Data Centers,” ACEEE Summer Study on Energy Efficiency in Buildings, Pacific Grove, CA, Aug. 13–18, pp. 76–87.
Schmidt, R. , and Iyengar, M. , 2007, “ Best Practices for Data Center Thermal and Energy Management: Review of Literature,” ASHRAE Trans., 113(1), pp. 206–218.
Nagarathinam, S. , Fakhim, B. , Behnia, M. , and Armfield, S. , 2013, “ A Comparison of Parametric and Multivariable Optimization Techniques in a Raised-Floor Data Center,” ASME J. Electron. Packag., 135(3), p. 030905. [CrossRef]
Sorell, V. , Escalante, S. , and Yang, J. , 2005, “ Comparison of Overhead and Underfloor Air Delivery Systems in a Data Center Environment Using CFD Modeling,” ASHRAE Trans., 111(2), pp. 756–764.
Iyengar, M. , Schmidt, R. , Sharma, A. , McVicker, G. , Shrivastava, S. , Sri-Jayantha, S. , Amemiya, Y. , Dang, H. , Chainer, T. , and Sammakia, B. , 2005, “ Thermal Characterization of Non-Raised Floor Air Cooled Data Centers Using Numerical Modeling,” ASME Paper No. IPACK2005-73387.
Demetriou, D. W. , and Khalifa, H. E. , 2011, “ Evaluation of a Data Center Recirculation Non-Uniformity Metric Using Computational Fluid Dynamics,” ASME Paper No. IPACK2011-52005.
Alkharabsheh, S. , Sammakia, B. , Shrivastava, S. , and Schmidt, R. , 2013, “ Utilizing Practical Fan Curves in CFD Modeling of a Data Center,” IEEE Semiconductor Thermal Measurement and Management Symposium (SEMI-THERM), San Jose, CA, Mar. 17–21, pp. 211–215.
Alkharabsheh, S. , Sammakia, B. , Shrivastava, S. , Ellsworth, M. , David, M. , and Schmidt, R. , 2013, “ Numerical Steady State and Dynamic Study Using Calibrated Fan Curves for CRAC Units and Servers,” ASME Paper No. IPACK2013-73217.
Alkharabsheh, S. , Sammakia, B. , Shrivastava, S. , and Schmidt, R. , 2013, “ A Numerical Study for Contained Cold Aisle Data Center Using CRAC and Server Calibrated Fan Curves,” ASME Paper No. IMECE2013-65145.
Alkharabsheh, S. , Sammakia, B. , and Shrivastava, S. , 2015, “ Experimentally Validated CFD Model for a Data Center With Cold Aisle Containment,” ASME J. Electron. Packag., 137(2), p. 021010. [CrossRef]
Bash, C. E. , Patel, C. D. , and Sharma, R. K. , 2006, “ Dynamic Thermal Management of Air Cooled Data Centers,” IEEE Intersociety Conference on Thermal and Thermomechanical Phenomena in Electronics Systems (ITHERM '06), San Diego, CA, May 30–June 2, pp. 445–452.
Kummert, M. , Dempster, W. M. , and McLean, K. , 2009, “ Transient Thermal Analysis of a Data Centre Cooling System Under Fault Conditions,” 11th International Building Performance Simulation Association Conference and Exhibition, Glasgow, UK, July 27–30.
Gondipalli, S. , Ibrahim, M. , Bhopte, S. , Sammakia, B. , Murray, B. , Ghose, K. , Iyengar, M. , and Schmidt, R. , 2010, “ Numerical Modeling of Data Center With Transient Boundary Conditions,” 12th IEEE Thermal and Thermomechanical Phenomena in Electronic Systems (ITHERM), Las Vegas, NV, June 2–5.
Beitelmal, A. H. , and Patel, C. D. , 2007, “ Thermo-Fluids Provisioning of a High Performance High Density Data Center,” Distrib. Parallel Databases, 21(2–3), pp. 227–238. [CrossRef]
Sharma, R. K. , Bash, C. E. , Patel, C. D. , Friedrich, R. J. , and Chase, J. S. , 2005, “ Balance of Power: Dynamic Thermal Management for Internet Data Centers,” Internet Comput., 9(1), pp. 42–49. [CrossRef]
Patel, C. , Bash, C. , Sharma, R. , Beitelmal, M. , and Friedrich, R. , 2003, “ Smart Cooling of Data Centers,” ASME Paper No. IPACK2003-35059.
Khankari, K. , 2010, “ Thermal Mass Availability for Cooling Data Centers During Power Shutdown,” ASHRAE Trans., 116(Pt. 2), pp. 205–217.
Khankari, K. , 2011, “ Rate of Heating Analysis of Data Centers During Power Shutdown,” ASHRAE Trans., 117(Pt. 1), pp. 212–221.
Sundaralingam, V. , Isaacs, S. , Kumar, P. , and Joshi, Y. , 2011, “ Modeling Thermal Mass of a Data Center Validated With Actual Data Due to Chiller Failure,” ASME Paper No. IMECE2011-65573.
Ibrahim, M. , Shrivastava, S. , Sammakia, B. , and Ghose, K. , 2012, “ Thermal Mass Characterization for a Server at Different Fan Speeds,” IEEE Intersociety Conference on Thermal and Thermomechanical Phenomena in Electronics Systems (ITHERM), San Diego, CA, May 30–June 1, pp. 457–465.
Erden, H. S. , Khalifa, H. E. , and Schmidt, R. R. , 2013, “ Transient Thermal Response of Servers Through Air Temperature Measurements,” ASME Paper No. IPACK2013-73281.
Erden, H. S. , Khalifa, H. E. , and Schmidt, R. R. , 2014, “ Determination of the Lumped-Capacitance Parameters of Air-Cooled Servers Through Air Temperature Measurements,” ASME J. Electron. Packag., 136(3), p. 031005. [CrossRef]
Alkharabsheh, S. , Sammakia, B. , Shrivastava, S. , and Schmidt, R. , 2014 “ Implementing Rack Thermal Capacity in a Room Level CFD Model of a Data Center,” IEEE Semiconductor Thermal Measurement and Management Symposium (SEMI-THERM), San Jose, CA, Mar. 9–13, pp. 188–192.
Shrivastava, S. , and Ibrahim, M. , 2013, “ Benefit of Cold Aisle Containment During Cooling Failure,” ASME Paper No. IPACK2013-73219.
Schmidt, R. , 2004, “ Thermal Profile of a High-Density Data Center: Methodology to Thermally Characterize a Data Center,” ASHRAE Trans., 110(2), pp. 635–642.
Schmidt, R. , and Iyengar, M. , 2005, “ Effect of Data Center Layout on Rack Inlet Air Temperatures,” ASME Paper No. IPACK2005-73385.
Karlsson, J. F. , and Moshfegh, B. , 2005, “ Investigation of Indoor Climate and Power Usage in a Data Center,” Energy Build., 37(10), pp. 1075–1083. [CrossRef]
Boucher, T. D. , Auslander, D. M. , Bash, C. E. , Federspiel, C. C. , and Patel, C. D. , 2005, “ Viability of Dynamic Cooling Control in a Data Center Environment,” ASME J. Electron. Packag., 128(2), pp. 137–144. [CrossRef]
Beitelmal, M. H. , Wang, Z. , Felix, C. , Bash, C. , Hoover, C. , and McReynolds, A. , 2009, “ Local Cooling Control of Data Centers With Adaptive Vent Tiles,” ASME Paper No. InterPACK2009-89035.
Chen, K. , Auslander, D. M. , Bash, C. E. , and Patel, C. D. , 2006, “ Local Temperature Control in Data Center Cooling: Part I, Correlation Matrix,” HP Enterprise Software and Systems Laboratory, Report No. HPL-2006-42.
Chen, K. , Bash, C. E. , Auslander, D. M. , and Patel, C. D. , 2006, “ Local Temperature Control in Data Center Cooling: Part II, Statistical Analysis,” HP Enterprise Software and Systems Laboratory, Report No. HPL-2006-43.
Abdelmaksoud, W. A. , Dang, T. Q. , Khalifa, H. E. , and Schmidt, R. R. , 2013, “ Improved Computational Fluid Dynamics Model for Open-Aisle Air-Cooled Data Center Simulations,” ASME J. Electron. Packag., 135(3), p. 030901. [CrossRef]
Arghode, V. K. , and Joshi, Y. , 2015, “ Experimental Investigation of Air Flow Through a Perforated Tile in a Raised Floor Data Center,” ASME J. Electron. Packag., 137(1), p. 011011. [CrossRef]
Bhopte, S. , Sammakia, B. , Iyengar, M. , and Schmidt, R. , 2007, “ Experimental Investigation of the Impact of Under Floor Blockages on Flow Distribution in a Data Center Cell,” ASME Paper No. IPACK2007-33540.
Vangilder, J. W. , Pardey, Z. M. , Zhang, X. , and Healey, C. , 2013, “ Experimental Measurement of Server Thermal Effectiveness for Compact Transient Data Center Model,” ASME Paper No. IPACK2013-73155.
Iyengar, M. , Schmidt, R. , Hamann, H. , and Vangilder, J. , 2007, “ Comparison Between Numerical and Experimental Temperature Distributions in a Small Data Center Test Cell,” ASME Paper No. IPACK2007-33508.
Fakhim, B. , Behnia, M. , Armfield, S. W. , and Srinarayana, N. , 2011, “ Cooling Solutions in an Operational Data Centre: A Case Study,” Appl. Therm. Eng., 31(14–15), pp. 2279–2291. [CrossRef]
Arghode, V. K. , and Joshi, Y. , 2014, “ Room Level Modeling of Air Flow in a Contained Data Center Aisle,” ASME J. Electron. Packag., 136(1), p. 011011. [CrossRef]
Simons, R. , Moran, K. , Antonetti, V. , and Chu, R. , 1982, “ Thermal Design of the IBM 3081 Computer,” National Electronic Packaging and Production Conference, Anaheim, CA, Feb. 23–25, pp. 124–141.
Chu, R. , Hwang, U. , and Simons, R. , 1982, “ Conduction Cooling for an LSI Package: A One Dimensional Approach,” IBM J. Res. Dev., 26(1), pp. 45–54. [CrossRef]
Hwang, U. , and Moran, K. , 1990, “ Cold Plate for IBM Thermal Conduction Module Electronic Modules,” Heat Transfer in Electronic and Microelectronic Equipment, Vol. 29, A. E. Bergles, ed., Hemisphere, New York, pp. 495–508.
Delia, D. , Gilgert, T. , Graham, N. , Hwang, U. , Ing, P. , Kan, J. , Kemink, R. , Maling, G. , Martin, R. , Moran, K. , Reyes, J. , Schmidt, R. , and Steinbrecher, R. , 1992, “ System Cooling Design for the Water-Cooled IBM Enterprise System/9000 Processors,” IBM J. Res. Dev., 36(4), pp. 791–803. [CrossRef]
Lei, N. , Skandakumaran, P. , and Ortega, A. , 2006, “ Experiments and Modeling of Multilayer Copper Minichannel Heat Sinks in Single-Phase Flow,” IEEE Intersociety Conference on Thermal and Thermomechanical Phenomena in Electronic Systems (ITHERM '06), San Diego, CA, May 30–June 2, pp. 9–18.
Dede, E. , 2014, “ Single-Phase Microchannel Cold Plate for Hybrid Vehicle Electronics,” IEEE Semiconductor Thermal Measurement and Management Symposium (SEMI-THERM), San Jose, CA, Mar. 9–13, pp. 118–124.
Iyengar, M. , David, M. , Parida, P. , Kamath, V. , Kochuparambil, B. , Graybill, D. , Schultz, M. , Gaynes, M. , Simons, R. , Schmidt, R. , and Chainer, T. , 2012, “ Server Liquid Cooling With Chiller-Less Data Center Design to Enable Significant Energy Savings,” IEEE Semiconductor Thermal Measurement and Management Sysposium (SEMI-THERM), San Jose, CA, Mar. 18–22, pp. 212–223.
David, M. , Iyengar, M. , Parida, P. , Simons, R. , Schultz, M. , Gaynes, M. , Schmidt, R. , and Chainer, T. , 2012, “ Experimental Characterization of an Energy Efficient Chiller-Less Data Center Test Facility With Warm Water Cooled Servers,” 28th Annual IEEE Semiconductor Thermal and Measurement and Management Symposium (SEMI-THERM), San Jose, CA, Mar. 18–22, pp. 232–237.
Parida, P. , David, M. , Iyengar, M. , Schultz, M. , Gaynes, M. , Kamath, V. , Kochuparambil, B. , and Chainer, T. , 2012, “ Experimental Investigation of Water Cooled Server Microprocessors and Memory Devices in an Energy Efficient Chiller-Less Data Center,” 28th Annual IEEE Semiconductor Thermal Measurement and Management Symposium (SEMI-THERM), San Jose, CA, Mar. 18–22, pp. 224–231.
Iyengar, M. , David, M. , Parida, P. , Kamath, V. , Kochuparambil, B. , Graybill, D. , Schultz, M. , Gaynes, M. , Simons, R. , Schmidt, R. , and Chainer, T. , 2012, “ Extreme Energy Efficiency Using Water Cooled Server Inside a Chiller-Less Data Center,” 13th IEEE Intersociety Conference on Thermal and Thermomechanical Phenomena in Electronics Systems (ITHERM), San Diego, CA, May 30–June 1, pp. 137–149.
David, M. , Iyengar, M. , Parida, P. , Simons, R. , Schultz, M. , Gaynes, M. , Schmidt, R. , and Chainer, T. , 2012, “ Impact of Operating Conditions on a Chiller-Less Data Center Test Facility With Liquid Cooled Servers,” 13th IEEE Intersociety Conference on Thermal and Thermomechanical Phenomena in Electronics Systems (ITHERM), San Diego, CA, May 30–June 1, pp. 562–573.
Eiland, R. , Fernandes, J. , Vallejo, M. , Agonafer, D. , and Mulay, V. , 2014, “ Flow Rate and Inlet Temperature Considerations for Direct Immersion of a Single Server in Mineral Oil,” IEEE Intersociety Conference on Thermal and Thermomechanical Phenomena in Electronics Systems (ITHERM), Orlando, FL, May 27–30, pp. 706–714.
Tuma, P. , 2010, “ The Merits of Open Bath Immersion Cooling of Datacom Equipment,” IEEE Semiconductor Thermal Measurement and Management Symposium (SEMI-THERM), Santa Clara, CA, Feb. 21–25, pp. 123–131.
Patterson, M. K. , and Fenwick, D. , 2008, “ The State of Data Center Cooling: A Review of Current Air and Liquid Cooling Solutions,” Intel, Digital Enterprise Group, White Paper.
Blough, B. , ed., 2011, “ Qualitative Analysis of Cooling Architectures for Data Centers,” The Green Grid, Beaverton, OR, Report No. 30.
Niemann, J. , 2008, “ Hot Aisle vs. Cold Aisle Containment,” American Power Conversion, West Kingston, RI, White Paper No. 135.
Niemann, J. , Brown, K. , and Avelar, V. , 2010, “ Hot-Aisle vs. Cold-Aisle Containment for Data Centers,” American Power Conversion, West Kingston, RI, White Paper No. 135, rev. 1.
Tozer, R. , and Salim, M. , 2010, “ Data Center Air Management Metrics-Practical Approach,” 12th IEEE Intersociety Conference on Thermal and Thermomechanical Phenomena in Electronic Systems (ITHERM), Las Vegas, NV, June 2–5, pp. 1–8.
Gondipalli, S. , Sammakia, B. , Bhopte, S. , Schmidt, R. , Iyengar, M. , and Murray, B. , 2009, “ Optimization of Cold Aisle Isolation Designs for a Data Center With Roofs and Doors Using Slits,” ASME Paper No. InterPACK2009-89203.
Emerson Network Power, 2010, “ Combining Cold Aisle Containment With Intelligent Control to Optimize Data Center Cooling Efficiency,” Emerson Network Power, Columbus, OH, White Paper.
Pervila, M. , and Kangasharju, J. , 2011, “ Cold Air Containment,” 2nd ACM SIGCOMM Workshop on Green Networking (GreenNets '11), Toronto, ON, Canada, Aug. 15–19, pp. 7–12.
Schmidt, R. , Vallury, A. , and Iyengar, M. , 2011, “ Energy Savings Through Hot and Cold Aisle Containment Configurations for Air Cooled Servers in Data Centers,” ASME Paper No. IPACK2011-52206.
Shrivastava, S. K. , Calder, A. R. , and Ibrahim, M. , 2012, “ Quantitative Comparison of Air Containment Systems,” 13th IEEE Intersociety Conference on Thermal and Thermomechanical Phenomena in Electronics Systems (ITHERM), San Diego, CA, May 30–June 1, pp. 68–77.
Demetriou, D. W. , and Khalifa, H. E. , 2011, “ Energy Modeling Of Air-Cooled Data Centers—Part l: The Optimization of Enclosed Aisle Configurations,” ASME Paper No. IPACK2011-52003.
Xu, Y. , Gao, Z. , and Deng, Y. , 2013, “ Analyzing the Cooling Behavior of Hot and Cold Aisle Containment in Data Centers,” 4th IEEE International Conference on Emerging Intelligent Data and Web Technologies (EIDWT), Xi'an, China, Sept. 9–11, pp. 685–689.
Sundaralingam, V. , Arghode, V. K. , Joshi, Y. , and Phelps, W. , 2015, “ Experimental Characterization of Various Cold Aisle Containment Configurations for Data Centers,” ASME J. Electron. Packag., 137(1), p. 011007. [CrossRef]
Muradlidharan, B. , Ibrahim, M. , Shrivistava, S. , Alkharabsheh, S. , and Sammakia, B. , 2013, “ Advantages of Cold Aisle Containemnt (CAC) System and Its Leakage Quantification,” ASME Paper No. IPACK2013-73201.
Kennedy, J. , 2012, “ Ramification of Server Airflow Leakage in Data Centers With Aisle Containment,” Tate Access Floors, Jessup, MD, White Paper.
Alkharabsheh, S. A. , Muralidharan, B. , Ibrahim, M. , Shrivastava, S. , and Sammakia, B. , 2013, “ Open and Contained Cold Aisle Experimentally Validated CFD Model Implementing CRAC and Servers Fan Curves on a Testing Data Center,” ASME Paper No. IPACK2013-73214.
Alkharabsheh, S. A. , Shrivastava, S. K. , and Sammakia, B. G. , 2013, “ Effect of Containment System Perforation on Data Center Flow Rates and Temperatures,” ASME Paper No. IPACK2013-73216.
Gebrehiwot, B. , Dhiman, N. , Rajagopalan, K. , Agonafer, D. , Kannan, N. , Hoverson, J. , and Kaler, M. , 2013, “ CFD Modeling of Indirect/Direct Evaporative Cooling Unit for Modular Data Center Applications,” ASME Paper No. IPACK2013-73302.
Vasani, R. , and Agonafer, D. , 2014, “ Numerical Simulation of Fogging in a Square Duct—A Data Center Perspective,” 30th Annual IEEE Semiconductor Thermal Measurement and Management Symposium (SEMI-THERM), San Jose, CA, Mar. 9–13, pp. 45–52.
Seger, D. , and Solberg, A. , “ Economizer Performance: Applying CFD Modeling to the Data Center's Exterior,” SearchDataCenter.com, accessed October 26, 2014, http://searchdatacenter.techtarget.com/tip/Economizer-performance-Applying-CFD-modeling-to-the-data-centers-exterior
Beaty, D. L. , 2004, “ Liquid Cooling: Friend or Foe,” ASHRAE Trans., 110(2), pp. 643–652.
Ellsworth, M. J. , Campbell, L. A. , Simons, R. E. , Iyengar, M. , and Schmidt, R. R. , 2008, “ The Evolution of Water Cooling for IBM Large Server Systems: Back to the Future,” 11th IEEE Intersociety Conference on Thermal and Thermomechanical Phenomena in Electronic Systems (ITHERM 2008), Lake Buena Vista, FL, May 28–31, pp. 266–274.
ASHRAE TC 9.9, 2011, “ Thermal Guidelines for Liquid Cooled Data Processing Environments,” American Society of Heating, Refrigerating and Air Conditioning Engineers (ASHRAE), White Paper.
Heydari, A. , and Sabounchi, P. , 2004, “ Refrigeration Assisted Spot Cooling of a High Heat Density Data Center,” Intersociety Conference on Thermal and Thermomechanical Phenomena in Electronic Systems (ITHERM '04), Las Vegas, NV, June 1–4, pp. 601–606.
Mulay, V. , Agonafer, D. , and Schmidt, R. , 2008, “ Liquid Cooling for Thermal Management of Data Centers,” ASME Paper No. IMECE2008-68743.
Schmidt, R. , and Iyengar, M. , 2009, “ Server Rack Rear Door Heat Exchanger and the New ASHRAE Recommended Environmental Guidelines,” ASME Paper No. InterPACK2009-89212.
Tsukamoto, T. , Takayoshi, J. , Schmidt, R. , and Iyengar, M. , 2009, “ Refrigeration Heat Exchanger Systems for Server Rack Cooling in Data Centers,” ASME Paper No. InterPACK2009-89258.
Iyengar, M. , Schmidt, R. , Kamath, V. , and Kochuparambil, B. , 2011, “ Experimental Characterization of Server Rack Energy Use at Elevated Ambient Temperatures,” ASME Paper No. IPACK2011-52207.
Iyengar, M. , Schmidt, R. , and Caricari, J. , 2010, “ Reducing Energy Usage in Data Centers Through Control of Room Air Conditioning Units,” 12th IEEE Intersociety Conference on Thermal and Thermomechanical Phenomena in Electronics Systems (ITHERM), Las Vegas, NV, June 2–5, pp. 1–11.
Fernandes, J. , Ghalambor, S. , Agonafer, D. , Kamath, V. , and Schmidt, R. , 2012, “ Multi-Design Variable Optimization for a Fixed Pumping Power of a Water-Cooled Cold Plate for High Power Electronics Applications,” 13th IEEE Intersociety Conference on Thermal and Thermomechanical Phenomena in Electronics Systems (ITHERM), San Diego, CA, May 30–June 1, pp. 684–692.
Goth, G. , Arvelo, A. , Eagle, J. , Ellsworth, M. , Marston, K. , Sinha, A. , and Zitz, J. , 2012, “ Thermal and Mechanical Analysis and Design of the IBM Power 775 Water Cooled Supercomputing Central Electronics Complex,” 13th IEEE Intersociety Conference on Thermal and Thermomechanical Phenomena in Electronics Systems (ITHERM), San Diego, CA, May 30–June 1, pp. 700–709.
Brunschwiler, T. , Rothuizen, H. , Paredes, S. , Michel, B. , Colgan, E. , and Bezama, P. , 2009, “ Hotspot-Adapted Cold Plates to Maximize System Efficiency,” 15th IEEE International Workshop on Thermal Investigations of ICs and Systems (THERMINIC 2009), Leuven, Belgium, Oct. 7–9, pp. 150–156.

Figures

Grahic Jump Location
Fig. 1

Power usage effectiveness (PUE) survey. More than 55% of the sample exhibit PUE greater than 1.8. Data adopted from the Uptime Institute Data Center Industry Survey [7].

Grahic Jump Location
Fig. 2

Multiscale thermal systems. The heat is essentially generated from a chip level inside the IT equipment. The heat transfers through multiscale subsystems from the chip level, server level, rack level, and data center room level system [16].

Grahic Jump Location
Fig. 3

Most common data center cooling scheme. Raised floor forms a plenum for the cold air supplied from the cooling units (CRAC/CRAH). Cold air enters the area above raised floor through perforated tiles. Server racks are used to house the IT equipment and provide the necessary structure for cooling through front and rear perforated doors (Photo courtesy of 42U Data Center).

Grahic Jump Location
Fig. 4

(a) Modular data center layout, (b) effect of plenum height on the airflow distribution, and (c) effect of tile open area ratio on the airflow distribution. Increasing the plenum depth and decreasing the tile open area enhance the uniformity of the flow in the tiles [35].

Grahic Jump Location
Fig. 5

(a) Rack and server CFD model, (b) temperature contours for 100 W dissipated power, (c) velocity vectors showing the air recirculation inside the rack, and (d) rack with internal blockages to prevent recirculation [50]

Grahic Jump Location
Fig. 6

(a) Temperature contours showing the baseline model and (b) temperature contours showing the optimized model. The inlet temperatures are reduced and the hot spots become less prominent by changing the plenum depth, cold aisle location, and height of the room [66].

Grahic Jump Location
Fig. 7

(a) Modular data center used for room level analysis and (b) inlet temperature response for room level model showing the effect of server heat capacity (HC). The server HC has a significant impact on the transient response and must be included in transient simulation for accurate estimation of the thermal time constant [57].

Grahic Jump Location
Fig. 8

(a) Experimental setup showing hot-wire anemometer probe to measure the air velocity, (b) tested tiles, and (c) measurements and CFD simulations for tile C with symmetric 25% perforation. The experimental results used to develop numerical model for an accurate prediction of downstream velocity of a tile using CFD simulations [38].

Grahic Jump Location
Fig. 9

(a) Sectioned view of multipass branching microchannel cold plate and (b) schematic of the experimental setup. This cold plate design shows good thermal characteristics, but advanced diffusion bonding techniques are needed, which can be challenging for high-volume production [112].

Grahic Jump Location
Fig. 10

(a) CAC and (b) hot aisle containment (HAC). Containment systems reduce the hot air recirculation and enhance the inlet temperature uniformity, which leads to energy savings (Photo courtesy of 42U Data Center).

Grahic Jump Location
Fig. 11

(a) Research data center layout, (b) cold aisle configurations, and (c) experimental results at the rack inlet for overprovisioned cold aisle case. It is recommended to overprovision CACs and to use a ceiling only containment system over doors only if full containment is not an available option [132].

Grahic Jump Location
Fig. 12

(a) Schematic of detailed CAC model, (b) schematic of detailed rack model, (c) results of validating the CFD model of CAC, and (d) the impact of leakage at high elevation of the racks. Detailed modeling of CAC panels and calibration of the pressure drops in cooling units and servers is important for accurate CAC simulations. Small overprovisioning does not prevent leakage [78].

Grahic Jump Location
Fig. 13

CFD model of indirect/direct evaporative cooling unit. High face velocity affects the life and the performance of the air filters. Air flow distribution improvement ideas should address this challenge [137].

Grahic Jump Location
Fig. 14

Rear door heat exchanger for cooling rack exhaust air: (a) schematic side view, (b) rack mounted example, and (c) data center application. The exhaust hot air goes through the heat exchanger and gets cooled before it recirculates into the cold aisle [141].

Grahic Jump Location
Fig. 15

(a) Specifics of the cold plate's geometry chosen for optimization, specifically design variables used as input, namely, serpentine channel width (not highlighted) and height (indicated as middle thickness); and influence of said parameters on (b) weight, and (c) thermal performance of the cooling solution [149].

Tables

Errata

Some tools below are only available to our subscribers or users with an online account.

Related Content

Customize your page view by dragging and repositioning the boxes below.

Related Journal Articles
Related eBook Content
Topic Collections

Sorry! You do not have access to this content. For assistance or to subscribe, please contact us:

  • TELEPHONE: 1-800-843-2763 (Toll-free in the USA)
  • EMAIL: asmedigitalcollection@asme.org
Sign In