Research Papers

Thermal Performance and Efficiency of a Mineral Oil Immersed Server Over Varied Environmental Operating Conditions

[+] Author and Article Information
Richard Eiland

Department of Mechanical and
Aerospace Engineering,
University of Texas at Arlington,
P.O. Box 19023,
Arlington, TX 76013
e-mail: richard.eiland@mavs.uta.edu

John Edward Fernandes

Department of Mechanical and
Aerospace Engineering,
University of Texas at Arlington,
P.O. Box 19023,
Arlington, TX 76013
e-mail: john.fernandes@mavs.uta.edu

Marianna Vallejo

2020 SW 4th Avenue, Street 300,
Portland, OR 97201
e-mail: marianna.vallejo@ch2m.com

Ashwin Siddarth

Department of Mechanical and
Aerospace Engineering,
University of Texas at Arlington,
P.O. Box 19023,
Arlington, TX 76013
e-mail: ashwin.siddarth@mavs.uta.edu

Dereje Agonafer

Fellow ASME
Department of Mechanical and
Aerospace Engineering,
University of Texas at Arlington,
P.O. Box 19023,
Arlington, TX 76013
e-mail: agonafer@uta.edu

Verrendra Mulay

Facebook Inc.,
Menlo Park, CA 425081
e-mail: vmulay@fb.com

1Corresponding author.

Contributed by the Electronic and Photonic Packaging Division of ASME for publication in the JOURNAL OF ELECTRONIC PACKAGING. Manuscript received February 16, 2017; final manuscript received August 1, 2017; published online September 7, 2017. Assoc. Editor: Baris Dogruoz.

J. Electron. Packag 139(4), 041005 (Sep 07, 2017) (9 pages) Paper No: EP-17-1020; doi: 10.1115/1.4037526 History: Received February 16, 2017; Revised August 01, 2017

Complete immersion of servers in dielectric mineral oil has recently become a promising technique for minimizing cooling energy consumption in data centers. However, a lack of sufficient published data and long-term documentation of oil immersion cooling performance make most data center operators hesitant to apply these approaches to their mission critical facilities. In this study, a single server was fully submerged horizontally in mineral oil. Experiments were conducted to observe the effects of varying the volumetric flow rate and oil inlet temperature on thermal performance and power consumption of the server. Specifically, temperature measurements of the central processing units (CPUs), motherboard (MB) components, and bulk fluid were recorded at steady-state conditions. These results provide an initial bounding envelope of environmental conditions suitable for an oil immersion data center. Comparing with results from baseline tests performed with traditional air cooling, the technology shows a 34.4% reduction in the thermal resistance of the system. Overall, the cooling loop was able to achieve partial power usage effectiveness (pPUECooling) values as low as 1.03. This server level study provides a preview of possible facility energy savings by utilizing high temperature, low flow rate oil for cooling. A discussion on additional opportunities for optimization of information technology (IT) hardware and implementation of oil cooling is also included.

Copyright © 2017 by ASME
Your Session has timed out. Please sign back in to continue.


ASHRAE TC 9.9, 2015, Thermal Guidelines for Data Processing Environments, 4th ed., ASHRAE Datacom Series, ASHRAE, Atlanta, GA.
Alkharabsheh, S. , Fernandes, J. , Gebrehiwot, B. , Agonafer, D. , Ghose, K. , Ortega, A. , Joshi, Y. , and Sammakia, B. , 2015, “ A Brief Overview of Recent Developments in Thermal Management in Data Centers,” ASME J. Electron. Packag., 137(4), p. 040801. [CrossRef]
Ellsworth, M. J., Jr. , and Iyengar, M. K. , 2009, “ Energy Efficiency Analyses and Comparison of Air and Water Cooled High Performance Servers,” ASME Paper No. InterPACK2009-89248.
Carbó, A. , Oró, E. , Salom, J. , Canuto, M. , Macías, M. , and Guitart, J. , 2016, “ Experimental and Numerical Analysis for Potential Heat Reuse in Liquid Cooled Data Centres,” Energy Convers. Manage., 112, pp. 135–145. [CrossRef]
Chu, R. C. , Hwang, U. P. , and Simons, R. E. , 1982, “ Conduction Cooling for an LSI Package: A One-Dimensional Approach,” IBM J. Res. Dev., 26(1), pp. 45–54. [CrossRef]
Fernandes, J. , Ghalambor, S. , Docca, A. , Aldham, C. , Agonafer, D. , Chenelly, E. , Chan, B. , and Ellsworth, M., Jr. , 2013, “ Combining Computational Fluid Dynamics (CFD) and Flow Network Modeling (FNM) Design of a Multi-Chip Module (MCM) Cold Plate,” ASME Paper No. IPACK2013-73294.
Ovaska, S. J. , Dragseth, R. E. , and Hanssen, S. A. , 2016, “ Direct-to-Chip Liquid Cooling for Reducing Power Consumption in a Subarctic Supercomputer Centre,” Int. J. High Perform. Comput. Networking, 9(3), pp. 242–249. [CrossRef]
ASHRAE TC 9.9, 2011, “ 2011 Thermal Guidelines for Liquid Cooled Data Processing Environments,” ASHRAE, Atlanta, GA.
Incropera, F. P. , 1999, Liquid Cooling of Electronic Devices by Single-Phase Convection, Wiley-Interscience Publication, New York.
Anderson, T. M. , and Mudawar, I. , 1989, “ Microelectronic Cooling by Enhanced Pool Boiling of a Dielectric Fluorocarbon Liquid,” ASME J. Heat Transfer, 111(3), pp. 752–759. [CrossRef]
Tuma, P. , 2010, “ The Merits of Open Bath Immersion Cooling of Datacom Equipment,” 26th Annual IEEE Semiconductor Thermal Measurement and Management Symposium (SEMI-THERM), Santa Clara, CA, Feb. 21–25, pp. 123–131.
Arik, M. , 2007, “ Enhancement of Pool Boiling Critical Heat Flux in Dielectric Liquids by Microporous Coatings,” Int. J. Heat Mass Transfer, 50(5–6), pp. 997–1009. [CrossRef]
El-Genk, M. S. , 2012, “ Immersion Cooling Nucleate Boiling of High Power Computer Chips,” Energy Convers. Manage., 53(1), pp. 205–218. [CrossRef]
Warrier, P. , Sathyanarayana, A. , Patil, D. V. , France, S. , Joshi, Y. , and Teja, A. S. , 2012, “ Novel Heat Transfer Fluids for Direct Immersion Phase Change Cooling of Electronic Systems,” Int. J. Heat Mass Transfer, 55(13–14), pp. 3379–3385. [CrossRef]
Pierce, L. W. , 1992, “ An Investigation of the Thermal Performance of an Oil Filled Transformer Winding,” IEEE Trans. Power Delivery, 7(3), pp. 1347–1358. [CrossRef]
Taha-Tijerina, J. J. , Narayana, T. , Tiwary, C. S. , Lozano, K. , Chipara, M. , and Ajayan, P. M. , 2014, “ Nanodiamond-Based Thermal Fluids,” ACS Appl. Mater. Interfaces, 6(7), pp. 4778–4785. [CrossRef] [PubMed]
Jones, P. , 2012, “ Data Center Dynamics,” Data Center Dynamics Ltd., London, accessed Nov. 15, 2013, http://www.datacenterdynamics.com/focus/archive/2012/09/intel-says-liquid-and-servers-do-mix
Fulton, S., III , 2017, “ Data Center Knowledge,” Data Center Knowledge, San Francisco, CA, accessed June 2, 2017, http://www.datacenterknowledge.com/archives/2017/05/01/data-center-cooling-how-practical-is-dunking-servers-in-oil/
Prucnal, D. , 2013, “ Doing More With Less: Cooling Computers With Oil Pays Off,” Next Wave, 20(2), pp. 20–29. http://www.grcooling.com/wp-content/uploads/2015/06/NSA-The-Next-Wave.pdf
Patterson, M. , and Best, C. , 2012, “ Oil Immersion Cooling and Reductions In Data Center Energy Use,” ASHRAE Summer Conference, San Antonio, TX, June 23–27.
Frachtenberg, E. , Heydari, A. , Li, H. , Michael, A. , Na, J. , Nisbet, A. , and Sarti, P. , 2011, “ High-Efficiency Server Design,” International Conference for High Performance Computing, Networking, Storage, and Analysis (SC), Seattle, WA, Nov. 12–18, pp. 1–11.
Li, H. , and Michael, A. , 2011, “ Intel Motherboard Hardware v1.0,” Open Compute Project, accessed Nov. 22, 2013, http://www.opencompute.org/projects/intel-motherboard/
Carraway, D. , 2013, “ Lookbusy: A Synthetic Load Generator,” Look Busy, accessed Aug. 18, 2017, http://www.devin.com/lookbusy
The Green Grid, 2008, “ The Green Grid Data Center Power Efficiency Metrics: PUE and DCiE,” The Green Grid Administration, Beaverton, OR, White Paper No. 6. http://www.premiersolutionsco.com/wp-content/uploads/TGG_Data_Center_Power_Efficiency_Metrics_PUE_and_DCiE.pdf
The Green Grid, 2012, “ PUE: A Comprehensive Examination of the Metric,” The Green Grid Administration, Beaverton, OR, White Paper No. 49. https://datacenters.lbl.gov/sites/all/files/WP49-PUE%20A%20Comprehensive%20Examination%20of%20the%20Metric_v6.pdf
Godfrey, D. , and Herguth, W. R. , 1996, Physical and Chemical Properties of Industrial Mineral Oils Affecting Lubrication—Parts 1–5, Society of Tribologists and Lubrication Engineers, Park Ridge, IL.
Eiland, R. , Fernandes, J. , Vallejo, M. , Agonafer, D. , and Mulay, V. , 2014, “ Flow Rate and Inlet Temperature Considerations for Direct Immersion of a Single Server in Mineral Oil,” IEEE Intersociety Conference on Thermal and Thermomechanical Phenomena in Electronic Systems (ITherm), Orlando, FL, May 27–30, pp. 706–714.
White, F. M. , 2011, Fluid Mechanics, 7th ed., McGraw-Hill, New York.


Grahic Jump Location
Fig. 3

Diagram of the two flow configurations through the immersion tank. “MB flow path” occurs when only the MB inlet and outlet valves are open. “PSU flow path” occurs when only the PSU inlet and outlet valves are open.

Grahic Jump Location
Fig. 2

Schematic of test setup and data collection equipment

Grahic Jump Location
Fig. 1

Intel-based Open Compute server in standard air-cooled configuration with air duct removed for visual purposes

Grahic Jump Location
Fig. 4

Typical test duration and establishing steady-state conditions for the server under test

Grahic Jump Location
Fig. 5

Impact of increasing oil volume flow rate on CPU die temperature along lines of constant oil inlet temperature for the MB flow path case

Grahic Jump Location
Fig. 6

Impact of increasing oil volume flow rate on CPU die temperature along lines of constant oil inlet temperature for the PSU flow path case

Grahic Jump Location
Fig. 7

Relation between system approach temperature and efficiency

Grahic Jump Location
Fig. 8

Surface contour of the normalized total system power consumption for the MB flow path case

Grahic Jump Location
Fig. 9

General relationship between component temperatures and total server power based on the data collected in the MB flow path case

Grahic Jump Location
Fig. 10

Comparison of trends for temperature-dependent pumping power for the current system and temperature dependent viscosity of transformer oil as predicted by Eq. (3)

Grahic Jump Location
Fig. 11

Temperature-dependent oil flow rates and cubic relation of pumping power to flow rate

Grahic Jump Location
Fig. 12

Competing trends for pumping power and server IT power over a range of oil inlet temperatures at constant operating oil flow rates



Some tools below are only available to our subscribers or users with an online account.

Related Content

Customize your page view by dragging and repositioning the boxes below.

Related Journal Articles
Related eBook Content
Topic Collections

Sorry! You do not have access to this content. For assistance or to subscribe, please contact us:

  • TELEPHONE: 1-800-843-2763 (Toll-free in the USA)
  • EMAIL: asmedigitalcollection@asme.org
Sign In