0
Photogallery

# Optimization of Enclosed Aisle Data Centers Using Bypass RecirculationOPEN ACCESS

[+] Author and Article Information
Dustin W. Demetriou1

Ph.D. Candidate and Research AssistantMechanical & Aerospace Engineering,  Syracuse University, Syracuse, NY 13244dwdemetr@syr.edu

H. Ezzat Khalifa

Professor of Mechanical & Aerospace Engineering,  Syracuse University, Syracuse, NY 13210

1

Corresponding author.

J. Electron. Packag 134(2), 020904 (Jun 11, 2012) (8 pages) doi:10.1115/1.4005907 History: Received July 28, 2011; Revised November 10, 2011; Published June 11, 2012; Online June 11, 2012

## Abstract

The work presented in this paper describes a simplified thermodynamic model that can be used for exploring optimization possibilities in air-cooled data centers. The model is used to evaluate parametrically the total energy consumption of the data center cooling infrastructure for data centers that utilize aisle containment. The analysis highlights the importance of reducing the total power required for moving the air within the computer room air conditioners (CRACs), the plenum, and the servers, rather than focusing primarily or exclusively on reducing the refrigeration system’s power consumption. In addition, the benefits of introducing a bypass recirculation branch in enclosed aisle configurations are shown. The analysis shows a potential for as much as a 60% savings in cooling infrastructure energy consumption by utilizing an optimized enclosed aisle configuration with bypass recirculation, instead of a traditional enclosed aisle in which all the data center exhaust is forced to flow through the CRACs. Furthermore, computational fluid dynamics is used to evaluate practical arrangements for implementing bypass recirculation in raised floor data centers. A configuration where bypass tiles, with controllable low-lift fans, are placed close to the discharge of CRACs results in increased mixing and is shown to be a suitable method for providing nearly thermally uniform conditions to the inlet of the servers in an enclosed cold aisle. Other configurations of bypass implementation are also discussed and explored.

<>

## Introduction

In the recent years, the information technology (IT) industry has undergone a shift in paradigm related to how the total cost of ownership of the data center is divided. The main objective of data center design and operation is to provide an acceptable thermal environment for the IT equipment, in order to maintain high levels of reliability. In the early 1990s, the cost of operating a data center was driven primarily by the high initial cost of purchasing the IT hardware. Numerous trends in the IT industry have led to a significant increase in data center’s energy consumption. Between 1999 and 2005, electricity consumption by data centers rose more than 39% [1]. Between 2000 and 2010, while worldwide IT hardware costs have stayed almost constant, the estimated cost of power and cooling has increased significantly. It has been reported that the nation’s data centers consumed 61  ×  109 kWh of electricity in 2006, which translates to 1.5% of the total electricity consumption of the United States [2]. However, not all of this electricity goes directly to powering the IT equipment. The power and cooling infrastructure that support the IT equipment consumes nearly 50% of that electricity. Salim and Tozer [1] performed an energy audit of 40 data centers, where they benchmarked the component-by-component energy consumption of both the mechanical and electrical systems. The results showed that, as a percentage of the total data center power, the cooling (refrigeration) system consumed between 15% and 35% and the CRAC/CRAH (computer room air handler) fan power consumed between 10% and 20%. They attributed the large fan power to low efficiency motors and limited use of variable frequency drives. Interestingly, they found that small data centers (<10,000 ft2 ) had large refrigeration power consumption, but typically employed lower efficiency direct expansion CRAC units, instead of a more efficient central chilled water system widely used in large data centers.

ASHRAE [3] compiled a set of guidelines for the acceptable condition of the air entering the IT equipment. The recommended guidelines are provided to assure the reliability of the equipment. The allowable guidelines give a maximum limit where the equipment can be operated for short periods of time and still maintain functionality. The updated guidelines expanded the acceptable range to allow data center operators to provide a higher inlet temperature (27 °C) to the equipment to reduce refrigeration power and allow for more economizer use.

Reliability concerns in data centers force the cooling infrastructure to typically be overdesigned, which leads to lower efficiency operation of the equipment. Even though off-design operation could have a significant energy impact, little work has been done on optimizing the cooling infrastructure with time-varying off-design conditions. Lui [4] attributes this to the limitations in currently available simulations models in accounting for the dynamic behavior of the equipment. Further, little work has included both the data center airspace and cooling infrastructure in a coupled fashion. However, a number of studies in the recent years have focused on experimentation and computational fluid dynamics (CFD) modeling of data center airspace to understand the complex air flow and temperature patterns [5–10]. Shah et al.  [11] performed an exergy analysis of the data center cooling systems and showed that over 20% of the exergy introduced into the data center is wasted in either recirculation patterns or CRAC inefficiencies. In terms of the cooling infrastructure, recent literature has shown that the current state-of-the-art in data center thermal management consists of a single sensor feedback signal, which acts as an indication of the heat being dissipated in the room and controls the temperature of the CRAC supply air [12]. Several researchers have studied the dynamic optimization of CRAC/CRAH operation [12-15].

One of the first studies to consider the data center and cooling infrastructure from a holistic view was done by Iyengar and Schmidt [16]. They developed an analytical design-point model to predict the thermal performance and energy efficiency of the data center cooling loop. Pelley et al.  [17] recognized that the development of a model that incorporates important dynamics was essential to understanding the data center energy consumption. As a starting point, they developed a simplified system level model of the data center subsystems, specifically, servers, power conditioning, cooling, networking, lighting, and security. Hellmer [18] studied design-point system models and their energy impact using hour-by-hour weather data for various cities. Several authors have recognized the need for dynamic models for analyzing the data center cooling infrastructure. Demetriou et al.  [19] developed and experimentally validated a coupled thermohydraulic model for a chiller-plant-based data center infrastructure. The model accounts for the off-design performance of all the major equipments in the data center cooling loop, from the cooling tower to the servers. The model allows for various dynamic forcing functions including variations in ambient weather conditions and IT loading. The hydraulic network takes into account the changes in flow rate distribution based on CRAH operation, through realistic CRAH valve operating strategies. The model was validated against an operating 5 MW data center, showing excellent agreement in the aggregate. Walsh et al.  [20-21] developed a simple thermodynamic and heat transfer-based model to evaluate various operating strategies in data centers from the cooling tower to the chip. The works of Bejan et al.  [22-24] have highlighted the importance of considering the fan power requirement when optimizing the design of combined thermal/fluid/heat transfer systems, in order to minimize entropy generation. Much of his work has focused on using simple physics-based models that capture the most important characteristics of the system to uncover fundamental design trade-offs.

In this paper, we will develop a simplified physics-based model of an air-cooled data center which includes a simplified thermodynamic/heat transfer-based model of data center air recirculation and thermal balance and their effects on cooling infrastructure power consumption for data centers that incorporate aisle containment. Subsequently, we will present the results of the model for a case where the servers have a constant flow, which maintains a specified server temperature rise for a given server power dissipation, and show the benefits of using a proposed bypass recirculation branch. After showing the benefits of bypass recirculation in conjunction with enclosed aisle configurations, and discussing several practical arrangements, a study using computational fluid dynamics is presented to study the effectiveness of several of these configurations.

## Model Description

The model developed in this work looks at the optimization of the data center cooling infrastructure when enclosed aisle configurations are used. In this section, a description of a simple analysis of the data center and cooling infrastructure is presented.

###### Thermal Analysis.

A simple model for a server, rack, or a group of servers/racks in an enclosed aisle is depicted in Fig. 1. In this model, all the servers have been aggregated into a single “super” rack operating in mode m at an IT power Pm . The super rack is cooled by a stream of mixed air $m·m$ entering the rack at an average mixed temperature $T¯mi$. The average rack exit temperature $T¯mx$ is given by Display Formula

$T¯mx=T¯mi+Pm/(m·mcp)=T¯m+ΔTm(1)$
(1)
The cold air at Ta being supplied by the CRAC to the underfloor plenum will be divided into an active cooling component, $m·a$, that eventually enters the enclosed cold aisle, and a leakage component, $λm·a$, that bypasses the racks and blends homogeneously with the data center air. Therefore, the data center’s exhaust air temperature, which is also the CRAC inlet temperature, will be given by Display Formula
$T¯x=Ta+Pm(1+λ)m·acp(2)$
(2)
If we consider a cluster of identical servers/chassis in racks organized in an enclosed cold aisle/hot aisle arrangement, the air entering the cold aisle could be a mixture of cold air discharged by the CRACs into the underfloor plenum and warm air that is bypassed around the CRAC and supplied to the underfloor plenum. We express the fraction of server total flow that is bypassed exhaust air by the symbol ϕ, or the fraction of cold supply air which passes through the CRAC is $ψ¯=1-ϕ¯$. The exhaust temperature must also satisfy energy balance at rack exhaust, i.e. Display Formula
$T¯x=m·mT¯mx+λm·aTam·m+λm·a=T¯mx+λ(1-ϕ¯)Ta1+λ(1-ϕ¯)(3)$
(3)
If we assume that all the recirculated air is at the average exhaust temperature, $T¯x$, then the average rack inlet temperature must obey Display Formula
$T¯mi=ϕ¯T¯x+(1-ϕ¯)Ta(4)$
(4)
An ideal case arises if by some means it was possible to supply every operating server with air at the same inlet temperature (i.e., if the flow entering the servers is thermally uniform) and composed of a fraction $ψ¯$ of cold air supply and a controlled fraction $ϕ¯=1-ψ¯$ of hot air recirculation. We note here the special case corresponding to $ϕ¯=0$ (i.e., the case of no bypass at all). This is the case of the enclosed aisles where air is supplied by the CRACs at the maximum possible temperature—the redline temperature, T* . This, however, does not necessarily correspond to an optimal solution in view of the increase in fan power to pump the larger amount of air ($ψ¯=1$) against the high pressure drop in the CRAC heat exchangers and filters, as will be shown later after we have completed model formulation.

The cooling infrastructure power optimization problem reduces to finding the optimum CRAC exit temperature, Ta , and its conjugate CRAC flow rate, $m·C$ that will minimize PC for a given level of IT power and outdoor conditions. This optimum depends primarily, if not entirely, on the performance characteristics of the CRACs (or CRAHs and their chiller plant), as long as Ta and $m·C$ satisfy the redline constraint, $T¯mi≤T*$.

###### Cooling Infrastructure Power Consumption.

The cooling power consumption PC comprises two major components that account for the dominant part of the cooling infrastructure power consumption [1,25], a refrigeration component, PR , that is expected to be higher the lower the CRAC exit temperature, Ta , and the higher the outdoor temperature, Toa , and an air moving (fan/blower) component, PF , that is expected to vary with the CRAC air flow rate. In the case where not all of the air is delivered through the CRAC ($ψ¯<1$ ) (i.e., the proposed bypass recirculation) the recirculated air must be thoroughly mixed with the CRAC cold air supply to ensure uniformity; therefore, it may be necessary to introduce this recirculated air into the plenum using a low-lift fan to overcome the modest ΔP in the plenum (typically about 10% of that in the CRAC/CRAH for the same flow rate). The total fan power consumption is then Display Formula

$PF≈KCNCV·C3ηF+KBNB(ϕ¯V·m)3ηF$
(5)
in which $V·C=(1+λ)m·a/(1+λ)m·aρρ$ is the CRAC volume flow rate, $V·m$ is the rack volume flow rate, ηF is the overall efficiency of the fans, assumed here to be driven by controllable variable-speed motors, NC and NB are the number of CRAC units and bypass fans, and KC and KB are the CRAC and bypass pressure coefficients. Therefore Display Formula
$PC≈PF+PR(Ta,Toa,Q·C)$
(6)
where the CRAC cooling capacity, $Q·C$ is given by Display Formula
$Q·C=Pm+PF+PP$
(7)
where PP is the sum of other parasitic cooling loads due to lights, wall heat transfer, etc., which we will neglect in this analysis. The power consumption of the refrigeration system is given by Display Formula
$PR=Q·CCOP(8)$
(8)
where coefficient of performance (COP) is the refrigeration system’s overall coefficient of performance. For a vapor-compression refrigeration system, the COP can be represented reasonably well by an expression of the form Display Formula
$COP=f(Te,Tc;ηc)$
(9)
where Te and Tc are the evaporator and condenser saturation temperatures, respectively, and ηc is the compressor/motor overall efficiency. It is convenient to express the real COP as a fraction, εr , of the ideal Carnot COP at the same Te and Tc Display Formula
$COP(Te,Tc;ηc)=ɛr(Te,Tc;ηc)TeTc-Te$
(10)
in which Carnot efficiency εr ≤ ηc accounts for all irreversibilities in the a real vapor-compression cycle. For this analysis, εr was obtained by computing the performance of a vapor-compression refrigeration cycle, with 5 °C subcooling and 0Â °C of superheat, which uses R-134a as a working fluid. Irreversibilities in the compression process were considered by using a map of the isentropic efficiency of a typical single-screw compressor with a variable volume ratio as given in Ref. [26]. The computed variation in εr is shown in Fig. 2.

The saturated evaporator temperature, Te , must be lower than the cold air supply temperature (CRAC exit temperature), Ta , and the saturated condenser temperature, Tc , must be higher than the outdoor ambient temperature, Toa (i.e., dry-bulb or wet-bulb temperature depending on the type of heat rejection system). We would expect the temperature difference between Ta and Te to depend on the evaporator or CRAC heat exchanger design (flow arrangement, UA, etc.), air flow rate, and evaporator load, $Q·C$, and we would expect the difference between Tc and Toa to depend on condenser design (flow arrangement, UA, etc.), coolant (air or water) flow rate, and the condenser load, $Q·o$, given by Display Formula

$Q·o=Q·C[1COP+1]$
(11)
Further, because the blowers in typical CRACs/CRAHs are placed downstream of the heat exchanger, the CRAC/CRAH exit temperature, Ta , will be higher than the air temperature exiting the CRAC/CRAH heat exchanger (evaporator for a CRAC) owing to the CRAC/CRAH fan/motor heating effect, which can be significant in air-cooled data centers. Therefore, the CRAC heat exchanger exit temperature is given by Display Formula
$Tax=Ta-PF(1+λ)m·acp$
(12)
It is this temperature, not Ta , which must be used in determining the appropriate evaporator temperature to be used in COP calculations. This is an important distinction when the fan/motor power dissipation is significant, as it is in a data center application.

In the present analysis, to compute Te and Tc we employed the following simplified expressions derived from well-known heat exchanger effectiveness-number of transfer units relationships for heat exchangers with one of the two fluids changing phase (i.e., boiling or condensing) [27] Display Formula

$Te=T¯x-(T¯x-Tax)1-exp[-(UA)e(1+λ)m·acp]$
(13)
Display Formula
$Tc=Toa+Q·o/Q·om·ocpom·ocpo1-exp[-(UA)cm·ocp,o]$
(14)
Here, cpo and $m·o$ are the specific heat and mass flow rate of the condenser coolant and Toa is the outdoor ambient temperature. Equation 14 was further simplified by observing that for a constant $m·o$ and (UA)c , $Tc-Toa$ is directly proportional to $Q·o$, i.e. Display Formula
$Tc-Toa≈χQ·o=χQ·C(1+Tc-TeɛrTe)Tc-χQ·CɛrTcTe≈Toa+χQ·C(1-1/1ɛrɛr)Tc≈[Toa+χQ·C(1-1/1ɛrɛr)]/[Toa+χQ·C(1-1/1ɛrɛr)](1-χQ·CɛrTe)(1-χQ·CɛrTe)(15)$
(15)
in which we ignored the dependence of εr on Te and Tc , and used instead an average constant value of εr ≤ ηc , typically 0.4–0.6.

Naturally, a water-cooled condenser is usually coupled indirectly to the outdoor air dry-bulb and wet-bulb temperatures through a cooling tower and the values of the condenser parameter χ for air-cooled condensers may be a better choice for the combination of water-cooled condenser and a cooling tower, using the ambient wet-bulb temperature, rather than the ambient (outdoor) dry-bulb temperature for Toa .

## Enclosed Cold Aisle, Constant Server Flow

The above-described model was tested for a typical air-cooled data center with a 1024 kW IT load. Typical values were assumed for model parameters and system characteristics, as given in Table 1. A typical leakage fraction, λ, of 25% was selected in this example [7].

Figure 3 presents sample results for a case with constant server flow (i.e., a specified server temperature rise of 10 °C). The figure shows the cooling infrastructure power consumption and the conjugate CRAC exit temperature for a range of CRAC air supply fraction, $ψ¯$ (i.e., the fraction of air that passes through the CRAC and is cooled).

It can be seen that there exists an optimal value of $ψ¯=0.48$ and its conjugate cold air supply temperature $Ta*=18.3 °C$ (291.15 K) that minimizes the power consumption, subject to the redline constraint. It is evident that a higher CRAC exit temperature does not always lead to lower power consumption in air-cooled data centers, where CRAC fan power consumption is a significant contributor to the overall cooling infrastructure power. It is worth noting here that several studies of data center virtualization have considered only the refrigeration portion of the total power consumption [28-29] ignoring the equally important CRAC fan power consumption. This may be justified in situations in which the refrigeration power consumption is dominant, as it would be when inefficient refrigeration systems are employed in very warm climates. In most other situations exclusive focus on reducing refrigeration power consumption without regard to the power consumed in moving the cooling air would lead to suboptimum, possibly misleading results, unless the CRAC flow rate (fan speed) is constant.

Next, we can examine the practical arrangement for achieving a thermally uniform inlet temperature—enclosing the cold aisle. With an enclosed cold aisle, no recirculated air can be entrained into the supply air stream, and the perforated tiles within the enclosure must provide as much air as the servers need. This air will enter the servers at a uniform temperature as high as the redline temperature T* . This is the point corresponding to the extreme right ($ψ¯=1$ ) of Fig. 3. Yet, paradoxically, the power consumption at this point is much higher than that at the minimum of the power consumption curve, which corresponds to a significant degree of bypass $ψ¯≅0.48$. This apparent paradox stems from the fact that if all the flow issuing from the enclosed perforated tiles emanates from the CRACs, considerable fan power will be consumed to overcome the relatively high pressure drop across the CRAC heat exchangers and filters, which is both wasteful and unnecessary in view of the relatively high temperature at which air enters the servers (in this case T*  =  27 °C). Based on this example, the implementation of a CRAC bypass branch in conjunction with aisle containment shows the potential of reducing the power consumption by 60% (207 kW versus 530 kW) compared to a conventional enclosed aisle.

How then can the benefits of an enclosed aisle and its associated uniform temperature at server inlets be realized while blending in a large percentage of recirculated air? The solution resides in the provision of a CRAC bypass recirculation branch similar to that shown schematically at the extreme left of Fig. 1 and discussed in Ref. [30]. It may also be possible to introduce recirculated air directly into the cold aisle by means of induction louvers and low-lift fans, along with mixing fans to “homogenize” the air inside the enclosure. The bypass branch and its controls may also be integrated directly into the CRACs themselves. This study will investigate several configurations using computational fluid dynamics.

Enclosing the hot aisle would produce similar benefits but may also provide some practical advantages: recirculation bypass can be provided directly by installing a number of actively dampered tiles in the floor of the hot aisle. The tiles may be equipped with low-lift axial fans to overcome the slight pressure difference in the underfloor plenum. In such a case, the hot aisle must be ducted to the inlets of the CRACs. It is surmised that the bypass-recirculated air will thoroughly mix with the cold air from the CRAC inside the plenum before issuing from the cold aisle perforated tiles to enter the racks.

Figure 4 illustrates the breakdown of power consumption for the refrigeration system and the fans (CRAC  +  bypass). For the special case of a conventional enclosed aisle without CRAC bypass ($ψ¯=1.0$), the refrigeration power is relatively low (∼19% of IT power) owing to the relatively high supply air temperature (Ta  =  27.0 °C), which improves the refrigeration system’s COP considerably. In this case, the entire flow required by the racks plus leakage passes through the CRAC, resulting in a high CRAC fan power use (∼34% of IT power). It should be noted, however, that in practice, many data centers are typically supplied with air at much lower temperatures (in the range of 12–18 °C) and consequently have higher refrigeration power consumption.

While the results depicted in Fig. 3 are for servers with a 10 °C temperature rise, the effect of the server temperature rise, ΔTm , on optimized cooling power consumption and its conjugate CRAC air supply fraction are shown in Fig. 5. As the server temperature rise increases (server flow decreases for the same server power), the cooling power consumption decreases. However, the advantage of bypass recirculation diminishes as the server temperature rise increases and disappears entirely at ΔTm ≅ 21 °C in this example case. However, enclosing the aisles together with bypass recirculation (when beneficial) also reduces the dependence of the optimum cooling power on server temperature rise as evidenced by the relatively small savings in power consumption (<10%) as the server temperature rise is increased from 8.0 °C (281.15 K) to 32.0 °C (305.15) in Fig. 5.

Not only do the enclosed aisles with bypass recirculation make for lower cooling infrastructure power consumption but also they facilitate IT load placement through so-called thermally aware, energy-optimized virtualization (TEV). The formidable problem of optimal workload placement, which is receiving increased interest by the IT industry and is the subject of numerous investigations (e.g., Refs. [28], [29], and [(31)), becomes much simpler, almost trivial when the aisles are enclosed. This is because there is no preference among identical servers in an enclosed aisle based on the thermal environment, which will be uniform in this case, making for a wide range of equivalent load placement possibilities within the data center, or at least those racks in the data center that share enclosed cold aisles.

## Investigation of CRAC Bypass Implementation Using CFD

One of the main advantages of enclosed aisle configurations is the ability to provide thermally uniform conditions to the inlet of all servers. The Enclosed Cold Aisle, Constant Server Flow section showed that the energy savings potential of the bypass recirculation branch in conjunction with an enclosed cold aisle relies on the ability to thoroughly mix the bypassed recirculated air with the chilled air provided from the CRAC to the underfloor plenum before providing this air through the perforated tiles, as depicted in Fig. 1. As previously discussed, there are a number of ways this can be done in practice. In this work, we will use the commercially available CFD package FLUENT to investigate several configurations of CRAC bypass. In all cases, the flow is assumed incompressible and the Reynolds-averaged Naiver–Stokes equations are solved using the SIMPLE-C algorithm. Second-order-accurate upwind schemes are used to solve the momentum and energy equations and the standard scheme is used for pressure interpolation. The realizable k–ε turbulence model was used with standard wall functions. A structured grid was created using the commercial software GAMBIT , where the typical cell size was (2 in. × 2 in. × 2 in.). It is worth noting that the cell size used in this work is much finer than what is typically used in the industry. VanGilder and Zhang [33] recommended a cell size of 6 in. × 6 in. × 6 in. However, prior to performing this work, a grid sensitivity study was conducted. Grid sizes of 2 in. × 2 in. × 2 in., 4 in. × 4 in. × 4 in., and 6 in. × 6 in. × 6 in. were studied. Both, the 4 in. and 6 in. grid sizes exhibited high numerical diffusion compared to the 2 in. cell size. This added numerical diffusion had a significant effect on underpredicting mixing in the data center. To this end, the (2 in. × 2 in. × 2 in.) cell size was used.

Figure 6 shows an isometric view of the computational domain, which measures (8.8 m × 3.7 m × 7.0 m). The domain consists of a single enclosed cold aisle with 16 racks (note: the enclosure is not shown in the figure to allow for a visual of the perforated tiles). An enclosed aisle with a nonporous roof and side barriers is assumed, which forces the tile airflow to be equal to the required server airflow. One and a half CRAC units feed a 36 in. deep underfloor plenum. The underfloor plenum feeds cooling air to the raised floor space through perforated tiles (∼85%), cable cut-outs behind the racks (∼14%), and distributed leakage over the raised floor (∼2%). The racks are modeled as black-box devices with a specified mass flow rate and heat addition. This approach does not require the details of the servers in the rack be resolved. Perforated tiles, cable cut-outs, and the raised floor are modeled as porous media with appropriate resistance coefficients. The bypass tiles are modeled as a specified mass flow rate that is delivered vertically into the underfloor plenum. The CRAC returns are modeled as outflow boundaries and the CRAC supplies are modeled as velocity boundary conditions with a specified velocity, temperature, turbulence intensity, and turbulence length scale. A symmetry boundary condition is applied to the front and right sides of the domain.

Figures 6 and 6 show two possible configurations for bypassing a fraction of the air into the underfloor plenum. In both configurations, it is assumed that floor tiles with low-lift fans are available to push the bypassed air into the underfloor plenum and overcome the modest pressure drop. Configuration 1, as seen in Fig. 6, places the bypass tiles in front of the CRAC units; whereas, configuration 2, as seen in Fig. 6, the bypass tiles are placed in the hot aisle. In addition, the study will use configuration 1 to investigate two different CRAC flow arrangements, specifically a CRAC with an installed turning vane that discharges the flow horizontally in the plenum and a down-flow CRAC where the flow is discharged vertically into the plenum.

As an example, consider a data center based on typical high volume servers where each rack in the data center has a temperature rise of 18 °C and power consumption of 16 kW. Referring to Fig. 5, the optimum operation for a data center configured with these racks is to bypass ∼32% of the flow around the CRAC and to cool the 68% of the flow that enters the CRAC to a temperature of 20.25 °C (293.4 K). For this study, it is assumed that the bypassed flow is distributed equally among the bypass tiles. Since the temperature uniformity of the air provided to the cold aisle is a consequence of mixing in the underfloor plenum, it is important to prescribe appropriate turbulence boundary conditions (turbulent kinetic energy or turbulence intensity and dissipation or length scale) at the discharge of the CRAC units. To this end, measurements were performed on the discharge of a CRAC unit with an installed turning vane using an omnidirectional hot-wire anemometer to estimate an appropriate turbulence intensity. A profile at the exit of the turning vane showed that while the velocity was relatively uniform, the turbulence intensity varied significantly between 20% and 70%. For simplicity, a turbulence intensity level of 50% and a turbulence length scale of 0.07Dh are used at the discharge of the CRAC units and bypass tiles.

Figure 7 shows temperature contours 8 in. below the raised floor in the underfloor plenum for configuration 1, (bypass near CRAC unit) for both the horizontal flow (a) and vertical flow (b) CRAC arrangements. Figure 8 shows temperature contours for configuration 2 (bypass in the hot aisle) with a horizontal flow CRAC. An examination of the results shows that configuration 1 with the horizontal flow CRAC unit results in a high degree of mixing; a consequence of the bypass flow and CRAC flow being discharged in a cross-flow arrangement, which promotes mixing far from the perforated tiles. In the case of the down-flow CRAC, the two jets are discharged parallel to one another and result in two distinct airflow paths with limited mixing. Figure 8 shows that the hot aisle bypass also results in limited mixing, mainly due to the close proximity of the bypass tiles to the perforated tiles.

While the underfloor temperature distribution helps to understand the resulting flow pattern, the real judge of the effectiveness of any of the proposed arrangements is in how uniform the temperature entering the servers in the cold aisle is. Table 2 gives the maximum and minimum temperatures experienced at the inlet to the racks of IT equipment in the cold aisle for each of the cases described above. The results show that even though configuration 1, where the bypass tiles are placed near the CRAC with an installed turning vane, does not result in a completely uniform temperature entering the servers, it proves to be the best of the arrangements studied here with a difference of only 1.6 K between the maximum and minimum temperatures at the inlet to the IT equipment.

## Conclusions

The simple analysis presented in this paper provides a flexible and fast tool for exploring optimization possibilities in enclosed aisle air-cooled data centers, and can be used for identifying optimal, energy-efficient designs and operating scenarios. The methodology embodied in this simple analysis can be used in the early stages of the conceptual design process to define energy saving approaches and near-optimum design and operating parameters such as flow rates and air supply temperatures, as well as to carry-out trade-off investigations of cooling infrastructure sizing and performance characteristics. While this simple model is not the substitute for detailed, higher fidelity analysis and optimization studies, its most useful benefits stem from its ability to limit the range of options and parameters to be explored in the resource-intensive more rigorous optimization analyses.

Through this simple model, we were able to: (1) highlight the importance of considering both the fan and refrigeration power consumption when analyzing the data center’s infrastructure power consumption, (2) identify the energy-saving potential of bypass recirculation in enclosed aisle configurations, (3) define optimal operating strategies for enclosed aisle configurations, and (4) assess the effect of server temperature rise on energy optimization.

A practical bypass tile configuration was proposed in which tiles with low-lift fans are placed near the CRAC units in the raised floor to inject the bypass flow into the underfloor plenum. Using computational fluid dynamics, this configuration was studied and showed that while a perfectly thermally uniform inlet temperature to the enclosed aisle could not be achieved, the configuration does a fairly good job at promoting mixing of the bypass air and CRAC air in the underfloor plenum. It is expected that in the future other configurations could be proposed to improve the uniformity of the inlet temperature. It is anticipated that installing a bypass branch within the CRAC itself would be a preferred approach.

While the analysis presented in this paper was focused on active cooling infrastructure utilizing vapor-compression refrigeration (direct expansion or chiller), it can be easily extended to include economizer operation. When economizers are used in series or in parallel with, or in place of vapor-compression refrigeration, the trade-off between refrigeration and air moving power consumption is expected to shift toward a greater emphasis on reducing air moving power usage since the power for active refrigeration in this case will be greatly reduced, or may be eliminated altogether under suitable climatic conditions. While the analysis presented in this paper is for a given ambient outdoor air temperature, it can be easily extended to cover annual energy consumption through the use of the well-known temperature-bin method [33], or through integration with hourly weather data [34-35].

## Acknowledgements

The work reported in this paper was performed with financial sponsorship from the IBM Corporation and the Syracuse Center of Excellence in Environmental and Energy Systems. Professor Khalifa also performed some of the work while at the Technical University of Denmark as the Otto Monsted Guest Professor. The authors express their deepest gratitude to Dr. R. Schmidt and Dr. M. Iyengar of the IBM Corporation for their insightful suggestions and valuable input.

COP

coefficient of performance

CRAC

computer room air conditioner

CRAH

computer room air handler

cp

specific heat (kJ/kg K)

Dh

hydraulic diameter (m)

K

pressure coefficient (Pa/(m6 /s2 ))

$m·$

mass flow rate (kg/s)

N

number of servers

p

pressure (Pa)

P

power (kW)

$Q·$

T

Temperature (°C or K)

UA

heat exchanger conductance (W/K)

$V·$

volume flow rate (m3 /s)

χ

condenser thermal resistance (K/kW)

εr

COP Carnot ratio

η

efficiency

φ

recirculation fraction (φ  =  1  −  ψ)

λ

leakage fraction bypassing the racks

ρ

density (kg/m3 )

ψ

cold air supply fraction (capture index)

a

air (cooling air)

B

bypass

c

condenser

C

CRAC

e

evaporator

F

fan

i

inlet

m

mixed

max

maximum

oa

outdoor air

R

refrigeration

S

server

x

exhaust

xa

CRAC coil exit

o

reference; optimum

*

redline conditions

## References

View article in PDF format.

## Figures

Figure 7

Temperature (in K) contours 8 in. below the raised floor with bypass tiles placed near CRAC for (a) installed CRAC turning vane and (b) no CRAC turning vane

Figure 8

Temperature (in K) contours 8 in. below the raised floor with bypass tiles placed in the hot aisle with installed CRAC turning vane

Figure 3

Optimization results (power normalized W.R.T. IT power)

Figure 2

Computed εr for a vapor-compression refrigeration system using R134a

Figure 1

Simplified schematic

Figure 4

Component-by-component energy (power normalized W.R.T. IT power)

Figure 5

Effect of server temperature rise (power normalized W.R.T. IT power). Note: This shows the minimum power consumption point for each value of server temperature rise

Figure 6

Data center geometry: (a) computational domain, (b) configuration 1—Bypass near CRAC, and (c) configuration 2—Bypass in hot aisle

## Tables

Table 2
Comparison of cold aisle rack inlet temperatures
Table 1
Assumptions used in the analysis

## Discussions

Some tools below are only available to our subscribers or users with an online account.

### Related Content

Customize your page view by dragging and repositioning the boxes below.

Related Journal Articles
Related eBook Content
Topic Collections