|
Changing Landscape Of Data Centers BY DONALD L. BEATY, P.E., FELLOW ASHRAE; DAVID QUIRK, P.E., MEMBER ASHRAE; JEFF JAWORSKI The first modern computers using integrated circuits were simple machines, and air-cooling was sufficient. As compute density increased, heat dissipation grew to be a concern for reliability and functionality, and air-cooling gave way to liquid-cooling. The invention of complementary metal-oxide semiconductors(CMOS) dramatically reduced heat dissipation, and the need for liquid-cooling all but vanished, for a time. But as the limits of compute performance and density are probed, designers continue to study thermal design and the art of thermal management to provide innovative and efficient ways to keep computers cool. 第一台使用集成电路的现代计算机还只是相对简单的机器，空气冷却已经可以满足散热的需求。随着计算密度的增加，散热量的增加也逐渐成为可靠性和功能性所需要关注的问题， 空气冷却（风冷）也逐渐被液冷取代。互补金属氧化物半导体的问世极大地减少了发热量，对于液冷的需求几乎消失了一段时间，但是随着计算机性能和计算密度的局限被探索，设计者继续研究热设计和热管理方法，以提供更创新有效的方式来保持计算机的冷却。 The takeaway from this history lesson should not be that thermal management is now well understood or time-tested, but rather that disruptive change has happened before, and may very likely happen again. 从这段历史经验教训所得出的结论并不是我们已经很好地掌握了热管理并经过了时间验证，而是颠覆性的技术革命以前已经发生过，而且很可能还会再发生。 Consider Figure 1. If a data center had been constructed in 1985 for the current technology, five years later that data center would have been grossly inadequate to handle the rapidly increasing heat production by the then-current technology. On the other hand, if a facility was built in 1990 expecting that the rate of production would only continue its 10-year trend of growth, the facility would have been grossly overcooled by 1995, after the introduction of CMOS-based processors. 通过图1可以看到，如果在1985年，为了适应那时的技术建造了一个数据中心，那么5年之后，这个数据中心将不足以处理5年后技术快速增长的所带来的热产出。相反地，如果在1990年，我们预计支持未来10年的增长趋势来配置基础设施，那么情况可能是配置的基础设施将会过度的冷却服务器，直到互补金属氧化物半导体的问世。 Therefore, when designing the HVAC for information technology equipment (ITE), it is prudent to consider designs and techniques to reduce the risk of obsoles-cence due to industry change of the transistors and related parts that make up servers. To do this properly requires at least a working knowledge of thermal management to understand how the IT equipment can affect the needs of the facility on the whole. 因此，在为IT设备设计通风空调的时候，需要重点考虑设计和技术，以降低由于服务器的关键部件比如三极管器以及相关部件的行业变化而导致的过时风险。要做到这一点，至少需要具备热管理的相关知识，以了解IT设备如何影响整个设施的需求。 Thermal Considerations for ITE Components Thermal management is the process by which a piece of ITE regulates itself to enable the delivery of reliable, consistent performance. Throughout the past 20 years (the CMOS era), the science of thermal management of the data center has evolved quite rapidly. This was due to the continuous increases in compute density and overall heat dissipation. To understand the whole picture, it’s best to start by understanding the component level of the ITE and abstract the implications from there. 基于IT设备元器件的热考虑 热管理是一种IT设备自我调节以实现可靠性和持续表现的过程。纵观过去的20年（CMOS时代），数据中心的热管理科学发展的非常迅速，这归功于计算密度以及总散热量的持续增长。为了理解整个过程，最好从理解IT设备的元器件级别开始，并从中得到一些启示。 Most ITE, such as servers and storage arrays, consists of similar components used to achieve different functions. These components typically include CPUs, memory, support logic, and storage. Each of these component groups has different behaviors as they relate to thermal design, and thus have different thermal specifications. It is these sets of thermal specifications that must be adhered to by the overall piece of ITE, which may greatly affect its design. 大部分的IT设备（如服务器和存储阵列），都由用于实现不同功能的类似组件组成。这些组件主要包括CPU，内存，支持组件和存储。每个组件群在不同的热设计里会有不同的表现，称之为热参数。整体的IT设备设计需要满足这些参数规格，这会极大的影响到最终的设计。 Each component may have different thermal specifications, but are typically characterized based on their: · Reliabilitylimit; · Damagelimit. 每个组件可能会有不同的热参数，但通常是基于它们的：
The reliability limit is the temperature at which the component will perform reliably in the long term, and thus is a target maximum for the component during normal operation. Short durations at or just above the reliability limit should not significantly impact reliability. 可靠性极限是部件长期可靠运行的温度，因此是部件正常运行期间的最大目标温度。短周期等于或者高于上限值不应该对可靠性造成严重的影响。 The functional limit is the temperature at which the component may cease to operate correctly, so exceeding this temperature is never recommended and can have unexpected results. Finally, the damage limit is the temperature at which permanent damage may occur. Therefore, never exceeding this limit is of paramount importance. 功能性极限是指组件可能停止正常工作的温度，因此不建议超过该温度，并且可能会产生意想不到的结果。最后，损伤极限是可能发生永久损伤的温度。因此，永远不要超过这个温度限制。 Understanding the limits of each component is a critical ingredient that influences the overall design of the ITE, since the overall reliability of the whole is at best only as good as the least reliable component. For example, it would be an ineffective ITE design if, during normal operation, a single component would rapidly approach its damage limit, while all of the other components were well within their reliability limits. 了解每个组件的限制是影响IT设备总体设计的一个关键因素，因为整体的可靠性必须建立在可靠的元器件基础上。如果在正常运行期间，单个部件会快速接近其损坏极限，即使所有其他部件都在其可靠性极限范围内，这也将是一种无效的设计。 Each of the three thermal limits is typically provided for most processors, whereas most memory specifies only a single temperature that is both the reliability and functional limit. Interestingly, for some DRAM memory, this limit may increase while functioning in an extended temperature range mode, which doubles the refresh rate to essentially halve the time that the data must be stored without corruption. This consumes some addi- tional energy, but may result in an overall decrease in energy by slowing a fan or allowin- g more efficient heat removal. 一般来说，大部分的处理器都会提供这三个热极限参数，而大部分的内存只会注明单一温度，它既是可靠性极限又是和功能性极限。有趣的是，对于某些DRAM内存，当在在扩展温度范围模式下工作时，此限制可能会增加，这会使刷新率翻倍，根本上将存储数据而不被损坏的时间减半。这会消耗一些额外的能量，但通过减慢风扇速度或允许更有效的散热来降低整体的能耗。 To design the ITE to efficiently address each of the components’ specifications, we must understand what drives the temperature of each component. Regardless of the cooling medium, each component is affected by the inlet temperature of the cooling medium, preheating of the medium from the inlet to the component, and the heat generated by the component itself. 要设计一个能够有效地满足每个组件的特殊参数的IT设备，我们必须理解是那些因素决定了每个元器件的温度限制。不管冷却介质是什么，每个元器件都会受到冷却介质的入口温度、从入口到部件的介质预热以及部件本身产生的热量的影响。 Since the preheating is due to the heat produced by other components in the ITE, the overall layout of components within the ITE is critical. However, there will be trade-offs associated with any layout optimization. For example, if the greatest sources of heat (CPU, memory) are distributed across the front of the inlet so they don’t have downstream impacts on each other, it may mean decreased available depth for full-sized PCIe cards in certain form factors. Figure 2 demonstrates this by comparing two possible layouts. 由于预热是IT设备中其他组件产生的热量所造成的，因此IT设备内组件的总体布局至关重要。但是，任何布局优化都会存在权衡。例如，如果最大的热源（CPU、内存）分布在入口的前面，这样它们就不会对彼此产生下游影响，这可能意味着在某些形状因素下，全尺寸PCIe卡的可用深度会降低。图2通过比较两种可能的布局来证明这一点。 Thermal limits and layout trade-offs are only two components of the overall design equation. In addition, each component may have various ways it can report its state, and also regulate its state. In particular, CPUs are the most tightly controlled components within the design. Whereas high performance and energy efficiency may sound like opposite needs, there have been increasing pressures to do both. This is often accomplished through the use of processor states, performance states, and thermal states. 热极限和布局权衡只是总体设计过程的两个方面，除此以外，每个组件可以有多种方式报告和调节其状态。特别是，CPU是设计中最严格控制的组件，尽管高性能和高能率听起来似乎是相反的需求，但兼容两者的需求越来越多。这通常通过使用处理器状态、性能状态和热状态来实现。 Processor states are essentially forms of sleep that the CPU can dip into for various durations to save energy and consequently regulate the amount of heat produced. While not sleeping, the CPU can also adjust its performance state, which typically adjusts voltage and frequency to regulate its throughput and heat. Finally, thermal states are essentially a time-out for the CPU, completely halting operation for brief periods of time only for the purpose of regulating the heat generated. 处理器状态本质上是CPU在不同的持续时间内进入休眠的各种形式，这样可以节省能耗，从而调节产生的热量。在不休眠的情况下，CPU还可以调整其性能状态，通常通过调整电压和频率来调节吞吐量和热量。最后，热状态本质上是CPU的超时，完全停止运行仅用于调节产生的热量。 Thermal Solutions for ITE Components Armed with a basic understanding of just some of the multitude of variables affecting the heating of ITE components, we can begin to discuss how they can be cooled. IT设备的热处理方案 大概了解了一些影响IT设备组件发热的变量，我们可以开始讨论冷却IT设备组建的方法了。 At their essence, all cooling techniques are about the removal of heat. In the simplest case, this may be just cool air absorbing heat from the surface of a component, and in the most complicated it may be a multistage removal process involving heat sinks, engineered fluids, and chilled water directly in the ITE chassis. But each method transfers heat from the ITE components into a medium that is then directed away from the IT equipment. 从本质上讲，所有的冷却技术都是关于散热的。最简单的，可以只是从部件表面吸收热量的冷空气，更复杂的，可能是一个涉及散热器、工程流体或者IT设备底盘中的冷冻水的多层级散热过程，但是每种方法都需要将来自IT设备组件的热量传递到冷却介质中，然后通过冷却介质把热量从IT设备中导出。 Air is the most obvious cooling medium since it is all around us. Thus, air-cooling has always been an attractive, affordable, and comparatively simple option. Take components, add a fan to move some air, and start cooling. 空气是最常见的冷却介质，因为它无处不在。因此风冷一直是一个很好的选择：可负担，相对简单。选定组件，通过增加风扇来带动空气进行冷却。 However, air does not have a very high specific heat, nor is it very dense. This, combined with air having a low thermal conductivity, means a lot of air must pass over a large surface area to remove a moderate amount of heat at a reasonable rate. To enable this, thermally conductive aluminum or copper heat sinks may be used to transfer the heat away from a hot component across a large surface area so it can be absorbed by passing air. 然而，空气没有很高的比热，密度也不高。利用低导热率的空气作为冷却介质，意味着需要大量的空气和较大的表面积，这样才能以合理的速度排出适量的热量。为了实现这一点，可使用导热好的铝或铜的散热器将热量从热部件转移到大面积的表面积上，以便空气散热。 To complicate matters further, the amount of energy required to power the fan(s) can grow quite rapidly when the volume of air must be large or if it must overcome a large static pressure. According to the fan laws that describe fan performance, airflow is proportional to fan speed, and power is proportional to fan speed cubed. Therefore, to increase the airflow of a given fan by two times might take an eight times increase in power. 更复杂的情况是，当风量需求很大或静压过大时，风扇的耗量会迅速增加。按照风扇性能的常规曲线，气流与风扇转速成正比，功率与风扇转速成正比。因此，对于指定的风扇，如果风量增加一倍，需要提供八倍的功率。 When the volume of air required and energy consumed to move it are too much to overcome, liquid cooling offers solutions. In contrast to air, liquids such as water are significantly denser and have a higher specific heat, which combine to produce a cooling capacity of around 4,000 times that of air by mass. In practical terms, it means much less water is needed to move the same amount of heat than air. 当需要的风量和由此带来的能耗太大而无法克服时，液体冷却给我们提供了解决方案。与空气相比，液体（如水）密度更高，具有更高的比热，这种物理特性使得相同质量的水的冷却能力约为空气的4000倍。这意味着冷却相同热量所需的水要比空气少得多。 Unsurprisingly, liquid cooling introduces its own set of unique challenges, such as adding additional weight and having the potential to leak. Each of the different approaches to liquid cooling presents further challenges and solutions. 不出所料，液体冷却也有其自身的特殊的挑战。例如，液体冷却会增加额外的重量，而且有潜在的泄漏问题。每种不同的液体冷却方法都提出了进一步的挑战和解决方案。 From a volumetric standpoint, the solution with the least liquid is a closed cooling loop within the ITE. In this scenario, the liquid is used to transport heat away from the components before being rejected to air in a water-to-air heat exchanger. This allows for efficient cooling of the components while not necessarily requiring design changes from a rack or facility perspective. 从体积的角度来看，使用最少液体的液冷方案在IT设备内是一个闭式的冷却回路。在这种情况下，液体作为热量的传输介质先将热量从组件中输送出去，然后在水-空气热交换器中把热量排给外部的空气。这样可以有效地冷却部件，而无需从机架或设备的角度进行设计更改。 Adding more liquid to the mix, another solution may use a cooling loop within the ITE that connects to an external technology cooling system (TCS) loop. Typically this loop is separate from the facility water, to which it transfers heat in a liquid-to-liquid heat exchanger. This is done because facility water and TCS water have different characteristics and needs, such as temperature range, water treatment, and pumping requirements. 另一种液冷方案会引入更多的液体，在使用IT设备内的冷却回路的同时，该回路会连接到外部冷却系统（TCS）回路与市政供水通过液体对液体的热交换从而排出IT设备的热量。通常情况下，该外部液体冷却回路与市政供水分离，因为市政供水和TCS回路用水具有不同的特性和需求，体现在水温范围，水压和水处理等方面。 A third option that incorporates even more liquid into the equation is immersion cooling, in which electronics are completely immersed in a dielectric fluid. While these arguably have the highest potential for a huge mess, they guarantee nearly 100% transfer of heat from the ITE to the cooling fluid as opposed to the facility air. Even more options exist that are hybrids or combinations of these techniques. 第三种方案是在冷却系统中引入大量的液体，即浸没式冷却。该冷却方式将电子元件完全浸没在电介质液体中，它无疑具有最大的散热潜能，该冷却架构保证了IT设备的热量全部被电介质液体吸收。还有更多的选择是这些技术的混合或组合。 ITE Thermal Management Overview Thermal management is the process by which a piece of ITE regulates itself, to enable the delivery of reliable, consistent performance. This means the thermal management of the system must optimize performance, efficiency, and even acoustics in the process of keeping the component temperatures within their limits. IT设备热管理概述 热管理是一个IT设备部件自我调节的过程，以实现可靠、一致的性能。这意味着系统的热管理必须在保持组件温度在其极限范围内的基础上，优化性能、效率，甚至噪音。 The general thermal management process begins by collecting sensor data from throughout the ITE. This may include temperature, power, fan speed, air pressure, and other system activities. This data comes from a variety of sensors that are included in CPUs, RAM, power supplies, hard drives, GPUs, PCIe cards, physical chassis (i.e., at the air inlet), etc., shown in Figure 3. 一般的热管理过程从整个IT设备收集传感器数据开始。这可能包括温度、功率、风扇转速、空气压力和其他系统活动。这些数据来自各种传感器，这些传感器包括在CPU、RAM、电源、硬盘、GPU、PCIe卡、物理机箱（即进风口）等中，如图3所示。 The baseboard management controller (BMC) collects this sensor data over standardized bus protocols, and uses algorithms to generate system responses in the form of fan speeds, CPU performance states, and other power settings. Based on the components within the ITE, these algorithms can be quite different, as the primary heat drivers may vary greatly, as can the locations and availability of sensors. These algorithms are sometimes further affected by boot options or system settings that allow the customization of the ITE to its specific application and environmental conditions. 基板管理控制器（BMC）通过标准化总线协议收集传感器数据，并使用算法以风扇速度、CPU性能状态和其他电源设置的形式生成系统响应。对于ITE中的不同组件，这些算法可能非常不同，因为主要的热驱动因素可能会有很大的变化，传感器的位置和可用性也会有很大的变化。算法有时也会受到引导选项或客户定制化系统设置的影响。 Although it can be defined somewhat easily, thermal management turns out to be an incredibly complex optimization problem in that it isn’t just a single optimization. Rather it is a continuous optimization across a wide spectrum of conditions that can include fan failures, extreme CPU use, swappable components, and various environmental conditions ranging from overcooled to undercooled to facility cooling failure. 虽然在某种程度上，我们可以很容易地定义热管理，但它是一个极其复杂的优化问题，因为它不仅仅是一个单一的优化。相反的，热管理是一种跨越各种条件的连续优化，包括风扇故障、CPU极端使用、可更换组件以及从过冷到欠冷到冷却失效等各种环境条件。 Closing Comments From the smallest component to the largest ITE chassis, thermal design plays an incredibly important role and influences design throughout the data center industry. It is not a straightforward problem of keeping equipment cool, but rather an optimization of keeping equipment within acceptable temperatures while enabling the right blend of performance and efficiency. 总结 从最小的组件到最大的IT设备箱，热设计发挥着极其重要的作用也同时影响了整个数据中心行业。这不仅仅是简单的如何保持设备冷却，而是在设备可接受的温度范围内，实现性能和效率的优化组合。 For the HVAC design engineer, the most important focus is the providing the right “entering” thermal conditions (air or liquid) to the ITE. For the ITE manufacturers, it is an optimization problem involving a lot of different pieces of equipment and trade-offs. 对于暖通空调设计工程师来说，最重要的关注点是为IT设备提供正确的“准入”热条件（空气或液体）。对于IT设备制造商来说，这是一个涉及大量不同设备组件和权衡的优化问题。 As the data center industry rapidly evolves based on changing need and advancing technology, thermal design must constantly adapt to these changing conditions. In fact, across the last 50 years, the biggest constant within the industry has been change itself. 随着需求的不断变化和技术的飞速发展，数据中心行业也在不断的进步，热设计必须不断适应这些发展。事实上，在过去的50年里，行业内最大的矛盾就是变革本身。 Will heat dissipation continue to grow upward? Or will another new technology revolutionize the CPU or memory, dramatically decreasing heat dissipation? 散热量会继续向上增长吗？或者另一种新技术会彻底改变CPU或内存，显著减少散热？ For the HVAC design engineer, these are challenging questions to answer. So regardless of the speculation, one safe bet is to assume that some form of scalability is necessary in the HVAC system design. 对于暖通空调设计工程师来说，这些问题是很有挑战性的。因此，不管猜测如何，可扩展性在暖通空调系统设计中是必要的。 翻译： 周婷 施耐德信息技术有限公司 产品经理 DKV（Deep Knowledge Volunteer)计划精英成员 编辑： 李擎 北京欣盛云路科技有限公司 高级运营经理 公众号声明： |
|
|
