Maurizio Salato, Vicor
The digital universe—a measure of all digital data created, replicated, and consumed in a year—will reach 40 zettabytes (40 x 1024 bytes) by the end of 2020—a 50-fold increase from 2010 . According to projections from Cisco Systems, annual Internet traffic alone will cross the 1 zettabyte mark by the end of 2015 . These trends are but two indicators of changes to the data environment that are driving scale in both medium- and large-size server applications.
Growing access, denser content, expanding sources
Consumer Internet traffic is representative of the unprecedented growth in worldwide data transmission. Cisco predicts a 23% compound annual growth rate (CAGR) for consumer IP traffic between 2012 and 2017, led primarily by users in the North America and Asia Pacific regions (see Figure 1). The growth rate of new users, however, is less than 5% in all regions worldwide (see Table 1).
Click image to enlarge
Figure 1: Annual consumer IP traffic worldwide is projected to grow at 23% CAGR—far faster than the new-user rate would support. Data source: Cisco Systems.
Clcik image to enlarge
Table 1: Market penetration of Internet access worldwide by region. Data source: United Nations.
Three key factors are contributing to rapid traffic growth far beyond the organic rate of new Internet users. Typical mobile phone users check their devices on average about 144 times per day, generating IP traffic 46% of the time . Thanks to public hotspots that offer essentially ubiquitous access, portable devices provide convenient 24/7 access to information, communication, and entertainment services to users outside the confines of office and home. Since 2009, portable devices accounted for a growing fraction of overall traffic—a trend that is accelerating and, on its current trajectory, will reach 30% by mid 2015 (see Table 2).
Click image to enlarge
Table 2: Mobile traffic as a percentage of total Internet traffic – 2009 – 2013. Data source StatCounter.
Content, too, has shifted from predominantly text-based to predominantly media-based. A picture may be worth 1,000 words but one image can easily occupy the data space of a half million words or more. Video is an ever-growing part of the data mix. YouTube uploads alone have grown from 20 hours of video per minute in 2009 to 100 hours per minute in 2013.
Content providers have made full television episodes and full-length films available online through a variety of anything-on-demand (XoD) portals. Most production facilities have switched over to high definition (HD) video format, which requires four to five times the bandwidth of standard definition (SD) video. Traffic for video clips and streaming television programming is expected to exceed web and Internet traffic in 2015 .
There are now more connected devices than there are people on earth and the device growth rate exceeds that of the human population. Cisco projects that there will be 6 billion devices in machine-to-machine (M2M) applications contributing to IP traffic by 2017.
Increasing server density
To support this unprecedented traffic growth, the worldwide average fixed broadband speed is increasing from 11 Mbps in 2012 to an expected 39 Mbps in 2017. Infrastructure projects like those supporting Google Fiber may result in fiber to the premises (FTTP) providing as much as 1 Gbps service.
In response, servers have adopted multi-core processors and increased the number of processors per board. Overall rack density has grown as well, from 7 servers per rack in 1996 to 20 servers per rack in 2010. Rack power has increased in kind, from kW/rack in 2000 to 10kW/rack in 2007 to beyond 20kW/rack for many new installations today.
For new facilities and those undergoing server upgrades, these trends have made it difficult to continue distributing power at rack level at 12V using single-phase AC-DC converters.
Challenges to 12V power distribution
Typical server racks based on 12 V power distribution use a power delivery unit (PDU) comprising EMI line filters and a Y-configured transformer with a 480V three-phase input and 277V single-phase output, which powers the rack’s AC-DC converters. To power a 10kW rack, the AC/DC converters, referred to colloquially as silver boxes, must combine to supply over 800 A to their IT loads. A schematic example is shown in Figure 2
Click image to enlarge
Figure 2: Schematic representation of high-density computing rack with single phase AC distribution and 12 V server motherboards
The silver boxes operate independently without synchronization, resulting in significantly richer harmonic content on their input-current waveforms. The converters’ power-factor correction (PFC) circuits correct input current waveform phase relative to the input voltage waveform, but as silver box manufacturers strive to increase power conversion efficiency, the harmonic content on the AC line increases .
For example, silver boxes that qualify for 80-Plus Gold certification deliver 92% peak efficiency and produce about 5% total harmonic distortion (THD) of their input-current waveform (with respect to fundamental frequency). AC/DC converters qualifying for 80-Plus Titanium certification deliver 96% peak efficiency but generate about 12% THD.
Moreover, silver boxes operate asynchronously, therefore the harmonic currents that are generated interact on the upstream AC line, and usually combine at the three-phase transformer within the PDU or the uninterruptable power source (UPS), generating an even wider spectrum of low and mid frequency harmonics (few Hz up to few kHz).
Recent studies [5, 6] have shown that when THD of the current waveform in a line transformer exceeds 5%, for every 2% THD increase an additional 1% total power losses are generated, typically in the PDU (or UPS, or both). For a 10kW system powering 80-Plus Titanium AC-DC converters, that corresponds to at least 350W dissipated in the PDU just due to current THD. System designers must size the PDU to accommodate the additional losses, increasing the installed cost of the rack, and affecting overall system reliability.
Eventually, as rack power continues to increase, 12V distribution starts to run into problems that are more fundamental. As individual motherboards add cores, memory, and I/O, the ability to subdivide power sources becomes limited and current maximums for practically and affordably sized bus bars and power-entry connectors negatively affect overall rack density. At 20kW/rack, 12V power shelves have to deliver a net 1.7kA, and rack power delivery requirements haven’t stopped there.
48 is the new 12
48V power distribution designs differ from 12V systems in important ways beyond the distribution scheme’s operating potential. Most notably, 48V distribution systems can replace the PDU transformer and silver boxes with a 400V/480V three-phase rectifier (a schematic example is shown in Figure 3. A modern rectifier produces about 3% THD and rarely exceeds 5% even under light loads. The reduced number of rectifiers (given the higher per-unit power) and the inherent lower harmonic content of the line currents they draw result in far lower current-waveform harmonics at system level.
Clcik image to enlarge
Figure 3: Schematic representation of high-density computing rack with 54 V DC distribution and integrated short-term backup
Server operators can take advantage of the economies of scale for existing 400V/480V three-phase AC-to-48V DC apparatus in use extensively for telecom and other existing 48 V applications. A typical 10kW unit requires only 2U (89 mm) of rack height and delivers conversion efficiency ≥ 97% and THD < 5%. By contrast, 400V/480V three-phase to 12V rectifiers are not practical, given the extremely high current output.
Conduction losses and practical limits on conductor size constrain the distance at which power can be economically transmitted (for 12V rack-based systems) to about 5kW. 48V power distribution using the same delivery infrastructure can deliver 20kW—enough to source a full server rack from a single three-phase rectifier.
A DC distribution strategy that eliminates single-phase AC in the rack also simplifies battery-backup implementations: battery stacks need not up convert through a UPS inverter that, in turn, drives AC-DC converters. Instead, 48V backup batteries can drive IT loads through a minimal control interface that manages switchover, battery charging, battery monitoring, and status reporting.
The growing shift toward 48V power distribution has system designers rethinking their board-power strategies. Several options exist but a few are simpler and smaller than those that 12V designs use. One example is Vicor’s 48V Intel VR12.5 compliant reference design, which can eliminate an interstitial conversion stage. Vicor’s approach avoids multi-phase conversion topologies, reducing component count, and allowing direct interface to sources (including backup) in the popular 36-60 V telecom voltage range. The component count reduction and smaller energy storage requirements allow designers to bring the power train closer to the processor, reducing losses and parasitic inductances that are roughly proportional to PCB-trace lengths.
For on-board loads other than processor and memory, single stage bucks are foreseen enabling 48 V distribution across the entire server board. With demand for increasing power density, thermal design is a growing concern. Packaging technologies such as Vicor’s Converter housed in Package (ChiP) platform are compatible with two-sided cooling, which can simplify the thermo-mechanical design.
Overall, 48V power distribution designs use less material than 12V schemes. They exhibit shorter bills of materials and require less copper at the rack level. With Vicor’s 48 V Intel VR12.5 compliant solution, they also avoid electrolytic capacitors. The net effect is higher reliability, better scalability, and greater power density.
Server farms have increased their power demand from 1 kW/rack in their early years to 20 kW today. Data traffic trends have accelerated and demands of 30 kW/rack are expected in the near future.
Practical limits on current magnitude and power delivery over a distance are forcing high-density server installations to switch from 12 V power distribution to 48 V designs. The switchover brings benefits that aren’t available to 12 V systems.
48 V power is particularly attractive when on-board converters are operating directly from 48 V. These topologies can reduce component count, energy storage, and losses while increasing reliability. New packaging technologies such as ChiP allow for two-sided cooling and simplify thermal designs.
48 V DC distribution provides high power density that can scale with foreseeable server deployments.
1. Digital Universe Study, IDC, Sponsored by EMC, December 2012.
2. Cisco Visual Networking Index: Forecast and Methodology, 2012-2017, Cisco Systems, May 2013.
3. Meeker, Mary and Liang Wu, Internet Trends D11 Conference (presentation), KPCB, May 29, 2013.
4. Surge in video will drive global data traffic to more than 60,000 Petabytes in 2016, ABIresearch, May 10, 2011.
5. Data Center power system harmonics: an overview of effects on data center efficiency and reliability, The Green Grid, 2013
6. The cost of Harmonic Losses and mitigations in distribution systems, 18th International Conference on Electricity Distribution, 6-9 June 2005