Like other people, I’ve been wondering how Oracle has integrated many of Sun’s technologies and hardware after buying that company. Oracle has just started a nationwide tour to inform us of just that. The first stop was in their neck of the woods, Palo Alto, California. The free seminar information, titled "Share the Vision: Build aMore Efficient and Powerful Datacenter with Oracle,” is found here, along with the agenda and locations.

Now that Oracle has Sun’s hardware, Solaris operating system, virtualization engine, and other components, they can provide well-integrated solutions for data centers, as this slide shows:

The plan is to increase server and storage performance by several times and even tens of times more in a few years.

 Server Performance Increase

 Storage Performance Increase

 In addition to Solaris, they have Red Hat–compatible Oracle Enterprise Linux (OEL) and support. Their virtualization is implemented through Oracle’s version of Xen software, Oracle VM. On top of Xen, Oracle added management and other software to make it a comprehensive offering for virtualization. The features of Virtual Iron, which Oracle acquired some time ago, are being integrated into Oracle VM. Both Oracle VM and Virtual Iron are based on Xen, so the integration should not be too hard.

 A presentation on cloud computing was a good tutorial. Many of Oracle’s customers are enterprises, which tend to use VMware’s virtualization solution. Oracle provides a feature to translate VMware VM file formats to theirs. Currently, private or on-premise clouds are implemented mostly with VMware, and public clouds (AWS and Rackspace) are implemented with Xen. Eucalyptus’s enterprise version has a feature that is the reverse of what Oracle VM does. It can translate VMs by VMware on-premise to AWS file formats to allow them to be transported to AWS public cloud (known as CloudBurst).

 Oracle view on Enterprise Evolution to Cloud

 Since Oracle now owns microprocessors, server and storage hardware, operating systems, virtualization engines, databases, and applications, they can fine-tune the entire system to make it very efficient and execute very fast. Although Oracle is still small compared with IBM in terms of revenue (Oracle $27B vs. IBM $96B), Oracle is now in a position to compete with IBM.

	- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
		Energy Efficient Cooling for Data Centers: A Close-Coupled Row Solution
	- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

 The seminar was well run and presented very useful information. If there was one thing I did not like about it, it was the lack of discussion of energy efficiency. Energy efficiency and green IT initiatives were mentioned several times during the day, but they were not discussed in detail. I asked one of the speakers to share some energy efficiency data with me. If and when I get it, I will publish it here.

When I started this blog about two years ago, there was a lot of discussion of direct current (DC) power distribution at data centers. It is very easy to see why DC is better than alternative current (AC) power distribution. AC power enters your data center from your utility. It then goes to a set of UPS that take in AC but convert the power to DC. This is because the power goes through a set of batteries (which only takes DC). Then, at the other side of the UPS, it is converted back to AC. Then the power is distributed to IT equipment via PDUs. This AC power is then converted again by the IT equipment for its internal use. Each conversion loses some percentage of power.

 If the AC power entering the data center is converted only once to DC power and distributed to IT equipment that takes DC as input, there would be no conversion loss. Unfortunately, a few years ago, Green Grid put out a white paper comparing the power distribution of DC and AC and concluding that there were very few differences between the two configurations. After reading it, I moved my focus away from power distribution. Now I am back on this subject because some friends who started a company in that area and Keizo Hoshijima of NTT Facilities pitched me the merits of DC distribution recently.

 I cannot forget another DC power distribution advocate, Dennis Symanski of EPRI. I met Dennis when he was a panelist in my Nordic Green panel session.

 Dennis Symanski

 At the conference, he gave a presentation on DC power distribution. Cooling is known to consume about 30–60% of power in a data center, but we seem to be getting a handle on that. Once cooling and other culprits are under control, then we can pay attention to power distribution to further cut power consumption.

 Recently, I had a chance to visit Dennis at EPRI to chat about the current status of DC power distribution at data centers. The following is an edited summary of our conversation.

Q: What do you do at EPRI, and what is your background?

A: My focus is to make any computer equipment energy efficient (EE). Prior to coming to EPRI, I spent 18 years at Sun working on international standards and regulations. EPRI is funded by utilities but is an independent nonprofit research organization. It strives to "do good for society,” and it is quite refreshing for me when I do not have to pay attention to revenues and stock prices.

	- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
		Energy Impact of Increased Server Inlet Temperature
	- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

Q: So what’s new in the DC power distribution area?

A: In 2006, we conducted a set of experiments on the use of DC power distribution at Sun (currently Oracle), along with LBNL, Ecos Consulting, and other vendors. Our conclusion was that DC power distribution increases data center EE and reliability. Since the experimentation, we formed an organization called DC Power Partners to further our efforts. The members include LBNL, EPRI, EMC, IBM, HP, Oracle (former Sun side), Intel, and even APC. (ZK; APC had a different opinion of DC power distribution before.) We have a once-a-month teleconference to discuss new technologies and installations to share information. In addition to the U.S. companies, some European and Japanese companies call in.

Q: You only cover technology aspects at your DC Power Partners?

A: EMerge Alliance works on DC lighting, led by Armstrong Ceiling for 24 VDC. DC Power Partners is joining the alliance. They can take care of sales and marketing, trade shows, and web promotion.

Q: Are there any major changes in the DC power distribution area?

A: There are no drastic changes. We are currently working to make sure the components that comprise DC power distribution comply with UL and/or FCC regulations. For example, connectors that are used for DC power distribution.

Q: I have no issue with the technology. What about the market? Which region of the world is more receptive to this technology?

A: In general, the telecom industry is keen on picking this up. Companies like AT&T, Verizon, and NTT. In Europe, the European Telecommunications Standards Institute (ETSI) has prepared a specification for 380 VDC. There is at least one CO in each city and dozens in large cities, so there are 10’s of thousands of COs in the U.S.

Q: But IT-centric data centers and COs are different, and the IT data centers may not be so enthusiastic about DC power distribution.

A: They used to be quite different, but, these days, they are becoming increasingly similar as they share the same kind of IT equipment. Look at AT&T, which needs to process IP traffic coming from iPhones and iPads.

Q: A couple of years ago, I read a white paper from Green Grid (GG) that said there were few differences between AC and DC power distribution. What is your take on that?

A: GG members do not include any direct current facilities equipment suppliers. Several of the GG IT equipment suppliers are researching direct current power supplies for their equipment.

Q: What do server vendors need to do to support DC power distribution?

A: All they have to do is to swap the AC power supply with the DC one. The direct current power supplies have a smaller component count, are more efficient and as a result, research demos may show them to be more reliable. It is not hard to create a data center solely with DC power distribution. IBM created such a data center at Syracuse University.

Q: Is there any data center with a solely DC power distribution system in the San Francisco Bay area?

A: Several are going in parallel. But one in the Bay Area will be announced in November.

Q: Right now, generated power is carried via the AC transmission system. If DC power enters a data center, we do not need to convert power at all. Can you transmit power via DC?

A: In the days of Edison vs. Westinghouse, we could not transmit DC power at 5,000 V or higher. These days the high DC voltage is transmitted over a long distance. A hydropower plant in northern Quebec transmits generated DC power to New England over several hundred miles, in addition to much-closer Toronto and Montreal. (ZK: High-voltage direct current is a technology for transmitting high voltage DC over thousands of miles)

Q: Is there anything else I should know about what EPRI is doing in the DC area?

A: We talked about the power supply for IT equipment. Those power supplies were very inefficient. The conversion rate was only 65% on the average. 80 Plus is an organization that promotes increasing the conversion rate to at least 80%. EPA adopted their specification, and EPRI is running a test lab for 80 Plus.

I got very useful information talking with Dennis. EPRI covers a large area of energy and its efficiency. I have not heard much about DC power distribution recently, but that does not mean efforts on its behalf were terminated. On the contrary, they are going very strong, including DC for home and power transmission. I will report on this subject from time to time in this blog.

I have done consulting work for MySQL as it enters the Japanese market for more than three years. I have not seen Marten Mickos for a few years. I wondered what his next move would be after he left MySQL. I heard he became CEO of Eucalyptus, which is gaining visibility in the crowded cloud computing market. I asked him to talk to me about his new venture. In spite of his busy schedule, he was nice enough to sit down with me at breakfast and share his views on Eucalyptus and cloud computing. I usually take my own pictures for this blog, but I forgot to do so in the excitement of talking with him. So the official picture from Eucalyptus is a little too formal for my blog. But, oh, what the heck!

Marten Mickos

The following is a summary of our talk.

Q: Correct me if I’m wrong: Eucalyptus provides a set of open source tools and utilities to create on-premise clouds. All the components are either from Eucalyptus or some other open source. In addition, it has a hook to pick different types of hypervisors and to run on major Linux distributions like Ubuntu and CentOS. It is largely compatible with the Amazon Web Services (AWS) environment. You can move your virtual machines (VMs) between Eucalyptus and AWS clouds.

A: Basically, you are right. We provide a platform for building on-premise cloud.

Our enterprise version supports multiple hypervisors, such as Xen, KVM, and VMware.

Q: Can you name your customers for the enterprise product? Is the pricing based on number of CPUs?

A: We have several customers that are large enterprises and government agencies. I cannot reveal their identities yet, because I need their consent. Licensing is based on cores and we charge about $300 per core.

Q: What is an on-premise cloud, and what is the difference between on-premise and private clouds? In the past, you were looking for a better term than on-premise cloud. Did you find one?

A: An on-premise cloud runs on your hardware infrastructure at your site. A private cloud is a cloud reserved for the use of one single organization. In many cases, if not all, you can use the two terms interchangeably.

Q: What is your license for the open source side?

A: It is GPL version 3.

Q: Who develops code and commits it to the main trunk? Who has the copyright of the code? Do you take others’ contributions as well?

A: We develop our code and have the copyright of the code but take in others’ contributions as well.

Q: So this is somewhat similar to what you did at MySQL?

A: Yes. But the difference is that MySQL was disruptive in the old database market, but Eucalyptus is innovative in the new market of cloud computing.

Q: You are a big proponent of open source, and you like what Rackspace recently did? I mean that it created OpenStack for clouds. NASA’s Nebula project uses Eucalyptus, and they contributed their code to the OpenStack project. Did they include Eucalyptus in OpenStack as well?

	- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
		Preventing Data Corruption in the Event of an Extended Power Outage
	- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

A: Yes, I like what Rackspace did. I do not think they included Eucalyptus, because our license is GPL and OpenStack is by Apache 2.0. With open source, users can naturally take your code, use it, modify it and redistribute it. In the cloud market there are new projects and companies being launched all the time.

Q: If everyone can do it, what you are doing? What are your differentiation and/or advantage points?

A: We see it as a benefit, not a problem, that open source gives users many freedoms. Already today we have tens of thousands of users who do not need our commercial support. But the most mission-critical installations are dependent on our support, bug fixes and further development of the product. Uniquely in this market at this time, we have a scalable, mature and functional product.

Q: Amazon uses Xen open source, yet has its own APIs and file format for VM. So open source is closed by a proprietary container and is no longer open source. What do you think of it?

A: Again I think it demonstrates the power of open source. Open source is needed in all major infrastructure deployments today. What Amazon is doing is perfectly within the licensing terms and principles of open source.

Q: What is Dr. Rich Wolski (the founder of Eucalyptus) like? Is he a typical researcher?

A: Not at all. The typical researcher does not produce very useful things like Eucalyptus. Also the typical researcher does not start a new business.

Q: Is cloud computing more energy efficient?

A: Yes it is more energy efficient. But we must remember that certain computation always requires a certain amount of energy. It is the fact that you have higher utilization rates in cloud computing and you can turn off servers that are idle for some time to save energy.

Q: What about the opposite of that? What if more demands come to you beyond your capacity? Do you load-balance among several different clouds? If so, what are the problems in doing that?

A: Yes, the vision of cloud computing includes cloud bursting, which allows you to move load dynamically from one cloud to another. Technically it is an advanced proposition and we are not there yet for the general public. There are intricate challenges around security, authentication and latency that the industry is working on.

Q: What geographical markets are you addressing now?

A: Four regions are very important: the U.S., the E.U., China, and Japan.

Q: Are you doing anything in those areas, other than the U.S.?

A: Eventually, we will be in all of the areas, but right now we have our hands full with the U.S.

Q: We are sitting and talking here in Silicon Valley. Do you plan to move Eucalyptus to this area?

A: In the days of cloud computing, the physical location of headquarters does not make a difference. The headquarters is where the CEO is.

Q: So it is like Air Force One. When the president gets on the plane, that plane becomes Air Force One.

A: Exactly. Now this cafe is Eucalyptus One, and I can make all the executive decisions from here! In reality, of course, a company makes decisions in many locations and by many executives. At Eucalyptus we work as a team and for this analogy to be perfect, we need to consider the locations of every employee of the company.

Q: One thing I remember about you, during your days at MySQL, is that you came up with good analogies. You said that regardless of which seating area (first class, business class, or economy class) in an airplane you are sitting in, everyone gets to the destination at the same time. This was in reference to MySQL vs. Oracle databases. Do you have a good one for cloud computing and/or Eucalyptus yet? That kind of description grabs people’s attention quickly and easily.

A: I am thinking about it but have not come up with anything yet.

I had not seen him for a while, but Marten remains as he was before, very nice and friendly. As I researched into Eucalyptus, I found many similarities with MySQL. They are both based on open source, but the business model has a commercial version as well. MySQL was run as a virtual company, which, come to think of it, is a cloud in a sense.

Cloud people tend to ignore the underlying infrastructure; data centers. Marten has a good sense of what is below the cloud. When he talked about turning off servers in the case of low loads, I thought about Power Assure, which has a feature to turn servers on and off as needed. If you can synchronize the operation of IT and facility (cooling and power delivery) equipment according to the loads you receive, you could save more energy at your data center.

There are a few technologies to make networking more energy efficient. But a more energy efficient Ethernet would have by far the biggest impact because of its ubiquity.

I visited Broadcom to meet with David Berry, senior marketing manager, and Wael William Diab, technical director, office of the CTO. David and I have talked about the general energy efficiency of networking equipment before. Wael is vice chair of the IEEE 802.3 Working Group

So I focused on finding out what 802.3az is all about. My core technology area started with software engineering and expanded to cover networking and embedded systems. I am by no means an expert in networking technologies but am dangerous enough to know some of the basics. I here confess that while I was summarizing my conversations with these two gentlemen, I studied and researched this subject extensively so I could keep up. If the whole thing is completely over my head, it is easy to let it go, but with some effort, I can understand the subject matter to some extent. Sigh.　　

Wael Diab and David Berry

The following is an edited version of my questions and their answers.

Q: My understanding is that energy efficient Ethernet (EEE) allocates networking resources when needed and turns them off when not needed. Can you give me a little bit more on EEE?

	- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
		Power and Cooling for VoIP and IP Telephony Applications
	- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

A: Traditionally, the most important things about networking were speed and cost. But energy consumption is becoming more important than either of those because opex is beginning to surpass capex. The Ethernet has typically been operated as always-on, even though traffic is not always on. A typical utilization is less than 10% and sometimes even less than 1%. But when traffic is back on, it tends to hit the peak, and the Ethernet has to quickly come back on at 100% capacity. When the PHYsical layer (layer 1 on the OSI model) is on, the layers above (layer 2, data link, and up) are on as well. When there is no traffic, the PHYsical layer (PHY) can be turned off, along with the layers on top of it, creating further energy efficiency. Additionally, we have implemented a way to negotiate when to turn on layer 2. In this way, layer 2 can sleep until absolutely necessary.

Q: Is the ratification of 802.3az done? Is the product based on the standard shipping?

A: The standard is at the final stage, and ratification is expected in September. However, prestandard versions are shipping by vendors, as has happened for other networking products before.

Q: Can this technology be implemented with firmware refresh?

A: Unfortunately, to support firmware refresh, a new PHY is required and, therefore, new equipment. We have a technology called AutoGrEEEn. AutoGrEEEn technology enables a device with a non-EEE MAC to seamlessly transition to EEE capability by implementing control policy assist engines and circuitry inside the PHY device.

EEE requires control for the PHY to be done via in-band signaling over the MAC/PHY interface. This requires a change to both the PHY and MAC silicon. A number of systems have the MAC and PHY as two different pieces of silicon with the MAC often embedded in a switching or controller-type device. These MAC-containing devices have associated drivers and software, and are often multiport devices. So, a transition to EEE may be hampered by additional development that involves replacement of the MAC-containing devices. AutoGrEEEn technology eliminates the need to change the MAC/PHY interface on the MAC silicon, and allows for rapid transition today with legacy non-EEE MAC silicon attached to AutoGrEEEn-enabled PHYs.

Q: Servers are refreshed every three to five years at data centers. How about networking equipment?

A: In general, the networking equipment refresh cycle is probably longer than that of servers. However, it is important to consider networking in conjunction with servers and other IT equipment. Servers get more powerful, with new chips every 18 to 24 months. For example, a newer server with more computing power may require 48 ports rather than 24 ports in the networking equipment to process more loads. Thus, the refresh cycle may not be that much longer than that of servers. In addition, new builds are happening all the time and they can take advantage of any new innovations in energy savings.

Q: I think 802.3az is for 10 Mbps, 100 Mbps, 1 Gbps, and 10 Gbps. Does it apply to 40G and 100G as well?

A: For 40G and 100G, there is another standard called 802.3ba, which was recently ratified.
Both 802.3az (EEE) and 802.3ba (HSE) started about the same time. EEE looked at copper
interfaces that already existed ton be enhanced for energy efficiency. By the way, the
energy efficiency technology we use is called
low power idle . When it is idle, it sends little energy and is very energy efficient.

To achieve this both sides go into a suspended state, effectively turning the PHYs off. To keep both sides synced at the PHYsical layer a refresh signal is occasionally sent. Because the refresh signal has a low duty cycle, it is very energy efficient. To leave the LPI state, either side can send a wake-up signal and the other side reacts to that.

In addition to PHYsical layer savings additional subsystems above the PHY can be turned off. Broadcom championed the development of a technology called layer 2 data link LLDP stateful negotiation for enhanced saving modes in the standard.

Q: Some people insist that saving on the networking side does not make a huge impact on data center power consumption. Some studies indicate that power consumed by networking equipment at a data center is only several percents compared with those of servers and storage equipment. What do you say to that?

A: It is true that the networking equipment by itself does not consume a lot of power from the perspective of a few single ports. However, this energy efficient technology (EEE) for networking enables energy efficiency across the entire data center which typically has 1000’s or more ports. Enabling EEE is like turning off a bunch of leaky faucets. One might not notice the savings at a single point in time but it will certainly show up in the monthly bill. Ethernet has also been proven to be an extremely scalable technology enabling port consolidation during bandwidth migrations cutting total power consumption (1G to 10G for example) while at that same time increasing performance typically by a factor of 10. In a sense, Ethernet networking is playing a much larger role in turning a data center into the next-generation energy efficient one.

Think of this analogy. When you try to establish a high quality of service (QOS), you may want to have the fastest server with the fastest CPUs in it. However, if you study this carefully, you would find out that the fastest CPUs alone are not the answer. It may be established with fast CPUs, memory, disk, and other components combined. Energy consumption at a data center can be considered in the same way. Networking connects IT equipment together and, by activating and deactivating according to needs, networking controls other IT resources (IT equipment). This energy efficient technology in networking should not be considered just for networking equipment but for all the IT resources and, consequently, the entire data center.

After talking with Wael and David, I pondered the following. My discussion with them reminds me of the argument about whether servers or networking are more important in computing. It is true that servers are the center of computing, and we tend to pay attention only to servers. However, networking ties together servers and other equipment, like storage, to create an IT system. Without fast, low-cost, and energy efficient networking equipment, even the fastest and most energy efficient servers could not deliver, alone, the most energy efficient computing for a data center. This means networking equipment needs to be energy efficient. On top of that, networking equipment should be able to allocate resources dynamically, as needed, and without delay, to ensure the energy efficiency of other IT resources.

When the IT energy efficiency discussion was in its infancy a few years ago, it was O.K. to discuss each category of IT equipment in a data center separately to see how much it consumed in order to control its consumption. As IT equipment is being integrated and coming to function as one system, it is time to consider the energy efficiency of the IT system as a whole.

There are a few more technologies for making networking equipment more energy efficient, but I will cover them sometime in the future.

I reported that container-based data centers are gaining some attention these days. At the recent DatacenterDynamics in San Francisco, Andreas Zoll of i/o Data Centers gave a talk on the modularization of data centers. What is the relationship between modularization and containers? Modularization is a broader category than containers; a container is one example of modularization. Actually, data centers have been constructed with customized components because each data center is different. As I tour more data centers, I find that is true for somewhat old ones. But at the same time, newer constructions attempt to use standardized components. Standardized components could allow the whole data center to be constructed cheaper because they could become a commodity, and the entire data center structure could be standardized, regardless of location and requirements.

Andreas Zoll

Zoll started his talk by giving the current status of data centers, which was not surprising to those of us who keep track of their trends and challenges. These are some of his key points:

• Continuing growth of densities (for computing, power, and cooling)
• Cooling requirements and focus
• High capex and opex
• Right-sizing hard

Container data centers could address most of those trends and challenges. He made these points:

• Rapid scalability
• Capex in line with actual demand
• Geographically agnostic
• Higher efficiency
• Repeatable
• Flexible

One of his slides shows the history of container-based data centers:

Container-based data center progression

He defined three generations of container data centers.

Generation 1:

• Box containing servers
• Vendor lock-in
• Niche audience only

Generation 2:

• Variety of cooling options
• Vendor-neutral configurations
• Customizable
• Not self-contained but requiring several vendors

Generation 3 (next generation that is not yet on the market):

• Purpose-built
• Open architecture
• Fully integrated
• Tailored for redundancy
• Maintainable
• Location agnostic
• Automated
• Internationally plug-and-play

	- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
		Data Center Projects: Project Management
	- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

Some people asked him to elaborate on his last point about the second generation and full integration in the third generation. The current generation of container comes either with IT equipment that has some power and cooling inlets or with power and cooling equipment. HP (IT equipment) teamed up with Active Power (power) to provide a whole solution. Dell’s version comes as a double-decker: the bottom is IT equipment storage and the top is a power and cooling box.

Regardless of configuration, power and cooling should be available as a modularized unit to hook the container up to the building. This reminds me of the fourth generation data center proposed by Microsoft.

What you can do with each container is limited. You still have to hook your container up to power and cooling before you put it into operation. So the entire data center should be designed to provide modularized power and cooling. How soon can you do that? It is a good idea, but I wonder how doable it is now.

While people were buzzing with this question, a VP came on stage and said i/o Data Centers would announce their latest container-based solution in two to three weeks. I suppose it will be the third generation. They planned the session well. They well knew that people would ask what "fully integrated” meant and that they would announce their new solution. That was very clever. We will see if their new solution is as good as their session planning.

After the departure of Andrew Fanara from EPA, a new person is in charge of Energy Star for data centers. She is Una Song.

She gave a talk on EPA’s efforts on Energy Star at the recent DatacenterDynamics in San Francisco.

Una Song

It was a good summary of Energy Star for data centers. Here’s a set of videos from her presentation. There was a little surprise during her talk. One of the participants started questioning why power generated with renewable energies is not counted in the computation of PUE. His claim is that power generated by clean energy should be given some credit. In an extreme argument, he says PUE could be 50, yet all the power fueling the data center might be from renewables. The absence of GHG emissions in that case should be commended rather than the bad PUE frowned upon. That interaction is captured toward the end of videos 2 and 3. This reminds me of the metric (called Data Center Performance per Energy, or DPPE) proposed by Japan’s Green IT Promotion Council.

	- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
		Energy Efficient Cooling for Data Centers: A Close-Coupled Row Solution
	- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

This metric takes the use of renewables into account. To refresh your memory, here is the definition of DPPE:

DPPE =

(IT equipment utilization * IT equipment ability)

/ (total power consumed – green power used)

DPPE was also mentioned in another session. That will come later.

1. Video of Una Song’s presentation on Energy Star for data centers-Video1
2. Video of Una Song’s presentation on Energy Star for data centers-Video2
3. Video of Una Song’s presentation on Energy Star for data centers-Video3
4. Video of Una Song’s presentation on Energy Star for data centers-Video4

I wrote about my first idea for a panel discussion in my previous blog, and here’s my second. When we analyze the power consumption ratio among servers, storage equipment, and networking, we find that servers consume far more power than storage and networking equipment combined. That is why the initial Energy Star effort at energy efficiency in data centers was for servers. EPA’s first version of the server energy efficiency rating system is completed, and the agency is working on the second version now.

Servers

There are several ways to curb server power consumption in data centers:

1. Turn off unused or unnecessary servers.
2. Consolidate servers via virtualization.
3. Implement power management.
4. Refresh hardware for better energy efficiency.

The second and fourth options above have been given a lot of attention. Is there anything more we could do to reduce consumption?

Storage

After we got a handle on server energy efficiency, we started looking at storage devices, which consume a sizable amount of power, even though it’s less than servers use. This is reflected by EPA’s recent effort to establish an Energy Star rating system for storage. There are a few energy efficiency technologies for storage devices:

1. deduplication
2. MAID
3. thin provisioning

Anything else?

Networking (revised and added)

We have not seen any effort on a rating system for networking energy efficiency yet by EPA. But IEEE has started working on one. Some of the energy efficiency in this space are:

1. IEEE EEE (Energy Efficient Ethernet): This is a specification for a more-efficient Ethernet.
2. WOL (Wake on LAN): A network message turns a computer on when it’s needed and off when it isn’t.
3. Network I/O Virtualization with Unified Fabric: Fibre Channel can be carried on top of Ethernet.
4. Network Controller I/O Offload: Offloads networking tasks from central CPUs to offloading engines.

When we analyze power consumption at data centers, depending upon the data we quote, we find that cooling consumes 30% to 60% of the power. So we tend to curb the power consumed by cooling. But when we step back and think hard, we realize that the IT equipment is generating intense heat. There is a study that indicates that a 1 W IT saving equates to a 2.84 W facilities saving.

	- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
		Ten Cooling Solutions to Support High-Density Server Deployment
	- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

So another idea for a panel is to gather people who are in the server, storage, and networking space and discuss how the IT space can be made more energy efficient. One panelist could be a vendor or someone who installs data center equipment. Another panelist could be a designer or planner of IT infrastructure who might be able to quantify the saving. If the quantification is hard for the planner, we could bring in a researcher.

I might ask the panelists:

• Can you run your equipment under much higher temperature? If so, how high can it be?
• Is there any other way to curb (server, storage, or networking) power consumption further, independent of the facilities environment?
• Can you save more if you coordinate your server with storage or networking equipment?
• How can you curb power consumption if you coordinate your IT equipment with the facilities environment?
• How about DC interfaces to your IT equipment to support DC power distribution?

What is your idea?

When NTT Facilities throws a party, it knows how to do it right. (NTT Facilities is one of many NTT companies, and it specializes in the inside infrastructure of data centers.) Following the conference presentation last week in Tokyo,  the other attendees and I were bussed to their research center, about an hour from downtown Tokyo. The research center is in the same place as the NTT Central Research Laboratories. 

It is a nice, cozy suburban environment without any distractions (nothing else to do except for conducting research).

There we saw the state-of-the-art technologies being developed by NTT Facilities. We saw three research projects (only a subset of all the research) going on for data centers:

1. Earthquake simulator
2. Airflow simulator
3. Environment simulator

When I say a simulator, it is not a software simulator but a real system consisting of a lot of iron. Let me briefly describe each.

Earthquake simulator

Japan has a reputation as the center of earthquakes, especially in the Tokyo metropolitan area. Since Tokyo is the center of economy and politics, earthquake damages to its infrastructure would be catastrophic, including for IT and communications. Yet not many businesses leave Tokyo because of that. The second-best idea is to get ready for imminent earthquakes. What does that mean for data centers and the equipment in them?

	- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
		Monitoring Physical Threats in the Data Center
	- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

Equipment, especially IT devices, is sensitive to shaking. For example, disk heads may crash if the equipment is shaken too hard. If there is a way to mitigate such shaking, the equipment may survive an earthquake of considerable size. But how do you prepare for that without actually shaking your racks full of sensitive IT equipment? As shown in the following picture, NTT Facilities developed an earthquake simulator.

It is a giant platform with a few arms that can move and shake the platform vertically and horizontally, as shown in the two pictures.

(Note: All the pictures in this blog are due to courtesy of NTT Facilities, Inc.)

The first picture shows the size of this simulator. The second picture shows some details from above, such as the vertical and horizontal arms.

NTT has offices throughout Japan, and each is equipped with meters to record earthquake information. In addition to the data recorded by Japan Meteorological Agency, it has its own recording of the major earthquake in the Kobe region, which was not known as a site for major earthquakes prior to 1995. Initially, NTT Facilities did not allow us to take pictures. I can say this: even if you could take pictures of this sophisticated simulator, it would be impossible to copy what they developed. It involves elaborate manufacturing technologies and precise software to repeat the exact earthquake waves.

Airflow simulator

The second tour was of a chamber for experimenting with the control of airflow in a data center. NTT Facilities has a product to contain a cold aisle, also known as aisle capping. Two aisles are set up, with and without capping, and airflow and temperatures at multiple locations are measured.

The picture shows an aisle without capping.

Environment simulator

The last tour was of a chamber for controlling the environmental parameters (temperature and moisture) to test CRAC units for outside. CRAC units are usually deployed in a pair, one inside and the other outside. The outside unit is affected a lot by outside environmental conditions. It would be nice to be able to change the parameters and test them in a simulated environment.

This picture shows this chamber, which can produce temperatures ranging from 14 °F to 122 °F. When we entered this chamber, it was set to 14 °F to simulate a typical temperature at Sapporo (the biggest northern city) in winter. We only stayed there for five minutes because it was too much.

This tour exemplifies the fact that Japan has a lot of technologies in progress and sometimes shows them to the public (as long as they speak Japanese) but is very shy in presenting them in English. I do not think it is because of their secrecy but the lack of language ability to do so. I will keep publishing interesting news.

I have covered many topics as they relate to energy efficiency for data centers. I want to extend that coverage by gathering the experts in each specific topic for a meeting. A panel discussion is a good way to do that.

I have a few ideas that I will use one by one in my blogs as proposals to potential conferences and panelists. One area I have spent a good deal of time on is metering in data centers. I have visited several metering companies and interviewed their executives. For most of those companies, I have written a blog post:

• Arch Rock
• Modius
• OSIsoft
• Sentilla

In addition, I have talked to Liebert/Aperture and SynapSense. Although they are all in the same monitoring and metering space, each company has a slightly different angle. Some measure via their own sensors and aggregate the data for display. Others do not gather data by themselves but exploit the data from other sources. So the functions can be roughly classified into three categories: measuring (via sensor), aggregating, and analysis and display.

	- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
		Energy Efficient Cooling for Data Centers: A Close-Coupled Row Solution
	- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

Also, in addition to measuring power consumption, different approaches can be taken to common functions like:

• alarm management
• asset management
• capacity analysis
• efficiency analysis
• air control automation

In the first panel, I would like to discuss the following so that a general audience can understand the needs of metering, learn which technologies are state-of-the-art, and assess the minimally necessary functions of metering. We would discuss the following:

• Why do data centers require monitoring and metering?
• What is a typical architecture for measuring, aggregation, analysis, and display?
• What are the minimally required functions of metering?
• What kind of standards should be defined for metering? What type of data should be collected? Granularity of data? Frequency of collection? Data formats (like XML)? Display formats?
• What extensions, such as e-waste and water, should be considered for existing metering?

An ideal panel will consist of three or four vendors and one researcher in the space. I plan to send this blog to the vendors I talked to before, at least one candidate conference, and a research organization, like a university.

Next, a proposal for a panel discussion of IT energy efficiency.

It has been reported that both the airline and the IT industries are equally responsible for the emission of greenhouse gases—2% each of the total.

The volcanic eruption in Iceland grounded planes in most European airspace for six days. The "Information is Beautiful” website reported interesting statistics.

They calculated the saving of CO₂ emissions resulting from flight cancellations during that period. The total is the saving realized by no flights minus the emissions caused by the eruption.

So this is the equation:

344,109 tons (saved) – 150,000 tons (added by the eruption) = 206,465 tons

This figure is only for European airlines. I know that many other flights between Europe and Asia and between Europe and the U.S. were cancelled. So the saving is much more.

This figure poses a few interesting questions. First, the cancellations caused many business activities and personal lives to be disturbed. But at the same time, we saved a lot of CO₂ emissions. We need to ask ourselves which is more important: to reduce CO₂ emissions or to preserve our business and personal activities. Of course, there should be some balance between the two. But how do you draw a line to decide the right balance? With the advent of the online world, we can communicate with people around the globe without leaving the comfort of our home. But that cannot replace physically being there with your business associates or loved ones.

	- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
		Making Large UPS Systems More Efficient
	- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

Second, IT is equally to blame for the emission of CO₂. What if we decide to curb the use of IT in order to curb the emissions from its use? We need to assess how much IT is saving CO₂ emission by telecommuting, replacing business trips, and so on. In addition, IT saves power consumption (thus, less CO₂ emission) by optimizing supply chains and fleet management. Of course, IT consumes power as it functions. IT is used not only for absolutely necessary functions but also for entertainment and enjoyment like online video sharing and SNS. Again, some people may think the latter functions are absolutely necessary, while others can do without them altogether. Again, it is hard to draw a line between the two.

On top of this, if you throw in developed countries vs. developing ones, the discussion becomes much more difficult to deal with. I wonder whether we will eventually come to some kind of agreement?

I have participated in and run several sessions on the topic of data center energy efficiency here in the U.S. as well as in Japan. I will moderate yet another such panel at SRI at 10:45 a.m. on April 28, less than a week away. This one is a little bit different from the ones I’ve been involved in before. The volcano ashes at one time threatened to postpone the conference, but it will be held as planned.

I’ve talked before about how to select a location for a data center. The criteria include:

• Power
• Network access
• No natural disasters
• Climate
• Tax breaks
• Access to human resources
• Space
• Market

The first two are essential. Without power or network access, no data center can be constructed. It is most beneficial to discuss universally applicable disciplines for energy efficiency, but it is also interesting to know how four data center operators from Norway, Iceland, and Finland are exploiting their regional environments for energy efficiency. My April 28 panel will also have two experts from Google and  from the Electric Power Research Institute (EPRI), who will talk about universally applicable disciplines, with flavors of region-specific environments.

The panel consists of:

• Moderator: Zen Kishimoto, principal analyst, AltaTerra Research (U.S.A.)
• Isaac Kato, executive vice president and chief financial officer, Verne Global　(Iceland)
• Kimmo Pentikäinen, director of Environmental Information, Elisa Corp. (Finland)
• Sindre Kvalheim, chief executive officer, LocalHost (Norway)
• Eric Teetzel, manager, Green Energy Program, Google (U.S.A)
• Dennis Semanski, senior project manager, EPRI (U.S.A.)

Incidentally, Isaac Kato lives in Boston and sort of commutes to Iceland. Verne Global’s data center is completely fueled by renewable energies from hydro- and geothermal, and it exploits the cold air there. The data center is west of the volcano and has not been much affected by the eruptions. And other operators have a lot of interesting stories to tell about their energy efficiency efforts. This is a must-see event.

You can register here.

The conference by itself will be interesting enough, but they are making it even more exciting by having opening and closing receptions. The opening reception is scheduled for Monday, April 26, 6:00 p.m.–8:30 p.m., at Stanford University’s Wallenberg Hall, 450 Serra Mall, Building 160. I plan to attend this reception and would like to meet with some of my readers there. The Silicon Vikings and the Consulate General of Finland will host the networking event. Drinks and hors d’oeuvres will be served. Please RSVP and find more info here.

	- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
		Deploying High-Density Zones in a Low-Density Data Center
	- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

The closing reception is scheduled for Wednesday, April 28, 6:30 p.m.–9:00 p.m., at Dinah’s Garden Hotel and Trader Vic’s Restaurant, 4269 El Camino Real, Palo Alto, CA. Hans-Martin Friis Møller, director of Water, Energy, and Environment for Grontmij　Carl Bro, will give an informal presentation on the worldwide challenges of water and how new technology and programs are being implemented. The presentation will be hosted by the Danish American Chamber of Commerce, Invest in Denmark, and Innovation Center Denmark. Hors d’oeuvres will be served, and there will be a cash bar. Please register and learn more here.

In early February, Bloom Energy made quite a splash with the announcement of their fuel cell–based systems. There are several types of fuel cells, each using a different type of electrolyte. Knowing that fuel cells are very hot in Japan, I searched organizations working on the same type of fuel cell, solid oxide. This type uses ceramic electrolytes and runs at a temperature between 850 °C and 1,100 °C.

My search found the New Energy Foundation (NEF) working with New Energy and Industrial Technology Development Organization (NEDO). I met Mr. Makoto Okuda, the director of the Fuel Cell Department of the Research and Planning Center. NEDO has eleven projects under way, including development and validation of solid oxide fuel cell (SOFC) and polymer electrolyte fuel cell (PEFC) technology, and hydrogen technology and infrastructure development.

Mr. Makoto Okuda

The PEFC type is for small-scale systems, and the SOFC is for large systems. The PEFC is planned for automobiles and small power systems for residential use. Since I want to compare Bloom Energy with what’s happening in Japan, I will focus on SOFC in this blog.

NEF classifies SOFC in these four categories:

• Residential—between 1 kilowatt and several kilowatts
• Commercial—between several kilowatts and several hundreds of kilowatts
• Industrial—between several hundreds of kilowatts and several megawatts
• Large installation—several megawatts and more

Small-scale SOFC will accomplish $10,000 per kilowatt in 2012 or so; a comparable solution is about $700,000 per kilowatt. A good-size data center would require an industrial-size fuel cell, which should be available around 2012.

Right now, NEF is conducting validation of the residential version of the SOFC technology. Participants include large gas companies like Osaka Gas and Tokyo Gas, electric power companies like Tokyo Electric Power Co., Nippon Oil Corp., and TOTO. Except for Nippon Oil, which uses LPG and kerosene, the fuel is natural gas. Fuel cell system vendors include Kyocera and TOTO.

I discussed the following with Mr. Okuda:

• Amount of investment
• Speed to market
• Elaborate validation

	- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
		Energy Impact of Increased Server Inlet Temperature
	- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

The Japanese government is investing $12M in SOFC technology; in contrast, Silicon Valley–based Bloom Energy has raised $400M in funding so far. Also, the speed-to-market is significantly different. Bloom Energy, in competition with many others, needs to reach the market at utmost speed, while in Japan several companies need to work together under the guidance of the government.

Finally, the Japanese effort spends lots of time on validation. The SOFC technology runs its system under high temperature (700–1,000 °C), and it is necessary to make sure the system does not break after operating at high temperature for a long time. Mr. Okuda did not question the quality of Bloom’s fuel cell, but he was interested in reviewing the results.

Finally, I may be able to visit Bloom Energy soon and will ask for such data.

IBM experiments with a liquid cooling system that moves heat away from sensitive computer components without chillers, cutting energy costs in half

Today’s supercomputers run hot, thanks to power-hungry microprocessors that enable sophisticated scientific research and complex financial transactions to be performed in the blink of an eye. As these microprocessors have become smaller and more powerful over time, they are generating even more heat, a problem that data centers generally address expensively with air conditioning and chilled-liquid cooling systems.

Whereas datacenters have for decades used cold liquids to transfer heat away from central processing units (CPUs), a team of IBM researchers in Switzerland is experimenting with a micro network of copper tubes that run through smaller, clustered computer servers and whisk away heat with the help of warm water. Liquid cooling, even with warm water, is 4,000 times more effective than air cooling at removing heat, they say.

Continue reading at Scientific American –>

I will be moderating a panel titled "Data Center Efficiency via IT Optimization, Standards, and Collaboration” at the BrightTalk Efficient Data Center Summit April 21, from 11 a.m. PDT, as you can see here. Energy consumption by U.S. data centers has been increasing, and some are running out of power for their growing number of servers and other IT equipment. To make the matter worse, IT equipment generates excessive heat, which must be rejected, and cooling requires more power.

Power crisis and energy efficiency, as they relate to data centers, have been discussed at many conferences and meetings. In this panel, we will take a different angle on three topics. One is IT energy efficiency improvement. A 2007 EPA report looked at five different scenarios for the past, present, and future of data center power consumption. Each scenario is defined by the energy efficiency of both IT and facilities. Of course, the facilities side is important, but the very reason to have IT equipment is to support business needs. IT comes before facilities, and we need to consider IT equipment energy efficiency as such. Each of the five scenarios has a different degree of the following to achieve a different level of energy efficiency:

• Turning off of unused servers

• Virtualization

• Power management

• Hardware refresh

• Storage equipment efficiency

	- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
		Avoidable Mistakes that Compromise Cooling Performance in Data Centers and Network Rooms
	- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

The EPA study was conducted in 2006. In the four years since then, IT energy efficiency technologies and practices should have become more innovative. First, we would like to explore that. Second, several data center organizations advocate different metrics, technologies, and practices for energy efficiency. Doesn’t it make sense to have one set of standards for everyone to use, eliminating the confusion? Third, to produce a single set of standards, all these data center organizations must get together and collaborate rather than continuing individually to develop their own, all of which may be slightly different.

Summing up, these are the three topics:

• IT energy efficiency optimization

• Standards

• Collaboration

We are fortunate to have four experts in data centers and data center energy efficiency for this panel:

• Richard Garrison, board member of the 7´24 Exchange California chapter and senior principal at Alfa Tech Cambridge Group

• Jon Haas, director of Eco-Technologies at Intel and board member of The Green Grid

• KC Mares, co-chair of the SVLG Data Center Energy Efficiency Demonstration Program and president of MegaWatt Consulting

• Dean Nelson, founder and chairman of Data Center Pulse and senior director of eBay

We expect a lively conversation on these topics. Again, you can join us for this webinar at your desk at 11 a.m. PDT on April 21. See you then.

Although the power usage effectiveness (PUE) metric is not without its problems, it is the de facto standard for measuring energy efficiency in a data center. Japan’s Green IT Promotion Council  wants to improve on PUE with its own metric, data center performance per energy (DPPE). The council (GIPC) wants to make this metric the world standard by promoting it to other organizations, such as The Green Grid. GIPC even published a white paper on DPPE in February.
GIPC’s definition of DPPE is somewhat convoluted, defining submetrics first:

• IT equipment utilization (ITEU)
• IT equipment energy efficiency (ITEE)
• PUE
• Green energy coefficient (GEC)

Conceptually,

DPPE = (IT equipment utilization * total IT equipment ability) / (total DC power consumption – green energy used)

= IT equipment real use / total power consumption

More precisely,

DPPE = ITEU * ITEE * 1/PUE * 1/ (1-GEC)

= (IT equipment utilization * IT equipment ability)

/ (total power consumed – green power used)

So they are saying that DPPE is a ratio of the actual IT equipment’s ability (because some of its total ability may not be used all the time) against the dirty power consumed.

GIPC claims that they came up with this metric after interviewing many data center operators and related companies.

Let’s look at two other metrics considered as improvements to PUE: data center energy productivity (DCeP) and corporate average data center efficiency (CADE). First, DCeP, under consideration at The Green Grid, is similar (but not identical) in spirit to DPPE.

DCeP is defined as:

useful work produced / total power consumption

This is different from DPPE in two ways:

1. DCeP does not take the type of power (dirty or clean) into consideration.
2. DCeP considers the output that is really useful.

Although I understand that the consideration of clean vs. dirty power may make sense, in reality we are not even close to the use of clean power for data centers yet. Although it is not possible to have all the power supplied by clean (green) power, if a data center is fueled by clean power alone, this ratio becomes infinity. Any metric that has a potential to become infinity may not be appropriate.

As for the second point, DCeP takes the software aspect somewhat into consideration. DPPE seems to simply count MIPS for given computing, regardless of what is produced. In an extreme case, you can run meaningless software to keep a server occupied and produce useless results. But I think DPPE is using this as a proxy for the useful work, as in DCeP. In addition, defining useful work has been a problem. That is why several proxies have been proposed for DCeP.

CADE, on the other hand, was developed by McKinsey, in conjunction with Uptime Institute.

	- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
		Energy Efficient Cooling for Data Centers: A Close-Coupled Row Solution
	- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

CADE is defined as:

CADE = facility efficiency (FE) * asset efficiency (AE) where

• FE = facility energy efficiency * facility utilization
• AE = IT energy efficiency * IT utilization

This also is somewhat similar, although not identical, to DPPE.

Like any metric, DPPE has its merits and demerits. The Green Grid and other organizations have their own metrics for improving PUE. If DPPE is promoted as the world standard for energy efficiency at data centers, GIPC needs to have open discussions with The Green Grid and others. Everyone has an opinion on energy efficiency metrics, and no single metric can express energy efficiency perfectly.

In my opinion, Japan has been following many of the U.S.-led initiatives and technologies to date. Whether or not DPPE is accepted as a world standard for data center energy efficiency, it is a good thing for Japan to express its view outside of Japan.