Following on my previous post, today I focus on automated meter infrastructure (AMI) and field area network (FAN). Local area network (LAN) and wide area network (WAN) are well-known terms and do not need explanation. There are a few XANs to indicate the scope of the area covered. For example, a metro area network (MAN), usually citywide, is larger than a LAN but smaller than a WAN.

Smart grid introduced the term FAN. FAN is part of AMI, the infrastructure connecting the end consumer to the utility. A smart meter aggregates home power consumption data and passes them on to a FAN, which then transmits that data to a WAN, which delivers them to the utility.

For the average consumer, the smart meter is the gateway to the utility and the interface with the home area network (HAN). Although there are other gateways, such as broadband cable, DSL, and fiber modems and connectors, the smart meter is gaining a foothold as a gateway. Since the utility installs it, the smart meter is already compatible with the utility’s communications protocols. For that reason, it is becoming the de facto gateway.

The meter technology is unique, and I am not sure whether typical ICT vendors can enter the field easily. Some vendors, like GE, stand out among the who’s who in providing meters. In my PG&E territory, we had an analog (dumb) meter from GE before, and our new smart meter is also from GE. A big difference between dumb and smart meters, aside from analog vs. digital, is the communications capability of the smart meter. The smart meter has at least three functions: it aggregates/stores the power usage information (hourly or more frequently), transmits the aggregated data to the utility via the FAN, and receives the signal from the utility to control devices and appliances on the HAN.

A dumb meter stores the power usage information by advancing hands in the analog indicators, and it needs no specific internal memory. On the other hand, a smart meter requires some kind of internal memory to record the usage according to the frequency of measurement. In addition, a smart meter needs a mechanism to transmit and receive data and signals. A transmission chip or firmware is embedded into the meter logic for that. ZigBee (mesh wireless technology) is becoming the de facto standard for smart meters. (ZigBee is not a company but a consortium that dictates the ZigBee specification. A company that wants to implement ZigBee pays a license fee and implements its own version according to the specification. NIST is very cautious about declaring a technology a standard. For example, NIST has designated IP as a communications protocol but no others yet. ZigBee, Wi-Fi, and WiMax are under consideration to become part of the standard. ZigBee has started working on supporting IP but doesn’t do so completely yet.

	- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
		Schneider Electric Improves Product Quality While Saving Over 2,500 Engineering Hours with Coverity
	- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

Data collected from each household are aggregated to a network access point (NAP) in the neighborhood. I have been looking for this point but have not found it yet. Since ZigBee does not support IP 100%, the communication between each smart meter and the NAP is via ZigBee.

Some information about PG&E’s smart meter can be found here.

Each NAP is now connected via wireless WAN. PG&E uses ZigBee for FAN, but other technologies could be wired  or wireless (WiMax) and BPL.

Although I have not abandoned the green IT/data center field, I have also started following smart grid. Smart grid includes the three areas of power, IT, and communications. In this post, I’ll briefly touch on how ICT is used in smart grid.

The power system consists of generation, transmission, and distribution. Let’s start with distribution, which also has three parts: the distribution network, field area network/automated meter infrastructure (FAN/AMI), and the home. Today I’ll talk about the home aspect.

	- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
		The Different Types of UPS systems
	- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

A smart meter installed at home provides power usage information as an aggregate. By observing my hourly power usage information, I can understand my usage. (By the way, I do not think we have any dynamic pricing yet, as the usage and the charge for it are proportional.) This is great progress, but I would like to see a more detailed breakdown. For example, there is a spike in the middle of the night when I shut down all but essential appliances like the refrigerator. I would like to know if this spike comes from the refrigerator’s defrosting cycle or something akin to it.

To support more-detailed information, each appliance and other electronic device must be able to report its power use to the smart meter or some other collection point in the house. To do this, we need a module to meter power usage as well as some kind of communication function. A new set of appliances and electronic gear may come with the metering chip or firmware installed, but we need a dongle for the existing ones. The dongle business would be only until our current appliances are all replaced by new ones that have the chip or firmware. As for communication, we need a mechanism to transmit the power usage information to the smart meter and/or some collection point. The communication should be two-way because, when demand and response (D/R) is implemented, each appliance and other electronic gear must receive a D/R signal and respond to it. Whirlpool, for example, has announced that it would include such a function in all its new appliances. The candidates for the communications technology include HomePlug, ZigBee, Wi-Fi, Z-Wave, and 6lowpan. None of them is designated as a standard by NIST, though.

Right now, the only way I can get my power usage information is to access PG&E’s website. The information is delayed more than 24 hours. Even though this is progress over the one-month-late information on the bill, I want to receive more minute and real-time information. Unless the data sent over to PG&E are available to me, I cannot do anything with them. However, if its own collection module is embedded in each appliance and electronic device, any power usage information, whether aggregate or individual, can be readily available. The metering frequency can be adjusted at our discretion. Once such data become available, they can be displayed on any of several dashboardlike software applications now available. Google and Microsoft provide free software, while companies like Opower sell their own versions. Opower works with utilities to provide its product. PG&E has not worked with either Google or Microsoft, stating that it wants to wait for the standard.

The home market may be large, but the technical barrier to entering it does not seem to be too daunting. The chip or firmware for metering power usage is a commodity. The communications protocols are well known and do not appear to be hard to implement. Once the power usage data are available, collection, aggregation, analysis, and display are very straightforward, and I do not see much differentiation in the technology itself. So probably the key to this market is how well each vendor and service provider can work with utilities. After all most consumers are not engineers and would like to have an easy solution. On top of that, power consumers deal with their utility, and adopting the solutions provided by their utility may feel easier.

By the way, as one of the speakers at SVLG’s energy summit, I find my interest in checking the hourly use of power waning. After all, I cannot see the breakdown or real-time usage. I will talk about the FAN/AMI area later.

Like other people, I’ve been wondering how Oracle has integrated many of Sun’s technologies and hardware after buying that company. Oracle has just started a nationwide tour to inform us of just that. The first stop was in their neck of the woods, Palo Alto, California. The free seminar information, titled "Share the Vision: Build aMore Efficient and Powerful Datacenter with Oracle,” is found here, along with the agenda and locations.

Now that Oracle has Sun’s hardware, Solaris operating system, virtualization engine, and other components, they can provide well-integrated solutions for data centers, as this slide shows:

The plan is to increase server and storage performance by several times and even tens of times more in a few years.

 Server Performance Increase

 Storage Performance Increase

 In addition to Solaris, they have Red Hat–compatible Oracle Enterprise Linux (OEL) and support. Their virtualization is implemented through Oracle’s version of Xen software, Oracle VM. On top of Xen, Oracle added management and other software to make it a comprehensive offering for virtualization. The features of Virtual Iron, which Oracle acquired some time ago, are being integrated into Oracle VM. Both Oracle VM and Virtual Iron are based on Xen, so the integration should not be too hard.

 A presentation on cloud computing was a good tutorial. Many of Oracle’s customers are enterprises, which tend to use VMware’s virtualization solution. Oracle provides a feature to translate VMware VM file formats to theirs. Currently, private or on-premise clouds are implemented mostly with VMware, and public clouds (AWS and Rackspace) are implemented with Xen. Eucalyptus’s enterprise version has a feature that is the reverse of what Oracle VM does. It can translate VMs by VMware on-premise to AWS file formats to allow them to be transported to AWS public cloud (known as CloudBurst).

 Oracle view on Enterprise Evolution to Cloud

 Since Oracle now owns microprocessors, server and storage hardware, operating systems, virtualization engines, databases, and applications, they can fine-tune the entire system to make it very efficient and execute very fast. Although Oracle is still small compared with IBM in terms of revenue (Oracle $27B vs. IBM $96B), Oracle is now in a position to compete with IBM.

	- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
		Energy Efficient Cooling for Data Centers: A Close-Coupled Row Solution
	- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

 The seminar was well run and presented very useful information. If there was one thing I did not like about it, it was the lack of discussion of energy efficiency. Energy efficiency and green IT initiatives were mentioned several times during the day, but they were not discussed in detail. I asked one of the speakers to share some energy efficiency data with me. If and when I get it, I will publish it here.

When I started this blog about two years ago, there was a lot of discussion of direct current (DC) power distribution at data centers. It is very easy to see why DC is better than alternative current (AC) power distribution. AC power enters your data center from your utility. It then goes to a set of UPS that take in AC but convert the power to DC. This is because the power goes through a set of batteries (which only takes DC). Then, at the other side of the UPS, it is converted back to AC. Then the power is distributed to IT equipment via PDUs. This AC power is then converted again by the IT equipment for its internal use. Each conversion loses some percentage of power.

 If the AC power entering the data center is converted only once to DC power and distributed to IT equipment that takes DC as input, there would be no conversion loss. Unfortunately, a few years ago, Green Grid put out a white paper comparing the power distribution of DC and AC and concluding that there were very few differences between the two configurations. After reading it, I moved my focus away from power distribution. Now I am back on this subject because some friends who started a company in that area and Keizo Hoshijima of NTT Facilities pitched me the merits of DC distribution recently.

 I cannot forget another DC power distribution advocate, Dennis Symanski of EPRI. I met Dennis when he was a panelist in my Nordic Green panel session.

 Dennis Symanski

 At the conference, he gave a presentation on DC power distribution. Cooling is known to consume about 30–60% of power in a data center, but we seem to be getting a handle on that. Once cooling and other culprits are under control, then we can pay attention to power distribution to further cut power consumption.

 Recently, I had a chance to visit Dennis at EPRI to chat about the current status of DC power distribution at data centers. The following is an edited summary of our conversation.

Q: What do you do at EPRI, and what is your background?

A: My focus is to make any computer equipment energy efficient (EE). Prior to coming to EPRI, I spent 18 years at Sun working on international standards and regulations. EPRI is funded by utilities but is an independent nonprofit research organization. It strives to "do good for society,” and it is quite refreshing for me when I do not have to pay attention to revenues and stock prices.

	- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
		Energy Impact of Increased Server Inlet Temperature
	- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

Q: So what’s new in the DC power distribution area?

A: In 2006, we conducted a set of experiments on the use of DC power distribution at Sun (currently Oracle), along with LBNL, Ecos Consulting, and other vendors. Our conclusion was that DC power distribution increases data center EE and reliability. Since the experimentation, we formed an organization called DC Power Partners to further our efforts. The members include LBNL, EPRI, EMC, IBM, HP, Oracle (former Sun side), Intel, and even APC. (ZK; APC had a different opinion of DC power distribution before.) We have a once-a-month teleconference to discuss new technologies and installations to share information. In addition to the U.S. companies, some European and Japanese companies call in.

Q: You only cover technology aspects at your DC Power Partners?

A: EMerge Alliance works on DC lighting, led by Armstrong Ceiling for 24 VDC. DC Power Partners is joining the alliance. They can take care of sales and marketing, trade shows, and web promotion.

Q: Are there any major changes in the DC power distribution area?

A: There are no drastic changes. We are currently working to make sure the components that comprise DC power distribution comply with UL and/or FCC regulations. For example, connectors that are used for DC power distribution.

Q: I have no issue with the technology. What about the market? Which region of the world is more receptive to this technology?

A: In general, the telecom industry is keen on picking this up. Companies like AT&T, Verizon, and NTT. In Europe, the European Telecommunications Standards Institute (ETSI) has prepared a specification for 380 VDC. There is at least one CO in each city and dozens in large cities, so there are 10’s of thousands of COs in the U.S.

Q: But IT-centric data centers and COs are different, and the IT data centers may not be so enthusiastic about DC power distribution.

A: They used to be quite different, but, these days, they are becoming increasingly similar as they share the same kind of IT equipment. Look at AT&T, which needs to process IP traffic coming from iPhones and iPads.

Q: A couple of years ago, I read a white paper from Green Grid (GG) that said there were few differences between AC and DC power distribution. What is your take on that?

A: GG members do not include any direct current facilities equipment suppliers. Several of the GG IT equipment suppliers are researching direct current power supplies for their equipment.

Q: What do server vendors need to do to support DC power distribution?

A: All they have to do is to swap the AC power supply with the DC one. The direct current power supplies have a smaller component count, are more efficient and as a result, research demos may show them to be more reliable. It is not hard to create a data center solely with DC power distribution. IBM created such a data center at Syracuse University.

Q: Is there any data center with a solely DC power distribution system in the San Francisco Bay area?

A: Several are going in parallel. But one in the Bay Area will be announced in November.

Q: Right now, generated power is carried via the AC transmission system. If DC power enters a data center, we do not need to convert power at all. Can you transmit power via DC?

A: In the days of Edison vs. Westinghouse, we could not transmit DC power at 5,000 V or higher. These days the high DC voltage is transmitted over a long distance. A hydropower plant in northern Quebec transmits generated DC power to New England over several hundred miles, in addition to much-closer Toronto and Montreal. (ZK: High-voltage direct current is a technology for transmitting high voltage DC over thousands of miles)

Q: Is there anything else I should know about what EPRI is doing in the DC area?

A: We talked about the power supply for IT equipment. Those power supplies were very inefficient. The conversion rate was only 65% on the average. 80 Plus is an organization that promotes increasing the conversion rate to at least 80%. EPA adopted their specification, and EPRI is running a test lab for 80 Plus.

I got very useful information talking with Dennis. EPRI covers a large area of energy and its efficiency. I have not heard much about DC power distribution recently, but that does not mean efforts on its behalf were terminated. On the contrary, they are going very strong, including DC for home and power transmission. I will report on this subject from time to time in this blog.

I have done consulting work for MySQL as it enters the Japanese market for more than three years. I have not seen Marten Mickos for a few years. I wondered what his next move would be after he left MySQL. I heard he became CEO of Eucalyptus, which is gaining visibility in the crowded cloud computing market. I asked him to talk to me about his new venture. In spite of his busy schedule, he was nice enough to sit down with me at breakfast and share his views on Eucalyptus and cloud computing. I usually take my own pictures for this blog, but I forgot to do so in the excitement of talking with him. So the official picture from Eucalyptus is a little too formal for my blog. But, oh, what the heck!

Marten Mickos

The following is a summary of our talk.

Q: Correct me if I’m wrong: Eucalyptus provides a set of open source tools and utilities to create on-premise clouds. All the components are either from Eucalyptus or some other open source. In addition, it has a hook to pick different types of hypervisors and to run on major Linux distributions like Ubuntu and CentOS. It is largely compatible with the Amazon Web Services (AWS) environment. You can move your virtual machines (VMs) between Eucalyptus and AWS clouds.

A: Basically, you are right. We provide a platform for building on-premise cloud.

Our enterprise version supports multiple hypervisors, such as Xen, KVM, and VMware.

Q: Can you name your customers for the enterprise product? Is the pricing based on number of CPUs?

A: We have several customers that are large enterprises and government agencies. I cannot reveal their identities yet, because I need their consent. Licensing is based on cores and we charge about $300 per core.

Q: What is an on-premise cloud, and what is the difference between on-premise and private clouds? In the past, you were looking for a better term than on-premise cloud. Did you find one?

A: An on-premise cloud runs on your hardware infrastructure at your site. A private cloud is a cloud reserved for the use of one single organization. In many cases, if not all, you can use the two terms interchangeably.

Q: What is your license for the open source side?

A: It is GPL version 3.

Q: Who develops code and commits it to the main trunk? Who has the copyright of the code? Do you take others’ contributions as well?

A: We develop our code and have the copyright of the code but take in others’ contributions as well.

Q: So this is somewhat similar to what you did at MySQL?

A: Yes. But the difference is that MySQL was disruptive in the old database market, but Eucalyptus is innovative in the new market of cloud computing.

Q: You are a big proponent of open source, and you like what Rackspace recently did? I mean that it created OpenStack for clouds. NASA’s Nebula project uses Eucalyptus, and they contributed their code to the OpenStack project. Did they include Eucalyptus in OpenStack as well?

	- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
		Preventing Data Corruption in the Event of an Extended Power Outage
	- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

A: Yes, I like what Rackspace did. I do not think they included Eucalyptus, because our license is GPL and OpenStack is by Apache 2.0. With open source, users can naturally take your code, use it, modify it and redistribute it. In the cloud market there are new projects and companies being launched all the time.

Q: If everyone can do it, what you are doing? What are your differentiation and/or advantage points?

A: We see it as a benefit, not a problem, that open source gives users many freedoms. Already today we have tens of thousands of users who do not need our commercial support. But the most mission-critical installations are dependent on our support, bug fixes and further development of the product. Uniquely in this market at this time, we have a scalable, mature and functional product.

Q: Amazon uses Xen open source, yet has its own APIs and file format for VM. So open source is closed by a proprietary container and is no longer open source. What do you think of it?

A: Again I think it demonstrates the power of open source. Open source is needed in all major infrastructure deployments today. What Amazon is doing is perfectly within the licensing terms and principles of open source.

Q: What is Dr. Rich Wolski (the founder of Eucalyptus) like? Is he a typical researcher?

A: Not at all. The typical researcher does not produce very useful things like Eucalyptus. Also the typical researcher does not start a new business.

Q: Is cloud computing more energy efficient?

A: Yes it is more energy efficient. But we must remember that certain computation always requires a certain amount of energy. It is the fact that you have higher utilization rates in cloud computing and you can turn off servers that are idle for some time to save energy.

Q: What about the opposite of that? What if more demands come to you beyond your capacity? Do you load-balance among several different clouds? If so, what are the problems in doing that?

A: Yes, the vision of cloud computing includes cloud bursting, which allows you to move load dynamically from one cloud to another. Technically it is an advanced proposition and we are not there yet for the general public. There are intricate challenges around security, authentication and latency that the industry is working on.

Q: What geographical markets are you addressing now?

A: Four regions are very important: the U.S., the E.U., China, and Japan.

Q: Are you doing anything in those areas, other than the U.S.?

A: Eventually, we will be in all of the areas, but right now we have our hands full with the U.S.

Q: We are sitting and talking here in Silicon Valley. Do you plan to move Eucalyptus to this area?

A: In the days of cloud computing, the physical location of headquarters does not make a difference. The headquarters is where the CEO is.

Q: So it is like Air Force One. When the president gets on the plane, that plane becomes Air Force One.

A: Exactly. Now this cafe is Eucalyptus One, and I can make all the executive decisions from here! In reality, of course, a company makes decisions in many locations and by many executives. At Eucalyptus we work as a team and for this analogy to be perfect, we need to consider the locations of every employee of the company.

Q: One thing I remember about you, during your days at MySQL, is that you came up with good analogies. You said that regardless of which seating area (first class, business class, or economy class) in an airplane you are sitting in, everyone gets to the destination at the same time. This was in reference to MySQL vs. Oracle databases. Do you have a good one for cloud computing and/or Eucalyptus yet? That kind of description grabs people’s attention quickly and easily.

A: I am thinking about it but have not come up with anything yet.

I had not seen him for a while, but Marten remains as he was before, very nice and friendly. As I researched into Eucalyptus, I found many similarities with MySQL. They are both based on open source, but the business model has a commercial version as well. MySQL was run as a virtual company, which, come to think of it, is a cloud in a sense.

Cloud people tend to ignore the underlying infrastructure; data centers. Marten has a good sense of what is below the cloud. When he talked about turning off servers in the case of low loads, I thought about Power Assure, which has a feature to turn servers on and off as needed. If you can synchronize the operation of IT and facility (cooling and power delivery) equipment according to the loads you receive, you could save more energy at your data center.

There are a few technologies to make networking more energy efficient. But a more energy efficient Ethernet would have by far the biggest impact because of its ubiquity.

I visited Broadcom to meet with David Berry, senior marketing manager, and Wael William Diab, technical director, office of the CTO. David and I have talked about the general energy efficiency of networking equipment before. Wael is vice chair of the IEEE 802.3 Working Group

So I focused on finding out what 802.3az is all about. My core technology area started with software engineering and expanded to cover networking and embedded systems. I am by no means an expert in networking technologies but am dangerous enough to know some of the basics. I here confess that while I was summarizing my conversations with these two gentlemen, I studied and researched this subject extensively so I could keep up. If the whole thing is completely over my head, it is easy to let it go, but with some effort, I can understand the subject matter to some extent. Sigh.　　

Wael Diab and David Berry

The following is an edited version of my questions and their answers.

Q: My understanding is that energy efficient Ethernet (EEE) allocates networking resources when needed and turns them off when not needed. Can you give me a little bit more on EEE?

	- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
		Power and Cooling for VoIP and IP Telephony Applications
	- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

A: Traditionally, the most important things about networking were speed and cost. But energy consumption is becoming more important than either of those because opex is beginning to surpass capex. The Ethernet has typically been operated as always-on, even though traffic is not always on. A typical utilization is less than 10% and sometimes even less than 1%. But when traffic is back on, it tends to hit the peak, and the Ethernet has to quickly come back on at 100% capacity. When the PHYsical layer (layer 1 on the OSI model) is on, the layers above (layer 2, data link, and up) are on as well. When there is no traffic, the PHYsical layer (PHY) can be turned off, along with the layers on top of it, creating further energy efficiency. Additionally, we have implemented a way to negotiate when to turn on layer 2. In this way, layer 2 can sleep until absolutely necessary.

Q: Is the ratification of 802.3az done? Is the product based on the standard shipping?

A: The standard is at the final stage, and ratification is expected in September. However, prestandard versions are shipping by vendors, as has happened for other networking products before.

Q: Can this technology be implemented with firmware refresh?

A: Unfortunately, to support firmware refresh, a new PHY is required and, therefore, new equipment. We have a technology called AutoGrEEEn. AutoGrEEEn technology enables a device with a non-EEE MAC to seamlessly transition to EEE capability by implementing control policy assist engines and circuitry inside the PHY device.

EEE requires control for the PHY to be done via in-band signaling over the MAC/PHY interface. This requires a change to both the PHY and MAC silicon. A number of systems have the MAC and PHY as two different pieces of silicon with the MAC often embedded in a switching or controller-type device. These MAC-containing devices have associated drivers and software, and are often multiport devices. So, a transition to EEE may be hampered by additional development that involves replacement of the MAC-containing devices. AutoGrEEEn technology eliminates the need to change the MAC/PHY interface on the MAC silicon, and allows for rapid transition today with legacy non-EEE MAC silicon attached to AutoGrEEEn-enabled PHYs.

Q: Servers are refreshed every three to five years at data centers. How about networking equipment?

A: In general, the networking equipment refresh cycle is probably longer than that of servers. However, it is important to consider networking in conjunction with servers and other IT equipment. Servers get more powerful, with new chips every 18 to 24 months. For example, a newer server with more computing power may require 48 ports rather than 24 ports in the networking equipment to process more loads. Thus, the refresh cycle may not be that much longer than that of servers. In addition, new builds are happening all the time and they can take advantage of any new innovations in energy savings.

Q: I think 802.3az is for 10 Mbps, 100 Mbps, 1 Gbps, and 10 Gbps. Does it apply to 40G and 100G as well?

A: For 40G and 100G, there is another standard called 802.3ba, which was recently ratified.
Both 802.3az (EEE) and 802.3ba (HSE) started about the same time. EEE looked at copper
interfaces that already existed ton be enhanced for energy efficiency. By the way, the
energy efficiency technology we use is called
low power idle . When it is idle, it sends little energy and is very energy efficient.

To achieve this both sides go into a suspended state, effectively turning the PHYs off. To keep both sides synced at the PHYsical layer a refresh signal is occasionally sent. Because the refresh signal has a low duty cycle, it is very energy efficient. To leave the LPI state, either side can send a wake-up signal and the other side reacts to that.

In addition to PHYsical layer savings additional subsystems above the PHY can be turned off. Broadcom championed the development of a technology called layer 2 data link LLDP stateful negotiation for enhanced saving modes in the standard.

Q: Some people insist that saving on the networking side does not make a huge impact on data center power consumption. Some studies indicate that power consumed by networking equipment at a data center is only several percents compared with those of servers and storage equipment. What do you say to that?

A: It is true that the networking equipment by itself does not consume a lot of power from the perspective of a few single ports. However, this energy efficient technology (EEE) for networking enables energy efficiency across the entire data center which typically has 1000’s or more ports. Enabling EEE is like turning off a bunch of leaky faucets. One might not notice the savings at a single point in time but it will certainly show up in the monthly bill. Ethernet has also been proven to be an extremely scalable technology enabling port consolidation during bandwidth migrations cutting total power consumption (1G to 10G for example) while at that same time increasing performance typically by a factor of 10. In a sense, Ethernet networking is playing a much larger role in turning a data center into the next-generation energy efficient one.

Think of this analogy. When you try to establish a high quality of service (QOS), you may want to have the fastest server with the fastest CPUs in it. However, if you study this carefully, you would find out that the fastest CPUs alone are not the answer. It may be established with fast CPUs, memory, disk, and other components combined. Energy consumption at a data center can be considered in the same way. Networking connects IT equipment together and, by activating and deactivating according to needs, networking controls other IT resources (IT equipment). This energy efficient technology in networking should not be considered just for networking equipment but for all the IT resources and, consequently, the entire data center.

After talking with Wael and David, I pondered the following. My discussion with them reminds me of the argument about whether servers or networking are more important in computing. It is true that servers are the center of computing, and we tend to pay attention only to servers. However, networking ties together servers and other equipment, like storage, to create an IT system. Without fast, low-cost, and energy efficient networking equipment, even the fastest and most energy efficient servers could not deliver, alone, the most energy efficient computing for a data center. This means networking equipment needs to be energy efficient. On top of that, networking equipment should be able to allocate resources dynamically, as needed, and without delay, to ensure the energy efficiency of other IT resources.

When the IT energy efficiency discussion was in its infancy a few years ago, it was O.K. to discuss each category of IT equipment in a data center separately to see how much it consumed in order to control its consumption. As IT equipment is being integrated and coming to function as one system, it is time to consider the energy efficiency of the IT system as a whole.

There are a few more technologies for making networking equipment more energy efficient, but I will cover them sometime in the future.

I reported that container-based data centers are gaining some attention these days. At the recent DatacenterDynamics in San Francisco, Andreas Zoll of i/o Data Centers gave a talk on the modularization of data centers. What is the relationship between modularization and containers? Modularization is a broader category than containers; a container is one example of modularization. Actually, data centers have been constructed with customized components because each data center is different. As I tour more data centers, I find that is true for somewhat old ones. But at the same time, newer constructions attempt to use standardized components. Standardized components could allow the whole data center to be constructed cheaper because they could become a commodity, and the entire data center structure could be standardized, regardless of location and requirements.

Andreas Zoll

Zoll started his talk by giving the current status of data centers, which was not surprising to those of us who keep track of their trends and challenges. These are some of his key points:

• Continuing growth of densities (for computing, power, and cooling)
• Cooling requirements and focus
• High capex and opex
• Right-sizing hard

Container data centers could address most of those trends and challenges. He made these points:

• Rapid scalability
• Capex in line with actual demand
• Geographically agnostic
• Higher efficiency
• Repeatable
• Flexible

One of his slides shows the history of container-based data centers:

Container-based data center progression

He defined three generations of container data centers.

Generation 1:

• Box containing servers
• Vendor lock-in
• Niche audience only

Generation 2:

• Variety of cooling options
• Vendor-neutral configurations
• Customizable
• Not self-contained but requiring several vendors

Generation 3 (next generation that is not yet on the market):

• Purpose-built
• Open architecture
• Fully integrated
• Tailored for redundancy
• Maintainable
• Location agnostic
• Automated
• Internationally plug-and-play

	- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
		Data Center Projects: Project Management
	- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

Some people asked him to elaborate on his last point about the second generation and full integration in the third generation. The current generation of container comes either with IT equipment that has some power and cooling inlets or with power and cooling equipment. HP (IT equipment) teamed up with Active Power (power) to provide a whole solution. Dell’s version comes as a double-decker: the bottom is IT equipment storage and the top is a power and cooling box.

Regardless of configuration, power and cooling should be available as a modularized unit to hook the container up to the building. This reminds me of the fourth generation data center proposed by Microsoft.

What you can do with each container is limited. You still have to hook your container up to power and cooling before you put it into operation. So the entire data center should be designed to provide modularized power and cooling. How soon can you do that? It is a good idea, but I wonder how doable it is now.

While people were buzzing with this question, a VP came on stage and said i/o Data Centers would announce their latest container-based solution in two to three weeks. I suppose it will be the third generation. They planned the session well. They well knew that people would ask what "fully integrated” meant and that they would announce their new solution. That was very clever. We will see if their new solution is as good as their session planning.

After the departure of Andrew Fanara from EPA, a new person is in charge of Energy Star for data centers. She is Una Song.

She gave a talk on EPA’s efforts on Energy Star at the recent DatacenterDynamics in San Francisco.

Una Song

It was a good summary of Energy Star for data centers. Here’s a set of videos from her presentation. There was a little surprise during her talk. One of the participants started questioning why power generated with renewable energies is not counted in the computation of PUE. His claim is that power generated by clean energy should be given some credit. In an extreme argument, he says PUE could be 50, yet all the power fueling the data center might be from renewables. The absence of GHG emissions in that case should be commended rather than the bad PUE frowned upon. That interaction is captured toward the end of videos 2 and 3. This reminds me of the metric (called Data Center Performance per Energy, or DPPE) proposed by Japan’s Green IT Promotion Council.

	- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
		Energy Efficient Cooling for Data Centers: A Close-Coupled Row Solution
	- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

This metric takes the use of renewables into account. To refresh your memory, here is the definition of DPPE:

DPPE =

(IT equipment utilization * IT equipment ability)

/ (total power consumed – green power used)

DPPE was also mentioned in another session. That will come later.

1. Video of Una Song’s presentation on Energy Star for data centers-Video1
2. Video of Una Song’s presentation on Energy Star for data centers-Video2
3. Video of Una Song’s presentation on Energy Star for data centers-Video3
4. Video of Una Song’s presentation on Energy Star for data centers-Video4

It is often said that smart grid is the power grid superimposed on information and communications technology (ICT). To support smart grid, it is imperative that communications be improved and used for smart grid.

At ConnectivityWeek several weeks ago, a representative from the Federal Communications Commissions (FCC) explained that their broadband plan includes a section on energy and the environment. More specifically, it considers smart grid from the communications perspective. The entire plan can be downloaded from here.

Chapter 12 (pp. 245–262) of the plan contains a section on how communications and broadband can help smart grid.

The entire plan is 376 pages long; I will only discuss chapter 12, which relates to the theme of my blog.

Chapter 12 has four subsections:

• Section 12.1 discusses how smart grid is implemented and reinforced with the application of broadband communications technologies. It gives a good overview of smart grid 101. If you want to get a glimpse of what smart grid is, especially from the communications perspective, this is a good section to read.
  

It recommends:

1. Enhancing the communications infrastructure.
2. Using commercial service providers (who are often considered not reliable enough for this crucial infrastructure).
3. Strengthening communications security.
4. Opening up the 700 MHz Public Safety Band for utilities to use.
5. Further opening up the Public Safety spectrum for smart grid.
6. Studying the requirements of utilities for smart grid.

• Section 12.2 covers home and building aspects of smart grid from the communications point of view. Smart meters installed at homes and other buildings would make power consumption information readily available for utilities and consumers. This section recommends:

1. Mandating utilities to provide consumers’ use data and maintain the privacy of this data at the state level.
2. Doing the same for the Federal Energy Regulatory Commission.
3. Doing the same for the Department of Energy.
4. That the Rural Utilities Service provide loans to rural utilities to implement smart grid in rural areas of the United States.

• Section 12.3 considers ICT as a whole in regard to power consumption and greenhouse gas emissions. It discusses PCs as well as data centers. This section recommends that:

1. The FCC start working on the communication industry’s energy efficiency and environmental impacts.
2. The federal government start working on the energy efficiency of data centers.

• Section 12.4 is about smart transportation. Since this is a report on broadband, rather than electric vehicles (EVs), this section is about how smart communications technologies can help reduce gas consumption by providing appropriate information, such as how to reroute to avoid traffic congestion.

	- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
		Making Large UPS Systems More Efficient
	- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

If you have time to read the entire report, it would certainly help you grasp what the FCC is considering in the way of improving the nation’s communications infrastructure. But if you are daunted by the length, chapter 12 alone would give you a good glimpse of how ICT could benefit smart grid. I have not seen any article or report discussing smart grid from the communications perspective before.

By the way, my friend in Saratoga, CA (which is pretty affluent), still complains of the lack of broadband services to his city. Saratoga is the community next to mine, but we have a choice of DSL and cable. My friend often tweets and expresses his opinion on the sad status of broadband services in the U.S. I just cannot imagine that that status is caused by the lack of technologies. It must be something to do with policies and politics. But it must be frustrating not to have broadband services at home.

I can relate to the frustration. Incidentally, my wireless router is acting up and drops the Internet connection sporadically. When I am not at home to fix it (cycle the power?), my wife gets angrier by the day. It is about time for me to consider a new router with 802.11n, which is somewhat pricey compared with the one with 802.11g. But again, this is a good excuse for me to purchase one.

This is a quiz. What is the relationship between C++ and smart grid? One of Fujitsu’s smart grid symposium sessions this week was a keynote speech by Bjarne Stroustrup of Texas A&M University.

Bjarne Stroustrup

He is the creator of the C++ language.

C++ was conceived from C and Simula.

 I would not mind talking about C++ and other programming languages for the next ten blog posts, but this is a blog about energy efficiency, green IT, cloud computing, and smart grid. It is Fujitsu’s fault for choosing him as a keynote speaker, which brought my mind back to pure IT.

In any event, as more and more objects are equipped with the ability to send messages back and forth, the "Internet of things” is everywhere.

The theme of Stroustrup’s presentation was how to bridge the gap between applications and hardware using C++ and C++0x (the latest standard version), as shown here.

Bridge the gap between applications and hardware using C++ and C++0x.

After talking about what is not an infrastructure, he listed the necessary conditions for being an infrastructure.

	- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
		The Different Types of UPS systems
	- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

What is required of infrastructure?

He also mentioned the difficulty in programming in the gap, which is supposed to connect applications and hardware. He said the interaction between the lowest layer of the infrastructure and hardware is very complex. For example, Intel has about 200 software engineers working on a compiler to produce appropriate code to link the lowest layer and hardware.

Programming in the gap is difficult.

 So the question is, how do you implement such "things” in an efficient way? And the answer is to use C++. See the next picture.

 Why C++?

Stroustrup did not say so explicitly, but he was talking about embedded systems. In the world of embedded systems, unlike the world of the Web, C and C++ still dominate. C can be very low-level language and interact well with hardware without requiring a lot of such resource. C++ is replacing C because C++ improves many of C’s problems, but I am not going there in this blog.

His conclusion as to how to bridge applications and hardware is given in the following slide.

When NTT Facilities throws a party, it knows how to do it right. (NTT Facilities is one of many NTT companies, and it specializes in the inside infrastructure of data centers.) Following the conference presentation last week in Tokyo,  the other attendees and I were bussed to their research center, about an hour from downtown Tokyo. The research center is in the same place as the NTT Central Research Laboratories. 

It is a nice, cozy suburban environment without any distractions (nothing else to do except for conducting research).

There we saw the state-of-the-art technologies being developed by NTT Facilities. We saw three research projects (only a subset of all the research) going on for data centers:

1. Earthquake simulator
2. Airflow simulator
3. Environment simulator

When I say a simulator, it is not a software simulator but a real system consisting of a lot of iron. Let me briefly describe each.

Earthquake simulator

Japan has a reputation as the center of earthquakes, especially in the Tokyo metropolitan area. Since Tokyo is the center of economy and politics, earthquake damages to its infrastructure would be catastrophic, including for IT and communications. Yet not many businesses leave Tokyo because of that. The second-best idea is to get ready for imminent earthquakes. What does that mean for data centers and the equipment in them?

	- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
		Monitoring Physical Threats in the Data Center
	- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

Equipment, especially IT devices, is sensitive to shaking. For example, disk heads may crash if the equipment is shaken too hard. If there is a way to mitigate such shaking, the equipment may survive an earthquake of considerable size. But how do you prepare for that without actually shaking your racks full of sensitive IT equipment? As shown in the following picture, NTT Facilities developed an earthquake simulator.

It is a giant platform with a few arms that can move and shake the platform vertically and horizontally, as shown in the two pictures.

(Note: All the pictures in this blog are due to courtesy of NTT Facilities, Inc.)

The first picture shows the size of this simulator. The second picture shows some details from above, such as the vertical and horizontal arms.

NTT has offices throughout Japan, and each is equipped with meters to record earthquake information. In addition to the data recorded by Japan Meteorological Agency, it has its own recording of the major earthquake in the Kobe region, which was not known as a site for major earthquakes prior to 1995. Initially, NTT Facilities did not allow us to take pictures. I can say this: even if you could take pictures of this sophisticated simulator, it would be impossible to copy what they developed. It involves elaborate manufacturing technologies and precise software to repeat the exact earthquake waves.

Airflow simulator

The second tour was of a chamber for experimenting with the control of airflow in a data center. NTT Facilities has a product to contain a cold aisle, also known as aisle capping. Two aisles are set up, with and without capping, and airflow and temperatures at multiple locations are measured.

The picture shows an aisle without capping.

Environment simulator

The last tour was of a chamber for controlling the environmental parameters (temperature and moisture) to test CRAC units for outside. CRAC units are usually deployed in a pair, one inside and the other outside. The outside unit is affected a lot by outside environmental conditions. It would be nice to be able to change the parameters and test them in a simulated environment.

This picture shows this chamber, which can produce temperatures ranging from 14 °F to 122 °F. When we entered this chamber, it was set to 14 °F to simulate a typical temperature at Sapporo (the biggest northern city) in winter. We only stayed there for five minutes because it was too much.

This tour exemplifies the fact that Japan has a lot of technologies in progress and sometimes shows them to the public (as long as they speak Japanese) but is very shy in presenting them in English. I do not think it is because of their secrecy but the lack of language ability to do so. I will keep publishing interesting news.

In a typical conference, people start to disappear after 3 p.m. It was quite different at ConnectivityWeek this week. At 5 p.m. (just before a reception with cocktails and food), Vint Cerf, Father of the Internet, gave a keynote speech. A pretty good-sized room, called Theater, was packed with people to hear what he had to say.

Vint started with the history of the Internet and said that in spite of popular belief (to survive nuclear attacks), the Internet’s initial objective was to share computing resources. There were only four nodes then.

He then described how the Internet should change to accommodate the new situation when every object has an identity and can communicate with any other object. You can easily guess what the new requirements will be like: large IP address space (32 bits for IPv4 vs. 128 bits for IPv6), and authentication/encryption of each packet for secrecy and authentication. Another point that struck me was that adaptability will be important because those objects may move around to exploit many different communication venues. You cannot assume only a handful of ways for communication. He said that the most difficult problem will be error caused by misconfiguration, which is hard to detect. A good way to detect misconfiguration early would be a good technology to develop.

	- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
		Energy Impact of Increased Server Inlet Temperature
	- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

I do not summarize what he said here, but you may want to see his entire presentation. It is worth 60 minutes of your time.

• Vint-Cerf-keynote-1
• Vint-Cerf-keynote-2
• Vint-Cerf-keynote-3
• Vint-Cerf-keynote-4
• Vint-Cerf-keynote-5
• Vint-Cerf-keynote-6

When Vint Cerf Speaks, Everyone Listens

I have covered many topics as they relate to energy efficiency for data centers. I want to extend that coverage by gathering the experts in each specific topic for a meeting. A panel discussion is a good way to do that.

I have a few ideas that I will use one by one in my blogs as proposals to potential conferences and panelists. One area I have spent a good deal of time on is metering in data centers. I have visited several metering companies and interviewed their executives. For most of those companies, I have written a blog post:

• Arch Rock
• Modius
• OSIsoft
• Sentilla

In addition, I have talked to Liebert/Aperture and SynapSense. Although they are all in the same monitoring and metering space, each company has a slightly different angle. Some measure via their own sensors and aggregate the data for display. Others do not gather data by themselves but exploit the data from other sources. So the functions can be roughly classified into three categories: measuring (via sensor), aggregating, and analysis and display.

	- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
		Energy Efficient Cooling for Data Centers: A Close-Coupled Row Solution
	- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

Also, in addition to measuring power consumption, different approaches can be taken to common functions like:

• alarm management
• asset management
• capacity analysis
• efficiency analysis
• air control automation

In the first panel, I would like to discuss the following so that a general audience can understand the needs of metering, learn which technologies are state-of-the-art, and assess the minimally necessary functions of metering. We would discuss the following:

• Why do data centers require monitoring and metering?
• What is a typical architecture for measuring, aggregation, analysis, and display?
• What are the minimally required functions of metering?
• What kind of standards should be defined for metering? What type of data should be collected? Granularity of data? Frequency of collection? Data formats (like XML)? Display formats?
• What extensions, such as e-waste and water, should be considered for existing metering?

An ideal panel will consist of three or four vendors and one researcher in the space. I plan to send this blog to the vendors I talked to before, at least one candidate conference, and a research organization, like a university.

Next, a proposal for a panel discussion of IT energy efficiency.

It has been reported that both the airline and the IT industries are equally responsible for the emission of greenhouse gases—2% each of the total.

The volcanic eruption in Iceland grounded planes in most European airspace for six days. The "Information is Beautiful” website reported interesting statistics.

They calculated the saving of CO₂ emissions resulting from flight cancellations during that period. The total is the saving realized by no flights minus the emissions caused by the eruption.

So this is the equation:

344,109 tons (saved) – 150,000 tons (added by the eruption) = 206,465 tons

This figure is only for European airlines. I know that many other flights between Europe and Asia and between Europe and the U.S. were cancelled. So the saving is much more.

This figure poses a few interesting questions. First, the cancellations caused many business activities and personal lives to be disturbed. But at the same time, we saved a lot of CO₂ emissions. We need to ask ourselves which is more important: to reduce CO₂ emissions or to preserve our business and personal activities. Of course, there should be some balance between the two. But how do you draw a line to decide the right balance? With the advent of the online world, we can communicate with people around the globe without leaving the comfort of our home. But that cannot replace physically being there with your business associates or loved ones.

	- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
		Making Large UPS Systems More Efficient
	- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

Second, IT is equally to blame for the emission of CO₂. What if we decide to curb the use of IT in order to curb the emissions from its use? We need to assess how much IT is saving CO₂ emission by telecommuting, replacing business trips, and so on. In addition, IT saves power consumption (thus, less CO₂ emission) by optimizing supply chains and fleet management. Of course, IT consumes power as it functions. IT is used not only for absolutely necessary functions but also for entertainment and enjoyment like online video sharing and SNS. Again, some people may think the latter functions are absolutely necessary, while others can do without them altogether. Again, it is hard to draw a line between the two.

On top of this, if you throw in developed countries vs. developing ones, the discussion becomes much more difficult to deal with. I wonder whether we will eventually come to some kind of agreement?

I’ve conducted several interviews with companies in the data center energy efficiency space. Sometimes, repeatedly bumping into someone from one of those companies at industry conferences and meetings prompted my interview. Sometimes, I have been introduced by someone who knows someone at such a company. My encounter with Craig Compiano of Modius was very different from the others.

As I have been reporting in my blog, I attended Data Center World in Nashville, TN, in early March. Craig gave a talk there regarding the importance of measuring and metering for data centers, from the Modius perspective.

At that time I was completely confused about this space (metering/measuring). Well, you put your own sensors or someone else’s at strategic locations (including both IT and facilities equipment) in a data center. Then, you collect and gather data, then you aggregate and analyze them. Finally, you display them in a dashboard. I never belittle the efforts to make a product out of any technology, but this seemed pretty straightforward—maybe too straightforward to consider as an independent space.

So after Craig’s talk, I went to him and asked this question. Maybe my tone of voice was not very friendly, but he gave me his answer. If I remember correctly, he told me that I needed to evaluate each company’s offering from many angles, including comprehensive device coverage, supportability, and scalability. Obviously, I was not convinced, and my face showed it. Craig said he could spare more time if I wanted. I thought I was going to do that if I had time. Then, at the Modius booth on the show floor, I bumped into Adam Waitkunas, whom I got to know by attending a few data center conferences and meetings. I told Adam my problem of not being able to differentiate those metering companies. He told me that he could arrange a meeting with Modius. So that was why I got to talk to Craig again on a hot San Francisco day.

When we sat down in his conference room, I was eager to find out the differentiation. He started drawing pictures on a whiteboard, as shown below.

Craig Compiano

It was deja vu. I remembered this scene from somewhere. Then it came to me. More than fifteen years ago, Jim Clark gave me the same kind of chalk talk about Mosaic Communications (later Netscape). Because of this deja vu, I knew that Craig is a technologist as well as a chief executive. And he certainly is. His view was quite interesting. The following is my version of his picture in a more abstract (less artistic) way.

Modius view of metering/measuring space. The box in light beige has been implemented by Modius, the one in light green is someone else’s, and the box in light red is being implemented.

His explanation follows.

What Modius did first was to build a middleware infrastructure to support several functions required at data centers. The middleware is enclosed by a dashed line in the picture. This middleware infrastructure is the cornerstone of the Modius technology; it allows scalability to support any number of data center locations around the globe. The database is also an important piece of the equation; it needs to collect, organize, and retain a huge amount of data for analysis, display and sharing with various applications.

A set of necessary functions (or applications) have been implemented on top of the middleware infrastructure. The five leftmost functions are those typically implemented at most data centers today:

• Real-time Alarm Management: Data set by various sensors regarding health of equipment and power consumption
• Asset management: Inventory of IT and facilities equipment
• Service desk: Help desk function
• Capacity analysis: Current and future power and cooling capacity
• Efficiency analysis: Real time analysis of the efficiency of DC power and cooling assets.

The last box is automatic air control, which controls the airflow and temperature of CRAC units on the basis of the inlet air temperature at each server in the data center. I am familiar with this technology because I edited the case study for the California Franchise Tax Board data center. Federspiel is marketing this technology. See in here under the header of "Control of Computer Room Air Handlers Using Wireless Sensors - Franchise Tax Board"

Modius is currently working on this technology, and has a product plan to extend their functions even further, but Craig is not sharing more information on that at this time.

Back to my basic questions. What is the differentiation? With this picture on the board, it was straightforward.

• Enough functions on top of the middleware structure. This is a necessary condition but is it sufficient? I am sure new applications will be added over time all fed by the same real-time source data. How many more functions are required to be enough? This needs more thinking and analysis.
• The infrastructure, including the middleware. Can it scale as an enterprise wants to add more locations and more applications into a single converged system? Does it have an easy interface to add new devices and equipment as they become available in a low-cost manner?

	- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
		Ten Cooling Solutions to Support High-Density Server Deployment
	- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

What about the big four in network/system management? Are they coming into this space anytime soon? Craig does not think so. The complexity and disparity of infrastructure assets being addresssed by metering/measuring companies and the highly varied consumers of the metric data is not necessarily tightly linked with network/system management. But who knows? If and when this space matures, this space may be merged into network/system management to form a new space.

So far Craig’s explanation was very useful to analyze the market space. If I had not approached him after his talk, would he have taken the time to share his thoughts with me? The talk with him has solved some of my confusions.