If you’re not familiar with log data, think of it as the Twitter feed of the datacenter – each piece of software and hardware emits a constant chatter of status updates. These status updates (short text messages called log messages) provide rich information on the state of the environment, and the actions of individual components within that environment. With log analytics, you not only have the ability to detect trends and issues in real-time, but because of that insight, you can resolve those issues faster.

Log data provides a critical yet greatly underutilized source of information on the health of your IT environment. When an IT issue occurs – say a transaction is failing, a service becomes unavailable, or a significant performance degradation occurs – IT workers must turn to the logs to identify the root cause of an issue. Log analysis is both critical and time-consuming. When we surveyed our customers, we found that the majority of our customers need to analyze logs every week. Even in the smallest IT installations, the volume of log data is large, and analyzing log data with traditional tools is a tedious, time consuming, and error-prone activity.

As the CTO for the Log Insight product, I’ve been eagerly waiting this moment where I can finally talk about what the Log Insight team has been working on! So enough background, let’s take a deep dive (demo-style) into what Log Insight does and how it can help you.

First, we’ve made deploying, configuring, and getting started with Log Insight incredibly easy, particularly if you have a vSphere environment. Just deploy the virtual appliance, point a web browser at it, and walk through a short configuration wizard. Because Log Insight supports all the variants of the syslog protocol, you can quickly configure hardware, OS, and applications to send their logs to Log Insight. To make it really easy for vSphere environments, we provide a utility that configures logging across your vSphere environment with a single step. In less than 30 minutes, you get your first insights! No need to bring in an expert. Want to retain more data? Just add a virtual disk. Want to support more concurrent queries? Add more vCPUs or more vRAM. vSphere makes it really easy!

Once the logs are flowing, Log Insight provides immediate insight into your vSphere environment through packaged knowledge that we’ve gathered from the VMware support and engineering teams. This packaged analysis shows up in Log Insight through a mechanism that we refer to as a “Content Pack.” The built-in vSphere Content Pack includes dashboards, saved queries, field definitions, and alerts. This image gives you a sample of one dashboard in the vSphere content pack:

Through this content pack mechanism, we are putting some of the same tools and knowledge in your hands that our own support teams use to resolve customer support cases. We find that customers learn something interesting (or even concerning) about their vSphere environment during their first use of Log Insight, even if they have never looked at a vSphere log before. For example, in the picture below, the chart on the right allows a customer to immediately see trends in vSphere alarms.

The content pack mechanism also allows third parties to easily create and distribute their own content packs. Every application or hardware component generates unique logs and those logs contain rich information – knowledge about the structure of messages into those logs and the most effective analytics and visualizations for those logs can be packaged into content packs. Content pack authoring is a big topic that we can cover in detail in a later post, but its super easy to export saved analytics and visualizations as content packs for others.

Underneath these colorful charts is a powerful engine that allows you to perform analytical queries on unstructured data. This provides significant agility advantages over a traditional database environment. In a traditional database environment, we are handicapped by the need to make data fit a schema. First we must foresee what questions we want to ask, then design a schema accordingly, and transform the data into that schema (aka ETL – Extract Transform, Load). This time- and resource-consuming process creates significant friction to asking new questions or generating new reports. Log Insight provides a tool that allows database-like analysis of data embedded in log streams without the traditional ETL overhead.

One way that this is achieved is through full-text search. Like “Google for your logs”, but with the additional ability to visualize and analyze the aggregate results of that query. The following screenshot shows how I quickly found all exceptions across my entire infrastructure that occurred in the past six hours.

But that is just the beginning. Where Log Insight sets itself apart is in the ability to dynamically apply structure to unstructured data, and construct database-like queries and visualizations without any query language. To understand this capability, let’s follow an example through the system.

Assume that I’m the administrator of a private cloud powered by vSphere. One of my internal customers reports performance issues experienced during vSphere administrative activities – such as launching virtual machines (VMs), reviewing VM inventory, etc. These management services are provided by VMware vCenter® Server, and most vCenter performance issues can be traced to performance issues with the 3rd-party SQL database that supports it. Suspecting SQL, I point my web browser at Log Insight and search for SQL-related log messages (image below):

Sure enough, I’m seeing some log activity that suggests that vCenter thinks the database server is too slow. The chart at the top gives me some further information – I’m seeing SQL logs happening periodically over the past six hours. To understand better the scope of the potential issue I’d like to answer questions like: “when did this error start?” and “is this only occurring in one datacenter?”

To understand when the error started, I can increase the time range of the search query. When I get to the last last seven days, I can see that I’ve been predictably having SQL performance issues since mid-day May 31st (image below).

Next question – does this affect any instances of vCenter beyond the one that I see in the first 50 results? All log messages that arrive within Log Insight are tagged with the hostname of the node that originated the message. The screenshot below shows the Fields view on the right side of the screen that allows us to quickly do a breakdown of a query by a field.

From the field breakdown chart on the right, we see that this error only occurs on this single vCenter server named ‘strata-vc.’

Now to the last question: “is there a particular query that is causing the issues?” I need to apply structure to the unstructured messages, because the information on the SQL query is embedded in the text of the message and has not been extracted as a field. But no problem! Watch how easy this is. With my mouse I highlight the embedded field in the message (see blue highlight in image below):

Note that when I select text in the message with my mouse, a new ‘Extract Field’ button appears. In this case I’ve highlighted the SQL query “BEGIN; select insert_stats_proc(?, ?, ?, ?); END;”. When I click ‘Extract Field’, Log Insight automatically constructs a pattern to dynamically extract that field. Log insight also gives me a preview of the results of applying the field extraction to the other messages in the view. The dark green highlighted text in the image below indicates the text in each message to be extracted to form the new field. I give this newly discovered field the name ‘slow_sql’.

Now that I have defined this new field, it becomes available on the list of fields on the right side. I can quickly expand it (image below) to discover that there are multiple SQL queries that are experiencing performance issues.

By clicking on the mini-chart, I can also promote it to the primary chart. Based on this visualization (shown in image below), I can see that there is one particular SQL query (insert_stats_proc()) that experiences the overwhelming majority of performance issues.

In a few minutes, Log Insight’s analytical capabilities have helped me to narrow down the scope significantly. Now I know that there is a performance issue caused by a database server. I know which database server. And I even know which SQL query. I could call my database administrator, and leave it to him to figure the rest out. But we’re not done yet! Let’s bring in VMware vCenter Operations Manager, which provides advanced operational analytics for numeric times series data. vCenter Operations Manager and Log Insight is integrated out of the box, starting with Operations Manager 5.7.1. So now, vCenter Operations customers now have the ability to analyze this critical log data and deeper insight into the health of their environment.

vCenter Operations Manager comes with built-in integration with vSphere, and has been monitoring the performance statistics across my vSphere environment. I’m going to use the integration between Log Insight and vCenter Operations Manager to solve two problems at once – first, to leverage vCenter Operation Manager’s advanced analytics to refine the possible root cause of the performance issues. Second to create an alerting mechanism so that in the future I’m notified immediately if any vCenter SQL query experiences slow performance. From the Interactive Analytics page, we create an alert (image below):

The alert dialog allows us to choose how the alert should be delivered. One mechanism is via email. The next mechanism is via a notification event sent to vCenter Operations Manager. From the screenshot below, you can see that I selected both options.

Within a few minutes, I receive an alert within vCenter Operations Manager, shown below.

Most interesting is the “Root Cause” section of the alert window – vCenter Operations Manager has applied its analytics to identify that the most likely culprit for the event was memory pressure on the cluster on which vCenter and the vCenter SQL database are running (actually a host within the cluster). This makes sense because in this cluster, vCenter and the SQL database share a compute and memory pool with other virtual machines. So I now know enough to take action – options include reducing the number of virtual machines in the cluster, creating memory reservations for the SQL database, or adding more RAM to the hosts in the cluster.

In summary, Log Insight extends VMware’s operational analytics into log data with software that is engineered to be very easy to use and quickly provides you with insight into your IT environment. Its integration with vCenter Operations Manager provides unified analytics across numeric and unstructured data, and comes pre-loaded with rich knowledge about how to identify and diagnose issues in your vSphere ecosystem.

Hopefully this blog post has given you enough of a taste of Log Insight and you’re ready to take the next step – check out the product. Log Insight is expected to be generally available for purchase in Q3, but no need to wait until then! A fully functional Beta version of Log Insight is now available for download. Download link and documentation to get you started can be found in the Log Insight Beta Community on My VMware. Or check out below the product videos we created that show demos of how to install and use Log Insight.

Building upon this work, the DMTF today announced the Open Software-Defined Data Center Incubator – a forum where the IT community can discuss and develop definitions, architectures and use cases for a software-defined data center that will be interoperable via open and standard interfaces. This is similar to the work it took on several years ago doing the same for Cloud Computing. The incubator establishes a venue for the industry to come together and create foundational whitepapers on what SDDC is, how it will be used and identify the gaps in the standards that will be needed to provide the interoperability and choice customers demand.

The goal of this group is to develop the foundational documents over the next 12 months that will enable the industry to create further standardization of management infrastructure for the next generation of data centers. The Incubator brings the industry together to collaborate and create the blue print for the next generation of IT which in turn puts customers on a path for improved agility, flexibility and new levels of automation.

I think the activity will help clarify the road to achieve the vision and the promise of SDDC. And at the same time, we’ll see the positive side effects of standards, which are increased choice for the customer, reduced cost for the vendors and improved interoperability of data center and cloud computing environments.

Are you on the path to building a SDDC? What tools do you need to get there? Please share your feedback and comments below.

Building upon this work, the DMTF today announced the Open Software-Defined Data Center Incubator – a forum where the IT community can discuss and develop definitions, architectures and use cases for a software-defined data center that will be interoperable via open and standard interfaces. This is similar to the work it took on several years ago doing the same for Cloud Computing. The incubator establishes a venue for the industry to come together and create foundational whitepapers on what SDDC is, how it will be used and identify the gaps in the standards that will be needed to provide the interoperability and choice customers demand.

The goal of this group is to develop the foundational documents over the next 12 months that will enable the industry to create further standardization of management infrastructure for the next generation of data centers. The Incubator brings the industry together to collaborate and create the blue print for the next generation of IT which in turn puts customers on a path for improved agility, flexibility and new levels of automation.

I think the activity will help clarify the road to achieve the vision and the promise of SDDC. And at the same time, we’ll see the positive side effects of standards, which are increased choice for the customer, reduced cost for the vendors and improved interoperability of data center and cloud computing environments.

Are you on the path to building a SDDC? What tools do you need to get there? Please share your feedback and comments below.

When the vCloud Hybrid Service architecture team began designing the service we followed two core tenets; simplify the physical aspects of the design and use VMware software wherever possible and practical. The goal was to deliver a high-quality cloud service that fully embraced the concepts of the software-defined data center.

vCloud Suite 5.1 was in development while we were designing the vCloud Hybrid Service architecture, and fortunately for us many of the new features in the vCloud Suite were directly applicable to our design tenets. For example, we saw an opportunity to use the vCloud Networking and Security (vCNS) software capabilities to radically change our networking design.

The resulting network design is simple from a physical perspective. Basic switching and routing capabilities are provided by hardware while more complicated functions are provided by the vCNS Edge appliance in combination with VXLAN. Throughout the course of our development period, we worked closely with the vCNS engineering team to refine our design and deploy it in production.

The vCNS software suite is a feature-rich-networking platform, and here are some of the numerous capabilities we used from vCNS Edge:

• Load Balancing – Used to provide Internet facing services to customers
• SNAT – Used to provide outbound access where required
• SSL and IPSec VPN – Remote access capabilities
• Routing – Routing between internal networks as well as to the Internet
• VXLAN – Isolate customer networks allowing for bring-your-own address space
• APIs – Manage and deploy virtual network infrastructure through automation

In combination, these capabilities provide resilient, high performance core networking services for our customers. Most of these features are also available for customers to employ directly in their vCloud Hybrid Service cloud environment, and all of the vCNS networking technology we are using is available today for on-premise deployments.

The end result is that we now have an agile, programmable network infrastructure that will provide an easy road to upgrade as we continue to expand the capabilities of vCloud Hybrid Service. I’ll cover other parts of our vCloud Hybrid Service architecture in future posts – but in the meantime, feel free to comment below around what’s important to you in a cloud service’s networking infrastructure.

VMware Horizon Mobile Overview

Perhaps even more profound than the BYOD trend is the change in how employees use their devices. Irrespective of who actually buys or owns the device, the corporation or the user, most employees tend to download personal apps onto these devices – Facebook, Angry Birds, Temple Run, etc. coexist with work email/PIM. It is fair to assume then that most devices will have both personal and corporate content (apps, data and services).

Given that the usage paradigms have changed, IT needs to rethink security and manageability of mobile devices. The old BlackBerry model of locking and wiping the device is no longer in line with how employees use their devices. IT administrators can now leverage VMware Horizon Mobile to isolate personal content from corporate content and only manage the corporate content on the device. The corporate content resides in a “workspace” whose lifecycle and usage is managed by IT. IT can customize what apps are in the workspace and what policies are applied to the workspace, provision the workspace to the user’s device over the air (OTA) and then manage its lifecycle remotely.

If you look at the latest mobile OS market share information, Android is way ahead of other mobile operating systems but if you consider the enterprise subset of that market share, iOS is the dominant platform. One of the reasons for Android not being dominant is its fragmentation, which makes it very difficult for IT to wrap their hands and heads around a comprehensive security and manageability story for Android devices. VMware Horizon Mobile leverages device virtualization to normalize that fragmentation and allows IT to deploy and manage its own Android workspace that looks and behaves the same on any Android device. However, in order to run this solution, you must have a VMware Ready device and hence, the importance of today’s announcement.

You can see the product in action here.

VMware Ready Program

VMware Ready designates VMware’s highest level of endorsement for products and solutions created by our established partners – and on the mobile side, a VMware Ready device is required to experience our dual persona solution. In the US, VMware is partnering with Verizon to enable a broad set of new and existing devices to become VMware Ready and it’s important to note is that existing in-market devices can be updated over-the-air (OTA) to become VMware Ready devices.

Indeed this is the case with LG Intuition and Razr M by Motorola. Both of these devices have been in the market for several months and recently received a software update. The update pushed out by Verizon included the right VMware technologies to enable them to run a second instance of Android and thereby, our Horizon Mobile solution.

Our customers should expect popular in-market devices will receive similar software updates resulting in a broad set of VMware Ready devices in the market. In fact, we anticipate many new devices will also launch as VMware Ready devices.

We’re looking forward to seeing many more VMware Ready devices in the market and if you’re an Android OEM interested in getting involved with the VMware Ready program, please contact us at mvp-oem-public @ vmware.com for more information.

On a more personal note… Over the weekend, I had some time to review some of the strategy documents and initial business plans that I put together when we first started on the mobile initiative. It is personally gratifying to see first hand how all this has come together and reflect on the journey that got us here.

It’s been a great experience getting to this point and we’re proud to walk hand in hand with LG to deliver the first VMware Ready device on Verizon Wireless. My sincere thanks to all the wonderful folks at LG who partnered with us and worked tirelessly through the various stages of our product development and betas to enable the LG Intuition as the first VMware Ready device on our partner’s network.

So what do you think of our approach to managing enterprise mobile users? Do you like the idea of managing just the corporate workspace and not the entire device? Share your thoughts with us in the comments section below.

VMware Horizon Mobile Overview

Perhaps even more profound than the BYOD trend is the change in how employees use their devices. Irrespective of who actually buys or owns the device, the corporation or the user, most employees tend to download personal apps onto these devices – Facebook, Angry Birds, Temple Run, etc. coexist with work email/PIM. It is fair to assume then that most devices will have both personal and corporate content (apps, data and services).

Given that the usage paradigms have changed, IT needs to rethink security and manageability of mobile devices. The old BlackBerry model of locking and wiping the device is no longer in line with how employees use their devices. IT administrators can now leverage VMware Horizon Mobile to isolate personal content from corporate content and only manage the corporate content on the device. The corporate content resides in a “workspace” whose lifecycle and usage is managed by IT. IT can customize what apps are in the workspace and what policies are applied to the workspace, provision the workspace to the user’s device over the air (OTA) and then manage its lifecycle remotely.

If you look at the latest mobile OS market share information, Android is way ahead of other mobile operating systems but if you consider the enterprise subset of that market share, iOS is the dominant platform. One of the reasons for Android not being dominant is its fragmentation, which makes it very difficult for IT to wrap their hands and heads around a comprehensive security and manageability story for Android devices. VMware Horizon Mobile leverages device virtualization to normalize that fragmentation and allows IT to deploy and manage its own Android workspace that looks and behaves the same on any Android device. However, in order to run this solution, you must have a VMware Ready device and hence, the importance of today’s announcement.

You can see the product in action here.

VMware Ready Program

VMware Ready designates VMware’s highest level of endorsement for products and solutions created by our established partners – and on the mobile side, a VMware Ready device is required to experience our dual persona solution. In the US, VMware is partnering with Verizon to enable a broad set of new and existing devices to become VMware Ready and it’s important to note is that existing in-market devices can be updated over-the-air (OTA) to become VMware Ready devices.

Indeed this is the case with LG Intuition and Razr M by Motorola. Both of these devices have been in the market for several months and recently received a software update. The update pushed out by Verizon included the right VMware technologies to enable them to run a second instance of Android and thereby, our Horizon Mobile solution.

Our customers should expect popular in-market devices will receive similar software updates resulting in a broad set of VMware Ready devices in the market. In fact, we anticipate many new devices will also launch as VMware Ready devices.

We’re looking forward to seeing many more VMware Ready devices in the market and if you’re an Android OEM interested in getting involved with the VMware Ready program, please contact us at mvp-oem-public @ vmware.com for more information.

On a more personal note… Over the weekend, I had some time to review some of the strategy documents and initial business plans that I put together when we first started on the mobile initiative. It is personally gratifying to see first hand how all this has come together and reflect on the journey that got us here.

It’s been a great experience getting to this point and we’re proud to walk hand in hand with LG to deliver the first VMware Ready device on Verizon Wireless. My sincere thanks to all the wonderful folks at LG who partnered with us and worked tirelessly through the various stages of our product development and betas to enable the LG Intuition as the first VMware Ready device on our partner’s network.

So what do you think of our approach to managing enterprise mobile users? Do you like the idea of managing just the corporate workspace and not the entire device? Share your thoughts with us in the comments section below.

The most basic question customers ask about hybrid cloud is whether it makes sense for their organization to adopt or not. As you can imagine, there are tradeoffs in security, performance, SLA guarantees, cost, and much more between private, hybrid, and public clouds. Each organization will have different constraints and priorities. Thus the decision of whether to move to hybrid or public cloud – or more specifically what apps, data, and services to move – will be different for each organization.

In my session, I talk about how to architect your datacenter to provide the flexibility to make changes over time. The right architecture will enable you to move between private, hybrid, and public cloud seamlessly, without impact to services and be completely transparent to users. And this is exactly the point of hybrid cloud: it’s about flexibility and choice. But in order to avoid lock-in, you need the right architecture.

But what does the right architecture look like? We’ve found that the answer is a self-service architecture. While it may seem a bit of a leap from hybrid cloud to self-service, the point is that the cloud that you use to run your infrastructure should be considered an implementation decision, not an architectural decision. And in order to achieve that, you need the right abstraction layer between your users and services and the backend infrastructure they utilize.

Ok, so how does self-service achieve this? It’s instructive to look at non-technical, real-life examples for inspiration. In my talk at CloudConnect Silicon Valley last month, I used the example of FedEx. As a customer, FedEx gives me a very simple interface with three input variables: where my package is going, how much it weighs, and when I want it to get there.

Assuming the price is right, I print my shipping label, put it on my box, and drop it at a drop location. This is all I see as a customer. However, on the backend, there is an amazing amount of complicated logistics to get my package to its destination on-time. But really, as a customer, do I care if FedEx uses a truck versus a plane to get my package to its destination? Do I care that they optimize to avoid making left turns? No – so long as my SLA is met, I don’t. Thus exactly how FedEx gets my package to its destination is an implementation detail that I as the customer don’t see or care about. More importantly, they can change this implementation detail as frequently as they’d like without affecting me (so long as my SLA is met!).

The question before us is how we can architect a datacenter such that we have the same clean separation between the customer (user) and the backend implementation chosen by IT. We believe that self-service is the answer our customers are looking for.

So what should you be thinking about if you want to build a self-service datacenter? The first piece of the puzzle is a self-service portal. This is a place where users can provision new services and manage their existing services. Like FedEx, each service would have an associated cost and users would need to decide if their business needs justify the cost of the service. If they do, then the user can start the process for provisioning that service. Whether that service is provisioned into the private cloud or the public cloud is of little concern to the user so long as the SLA they specified is met. And that’s exactly the point: all the user sees is the self-service portal, they’re unaware of exactly where this service has been provisioned.

It’s important to note the importance of this self-service interface. Typically in today’s IT environment, all requests for new services are made through a ticketing system. Within this system, the user requesting the service must provide very detailed specs for every aspect of their service, sometimes including the physical hardware or virtualization solution their service will run on. Because the user is deeply involved in specifying the backend infrastructure for running their service, it means IT has little wiggle room in case it decides to change vendors or technologies. This significantly ties IT’s hands and results in “silos” being created – different technology stacks for different applications (e.g. the Windows stack is different top-to-bottom from the Linux stack is different from the Tier 1 apps stack).

In the end, users are good at creating and running their services, not managing the underlying infrastructure their services run on. In the self-service model, IT handles the infrastructure choices and users can focus on their services. This gives IT the freedom to change technologies, move from the private cloud to the public cloud or vice-versa, test out new ideas on a small percentage of the services before rolling the change out to all of them, etc. And there are many benefits with having a clean separation between service and infrastructure implementation!

So what else should you be thinking about in build a self-service datacenter? Well, questions like “how can I automate the provisioning of services?” or “how can I prevent this self-service datacenter from turning into total chaos?” or “how do I operate a self-service datacenter?” should be top of mind. The answer in two words to all of these questions is: management tools. In particular, you’re looking for two types of management tools: cloud automation and cloud operations.

Cloud automation tools take care of wiring up and connecting all the disparate services you have in your datacenter. To provision a service, you need to contact vCenter Server to deploy the VM from a template, talk to the networking gear to set up a vLAN, portgroups, and configure network settings for the service, connect with the storage backend to provision a new LUN or NAS mount, talk to vShield or some other security service to configure firewalls and anti-virus, and much more. Previously, all of these were tasks manually performed by IT admins, which could take days. Cloud automation allows you to automate all of this so that it can be done in seconds. At the same time, to prevent the “total chaos” scenarios so many IT admins worry about in self-service scenarios, you need to be able to specify and apply policies to different services and workflows.

VMware’s vCloud Automation Center (vCAC) solves the cloud automation problem. It is designed specifically for hybrid cloud scenarios, where a company’s datacenter spans private cloud, public cloud, and even (gasp!) physical servers. There are three primary components to it: a fully-configurable self-service portal, automated workflows, and policy-based governance. The self-service portal is straightforward enough – this is the web page where users go to provision and manage services and you have very granular control over the look and functionality of it. In terms of automated workflows and policy-based governance, vCAC allows you to create “blueprints” that specify the steps that are taken after a user selects a service in the self-service portal. Sometimes that service request will need approval, say from the user’s manager or the LOB’s VP. Other times the request is small enough that no approval is needed. This is completely configurable by IT, and this is just one example of how the right cloud automation tool can allow you to avoid the theoretical chaos associated with self-service architectures by inserting the right control points. Assuming the request is approved, vCAC then provisions the service. As mentioned in the preceding paragraph, this involves connecting to APIs for many different infrastructure components, including virtualization, network, storage, and security. Again, this is all automatic based on the blueprint specification.

vCAC and other cloud automation tools are the glue that hold the system together. Moreover, they abstract away the specifics of the infrastructure from the user. All the user sees is the self-service portal. Behind the scenes, vCAC orchestrates a lot of components and potential complexity, but due to the data-driven nature of the blueprint, it makes it very easy for IT to manage it all. You can simply specify what components are involved and what you want to happen, and vCAC takes care of it for you. This is where the benefits of the architecture start to show themselves. While a blueprint might call for deployment in a private cloud, the admin could easily change it to a public cloud without any users knowing. The user would only see the service name and some of its characteristics, but would not know where it was being provisioned. Thus vCAC provides that separation layer that gives IT flexibility in moving between private cloud and public cloud (and back!).

Once services are provisioned, you then need to think about how you’ll manage their ongoing operations. This is where cloud operations tools come in. vCenter Operations Management Suite (vC Ops) is VMware’s solution in this space. Like vCAC, it was designed from the ground up for hybrid cloud scenarios – both private and public cloud (and yes – even physical servers too!). vC Ops covers a wide range of functionality, including performance, capacity, configuration, compliance, log analysis, in-guest monitoring, and much more. For this discussion, the most important items to think about are performance, capacity, and compliance. Performance issues are certainly top of mind in hybrid cloud scenarios. Public clouds often give you an SLA, but how do you know the provider is meeting that SLA? Or if there is a problem, is it on the provider’s side or yours? vC Ops has powerful analytics to help answer these questions.

With regard to capacity management, a self-service model necessitates you think differently. Typically, capacity management is done by IT on a per-request basis. A new request comes in, IT works with the user to understand infrastructure requirements, and then new hardware is bought and provisioned for that service. And this can take months. If we want to provision new services in seconds, we need to have the physical hardware ready before the request comes in. But how can we do that? Well, FedEx does it. I mean, have you ever gone to a FedEx drop location and found that all their trucks were full and they couldn’t take any more packages? Of course not! FedEx analyzes historical trends to understand customer demand and ensures that enough trucks are ready as the packages come, even for busy times like the winter holidays. Similarly, IT should start doing capacity trending – analyzing usage to understand when and where capacity shortfalls will occur. vC Ops provides these capabilities and will accurately forecast future capacity availability.

Compliance is another big issue for the hybrid cloud. IT needs to ensure that its datacenters are in compliance, irrespective of whether they’re in a private or public cloud. vC Ops provides automated configuration information collection and compliance assessment against that configuration. It will automatically flag items out of compliance and can automatically remediate them if IT so desires. This way admins can be assured that no matter where a workload is running, it will be in compliance.

You also no longer need different management tools for different environments. vCAC and vC Ops work across all your environments – private, hybrid, and public clouds – so you can create blueprints or check compliance in the same way and from the same screen regardless of the underlying infrastructure. This means that your admins can quickly get up-to-speed on new environments.

Together vCloud Automation Center and vCenter Operations Management Suite are crucial for enabling a self-service datacenter and with it, offer greater flexibility in your hybrid cloud strategy. Have you given them a try?

It’s great to see the public launch of Pivotal today. The mission — to build a new platform for a new era — is bold but appropriately targeted at some of the biggest fundamental changes in application technologies.

Pivotal is now a separate entity, bringing several teams and technologies from both VMware and EMC — including Greenplum’s Hadoop (now Pivotal HD), Greenplum Database (fused with Hadoop as a new database known as HAWQ), CETAS, Pivotal Labs, Gemfire in-memory database, the Spring Application Framework and the Cloud Foundry PaaS platform.

The goal of the platform is to enable the new wave of predictive big data applications — those which pull in vast quantities and sources of data — including high rate real time sources, and can make decisions in real time based on incoming data and learned historical behavior. Combine these technologies with the ease of application development and delivery through platform as a service for  and you have a powerful platform.

Is also great to see GE’s endorsement and investment.  While the application of these technologies to the web giants with huge user bases is clearer, GE is a good example of where traditional businesses will leverage real time analytics to fundamentally change their business. In their case, putting sensors on every product from jet engines through consumer appliances will give a full connected customer experience.

I’m often asked how the Pivotal initiative interacts with VMware’s big data efforts. VMware continues its focus on building the best infrastructure for big-data, enabling our partners’s big data products on a virtualized platform. This allows mixed workloads and multi-tennancy of Hadoop and key big-data applications on a  common infrastructure platform. Project Serengeti continues to be developed at VMware, and is our reference implementation and glue to allow Hadoop to be deployed rapidly on vSphere with key integration capabilities to allow elastic grow/shrink, integrated high availability. We continue to work with the Hadoop community and key partners on the integration of big data solutions with project Serengeti and vSphere.

Congratulations on the launch and I look forward to continue working with the team at Pivotal.

You can see today’s webcast here, and find information on the accompanying blog.

It’s great to see the public launch of Pivotal today. The mission — to build a new platform for a new era — is bold but appropriately targeted at some of the biggest fundamental changes in application technologies.

Pivotal is now a separate entity, bringing several teams and technologies from both VMware and EMC — including Greenplum’s Hadoop (now Pivotal HD), Greenplum Database (fused with Hadoop as a new database known as HAWQ), CETAS, Pivotal Labs, Gemfire in-memory database, the Spring Application Framework and the Cloud Foundry PaaS platform.

The goal of the platform is to enable the new wave of predictive big data applications — those which pull in vast quantities and sources of data — including high rate real time sources, and can make decisions in real time based on incoming data and learned historical behavior. Combine these technologies with the ease of application development and delivery through platform as a service for  and you have a powerful platform.

Is also great to see GE’s endorsement and investment.  While the application of these technologies to the web giants with huge user bases is clearer, GE is a good example of where traditional businesses will leverage real time analytics to fundamentally change their business. In their case, putting sensors on every product from jet engines through consumer appliances will give a full connected customer experience.

I’m often asked how the Pivotal initiative interacts with VMware’s big data efforts. VMware continues its focus on building the best infrastructure for big-data, enabling our partners’s big data products on a virtualized platform. This allows mixed workloads and multi-tennancy of Hadoop and key big-data applications on a  common infrastructure platform. Project Serengeti continues to be developed at VMware, and is our reference implementation and glue to allow Hadoop to be deployed rapidly on vSphere with key integration capabilities to allow elastic grow/shrink, integrated high availability. We continue to work with the Hadoop community and key partners on the integration of big data solutions with project Serengeti and vSphere.

Congratulations on the launch and I look forward to continue working with the team at Pivotal.

You can see today’s webcast here, and find information on the accompanying blog.

Read the full paper for detailed results and to learn about performance best practices for deploying Hadoop on vSphere.

Read the full paper for detailed results and to learn about performance best practices for deploying Hadoop on vSphere.

I recently had the pleasure of attending Venture Beat’s Mobile Summit and co-chairing two sessions on mobile’s effect on the enterprise with Gaurav Tewari, Director at SAP Ventures, and moderated by Dylan Tweney, Venture Beat Executive Editor. The sessions were roundtable discussions and focused on surfacing how mobile is impacting the enterprise and what issues and concerns are on executives’ minds. These were very engaging conversations that resonated strongly with me and I wanted to share some of the highlights and perspectives with a wider audience.

Security in a BYOD world was certainly a hot topic with varying perspectives. There was a surprisingly heated debate on the level of threat and vulnerability with mobile platforms. It was commonly agreed that the bulk of publicized attacks have been on corporate and cloud infrastructures, as well as Windows PCs. It was also agreed that newer platforms have better built-in hardening. The actual mobile platform threat was a hot topic of discussion. Some folks felt the risk was overblown and others touted imminent disaster due to out-of-date firmware and unpatched vulnerabilities, particularly calling out Android platforms. However, it was certainly agreed that as we move to mobile devices connected over Wi-Fi/the public internet and connecting to public cloud-based services, traditional corporate LANs become much less of a safeguard and end to end protection over an uncontrolled and potentially insecure communication path is becoming the norm.

The growing sophistication of attacks was also brought up. There was a clear consensus that security needs to be designed in by experts and applied at a fine granularity (app or data level) while still delivering a great native user experience. With a varied and heterogeneous mix of devices, diverse application frameworks and more insidious attacks, an overarching security framework for applications was a key agreed upon and as yet unmet need.

BYOD is considered table stakes today, although several folks expressed that the pendulum may swing back to corporate-owned devices. With BYOD plans, the perception is that it doesn’t really save money, but it does provide flexibility, enhance productivity and user satisfaction. 80% of companies are doing it, but only 10% actually have well thought out policies. Of particular interesting concern were MDM style device management ramifications coming as unwelcome surprises for users and the impact on their personal device and content. Here is an interesting viewpoint from Cesare Garlati, who spoke about a personal experience where his child was trying to unlock and play a game on his tablet without supervision. He tried the wrong password too many times and triggered a full remote wipe via IT policy, deleting family photos and music. It was a clear point of consensus that for BYOD use cases, fine grained policy and controls that are limited to business applications and content is required.

It was very clear that mobile is here to stay and is at the forefront of enterprise thinking. A comment was made and corroborated that the best IT departments, spur and enable mobile applications and encourage rapid innovation/iteration for business ROI. It’s not a hyperbole – mobile really enables us to not just automate physical world processes, but to create new and improved valuable workflows and operations. Enterprises are starting to use the full range of capabilities available – cameras for augmented reality, GPS location awareness as integral parts of advancing enterprise processes and backed by powerful cloud services. Attendees cited examples – looking at units of various types in the field by cameras and pulling up maintenance records. Another example was tablets used for factory inventory on forklifts. In this case, expensive hardened PCs being replaced by iPads – disposable, more functional, stateless devices.

It was also brought up that IT in many companies has a history of providing cumbersome user interfaces for applications resulting in lack of use and lower productivity. Mobile gives a fresh start and can and should be used to really improve employee lives and foster use. One company was cited as having turned this cycle around by providing a “loss leader” mobile application to pre-order lunch from the cafeteria and placing it in an enterprise apps store alongside other business apps. The use exploded and encouraged use of the new business apps as well. This example illustrates a common tenant – good mobile reduces friction in the system and saves time.

There was also an interesting discussion around the role of the CIO and how some companies are experiencing a move towards CMO/BU driven IT budgets because business is moving faster than ever. We have seen a similar phenomenon play out before during the transition to the PC era. CIOs need to encourage and enable mobile applications and innovations or get rolled over. The business health requires it!

These observations all resonated strongly with me and are very well aligned with where VMware is headed with Horizon Workspace. To recap, it’s becoming a mobile world and mobile solutions are transforming the enterprise landscape. IT needs to embrace these trends and it is rapidly becoming a prerogative to manage users’ business identities, their business applications and data, not their physical devices. This content must be delivered with native user experience to the device(s) of choice safely, securely and in a compliant manner while not precluding independent personal use. Security needs to be driven end to end – applications to cloud services over insecure public infrastructure and this is ideally a platform feature.

At any conference like this, a number of attendees live and breathe mobile and are at the forefront of enthusiasm for their chosen technologies. Some memorable phrases I heard were “it’s not mobile first, it’s mobile only” and “HTML 5, nice thought, not reality”. And lastly, I can’t finish without introducing you to my new friend Romo, a great use for your old iPhones and iPods – turn them into robots!

I recently had the pleasure of attending Venture Beat’s Mobile Summit and co-chairing two sessions on mobile’s effect on the enterprise with Gaurav Tewari, Director at SAP Ventures, and moderated by Dylan Tweney, Venture Beat Executive Editor. The sessions were roundtable discussions and focused on surfacing how mobile is impacting the enterprise and what issues and concerns are on executives’ minds. These were very engaging conversations that resonated strongly with me and I wanted to share some of the highlights and perspectives with a wider audience.

Security in a BYOD world was certainly a hot topic with varying perspectives. There was a surprisingly heated debate on the level of threat and vulnerability with mobile platforms. It was commonly agreed that the bulk of publicized attacks have been on corporate and cloud infrastructures, as well as Windows PCs. It was also agreed that newer platforms have better built-in hardening. The actual mobile platform threat was a hot topic of discussion. Some folks felt the risk was overblown and others touted imminent disaster due to out-of-date firmware and unpatched vulnerabilities, particularly calling out Android platforms. However, it was certainly agreed that as we move to mobile devices connected over Wi-Fi/the public internet and connecting to public cloud-based services, traditional corporate LANs become much less of a safeguard and end to end protection over an uncontrolled and potentially insecure communication path is becoming the norm.

The growing sophistication of attacks was also brought up. There was a clear consensus that security needs to be designed in by experts and applied at a fine granularity (app or data level) while still delivering a great native user experience. With a varied and heterogeneous mix of devices, diverse application frameworks and more insidious attacks, an overarching security framework for applications was a key agreed upon and as yet unmet need.

BYOD is considered table stakes today, although several folks expressed that the pendulum may swing back to corporate-owned devices. With BYOD plans, the perception is that it doesn’t really save money, but it does provide flexibility, enhance productivity and user satisfaction. 80% of companies are doing it, but only 10% actually have well thought out policies. Of particular interesting concern were MDM style device management ramifications coming as unwelcome surprises for users and the impact on their personal device and content. Here is an interesting viewpoint from Cesare Garlati, who spoke about a personal experience where his child was trying to unlock and play a game on his tablet without supervision. He tried the wrong password too many times and triggered a full remote wipe via IT policy, deleting family photos and music. It was a clear point of consensus that for BYOD use cases, fine grained policy and controls that are limited to business applications and content is required.

It was very clear that mobile is here to stay and is at the forefront of enterprise thinking. A comment was made and corroborated that the best IT departments, spur and enable mobile applications and encourage rapid innovation/iteration for business ROI. It’s not a hyperbole – mobile really enables us to not just automate physical world processes, but to create new and improved valuable workflows and operations. Enterprises are starting to use the full range of capabilities available – cameras for augmented reality, GPS location awareness as integral parts of advancing enterprise processes and backed by powerful cloud services. Attendees cited examples – looking at units of various types in the field by cameras and pulling up maintenance records. Another example was tablets used for factory inventory on forklifts. In this case, expensive hardened PCs being replaced by iPads – disposable, more functional, stateless devices.

It was also brought up that IT in many companies has a history of providing cumbersome user interfaces for applications resulting in lack of use and lower productivity. Mobile gives a fresh start and can and should be used to really improve employee lives and foster use. One company was cited as having turned this cycle around by providing a “loss leader” mobile application to pre-order lunch from the cafeteria and placing it in an enterprise apps store alongside other business apps. The use exploded and encouraged use of the new business apps as well. This example illustrates a common tenant – good mobile reduces friction in the system and saves time.

There was also an interesting discussion around the role of the CIO and how some companies are experiencing a move towards CMO/BU driven IT budgets because business is moving faster than ever. We have seen a similar phenomenon play out before during the transition to the PC era. CIOs need to encourage and enable mobile applications and innovations or get rolled over. The business health requires it!

These observations all resonated strongly with me and are very well aligned with where VMware is headed with Horizon Workspace. To recap, it’s becoming a mobile world and mobile solutions are transforming the enterprise landscape. IT needs to embrace these trends and it is rapidly becoming a prerogative to manage users’ business identities, their business applications and data, not their physical devices. This content must be delivered with native user experience to the device(s) of choice safely, securely and in a compliant manner while not precluding independent personal use. Security needs to be driven end to end – applications to cloud services over insecure public infrastructure and this is ideally a platform feature.

At any conference like this, a number of attendees live and breathe mobile and are at the forefront of enthusiasm for their chosen technologies. Some memorable phrases I heard were “it’s not mobile first, it’s mobile only” and “HTML 5, nice thought, not reality”. And lastly, I can’t finish without introducing you to my new friend Romo, a great use for your old iPhones and iPods – turn them into robots!

But if it is so obvious, why is IT still run using central planning?

All IT projects still go through a central planning model.  Anyone wanting to make changes must fill out a long change request and all changes must be centrally approved.  New hardware and resources are planned months in advance and provisioned after much discussion and planning.  When you start thinking about it like this, is it any wonder that IT can’t keep up with business demands?  The reality is that we need to start thinking differently about how we run IT, and cloud offers us a tremendous opportunity to accomplish this.

So what is cloud, anyway?  It’s all about self-service.  Cloud allows users to get the resources they need at the push of a button.  Indeed, that’s really what makes a market economy work so well.  People figure out what they need and they go and get it. They don’t ask anyone and they don’t wait for something to happen.  They go to a store to buy what they want and may even pay a premium to get it faster.

How is self-service realized in a cloud?  The first key ingredient is virtualization. Not only do you need compute virtualization, but you must also have storage and network virtualization.  Virtualization abstracts software from the hardware it’s running on, and as all of these resources are virtualized, it enables full control of everything through software.  Then you need to automate.  You leverage the control in software to effectively script everything.  This gives you the “press of a button” functionality. A single button can now do an amazing amount of work that would take forever if done manually.  But due to automation and abstraction in software, it can be done automatically in a moment’s time.  The next step is to create a “portal” where users can come to self-service.  This portal allows them to provision resources, manage the lifecycle of existing resources, and much more – it gives them all the options at a push of a button.

It’s not just a technology issue; it’s also a mindset issue. Up until this point, the flow for IT has been to receive a request, then after much planning, provision resources for that request. So it’s “receive request, then provision.” In the new self-service world, requests will come in whenever they do, and IT must be prepared for them. Thus the mentality must be “provision, then receive request.” This is the fundamental change in mindset that must occur.

Now, I know what most IT admins reading this are thinking: “wouldn’t this lead to total chaos?”  Well, let’s look back at Paris.  We know that it does not have a central planning economy, nor does any Western, industrialized city.  This means that the government has no direct say or knowledge of most of the activities that occur in the city.  But is there total chaos?  Of course not.  It’s just the opposite.  We have some of the most advanced cities in human history.  And why is this?  Well, it’s because the city government has control in the right places.  If you want to create a restaurant, then you need to get approval.  If you want to do major exterior work to a building, you need approval.  But small things like changing the menu at your restaurant or updating the interior of a building don’t need approval.  So the government strives to achieve the right balance between adding control points and letting people do what they want without oversight. (And, sure, we all can argue as to whether these city governments get it right!)

We can do exactly the same thing in IT, and that’s where management comes in.  Enterprise management tools can give you exactly the type of governance you’re looking for.  They will allow you to create policies up front that will be checked each time a self-service request is made.  For example, small requests for a couple of VMs do not need approval, but for three to eight VMs you need the employee’s manager’s approval.  And for 10+ VMs, you need a VP to sign off.  These are just examples, but you get the idea.  The right management tools give you the right level of control.

In addition to governance, management tools also give you operational control.  No longer will IT be getting specific requests for resources up front.  Instead, employees will just grab them through self-service.  This means that IT has to rethink issues like capacity planning.  Typically capacity planning is done based on planned, upcoming projects, ensuring that additional capacity is ready by the time the project kicks off.  Instead, in the new cloud model, IT will have to look at the aggregate growth of the overall enterprise cloud, and provision additional capacity based on that trend.  Again, this aligns closely with real city governments.  They don’t special case every family moving in and out of the city.  They look at aggregate trends to ensure there’s adequate housing stock growth in the upcoming years.

In my view, while cloud certainly involves technology changes, probably the biggest change is that of culture and mindset.  A central planning model for IT is simply no longer tenable.  IT has to start thinking of itself as a government overseeing a free market.  When you make this mindset shift, you realize that a government should no longer focus on dictating what’s on the menu of each restaurant, but instead focus on ensuring the roads are in good shape and that the electricity and plumbing work.  In IT terms, this means ensuring that the virtual infrastructure is up and running, that it’s meeting the SLAs as negotiated with users, and that there is a robust self-service user portal.  If you can build this virtual city for your users, you will allow them to move as fast as they can and to unleash innovation – and we, at VMware, are excited and ready to help you build and manage your own virtual city. To hear more, you can watch the replay of my keynote here.

So, who feeds Paris?  The right answer is to let Paris feed itself.

As I’ve discussed in previous posts, most big-data environments consist of a mix of workloads. Serengeti’s mission is to enable as many of the big-data family of workloads into the same theme park, all running on a common shared platform.

Supporting mixed workloads is a key capability for big-data. In my customer discussions I see a mix of Map-Reduce, HBase, Solr, numerical analysis (R and SAS), and increasingly more of the Big SQL engines such as Impala, ParAccel, and Pivotal Hawq.

Support for HBase in Serengeti

By definition, we can deploy a whole range of workloads on the virtualized cluster. For example, we can deploy SAS on the same physical nodes as Hadoop, using the same resources at different times for each purpose. To deploy and configure HBase as a holistic distributed system we included HBase specific cluster configurations in this release.

Highlights of this new support include:

• The ability to deploy an HBase instance, with full integration to map-reduce, exposing the Thrift and REST APIs
• HMaster HA, in an active and hot standby configuration using VMware HA
• Elastic scaling allowing the cluster to expand with a single command

Sub-Saharan Africa, Central America or Asia?

We continue to work with our key Hadoop parters to strengthen support for Hadoop and Big-Data applications in a virtual environment. In addition to Apache Hadoop 1.0. Hortonworks HDP-1.0, Cloudera CDH3, Greenplum GPHD-1.2, we have added support for MapR Hadoop distributions, and Cloudera CDH4 .

New Support for Cloudera:

• We now support the ability to deploy a CDH4 cluster, using either HDFS1 or HDFS2. 
• Name node federation: support for the new federation capabilities in HDFS2
• Configuration of the new Namenode HA in active/hot standby mode
• Dynamic support for core Hadoop configurations, allowing updates to the config after the cluster is deployed

New Support for MAPR:

• We can now deploy a full MapR cluster, with the MAPR CLDB, FileServer, JobTracker and Tasktracker
• We can deploy the MapR control system for monitoring and control of the cluster
• Support for elastic growth by adding more File Server and task-tracker nodes

Special support for Temporary Data

One of the key things we’ve learned about Hadoop is that it has significant ephemeral data use. This is typically used for stages like map output, reducer input, and sort spills. I covered this in some detail in this post.

In Serengeti 0.8.0 we can now provision a shared file system service specifically for the shared data. This makes it easier to separate out the compute VMs from the datanodes, making them stateless – with the compute job input/output going into either HDFS, MAPR or Isilon distributed file systems, and the temporary data going to local disks.

How to Learn More

We published the new release of Serengeti on our main project site, including more detail on these key areas. Feel free to follow-up with comments or questions on this new release.

As I’ve discussed in previous posts, most big-data environments consist of a mix of workloads. Serengeti’s mission is to enable as many of the big-data family of workloads into the same theme park, all running on a common shared platform.

Supporting mixed workloads is a key capability for big-data. In my customer discussions I see a mix of Map-Reduce, HBase, Solr, numerical analysis (R and SAS), and increasingly more of the Big SQL engines such as Impala, ParAccel, and Pivotal Hawq.

Support for HBase in Serengeti

By definition, we can deploy a whole range of workloads on the virtualized cluster. For example, we can deploy SAS on the same physical nodes as Hadoop, using the same resources at different times for each purpose. To deploy and configure HBase as a holistic distributed system we included HBase specific cluster configurations in this release.

Highlights of this new support include:

• The ability to deploy an HBase instance, with full integration to map-reduce, exposing the Thrift and REST APIs
• HMaster HA, in an active and hot standby configuration using VMware HA
• Elastic scaling allowing the cluster to expand with a single command

Sub-Saharan Africa, Central America or Asia?

We continue to work with our key Hadoop parters to strengthen support for Hadoop and Big-Data applications in a virtual environment. In addition to Apache Hadoop 1.0. Hortonworks HDP-1.0, Cloudera CDH3, Greenplum GPHD-1.2, we have added support for MapR Hadoop distributions, and Cloudera CDH4 .

New Support for Cloudera:

• We now support the ability to deploy a CDH4 cluster, using either HDFS1 or HDFS2. 
• Name node federation: support for the new federation capabilities in HDFS2
• Configuration of the new Namenode HA in active/hot standby mode
• Dynamic support for core Hadoop configurations, allowing updates to the config after the cluster is deployed

New Support for MAPR:

• We can now deploy a full MapR cluster, with the MAPR CLDB, FileServer, JobTracker and Tasktracker
• We can deploy the MapR control system for monitoring and control of the cluster
• Support for elastic growth by adding more File Server and task-tracker nodes

Special support for Temporary Data

One of the key things we’ve learned about Hadoop is that it has significant ephemeral data use. This is typically used for stages like map output, reducer input, and sort spills. I covered this in some detail in this post.

In Serengeti 0.8.0 we can now provision a shared file system service specifically for the shared data. This makes it easier to separate out the compute VMs from the datanodes, making them stateless – with the compute job input/output going into either HDFS, MAPR or Isilon distributed file systems, and the temporary data going to local disks.

How to Learn More

We published the new release of Serengeti on our main project site, including more detail on these key areas. Feel free to follow-up with comments or questions on this new release.

Outside of VMware activities, I spent three full days at the conference visiting other booths, talking to partners, customers and listening to sessions. Since there was so much to take in at the show, I’ve highlighted the top three observations that stood out the most to me during my time there. One was to be expected, the second was refreshing and the third was truly innovative.

Observation #1 – Enterprise Focus

In all the years I’ve been going to MWC, I’ve never seen as much focus on the enterprise opportunity as I have this year. The conference started off with the big attention-grabbing news that AirWatch raised a $200m funding round. Whoa, $200m?!!? That announcement and AirWatch’s booth were definitely “the talk of the town” amongst the enterprise mobility vendors at the show.

The other big enterprise-centric news was from Samsung with their Knox announcement. Knox is Samsung’s dual persona solution, similar to our Horizon Mobile solution, and it will be interesting to see how this plays out in the months ahead. If you are a Samsung shop, this solution promises all the benefits of our Horizon Mobile solution.

I also noticed there were more enterprise mobility vendors with booths at MWC this year than before. Air Watch, Soti, Fiberlink, Citrix and Red Bend were all well represented in the halls – and many other vendors chose to just get meeting rooms for customer discussions and demonstrations.

It certainly felt like the year of enterprise mobility at MWC 2013.

Observation #2 – Third OS ecosystem

iOS and Android are currently the premier mobile operating systems which leaves a lot of others vying for the third spot, including Microsoft, BlackBerry and numerous others. In the “others” category, the two that I was eager to see were Firefox and Ubuntu. The Firefox guys seem to be a bit ahead with about 18 committed carriers and LG, ZTE, Alcatel and Huawei as committed OEMs in their corner. However, the Ubuntu demos were solid. Their touch UI was very well thought through, and you can watch Mark Shuttleworth demo it here. I was especially impressed with their design aesthetics.

As many of us in the industry know, it’s a huge effort to create an ecosystem so I wish all these guys the best! I’m glad to see competition because at the end of the day, more competition means more innovation, which is always better for consumers.

Observation #3 – Yota phone

The device that really caught my attention was from Yota. It has two screens – one is your normal phone screen with Android apps and on the back they have a low-power electronic paper display that is ideal for reading websites, etc. without draining the battery. It’s super innovative compared to much of the competition, and I think this will get a lot of pickup. Very cool stuff!

In all, I really enjoyed checking out the latest phones and technologies at MWC and watching the mobile space grow. Our industry has come a long way and 2013 will be another fun ride. If you were at MWC, what were some of your favorite highlights and what do you hope to see change in the mobile space this year?

Side note: Probably the best experience of the visit – I was able to get tickets to see FC Barcelona play Real Madrid on the Tuesday of the show. I encourage you all to watch a soccer game in Barcelona if you have the opportunity. It was an unbelievable experience with 100,000 fans!

photo by Carrie Fei

Before our panel, I joined over 1000 women at the opening keynote given by Sheila Marcelo, Founder and CEO of Care.com.  All I can say is “wow.”  Sheila spoke to her focus on not just hard work, but passion for building products that touch people.  She demonstrated tremendous humility when talking about how she succeeded and failed along her career trajectory which I think reminded the participants that there’s no perfect path.  We each have to find our passions and learn along the way.

We had a lively group at our panel session.  Around 5o women attended – most were business school students and many had technical backgrounds.  The first part of the discussion focused on career paths and what it means to be a woman in tech.  My guiding principle has always been to focus on the quality of your work, working on cool projects and with cool/fun people.   The fact that you’re a “girl” should not come into play, but the reality is that we do think and behave differently and have to accept that we have to adapt to the predominantly male work environment in the Tech industry.  One example I discussed was applying for jobs.  As cited by the Anita Borg Institute‘s report “Solutions to Recruit Technical Women“, women tend to not apply for a job if they do not qualify for ALL of the description requirements whereas men will apply if they meet only a percentage of the requirements.  My co-panelists unanimously agreed that we need to push ourselves and our female colleagues to “lean in” when it comes to applying for roles of which we might not feel we’re completely qualified.  Whether this is for a job at a new company or the opportunity to grow or change roles at your existing company, we need to go for it more to get ahead as leaders – both administrative and technical – and we need to encourage and support each other to get there.

Another hot topic was near and dear to my heart:  How do we stay innovative as our companies scale up?  Each panelist had so many good points to offer.  I spoke about how important it is to keep channels open to ensure new ideas come forward – the primary mission of my team – but also how important it is to be selective and not invest in everything.  At VMware, we try hard to explore new ventures and give them time to prove themselves worthy for more investment, or not.  When we decide to not invest, we see the ending of such an exploration as a success because we made an educated decision not to invest.  Other panelists spoke to how important it is to go for it vs. taking too long to decide to invest and missing an opportunity to stay ahead of your competition, how to pay attention to trends and adapt your products to the leading edge (e.g., touch screens/mobile), as well as how important it is to fail fast and move on.

The panel was truly a terrific experience and I enjoyed the conversations I had with many of the attendees just after we wrapped up and during our networking lunch.  The latter ended up being a lively discussion about careers, how women can do a better job self promoting their talents, and a deeper discussion on how companies are innovating today.  I want to thank the two talented women, Carrie Fei and Luciana Baigun, for inviting me to participate and congratulate them for coordinating such a wonderfully organized event!

Consumerization of IT doesn’t mean encouraging your users to use unaccountable, non-compliant consumer services to get their work done. But it does mean that IT has competition! IT is no longer the only game in town. Users are far more sophisticated in 2013 – they use tools and technologies in their personal lives and woe to the IT organization that doesn’t provide access to modern easy to use, business compliant technologies that enable a productive workforce. Fail to do that and your IT org is on the road to obsolescent.

This brings me to BYOD, which is a great advance for users – buy and use what you want, when you want it. No longer are you tethered to an uncool corporate-sanctioned device that was approved 3 years ago.

And lastly, the multi-device workspace. We use many different devices throughout the day. Most users have 2-3 devices that they use at different times and places depending on context. Some folks call this the “post-PC” era, but as I’ve mentioned before I consider it the “Post PC only” era.

So given this landscape, what do businesses need today from an End-User Computing strategy? They need to embrace all of the above and do so in a way that is manageable, cost-effective, secure and compliant. They should also give users access to their “stuff” across all their devices safely, securely and efficiently. In order to accomplish this, IT needs to embrace technologies that will give them the ability to manage user business identity, and deliver business applications and data across a multitude of devices. In this new and modern world, identity and policies should follow a human, not a device – and IT needs to move away from physical device management. VMware has been hard at work to address this challenge for IT and users, and today I’m excited to share more details about the products and innovations that advance our EUC vision.

Today we launched the Horizon Suite – the platform for workforce mobility that will efficiently and easily deliver a virtual workspace to end-users that spans all their devices. The suite will consist of Horizon Workspace, Horizon Mirage and Horizon View. These products and their technologies, individually and collectively as a suite, will deliver the tools you need for today’s multi-device, mobile workspace. Now I’d like to discuss some of the technical innovations within each of these products.

Horizon Workspace 1.0 will present users with a single identity that spans your devices, business applications, files and data, and even your View desktops. The user interface is a self-service portal that with a single click delivers SaaS, Windows desktop, ThinApp packages and mobile applications to your devices in the most efficient way possible. IT entitles and tracks users and their applications via a centrally stored and administered catalogue and accompanying portal. Horizon Workspace also will contain an enterprise-grade file synch service that automatically synchronizes your files across your devices that enables rich collaboration. And for IT, the master copy of the data will be securely contained and tracked behind the corporate firewall, providing security and compliance. The product will be available later this quarter and I encourage you to check it out.

One of the constituents in a multi-device workspace is still good old Windows PCs and their applications. VMware and our customers haven’t forgotten about Windows just yet! We are continuing our investments in products and technologies that facilitate bridging between the Windows environment and the multi-device workspace, as well as provide a more easily predictable and easily managed Windows experience. Horizon Mirage 4.0 is the first major release of our Mirage image management software since the VMware acquisition of Wanova. It will feature the most requested new capability – production grade application layering. And yes, Horizon Mirage will now have the ability to package applications and groups of applications in separate, independently manageable layers. This is state of the art image management for enterprises of all sizes with diverse workforces and allows IT to manage Windows and Windows applications more like modern device firmware. Mirage operates on both physical and virtual images and enables centralized image management with local, disconnected execution. Great stuff!

Lastly, Horizon View 5.2 is the newest version of our trusted VDI product. It can run Windows clients in VMs on vSphere in the datacenter, and deliver the graphics and user interface remotely over the LAN or WAN. VDI has been a great technology that addresses many use cases, such as Mobile Secure Desktops, Business Process Desktops, and Follow-Me Desktops. The multi-device workspace needs Windows desktops and Windows applications too, especially when you are running on non-Windows devices such as iPads or Android tablets – and VMware has been putting a lot of emphasis on enhanced mobile user experience for tablets through our user interface virtualization innovations, now referred to as Unity Touch. Horizon View 5.2 will be fully integrated with Horizon Workspace, providing single sign-on and access to your desktop through the self-service portal. That access can be via the traditional full function View client available for most platforms and now will also be available in just the browser via our “Blast” HTML 5 protocol. Horizon View 5.2 will also contain additional enhancements including 3D hardware accelerated graphics, vSphere SE-Sparse storage consolidation and improved scaling and performance.

Hope you enjoyed meeting our Horizon family! All the products and their innovations are expected to be available later this quarter. Please keep us posted on your feedback and experiences as you use these products. Do you think the Horizon Suite will address the IT challenges that exist in your environment?

The previous incarnation of this application was called VMware Zimbra for Android and made available as a fling. Based on customer feedback, we decided to productize and release the app via Google Play. This will also be the default email/PIM client for our VMware Horizon Mobile initiative.

Horizon Mail supports any Microsoft ActiveSync compliant email server and also supports the VMware Zimbra Collaboration Suite (ZCS). With ZCS as the backend, Horizon Mail offers several additional ZCS-only features such as Briefcase, Saved Searches, and others that are not available in any of the Android email applications in the market today. Horizon Mail was tested with Microsoft Exchange 2003, 2007 and 2010 and Zimbra Collaboration Suite 6.x, 7.x and 8.x. It supports Android 2.x, 3.x and 4.x.

There are a lot of cool features in VMware Horizon Mail, and some of the highlights include:

• Comprehensive capabilities for Email, Contacts, Calendar, Tasks
• Best-in-market support for VMware Zimbra Collaboration Suite with features such as Briefcase, Saved Searches, etc.
• Clean, elegant but powerful user interface
• Create widgets to access email, contacts and calendar from home screen
• Conversation mode
• Comprehensive folder support with ability to automatically sync individual folders
• Local and server-based search
• Integrate with native PIM to view personal and “work” Contacts and Calendar together
• Support for ActiveSync policies such as PIN, etc.
• Support for certificate-based authentication and S/MIME for encrypted emails
• Enabled for multi-window capability on select devices
• GAL search
• And much more…

I encourage you to download the application, check out all the features and give this app a spin! Here’s what you’ll see while you’re in VMware Horizon Mail:

Inbox view

Conversation view with selected messages to show context sensitive menus

Click on Inbox to easily switch to Calendar, Contacts, Tasks or Briefcase views

Folder Management – Long press on folder to sync contents

Calendar view – Combine work and personal calendars into a single view

Contacts view – Combine work and personal contacts into a single view

Our team is excited to deliver this new application to our mobile users, and we look forward to hearing your feedback! Please join the VMware Horizon Mail discussion [forum] where you can engage with other users, and interact with the team developing the app to submit bugs and feature requests. (Note: Currently the user interface is optimized for smartphones.)

OVF 2.0 was recently released and just announced by the DMTF. It has many new and improved features and capabilities benefitting both end-users and cloud service providers. This specification is another strong step in improving the interoperability and portability of workloads for virtualization and cloud computing.

Some of the new improvements to OVF 2.0 include:

• Improved support for network configuration
• Package encryption for safe delivery
• Scaling and deployment options
• Support for basic placement policies, including Affinity and availability placement
• Shared disks
• Advanced device boot order
• Advanced mechanisms for passing data to guest

These updates add security with support for a new section – EncryptionSection – which will be the focal point for the encryption functionality. This new section provides a single location for placing the encryption algorithm related markup along with the corresponding reference list that points to the encrypted OVF content.

Now in OVF 2.0, each logical network will contain a set of networking attributes that should be applied when mapping the logical network at deployment time to a physical or virtual network. Networking attributes are specified by embedding or referencing zero or more instances of network port profile. The network port profile is a new and flexible mechanism which allows network configuration information to be embedded within the OVF or it can be represented in an external file that will be referenced by the OVF. Lastly, the network port profile provides a more comprehensive set of configurations and properties which allows for reservation requests of bandwidth for example. (For more information on what’s new in OVF 2.0 also take a look at the OVF 2.0 FAQ.)

Given the newness of this specification, it will take time for vendors to add support for OVF 2.0 and include in their future products. However, as adoption of cloud and the need for interoperability continues to grow, I expect we won’t be waiting too long!

I am amazed by the changes our industry has undergone over the last decade. Virtualization has become the default technology upon which the majority of the world’s server applications run. What’s more, virtualization’s impact is rapidly extending into storage, networking, security, and every other aspect of the modern datacenter. The resulting “software-defined datacenter” is clearly the architecture of the future, enabling the efficient private, public, and hybrid clouds that are becoming part of every company’s IT strategy.

There is also a renaissance in how we consume applications. We’re in a world where we continue to use Windows applications, but augment them with web, SaaS, and mobile applications. It’s also a multi-device world where tablets and smart phones join PCs and laptops as our omnipresent technology sidekicks. The grand challenge is to help organizations offer these new choices to their employees while keeping company information safe and secure. Throw in dramatic changes in the way we write applications, analyze massive amounts of data, share files, or communicate with one another, and you can understand why I’m so excited about today’s technology world!

It is indeed an incredible time to be in our industry, and the challenges ahead will keep engineers busy for many years to come. And engineers are at the core of my proudest accomplishment at VMware: helping to build and develop one of the best system software teams anywhere. This team is well poised to continue to change the world of enterprise computing.

It is this team-building experience that I’m hoping to leverage in my role at General Catalyst, extending it into an even broader technology domain. My primary focus will be finding, supporting, and developing great technical entrepreneurs as they build the products and companies that they’ve always dreamt of building. These companies will bring the same tremendous energy, creativity, and innovation to these and other challenges, just as VMware has for so many years.

Again, it’s been an incredible journey at VMware, and I look forward to my ongoing role as a technical advisor to the company. I’ve appreciated every single day of the last 11 years, and here’s to an exciting future ahead for each and every one of us!

Sincerely,
Steve Herrod (@herrod)

Passthrough mode is the most straightforward way to enable guest-level RDMA. With passthrough (which we call VM DirectPath I/O), a physical PCI device can be made directly visible to the guest operating system running within the virtual machine. We published a research note showing that this approach delivers very good InfiniBand latencies (under 2us) and excellent bandwidths over a wide range of message sizes. There is a downside, however: Punching through the virtual machine abstraction in this way disables several platform features, most notably vMotion (live migration) and Snapshots.

Many of the HPC customers I’ve talked with about this aren’t too concerned with these limitations, primarily because their bare-metal environments for the most part don’t offer these features and so they aren’t losing capabilities when they transition to a virtual environment. However, in the Office of the CTO we take a longer view — that’s our job. And what we see is that both vMotion and Snapshots can be used to offer new capabilities in virtualized HPC environments that are either difficult or impossible to implement in bare-metal environments, features like reactive or proactive fault tolerance and dynamic resource management. There is a full description of those features in the first part of this presentation, for those interested. In addition, it is clear that if RDMA is to be deployed in Enterprise datacenters (using RoCE, InfiniBand, or iWARP), then enabling a widely-used feature like vMotion is going to be very important.

My colleague Bhavesh Davda and our intern, Adit Ranadive, worked closely together this summer to design a solution to this problem, which they discussed in a video interview back in August. More recently, they’ve described their work in a paper titled Toward a Paravirtual vRDMA Device for VMware ESXi Guests, which is included in the Winter 2012 VMware Technical Journal that was just released last week. The paper describes the design of a virtual device that supports standard, Verbs-level access to RDMA within a guest operating system while maintaining the ability to perform vMotion and Snapshots, and enabling direct datapath access to the hardware, which is needed to deliver high performance. The development of the prototype is underway — watch this space for performance results and other updates.

From talking to dozens of customers, it is obvious that seismic changes are brewing. Perhaps the biggest of them all is how employees are using their smartphones and tablets, and the driving force behind this is the application ecosystem. The app ecosystem, which was largely unknown in the early Blackberry days, is flourishing on iOS and Android so much so that over a million applications are now available for your mobile devices. According to analyst, users are downloading an average of 40+ apps onto their devices. Just rewind a few years and contrast that with your Blackberry device and you will notice how usage patterns are changing. VMware is noticing that no matter who buys the device (corporation or employee), most phones will have personal applications and data alongside corporate applications and data – making all devices dual purpose. Users expect to have the freedom to download any apps they want to, whenever they want to – and organizations that try to prevent this employee behavior is a losing the battle. This user freedom is having a profound impact on enterprises and how they think about security and management of mobile devices.

Now let’s look at this from IT’s perspective. In the pre iOS era, employees used their Blackberry devices mostly as an email device. The mobile app ecosystem wasn’t in place then so employees largely used what came with the device. Given that the device was corporate issued, IT leveraged the policy framework that RIM provided to manage the entire device – device password, device wipe, etc. came into our vocabulary then.

With the introduction of the iOS, the app ecosystem was born. Users wanted to download games, navigation apps, culinary apps and sky was the limit and people started downloading a lot of apps to their mobile devices. Once Active Sync clients were available on iOS and Android devices, they become legitimate corporate devices from a user standpoint but for IT to support these devices, security was required. Therefore, the first wave of security and management for iOS and Android simply replicated the Blackberry model of securing and managing the device. The mobile device management (MDM) industry was thus born.

Once IT organizations started purchasing and rolling out MDM solutions, they realized that these solutions were not quite what is needed. They then realized that the usage patterns changed and therefore, their old-school methods of securing and managing devices were no longer in line with usage patterns or user expectations. That being said, my first prediction is that the MDM bubble will burst in 2013 as new ways to secure and manage corporate content on mobile devices will allow end users and IT to reach a truce – security and manageability without compromising usability or device capabilities!

A significant percentage of the million+ apps available for mobile devices today are consumer oriented. Many ISVs have now developed mobile front ends to their apps but what is largely missing is the quantity of mobile applications created by enterprises for enterprise use. As the proliferation and acceptance of mobile devices as tier 1 platforms increases, my second prediction calls for a significant up-tick in enterprise mobile apps – apps of enterprises, for enterprises, by enterprises. I expect you’ll see users in 2013 not only using corporate email and consumer apps but also dozens of internally developed apps.

IT will start to re-think how they develop internal applications. In the PC era, large teams of IT developers developed big monolithic apps over a period of 12-18 months and rolling it out using standard deployment methodologies. These apps have generally been hard to use and maintain as they tried to do everything for everyone. However, based on the types of apps employees are using in their consumer life, the usability bar has risen to new levels. Therefore, IT will need to develop dozens of smaller single-purpose apps with usability at the front and center of app development cycle rather than simply putting lipstick at the very end of the development cycle.

While I’m excited to see how these trends shake out next year – it’s important for us to keep in mind that the smartphone and tablet industry is still in its infancy with things evolving at a rapid pace. It is a great time to be part of such paradigm shifts and here’s hoping 2013 is successful to each of us!

In 2012, I saw much more activity within core enterprise and business. There are a growing number of enterprises that are already heavily invested in the use cases –  but by volume, most customers now have some form of big data proof-of-concept underway. These proof of concepts typically start with a thesis of how competitive advantage can be gained through insight from the data. A proof of concept can quickly validate the theory, and helps sell further investment in the analytics platform, and it snowballs from there.

VMware made awesome progress this year in making vSphere a great platform for big data, with the mission of allowing all varieties of big data storage and analytics frameworks to run on a common virtual infrastructure platform. In support of this, we’ve teamed up with the Hadoop community to validate virtual infrastructure as a differentiated Hadoop platform and make the combination of Hadoop and virtualization better than the sum of its parts. The highlights for this year include:

• Our engineering team showcased performance results of Hadoop running on virtual infrastructure with little or no overhead.
• Through Project Serengeti , we were able to make Hadoop dead-simple to provision and scale, allowing us to deploy a new Hadoop  cluster in less than 10 minutes.
• We enabled Hadoop to be elastic so that it can co-exist with other big-data workloads, allowing dynamic grow/shrink of Hadoop nodes in concert with other big-data workloads – ultimately creating elastic Hadoop on demand.

Now onto my predictions of how 2013 will unfold. Drumroll, please!

Prediction #5) We will all know at least one colleague who is bragging about a Petabyte stockpile of new data.

We’re seeing a growing list of new sources of data, most of it being machine generated. It’s estimated that in 2013, we’ll produce 4 Zetabytes (that 4 million petabytes) of new data. Over 80% of that will be unstructured – in the form of files, documents, media, logs, and other types.  That will amount to a jaw dropping 1 quintillion new objects.

The current research is showing a growth rate of between 50-60% per year of these new types of data. As an example, one customer I’ve been working with is building out an architecture to store every single key-click, mouse-over and application log event for every user for two years. This will give them tremendous insight into what their customers’ interests are, and allow them to do sophisticated targeted marketing. Keeping this data amounts to an estimated storage stockpile of 200 Petabytes!

The economics alone is a forcing function towards new storage architectures. If we store 1 Petabyte today in a regular storage system, that’s typically a storage investment of several million dollars. The challenges are the costs of storage, the administrative overhead of managing this much data, and bringing enough computation to the data in a way that we can reasonably filter, organize and analyze the data.

Prediction #4:  ‘Delete’ will become a forbidden word

There’s definitely a mindset change about keeping data – with a change from storing important data to keeping ALL data. The problems is that we don’t know up front what questions we want to ask of the data, so if we don’t keep that information we are precluded from doing whatever insightful analytic that could have been the “killer usecase”. If we keep all data, then we can keep open all options for interesting analytics. The data scientists can develop new theories and models, and go back in time to understand these new models.

I believe we’ll see a growing number of companies who follow the same path. They will setup sufficiently large-scale data stores and scale-out analytic tools so that keeping all data is affordable and practical.

Prediction #3: There will be a mad dash for software-defined storage

I predict we’ll see a flurry of new technologies and companies that will claim to offer different renditions of software-defined-storage, aimed at storing this mass of data. The traditional model of whole-system storage hardware will change in light of the volume of data tilting heavily towards new data types, and a blurring of the line between compute and data.

The growth rate of traditional data (customer records, transactions, history) just doesn’t grow at anywhere near the rate of the new data. Traditional enterprise data is only growing at 20% or so – but as we saw, the amount of new data being stored is growing in the order of 50% year over year. This means that there will be  two key shifts within the storage industry – a move towards more commodity-based storage that can potentially take the place of traditional storage, and a new set of high-scale storage architectures aimed at storing all this new data.

The chase will come from multiple dimensions:

• Software-defined SAN, to provide cheaper places to store blocks. A software-defined storage approach means that we can use software to provide the high reliable, feature-rich storage services on commodity hardware. There are a number of startups entering this space with either pure-software or software-hardware appliance combinations, and as announced at VMworld in 2012, VMware is also developing technology in this space. These storage solutions will provide moderate scale storage, and be aimed at providing the storage feature set (things like data recovery, replication, policy based placement). They will make heavy use of SSD and Flash to deliver high IOPS performance.
• Software-defined NAS, in particular scale-out NAS. Software-defined NAS can run on commodity servers, and able to scale up to the order of 10 Petabytes. I expect the NAS market will experience a resurgence in the big-data space. Scale-out NAS architectures allow for the capacity requirements of big-data, by clustering many nodes together to provide multi-petabyte configurations. Scale-out NAS offerings will have an advantage of their traditional access methods – making it easy to ingest and export data from the system through standard mechanisms such as NFS. Scale-out NAS combined with virtualization means that you can bring compute to the data, making it a viable platform for data-parallel workloads such as big-data.
•  Software-defined Object stores. These are radical new object stores that claim to scale to 100 Petabytes, with interesting cloud-replication capabilities. These storage architectures trade-off some of the typical constraints that limit scale, including POSIX semantics and consistency. For big-data, we often don’t need those fine-grained semantics, but we do need to store data at scale. It’s clear that Hadoop’s HDFS is one strong player in this space with the ability to cluster several hundred servers together. A growing number of technologies exist in this space and will be positioned as commercial solutions to big-data storage, with a strong emphasis on scale and multi-cloud replication.

Prediction #2: The default infrastructure for Big Data will change

We should expect a tipping point in network infrastructure, 10GBE networks and high-bandwidth switch topologies. Cost metrics will afford the majority of new big-data installations to take advantage of 10GBE, resulting in a different set of assumptions about optimal big-data systems. Cross-sectional bandwidth within a rack of 1Tbit will ease focus on data locality, and put the emphasis more on designing storage topologies for availability. In 2013, data and compute can be anywhere in a switch domain with little or no performance difference. Beyond 2013 we’ll see more interesting flat networks evolve, which will even further relax the locality requirement.

Additionally, the decreasing cost of flash and the increasing availability of software to take advantage of multiple tiers of storage will mean that flash will be an integral part of every storage architecture. Hot blocks will be placed automatically on SSD, and writes will be buffered by SSD to give much lower latencies. In some cases, entire applications data sets will be moved to flash based storage tiers.

Prediction #1: The focus on big data use cases will shift heavily towards real-time

Businesses are starting to realize they now have a significant and new competitive advantage with the ability to make real-time decisions based on their own data.

A few of the top use cases include:

• Personalized, targeted marketing (such as new retail): Rather than just acting on buying patterns, retailers will be able to mash up large amounts of historical data and recent real-time events (what did you buy just a few minutes ago, where are you, where did you come from, what did you tweet?) and deliver customized offers targeted accurately at buyers’ needs.
• The predictive enterprise: Real-time decisions replace age-old processes for running the business become the enterprise “brain.” Stock and shipping calculations become dynamic and adaptive, responding to predicted trends and swings based on real-time inputs. Stocks can be reduced, and pricing can become dynamic based on spot markets and needs.
• Automatic failure analysis and predictive maintenance through a closed feedback loop from embedded sensors and metrics: The same technology that is used for elite cases such as nuclear reactor monitoring is being applied to everyday uses – your car, home appliances become part of a predictive failure analysis system and will proactively alert for up coming situations requiring attention. In addition, enterprises are increasingly looking to use this technology to improve performance and availability of their applications (in fact, VMware’s vCenter Operations is a good example of this type of analytics).

As a result, in 2013 I predict we’ll see an emergence of the frameworks and technologies required to implement these systems. The significant component will include:

• Real-time in-memory databases: These databases will be able to ingest the extreme rates of events that come from sources such as social: Twitter and Facebook feeds, machine generated metrics, and large-scale user-driven interactions. These databases are able to incorporate this real time data with learned behavior, and react in real time. Examples include SPARK and SHARK from UC Berkeley, Gemfire from VMware,
• Frameworks for programming event driven actions: The Storm project from Twitter, some new entrants based on NoSQL, such as Continuuity.
• Frameworks for implementing machine learning: The programming models for machine learning typically involve iterating over steps in data in rapid succession, where subsets of the data reside in memory. Platforms such as SPARK from UC Berkeley provide the reliable datagram store and iterative programming models that are needed.

Almost every application being built to incorporate these techniques is hand-rolled. In 2013, we’ll see startups emerging with new PaaS-like frameworks to aid in the development of these real-time applications.

As the need shifts from a monolithic map-reduce powered platform to a hybrid of real-time, batch and machine learning, there will strong need for running multiple framework types on the same cluster. We believe that virtualization will play a central role in creating that common distributed platform, and we see a growing number of enterprises in 2013 standarding on virtualization as the platform for their big-data solutions. I can’t wait to see how all this plays out next year!

Before I start, let’s get one no-brainer prediction out of the way: 2012 will be remembered for Nicira, and not Nibiru. Yes, software-defined networking (SDN) will trump planetary collisions and the entire Mayan Apocalypse. Allwyn has a great blog entry that explains how SDN is just the beginning of a much bigger trend around network virtualization and the software-defined datacenter — it’s well worth reading.

And, now, on to the main predictions, all of which are my opinions based on public information with no wink-wink-nod-nods based on any inside information whatsoever.

Prediction: HPC Public Cloud

This coming year will herald the beginning of much broader use of public clouds for running HPC workloads. This is in part because big players like Amazon (Cluster Compute), Google (Compute Engine), and Microsoft (Big Compute) are all now catering to HPC workloads with their cloud offerings. But a few large players do not an ecosystem make. It is much more interesting — and revealing — to see small companies like Bitbrains in the Netherlands and EngineRoom.io in Australia using virtualized cloud infrastructure to offer not just compute cycles, but a full range of tuning and other services to enable effective and successful use of cloud for HPC workloads.

Bitbrains IT Services designs, builds and supports Cloud Computing solutions based on VMware products for companies that require high levels of continuity, reliability, and scalability of their complex and business-critical applications. With respect to HPC specifically, Bitbrains engineers and integrates complex risk-calculation clusters for large finanical institutions using algorithmic solutions from a variety of partners and offers a risk-management-as-a-service solution that scales to over 1000 cores.

EngineRoom.io focuses on enabling companies to derive insights and revenue from their data by giving them the capacity to aggregate structured and unstructured data, to handle high-velocity streaming data, and to process these large datasets at scale. They are applying their expertise to solve customer problems in a number of HPC verticals, including Media and Entertainment, Life Sciences (bioinformatics & medical image analysis), Financial Services, and Oil & Gas.

I’ve talked recently with the folks at both Bitbrains and EngineRoom.io and was impressed by their energy and their expertise. These companies have a deep understanding of both our platform and the technical requirements of HPC workloads, allowing them to bridge the gap between the two and offer real value to their customers. While the completely self-service, zero-touch offerings from Amazon and others have their place, it’s the full-service offerings like those offered by Bitbrains, EngineRoom.io, and similar companies that will enable cloud computing to address the problems of the so-called missing middle [PDF] and open HPC techniques to a much broader, unserved market.

This year, IBM acquired Platform Computing, provider of the popular LSF distributed job scheduler. In addition, Intel acquired both Qlogic’s InfiniBand assets and Cray’s HPC interconnect technology. And two years ago, Oracle made a strategic investment in Mellanox, the primary provider of InfiniBand and RDMA technologies for the HPC market. While it is true that both IBM and Intel are big HPC vendors, it would be a mistake to view these actions as simply consolidation within the HPC ecosystem. Instead, view this as tangible evidence that the convergence of requirements between Enterprise and HPC is accelerating and savvy Enterprise vendors are positioning themselves to address new Enterprise requirements by drawing on technologies fired within the performance-critical crucible of High Performance Computing.

I predict this trend will continue, with additional acquisitions of traditional HPC assets over the coming year and beyond. These assets will be repurposed and expanded to more quickly solve a variety of mainstream cloud computing challenges (e.g., provisioning, monitoring, management, and scheduling at massive horizontal scale) because these HPC components have been architected to address the difficult challenges of scalability and performance — two critical mainstream cloud computing issues.

Have a different opinion about any of these predictions? Post a comment!

In an industry where new technology seems to leapfrog what was just new only a moment ago, it is critical for software companies to think strategically about their innovation programs.  VMware has maintained its culture of innovation throughout its history and we plan on continuing our track record of introducing game-changing technology solutions to the IT industry for years to come.

In the past year VMware demonstrated its commitment to innovation in many different ways. We introduced the concept of the software-defined datacenter to our industry – along with offering new and unique methods to manage datacenters at the scale and performance our customers require. VMware also had its very first Open Innovation contest where we invited our community to submit Fling ideas (see who won here!) and we will continue to involve our customers and partners in our innovation process as we enter into the new year.

Also in 2012, we showcased for the first time some of the innovation projects incubating inside of the Office of the CTO in the Innovation Lounge at VMworld.  There was a lot of buzz about these ideas, including one that involved A Social Media Approach to Managing Virtual Infrastructure and another on Environmentally Aware Computing.  We were also excited to introduce the Next Generation Education Environment - a new cloud scale platform for hosting and delivering education programs including VMware’s Hands on Labs (see below for a way to get priority access to these labs!).  These are just a few of all the cool things we have brewing inside VMware and I will enjoy sharing more with you next year.

As we enter 2013, there are many trends in innovation that I will be thinking about for VMware.  There is a constant need to balance time to deliver on current product requirements while ensuring we’re thinking about the next big thing.  More and more companies are coming up with creative ways to allocate percentages of time to engineers to allow for creativity and innovation.  At VMware, it’s important to offer our employees the opportunity to contribute to innovation and I look forward to sharing more details about the programs that allow for these opportunities in my upcoming blogs.

Another growing trend I see happening in our industry in 2013 is creating workspaces that allow for more creative thinking and collaboration.  Spaces should allow for open communication, spontaneous brainstorming and evoke an atmosphere of fun.  From lots of whiteboards to foosball tables and ping pong, VMware encourages creative workspaces – where they’re designed with innovation in mind. I expect more companies to adopt this approach when developing the office environment.

Lastly, in the spirit of Open Innovation, I predict more organizations will pop up to sponsor and host innovation meet-ups.  These events are a great way to get different perspectives and often lead to new ideas that may not have happened in closed settings.  VMware amongst other Bay Area companies started hosting such meet-ups and events in 2012. We were fortunate enough to host the popular Bay Area Girl Geek Dinner along with MassInnovation Nights in our Cambridge, MA office.  We will definitely be hosting more events like these soon and I think you’ll see more companies do the same in 2013.

As I look back on 2012, I am proud of what VMware was able to accomplish as a company and I’m excited about what’s to come in 2013 on the innovation front – both at VMware and in the industry at large! What do you expect to see that’s innovative next year? I would love to hear your predictions as well as any of your own innovation plans for 2013 – the first 20 people to comment on this blog and share your innovation plans gets to the head of the line (in front of the thousands of people waiting!) to get access to VMware’s beta Hands on Labs!

But it won’t stop there. It never does. I expect we’ll see even more change in 2013, and here are some of my predictions for the industry as I look into my IT crystal ball:

Big Data (or, more appropriately, “Big Analytics”): I’ve heard it and you’ve heard it – there’s an explosion of data out there! But what companies tried to figure out in 2012 was how to deal with massive amounts of data. VMware made strides in this area with Project Serengeti, our latest version of VMware Data Director and more. The adoption of big data technologies continues to grow as companies get their arms around their data dilemmas along with the opportunities available to those that make sense from it all. My colleague Richard McDougall will share his 2013 predictions for the big data space, so stay tuned that blog.

Networking & Security: We all saw the SDN space explode in 2012. It has become the talk of the town, and VMware has made great strides with VMware vCloud Network and Security offerings. We took that conversation to a whole new level with the acquisition of Nicira. As one of the core components of hybrid cloud computing, we took steps to remove “networking” as the bottleneck behind workload mobility across multiple clouds and organizations will experience the benefits of this next year. Check out Allwyn Sequeira’s predictions for the networking and security space in 2013.

Storage: What a year for storage! The ecosystem continued to make strides with storage technologies to support highly virtualized and cloud environments. At VMworld 2012 you heard of the early work around VMware’s storage directions including our Distributed Storage preview, and I expect to see even more customers virtualize their storage environment and move towards a more policy-based management approach. These steps are just the first of many announcements you’ll see from VMware and our partners in 2013 as we continue on the path towards software-defined storage. Look out for a blog from Christos Karamanolis where he’ll share his 2013 storage predictions.

Management: Can you say heterogeneity? That seems to be where it’s at these days. While VMware and others feel a homogenous environment is the most efficient approach, we recognize customers will have heterogeneous pools of infrastructure…and this isn’t going away. The ability to deploy and move x86 workloads between private and public clouds (VMware, AWS, OpenStack and others) is becoming a standard practice for organizations of all sizes as they benefit from the agility, flexibility, efficiency and reduced costs of a hybrid cloud environment. But with multiple clouds comes the need to manage this multi-cloud world. VMware announced updated cloud management products at VMworld Europe, and we’re supporting our customers manage their heterogeneous environments – and I expect the industry to continue on this path in 2013. Look out for a blog from my colleague Kit Colbert where he’ll share his predictions for the management space in 2013.

Hybrid Cloud: As the adoption of cloud grew in 2012, our customers’ workloads were not only in private clouds but in public clouds too. A special shout out goes to the recent vCloud Connector 2.0 release for the help there! We’re seeing enterprises move to a hybrid cloud model, leveraging and benefiting from reliable resources offered by our vCloud service provider partners. We welcomed T-Systems and OVH to VMware’s vCloud Datacenter Services program in 2012 – and also introduced vCloud Service Evaluation, where customers can “test the waters” with our vCloud partners. I expect to see even more customers adopt a hybrid cloud model in 2013 as they see the awesome benefits experienced by their peers this past year.

Mobile Workforce: We continue to enable the mobile workforce to use the device of their choice while staying productive, and I expect you’ll see more advancement here in 2013. At VMworld, we spoke about the innovation happening in support of our end-user computing vision including all the awesome work happening on the mobile side. The ability to manage a desktop and device will continue to be a hot topic for enterprises next year, as employees are on the move with their laptops, smartphones, iPads and more –and vendors will continue to offer and innovate solutions to address this rapid change. Scott Davis also shared his predictions for the end-user computing space.

Small and Midsize Business: If you’re an IT pro at a small and midsize business (SMB), you know all too well that the demands on IT – keeping business up and running, balancing IT supply with IT demand, and being able to respond quickly to business needs – are the same regardless of the size of your business. Fueled by growing demand for business agility and high availability/disaster recovery, virtualization has gone mainstream amongst small and midsize business.  The challenge SMBs face is to do it all with a small team.  In 2013 advances in cloud-based management solutions will make it even easier for companies to solve these problems and focus on growing their business. You can read more about the unique ways small IT teams are meeting the demands of their business on our SMB blog.

All in all, we’ve had a great 2012 and I’m even more optimistic about what’s ahead in 2013 as VMware and our partners make even more strides across these and other fun technology areas. I look forward to watching our customers benefit from these cool developments and watch their environments evolve.

The acquisition of Nicira was the “shot heard around the world”, bringing together leading network virtualization solutions for both VMware and non-VMware stacks. And it’s clear that VMware, along with our ecosystem partners, can massively simplify the way customers provision and consume networks at scale, in private, hybrid and public clouds.

As I look ahead to 2013, certain trends begin to take shape – and only time will tell how many of these are here to stay. Here are my predictions for next year:

Prediction #1: The software-defined datacenter vision took the industry by storm in 2012. It represents a prescriptive model that brings the benefits of virtualization to the rest of the datacenter. Expect to see the move towards a software-defined datacenter accelerate in 2013. Networking and infrastructure security represent some of the stickiest issues when it comes to the drive to a more agile data center. And because of this strong customer interest in SDDCs, you’ll also see more networking vendors and startups modify their roadmaps to steer towards a software-defined networking strategy.

Prediction #2: In 2013 we’ll see network virtualization established as the real goal of SDN. What we’re really after is the ability to provision networks and network services on demand, instantaneously in the context of a compute/storage standup, and be able to do so without requiring manual configuration/re-configuration of the underlying infrastructure. The following simplified cloud consumption model depicts what network virtualization is all about – enabling the consumption of L2 and L3 networks logically to support compute & storage connectivity needs; likewise the consumption of L4-7 services, gateways to physical networks and the internet. This consumption model can be leveraged by various cloud management systems, like vCloud Director, Open Stack, etc.

Prediction #3: 2013 will also herald the wholesale virtualization of network and security infrastructure services, including firewalls, load balancers, IDS/IPS systems. We have seen the initial signs of this trend in 2012, and I expect to see this continue un-abated in 2013. One ramification of this trend is the de-fragmentation of security services. Historically, the security industry has been extremely fragmented – with perimeter security, network security, workload/guest security, data security, app security, user security. With virtualization and cloud consumption models, we will begin to see several of these functions come together in the virtual/cloud plane, as depicted in the illustration above.

Prediction #4: In 2013, we will see the emergence of a network virtualization solution across hybrid cloud stacks, like VMware vCloud, Open Stack, and Cloud Stack. We know this is desired by cloud operators who have multiple stacks, yet want a common network virtualization solution. Customers are also asking for this as they move towards a multi-cloud world. With the integration of the VMware and Nicira teams, this is an exciting area of focus for us.

Prediction #5: We’re also beginning to see the evolution of the physical network fabric in cloud and enterprise datacenters. Historically, service provider datacenters were L3-centric, while enterprise datacenters were L2-centric. L3 has the benefit of scale and fault detection, while L2 nicely supports enterprise apps that rely on broadcast domains (Microsoft apps, vMotion, etc). In 2013, I expect to see the architecture depicted in the graphic below within cloud datacenters (enterprise or service providers). The physical network fabric is becoming fast, flat and fat (see this prior post), with L3 to the Top of Rack (ToR), full mesh, with ECMP to effectively leverage all the paths between the leaf and spine. We are also beginning to see the leaf become dedicated to workloads (C=Compute, S=Storage, P=Physical) or to services (M=Management functions, FW=Firewall, LB=Load Balancer), per the following illustration. This “butterfly” architecture provides a lot of flexibility and simplicity in the realization of datacenter racks.

Prediction #6: What about L2 services in the above architecture? That question brings us to the next major trend – 2013 will be the year of L2 overlays. Millions of dollars have been poured into overlay technologies like VXLAN, NVGRE and STT, and we expect to see the entire industry release products (if they haven’t done so already), whether they are NICs, ToRs, or hypervisor support. L2 overlays over the L3 fabric above will become an important pattern in the data center. While there can be debate on the overlay technology, the more important observation is that L2 overlays over “L3 to the rack” architectures are a compelling tool in the war chest. You can read more about tunneling in this recent post by Martin Casado and Bruce Davie.

Prediction #7: The tools used by the new guard are beginning to evolve too. REST becomes the new CLI. Chef/Puppet become the new Perl/bash. In 2013, I expect that the centralization of control e.g. SDN controllers, will lead to chef/puppet/rest recipes beginning to replace those timeless network engineering scripts. The exciting news for network engineers in enterprises as they evolve to becoming “cloud architects”, is that they can now create recipes to build out service provider style transport networks with minimal configuration churn and potential for “fat fingering,” yet rapidly provision L2 over L3 overlays & virtualized L4-7 services to meet the demands of their business units or application developers.

Prediction #8: I expect we’ll see software-defined security infrastructure get the same kind of attention in 2013, that SDN had in 2012. With the move from web browser-server architectures to mobile-cloud architectures, classic security paradigms based on desktop security and web server security must evolve to focus on security within the cloud i.e. authentication, authorization and tenant edge security at the cloud tenant edge, and logical security within the virtual datacenter. Likewise, the evolution of network virtualization offers a new substrate to logically insert netsec services, all automatable into CISO-friendly workflows.

Next year promises to be an exciting time for the industry – one packed with disruption, innovation and transformation. Wishing all our readers a wonderful 2013!

/Allwyn

It’s been an exciting year of changes and innovations in End User Computing land. In the industry at large, we saw the continued, inescapable march of BYOD and end user choice, featuring Apple iPad domination of the tablet landscape, Macs in the enterprise and a duopoly of Android and iPhone in the smartphone world. We also saw the arrival of Windows 8 and while not emerging to widespread acclaim or the Microsoft domination of yesteryear, it was not a complete dud either. Microsoft has made a big and aggressive bet with its new, hybrid touch/keyboard user interface and application compatibility/availability ecosystem challenges and time will tell how it plays out in both the corporate and consumer worlds.

From the VMware perspective, 2012 brought View 5.1 building on the significant momentum of View 5.0 and the groundbreaking acquisition of Wanova and subsequent launch of VMware Mirage.  The combination of VMware View and Mirage is an industry first pairing that dramatically redefined the VDI market – no longer is the market restricted to virtual desktops, but central image management operational benefits can be extended to more types of client end point systems – including physical, virtual, tethered desktops and roaming laptops (Mac and PC).  And at VMworld 2012, we showed the progress that we’ve made with a technical preview of the industry’s first integrated platform for the mobile, multi-device workspace. Now in beta release, we call this technology the Horizon Suite and we demonstrated self-service access to your apps and data across your devices including mobile-centric containers for management and security.

But enough about 2012, time to take a peek into the future…

2013

I expect we will continue to see a duopoly of iPhone and Android devices, dark horse being Windows 8 Phone. Blackberry 10 looks interesting, but likely too little, too late. It’s a pretty safe bet that some combination of Google and Samsung will solidify their dominance in the Android ecosystem. And while Apple will continue to have more elegant, integrated and desirable devices, I predict that Google will widen the gap with their backend cloud services. The first significant and clear service-based skirmish was the widely reported iOS Map fiasco of 2012. What got lost in the noise around the Apple decision to remove Google in favor of its own immature offering was that they apparently felt forced there due to the lack of voice directions in the Google Map app version for iOS. I predict Apple will use its sizeable war chest to acquire significant players who can drive delivery of valuable cloud based services for their iPad/iPhone franchises.

The diagram below pre-dates Apple’s iCloud and Google Play, but is a pretty good illustration of the landscape anyway…

While I’m discussing the duopoly, both Apple and Google (and Microsoft) are playing the game of integrated cloud service silos across their devices. I predict that 2013 will see a revolt against these locked in silos, raising pressure on these incumbents. More open cloud services will become desirable and provide opportunities for players with truly multi-device technologies that work across the eco-systems.

An EUC predictions blog for 2013 can’t get away without a prediction about Windows 8. My belief is that Windows 8 will have modest adoption with consumers and very light adoption with enterprises. For enterprises, the new hybrid user interface makes the devices too different a user experience and incompatible with existing apps, but also not as intuitive as a pure touch tablet. I predict there will be little uptick in the enterprise and Windows 7 and XP will continue to be the standard for traditional enterprise PCs. April 8, 2014 is the current official sunset of support for Windows XP. Some will ignore it and use it till it dies; others will do a crash migration to Windows 7 or non-windows platforms as the date gets closer. Regardless, Windows role in the enterprise will also continue to gradually recede in importance, becoming more invisible and more of a runtime environment for legacy applications. For consumers, I also predict that Windows 8 will also have only modest success. For tablets, not enough of a tablet experience moving forward and not enough applications for it as well as the flip side of not enough compatibility for traditional PC uses. Of course, it will be the default with new consumer PCs and make the bulk of its progress there.

Back to the theme of mobile devices and their dependency on cloud based, backend services, I predict that in 2013 we will see mainstream recognition that computing is no longer primarily done on one or more client systems like a PC, but moves further towards a world where the bulk of important computational processing is performed in the cloud and that your end point devices, especially smart phone and tablets broaden from the user interface role and take on more of the flavor of “remote controls” for your cloud services. I expect smartphones and tablets to morph into remote controls for additional services. We’re already seeing wallet apps for credit card payments, airline boarding passes, coupons, etc. Why carry a separate badge or smartcard when everyone has a smartphone and the phone is fully capable of running sophisticated authentication applications that can interface with backend services and securely identify the user? Biometric authentication via smartphone camera or touch anyone? And the user interface will improve and evolve as well. Speech as input is beginning to enter the mainstream as a user interface exemplified by Siri and Google and the advanced technologies and companies behind those technologies.

photo by Time, Inc.

Which brings me to the so called “Internet of things”; why stop at being a remote control for virtual computer services? In 2013, we will see more significant smartphone and txt message driven cloud services that allow us to control our physical devices and environments. I think we will move from remotely controlling your DVR, thermostat, baby monitors and video camera surveillance to more advanced uses and communication between the devices themselves, all controllable via your phone or tablet. One can find on the market today thermostats that allow full climate control and electric monitoring from the iPhone/Androids and performs operational analysis in the Cloud on your energy consumption patterns; baby monitors that are controlled by smartphones now with video and alerting; Sleep monitoring, Pedometers and more. The common thread here is intelligent aggregation of diverse information sources from the physical world filtered and processed in the cloud and delivered in an intuitive and useful manner through a smartphone app. Where will this all lead?  How about a remote starter for your car and its heating system? Or keyless entry? From your smartphone. Or a Scheduler for controlling household devices such as lights? One of my colleagues told me he is looking at a new house that came pre-equipped with an iPad app to control many features. When you start thinking about historically dumb devices with analog controls having digital interfaces, coupled with ubiquitous communication and simple control messages via web or SMS, it’s not a big step to think about distributed control and where that can lead convenience wise. And the input can actually be voice! “Siri, dim the lights!” In 2013, these scenarios will move further from novelty to mainstream desirable reality.

Our relationship with privacy will also continue to evolve in unexpected ways. Of course, expect more malware exploits and debacles like the Petraeus scandal as people fumble over themselves learning to retain privacy in the brave new world.  We’ll be looking for more efficient ways to interact with that Internet of things such as Siri and that ease of access will force the need for securing not only IT resources but also our personal info-sphere better than we’ve done in the past. An illustrative example from another colleague – while waiting for their spouses to return from “shopping”, his son in law used a Siri ‘Find my Wife’ request triggering the Apple Find My Friends app with GPS to identify that their wives were actually spending time at a local spa. You’ve been warned!

For some of my predictions, 2013 may be too early. But I stand by these as where I think the world is headed over the next year or so. A more interconnected world, dominated by cloud computing and a connected web of physical devices, all controlled by our smartphones and tablets.

• OVF 1.1 became the first national and first international standard for virtualization and clouds.
• The industry recently released the Cloud infrastructure Management Interface Standard (CIMI) as well as an OVF 2.0 specification on the way.
• We saw the launch of a number of interoperability events that start to test early implementations of cloud interface specifications.
• OVF is now supported by industry players beyond VMware, including Citrix, Microsoft and Amazon

2013 will be an important year for cloud as customers continue on their journey from virtualized datacenters to a hybrid cloud model. Many organizations have implemented some cloud infrastructure – some private and some public cloud – and I believe customers will look at how to pull all of this together via hybrid cloud. When implementing a hybrid cloud environment, customers will view performance, functionality, portability, security and standards as increasingly important attributes.

We’ve also seen the cloud market and available solutions grow rapidly in 2012, including tools, platforms and interfaces. I predict customers will continue to maintain a close focus on the interoperability of their cloud solutions in 2013, even more than in previous years. Customers today want choice and do not want to be locked in to a single platform or vendor – so when running multiple clouds (VMware, AWS, OpenStack), interoperability will be key to a successful multi-cloud environment.

As I said in a previous blog, I think our industry will continue to make good progress in 2013 on interoperability testing – and more products will start supporting a set of open standard interfaces. However, I do not see a debate or the consolidation to a single set of standards happen in 2013. I expect that the debate will continue and that it may be until 2015 before we see industry and customer agreement on those standard and required interoperability interfaces.

It’s never a dull moment in the land of cloud standards, and I’m looking forward to witnessing and being a part of the progress that will take place in 2013!

Because the topic is of broad interest, I’ve published the slides in SlideRocket and added an audio track that explains why virtualization should not be viewed simply as a tax one pays to move into a cloud environment, but rather as an approach that can enable new capabilities that are either difficult or impossible to deliver in an un-virtualized HPC environment; a number of these capabilities speak directly to the most common objections raised to creating centralized computing facilities. I also give examples of what virtualized HPC deployments look like and provide a summary of our latest guidance on how well HPC applications perform on our platform.

You can access the Slide Rocket presentation here.

I want to extend a huge thank you to our community for your participation in this contest.  Given the high level of interest and success, we are already planning our next Open Innovation event.  Be on the lookout for more information after the start of the New Year!

Written by guest author: Joe Baguley

In my role I’m able to spend a lot of time talking to CIOs and senior IT leaders from our EMEA customer base, and have noticed a trend amongst these customers. It’s one that I discuss quite a bit: IT can be and is being used as a strategic weapon.

There seem to me to be two types of IT organization – one that has a leader who reports to the CFO, and one where the leader reports to the business or directly to the board. When reporting to the CFO, IT is generally seen as a necessary evil because it’s part of the cost of doing business and a cost that must be reduced – a strong theme in IT throughout the last few years! But when IT reports to the business or board, I’m seeing some exciting things happening in our industry – because these are the organisations that view IT as a weapon for their business and one that can directly affect the performance of the business for the better.

Recent history is littered with big name firms that were blind-sided by someone using technology to change their market, and were too late to react. I am told by IT leaders that this inability to react quick enough is due to existing organizational structures and the silos they work in. How can a company rapidly deploy a new solution for the business when its existing processes require gaining agreement from all the necessary stakeholders in IT (compute, storage, networking, security, DBAs etc.) which typically takes months to do along with numerous customer conversations? They are definitely looking for ways to change and be more agile.

However, some are embracing change and reacting. An IT leader at an’old’ large investment bank described their business to me as one that is “now a technology company that does banking”.
They turned their IT organisation on its side as a first step. Taking the original siloed empires and combining them into stacking layers – at the top “End User”, then “Platforms” which puts the Developer, Application and Database teams together who then use services provided by the “Infrastructure” team consisting of Compute, Storage and crucially Networking.

While banking is arguably a long way from DevOps due to segregation of duties and other regulations, this grouping will greatly help the bank’s agility in terms of responding to business needs. By aligning priorities between previously differently-goaled and measured teams, the creation of more siloed technology stacks in the future will be avoided.

This is exactly the kind of world VMware is building for its customers via the software-defined datacenter!

We are also starting to see the industrialisation of IT – building variant, supportable and agile solutions out of standardised components managed with a new policy-based automation at scale. But to do this requires a leader with a clear vision of how to evolve to a service-focused mentality and can clearly communicate business value and revenue results from IT investment as opposed to just cost savings.

I met with an oil company whose CIO now reports into the head of exploration technology. They are actively looking into how they can use technology to get ahead of the competition (VDI was a hot topic here, as they try to deliver high-end compute resources to increasingly remote and badly connected places in Africa) and how they can start making architectural choices now to prepare for the future.

I also had the pleasure recently of presenting at IPExpo with Sega about how they have used hybrid vCloud with a European vCloud provider to increase the scale of their games testing process and increase the scale of testing – the end result being they are bringing better quality games to market faster than their competitors, with greater insight than before into the cost of such activity – a project led by business need, rather than cost optimisation. It’s a very compelling story, and you can read more about it here.

I’m also seeing a growing interest amongst our VMUG EMEA members in VMware’s new vCloud Automation Center product (formerly DynamicOps) and its inclusion in the vCloud Suite. These VMUG members see that they, as the virtualisation experts, are now in a prime position to be the future leaders of IT. Their trailblazing experience in virtualising and automating compute has become increasingly relevant as both storage and networking are now becoming virtualised too.

Do you see yourself as someone that can make IT strategic and have it be a force of change for your business? Or do you think IT needs to been performed only in a cost effective way?

In addition, the DMTF released a whitepaper earlier this year that presents several use cases that could benefit from standards interfaces and data formats. These uses cases explore a variety of scenarios that highlight the need for interoperability, including the need for cross-platform tools to validate compliance controls and security procedures adherence. Other use cases looked at the ability to verify controls around geo-location. I encourage you to read through this whitepaper, to see if you agree with the scenarios and decide if they reflect your current IT environment.

Following the whitepaper, the DMTF recently published a draft of a new specification called “Cloud Auditing Data Federation (CADF) – Data Format and Interface Definitions Specification.” This specification provides a data format and interface definitions that support the federation of normative audit event data along with domain specific identifiers and standardized event classifications to and from public clouds and / or hybrid clouds. This data can then be used to create customized reports and logs. Today there are many logs and other interfaces from which you need to understand and decode to find the needed information and there are many different codes and messages that must be understood to get a true picture. With this new standard, there is hope that over time cloud providers will be able to report information in a consistent way through a standard REST-based interface. This will enable better verification and contractual commitments by cloud consumers.

Adoption of the CADF specification as well as other open standards from cloud providers’ management platforms such as Cloud Infrastructure Management Interface (CIMI) and the Open Virtualization Format (OVF) would also go far to instill greater trust in “cloud hosted applications.” Cloud consumers rely on their service provider to deliver specific services along with associated service levels – however, once this specification is implemented, consumers will be able to have complete confidence in the provider to deliver without fail. This would be a significant step forward in fulfilling the promise of an open cloud marketplace, which I know customers are ultimately after.

What do you think about the DMTF’s whitepaper and new specification? Will these efforts help you on your cloud journey? I welcome your feedback and comments.

It has been very exciting to see how fast things have changed since I first visited Japan three plus years ago to discuss mobile virtualization. iPhone had just arrived and Android was still “interesting.” Even though feature phones are still prevalent in Japan, Android and iOS have taken off like a rocket ship.

These cool devices with powerful operating systems have kicked off two significant trends:

1. Users are downloading more applications onto their mobile devices than ever before.
2. BYOD! Enough said!

Consequently, we are seeing devices used for multiple purposes – work and personal i.e. dual persona devices. Almost all devices will have personal content and corporate content (apps and data) no matter who actually paid for the device – corporation or employee. So the critical thing is to completely isolate personal from corporate to prevent data leakage and have IT only manage the corporate content. By leveraging virtualization, we are able to offer the most secure isolation of these environments. So we are excited to offer a solution with our partners to address customer requirements to manage and protect corporate content without compromising consumer experience.

Please visit us at the vForum in Tokyo this week to learn more about the product, discuss your needs and how we can address them, and to see cool demos working on the beautiful Motorola handsets. We will have breakout sessions to dive into product and architecture details as well.

A group of women from VMware recently attended the Grace Hopper Celebration in Baltimore.  This is an important event that brings together technical women for several days to discuss technology trends, get career support and learn from thought leaders.  While I was not able to attend this year’s event, I had a chance to catch up with some of those who did and want to share their direct feedback and experiences:

“This was my second time attending the GHC and it was as energizing as the first time! It is an amazing feeling to be surrounded by so many accomplished women each with a unique story.  I was lucky to be included in a session led by Rebecca Shambaugh, author of  “It’s Not a Glass Ceiling It’s a Sticky Floor”. The session really gave me a different perspective.  The exchange with the other women attending the session gave me lots to think about how we each have found different ways to balance our lives and not give up on our careers goals. I really encourage whoever attends to be open to sharing your experiences and listen to others’ stories. You might be surprised how much more you will get out of the conference beyond just technical information.”

“I have been spending some of my time working on ways to attract more women to VMware R&D and there were many eye-opening moments for me at this year’s Grace Hopper Celebration. I got a much greater appreciation for the young women who are eager to learn from women in the industry, and what an impact these conversations can have on the lives of women everywhere — especially in countries where there are few women in science. Most importantly, I learned what a profound impact these connections can have on women already established in the industry.”

Sharing career perspectives.

“I could not have been more proud to be a member of the VMware team that traveled to Baltimore for the Grace Hopper Celebration this year. From a University Recruiting perspective, one of the great benefits related to attending the conference is the opportunity to engage with students from top-notch Computer Science and Engineering programs that we don’t always have a chance to visit frequently during recruitment seasons.  Being able to meet students from different schools, allows us to discover innovative programs for new partnerships in the future. If you attended the Grace Hopper event this year and are interested in VMware’s career opportunities, I encourage to apply to either our New College Graduate or Intern positions here. We’re interested in learning more about you and your interests, especially if we did not have a chance to catch up in Baltimore one-on-one. Finally, if you missed us this year, you’ll know where to find us next fall…Minneapolis, here we come!”

This year, I attended my second Grace Hopper Conference and was, once again, blown away by the sheer energy and scope of the conference. This year, sessions included everything from “Lessons Learned” for fresh graduates, to “Tips & Tricks” for working professionals in the industry (between 3-7 years) and more senior management and technical tracks. There was a particularly big focus on Security and Innovation this year with speakers such as Vivek Wadhwa, Meg Layton and Michelle Guel. It was of particular interest to several of us developers that the SANS Institute’s annual conference was right next door – too bad I didn’t bring my little sniffer with me – heh, heh. We made sure to encrypt *everything* we possibly could.  For me, GHC was a treasure trove of great female and male role models – how often do you get a chance to meet, shake hands with and even go out to dinner with such giants of the industry?!  And I came away with a huge list of contacts, knowledge and connections that span the globe and industries – connections that triggered ideas and concepts in my information-packed brain.

VMware interns enjoying the event.

Since the release of Serengeti, VMware has learned a tremendous amount from our customers about using virtualization as the platform for big data workloads and Hadoop. These customer conversations provided us with solid reasons to virtualize Hadoop and other big data workloads.

Today, we’re introducing the third significant release of project Serengeti. In this blog I cover some of the new headlines of the work accomplished since the Hadoop Summit in June, which include:

• Support for Dynamic Elastic Scaling
• Hive JDBC connections
• Data upload/download interface
• Ability to configure infrastructure topologies
• Placement controls for Hadoop nodes on Physical
• Hadoop tuning configurables
• A community contributed UI

At this October release, we’re also supporting Cloudera CDH3u3, Greenplum GPHD 1.2.0.0, Hortonworks HW 1.0.7, and Apache 1.0.1.

The Hadoop Users

There are two key users that interact with the Hadoop system:

• The Data Scientist: This would typically be  the line of business person who is tasked with analysing and providing intelligent insights from the data. Their primary concerns are to have quick time to insights on the data. Here, getting quick access to a cluster on demand, and getting reasonable performance from their cluster are primary concerns.
• The IT Guy: This would include the architect, admin and CIO. They run the infrastructure and their primary concerns are keeping up with the demands of the business, cost efficiency, and keeping the services available.

Virtual Hadoop helps the data scientist get rapid access to new virtual clusters on demand, reducing the time taken to provision new data environments. For the IT guy, we can provide a common infrastructure on which to run many big data workloads, removing the need for silos of different solutions, and reducing the cost and complexity of the systems required to host big data workloads.

Multi-tenancy, Mixed Workloads and Elastic Hadoop

During our investigations of how to run Hadoop on virtualization, we found that there are several new ways to configure Hadoop that can take advantage of the software-definable underlying resources. Consider today that Hadoop is typically configured on a relatively static set of physical resources, and takes over 100% of those physical resources. This makes it hard or nearly impossible to share those resources with other applications. Typically, a whole physical host would be assigned to either Hadoop or something else, leading us to build vertical silo’s of application-specific clusters.

In a virtual configuration, it’s of course easy to share those resources amongst multiple applications. But first, we should discuss what might motivate us to want to do that.

While discussing big data with our customers, we’ve seen a continuous pattern of multiple components and applications that make up a full big-data application. For example, there is typically Hadoop, layered Hadoop components (such as Hive, etc.), several flavors of NoSQL (including Cassandra, hBase, MongoDB), fast databases (in memory technologies such as GemFire, Redis, Memcache etc.), scale-out column oriented SQL databases, and then other non-map-reduce applications which are part of data integration and visulation.  Some of these are batch oriented, like Map-Reduce, but many are long-running apps that need the traditional services of an application deployment and life cycle management system.

The main point is that rarely do I just see Hadoop. It’s always a mix of different application types. The other types of applications that complement Hadoop should be able to share the same cluster as Hadoop, but may leverage existing application deployment and management frameworks, such as those provided by virtual infrastructure.

Since the resource demands of each workload shift over time, we can make much better use of the infrastructure if we can mix these workloads on the same cluster. In the ideal scenario, we can dynamically size of the amount of resources available to each.

Frameworks for Mixed Big Data Workloads

Given the need to support mixed types of big data workloads, there is an industry level focus on creating a shared infrastructure platform that can support concurrent operation of these different application types. There are several efforts underway in the community to address this, with a diverse set of approaches. Some notable examples are:

• Apache YARN – which adds the some capabilities of running other non-map-reduce workloads inside the Hadoop framework. YARN allows multiple application types to run in YARN containers, of which map-reduce is just one of a few types. The initial work shows map-reduce and MPI jobs running along side of each other on the same cluster. The Hadoop resource manager is used to partition the resources according to policy. Arun discusses the need to run mixed workloads on the same cluster in his InfoQ article. He cites some examples of Map-reduce, Twitter Storm, Apache S4 and OpenMPI sharing the same platform environment.
• MESOS – a resource-brokering framework, which allows Hadoop to share the resources with other application frameworks. Started originally at UC Berkeley, MESOS allows multiple Hadoop frameworks to run along side of non-Hadoop workloads on the same cluster, integrating with the native resource and configuration management capabilities of the underlying OS to provide some level of isolation between frameworks.

• Serengeti – VMware’s mixed workload framework that leverages Virtualization to allow Hadoop to run along side of other applications in the same cluster. We can use virtualization to provide the runtime containers for many types of applications concurrently on the same platform, and leverage the strong isolation that virtual machines provide.

Common to all of these approaches is a need to provide the three standard levels of isolation:

• Configuration Isolation: the ability to allow each application type to be configured independently of each other. This extends beyond mixed workloads, and is useful for even different versions of the Hadoop environment on the same cluster – commonly to support development, test and production.
• Resource Isolation: the ability to independently control the quality of service of application environments. This is most commonly used to prevent the “noisy neighbor problem”, so that a busy application doesn’t impact the performance of another. A resource management framework is used to set priorities or amounts of resource available to each application.
• Fault Isolation: the ability to prevent the failure of one environment from impacting another. For example, if someone runs an experimental version of Hadoop in test, we don’t wan that to impact a production version that is running daily operations.

If we contrast the approaches, we see that YARN is bringing some new workloads to Hadoop, but will continue to be quite restrictive in what it can run and control, since it lacks some of the key foundational primitives to run generic applications (e.g. block or Posix storage, generic networking, etc.). MESOS is heading towards a similar place to managed virtualization, i.e. a common infrastructure approach to running different workload types, with an emphasis on resource management.  In the end, a full virtualization platform has a strong blend of the provisioning, infrastructure services (e.g. storage), management and distributed resource controls required for a wide variety of applications.

The Need for Elasticity

Most big data workloads utilize a scale-out distributed architecture. This means that if we mixed these workloads, we will ultimately need to vary the amount of resource allocated to each by scaling the number of nodes in each workload.  In one recent discussion with a public social application provider, the need to have differentiated quality of service for their production hBase and map-reduce workloads was called out. Often, the map-reduce workloads would impact the performance of hBase, so they wanted stronger isolation between hBase and map-reduce. Both of these workloads are scale-out architectures, and the actual number of nodes is a function of the application demand for a given time of day, and stage of the applications growth pattern. Another customer cited the need to support a scale-out SAS environment for numerical analysis and map-reduce. In a third customer example, there is a need to run more Hadoop map-reduce on a large standby cluster, that is mostly idle when it’s not be used for fail-over workloads. They want Hadoop to be able to shrink, but still be available for some access to the data, and then grow again.

Scaling workload up and down is not a new science – it’s something we’ve done for years in virtual and cloud environments for stateless applications. The hard part is scaling an environment which contains a large amount of distributed state. For example, if we want to scale Hadoop, each node of Hadoop may contain upwards of 20Terabytes of data, which means that growing Hadoop onto a new node will require re-balancing and replication of that data onto the node. Likewise, shrinking of the environment will require evacuation of that 20Terabytes of data and re-replication onto other nodes. This can be done, but it’s expensive and not something that could be done often. This is why Amazon elastic map-reduce has limited purpose, because when you rent the cluster, it’s empty. When you finish using the cluster, you have to do something with the data, since the cluster goes away. The blob service provides somewhere to store some of the state, but its limited bandwidth limits the size of the data that can be practically used by the map-reduce cluster.

Enabling Elasticity and Multi-tenancy in Hadoop

The ideal elastic map-reduce would allow us to rapidly grow and shrink the amount of map-reduce on the cluster, without the constraint of moving massive amounts of data. The data would be always available by a resident service, and the amount of map-reduce can vary as required.

As VMware explored these needs, we found an interesting set of deployment options appeared for how Hadoop can be configured in a virtual environment that supports these needs. Since Hadoop is a combination of both a distributed file system and a scale-out distributed runtime system, we are able to consider these layers separately, which open up new options for scaling and multi-tenancy.

We considered three primary ways to run Hadoop in a virtual environment, which would allow varying amounts of elasticity and multi-tenancy.

1. Traditional Hadoop – In this model, we can easily virtualize Hadoop the same way it run’s on physical systems today. We would run one large Linux instance on each host, which could contain the runtime portion (the task-tracker) and the distributed file system node (the data-node). Both of these services run as a JVM inside the same Linux instances, and communicate with each other via domain sockets. This model allows us to provision a virtual Hadoop cluster rapidly on virtual, but has the limitation that it’s hard to scale up/down because of the data in each VM (the 20Tetabytes in our example). It would also be difficult to scale the amount of memory up/down if we want to share the hosts’ resources with other applications.
2. Separated Compute and Storage – In this model, we place the run-time nodes (task-tracker) and distributed file system node (data-node) in separate virtual machines. Surprisingly, this model is quite easy to configure, and does not required changes to the Hadoop source. We run the Hadoop file system as a distribute service on the cluster, to manage the disks on each node and present an always available file system service to the Hadoop runtime nodes. We can then run any number of runtime nodes (virtual machines with the task-tracker inside) against that data service. To expand the size of the virtual Hadoop cluster, we simply add more runtime virtual machines, and to contract we simply power off some of the virtual machines. The data stays resident in the data layer.
3. Separated Compute and Storage with Multiple Tenants – Taking this model to the next step, it’s possible to run multiple Hadoop runtime clusters against the same data service. This allows us to build a multi-tenant Hadoop service, by offering virtual Hadoop clusters each with it’s own set of virtual resources. Each tenant can run their own version of Hadoop, Linux or environment, fully isolated with distinct versions, resources and configuration.

Virtual Hadoop and Serengeti

We have been configuring and exploring these architecture in the lab, and extending project Serengeti to support these models.

We have added several new capabilities to Serengeti, including the ability to specify separate compute and data clusters, the ability to provide Hadoop tuning into the configuration, and the ability to deploy a Hive server.

Separated Compute/Data Clusters in Serengeti

Also in milestone 2, we introduced the capability of specifying a deployment with separated compute and data-node virtual machines. This lays the foundation for building our elastically scalable Hadoop clusters.

A blog by my colleague Tariq Magdon-Ismail, discusses the architectural implications to separated compute and data for Hadoop, and some performance measurements that answer some of the key questions about deploying Hadoop inside virtual machines in model 2.

Dynamic Cluster Resizing

The big theme for this week’s introduction of Serengeti milestone 3 is dynamic scaling. In this release, we are fully embracing virtual resources as the backing for Hadoop, by allowing Hadoop to hot add/remove Virtual Hadoop nodes on the fly. This gives us the capability to rapidly grow or shrink a Virtual Hadoop cluster, based on the changing needs of the application and user mix being serviced.

See here for a demo of Serengeti dynamic Hadoop Scaling.

New to milestone 3 is a Virtual Hadoop Manager (VHM), which provides a dynamic interface between Serengeti and vSphere vCenter. The VHM allows Serengeti to power up/down Hadoop nodes on the cluster, which allows dynamic grow/shrink of the Hadoop cluster while jobs are running.

We leverage Serengeti’s capability to deploy
Hadoop in a separated mode so that we have multiple virtual machines for the task-trackers and data-nodes. By default, Hadoop will use all of these provisioned nodes to run Hadoop. With VHM, we interface with Hadoop’s job tracker to vary the active number of virtual machines dynamically. It does this by using an algorithm to determine the correct number of powered on task-trackers per host, and then updating the exclude list on the job tracker to change the active scheduler configuration.

The architecture of VHM is shown in the diagram above. The VHM interfaces with vCenter to gather statistics about each host, information from Hadoop about the current task-tracker configurations, and inputs from the Serengeti user interface. These inputs are processed by a pluggable algorithm, and output as actions against vCenter (for power on/off) and to Hadoop’s job tracker (for node activate/deactivate).

Changing the size of the cluster is through a new command and a user-interface component:

serengeti> cluster limit --name myHadoop –activeComputeNodeNum 8

To enable all the TaskTrackers in the “myHadoop” cluster, use the “cluster unlimit” command:

serengeti> cluster unlimit --name myHadoop

The VHM is open sourced in Serengeti M3.

Several other key features are included in Serengeti M3, including the ability to upload infrastructure topologies, hints for placement control, and a new user interface.

Hadoop using External Distributed File System

There are a growing number of environments where the compute portion of Hadoop connects to a distributed NAS solution, which replaces HDFS. An example of this is the Isilon scale-out NAS system from EMC. In this configuration, just the job-tracker and task-trackers (the compute portion) of Hadoop are deployed, and the scale-out NAS provides the Hadoop compatible storage. There’s a great demo of the Isilon-Serengeti integration here.

To facilitate these configurations, Serengeti now supports deploying Hadoop with its compute nodes (task-tracker) deployed in separated virtual machines.

To support the Isilon example, we want to have just a compute cluster. In the cluster configuration file, a compute-only cluster can be described by specifying the Hadoop master node to contain just a job-tracker, and the worker nodes to contain just the task-trackers. Here is an example of this configuration:

{
    "externalHDFS": "hdfs://hostname-of-namenode:8020",
    "nodeGroups": [
        {
            "name": "master",
            "roles":  [
                "hadoop_jobtracker"
            ],
            "instanceNum": 1,
            "cpuNum":  2,
            "memCapacityMB":  2048,
        },
        {
            "name": "worker",
            "roles": [
                "hadoop_tasktracker",
            ],
            "instanceNum": 4,
            "cpuNum":  2,
            "memCapacityMB": 1024,
            "storage": {
                "type": "LOCAL",
                "sizeGB": 10
            },
        },
        {
            "name": "client",
            "roles":  [
                "hadoop_client",
                "hive",
                "pig"
            ],
            "instanceNum": 1,
            "cpuNum": 1,
            "storage": {
                "type": "LOCAL",
                "sizeGB": 10
            },
        }
],
“configuration” : {
}
}

Placement Policies

In milestone 2, we also support directives to affect the placement of virtual Hadoop nodes on specific hosts. This is important when creating specific relationships between compute and data nodes, now that we allow separation of compute from data. For example, we often want to create one data node per physical host, and then a variable number of compute nodes per host. To allow this to be described, we provide the placementPolicies directive.

{
  "nodeGroups":[
    {
      "name": "master",
      "roles": [
        "hadoop_namenode",
        "hadoop_jobtracker"
      ],
      "instanceNum": 1,
      "cpuNum": 4,
      "memCapacityMB": 2048,
    },
    {
      "name": "data",
      "roles": [
        "hadoop_datanode"
      ],
      "instanceNum": 4,
      "cpuNum": 2,
      "memCapacityMB": 1024,
      "storage": {
        "type": "LOCAL",
        "sizeGB": 50
      },
      "placementPolicies": {
        "instancePerHost": 1
      }
    },
    {
      "name": "compute",
      "roles": [
        "hadoop_tasktracker"
      ],
      "instanceNum": 8,
      "cpuNum": 2,
      "memCapacityMB": 1024,
      "storage": {
        "type": "LOCAL",
        "sizeGB": 10
      },
      "placementPolicies": {
        "instancePerHost": 2,
        "groupAssociations": [
          {
            "reference": "data",
            "type": "STRICT"
          }]
      }
    },
    {
      "name": "client",
      "roles": [
        "hadoop_client",
        "hive",
        "pig"
      ],
      "instanceNum": 1,
      "cpuNum": 1,
      "storage": {
        "type": "LOCAL",
        "sizeGB": 10
      }
    }
  ],
  "configuration": {
  }
}

Hadoop Configuration Tunings

Serengeti now allows tuning of the site and instance configurations. Serengeti provides a simple and easy way to tune the Hadoop cluster configurations including attributes in core-site.xml, hdfs-site.xml, mapred-site.xml, hadoop-env.sh. You can set the configurations when provisioning a Hadoop cluster or modify the configuration afterwards. Configuration changes can be done at cluster level or node group level. In cluster level, changes will apply to all the nodes in a cluster. In node group level, changes only apply to a group of nodes.

This is done through the spec file with the cluster config command:

{
  "nodeGroups" : [
    {
      "name": "master",
      "roles": [
        "hadoop_namenode",
        "hadoop_jobtracker"
      ],
      "instanceNum": 1,
      "instanceType": "LARGE",
      "cpuNum": 2,
      "memCapacityMB":4096,
      "storage": {
        "type": "SHARED",
        "sizeGB": 20
      },
      "haFlag": 'ft',
    },
    {
      "name": "worker",
      "roles": [
        "hadoop_datanode",
        "hadoop_tasktracker"
      ],
      "instanceNum": 3,
      "instanceType": "MEDIUM",
      "cpuNum": 2,
      "memCapacityMB":2048,
      "storage": {
        "type": "LOCAL",
        "sizeGB": 30
      }
    },
    {
      "name": "client",
      "roles": [
        "hadoop_client",
        "hive",
        "hive_server",
        "pig"
      ],
      "instanceNum": 1,
      "instanceType": "SMALL",
      "memCapacityMB": 2048,
      "storage": {
        "type": "LOCAL",
        "sizeGB": 10
      }
    }
  ],
"configuration": {
    "hadoop": {
      "core-site.xml": {
        // check for all settings at http://hadoop.apache.org/common/docs/r1.0.0/core-default.html
        // note: any value (int, float, boolean, string) must be enclosed in double quotes and here is a sample:
        // "io.file.buffer.size": "4096"
      },
      "hdfs-site.xml": {
        // check for all settings at http://hadoop.apache.org/common/docs/r1.0.0/hdfs-default.html
      },
      "mapred-site.xml": {
        // check for all settings at http://hadoop.apache.org/common/docs/r1.0.0/mapred-default.html
      },
      "hadoop-env.sh": {
        // "HADOOP_HEAPSIZE": "",
        // "HADOOP_NAMENODE_OPTS": "",
        // "HADOOP_DATANODE_OPTS": "",
        // "HADOOP_SECONDARYNAMENODE_OPTS": "",
        // "HADOOP_JOBTRACKER_OPTS": "",
        // "HADOOP_TASKTRACKER_OPTS": "",
        // "JAVA_HOME": "",
        // "PATH": "",
      },
      "log4j.properties": {
        // "hadoop.root.logger": "DEBUG,console",
        // "hadoop.security.logger": "DEBUG,console",
      }
    }
}

HA and FT improvements

In Milestone 2, you no longer need to enable Fault Tolerance (FT) in the vSphere client. The setting is done by Serengeti automatically if you want to do so. When you enable High Availability (HA) for namenode or jobtracker, the service is configured to be monitored as well. It makes the HA and FT setting extremely easy.

The haFlag attribute now accepts “on” for HA protection, “off” for no protection, or “ft” for FT protection.

Hive server

Hive server can be deployed and configured automatically. It provides JDBC/ODBC remote connection. Just add “hive server” as one of the roles on the node you want it to run on.

Rack Configuration Profiles

There are now fully customizable configuration profiles to allow flexible topology descriptions, which allow:

• Setup of Physical Rack-Hosts mapping topology to Serengeti.
• Fully control the placement of Hadoop nodes including group association and rack policy.
• Define Hadoop topology including RACK_AS_RACK, HOST_AS_RACK and HVE.

With this week’s Serengeti release, VMware now supports several new capabilities for big data in a virtual environment. We look forward to interacting on some of the new proof of concepts and deployments around the latest release of Serengeti!

In his joint presentation at Hadoop Summit 2012 titled “Hadoop in Virtual Machines”, Richard McDougall talked about  the benefits and challenges of virtualizing Hadoop. In particular, he introduced the idea of separating Hadoop’s compute runtime from data storage  on virtual infrastructure and touched on why this architecture is both desirable and feasible in a cloud environment. In this blog post I hope to examine this topic a bit further and present some initial performance results.

The Evolution of Virtual Hadoop

The common approach to virtualizing many applications is to perform a P2V (physical-to-virtual) migration where the physical deployment is directly cloned into virtual machines. Hadoop is no different. While this is a reasonable first step, there are a few drawbacks that make this model less than ideal in a cloud environment. The first has to do with elasticity. Adding/removing Hadoop nodes to increase/reduce compute resources available to jobs is a cumbersome, coarse grained, activity due to the tight coupling between the compute runtime and data storage layers (see below.) This makes it difficult to dynamically scale the virtual cluster to either make use of spare physical capacity or relinquish it to another workload.  In other words, the virtual Hadoop cluster is not elastic.

In order to gain elasticity, we need to be able to separate the compute layer from the storage layer so that each can be independently provisioned and scaled (the details of which are the subject of the following sections.) This is the next leg in the journey to virtualize Hadoop. This level of flexibility enables Hadoop to more efficiently share resources with other workloads and consequently raise the utilization of the underlying physical infrastructure. Moreover, this split architecture allows for more efficient hosting of multiple tenants, with their own private virtual clusters. It goes beyond the level of multi-tenancy offered by Hadoop’s built-in scheduler and security controls by relying on the hypervisor for much stronger VM level security and resource isolation guarantees. Further, since each compute cluster is independent, each tenant could have their own version of the Hadoop runtime. All these characteristics combine to form a very flexible, elastic and secure service that is the end goal to providing Hadoop-as-a-service.

Data Friction

Why is quickly scaling a Hadoop cluster difficult? In typical Hadoop deployments today, the compute and storage engines (known as the DataNode and TaskTracker) run inside each node, so the lifecycle of a node is tightly coupled to its data. Powering it off means that we lose the DataNode having to replicate the data blocks it managed.

Similarly, adding a node would mean we need to rebalance the distribution of data across the cluster for the optimum use of storage.

This decommissioning/provisioning and the large volume of data transfers involved makes responding to changing resource pressures slow and cumbersome. A further complication is that compute and storage capacity requirements change at very different velocities and we cannot respond to these different needs independently or efficiently. A potential solution is to scale storage by adding VMs (to add DataNodes) but scale compute by growing/shrinking a VM by adding/removing virtual CPU and memory resources. This would require close coordination between the hypervisor, guest OS and Hadoop and introduces some complexities that are beyond the scope of this blog. Another potential solution is to separate Hadoop’s compute layer from its data layer. This is a more natural solution, since Hadoop inherently maintains this separation via independent DataNode and TaskTracker JVM processes loosely coupled over TCP/IP.

Separating Compute from Data

The primary reason for having co-located DataNodes and TaskTrackers, is to maintain compute-data locality. This is  important in physical clusters because of network bandwidth limitations and the impact fetching large amounts of data from a remote node would have on job performance. But, virtual networking between VMs in a host is free of physical link limitations and throughput purely a function of CPU and memory speed. This coupled with the work we are doing on virtual topology awareness for Hadoop, so that nodes are aware of physical vs virtual host locality, opens up the prospect of being able to split compute and data into separate virtual nodes in our quest for the elastic elephant.

An added benefit with this approach is that the operational model is firmly established. It is a widely accepted technique in distributed application architectures in the cloud to add/remove VM instances as a way of scaling the workload. But a key question remains: what about job performance?

Performance

In the co-located setup data packets transfer between the DataNode and a Task over a loopback interface within the Guest OS. But when you split the JVMs into two separate virtual machines packets that are sent by Hadoop pass through many more layers before they get to their destination. Lets look at the typical path taken by a message that is transmitted from VM. The message is first processed by TCP/IP in the guest operating system. After the required headers have been added to the packet, it is sent to the device driver in the virtual machine. Once the packet has been received by the virtual device, it is passed on to the virtual I/O stack in ESX for additional processing, where the destination of the packet is also determined. If the packet is destined for a VM on the same host, it is forwarded over vSwitch to that virtual machine and not sent on the wire (yellow path in the figure above.) All other packets are sent to the physical NIC’s device driver in ESX to be sent out on the physical network (red path above.) Unlike VM-to-physical machine communication, VM-to-VM throughput is not limited by the speed of network cards since packets travel through memory. It solely depends on the CPU architecture and clock speeds. For example, with two Intel Xeon @ 2.80GHz processors running ESX 4.1, Linux VM- to-VM throughput approaches 27Gbps, which is nearly three times the rate supported by the 10Gbps network cards available on the market today.

Given the impressive raw virtual network bandwidth we can drive within a host today, we’ve started to gather some data on what impact it would have to Hadoop running in a split configuration. Following is the result of a 4 node TeraSort run where each physical node is an 8 core, 96GB machine with 16 disks. The HDFS replication factor was set to 2. Note also that these experiments were conducted with Hadoop 0.23 where the NodeManager replaced the TaskTracker from older versions.

In terms of both elapsed time and CPU cycles/byte used the split configuration is within 10% of the combined case. The primary reason we are focused on cycles/byte here instead of just CPU utilization is that we want to quantify the tax imposed by the split architecture on data movement.  In addition, CPU cycles afford us a way to normalize out CPU frequency and core count variations across different machines and tests, so that results are directly comparable. Note, even though the graphs above suggest that the split case is faster than combined, this difference is within experimental noise and is not statistically significant.

Apart from the TeraSort experiments above, we also ran TestDFSIO on a single node to understand the performance of raw I/O.

As with TeraSort, bandwidth in the split case is within 10% of combined, but cycles/byte is much higher. As an optimization we’ve done some early prototype work using VMCI (instead of TCP/IP) to communicate between Tasks and the local DataNode. With VMCI we were able to reduce the cycles/byte needed for the TestDFSIO read test.

Just the Beginning…

While this is still early days, preliminary data suggests that a split Hadoop model over virtual networking is quite viable. It adds the necessary flexibility to Hadoop with a tolerable impact on performance, and enables the cloud delivery of Hadoop. But, we still have more work to do to understand and tame this elastic beast. Watch this space!

It has been a very busy week at the OpenStack Summit. Unfortunately, I am not there but Quantum is.

What is Quantum?

Quantum is the networking-as-a-service capability and most talked about aspect to OpenStack’s Folsom release. Quantum has become a game-changer when it comes to enabling advanced networking in OpenStack environments, and the Nicira team at VMware has been leading the charge both in open source Quantum development and in deploying Quantum in production OpenStack environments.

Like many cloud stacks, the initial focus of OpenStack was on compute (Nova) and storage (Glance, Swift) with networking as a sub-component within Nova, offering very limited options of a “flat” network model (all tenants on the same public network) or “VLAN” network mode (VMs from each tenant are placed on a different VLAN).

OpenStack Quantum overcomes both types of limitations by providing:

• A flexible tenant-facing API to provisioning “virtual networks,” “virtual subnets,” and “virtual ports” to define network topology, addressing and packet forwarding policies
• A “plugin” architecture that lets cloud operators choose different back-end technologies to implement Quantum networking.

VMware’s Quantum Leadership

The Nicira team at VMware was instrumental in starting the Quantum project at the Diablo OpenStack Summit and has led the project both in terms of employing Quantum core developers and the Quantum Project Technical Lead (PTL). These efforts include significant open source contributions of code, documentation, community support, and OpenStack advocacy at conferences around the world.

More importantly, VMware is the clear leader in helping customers successfully leverage Quantum to bring advanced networking to production OpenStack deployments. The Nicira Network Virtualization Platform (NVP), which has been generally available for well over a year, powers many of the most advanced OpenStack deployments to date, including the largest OpenStack cloud in the world at Rackspace. NVP enables enterprise class networking at cloud scales using overlay network virtualization combined with a highly scalable and fault tolerant network control plane. Add in a healthy dose of management capabilities for monitoring and trouble-shooting cloud networks and integration with OpenStack Quantum and you have a truly powerful solution.

VMware at the OpenStack Summit

Quantum is a big deal, and we are very excited about the impact OpenStack Quantum has already had on the OpenStack community and are looking forward to an even broader engagement during the Grizzly release cycle. VMware CTO Steve Herrod gave a talk today highlighting other ways VMware is positively engaging with the community. One of the things he hit on was how VMware will improve support of the vSphere hypervisor (ESX) in OpenStack compute (Nova) by preparing a series of code commits. This is the latest evidence of VMware’s history of commitment to openness as well as to the broader interoperability needs of modern cloud computing. To hear more about VMware’s vision of the software-defined datacenter in a multi-cloud world, check out Steve’s blog here.

Management must change in a fundamental way. In today’s cloud era, there are faster moving parts, a requirement for greater scalability, a substantial need for self-service – and a core requirement of deep automation. VMware brings new capabilities to our customers to better manage their software-defined datacenters – with vCloud Automation Center, vCenter Operations Management Suite and vFabric Application Director. I address these areas in greater detail below, and in my keynote.

Self-Service is core to Cloud

• VMware is delivering self-service to customers with vCloud Automation Center. We recognize that cloud is about self-service access and automation; we are delivering tools to do this.
• vCloud Automation Center is a portal where users can select from a set of applications that will deploy and run automatically with full policy linkages. IT can enable users to move faster.
• Once the applications are deployed, users can manage and monitor apps on their own, in vCloud Director-based public/private clouds or external clouds such as AWS.

Application Management must evolve

• With the growth of applications in a software-defined datacenter, the challenge for users becomes ‘how do I manage all of these applications’ since practically any application can be run on vSphere.
• vFabric Application Director provides modular blueprints to users so they can easily compose applications, configure properties and settings inside the application, and ultimately have the applications deploy automatically.
• We’ve built the Cloud Application Management Marketplace for customers to download application blueprints from other software vendors and also share their own blueprints.
• If you want to learn more about our application management strategy and the supporting app management products that launched today, check out my colleague Preeti Somal’s blog.

Cloud Operations needs to be automated

• After your cloud is up and your environment is likely getting larger and change is happening more quickly, we know customers will want to automate the operations and analytics.
• vCenter Operations helps customers figure out what’s happening in their environment. The product has integrated and automated performance, capacity, and configuration management capabilities along with deep analytics.
• Some of the cool new features in the vC Ops product include:
  • Custom Groups which let’s users slice and dice views of their environment from a business perspective. This will help ensure your important business applications are healthy.
  • Best Practices have also been embedded in the product. We published our own deep knowledge of vSphere to help with troubleshooting.
  • Application Visibility: You can now determine if there’s a problem at the infrastructure level or the application level.
  • Single Pane of Glass: Customers now have a single dashboard to monitor the state of their environment across multiple clouds, private to public.

Multi-Cloud world

• VMware is enabling our customers to operate across multiple clouds – VMware, AWS, OpenStack, physical, you name it. While we feel it’s best to deal with a homogenous set of resources, that isn’t always the reality and we’re helping customers bridge the gap.
• Multi-hypervisor Manager is a new feature I previewed today in my keynote. Users will be able to manage both VMware and Hyper VMs while staying in their reliable VMware environment.
• My colleague Kit Colbert today blogged about VMware’s support for heterogeneous management and how we’re making this a reality for our customers.

¡Ay, caramba! Lots happening at VMware on the management front and we couldn’t be more excited about where it’s all headed. If you’re at VMworld Europe this week, I encourage you to check out all the management spotlight and breakout sessions, and walk around the VMware booth to learn more about what we’re doing in this space.

Now, it’s time to get some tapas…

]]> http://cto.vmware.com/software-defined-datacenters-clouds-puppetconf-2012/ Just over a week ago I had the pleasure of speaking at <a rel="nofollow" target="_blank" href="http://www.puppetconf.com">PuppetConf 2012</a> .  PuppetConf is a great conference focused, somewhat unsurprisingly, around <a rel="nofollow" target="_blank" href="http://puppetlabs.com/puppet/what-is-puppet/">Puppet</a> , a declarative, model driven framework for automating systems and IT management.  The conference brings together a community of pretty hard core systems admins from the <a rel="nofollow" target="_blank" href="http://en.wikipedia.org/wiki/DevOps">DevOps</a> world and beyond.  Automated management is clearly a key ingredient in building <a rel="nofollow" target="_blank" href="http://cto.vmware.com/the-software-defined-datacenter-meets-vmworld/">software-defined datacenters</a>  and Clouds, and something that we, at VMware, have more than a passing interest in. Preparing for it made me think.  A lot!  I was after a prism of simplicity, if you will.  A way of capturing the Cloud and software-defined datacenter concepts in a simple way, that allowed one to understand how we got to where we are, and then to easily extrapolate.  I like simple ideas.  For me the search for simplicity is at the heart of everything.  You can build highly functional, and almost inevitably, complicated systems using simple components.  And by understanding simple components and how they are [simply] combined, you can understand complicated systems.  If you can&#8217;t understand or express something in simple terms, then it is probably too complicated, and you are doing something wrong! As I mentioned in an <a rel="nofollow" target="_blank" href="http://cto.vmware.com/cloud-changes-everything/">earlier post</a> , you can view the emergence [...] http://cto.vmware.com/?p=1361 Mon, 08 Oct 2012 17:45:09 +0000 Just over a week ago I had the pleasure of speaking at PuppetConf 2012.  PuppetConf is a great conference focused, somewhat unsurprisingly, around Puppet, a declarative, model driven framework for automating systems and IT management.  The conference brings together a community of pretty hard core systems admins from the DevOps world and beyond.  Automated management is clearly a key ingredient in building software-defined datacenters and Clouds, and something that we, at VMware, have more than a passing interest in.

Preparing for it made me think.  A lot!  I was after a prism of simplicity, if you will.  A way of capturing the Cloud and software-defined datacenter concepts in a simple way, that allowed one to understand how we got to where we are, and then to easily extrapolate.  I like simple ideas.  For me the search for simplicity is at the heart of everything.  You can build highly functional, and almost inevitably, complicated systems using simple components.  And by understanding simple components and how they are [simply] combined, you can understand complicated systems.  If you can’t understand or express something in simple terms, then it is probably too complicated, and you are doing something wrong!

As I mentioned in an earlier post, you can view the emergence of Cloud systems through this prism, splitting the overarching trend into a couple of simple interacting threads – that of the evolution of once monolithic applications into ever more disaggregated and distributed services, and the evolution of infrastructure toward an ever more distributed fabric of resources connected with high speed, low latency networking.  These threads are long lived, but what has happened on the infrastructure side of the house in the last 5 to 10 years is quite profound.

Firstly the wholesale adoption of Intel x86 based servers is making the data center much more homogeneous from an infrastructure perspective.  Whilst this adoption was driven by cost, the net effect of greater homogeneity is that management, and especially automation, becomes much easier.  The smaller the number of types of things, and the slower their rate of change, the easier they are to manage, and the more effective and sustainable any automation of that management is.

Secondly the pervasive adoption of server virtualization is separating the application from the underlying infrastructure.  This separation yields simpler resource allocation/isolation and application mobility.  Whilst the adoption of server virtualization was primarily driven by the desire for greater efficiency, its most profound impacts are now being felt as a consequence of this mobility.  Once you have mobility, you can move workloads to the best place for them to run at any given time.  You can instantly provision them, you can move them from low capacity to high capacity as needed, you can move them off hardware that needs maintenance with no interruptions, you can move from one location to another for business continuity, and you can move them from within your business to the outside (hybrid Clouds) for burst capacity or vice versa when bringing new applications in house.

But there is a fly in the ointment!  Not everything has been effectively virtualized yet.  Whilst the application – encapsulated within a virtual machine, together with its operating system – has been liberated from its bonds to the physical server, constraints around storage, and in particular networking remain.  Software-defined networking (SDN) is effectively addressing this, further decoupling the application from the physical infrastructure, and increasing mobility.  It is for this reason that we purchased Nicira.

But what server virtualization and SDN combined show us, is another thread, that of separating the infrastructure operating software from the physical infrastructure.  The hypervisor does this for servers, and SDN control software does this for the network.  This, together with the ongoing evolution of storage virtualization, is what we call the software-defined datacenter.   We (as an industry, and VMware specifically) are effectively creating a layer of software, a meta-operating system, between the applications and the physical infrastructure.  This meta-OS will ultimately hide the complexity of underlying infrastructure from the applications and those who manage them, automatically placing applications on the right pieces of infrastructure, at the right locations, to deliver the right quality of service, at the right price, flexibly, safely, securely and compliant.

The software-defined datacenter provides the technology foundations for the Cloud, and provides the capabilities that enable the Cloud business model of self-service, instantly provisioned, pay as you go, elastic IT.  We are still in the early days of the meta-OS for the software-defined datacenter, but it will change the management of infrastructure, making it more or less fully automated and something that the average business, and average IT admin will no longer care about.  15 to 20 years ago, people cared intimately about the operating system scheduler and how it mapped application processes and threads onto processors.  Today, in the vast majority of cases, no one cares.  The operating system simply does it for you.  Similarly we are rapidly heading towards a point in time where no one will care which server any given application is running on and how they are connected.  The meta operating system for the software defined data center will take care of this for you.

So what about applications and PuppetConf?  How do they fit into this?

Well, if it is simplicity that enables better management and scalable, sustainable automation, then it is the virtual machine that provides that simplicity for managing applications from a day to day perspective.   Once an application [instance] is within a virtual machine (VM) it looks much like any other application in terms of most daily operations – provisioning, resource and performance management, availability management and so forth.  For most operations, managing the VM is a good enough proxy for managing the application instance itself.  Thus massive automation of the management of essentially homogeneous VMs, something that is relatively simple, results in the massively automated management of heterogeneous applications – something that is hard.

The next step in this journey is to be able to move from individual script based configuration of [relatively] simple OS and application stacks, to managing the configuration and topologies of complex, distributed applications as a whole, improving delivery and update times. This why we introduced vFabric Application Director, partnered with Puppet Labs and invested in them to complete the picture.  And it is why I was so excited to get to spend some time with the Puppeteers.  That, and the fact that such things are just plain cool.   The level of engagement of the Puppet community is energizing and infectious, and participating was a lot of fun.  Thanks to PuppeLabs for inviting me, and to Puppet community for listening and for feedback!

The Myths of DaaS

There is a lot of misinformation and FUD surrounding DaaS in the marketplace today. Some are the result of loose interpretation of licensing restrictions while others are more self-serving that the only DaaS achievable today comes in a form that a vendor has been selling for years and has decided to now attach the term ‘DaaS’ to describe it. We’ve heard all these claims and expect to hear more as the market continues to mature

We intend to dispel these myths and back it up by showing recent innovations that are making true-DaaS a reality.

What’s New

This year, at VMworld in the US and Barcelona we are showing two Technology Previews which help to enable multi-tenant DaaS to ride atop existing IaaS and methods to fully leverage vClouds. The level of interest from partners around accelerating their Time to Market (TTM) of DaaS and the desire to move quickly into product offerings is being spurred by similar macro trends we’ve seen in the traditional VDI market.

• The appeal of centralizing non-business differentiating tasks such as the operational management complexity of the PC so that it can be consumed “as-a-service” like other modern IT services
• The proliferation of new mobile devices incapable of native access to the ‘long tail’ of OS-specific (legacy) Windows applications
• New technology enablers that continue to drive down the CapEx investments required and subsequently benefit either an on-premises or hosted service subscription offerings
• Conversion of what is traditionally an upfront CapEx investment for VDI into a much more predictable monthly OpEx.

We are very excited to be part of solutions that are available now through our Service Provider partners. Through partnership with Desktone and with Dell’s DVS Simplified DaaS offering based upon their platform, we are able to extend the scalability of vSphere and the user experience of View to DaaS offerings in the market today.

The 8 Key Features of DaaS

With the choice not being ‘If’ DaaS, but rather ‘what DaaS’, we thought it would be good to reiterate what we believe to be the 8 key features of any DaaS offering – regardless of what vendor you choose:

1. Builds upon the foundation for private/public cloud – DaaS is a new and demanding workload for your private/public cloud infrastructure. It’s imperative that you start by building on a solid, proven foundation that is backed by a broad ecosystem.
2. Secure Multi-Tenancy – Multi-tenancy in our vocabulary goes far beyond what is possible with session based desktops running atop Windows. Trust in a cloud-orchestration framework that has been proven to keep your desktops as safe and secure as the rest of your cloud resources.
3. Elastic Resources – If you’re going to effectively scale-up/scale-down based on capacity to demand – and if you’re going to do so in such a way to maintain licensing compliance, how you manage resource elasticity will be critical.
4. Automated Provisioning – It’s possible to provision new desktops via scripts, but policy based automated provisioning delivers the resources on-demand your users expect.
5. Identity Management / Federation – Establishing trusts between a foreign identity service (AD domain) and a customer’s production AD is never a pleasant conversation. Ensuring proper isolation is in place means more than just having tenants in separate Organizational Units.
6. Role Management – DaaS isn’t VDI. Your users shouldn’t be concerned with tiered data-stores, your IT admins don’t need to worry about tenant creation. Establishing appropriate roles and workflows help optimize efficiency.
7. Self-Service Portal – As you look to scale to hundreds of tenants and thousands of desktops, enabling self-service will be the only way to effectively scale your operation. Make sure it’s simple and reliable.
8. Reporting, Metering & Chargeback – When producing a service offering, customers will demand detailed, accurate and timely reporting – including a track of charges incurred in real-time. These are capabilities already instrumented in your IaaS offering, don’t reinvent the wheel!

Accelerating beyond Technology Previews

It’s always exciting to show a technology preview to customers to see their reactions, but it’s even more satisfying to see advanced development activities accelerate from previews to future products and features. We will continue to innovate around DaaS and want to recap two technology projects that are moving forward quickly to help you achieve your DaaS objectives.

• VMware View Agent Direct-Connection Plugin– With View Agent Direct-Connection, we enable VMware View Service Provider (VSPP) customers’ additional flexibility in realizing the advanced user experience benefits of VMware View without a requirement for a specific brokering or connection management model. Using View Agent Direct-Connection, we leverage the extensible model of the VMware View virtual desktops and the ubiquitous nature of the VMware View clients. With an open client API, we enable our partners to integrate and the results are amazing.We previewed this at VMworld US in the Bring-Your-Own-Device Hands on Labs. Hundreds of concurrent users accessed their HoL leveraging this technology from almost any device imaginable. Attendees very own PC’s, Macs, Android tablets and iPads were all connecting to the Hands on Lab to get a preview of the latest VMware technologies.We are pleased to have partners such as Desktone leveraging this technology in their platform which is powering premier service provider DaaS offerings today including Dell, Navisite, Quest and many more!

  Learn more in this interview with Scott Davis, CTO of End User Computing and Peter McKay, CEO of Desktone.

• Project Catalyst– In this advanced development project, we are showing how VMware technologies can fully support the 8 key features of DaaS. Our intentions with this project are to articulate how VMware View and the multi-tenant nature of the vCloud Infrastructure Suite deliver on the promises of bringing a cloud orientation to DaaS just as a traditionally virtualization-centric workload has been to VDI.The early results of this project have been nothing short of incredible. From automated self-service provisioning of desktops as well as supporting infrastructure components and automation through fully leveraging the multi-tenancy of the vCloud Suite we are helping to illustrate why building atop a VMware vCloud solution can pay dividends in your DaaS offering.We’re not stopping here! – Project Catalyst continues to integrate other new and emerging technologies across our End User Computing portfolio. Technologies such as vCenter Operations for View and VMware Mirage demonstrate why by building top the foundational elements of VMware cloud infrastructure will rapidly enable adoption within a DaaS offering.

  Hear more from Scott Davis, CTO of End User Computing at VMware in this discussion with the lead architect of Project Catalyst, Andre Leibovici.

We are excited to be sharing how VMware can help you accelerate your move to virtual desktops with VMware View. We’re proud to have technology partners such as Desktone leveraging View features as part of a powerful DaaS solution already deployed at premier VMware service provider partners such as Dell. We recognize the power of this ecosystem.

Additional partners and customers approach us daily with new and innovative value-added services they wish to deliver as part of a DaaS solution and we welcome the opportunity to work with you to accelerate the shift of desktop workloads to the cloud.

Summary

Starting now and as evidenced by our partner offerings – DaaS is a reality today. As we have shown with Project Catalyst, VMware intends to continue innovate in this space in support of our partners and our customers to help you realize the benefits of what a VMware and vCloud based DaaS offering can achieve.

The opportunity for Desktop as a Service is real. It will help customers accelerate towards realizing the benefits of virtual desktops. It will help partners deliver a new and powerful business solution atop existing cloud infrastructure. VMware intends to continue to innovate at how we can take desktops from virtualized infrastructures to the cloud and we can’t wait to share with you what we’re working on next!

During the keynote, we demo’ed the next version of Serengeti, which allows Hadoop to be elastically scaled on a virtual platform. We showed a scenario of mixed workloads on the same platform, allowing time-based shifts in the amount of resources assigned to Hadoop and other workloads. You can see the highlights from the keynote here, and additional information on compute/data separation in an up coming blog.

In addition, Jeff Buell and I presented on the work VMware is doing to make virtualization the best place for Hadoop, including project Serengeti, HA and recent performance results. You can view our slides here.

During the conference, I also had the opportunity to chat with some big data superstars about their experiences and outlook on trends in that space:

• Video interview with Amr Awadallah – CTO, Cloudera 
• In this discussion, I asked Amr, the CTO of Cloudera some questions about the types of systems they are seeing deployed for Hadoop and Analytics. We dive into some detail about the rules of thumb for systems architecture, storage approaches, and the introduction of 10 gigabit ethernet.

• Video interview with Jim Kascade – CEO, Infochimps 

• Jim shares a great set of use cases for analytics and analytics enabled applications, including a discussion about the platform architectures required for building real time applications.

• Video interview with Zubin Dowlaty – CEO, Mu Sigma 
• Zubin and I discuss the common patterns of analytics applications, what we are seeing today and the trends towards real time.

Thank you to Zubin, Jim and Amr for your valuable insights! For those who will at VMworld Europe in a few weeks, check out the Big Data sessions and booths to learn even more.

I gave my talk to share some specific data about HPC performance in virtual environments, but I first described what new capabilities can be gained by virtualizing an HPC environment. I felt this was necessary because in HPC discussions it is often the case that virtualization is only discussed in a negative context: how much performance is it going to cost me? That’s a legitimate question, but it is also important to understand what additional capabilities might be gained in return to do a proper cost-benefit analysis.

Those additional capabilities include:

• “Bring your own software stack” — Rather than mandating what OS distribution will be run across an entire HPC cluster, a virtualized HPC environment can run whatever operating system, middleware, and applications each user needs. This is especially important in multi-department environments where different groups require different software configurations or have requirements that change over time. For example, organizations attempting to centralize their HPC compute resources to increase efficiencies and cost savings, can use virtualization to help address a significant concern of groups being asked to give up control of their dedicated resources: that they will be forced to use a standard software environment that does not match their specific requirements.
• Separate workloads — Applications can be isolated from each other by running them within their own virtual machine instances. This separation provides both security and fault isolation between running jobs, which is not possible in bare-metal environments where slot-based batch schedulers run jobs from multiple users within the same operating system instance. This isolation is especially important in environments where multiple jobs must be run on each physical machine to use all available compute resources and where data leakage between jobs is unacceptable or where cross-job interference significantly affects efficiency.
• Use resources more efficiently — In bare-metal HPC environments, jobs that are launched by batch schedulers are constrained to run where they are placed and they remain on those resources until they either complete, fail, or are killed. In contrast, virtual environments allow running workloads to be migrated from one physical machine to another, allowing policy-based, dynamic load balancing to  use resources more efficiently.
• Protect applications from failures — Since the virtual machine abstraction encapsulates the full state of the operating system and application, virtualization can be used to checkpoint applications so they can be restarted if they fail due to hardware or software problems. In addition to supporting this “fail and recover” approach, virtualization could also use live migration to proactively move applications (or pieces of MPI applications) from failing to healthy machines. This “move and continue” approach would reduce the need for full checkpoints, which require significant time and space to execute and which result in redoing lost work, which can be expensive if third-party software licenses are involved.

I focused on performance in the second half of the talk, making the point that single-process HPC applications across a range of vertical markets (Life Sciences, Digital Content Creation, Electronic Design Automation) generally show slowdowns of about 0-5% relative to bare-metal performance. For specific Life Sciences data, I referenced the paper Pragmatics of Virtual Machines for High-Performance Computing: A Quantitative Study of Basic Overheads by Cam MacDonell and Paul Lu of the University of Alberta. Here are their results for HMMer, an important biosequencing code:

And for GROMACS, a molecular dynamics code:

I then discussed distributed applications, starting with Hadoop which is a workload we have seen run faster in some cases when virtualized. We previously published a technical white paper about this, which can be found here.

Turning to more challenging cases, I showed the results of Intel experiments in 2009 showing HPCC and STAR-CD performance using InfiniBand in passthrough (VM DirectPath I/O)  mode. HPCC generally ran well with two exceptions, one of which we have explored and understand (MPIRandomAccess should be run with large pages to reduce TLB miss rates) and the other (NaturallyOrderedRingBandwidth) which has not yet been investigated. The HPCC results are shown below for two, four, and eight-node configurations.

STAR-CD, a computational fluid dynamics code,  ran with an overhead of about 15% in the tested configuration. The results for an eight-node case are shown below. As mentioned in the slides, STAR-CD is moderately latency sensitive due to the number of small messages it exchanges. Other less sensitive applications will see less overhead and more sensitive applications will see more.

Intel’s tests were run on DDR InfiniBand using ESX4, both of which are dated technologies today. To partially address this, I also shared our results from QDR InfiniBand experiments in 2011 that showed we could achieve ping-pong latencies under two microseconds with passthrough mode using vSphere 5.1. These results were recently reported in a research note, available here. We are also working with Intel and Mellanox to run additional MPI tests with newer hardware and software and expect to report those results soon.

My message to ISC Cloud attendees was that today’s cloud is not tomorrow’s cloud: As virtualized performance continues to advance and as cloud providers see a business value in deploying high-bandwidth, low-latency interconnects, the number of applications that will run well in a cloud environment will continue to expand. In the meantime, many single-process applications run very well in virtualized environments, MPI overheads may be acceptable for some applications, and virtualization can offer new capabilities not available in traditional, bare-metal HPC environments.

For the first time at a VMworld event, we had an Innovation Lounge on the Solutions Exchange which showcased posters from the three idea pitches, as well as seven other posters that represented other innovative projects underway VMware. The engineers behind each of these innovation projects accompanied their poster, and answered numerous conference attendee questions throughout the week. We even had one of our star summer interns showing off his cool robot that is under development for touch screen device testing.  The lounge also included information and demos about some of the innovation programs we run inside VMware to get new ideas cooking amongst attendees such as our Academic Program and one of our soon-to-be-released new projects, the Next-generation Education Environment (NEE) which is a cloud-scale solution for education providers.  Check out this video I did touring the lounge so you can see more.  Based on the popularity of the innovation demos and the lounge, you can expect to see more of this at future VMworlds!

Also new this year at VMworld was the opportunity for the Diamond sponsors to showcase new ideas they’re developing onstage during the General Session.  EMC, NetApp, Cisco, HP and Dell each got four minutes to pitch a new idea on how their product(s) can work best with VMware and get audience ratings.  It was a great way to learn about the innovation happening within our ecosystem and what customers can take advantage of from a technology perspective. You can see this along with all the cool demos Steve shared in the Day Two keynote here

Many of the VMware breakout sessions were focused on sharing previews of our new technology innovations either recently or soon to be released.  The Mirage project, which is an advanced technology for abstracting user profiles from the operating system, was just one example and the VMworldTV guys captured it well here.

Another innovation highlight at VMworld was the new way VMware is driving innovation by announcing our very first Open Innovation Contest. We’re asking our community to send us their best Fling ideas, and the contest winner will have their Fling built by VMware engineers. The winner also gets a free pass to VMworld 2013.  There are already close to 60 Fling ideas on the contest site, so check them out and vote for your favorites or suggest your own!  The contest runs through October 31, 2012.

Did you see something especially innovative at VMworld in San Francisco? I would love to hear from other attendees about what inspired them this year.  Meanwhile, I’m looking forward to doing this all over again at VMworld EMEA in Barcelona next month!

Bigger Virtual Machines

First, monster VMs got a bit more monstrous in vSphere 5.1. While we continue to support up to 1 TB of memory per VM, we have doubled the number of supported vCPUs to 64, which will be a boon to those wanting to run large-scale OpenMP or other threaded applications in a virtual environment. Those of you who regularly read the HPC entries on the CTO blog will recall that last year we presented a paper on vNUMA at the HPCVirt 2011 Workshop in Bordeaux that showed SPECOMP performance results running in virtual machines with up to 64 vCPUs, so you perhaps aren’t surprised to see this enhancement. That paper, by the way, is here and I recommend reading it if you intend to run NUMA-sensitive workloads in VMs that will span more than one physical socket.

Performance Counters

Second — and this is very cool — we’ve introduced support for virtualized CPU performance counters, which means tools like the Performance Application Programming Interface (PAPI) should now be able to enable profilers to gather performance data on applications running within VMs on vSphere. In fact, we’ve been funding the University of Tennessee to do exactly that as part of our VMware Academic Program (VMAP).

Our implementation provides several options to control how these performance counters function in a virtual environment. By default, we count only guest instructions for “instructions retired” and “branches retired” — these events do not increment when executing hypervisor instructions. All other events increment whenever the physical CPU is executing either guest or hypervisor instructions on behalf of the virtual machine. It is also possible to ignore all hypervisor code when incrementing the counters or to increment the counters regardless of whether guest or hypervisor code is executing. This flexibility supports different use-cases for performance data in a virtualized environment — from end-users interested in application performance to implementors interested in understanding virtualization overheads.

Enhanced vMotion

Enhancements to vMotion in vSphere 5.1 provide a new level of flexibility for live virtual machine migrations. Specifically, we can now support vMotion without the need for shared storage, a situation that arises on HPC clusters in which important application state is resident on the local nodes of a cluster, usually for performance reasons.

Extended vMotion supports local datastores

With enhanced vMotion, a migration copies both the virtual machine memory and its disk over the network to the destination host. This type of migration can be done within DRS clusters in small environments or across DRS clusters as would often be the case in large virtualized HPC compute farms.

PCI Device Sharing (SR-IOV)

Single Root I/O Virtualization (SR-IOV) is a part of the PCI standard that enables one PCI Express (PCIe) adapter to be presented as multiple separate logical devices to virtual machines. The hypervisor manages the physical function (PF) while the virtual functions (VFs) are exposed to the virtual machines. Previously, one could use VMware DirectPath I/O (passthrough) to expose an entire InfiniBand HCA to one virtual machine, but the HCA could not be shared between VMs as is sometimes useful in multi-tenant HPC environments or in environments in which individual MPI ranks are not able to consume the increasingly large amount of parallelism available from modern multicore systems. SR-IOV enables this sharing.

Note that using SR-IOV to gain direct guest access to a virtual slice of an HCA still disables use of vMotion and Snapshots — the same restrictions placed by use of VMware DirectPath I/O. This is why the research work being done on vRDMA by my colleague Bhavesh Davda in the Office of the CTO is so important: it aims to deliver CPU offload capabilities as well as bandwidths and latencies that are similar to those achievable with passthrough while maintaining vMotion and Snapshot capabilities, both of which can enable interesting use-cases for virtualization in HPC environments.

Auto Deploy

vSphere 5 included our first release of Auto Deploy, which supports provisioning stateless/diskless ESXi hosts using PXE boot and host profiles to customize each host. With vSphere 5.1 we’ve added two additional modes — stateless caching and stateful installs. Stateless caching stores a backup copy of the in-memory image on a dedicated boot device (local disk, SAN, USB) and will use this cached copy on reboot in cases where the PXE infrastructure or other components required to run in stateless mode are unavailable. The stateful install mode allows administrators to leverage Auto Deploy’s provisioning capabilities to configure a new host on first boot and then subsequently boot the system from its dedicated boot device. In such a scenario, the system will always try first to boot from its boot device and will only fail over to a network boot in the event of an error.

More Information

For additional information on the features mentioned above as well as the other enhancements and new capabilities included in vSphere 5.1, see the technical white papers available here.

What does all this mean for traditional Windows? Well, that’s a whole blog entry in and of itself… However, in brief, Windows has served many roles or functions for us in a monolithic manner when a “user” meant the same thing as a single physical device. As we move forward with a heterogeneous multi-device world, these roles are teased apart – broadly, some parts of the roles become properties of each of our devices, others become properties of the user and their environment that spans devices, and others become properties of applications and comprise the runtime environment within which they execute. The Windows core OS properties that are part of the device take on more of the characteristics of device firmware and managed as image updates. User settings and environments will be cloud workspace based and span all of our devices and application runtimes will be closely associated with the application itself.

In today’s keynote at VMworld 2012, you will have seen firsthand the progress that we’ve made on our path to this multi-device workspace. Steve Herrod gave a technical preview of the industry’s first integrated platform for the mobile, multi-device workspace. We call this the Horizon Suite and we’ll be sharing many of its exciting capabilities that provide self-service access to all your apps and data across your devices including mobile-centric containers for management and security. You’ll also learn about the broadening of our Windows product portfolio, delivering both Windows as a Service for the multi-device world and Windows image management solutions for the Mobile workforce with View and Mirage technologies.

And In my role at VMware, I have the pleasure of helping to guide a lot of our EUC advanced development agenda and I am looking forward to representing these efforts by sharing and previewing some of the new technologies we’ve been working on that will drive us further towards these goals, some of which might become part of our future products. A few of the topics I’ll be discussing and previewing include:

Idealized Persistent Virtual Desktops with View, Wanova Mirage and Project Octopus (a.k.a. Horizon Data). These technologies are all powerful in and of themselves, however we see the combination of them as a game changer in the VDI market. When used together, it will be possible to deliver capex-efficient, persistent/stateful virtual desktops that enable efficient base image diversity and the ability to recompose and centrally manage the base Windows OS while preserving user customizations. This includes user installed applications and Octopus based “My Documents” in the cloud! In other words, persistent virtual desktops with all the capex and opex benefits previously achievable only with stateless/non-persistent VDI.

DaaS Technologies – The term Desktop as a Service (DaaS) is used to describe Windows desktops delivered as a service from public, private or hybrid clouds. VMware View is a robust, mature Enterprise VDI platform, but was not designed off the shelf for cloud platform multi-tenancy and resource elasticity. The new View Agent Connect component has been built specifically to enable our service provider partners looking to offer DaaS to realize the user experience benefits of VMware View and vSphere such as 3D graphics, PCoIP and broad peripheral support for VMware View Clients. With View Agent Connect, DaaS providers can offer a rich, PC-like experience to users anywhere. Taking this even further, Project Catalyst is a prototype of a VMware integrated end-to-end DaaS platform utilizing View and vCloud Director with full multi-tenancy, elastic resource provisioning, self-service portal operation, and more.

Managed Mobile Workspace (a.k.a. Project Fuji) – We are now securely delivering apps, files and data via Horzion Mobile which is extending its reach to iOS devices, providing policy-based management and security for collections of corporate resources on both iOS and Android devices. Horizon Mobile features are designed to enable policy specification, implementation, enforcement & remediation for the mobile workspace.

User Interface Virtualization (a.k.a. Project AppShift) – One of today’s main uses for VDI and application remoting is delivering Windows desktops and applications to non-Windows tablet devices in a multi-device workspace. However, no matter how good the remote graphics protocol is, trying to use a point and click interface on a gesture oriented tablet can be a frustrating user experience. VMware is advancing a unique technology that we refer to as “User Interface Virtualization.” With this technology approach, we’re able to transform numerous aspects of Windows desktop and application interfaces into a rich tablet friendly user experience. Like other virtualization technologies, our approach is completely transparent to existing Windows applications.

I can’t wait to talk to you about our vision, upcoming offerings, and share sneak peaks of these exciting new technologies with you in VMworld session EUC1351 “Directions in End User Computing; the Post PC Era,” offered on Tuesday Aug 28th at 10:30AM and repeated on Wednesday Aug 29th at 8:30AM.

When VMware talks to enterprises about mobility, three things typically come up in the conversations:

1. BYOD/Usability: The release of the first iPhone in 2007 ushered in two fundamental changes that will not likely change any time soon. The first is usability – clean beautiful small apps that do a few things really well. The second is BYOD. Employees were using better devices in their consumer life than at work so their preference was to use their personal devices for work as well. As a consequence of these two changes, the days of IT telling employees what device to use and what they can and cannot do on these devices are long gone.
2. Data Leakage: This is perhaps the biggest and primary concern for IT folks who are tasked with defining any company’s mobile strategy. Smartphones and tablets are truly mobile and go with us to places where laptops have not gone. Consequently, the idea that users could lose these devices along with the content on them is a major concern. Further, as employees download and use other mostly-personal apps on these devices, there is a chance data could be leaked accidentally by uploading content to a vendor’s cloud.
3. Apps: Most of the enterprises we talk to indicate that they will be developing more applications for mobile devices so users can be productive no matter what device being used. We know users will be running dozens of apps in addition to email/PIM, which means that there is potentially more content on the device that needs to be provisioned but also secured and managed. This combined with all the personal apps that also reside on the device means that every device is dual purpose – work and personal.

The first wave of vendors who offered solutions to address mobility concerns took a very RIM/Blackberry approach by securing the entire device and controlling what apps and services users can deploy on these devices. Given the dual-purpose nature of devices now, IT administrators are finding out that trying to lock down and control the entire device is not resonating with their users – one example being the device passcode. IT administrators want to enforce the passcode for all devices connecting to corporate assets. This requirement seems fair and painless, but at the same time forces users to enter the passcode even if they want to use the device for personal reasons – Facebook, Angry Birds or any other non-work app that resides on the device. In addition, IT also has the ability to wipe the entire device, so an employee’s personal pictures, videos, and other interesting content could be gone in an instant! As you can imagine, the employee response to all this is typically #UsabilityFailure!

IT administrators are now forced to reconsider how they think about security and management of mobile content. They need to enforce security without compromising usability. The challenge from an IT standpoint then is how to isolate personal apps and data from corporate apps and data, how to protect corporate apps and data and manage corporate apps and data on devices that have both personal and corporate content.

VMware’s Approach

VMware believes that providing dual persona (work and personal) functionality is the future of mobile computing, where users and IT both get what they need, and the solution is VMware Horizon Mobile.

Our initial foray was for Android devices and you can see the latest and greatest here.

However, we are now extending the solution to iOS devices. IT administrators will be able to create an iOS workspace, which is a collection of applications and data and services, set policies on that workspace and be able to secure and manage it.

You can watch a demo of this here. You will see how policies *only* apply when a user accesses corporate content. You will also see how we protect data. Data is protected at rest with encryption, data is protected in transit, and data access is controlled so only work apps can access enterprise data.

We’ve shown how you can deploy other critical applications into the workspace. Unmodified native iOS apps can be added to the workspace and they will automatically inherit the security policies defined by IT. All without compromising usability and maintaining native look and feel.

With VMware Horizon Mobile, security and usability go hand in hand – and users and IT will both win with this solution. If you’re at VMworld this week, stop by the VMware booth to see this technology in action.

My talk today on Day 1 focused on the software-defined datacenter (SDDC). A software-defined datacenter is where all infrastructure is virtualized and delivered as a service, and the control of this datacenter is entirely automated by software.

VMware is making the software-defined datacenter a reality with the vCloud Suite announced today. The vCloud Suite brings together what customers need to build, operate and manage cloud infrastructure – virtualization, software-defined datacenter services, policy-based provisioning, disaster recovery, and applications and operations management.

During my keynote, I spoke about the vCloud Suite technologies available today and in the future. Here are some of the highlights:

Software-Defined Compute

• Customers recognize that vSphere is the best place to run all applications. With the latest release of vSphere 5.1 there are numerous new features and technical enhancements that all applications can benefit from.
• Monster VM capabilities are back and better than ever! vSphere 5.1 now supports 64 VCPUS and it also is the first to hit record performance numbers – 1.05M IOPS from a single VM. It’s all about I/O, and you can check out our Performance team’s blog to learn more about this awesome benchmark.
• In addition to today’s applications we are also thinking about future applications. vSphere is an asset to both PaaS environments and Big Data, and is backbone of VMware’s Open PaaS offering, CloudFoundry. We’re also making Hadoop run well on vSphere, as you’ve heard from our recent announcement of Project Serengeti. For developers thinking about next-generation applications, think about vCloud Suite.

Software-Defined Storage & Availability

• We’re making storage a first-class citizen in the datacenter. I’ll show a preview of our Virtual SAN Technology, Virtual Volumes and Virtual Flash. At a high-level these efforts let network attached storage better leverage the server resources – and both VMware and our partners are making this possible.
• On the Availability front, I encourage all SMB attendees to check out Russ Stockdale’s VMworld Spotlight Session where you’ll have a chance to see a tech preview of DR to the Cloud for SMBs.

Software-Defined Networking & Security

• We’re enabling logical networks to work best in virtual environments, and with the ecosystem. vCloud Security and Networking is paving the way for L2 and L3, and we see real value in how you plug in L4-L7 services into logical networks, automate all this and then pull these services together. This is available now and I’ll demo this capability on stage. For a deeper dive on this area, check out the blog published today by my CTO colleague Allwyn Sequeira.

Management

• Management wraps all of these software-defined services together! We give our customers the ability to create clouds, deploy applications and pay attention to ongoing operations.
• vCloud Director 5.1 and vCenter 5.1 feature new enhancements and together make your infrastructure available as Virtual Datacenters. Customers will be pleased to know that the vCloud API is more extensible and can now be used with other clouds.
• vFabric Application Director can be used to deploy applications via an easy-to-use blueprint and is a key part of the vCloud Suite. vCenter Operations continues to handle the ongoing operations of a dynamic environment within a SDDC.
• To hear more about managing a SDDC, please read another great blog published today by Kit Colbert, a long-time engineer at VMware.

Multi-cloud

• We are also making vCloud Connector part of the vCloud Suite in order to address the need for management of hybrid cloud environments. This is particularly timely as our vCloud Datacenter Service Partner list grew today. We welcome T-Systems as the latest to share our hybrid cloud story.
• We believe standardization across the datacenter is critical if you’re a customer looking for efficiency and reduced complexity. However, we recognize the world has become more heterogeneous and we see a need for open platforms.
• We are extending automation and orchestration (via DynamicOps) as well as software-defined networking (via Nicira] to OpenStack, other clouds and even physical infrastructure.

Innovation

What’s the secret sauce behind a lot of these technologies? VMware Innovation. Innovation is a big focus for the company, especially VMware’s R&D and Office of the CTO. A lot of the cool innovative projects happening at VMware are featured in the Innovation Lounge and in Julia Austin’s Spotlight Session. I encourage attendees to check it out.

A lot going on this week at VMworld 2012! If you aren’t in San Francisco, you can tune into watch my live keynote at VMware NOW or catch the replay afterwards. I’m looking forward to some great conversations with our customers and partners this week.

In previous posts, I wrote about the need for a new networking and security architecture, and its role in the SDDC. As we spent time working through the product release, and integrating with several partners, we began to see certain patterns emerge in realizing the software-defined networking and security architecture. The following picture is a fairly succinct view of the stack – it should come as no surprise that some of these patterns are similar to the corresponding compute and storage stacks.

The Software-Defined Networking & Security Stack

The SDN and SDSec stack is built around policy-based automation, leveraging centralized management and control with distributed network and security functions, while abstracting and untethering the stack from the underlying physical hardware. Key attributes of software-defined networking and security include:

• ABSTRACTION. The network is abstracted as a set of network ports (and virtual NICs on the VM side). Security is abstracted as a set of port firewalls and end-point introspection. These abstractions are instantiated via virtual switches (vSwitch) and virtual firewalls (vShield) respectively and deployed in a scale out fashion on each host hypervisor. The network and security virtualization layer effectively untethers the VDC from the underlying physical network and firewall architecture, and provides a logical foundation to build the stack.
• POOLING. vSwitches are pooled into virtual distributed switches (VDS). Ports are pooled into port groups. Logical networks leverage these port groups and can be instantiated across the data center (VXLAN). Port firewalls are realized in vShield App or Edge. VM-based security is available via vShield Endpoint and endpoint security partner offerings. Because of the scale out nature of deployment, these pools are elastic and data center wide, and available on demand to be allocated to tenants or app owners.
• SERVICE INSERTION. The platform must be extensible to enable instant insertion of additional abstractions into the virtual plane e.g. encryption, intrusion detection & prevention, anti-virus, application delivery controllers, data leakage prevention, wan optimization control, monitoring tools, and other L4-7 services. The network and security virtualization layer provides a logical context for the instant insertion of such services.
• AUTOMATION. Just as VMs were the container for server virtualization, the Virtual Data Center (VDC) is the container for the SDDC. VDC deployments are completely automated via vCloud Director, and handle policy-based deployment of compute and storage, delegating networking and security deployments to the vCloud Networking & Security sub-system. A centralized command and control mechanism (vShield Manager) takes inventory of all the abstractions & pools, and is responsible for managing and mapping these pools into the needs of higher level entities like tenants or apps, and aligning with higher level virtual containers. Notions of multi-tenancy, isolation, elasticity, and programmability via RESTful interfaces are also handled here.

Software-Defined Networking & Security Lineup at VMworld

There are several noteworthy things happening at VMworld this week coming from VMware on the networking and security front:

• Today we launched the VMware vCloud Networking and Security Suite 5.1, which brings together new and updated versions of all the key components into a unified SDN & SDSec suite. It is being bundled into the vCloud Suite, making it easier for customers to seamlessly automate their networking & security needs as they build, operate and manage cloud infrastructure.
• VMware also made several key advances in various areas of the networking & security stack – including the VDS, VXLAN, vShield App, Edge and Endpoint, and vCloud Networking. All of these technologies are featured in sessions this week and demoed at the VMware booth.
• During today’s VMworld keynote, Steve Herrod demoed how the vCloud Suite paves the way for new datacenter SDN and SDSec architecture. Pre-eminent “Virtual Network Engineer” Serge Maskalik helped out on stage – effectively compressing weeks of intricate network and firewall engineering into a matter of minutes leveraging the power of the new architecture. This is the wave of the future!
• The VMware Ready^TM for Networking and Security Program also launched today. More than forty of our networking and security ecosystem partners will now be able to integrate their offering into the new architecture. This program and the vCloud Ecosystem Framework, allow our partners to further extend the SDDC – a big win for our partners and customers.
• If you’re in San Francisco this week, there are several deep dive sessions in the networking & security tracks you can attend to learn more about this topic. I plan to provide even more color in my SDN breakout session on Wednesday (INF-NET2313).
• Finally, it’s great to have Nicira join the VMware team at VMworld 2012! Nicira’s leadership in software-defined networking, adding network virtualization to other hypervisors and cloud management systems like OpenStack, and early wins in cloud scale datacenters makes for an exciting coming together of two awesome teams.

Network and Security Ecosystem

I spoke about abstraction, pooling, service insertion, and automation as the key building blocks for the SDN and SDSec stack. These will continue to be especially important as VMware continues to integrate additional partner offerings. Ideally these specialized functions are virtualized, distributed, scale out, multi-tenant and manageable by the respective element manager, for the most seamless integration into the stack.

SDNs (and SDDC) enable a new opportunity for the networking ecosystem – there is the need for a whole new class of networking hardware to realize fast, fat and flat, programmable, converged fabrics, where massive scale requirements will call for a new class of hardware and ASICs, especially at the spine/core. Also, programmability of such fabrics will unleash a whole new wave of network-aware applications and innovative deployments, and increase the opportunity rather than curtail it, much like we’ve seen with compute and storage.

VMware definitely couldn’t do all this alone! We value our partners and the joint work we’re doing together to address customer pain points. Many of our security and networking partners are at VMworld this week, showcasing their compelling solutions for the SDDC in breakout sessions, demos and more. On Tuesday I am hosting a SDN & SDSec partner breakout [INF-SEC3460]. I encourage you to attend to learn more about the tremendous work happening in the ecosystem.

As you can see, there is a lot going on in the space! VMware is serious about software-defined networking and security, and helping our customers enable automation of the data center, including interconnecting heterogeneous pools on compute and storage. I’d like to thank the VMware networking and security teams who have worked hard to showcase all of this at VMworld in the keynotes, breakout sessions and deep dives in the network and security track and the hands-on labs.

I look forward to meeting some of you at VMworld, and hearing your thoughts and ideas about networking and security!

As we’ve partnered with our customers to transform their IT, we’ve tailored our strategy to address what we’ve learned. First of all, VMware has moved beyond managing server virtualization to managing the datacenter as a virtual entity – that includes networks, security, storage and pools of VMs.  This virtual entity is a software-defined datacenter, which you’ve heard me talk about.  Secondly, as we get deeper and deeper into this transformation, we hear very clearly from our customers that they will continue to have heterogeneous infrastructure in their purview and need help dealing with the associated challenges. Several recent VMware investments reflect our recognition of this reality and our strategy to address it. These cut across application development, cloud management, and cloud networking, which are arguably the three most critical areas for interoperability. Here is a simple visualization of our current strategy:

We started down this path with our substantial investment in the popular Cloud Foundry open source project, an open approach to Platform-as-a-Service (PaaS). Our Open PaaS strategy is a key enabler for our customers, allowing them to easily create, deploy, and manage modern applications across almost any cloud environment. Today Cloud Foundry is easily deployed on Amazon Web Services, OpenStack-based clouds, laptops, as well as on VMware-based infrastructure.

You’ve recently seen VMware announce two acquisitions that continue this trend, addressing customer challenges around cloud management and cloud networking respectively. DynamicOps focuses on self-service and automated provisioning and management of applications across VMware- and non-VMware-based infrastructure. Nicira delivers software-defined networking capabilities to VMware- and non-VMware-based infrastructure as well.

As with Cloud Foundry, these new VMware solutions address more than just the VMware portion of our customers’ datacenters. Both of these solutions address, and in fact enhance, the use of heterogeneous “pools of infrastructure”. These “pools” include VMware-based private and public clouds, but also portions of IT’s responsibility that are powered by Open Stack, CloudStack, Amazon Web Services, or even (gasp!) physical infrastructure.

As VMware executes on our commitment to open platforms, you’ll see us continue to integrate infrastructure components and management through an API-centric strategy. Nicira’s integration with OpenStack and CloudStack are examples. The foundations and APIs we have built there will allow us to bring together the existing VMware networking roadmap with Nicira’s technology and deliver the benefits to more clouds. Another example is VMware vCloud Director with the vCloud API (released under the MIT license), and we are proud to be the first cloud platform to deliver software-defined networking and virtual datacenters with version 1.0 of vCloud Director two years ago.

And we also recognize that these communities are do-ocracies. Code speaks far more than words, and you’ll see our commitment to this strategy in the contributions you’ll continue to see from VMware. These contributions will be in the abovementioned technologies, but you’ll see investments in other areas as well. We recognize that customers are asking for better integration, between the new cloud distributions and VMware’s enterprise products.  Developers who want to work with us should get in touch to get more closely involved.

The software-defined datacenter delivers cost and agility benefits to the whole datacenter in the same way that server virtualization does for compute. We drove the benefits from that first foray into virtualization 15 years ago and we are dedicated to extending those benefits now across the entire datacenter, in a way that supports our customers’ need to manage multiple clouds. It’s efficient, it’s flexible, and we are committed to keeping it open.

There’s a lot more to tell about our strategy and how we’ll continue this journey. Join me at VMworld or online at VMware Now to hear even more!

Let’s start by discussing a few scenarios, beginning with so-called “Data at Rest.” Keeping data behind the firewall in the data center rather than on an endpoint is a key security benefit of VDI. In a recent blog post, Shawn Bass makes a case that the availability of disk encryption mitigates this problem and use case and also points out that live data compromises triggered from a web browser can occur with both models. On the latter point, I agree. Neither VDI or data encryption prevents live data exploits per se. And VDI is not a cure all for every security threat. However, on the data at rest aspect, physical theft is not the only threat and any historian of VDI will tell you that developer outsourcing was perhaps the key driver that launched VDI commercially in the 2005-2008 period. The main reason was the ability to eliminate access to source code at rest on the endpoint out of the country. This in fact can be a very major benefit where geo-political and outsourcing arrangements impact the use-case. Can that data still be stolen even with VDI? Yes, but you don’t leave your house unlocked because certain burglary tools can compromise it. Also, disk encryption is not used in many instances for a number of valid reasons including performance overhead, complexity, and incremental costs. Also, unless the encryption is deployed with centralized access controls, terminating access to the data can be a challenge when compared to VDI. As far as copies stored on Cloud data services, these can be blocked and have some security controls in place, while laptop and PC loss or theft yields immediate access to the disk. “Data in Transit” shouldn’t be ignored either. It can be spoofed and sniffed and encryption technologies are a must when going on the public Internet. This is another area where the attack surface is much larger than when contained in the datacenter. As a side note, even with a well secured VDI environment today, high fidelity smartphone cameras and apps that include OCR of images, it is becoming much easier than ever to steal source. So while nothing is foolproof, VDI can be useful in reducing the chance of data loss and greatly increasing the effort involved in certain kinds of data theft. And with VDI, datacenter targeted solutions such as RSA Archer created to manage risks, demonstrate compliance, secure and alert can also be applied.

Another prevalent use case for VDI is Mobile Secure Desktops, the ability to safely and securely access your corporate desktop environment from outside the enterprise, in particular from home or from WiFi access points such as internet cafes. Accessing your corporate desktop over a secure dedicated channel via a remoting protocol from a desktop or tablet is far more secure than putting that same device on the corporate LAN via VPN technology. With VPN technology, if the end point becomes infected, such as via home web surfing on a shared computer or over a public WiFi, you are exposing the corporate LAN to whatever is executing on the client device. Some newer VPN deployments provide port-level restrictions, limiting access to specific individual resources, but customer feedback has shown this to be complex to manage in practice. With VDI and View in particular connected with PCoIP over the View Security Server, the risk and attack surface is far less, since access is limited to a remote graphics protocol over a dedicated, signed and secured channel. Careful controls are automatically applied to ensure that users can only access central resources if they are strongly authenticated and can only access the specific desktop(s) they are authorized to access based on their identity and entitlements.

Another aspect of a View environment with demonstrable security benefits is its centralized system image administration and network controls coupled with rapid remediation techniques. A well-managed View environment reduces the number of threats in these domains. Some specific examples:

• Windows OS is only exposed to a managed vs. unmanaged network with central auditing.
• Less opportunity for “Man In the Middle” attacks, Port Scan/Exploits, Worms, DNS hijacking
• Controlled network access via vShield Edge/App to and between Windows environments. One key thing that cannot be effectively done in a scalable manner with physical desktops is the prevention of pivot attacks. Note that a particular physical desktop may not be the initial target. It may be that the attacker just wants a way into the network and they will take whatever method they can. If that initial target is compromised and you use vShield App to prevent the virtual desktops from “talking” to each other, the attacker is going to be limited where they can pivot to get to the next target. If added controls are put in place to limit access only to needed resources, then they are even that much more secure. So by using vShield App along with View, you can greatly increase your security over physical desktops and personal firewalls.
• Centralized, agentless AntiVirus to deter in-guest tampering offers additional benefits in a VDI environment. This includes zero footprint in the VM, management (including updates) outside of the guest OS, and the ability to scan even for desktops that are powered off. An example is Trend Deep Security.
• VDI is a unique enabler for bringing Datacenter caliber governance, risk and compliance solutions to desktop workloads. The ability to deploy a fresh security scanned desktop each time the user logs in and to perform forensic analysis are key. Mike Foley from RSA writes about these capabilities here.
• Patch management via linked-clones/templates and enable faster response in the event of compromise. As an added precaution, VDI enables refresh on logoff, ensuring users always connect to a brand-new and clean desktop, however that (currently) comes with a trade-off, such as inability to persist user-installed applications.
• Central AD policy configuration & compliance enforcement
• By reducing the attack surface and client access methods to Windows
  • Whereby which physical access to the Windows device is never granted.
  • Whereby a secure display protocol method is the sole method of console access
• Also, by only executing the image within the data center, VDI can dramatically reduce threats stemming from physical access/possession of an endpoint. Some specific examples:
  • Theft or compromise from the client device of unstructured user data at rest (My Documents, Offline Address Book).
  • Compromising locally stored encryption keys, caches, host name tables, etc.
• And last, but not least, by combining security best practices in a centralized environment, we reduce the complexity of distributed endpoint management.

Hopefully I’ve now shown you a myriad of ways that a well-managed View VDI environment in the data center is a more secure place to run Windows than a physical client environment. Even so, I do want to reiterate that View and VDI is not a comprehensive security solution. VDI can and should be one of the tools for securing the corporate environment, but it is certainly not the only one. And I want to reiterate that VDI does nothing to directly improve the security of the Windows Operating System, nor a web browser running on it. Also, we recognize that some security benefits of VDI are shared with other forms of Desktop Virtualization, such as RDSH (Terminal Server).

In conclusion, while View and VDI is not a security panacea, it can significantly improve protection against some common exploits and can provide a more secure environment for your Windows desktops.

But I’m getting a bit ahead of myself. Let’s step back and fit this acquisition into context of VMware’s broader vision and product offerings.

Delivering the Software-Defined Datacenter

In a recent blog, I discussed the notion of a software-defined datacenter, the foundation of cloud computing. Cloud computing is about agile, elastic, efficient, and reliable services, and it can only be achieved through intelligent software that abstracts out hardware resources, pools it into aggregate capacity, enabling automation to safely and efficiently dole it out as needed by applications.

Tenants or customers utilizing the software-defined datacenter can have their own virtual datacenter with an isolated collection of all the compute, storage, networking, and security resources that they are used to. Furthermore, this virtual datacenter can grow and shrink to efficiently utilize physical resources. But most importantly, the time to deploy these resources can be orders of magnitude faster than in most of today’s IT environments. This is what the software-defined datacenter is all about, and it is the architecture for the cloud. This acquisition advances our software-defined datacenter strategy.

For the past 14 years, VMware has led the way towards the software-defined datacenter with our battle-tested server virtualization capabilities, as well as through vSphere’s software-based storage and availability offerings. Nicira will help us advance our software-defined networking activities substantially. Today we offer VMware vSphere’s virtual switching, vCloud Director Networking, vShield Network and Security services, and collaborate with the industry on the VXLAN protocol. This acquisition expands VMware’s networking portfolio to provide a full suite of SDN capabilities and a comprehensive solution lineup for virtualizing the network – from virtual switching to virtualized layer 3-7 services.

Some Challenges with Networking in the Cloud

Cloud computing has put a real strain on traditional approaches to networking. Managing networks and networking services in a cloud environment is complex and time-consuming: to provision and configure the networking services for workloads in the cloud, customers often have to deal with creating and managing thousands of VLANs and VLAN rules. As a result, while provisioning a VM may take only 2 min, provisioning the associated network and networking services can add days or even weeks to the process. Furthermore, cloud computing benefits from applications’ ability to move all around a datacenter (or even across datacenters). However, physical network topology limits workload mobility within the scope of a top-of-the-rack switch and a handful of servers. The industry clearly recognizes the need and opportunity to transform networking for the cloud, and thus to enable even more agility and efficiency in its operation.

Enter Software-Defined Networking

To address these challenges, Nicira’s software-defined networking starts by virtualizing the network, decoupling the logical view of a network from its physical implementation. It does so by creating an abstraction layer between server hosts and existing networking gear which decouples and isolates virtual networks for specific networking hardware, turning it into a pool of network capacity. This enables the on-demand, programmatic creation of tens of thousands of isolated virtual networks with the simplicity and operational ease of creating and managing virtual machines. The resulting business value comes from more agile, efficient, flexible, and robust networking configurations. You can more about network virtualization here, and the included picture should look familiar:

Embracing Multiple Hypervisors and Multiple Clouds

Customers want networking solutions that work across their entire datacenter, managing communication of both VMware- and non-VMware installations as well as (gasp!) physical hardware. Nicira is the pioneer in software-defined networking, but it is important to note that they are also the leader in network virtualization for heterogeneous hypervisor and cloud environments. They are major contributors to the networking capabilities of other hypervisors (via the Open vSwitch community) as well as to the “Quantum Project”, one of the key subsystems of OpenStack.

I can imagine skepticism as to whether we will continue this substantial embrace of non-VMware hypervisors and clouds. Let me be clear in this blog… we are absolutely committed to maintaining Nicira’s openness and bringing additional value and choices to the OpenStack, CloudStack, and other cloud-related communities. It’s worth noting that this builds upon the openness delivered by our other recent acquisition of DynamicOps, a leader in cloud automation solutions for heterogeneous environments. It also builds upon our experience with the SpringSource community as well as our stewardship of the CloudFoundry Open Platform as a Service Project.

Similarly, Nicira’s network virtualization supports a broad variety of NICs, switches, appliances, networking APIs, and fabric types. A customer’s ability to gain new networking virtualization capabilities without changing out their existing infrastructure is paramount. This is an area VMware is very comfortable with today through our broad partnerships and hardware compatibility programs around servers and storage. We will continue to maintain our open approach and provide access to our networking technology and APIs to our partners to allow them to add new value through both hardware and software advancements, as well as by providing compatibility with legacy systems. In this context we look forward to continuing to work closely with current and future ecosystem partners such as Cisco.

We’ll be sharing even more exciting news about how we plan to engage and support these different communities in the near future. In the meantime, you can visit the blog of Nicira co-founder and CTO Martin Casado here.

Looking Forward

In closing, this is an incredibly exciting time at VMware, but also for our collective industry. Customers clearly demand cloud computing, and we can deliver it via the software-defined datacenter. With this acquisition, we bring together two pioneering teams in network virtualization to accelerate the realization of the software-defined datacenter and the benefits it will deliver to businesses and service providers alike.

I’ll close now with a shameless plug to join us at VMworld where we’ll speak even more about VMware, Nicira, and the exciting times ahead!

I recently sat down with Steve Muir, VMware’s Director of VMAP, to hear more about the program and would like to share our conversation with the community.

Julia: Now that you’ve been at VMware for just about a year, can you tell me how you would describe the VMware Academic Program to the community?

Steve: VMAP is an initiative of the Innovation team within the Office of the CTO.  We foster VMware’s relationships with the academic community.  While the goals and programs of VMAP are primarily research oriented, we also have various efforts to support teaching and education.

Julia: Are there specific focus areas of VMAP?

Steve: There are three primary elements in the program: research collaboration, community engagement and technology initiatives.  In research collaboration we support a number of universities’ research activities, either joint research projects or broad sponsorship of a particular department or research lab.  Our community engagement efforts provide sponsorship for a wide range of academic conferences every year, and we work closely with our colleagues in University Relations to reach out to student groups at various universities.  Technology initiatives are focused on making VMware technology and related educational content available to academic users.

Julia: A lot of tech companies simply write checks to sponsor research. How is VMAP going beyond that when it comes to supporting research activities?

Steve: Our mission is to establish strong, working relationships with academic researchers – well beyond just writing checks.  We have a number of universities engaged in collaborative research projects with our core engineering team.  The close proximity of several of our R&D labs to some of the top research universities in the world has allowed for a very fluid interaction with these teams.

We target specific areas of interest (such as security, networking, etc.), and provide the opportunity for researchers to partner with teams who are actually shipping code.  Our projects are solicited through an annual Request for Proposals or initiated by our R&D colleagues based upon their existing relationships. Ultimately, our collaborative work results in joint publications and quite often influences VMware product and feature offerings, e.g., this paper on Hierarchical QoS just published at USENIX.

Julia: I’m sure you get regular feedback from our academic partners about what they get out of the program.  What types of benefits do they cite from their collaboration with VMware?

Steve: Our academic partners truly appreciate the insight they get into real customer challenges, and at scale far beyond the systems that researchers typically have access to. We help guide research towards the most relevant challenges, and consequently maximise the impact of that research.  We obviously also have an exceptionally talented and experienced workforce that can provide knowledge and perspective on a wide range of systems research areas, but perhaps most important are the relationships built between researchers, students and VMware engineers, often leading to joint publications or a starting point for a PhD thesis.

Julia: Who are some of the collaborators you’re working with these days?

Steve: We have ongoing relationships with numerous universities, not just in the US but internationally.  Through our 2011 RFP, on ‘Performance in Cloud and Virtualized Environments’ we funded projects at the University of Boston, University of Tennesee and University of Toronto, and have already seen those projects bear fruit.  We will shortly announce the recipients of our 2012 RFP, for cloud and virtualized system security, for which we had a great level of interest.  Some of our other research partnerships, driven by existing relationships, are with MIT, CMU, Berkeley, Ohio State University, Georgia Tech, Karlsruhe Institute of Technology, Technion Israel Institute of Technology and Imperial College, London.

Julia: Part of being in the academic community is attending and participating in conferences.  Are there specific conferences VMware sponsors, and what role do we play at those events?

Steve: Just as VMware’s corporate interests are broader than just virtualization and our global presence has expanded to various countries, VMAP’s conference sponsorship and participation evolves and looks to new areas.  For example, with our current focus on Big Data and other database topics we attended the recent ACM SIGMOD conference for the first time and made some great connections with researchers in this area.

Internationally we supported the ACM PLDI conference in Beijing, where VMware Fellow Ole Agesen was the keynote speaker, and Eurosys in Switzerland, as part of our global outreach. We also co-sponsored conferences outside the academic sphere to engage with different communities – for example, this year we partnered with our Ecosystem Engineering organization to co-sponsor EclipseCon.  Our conference enables more students to attend by providing travel grants, and many researchers enjoy talking to VMware engineers about how their research relates to our business and technical challenges.

One great thing that we hear about our participation at conferences is that we aren’t there just to recruit.  We often have our own papers to present, lead Birds of a Feather (BoF) sessions, and our engineers really enjoy the chance to engage with the research community.   This level of participation has been a great way to demonstrate our support for the academic community, and highlight our work in solving hard problems through innovation.  It also provides great professional and personal development opportunities for our engineers, and helps us develop relationships with professors and students.

Julia: I always get excited when I hear one of our engineers just had a paper accepted to a conference.  What have been the most successful conferences for VMware this year?

Steve: We’re extremely proud of the folks that work so hard to reach these achievements.    In March we sponsored VEE and ASPLOS, held at the Royal Society in London, and had several papers presented in the conferences and associated workshop.  Just last month we sponsored the USENIX Annual Technical Conference, in Boston, where we presented two papers on storage QoS, based on joint work with faculty and students at Rice University, and one on hypervisor performance optimisation.

Looking ahead a little, one of the most interesting conferences for us every year is the USENIX Large Installation System Administration conference (LISA).  It is unique in bringing together researchers and practicing system administrators, which are both communities that are very interested in what VMware is doing.  LISA has a different feel than a traditional academic conference, and it’s always a great way to wrap up the conference season; we had a very well attended BoF last year, and hopefully will have a paper or two this year.

Julia: Who at VMware can participate in VMAP?

Steve: The VMAP team runs the program, but our success is heavily dependent on the participation of our VMware colleagues in R&D.  Publishing papers, representing VMware at conferences and engaging with students are all things that any VMware engineer can do, and they do it very well.  We encourage everyone at VMware to take advantage of the opportunities we can provide them to get out and interact with the academic community.

Julia: It’s great to see the progress being made with VMAP. How can the community get more information about the program?

Steve: There has been a lot of progress made, but there’s also much more we plan to do in the future.  The VMware Academic Program hosts a lot of information about different programs here. Our software licensing program for academic users is described in detail here. You can also follow us on Twitter at  @vmwacademic.

The 2nd lecture was an introduction to system-level virtualization. I covered the evolution of x86 virtualization starting with VMware’s binary translation (BT) mechanism which later morphed into a more conventional trap and emulate approach for handling CPU virtualization once the processor vendors added support to correctly handling deprivileged OS execution including x86 ISA oddities. We then briefly discussed several memory-related technologies, including shadow page tables, extended/nested page table support, transparent page sharing, ballooning, swapping, and compression.

Between questions and the content I presented, I wasn’t able to cover the other major virtualization components I had intended to present in the 2nd lecture — live migration, networking, HA, DRS, FT, etc. Happily, I have two more lectures and should be able to finish the virtualization overview tomorrow before moving on to discuss the real topic at hand — use of virtualization for HPC.

It’s been a rewarding experience so far — lots of good questions from the students during class and lots of good conversations outside of class, including at this afternoon’s poster session which included more than 50 submissions.

Before I presented our performance results, I spent the first part of the talk explaining what I see as an important convergence between HPC and Enterprise IT requirements, a convergence that can be very beneficial to both constituencies if we pay appropriate attention to the trends. The convergence offers the HPC community a way back into the mainstream of IT where vendor investments, which are generally proportional to market opportunity, are being applied to address an increasing number of problems of concern to HPC users and administrators.

insideHPC has posted a video of the talk and a link to the slides here.

HPC in the Cloud has posted a good summary of the talk here.

• First off, it’s clear that innovation is on every company’s strategic agenda no matter what industry. The key is how each company develops a culture of innovation that includes both revolutionary and evolutionary generation and implementation of ideas.
• Full solutions, including services, are fundamental to product survival. It’s not the product itself, but the utility of that product that will differentiate (Drucker).
• There are lots of productivity opportunities when social media, crowd sourcing, and the wider adoption of mobile enable faster turnaround of idea generation, implementation and/or the “killing” of innovation projects.  Innovative organizations must plan for and encourage experimentation and be willing to not only allow room for failure, but celebrate and reward those who learn from failure!
• Open Innovation and engaging customers and partners in your innovation processes is critical. With open innovation comes co-creation, which takes place across the entire life cycle; building things with your ecosystem of customers, partners and possibly the academic research community from concept to implementation and support.
• When engaging customers in the innovation process, it’s important to not only create a platform for engagement but also to choose the right customers and incentives. Ensure you actually use your customers’ ideas and communicate that back to them – because “people [customers] are more likely to embrace what they are a part of creating.” (Mohan)
• Social media and Gamification are changing the way we work, play, and bring products to market. The landscape is changing and the next generation of innovators will not work in the traditional way we have been used to working. Mobility, immediate time to market, crowd sourcing and working smarter while having fun is a critical aspect of how companies will be able to attract and hire innovative talent.
• Social media has increased the level of engagement of our customers  within the ecosystem – we see noticeably more discussions happening over Twitter, Facebook and the blogosphere. So, it’s essential to have community sites and public forums where your customers/partners can collaborate and support each other and also be recognized and rewarded for being helpful and responsive via badging systems, awards, etc.  VMware Community is a great example of this, btw, and our vExperts not only have the satisfaction of helping their peers, but earn a number of exclusive benefits.
• There were several good reminders by speakers that “Social” means more about a conversation and not just information exchange.  We need to resist the urge to productize conversations and keep in mind both content and context.  We shouldn’t just listen, but engage in conversation and consider how that problem could lead to a solution that’s the next big thing.  We also need to ask what the risk is of not engaging.  Customers know what they want and now more than ever have the medium to say what they want through digital media, at a grand scale.  Will your company be able to do this?
• The adoption of digital media to engage customers and foster the innovation process internally by all companies is no longer optional.  More and more, customers and consumers are demanding open and transparent communication. “Friending” and “tweeting” and sharing your company products and culture via Instagram are still very new but is happening in pockets around us. This new medium of communication with the customer base can be a daunting experience for companies that are not prepared for the explosion of digital media – and this bottom-up influence is changing the game when it comes to how product innovation roadmaps are developed.  This is a huge opportunity for companies!
• Finally, social media is now being seen as a means towards ideation and social responsibility.  At WIFNY, there were lots of mentions on how social media is not just a means to innovate, but also a means to change the world.  There are numerous great examples of companies developing ideas that are not only increasing revenue or reducing costs, but also good for the environment and communities in need. – one being the inflatable turbine project that was a result of a social media contest run by GE.  I especially liked this example because it not only demonstrated GE’s open innovation model, but the goal of this particular contest was to modernize the grid and transform global energy.  The Challenge generated nearly 4,000 ideas and the five companies that were funded each received $100K!

If you’re interested in innovation I encourage you to attend next year’s conference. WIFNY not only allows the attendees to hear terrific and often very entertaining (thank you Sir Ken Robinson!) speakers, but it also provides ample opportunity to network with others who are tasked with driving innovation across their organizations or departments – from every possible industry sector you can imagine and from all sizes of organizations.  Many of the people I met were dubbed Innovation leaders of some sort for their companies/orgs only in the past 6-12 months.  Another interesting observation was that, from my eyeball survey, this is one of the first conferences I’ve been to in ages where there was a fairly even attendee gender split – either I spend too much time at nerd conferences, or the wold of innovation is doing a nice job of balancing things out :)

Did you attend this year’s WIFNY as well? If so, I’d be very interested in hearing about your observations and takeaways. Feel free to comment below.

Thanks to WOBI for putting on such a great event!  I’m looking forward to WIFNY’13!

Ole’s talk focused on why virtualization should not be ignored by today’s software developers, language designers and implementers – a topic that is important to us at VMware! He discussed how virtualization affects guest software and got into more detail about the performance of virtualization software.

You can download Ole’s presentation here. And if you’re a developer and want to chat more about this topic with Ole, feel free to leave a comment below.

VMware is seeing cloud change how businesses are run, how IT runs data centers, how developers write applications, and how we think about data. Our industry is in the midst of a data renaissance where a rapid proliferation of new uses of data drives new technologies to manage and get more value from data. The traditional relational database is becoming a thing of the past, which once was the main vehicle for serving the needs of online applications with optimized engines to store longer-term data for business intelligence and analytics. But times are a changing! No longer is there a “one size fits all” when it come to data and analytics technologies.

The four primary data requirements we see stretching the limits of traditional approaches to data and analytics include:

• Big Data: the need to store and get analytics from gigabytes, terabytes or petabytes of unstructured or semi-structured data
• Fast Data: the increasing need for low latency interactions with large sets of data, often driven by today’s mobile and social apps
• Flexible Data: the need to adapt data access to the most appropriate model for each application
• Cloud Delivery: the growing demand to access data as a service, provisioned on the cloud of your choice.

VMware is working on several key initiatives to deliver the best platform for Big, Fast and Flexible Data in the Cloud, and hopefully you’ve been tracking our news in this space. Last week we announced Serengeti, a new open source project designed to enable Big Data applications to run on public and private clouds. Serengeti enables the rapid provisioning of highly-available Hadoop clusters on a virtual platform. My CTO colleague, Richard McDougall, blogged about Serengeti as well as the most recent update to the Spring for Apache Hadoop project. These announcements closely follow VMware’s acquisition of Cetas, a cloud service provider for Big Data analytics and predictive intelligence systems that can operate at a scale of hundreds of terabytes and billions of events.

What are the big data challenges you’re facing? How is cloud affecting your data requirements? VMware has a lot happening on the Big Data front, and you’ll continue to hear from us in this area.

One of the other great aspects of these stories is the fact that each idea came from very different technology areas and from all levels of the company. At VMware, anyone can take their dream to reality.

I hope you enjoy this piece as much as we do!

While at the Forum, I was happy to announce that the remainders of the DMTF VMAN specifications have been adopted by ANSI. These specifications allow for a standard set of models and interfaces for managing virtual machines and their resources. I anticipate even greater adoption of these specifications in the years to come.

The big news at the Forum was that FedRAMP was launched on June 6th. The Federal Risk and Authorization Management Program (FedRAMP) is a government-wide program that provides a standardized approach to security assessment, authorization, and continuous monitoring for cloud products and services across government agencies. The goals of the program, from their website (www.FedRAMP.gov) are to:

• Accelerate the adoption of secure cloud solutions through reuse of assessments and authorizations
• Increase confidence in security of cloud solutions
• Achieve consistent security authorizations using a baseline set of agreed upon standards to be used for Cloud product approval inside or outside of FedRAMP
• Ensure consistent application of existing security practices
• Increase confidence in security assessments
• Increase automation and near real-time data for continuous monitoring

The FedRAMP program is just starting and may have a few kinks to be ironed out, but it’s a good first step in the right direction!

One of the two panels on which I participated was USG Cloud Computing Technology Roadmap Priority Action Plan (PAP) Progress and Examples. These are documented more fully in NIST Special Publication 500-293

Here is a list of the top ten requirements from that specification:

• Requirement 1: International voluntary consensus-based interoperability, portability, and security standards (interoperability, portability, and security standards)
• Requirement 2: Solutions for high-priority Security Requirements (security technology
• Requirement 3: Technical specifications to enable development of consistent, high-quality Service-Level Agreements (interoperability, portability, and security standards and guidance)
• Requirement 4: Clearly and consistently categorized cloud services (interoperability and portability guidance and technology)
• Requirement 5: Frameworks to support seamless implementation of federated community cloud environments (interoperability and portability guidance and technology)
• Requirement 6: Technical security solutions which are de-coupled from organizational policy decisions (security guidance, standards, and technology)
• Requirement 7: Defined unique government regulatory requirements, technology gaps, and solutions (interoperability, portability, and security technology)
• Requirement 8: Collaborative parallel strategic “future cloud” development initiatives (interoperability, portability, and security technology)
• Requirement 9: Defined and implemented reliability design goals (interoperability, portability, and security technology)
• Requirement 10: Defined and implemented cloud service metrics (interoperability and portability standards)

At the Forum, we also got a glimpse of some other NIST programs, including an update on their Cyber Security Center of Excellence, work on their health records in the cloud, new identity management standards activities and work that is being started on standards for Big Data.

It is clear that there’s a lot going on at the NIST, and that interoperability, portability and security standards are a key part of that and the governments top priority to achieve the vision of cloud computing.

Title:

Achieving ultra-low latency in the Cloud: How low can we go?

Abstract:

Are cloud computing and virtualization only useful for running throughput workloads or can latency-sensitive applications be run as well? This is the question we have been examining within the Office of the CTO at VMware. In this talk, I will present our early results using InfiniBand RDMA in a vSphere virtual environment and discuss the prospects for future performance improvements and functional enhancements.

I plan to present the results of InfiniBand experiments we conducted last year and am hoping also to have some preliminary numbers to share running a CFD workload on vSphere.

There’s no question that the amount of value being extracted from data is increasing – almost every customer I speak with is building new technology to gain new or competitive insights from tapping large volumes or rates of data. In the last few posts, I have introduced VMware technologies and products that provide data services to new applications.

We see four major axes along which data requirements are stretching the limits of traditional approaches to data analysis:

• Big Data – The need to store and compute against hundreds of gigabytes of unstructured or semi-structured data
• Fast Data – The increasing need for low latency interactions with large sets of data, often driven by today’s mobile and social apps.
• Flexible Data – The need to adapt data access to the most appropriate model for the application.
• Cloud Delivery -The demand to access data as a service, provisioned on your cloud of choice.

The amount of data being stored is growing at an unprecedented rate, and much of this data is either unstructured or semi-structured. Database systems often have rigid schemas making it hard to store today’s range of data types (logs, sensor data, binary blobs). The sheer volume of data typically outscales current structured technologies and the cost of traditional business intelligence systems is prohibitive at this scale

The Hadoop platform has rapidly evolved to meet the needs in these areas, and we are seeing a significant growth in adoption of Hadoop by VMware customers.

Hadoop gives the ability to store massive amounts of data in a reliable datastore, and Map-Reduce provides a data-parallel programming framework to compute against that data. We have observed that the majority of our customers are using many of the higher level ecosystem tools, which utilize the power of the underlying data-parallel Hadoop platform through familiar data access methods – such as Hive for query access, or Pig for script based data processing.

Today I’m pleased to announce that VMware is introducing several new initiatives in the open source Hadoop community to enable big data applications to be deployed more easily, rapidly and in a more agile manner.

Why Virtualize Hadoop?

I’m asked this question often, although recently the reasons to virtualize are already compelling and the questioning quickly moves to ‘how’ rather than ‘why.’ Here are a few of the use cases we’ve been working on with customers:

1. Consolidation/Sharing of a big-data platform- A full big-data platform typically consists of the Hadoop distributed file system and core Map-reduce, hBase, Pig, Hive, Sqoop and a big-SQL database using traditional SQL or distributed SQL (like Greenplum DB) for more regularly accessed semi-structured data. A good strategy is to architect a common shared platform, on which all of the big-data technologies can reside. By virtualizing, all hardware nodes can be common, eliminating the need for special hardware for master services (the NameNode) so that if multiple clusters are deployed, you no longer need to provision and special servers for each of the master services.
2. Rapid Provisioning – Yes, it’s possible to deploy a complete Hadoop cluster in under 10 minutes! Automated provisioning of Hadoop clusters is possible through the vSphere virtualization APIs, so that all nodes and layered Hadoop ecosystem stack components can be quickly and easily deployed.
3. Resource Sharing – Since other workloads can run on the platform, it’s possible to vary the amount of resources assigned to Hadoop at any point in time. Several customers have asked if they can use nodes during off-peak hours to run Hadoop jobs. By varing the number of virtual Hadoop nodes, it is possible to grow or shrink the Hadoop cluster size, allowing scenarios such as time-shifted host sharing.
4. High Availability – The entire Hadoop stack can be made highly available. We can leverage vSphere’s built-in HA to protect all of the locations vulnerable to single-point of failure problems in the stack, including the core master services, HDFS NameNode, Map-Reduce Jobtracker. Since the big data platform includes other services, it’s also important to protect all of the other components of the stack against failure – including the hBase SQL meta-data database, the Oozie or Spring Batch master batch scheduler daemon, the HCatalog database, and depending on the distro, each of the management services.
5. Security – As we discuss later, it’s possible to split HDFS from the Task tracker virtual machines, meaning that data is no-longer shared and vulnerable inside the compute node. By putting HDFS into separate nodes, you provide higher levels of security, since only authorized Hadoop users can access the HDFS service over the virtual network.
6. Versioned Hadoop Environments – Since each cluster is virtual, it is possible to run multiple clusters, each with different version levels of Linux, the Linux application environment or Hadoop on the same physical cluster. We hear this requirement today, and foresee that it will be even more important as migrations to Hadoop 2.0 occur, warranting the need for mixed 1.0 and 2.0 runtime environments against the same data and within a cluster.

Common Misconceptions

There are a few common misconceptions encountered by those who are considering running Hadoop in a virtual environment:

• Isn’t there a large performance overhead?- We’ve done several deep studies of Hadoop in a virtual environment, and this is an ongoing focus area for us with our Hadoop partners. Our results show that on average Hadoop performs within a few percent of that of a pure-physical environment. In some cases, we see slightly better performance due to better scaling because of the ability to partition the machine into several smaller nodes. For more information, see Jeff’s post on Hadoop performance and the resulting white paper. The following graph from Jeff’s study shows the runtime of jobs on virtual compared to physical – so 1.0 is the same, and lower than 1.0 is faster. It shows that most workloads run close to native.
• Doesn’t vSphere use shared SAN storage only?- vSphere provides support for local or shared storage. While it’s true that HAdoop is designed for local storage, it’s entirely possible to put hadoop onto a SAN. The SAN does however have different cost and performance metrics – for small clusters of tens of gigabytes and low tens of nodes, there is enough performance to run Hadoop satisfactorily.One way to think of the different types of storage is to compare the important characteristics related to big data – mostly the cost per gigabyte and the amount of available access bandwidth. In the example below, we can see the approximate return on $1M spend in the different types of storage.Larger clusters may warrant the use of local storage for more scalable bandwidth and a ten-times reduction on the cost per gigabyte of provisioned storage. For Serengeti, we recommend the use of a hybrid storage configuration with shared storage for the OS images and other virtual machines and local disks for HDFS datanodes.The following diagram shows how local disks can be mapped into a virtual machine. In this example, the OS resides on shared storage with the HDFS datanode storage on local disks.

  
  Shared Storage (San/NAS) – Local Disks

Our Hadoop announcements Today

Today we are introducing three new Hadoop initiatives:

• Project Serengeti – Rapid and Automated provisioning of Hadoop on vSphere
• Spring for Apache Hadoop
• Integrations with vSphere and Apache Hadoop to deploy a highly available Hadoop platform

Project Serengeti – 10 Minutes to Provision a Highly Available Hadoop cluster On-Premise

Serengeti

Project Serengeti is a highly available, managed Hadoop platform. Using the best of breed of open source Hadoop and Virtualization features, Serengeti can rapidly deploy a highly available Hadoop cluster. You can deploy one or many virtual Hadoop clusters onto a vSphere farm, using traditional shared storage or local-storage.

Serengeti can deploy multiple Hadoop distributions and common Hadoop components such as HDFS, MapReduce, Pig, and Hive on a single virtual platform.

Serengeti takes advantage of vSphere high availability to protect the Hadoop master nodes in a virtual machine. The master node virtual machine is monitored by vSphere and if there is a hardware or software failure, vSphere will automatically start another virtual machine to reduce the unplanned down time.

Serengeti consists of a provisioning engine, the Apache Hadoop distribution, a virtual machine template and is preconfigured to automate Hadoop cluster deployment and configuration. With Serengeti, you can save time in getting started with Hadoop because you don’t need to install and configure an operating system, or download, install and configure each software package on every node in the cluster.

Serengeti uses a template document to configure all the nodes – which means it’s as easy as editing a document to specify the configuration of the cluster. It then uses Chef and the underlying templates to deploy each of the nodes. Serengeti is a full life-cycle tool which can both provision and update the configuration of the cluster – so that, for example, additional nodes can be added to the virtual cluster after it’s created.

Hadoop Distributions

Our strategy is to provide the Apache Hadoop distribution by default, and allow upload and deployment of popular Hadoop distributions. At the time of writing, Serengeti 0.5 supports Apache 1.0, CDH3, Hortonworks 1.0 and Greenplum HD 1.0. It’s easy to update the distribution inside the appliance.

A Quick Tour Through Serengeti

Serengeti is installed as a single virtual appliance. Once installed, it provides a pre-configured environment with a familiar command line interface for the management and provisioning engine.

$ ssh serengeti@serengeti-vm

$ serengeti

serengeti>

Creating a cluster is as easy as specifying the cluster name. This will create a default cluster of one master node (name node and job tracker), three worker nodes and one client VM from which we can run the Hadoop client commands:

serengeti> cluster create --name myElephant

serengeti> cluster list –name myElephant

name: myElephant, distro: cdh, status:RUNNING
  NAME    ROLES                                 INSTANCE   CPU MEM(MB)  TYPE   SIZE(GB)
  -------------------------------------------------------------------------------------
  master  [hadoop_NameNode, hadoop_jobtracker]  1          2   7500     LOCAL  50

name: myElephant, distro: cdh, status:RUNNING
  NAME    ROLES                                 INSTANCE   CPU MEM(MB)  TYPE   SIZE(GB)
  -------------------------------------------------------------------------------------
  master  [hive, hadoop_client, pig]            1          1   3700     LOCAL  50

    NAME                HOST                              IP              STATUS
    ----------------------------------------------------------------------------
    myElephant-client0  rmc-elephant-009.eng.vmware.com   10.0.20.184     Ready

$ ssh rmc@rmc-elephant-009.eng.vmware.com 

$ hadoop jar hadoop-examples.jar teragen 1000000000 tera-data

By default, Serengeti creates virtual machines on the default resources and storage pools. In most circumstances we’ll take advantage of vSphere’s resource groups to control which resources, storage and networks are used for the cluster. We can do this by providing vSphere pools for storage, network and cpu/memory resources through the same CLI.

We anticipate that many will want to customize their Hadoop configurations – including the distro, node roles, software stack components installed, size of nodes etc. To facilitate this, Serengeti provides the notion of a cluster spec file. This file contains a customizable JSON representation of the cluster configuration. In the example below, you can see how can easily describe a virtual cluster topology with a combined NameNode/Jobtracker, five Worker nodes which contain HDFS datanode and Tasktracker, and a single Hadoop client VM.

[
  "distro":"apache",             Choice of Distro
    {
      "name": "master",
      "roles": [
        "hadoop_NameNode",
        "hadoop_jobtracker"
      ],
      "instanceNum": 1,
      "instanceType": "MEDIUM",
      “ha”:true,                 Choice of Shared Storage or Local Disk
    },
    {
      "name": "worker",
      "roles": [
        "hadoop_datanode", "hadoop_tasktracker"
      ],
      "instanceNum": 5,
      "instanceType": "SMALL",
      "storage": {               Choice of Shared Storage or Local Disk
        "type": "LOCAL",
        "sizeGB": 10
      }
    },
    {
      "name": "client",
      "roles": [
        "hadoop_client",
        "hive",
        "pig"
      ],
      "instanceNum": 1,
      "instanceType": "SMALL"
    }
]

I won’t go into all the detail here, but wanted to point out that the configuration is completely flexible. Through JSON and command line scripting, it’s easy to automate and tailor the deployment to your needs. The full specification of the cluster spec is here.

What about Data Locality?

It’s easy to create a virtual cluster, but you may want to ensure the cluster is deployed with local storage to obtain full data locality and the bandwidth of local disks. Serengeti supports both types of storage – shared via SAN/NAS, or local disks. When we use local disks for the cluster, the nodes are distributed evenly across the available data stores, so that the HDFS datanode has access to local disks just as it would in a physical environment. The only difference is that you can specify how much of the local disk to use – either part of the disk or the whole disk for each virtual node. This allows for full locality, with the option of putting multiple clusters across the same datanodes.

Using Different Distro’s

Serengeti has a simple mechanism for adding other distro’s: you can easily add the desired distro as a tarball into Serengeti’s distro directory, and then configure the distro manifest to point to it. Following is an example of the distro configurations.

{
   "name" : "cdh",
   "version" : "3u3",
   "packages" : [
    {
       "roles" : ["hadoop_NameNode", "hadoop_jobtracker",
                  "hadoop_tasktracker", "hadoop_datanode",
                  "hadoop_client"],
       "tarball" : "cdh/3u3/hadoop-0.20.2-cdh3u3.tar.gz"
    },
    {
       "roles" : ["hive"],
       "tarball" : "cdh/3u3/hive-0.7.1-cdh3u3.tar.gz"
    },
    {
       "roles" : ["pig"],
       "tarball" : "cdh/3u3/pig-0.8.1-cdh3u3.tar.gz"
    }
   ]
 },

Open Source Model

The Serengeti project is released under the Apache 2.0 license and build upon other open source technologies such as IronFan, Chef, Fog, RabbitMQ and Spring. The main project page for Serengeti is http://projectserengeti.org where you can find links to the source code as well and other useful resources. We greatly welcome contributions and will provide help getting started with Serengeti through the user mailing list.

Video Demonstration of Serengeti

See also our full user guide for Serengeti.

Hadoop Virtualization Extensions

In a virtual configuration, there can be multiple HDFS nodes or Task-tracker nodes per physical host. In this case, these nodes perform as if they were on the same node and share the same failure domain, however the current Hadoop Topology model doesn’t know that these VMs are related. The current Hadoop model describes topology as Datacenter, Rack and Host, so that it can place compute tasks close to their data, and to ensure that data is replicated in a specific way (by default, to another host, and another one in a different rack).

When virtualizing Hadoop, there are several potential topologies, all of which have different virtual hosts with their own IP address that reside on the same physical host. Option 1 has a single combined compute (task-tracker) and data-node in the same VM, which works well with the existing topology code. Option 2 introduces multiple virtual hosts, and option 3 splits the compute node from the data-node. Option 2 and 3 both require additional topology description.

VMware is currently working with the Open Source community to make changes to the schedule and HDFS layers, to add another layer of topology awareness. To model the new failure and locality we are proposing introducing a new layer in the topology hierarchy.

The new layer, shown in red, is called a Node Group and represents the hypervisor layer. All hosts/virtual machines under the same Node Group run on the same physical host. The schedule can use this information to place tasks on the same physical host, even when they are in different virtual hosts. Likewise, the HDFS file system can ensure it places its replicas on separate physical hosts, to guarantee replicas are in a different fault domain.

At the time of writing, these proposals are being discussed in the Apache Hadoop community, under the following JIRA’s:

• HADOOP-8469 (Umbrella JIRA)
• HADOOP-8469
• HDFS-3495
• MAPREDUCE-4310
• HDFS-3498
• MAPREDUCE-4309
• HADOOP-8470
• HADOOP-8472

Making Hadoop Highly Available

There’s been quite a bit of buzz lately about Hadoop HA. We’ve invested a significant amount of time looking at this area, and have tested VMware’s High Available (HA) and Fault Tolerance features with Hadoop.

The issue is that several services in a Hadoop cluster are single points of failure – if that service goes down, so does some or all of the Hadoop cluster. Some examples include onNameNode, Job-tracker, Hive Meta-data-database, HCatalog MetaDB, and any management servers (like Ambari or Cloudera CMS).

vSphere HA: Robust, Automatic VM Restart after Failure

The vSphere HA facility allows for automatic restart of failed services. The vSphere service monitors the state of the application in each VM with HA enabled using a heartbeat connected to the core of the application. If the heartbeat is lost, then that instance of the VM is powered off, and another instance is started on an available host. This protects against hardware failures, OS crashes, and application failures.

Since the HA facility is available and very easy to enable, customers have asked if they can use this industry standard HA approach, rather than learning how to set up and maintain other new cluster-failover mechanisms. We have tested VMware HA with NameNode and Job Tracker, and expect it to work for all of the other services in the stack where HA is required around the additional databases and services.

HA for NameNode in Hadoop 1.0

For Task-tracker and HDFS NameNode, we’ve worked with Hortonworks to build and test a full NameNode HA solution for Hadoop 1.0. There are two key components to this work, a client patch to make the clients aware to wait for NameNode to restart upon failure, and a monitor that links NameNode’s health to vSphere’s HA Heartbeat.

To complete the HA solution, the failure of the HDFS NameNode not only needs to be detected and repaired, HDFS clients also need to behave gracefully, while the HA system detects and repairs the fault in the HDFS master. To support this, the standard HDFS client’s existing retry logic was extended to support a configurable window during which it regularly attempts to reconnect to the NameNode. Assuming NameNode service is restored within that window, clients seamlessly continue their work, allowing HA to be transparent to HDFS. For NameNode health monitoring, Hortonworks has contributed an open source monitoring agent to the Apache Ambari project.

Using VMware Fault-Tolerance for Master Services

In addition to vSphere HA, the Fault-Tolerance feature provides continuous availability for virtual machines. This is different to HA in that the virtual machine continues running even in the event of hardware failure. This is facilitated by replicating the state of the running virtual machine to a paired VM on another host, in effect running in lock-step. This facility has the advantage that there is zero downtime in the event of hardware failure, and requires no changes to the Hadoop software stack.

It’s very easy to enable FT for master services, through the vSphere administration client, or through Serengeti’s configuration. Shown below is an example of how to enable FT through the UI.

While there are advantages to using FT, the solution is only appropriate for small to moderate sized clusters. We have tested Hadoop NameNode with vSphere FT and found that it introduces only a small overhead, just 2% impact to job runtime. At the time of writing , we estimated that it will suffice for clusters up to a few hundred nodes. We’ll be publishing a more extensive report on this in the future.

We anticipate that this solution fits the needs of most enterprise Hadoop deployments, while the full HA solution is capable of providing HA for even the largest clusters.

Using VMware HA/FT on Just the Master Services

It’s also useful to know that you can choose to use the vSphere HA/FT services on just the master nodes, and still get significant value. That may be very useful if you have an existing Hadoop cluster on physical, you can now find a vSphere service someone in your organization, and request to put the Name-node, Job-tracker et al on the vSphere cluster, but leave the compute/data nodes on physical.

Video Demonstration of Hadoop HA

See also our white paper on the VMware Hadoop HA Solution.

Spring for Apache Hadoop

VMware is also announcing updates to Spring for Apache Hadoop, an open source project first launched in February of 2012 to make it easy for enterprise developers to build distributed processing solutions with Apache Hadoop. These updates allow Spring developers to easily build enterprise applications that integrate with the HBase database, the Cascading library, and Hadoop security. Spring for Apache Hadoop is free to download and available now under the open source Apache 2.0 license. Costin has a full writeup of the Spring for Apache Hadoop project.

Visit us at Hadoop Summit

Sessions:

Apache Hadoop and Virtual Machines: Thursday, June 14 10:30 – 11:10am

Virtual Machines are a mainstay in the enterprise. Apache Hadoop is normally run on bare machines. This talk walks through the convergence and the use of virtual machines for running ApacheHadoop. We describe the results from various tests and benchmarks which show that the overhead of using VMs is small. This is a small price to pay for the advantages offered by virtualization. The second half of talk compares multi-tenancy with VMs versus multi-tenancy of with Hadoop`s Capacity scheduler. We follow on with a comparison of resource management in V-Sphere and the finer grained resource management and scheduling in NextGen MapReduce. NextGen MapReduce supports a general notion of a container (such as a process, jvm, virtual machine etc) in which tasks are run. We compare the role of such first class VM support in Hadoop.

Richard McDougall, CTO, Application Infrastructure, VMWare & Sanjay Radia, Member of Technical Staff, Hortonworks

Big Data on a virtualized infrastructure: making Hadoop more elastic, multi-tenant and reliable: Thursday, June 14 1:30 – 2:10pm

Big Data and virtualization are two of the most exciting trends in the industry today. In this session you will learn about the components of Big Data systems, and how real-time, interactive and distributed processing systems like Hadoop integrate with existing applications and databases. The combination of Big Data systems with virtualization gives Hadoop and other Big Data technologies the key benefits of cloud computing: elasticity, multi-tenancy and high availability. A new open source project that VMware will announce at the Hadoop Summit will make it easy to deploy, configure and manage Hadoop on a virtualized infrastructure. We will discuss reference architectures for key Hadoop distributions and discuss future directions of this new open source project.

Richard McDougall, CTO, Application Infrastructure, VMWare

Live Technology Demonstrations at VMware’s Booth

• Project Serengeti
• Separating Compute and Data
• vFabric Data
• vFabric Cetas

Arctic Row is an adventure and an effort to be the first to achieve a feat, but it is also an expedition to benefit the Arctic ecosystem, the most pristine and vulnerable in the world. The route across the ocean was ice covered just 7 years ago; a mark of global climate change. The publicity that the team receives for their adventure will be a platform from which they can raise awareness of the changes. The rowers have also partnered with Professor Hopcroft from the University of Alaska Fairbanks to conduct scientific research whales olfaction in feeding habits in the Arctic Ocean and will collect plankton samples to increase understanding of this key food source for marine mammals. Without Arctic Row, the research lacked sufficient grants (it costs $25,000/day to charter a 24-hour research vessel). It turns out that the relatively slow rowing speeds create ideal conditions for plankton collection, and Arctic Row offered to do the research at no cost. The New York Times featured their scientific plans on the front page of their website in March.

You can learn a lot more about the expedition at http://www.arcticrow.com/ and you can join them in their journey along with the VMware Foundation by contributing financially and posting comments at indiegogo.com/arcticrow until June 15. Arctic Row will use the proceeds to pay for scientific and expedition costs and to create an educational documentary film on the Arctic. Without Arctic Row’s research and film, mankind will have a diminished understanding of how to protect Arctic animals as climate changes.

If your organization runs such workloads and you are interested in hearing how they can be virtualized, consider voting for the following VMworld proposal. Public votes can be cast here (free registration required) and voting closes on June 8th.

1984 A self-service model for supporting engineering, scientific and other technical applications in a vCloud environment

As the use of virtualization technologies continues to increase in organizations, there is a class of workloads that to date has been left largely unvirtualized. These technical or High Performance Computing (HPC) applications often exist as unvirtualized islands of computation within the larger organization,  often separate from Corporate IT. These workloads are typically not virtualized due to performance concerns and due to lack of information about how one might construct a virtualized cluster environment to support such applications.

In this presentation we will briefly discuss the main value propositions for virtualizing such workloads and will show that a large class of these HPC applications can be virtualized now with little performance overhead. We will then describe a proven vCloud-based solution architecture that provides a self-service portal for research and engineering groups to dynamically instantiate compute clusters while allowing the underlying hardware to remain under the control of Corporate IT. This solution has been deployed and is currently in use at a major global pharmaceutical firm. This solution is general in nature and has applicability across a wide range of business types.

For some quick business context, VMware’s growth in China is driven by a strategy tailored for that region – we deliver local products with local partners that meet the local market requests. And wherever possible we are working to embed our products into local provider solutions. This strategy appears to be key to working with the China government, and has so far proven to be unique and successful.

My trip kicked off with a joint press conference with one of our key Chinese partners Inspur, to share details about our latest partnership news. The two companies will work together to provide Chinese customers with localized cloud solutions that consist of VMware’s cloud stack products packaged with Inspur solutions under the Inspur brand – with a goal of expanding the adoption of cloud in China. It’s great to see our partnership with Inspur strengthen as we work hand-in-hand to make cloud a reality. What’s great is that we’ve just replaced Xen as the component of their IaaS cloud offering as well.

I was also pleased to be keynoting the Bejing VMware Solutions Symposium, which is an event similar to our VMware Forum series. Hundreds of local customers and partners attended, and I shared VMware’s views on the Software-Defined Datacenter. I spoke about this concept earlier this month at Interop, and this is where the world is headed in terms of how IT operates and functions.  In a software-defined datacenter, customers have a full stack of data enter capabilities – storage, networking, compute, security, management – that look like software concepts. Customers are able to pull together all of their datacenter resources and allocate out as VDCs. Isolation is done in a safe way and all applications, including Hadoop, HPC and high-latency apps, can run extremely well in this environment. Ultimately customers end up with one platform that has fewer silos, which is of great benefit.

Attendees learning about the Software-Defined Datacenter
at the VMware Solution Symposium keynote

Next was an intimate CIO roundtable with VMware customers from various industries, co-led by David Sung (VMware President of Greater China region) and Yanbing Li (VP, CPD & Global Sites) and me. We spoke more about the Software-Defined Datacenter, where VMware is headed and how we provide value to our customers. We heard the audiences’ experiences and viewpoints around various topics including private cloud, BYOD, desktop virtualization and SaaS. Interestingly, the conversations we had in the roundtable resembled many that I have at other CIO events – customers are still diverse in terms of their virtualization and cloud adoption – some cutting edge and some just dipping their toes in the water. But it was great to hear their stories, questions and concerns and address what’s top of mind for folks as they figure out how cloud fits into their world. One thing I continue to reiterate is that the end state is compelling and you can get there in a series of steps. The journey depends on what state your company is in, and there’s no single path. I was energized to see that these Chinese customers are eager to learn more about cloud, figure out how they can mirror the IT transformations happening around them, and are looking to VMware to lead them on the cloud journey.

CIO roundtable

But that wasn’t the last of the keynotes! The next day I had the privilege of being a keynote speaker at the China Cloud Computing Conference (CCCC). This year’s event focused on driving cloud computing development, technology innovation and adoption, while showing the achievements of cloud computing in China, and exchanging experiences and methods for cloud practices. The conference was supported by various Chinese government departments and the Chinese Institute of Electronics, and it was fascinating to see how big of a participant the Chinese government was in this event. They’re making cloud an IT priority, making it a key aspect of their 12^th 5 year plan. In addition to cloud, I was impressed to also see a heavy focus on Big Data. That topic continues to be a hot one as our CTO Richard McDougall has written about.

My talk at the CCCC again focused on the Software-Defined Datacenter. When finished I turned the floor over to Yanbing Li to talk about how VMware China is making the Software-Defined Datacenter a reality. VMware has over 500+ in our China R&D organization, with teams driving the future of cloud with the great work they’re doing around VXLAN (Software Defined Networking and Security), vCenter Operations (Management for the Software Defined Datacenter) and vShield Edge (Software Defined Security).

I closed out the trip with my favorite part of the visit – spending time with the local engineering team. VMware has an extremely talented set of employees in China and it was great to hear more about their contributions to the VMware portfolio across all three layers – Cloud Infrastructure & Management, End User Computing and Cloud Application Platform – with key projects like View Mobile Client, Zimbra, Cloud Foundry, vHadoop, VXLAN, SMB-Mozy, and much more. VMware China is growing tremendously and it’s great to have such talented technologists in our R&D organization.

Cloud is alive and growing in China, and I’m already looking forward to my next trip!

What has struck me over the last few years is the consumerization of enterprise IT as a whole.  We often think about the adoption of consumer technology within the enterprise, especially smart phones, tablets, and even the x86 architecture itself.   But what we perhaps tend to think less about is the tendency toward consumer behavior and expectation.

Consumer behavior is coming, and coming fast to the enterprise, as hi-lighted by Paul Maritz in his EMC World keynote, and reported by Giga-OM.  If we think about Cloud, Cloud is really about convenience and instant gratification – “I want this service! I want it now.  I’ll pay for what I use, and then throw it away when I’m done.”  This is typical consumer behavior and expectation.  Not traditional enterprise behavior.  The Bring Your Own Device (BYOD) trend is really about getting the services and data I want, any time and any place, on the device of my choice.  After all I know how to be productive and, guess what, the devices I have are newer and better, and are replaced every year!  And then I’d like to do this socially please.  I’d like to work with my team using the kinds of mechanisms I’ve grown used to and that facilitate easy and effective collaboration.  The enterprise is being morphed by both consumer technology and perhaps even more so, by consumer expectation and behavior.

The likely nature of the work force of the future can be illustrated by looking at my own family…

I have 4 daughters.  My eldest is 21 and doing a PhD.  She’s smarter than me.  Speaks a language I don’t understand any more.  Lots of stuff about biochemistry and genetics.  She’s already toyed with the idea of starting life-science services in the Cloud.  Yikes!   My youngest is 2.  The first thing she does when she sees a new electronic device is to go up to it and swipe her little index finger along the bottom.  Everything is an iSomething.  At 2!  But my 19 year old is the most interesting in terms of pointing the way to how the near term future workforce will expect to do things.  She does not read email.  She does not write email.  She does everything using messaging.  She does that messaging on a social site, and she does the vast majority of it on her mobile phone.

My 19 year old is a millennial.  And she is truly representative of the workforce of the future.  The very near future.  IT organizations have a choice.  They can sit on their thrones, like Cnut (Canute), demonstrating futility in the face of inevitability, or they can figure out how to take advantage of these shifts, safely and securely.

Which brings me back to my trip to Turkey, where one senior executive told me about his boss.  His boss recognized the inevitability of these changes and created/chartered a team to formulate and drive a strategy to take advantage of these.  No one on that team was older than 30!  They were the Millennial Committee.  And it wasn’t long before they had successfully deployed their own social platform inside their business.

So what are you doing in your organization to embrace all of this?  (Hint – talking to us might be a good idea! :o)

Wanova Mirage is an innovative image replication and layering technology that combines sophisticated, layered image management with client-side execution and persistent caching. It was designed to centralize the management of desktop images in the data center and to deliver those images efficiently to physical Windows systems for a native user experience, as well as to virtual machines for execution.

Both VMware View^™ and Wanova Mirage provide centralized desktop image management, all or nothing patching and push button image resets. The key difference in these technologies is that VMware View images execute on servers in the data center and use a remote graphics protocol for the user interface, while Wanova Mirage images are transmitted and  cached locally for runtime execution on the client systems. This makes Mirage well-suited for executing managed images on disconnected laptops, including MacBooks running VMware Fusion^®.  And VMware View remains the ideal choice for executing images securely in the data center and delivering the user interface to remote devices such as tablets and thin clients. In this manner, the two products stand independently and when put together address a much broader swath of the market than either is capable of individually.

How Does Wanova Mirage Work?
The Wanova Mirage client splits the desktop image into a set of discreet and independently managed layers, including the operating system, corporate/departmental installed applications, user-installed applications and user data and settings. Wanova Mirage then efficiently replicates these layers to the data center and keeps them synchronized. The data center replica of these layers are efficiently stored in a central repository and aggregated into an object referred to as a Centralized Virtual Desktop (CVD).

A CVD represents the desktop image and can be efficiently stored and managed, modularly restored or redeployed to any physical or virtual system. Individual layers can be repaired or regenerated independently. While the CVD exists in the data center, bi-directional asynchronous synchronization technology maintains a cached replica of the image for execution on either native physical hardware end points or in a virtual machine container.

The CVD layers are maintained and managed independently and acts as an intelligent, bi-directional cache. The base image and corporate/departmental applications layers are controlled and managed by IT.  Changes made in the data center are automatically propagated to an employee’s desktop or laptop. User-installed applications that are added on an employee’s desktop or laptop are stored in the personalized user layer and are replicated back to the data center. The image replication and synchronization mechanisms were carefully designed and optimized for the WAN, utilizing sophisticated de-duplication / compression, CPU & network bandwidth throttling, and single-instance file storage in the datacenter.

How Does Wanova fit into the VMware EUC Product Portfolio?
As a standalone product, Wanova Mirage broadens our EUC offerings by bringing many of the operational and centralized management benefits associated with VMware View to laptops and physical systems. This means both native user experience through local execution and disconnected access. The benefits include push-button image restore/recompose with user state preservation, desktop DR, far more reliable patch management and a variety of migration use cases. Wanova Mirage enables very sophisticated migration scenarios; including efficient automated zero touch mass migrations from Windows XP to Windows 7 and hardware upgrades or refreshes. 

Wanova Mirage’s underlying technology is also highly complementary to View use cases, by bringing efficient image diversity and management to VMware View and VMware View Composer. Wanova Mirage’s layered image management is highly complementary to VMware View Composer provisioning and runtime storage technology by enabling comprehensive personalization with user-installed and departmental application layers coupled to a common base image.

For example, with Wanova Mirage handling desktop resiliency, it is now feasible for direct-attached storage based Composer pools to serve persistent desktops.  And for stateless desktops, departmental layers can be preserved across recompose/refresh pool operations meeting image diversity requirements.

Wanova Mirage also completes our vision of extending VDI to locally executed and offline use cases, enabling seamless two-way replication of managed Windows images, applications and persona between the data center and physical laptops, as well as delivering them for execution as VMs on Windows, Mac with VMware Fusion and Linux. Wanova Mirage’s automatic replication and synchronization is well suited to this use case and makes VMware View completely compatible with offline execution for roaming laptops and tethered desktops where WAN connectivity precludes a remote graphics protocol.

Wanova Mirage also facilitates both branch cache for Remote Office Desktops and desktop DR Always On Desktop use cases. Wanova’s efficient replication of images to the alternate locale can facilitate two forms of branch cache and disaster recovery – 1) runtime instantiation of the image into virtual desktops for delivery by VMware View or 2) local execution on native endpoints with Wanova Mirage.

Just the Beginning
And this is just the beginning. As Windows evolves into a runtime environment for Windows applications delivered from the cloud, the Wanova Mirage technology presents a great building block for delivering Windows Applications as a Service through VMware Horizon and enabling managed service provider offerings from the cloud. We are very excited to have Wanova Mirage as part of the VMware EUC platform.

Welcome Wanova!

As a great validation of VMware’s decision to acquire Cetas, that team has been honored with the TiE50 award in the Software category. This award acknowledges their achievements and underscores the recognition we are getting in the Big Data Analytics space. We are thrilled to receive this prestigious award and it is a solid recognition of all the innovative and hard work done by the team. TiE is a leading entrepreneur-focused organization doing a tremendous job of fostering creativity and innovation amongst entrepreneurs while identifying the technologies and companies that rise to the top in their respective focus areas.

More and more business users are demanding instant access to data and insights from the data; they want the direct ability to perform analytics on their data thus democratizing the process. With in-depth analytics, companies can better understand their users’ behavior and make decisions that help to optimally monetize their users’ activities. Recently, companies have been working with data scientists and do-it-yourself methods using Hadoop in order the harness the power within their data.  But these efforts are largely experimental and lack the depth that purpose-built, machine learning algorithms can bring to the analytics world. For that reason, companies are increasingly turning to productized analytics offerings like Cetas which can offer real-time and in-depth analytics to end users. This provides business and IT users with instant business intelligence by surfacing trends and patterns never before understood, enabling immediate revenue-enhancing decisions.

Having real-time and deep analytics capabilities that are directly accessible to business and IT users greatly empowers an organization to make fully informed critical decisions rather than making those decisions in a vacuum or in delayed fashion. I believe all companies must take steps to create such an environment for their users.

Learn more about VMware/Cetas’ Cloud & Big Data analytics offering on the website.

Perhaps I’ll have to make do with applying a pendulum metaphor to something that’s been on my mind lately as I get more immersed in the subject of enterprise mobility. I’ve been thinking a lot about how the distribution of computing power as delivered by applications has shifted back and forth from centralized to decentralized models over the last 20 years. The proliferation of powerful mobile devices forced the industry to reconsider decentralization as a result of new apps which in another era could have been dismissed as “fat clients”. If you represent this back and forth shift as a pendulum, you’d see complex, heavyweight client applications talking to often equally complex server resources on one end, and thin, rendering vehicles like browsers talking to complex server on the other. The pendulum started swinging from the right side in the mainframe era, moved dramatically to the left for client/server and then swung right back for the Web.

Now it seems that the massive expansion in mobile applications are shifting the pendulum back to the left, towards something that could easily be described as “the revenge of client/server” (which could also be the title of a really bad movie about IT people). It’s not quite a full throwback to the days of PowerBuilder and VisualBasic, but it does represent a shift in both the way in which applications are built, and the way in which they are ultimately delivered to the end user. That shift suggests that there’s still some contention around the basic concept of the Internet Browser that demands applications should always be centralized away from the user and accessed through simple windows (pun fully intended — note lower case w) that simply render stuff on the screen.

This topic even came up in a recent interview with Marc Andreessen in Wired Magazine.

About halfway through the article, Marc is asked about his infamous comment back in the browser war era that Microsoft Windows would “become a poorly debugged set of device drivers” and that access to all applications would be done through a browser. Turns out he borrowed that line from Bob Metcalfe! Still, as a former Netscape engineer, I was curious how his views may have evolved 10+ years now that he’s dedicated himself to investing in companies that are trying to bring us the next big thing. During the interview, Marc revisits the idea and reaffirms he still believes that computing wants to be centralized. He goes further to suggest that “this is what technology wants” and along the way qualifies the statement by assuming that infinite and ubiquitous bandwidth would need to be available for this to take hold (a pretty big assumption, to say the least).

Well, if this is truly a pendulum phenomenon where we are always in the midst of a swing from centralized to decentralized computing, it would seem like Marc and others believe the ball is on the way back towards the centralized model and that consequently anybody betting on a future of rich, standalone client apps is going to be disappointed.

But why? What’s so bad about decentralization? Or better yet, is there anything better about a more “client/server” type model? Well, let’s start with the bad news. One of the main challenges that Marc and others raise about rich client apps is the challenge of updating and managing apps on remote devices. As a guy who has 10+ apps needing upgrades on his phone at any given time, I can appreciate the challenge this poses. But guess what? The reason I don’t update those apps all the time is because for the most part, whatever I’m using is working. Yes, I may be missing features and bug fixes, but in the end, I will happily update them when I feel it’s necessary. Let’s say we’re talking about enterprise apps, and worse yet, ones where the updates are related to security and have to be applied. There’s an answer for that one too. The task of remotely delivering and updating applications on managed devices is something that is not only technically possible, it’s already being tackled by some vendors. Whether or not the management model for apps is perfect today is another matter. The point is that manageability of apps is not in and of itself a reason to resist the idea that feature rich apps that run locally on a device are a good idea.

In addition to focusing on the challenges posed by rich clients, its worth taking a look at some of the benefits of this model. After all, there’s a reason why our pendulum has shifted away from the web and back to the other side. Perhaps the biggest benefit is in functionality. Rich client apps being built today not only are capable of accessing powerful device components like cameras, accelerometers, GPS, and microphones to deliver experiences that are simply not possible on PCs. Regardless of whether they’re built in native frameworks or in HTML5, the fact remains that apps that are smart enough to run locally while leveraging remote services where necessary provide the user experience, especially when you consider the reality that no matter how ubiquitous bandwidth gets, apps will need to gracefully handle the occasionally transient nature of any wireless connection.

Having explored this pendulum analogy there’s still one last question to answer. What’s the force that keeps the pendulum in motion? In a Foucault pendulum, gravity acts as the “restoring force” that pushes the pendulum back towards equilibrium. That force plus the mass of the pendulum is what swings it past equilibrium to the other side keeping the thing in motion forever (or for a very very long time if you don’t grant me air friction and other technicalities :). Well, it turns out that users like you and me are the restoring force that keeps the pendulum in motion. Our desire to use technology to improve our personal and work lives leads us to choose things we like and leave behind things we don’t. This is in sharp disagreement with Marc’s observation that “what technology wants” is what will ultimately lead us back towards a server-centric model. Perhaps we’ll get to a world where wireless connectivity truly is ubiquitous and rich apps can be created that run entirely in some cloud while magically beaming down three dimensional holograms with the directions on how to get to your next meeting. Until then I’ll continue to enjoy the resurgence of client apps and wait for the pendulum to start swinging again.

More recently, I took up a challenge to see how much we could learn about Hadoop I/O in a very short time, using our dynamic tracing framework. The results were quite interesting.

To ensure I position this work correctly, it’s really a work-in-progress study that I’d like to engage a discussion around.  It’s what we’ve learned by digging into the architecture, to help us make engineering decisions. I hope it’s helpful, and really want to get your feedback, so we can steer future iterations of these investigations.

The data in this post is a lead up to some simple modeling of Hadoop in a cluster, so we can rationalize how Hadoop will scale in different topologies.

Why Analyze Hadoop?

To design our Hadoop architecture, we have conducted many studies of the Hadoop stack. In this post, I want to share an initial analysis to build a better understanding the resource and topology requirements are of a distributed Hadoop system. The more we discussed with Hadoop committers, it became clear that there needed to be a shared view on what Hadoop I/O actually is. For example, there is much discussion about the need for bringing compute and data together, but when you dig in, we find that there are several types of I/O in the map-reduce and Hadoop model, with very different needs for locality, performance and underlying storage.

To learn more, I decided to invoke some of my dynamic tracing lessons learned, applied to a distributed Hadoop stack – by using the vProbes introspection framework that is built into the VMware hypervisor.

Before we get into the details, lets review some of the questions we are trying to answer. At the architecture level, we are evaluating some key options – for example what happens if Map-Reduce and Datanode are running in different VMs or on different hosts. In this case, some of the data is accessed remote, but as shown by this analysis, much of the access is to ephemeral data, which can always be local. More on this topic later.

To fully understand these dynamics, I’m seeking to categorize the major types of I/O into groups, and then be able to describe the types of I/O for each.

Some of the questions that we are looking for answers are:

• What are the major categories of Hadoop I/O?
• How much I/O is generated per unit of compute?
• How much I/O should we design for on each host?
• What is the nature of that I/O — is it sequential, random, what block size?
• For Hadoop HDFS, what is the impact of I/O being remote from the host?
• What categories of I/O are remote vs. local if the task is run on a remote node?
• Would 10Gb Ethernet help?

This resulted in a mini-study of Hadoop’s I/O, which I’d like to open as a starting point for discussion. Again, this is early observations, and I’d like to solicit opinion from the community and progress the analysis with feedback.

Keeping it Simple

To enable us to get visibility into the I/O categories, I’ve deliberately simplified the workload down to very basic configuration. In fact, I’ve used a ridiculously small example of Terasort. This allows us to track and visualize the I/O across all tasks, and zoom in on a task to analyze and categorize the detailed I/O.

The Hadoop example I’m using for analysis is Terasort with 1G/byte of input data, using two map slots and one reduce slot.

Map-Reduce Recap

As background to this analysis, lets recap the architecture of Hadoop Map-Reduce and the phases we expect the workload to go through. For our Terasort example, Hadoop Map-Reduce will implement several phases as shown in the following diagram:

Terasort in Action

Running Terasort in our small controlled study takes just under 10 minutes, with a 1 billion byte input. Nothing stellar, this configuration isn’t optimized for massive throughput, rather we’re hoping to introspect each step.

I have just two map slots configured, which means that at most, two tasks will run in parallel. Since each task processes 64MB (the default HDFS block size), then we should expect 16 tasks.

Parsing the logs of the job output, we can see the detail of what tasks ran. To make it simple to digest, I ran the job output log through a simple perl script to reformat into per-step stages:

$ log.pl job_201201261301_0005_1327649126255_rmc_TeraSort 
           Item    Time  Jobname           Taskname Phase Start-Time End-Time Elapsed
            Job    0.000 201201261301_0005
            Job          201201261301_0005
            Job    0.475 201201261301_0005 PREP
           Task    1.932 201201261301_0005 m_000017 SETUP
     MapAttempt    3.066 201201261301_0005 m_000017 SETUP
     MapAttempt   10.409 201201261301_0005 m_000017 SETUP SUCCESS 1.932 10.409 8.477 "setup"
           Task   10.966 201201261301_0005 m_000017 SETUP SUCCESS 1.932 10.966 9.034
            Job          201201261301_0005 RUNNING
           Task   10.970 201201261301_0005 m_000000 MAP
           Task   10.972 201201261301_0005 m_000001 MAP
     MapAttempt   10.981 201201261301_0005 m_000000 MAP
     MapAttempt   65.819 201201261301_0005 m_000000 MAP SUCCESS 10.970 65.819 54.849 ""
           Task   68.063 201201261301_0005 m_000000 MAP SUCCESS 10.970 68.063 57.093
     MapAttempt   10.998 201201261301_0005 m_000001 MAP
     MapAttempt   65.363 201201261301_0005 m_000001 MAP SUCCESS 10.972 65.363 54.391 ""
           Task   68.065 201201261301_0005 m_000001 MAP SUCCESS 10.972 68.065 57.093
           Task   68.066 201201261301_0005 m_000002 MAP
           Task   68.067 201201261301_0005 m_000003 MAP
           Task   68.068 201201261301_0005 r_000000 REDUCE
     MapAttempt   68.075 201201261301_0005 m_000002 MAP
     MapAttempt  139.789 201201261301_0005 m_000002 MAP SUCCESS 68.066 139.789 71.723 ""
           Task  140.193 201201261301_0005 m_000002 MAP SUCCESS 68.066 140.193 72.127
     MapAttempt   68.076 201201261301_0005 m_000003 MAP
     MapAttempt  139.927 201201261301_0005 m_000003 MAP SUCCESS 68.067 139.927 71.860 ""
           Task  140.198 201201261301_0005 m_000003 MAP SUCCESS 68.067 140.198 72.131
[snip…]
  ReduceAttempt   68.112 201201261301_0005 r_000000 REDUCE
  ReduceAttempt  795.299 201201261301_0005 r_000000 REDUCE SUCCESS 68.068 795.299 727.231 "reduce > reduce"
           Task  798.223 201201261301_0005 r_000000 REDUCE SUCCESS 68.068 798.223 730.155
           Task  798.226 201201261301_0005 m_000016 CLEANUP
     MapAttempt  798.241 201201261301_0005 m_000016 CLEANUP
     MapAttempt  806.113 201201261301_0005 m_000016 CLEANUP SUCCESS 798.226 806.113 7.887 "cleanup"
           Task  807.252 201201261301_0005 m_000016 CLEANUP SUCCESS 798.226 807.252 9.026
            Job  807.253 201201261301_0005 SUCCESS 0.000 807.253 807.253

From the log, we can see that we actually run nineteen tasks. There are 16 map tasks and one reduce tasks, plus two extra admin tasks (task 000017 and 000016), which are launched to prep for the job and to cleanup after the job.

The same information is better represented by a swim-lanes diagram. This allows us to see the time graph of the tasks as they are launched and when they complete. We can see that at most there are three tasks running, the two map tasks, and a reducer that is started after the completion of the first task.

We’re going to dig into what the individual map and reduce tasks do.

Enter vProbes

In 2008, vProbes first discussed in public — as a dynamic tracing framework that allows dynamic instrumentation of a running virtualization system without code modification. vProbes allows probe points  to be inserted into the virtualization kernel, the guest operating system kernel, and the guest application code.

For the purpose of this discussion, it’s a unique way of probing multiple guest operating systems at once. For this experiment, we’re looking at the system calls of the task trackers and datanodes in the mini Hadoop cluster. This lets us see what I/O is written by OS calls on each service, giving resolution of the I/O timing, size distributions and detail of the pathnames of each operation.

The vProbes framework is implemented in several VMware products, including VMware fusion – this data is from an experiment running vProbes on Fusion 4.1.1. It was enabled by setting vprobes.allow in the Fusion configuration file /Library/Application Support/VMware Fusion/config. I built the vprobes binary from the community source repository.

Once installed, we can probe the guest operating system.

Probing our Hadoop Nodes

Using probes around the system call entry and return functions, we can gather information about I/O, and reach into the arguments of each system call to get further detail about each operation.

GUEST:ENTER:system_call {
    string path;
    comm = curprocname();
    tid = curtid();
    pid = curpid();
    ppid = curppid();
    syscall_num = sysnum;
    if(syscall_num == NR_open) {
	path = guestloadstr(sys_arg0);
      syscall_name = "open";
      sprintf(syscall_args, "\"%s\", %x, %x", path, sys_arg1, sys_arg2);
    …
}
GUEST:OFFSET:ret_from_sys_call:0 {
    printf("%s/%d/%d/%d %s(%s) = %d \n", comm, pid, rtid, ppid, syscall_name,
                                              syscall_args, getgpr(REG_RAX));
}

vProbes allows a powerful set of aggregations and reductions at the probe site, but in this case I’m brute forcing the system to give me a trace of all I/O, which we can analyze later. We yield the following output from each Hadoop node:

java/14774/15467/1 open("/host/hadoop/hdfs/data/current/subdir0/blk_1719908349220085071_1649.meta", 0, 1b6) = 144 <0>
java/14774/15467/1 stat("/host/hadoop/hdfs/data/current/subdir0/blk_1719908349220085071_1649.meta", 7f0b80a4e590) = 0 <0>
java/14774/15467/1 read(144, 7f0b80a4c470, 4096) = 167 <0>

Each line gives us the process name, pid, thread-id, system call name, arguments and return code for each system call of the node. Tracing with vProbes made this much simpler than using regular system call tracing, since we could gather all information from one place, and didn’t need to worry about trying to attach to child processes which are spawned by the multiple layers inside Hadoop that creates JVMs (Hadoop creates an on-the-fly script named taskjvm.sh to launch new task JVMs).

High Level Summary of Hadoop’s I/O

As mentioned, we’re running terasort on a billion bytes of input. For completeness, the command lines used for the study are:

$ hadoop jar hadoop-examples-0.20.204.0.jar teragen 10000000 teradata

<begin trace>

$ hadoop jar hadoop-examples-0.20.204.0.jar terasort teradata teraout

We are just studying the output of the second command – the terasort.

The Hadoop job output shows us a summary of the Job:

Job Counters
     Launched reduce tasks=1
     SLOTS_MILLIS_MAPS=1146887
     Launched map tasks=16
     Data-local map tasks=16
     SLOTS_MILLIS_REDUCES=766823
   File Input Format Counters
     Bytes Read=1000057358
   File Output Format Counters
     Bytes Written=1000000000
   FileSystemCounters
     FILE_BYTES_READ=2382257412
     HDFS_BYTES_READ=1000059070
     FILE_BYTES_WRITTEN=3402627838
     HDFS_BYTES_WRITTEN=1000000000
   Map-Reduce Framework
     Map output materialized bytes=1020000096
     Map input records=10000000
     Reduce shuffle bytes=1020000096
     Spilled Records=33355441
     Map output bytes=1000000000
     Map input bytes=1000000000
     Combine input records=0
     SPLIT_RAW_BYTES=1712
     Reduce input records=10000000
     Reduce input groups=10000000
     Combine output records=0
     Reduce output records=10000000
     Map output records=10000000

The key items for our I/O study are the byte counters. Recall that we’re sorting one billion bytes of input data, so we would expect at least double this amount of I/O – for the read + output. This is faithfully represented by the HDFS_BYTES counters, showing 1B bytes read and 1B bytes written. We are however also seeing another 5,784,885,250 bytes read and written to  files. So our 1B byte sort resulted in 7,784,885,250 bytes of actual I/O. The reason for this is the extra ephemeral I/O that Hadoop does during sorting, staging and copying data between phases. Keep in mind this not a tuned Hadoop instance, so arguably we may be spilling during sort more often than we should, but there is still a significant amount of temporary I/O going on in even a well tuned system.

Digging into the breakdown between persistent data (HDFS) and ephemeral I/O teaches us a lot about the overall Hadoop I/O model, and helps us understand the implications of the storage bandwidth required as we scale-out the workload across multiple hosts and racks in a distributed system.

Making Sense of the Ephemeral Data

Categorizing the ephemeral data helps us understand the nature of the I/O, and rationalize how it might change if scaled up, or if the task was run on a remote node. It also brings to light some future optimizations that may be possible, and possibly raises some questions about why some of this data isn’t stored in HDFS.

I circulated the I/O observations around several Hadoop committers and we came to the following initial grouping of I/O categories:

• DFS Input data: Map input data that is read from the distributed file system (HDFS)
• DFS Output data: The output written by the reducers as the output of the job
• Sort Spills: Data that is written and read from disk as a result of overflow from a sort operation that didn’t fit in memory. Spills may occur on the map and reduce tasks.
• Logs: Output of the jobs, stdout, etc.
• Job staging: The environment for the job. Copies of shared libraries, jar files, config files etc.
• Map Output: Output of the map into an intermediate file on the Map task, that is later read during the shuffle phase
• Shuffled map output: The shuffled map output on the reducer, that is read from multiple map tasks. The Map output is held temporarily in the map output intermediate file on the map task, and made available by the task-tracker that hosted the job through a HTTP service with a private Jetty server.
• Combined, reduced output: the output after the reducer combines the shuffled key-values.

The Map Task

Each Map task reads it’s split of the input data from HDFS and implements it’s map function on the data. In the Terasort case, the map stage simply reads the input into key-values and sorts its split of the key-values. This results in reads across the network from HDFS, I/O to temporary spill files, and write of the output into the mapper’s intermediate output file (the one that the task-tracker will expose through HTTP to the reducer for the shuffle phase).

By aggregating and sorting the I/O’s by pathname, we can see that most of the I/O is to these well known destinations, and with a longer tail of I/O to the task staging, log, etc,… The path file.out is our map output, which we would expect to be 64Mbytes. The spillx.out is our sort spills.

By looking at the I/O attributes, we can learn a little about the I/O patterns for each component. In this case, some aggregation of I/O sizes reveals the logical I/O sizes before they are serviced by Linux ext4 fs.

Since this is a bandwidth-intensive app, we’d expect larger I/O sizes. Interestingly, we’re seeing most of the I/O performed in small I/O sizes. The data in the I/O size graph is by bucket, and per the traces, most of the I/O is happening at an odd size – 4090 bytes, mostly likely a multiple of the key-value size?

Since there is a large number of I/Os at a smaller size, it makes more sense to view the total bytes by I/O size, so that we can see how much I/O was generated in any one I/O bucket. This way, if a few very large I/Os are performed, we’ll be able to see if they represent any reasonable fraction of the total. However, we see that the majority of the bytes transferred are those with smaller sizes.

Since it wasn’t really evident that Hadoop’s I/O is sequential from this data, we went a step further and sliced the data another way. By looking at the offsets per file descriptor, you can aggregate adjacent I/Os, allowing a similar graph to be drawn of logical I/Os – those which are identified by any connected sequential I/O. This time the data looks quite different.

Through the Logical I/O graph, we can see what was hidden, that the actual I/Os are very sequential. In fact, the entire output is written sequentially, and much of the sort spill I/O is in I/O sizes between 128k and 4Mbyte.

The Reducer

The reducer starts it’s work by copying key-values from multiple map-nodes, in a phase known as shuffle. Data from map output is kept around and exported by the task-tracker, and read by the reducer over the network. It’s interesting that this data doesn’t just go into HDFS.

In this Terasort, there are no reducer spills. Data is read from the mappers into the mapx.out files, and then combined into the intermediate.1 file, sorted and then written back into HDFS.

The Data Node

The HDFS data-node is where the majority of the distributed I/O occurs. A client read from the distributed file system first consults the name-node to locate the relevant data-node for the block in question, then connects directly to the data-node to read that block.

The datanode stores that block in to key files – a Linux file containing up to 64Mbytes of the block, and a smaller file used to store the checksum of that data. For terasort, we see two clusters of I/O sizes, based on the I/O to/from the block and the read/write of the checksum.

Most of the I/O is performed in 64Kbyte chunks. This is because the I/O is initiated to/from the client using sendfile, with a buffer size of 64k. As a result, the data node is relying on the underlying file system to aggregate these requests into bigger I/Os.

When we perform the same logical I/O sequentially tests, we can see that the majority of the I/O is in fact sequential.

We haven’t done any experiments with Direct I/O yet, but it is something that has been asked. One would speculate that we’d not be fulfilling the true I/O bandwidth of the I/O subsystem if we used Direct I/O, since 64k is typically not large enough to get the best sequential bandwidth from a modern disk.

 Summary

Tying all of the data together, we begin to get a model view of a typical Map-Reduce job and it’s I/O requirements. Our job goes through these phases:

• Reads it’s input data from HDFS (ideally from a local datanode, but only 12% of overall I/O)
• Spills at high bandwidth during initial sort (guaranteed to be local, since it’s using local temporary storage on local disks)
• Writes map output (also guaranteed to be local, on temporary disks)
• The reducer reads that across the network (almost 100% probability it’s from a remote host)
• The reducer spills (guaranteed to be local disk)
• The reducer creates an intermediate file (also local)
• The reducer writes back to HDFS (one copy should be local through the replica placement engine, two more remote copies)

Interestingly, we think of Hadoop as always needing to have it’s data local as litterally meaning it’s task must run where the block’s data node is, however the consequence of running the task on a remote datanode is much more diluted that we might think. Running remote in this example results in only 25% of the total storage being accessed across the network.

This leaves a few interesting questions open:

• Is there really such a big impact for running jobs remote? Perhaps it’s only an issue with 1Gb networks, but with 10GB networks and fast top of rack switches, locality doesn’t matter any longer?
• We’re making extra copies of data and using a different transfer framework for map output. Why doesn’t map data simply go into HDFS, removing the need for the additional web proxy and making it easier to optimize locating the reducer?
• What are the actual bandwidth requirements of Hadoop on local storage and network, both at the host level and when we aggregate into rack and system? From this data, it’s possible to create some back of the envelope sizing models that allows us to rationalize how much actual bandwidth is required for local disks, host network, top-of-rack switch and between racks.

Let me know your thoughts. I’ll followup next with some implications for the storage and network, and a simple model to point towards answers to the above.

While VMs can be provisioned in minutes, provisioning the rest of the data center (storage, networking, security, etc.) takes days – the SDD vision, with the VDC container holds the promise of bringing this down to minutes, unleashing a whole new level of agility in the data center.

The SDD concept builds upon the Software Defined Networking (SDN) movement, which is the latest area to gain the attention of the IT industry, including researchers, investors, vendors, enterprises and cloud providers. Between the recent Open Networking Summit and Interop, we are seeing an extreme makeover within the networking industry – we haven’t seen so much excitement around networking in a long time.

Networking (and security) architectures have just not kept pace with the agility, efficiency and elasticity requirements of modern data centers. This is where SDNs (and Software-defined Security – SDSec) hold promise. The Open Networking Foundation (ONF) has done a tremendous job in rallying the industry behind the need for SDNs, and jumpstarting the movement to a new era of networking.

The Open Networking Summit was a celebration of the momentum in the space, and rightfully so. By far, the hit of the show was Google’s public disclosure of their dramatic use of OpenFlow to control their backbone traffic; Google effectively replaced routers with some Marvel chips, an ATCA chassis, and traffic engineering in place of routing, all using open source. Remarkable!

However, in the midst of all the excitement, one topic stood out – we as a collective whole, are not clear on what SDN really means – is SDN the same as OpenFlow? Is SDN a reference architecture? Is SDN a marketecture? Is it prescriptive? Here is a summary from Nick Lippis: What I Learned at the Open Networking Summit about Software-Defined Networking

At the ONS closing panel of ONF members hosted by Dr. Guru Parulkar, Executive Director of ONF, we talked about two possible views on SDN:

On the left is the classic view of SDN and OpenFlow. The OpenFlow Controller controls OpenFlow devices, and exposes tenant-specific slices to applications above. This enables research slices, Google Traffic Engineering, and other innovative applications. To a large extent, OpenFlow is used to replace the existing control plane in networking gear, as long as such gear have OpenFlow agent support.

On the right is a more pragmatic view of the SDN stack. Instead of using the OpenFlow protocol to control OpenFlow devices, we enable network virtualization by provisioning on-demand overlays on top of existing networking gear and fabrics. VMware has teamed up with several networking, NIC, ToR, switch, and silicon vendors to create high performance VXLANs. These are controlled by the network virtualization controller, which makes logical networks available to higher level applications such as vCloud Director.

In this model, network virtualization follows the footsteps of server virtualization:

1. Abstraction: We abstract switches, switch ports and NICs (vSwitch, vPort, vNic)
2. Pooling: We pool together switches (VDS), switch ports (VXLAN), thus creating the notion of an elastic Ethernet, than VMs can join/leave on an as-needed basis
3. Slicing: We can assign isolated VXLANs to different tenants, or lines of business, or app owners. Each VXLAN has a tenant id (VNI).
4. Finally, virtual data centers (VDCs) can consume these logical networks (VXLANs), on demand to interconnect compute, storage, etc.

In this model, we do not try to influence the specific path a flow will take through the network. Rather, we rely on ECMP to spread the flows across the fabric, leveraging as many of the paths as possible. Also, in addition to layer 2 network virtualization, we have begun to repeat the same abstraction/pooling/slicing/consumption paradigm for L3-7 network services like firewalls, load balancers, IPAM, VPNs, all logically inserted into the virtual plane.

Note that, as OpenFlow makes its way into fabric gear, we could go one step further, and pin VXLAN flows, if needed, leveraging the OpenFlow protocol.

The key in this architecture is to allow innovation to simultaneously proceed, above the SDN layer (Network Services), within the Controller, and beneath the overlay.

I don’t believe the left and the right views are in conflict with each other. But having a reference architecture with identified interfaces, especially the “north-bound SDN APIs”, will be key to unleashing the power of SDNs…

These are the early days of the next generation of networking, as SDNs take shape. Regardless, Software-defined Data Centers, including SDN and SDSec, and the evolution from VM containers to VDC containers, is on its way to becoming a reality – it is just too compelling!

/Allwyn

At the beginning of the year, I posted that we are in the midst of a data renaissance  — where a rapid proliferation of new uses of data is driving new technologies to manage data. The traditional relational database had once been the main vehicle for serving the needs of online applications with optimized variants used to store longer-term data for business intelligence and analytics. No longer however, does one size fit all when it comes to data and database technologies.

The unprecedented volume of information being collected and analyzed is driving new Big Data technologies. Extreme performance requirements and changing hardware architectures necessitate new Fast Data solutions, like in-memory databases. Applications that interact with data in a variety of formats and require frequent changes to their data model have given birth to NoSQL and other solutions for managing this Fluid Data.  Finally, the move to cloud computing is accompanied by a need to deploy and virtualize data technologies for Cloud Delivery. As VMware works to transform IT for the Cloud, we’ll increasingly look to address the challenges and opportunities associated with Big Data, Fast Data and Fluid Data.

There is a growing opportunity to augment application’s capabilities with analytics. In this context, I am excited to announce VMware’s acquisition of Cetas Software, an early stage Cloud Data Analytics company. The Cetas team has been driving the democratization of analytics and business intelligence by dramatically lowering the barriers to entry for all companies, from small businesses to large enterprises.  The Cetas analytics service provides next generation Big Data analytics and predictive intelligence systems that can operate at a scale of hundreds of terabytes and billions of events for online companies and enterprises. The key use cases served by Cetas are Online App Analytics, IT & Operational Analytics and Enterprise Hadoop Analytics. VMware will continue to invest in the Cetas technology and invite interested users to sign up for the service here.

Stay tuned for more news about how we will be integrating Cetas into VMware’s cloud application platform and services. For more information about Cetas, visit www.cetas.net, the Cetas blog, and follow on Twitter at @CetasAnalytics.

I presented at the Public Sector Cloud Summit on behalf of the DMTF last week. The event was sponsored by the Cloud Standards Customer Council, and was kicked off with a keynote by Dawn Leaf, NIST’s Senior Executive for Cloud Computing. One of the big takeaways from Dawn’s presentation was that the new focus at the NIST will be on reference architectures. NIST certainly made great progress early on by developing the definitive definition of cloud computing. She stated that their next big contributions will be on standardizing reference architectures so that we can have a common understanding on which to build. The NIST will also be supporting the ongoing efforts of the Fed RAMP program to standardize security assessments of cloud solutions across the various US government agencies.

David McClure, associate administrator of the U.S. General Services Administration (GSA) Office of Citizen Service also gave a great keynote that addressed GSA cloud activities. He shared some of the successes in the US government adoption of cloud computing, as well as a number of challenges ahead. The creation and adoption of cloud standards as it relates to security, interoperability and portability were high on his list.

There was also a great presentation by VMware’s Andy Tait, former Deputy Director for the G-Cloud, UK Cabinet Office, on the state of the UK G-Cloud initiative which is also driving for standards-based interoperable clouds.

My presentation at the event covered interoperable clouds, the progress that we’ve made so far, and the opportunities still ahead. I wanted to educate the audience that we have made great progress on many key IaaS standards and to review the areas that we need to complete so that we can have truly interoperable clouds.

All of the presentations are available to the public and can be downloaded here:
http://www.cloudstandardscustomercouncil.org/March2012/agenda-pr.htm

It was amazing for me to see how many organizations across the globe are involved or have some activity around cloud computing. Organizations from China, India, Japan and across Europe have started work on cloud standardization. Many standards development organizations that weren’t involved six months ago have launched new cloud computing activities. This is a good sign and shows that there is global focus on cloud standards and interoperability. These efforts are sorely needed and will help to improve the security, portability and interoperability in clouds.

As I said in my previous post, we need to solve the challenges ahead of us and leverage the work already available to our community. A great example is the Cloud Security Alliance, which has focused its effort on best practices around cloud computing security and has not attempted to create new standards since plenty already exist in that space.

I recommend that organizations and groups that are just getting into this space also utilize the work that has been done, just as the CSA did. There are several inventories of standards from ISO, ITU-T, NIST and others like the public wiki at www.cloud-standards.org  of which folks can take advantage. That way, we as a community can focus our time on the gaps in standards and drive adoption of open secure cloud standards.

In this video I interview Juniper Networks’ Chris Hoff at RSA 2012, and we talk more about the security industry.

The annual RSA conference kicks off today, and our team at VMware is fired up about ongoing developments in the security space. To provide some perspective on VMware’s approach, our CTO Steve Herrod is keynoting today at the Cloud Security Alliance summit. I’d like to give you a glimpse of the commentary he has planned.

Today, IT is in a state of great change that has deep implications for the security community. We can see that data and applications are being accessed in a variety of new ways – SaaS applications are firmly entrenched, mobile applications are a demanding participant in any IT architecture; and simultaneously,   existing applications are being housed in virtual containers. Effective support and control of these diverse tools demands an updated approach to IT security.

Two distinct areas require focus in a modern IT security strategy:

• the infrastructure that supports your IT assets
• access to these assets

Consider the supporting infrastructure

Existing applications and data are being virtualized at a rapid pace. Over 50% of servers are now virtualized, on pace to reach 80M virtual machines by 2014. With the virtual server installed base growing at this rate, a traditional security strategy based on physical assets is being stretched irreversibly.

As virtualization becomes more deeply embedded, datacenter density is increasing and the days of an application being tied to a single piece of hardware are long gone.  Furthermore, as the datacenter network inexorably moves towards 10GbE access layer and 40/100GbE cores, physical firewalls/security will not be able to keep up.  Here is an insightful note from Jon Oltsik:

http://www.networkworld.com/community/node/79866

Add the emergence of hybrid clouds to today’s IT landscape and one can see the need for a wider and more complete approach to security.

The best way to manage the scale of virtualization, consolidation and high performance networking, the emergence of hybrid clouds and the requirements of new applications is through the establishment of logical trust zones for all applications and data. This approach untethers application deployment from the constraints of physical network segmentation and clustering, moving firewalls into the virtual plane and enabling linear scale out access control. Logical infrastructure isolation and access control is the most efficient way to manage security in a highly virtualized environment.

How do we access the assets?

The trends around BYOD (Bring Your Own Device), any device anywhere, multiple personas and web-centric computing are contributing to an essential crumbling of the traditional security perimeter. Users will ultimately abandon traditional static, physically limiting access to their applications and data.

Access to apps/data in the new world will focus more on the authenticity of the persona (versus the device), and the establishment of dynamic authorization directives (e.g. OAuth2, identity-aware firewalls). Likewise, on the datacenter side of the equation, the landing spot for access will extend the traditional physical DMZ and include logical, tenant-specific DMZs fronting logical apps (e.g. 3 tier apps, VDI pools).  Apps/data fronted by such virtual DMZs with policies to restrict access based on authenticated and authorized personas will become the norm in the emerging world order.

In short, a new security architecture has emerged to support virtualized IT infrastructure and user behavior around access to IT assets. Fundamental to this new architecture are:

• “Need to know” access based on authenticated & authorized personas
• Logical trust zones for apps, and data privacy
• Logical infrastructure isolation and access control
• Virtual DMZ to buffer access from any device anywhere

Each vApp (collection of related apps, servers and data) gets its own secure container, including a virtualized DMZ to restrict access. Once this proactive security architecture is in place, traditional threat defense like AV, IDS/IPS, DLP, etc, can be virtualized, inserted and enforced at such logical trust zone boundaries.

One can observe overall that security is moving closer to IT assets and becoming logical, vs. the traditional physical approach. To a large extent, there seems to be a parallel with what’s going on with the virtualization of networking, per my last posts on logical networks and VXLANs. This transformation leads to better security than was possible before…and the change is taking place right now.

Steve Herrod, VMware CTO, will discuss these trends and requirements in his Cloud Security Alliance keynote on Monday, February 27th at 12:30PM.

Phew, nothing like spending time at VMworld, which has become the new mecca for IT professionals everywhere – thanks again to everyone for the support and encouragement, egging us to keep on truckin’!

In my last post, I talked about the need to have logical networks, edges and trust zones on a per tenant basis, and the need to map these onto provider networks that are increasingly becoming fast, fat and flat. At VMworld 2011, we announced some key advancements that make these concepts a reality; VMware and partner booths demonstrated a variety of Software Defined Networking solutions, leveraging VXLANs, vCloud Director, vShield and vSphere/vCenter, all part of the emerging Cloud Infrastructure Suite – we are getting closer to what Paul Maritz refers to as the “invisible infrastructure”…

Let’s talk through some of the advancements in network & security virtualization, realizing elastic Virtual Data Centers (VDCs), and VMware’s ecosystem framework…

VXLAN – the protocol

One of the bigger announcements made at VMworld, was the VXLAN initiative. See Steve’s post on the subject, and this writeup from Mallik Mahalingham, Principal Engineer at VMware, who has spearheaded this effort at VMware.

From a wire-protocol perspective, VXLAN is essentially a MAC-in-UDP frame format, including a 24 bit segment Id. Effectively UDP gets you to the right ESX host/controller, the segment Id gets you to the respective Org/Tenant, and the “inner” MAC gets you to the right vNIC/VM in the Org. Additionally, tenant broadcasts are converted to IP multicasts (Protocol Independent Multicast – PIM).

Here are a couple of posts from networking blogger Ivan Pepelnjak, one on VXLAN versus OTV/LISP, and one on vCloud Networking. Worth a read.

VMware has been collaborating with Arista, Broadcom, Brocade, Cisco, Emulex, Intel and several industry players to work towards standardizing this protocol. With the broad collaboration, expect to see VXLANs become the currency for multi-tenant data center networking (enabling multi-tenant data center fabrics), with NIC vendors providing native offload/acceleration of VXLAN frames, Top of Rack and switch/router vendors providing fast, fat and flat implementations of the protocol, and vendors building VXLAN-VLAN gateways to enable high performance, mixed environments.

What’s nice about VXLANs is that it leverages existing infrastructure, yet can take advantage of advances in data center fabric and server NIC technologies as these become available. However, the really big deal here is that the broad (virtualization, system/NIC, and networking vendors) convergence on the frame format, enables the industry to move on, and focus on innovation in the data center fabric, within the virtualization layer, and in delivering capabilities to cloud tenants/orgs.

Building logical networks on the VXLAN foundation

So, on the infrastructure provider side of the equation, VXLANs provide us the capability to realize isolated, multi-tenant broadcast domains across data center fabrics. Let’s talk about how VMware and partners leverage this capability in emerging provider networks to provide elastic, logical networks to tenant/org VDCs.

The Virtual Distributed Switch abstracts the data center fabric and provides a sea of ports. vCloud Director (VCD) creates an Org Virtual Data Center (VDC), including allocating compute and storage resources. Tenants/orgs can now provision their own logical network to connect these resources. VCD delegates networking/security control to the vShield Manager, which in turn creates a VDS port group backed by a VXLAN, maps the tenant id to the VXLAN segment id, and connects org VMs to the respective ports in the port group. Additionally, vShield Edge provides multicast services, and maps tenant broadcasts into provider multicasts (using PIM). We now have VXLAN backed logical networks, which are elastic (add/delete vNics/ports on an as-needed basis).

With networking constraints out of the way, VDCs can now span cluster, pod and subnet boundaries, removing one of the major limitations in the data center. The concept of elastic VDCs was an important part of the newly released vCloud Director 1.5.

Two very cool options, which highlight the power of the VXLAN construct and logical networking:

• VXLANs can span multiple VDS and/or N1K switches (demonstrated at VMworld, using vCloud Director & vShield Manager to federate across VDSes)
• With VXLAN-VLAN gateways, you can have VMs and physical servers share the same broadcast domain!

Towards secure, elastic hybrid clouds

The elastic VDC constructed above can likewise be instantiated in cloud infrastructure hosted by a VMware Cloud Provider partner. vShield Edge can be used to instantly provision a secure, L3 tunnel to the remote VDC. VMs can now be moved between the local VXLAN and remote VXLAN as needed.

What about securing these VDCs, whether local or  remote? This is where the rest of the vShield portfolio, and our  ecosystem partners come into play. The following is a depiction of the different elements of the solution:

vShield Edge provides perimeter security for the VDC construct. vShield App allows creation of elastic trust zones, whether these be network-centric (break up the logical network into smaller, isolated domains), app-centric (zoning based on Web/App/DB tiers), data centric (zoning based on sensitive data discovery), or identity based (“need to know” access based on Active Directory roles, for example). Finally, vShield Endpoint enables guest VM protection.

It was very gratifying to see Los Alamos National Laboratory (LANL) talk through how they had deployed ALL of the above. Last year LANL was the first government agency to deploy their Infrastructure on Demand (IoD) service leveraging vCloud Director for self-service consumption, and vShield App for micro-segmentation of their VDCs. Leading up to VMworld, they leveraged VXLANs and vShield Edge to extend to a data center hosted at Terremark, building one of the first government hybrid clouds based on the VMware Cloud Infrastructure Suite. Kudos to Anil Karmel (Cloud and Virtualization Architect) and team for standing up these powerful environments.

While vShield Edge, App and Endpoint provide foundational protection and zoning at the perimeter, interior, and VM boundaries respectively, we are working very closely with the networking and security ecosystem (we already have working solutions with Cisco, Trend Micro, and RSA for example, with many more to come) to insert purpose-built functionality at logical boundaries, while seamlessly integrating into the management plane via vShield Manager, which in turn enables these services to be available RESTfully. The combination of logical networking and security, with integrated ecosystem offerings and programmable services, should provide the much needed advancement to support virtualization and cloud needs.

SUMMARY:

We are seeing a massive transformation in the way networking and security is being re-architected in modern virtualization/cloud data centers. Logical networks, edges and zones abstract the underlying infrastructure, and untether higher layers from the need to be infrastructure aware. The virtualization layer serves to overcome the impedance mismatch between the provider and consumer, potentially unleashing a whole new wave of innovation – secure, elastic VDCs become the new currency for private and hybrid clouds.

Let me end with a quote from Jon Oltsik’s note at Network World:

…VMware’s vision is spot on. Networks are cool and all but ultimately they exist to move data and application bits around. VXLAN is a new way to make sure that applications and networks can do this in a more integrated and efficient way.

Thanks Jon! We couldn’t have said it better…

/Allwyn

In my last post, I hinted about the changes happening in the data center, especially with respect to networking and security architectures and deployments. To say that there are transformational changes going on in the industry in this area, is an understatement – the premier networking event Interop goes live this week in Las Vegas, and will showcase some of these trends. One of sessions of interest is hosted by the recently formed Open Networking Foundation, which will also hold an informational session Wednesday, May 11th at 11 – 11:45 to highlight the ONF vision and the future of Software Defined Networking (SDN).

The rampant adoption of server virtualization and consolidation, the emergence of server hosted desktops, along with growing interest in private and hybrid clouds, is highlighting the shortcomings of current networking & security architectures.

Following representation is a simplified view of existing data center networking architectures:

       Virtualized servers are connected to virtual switches (1), which are connected to Top of Rack (ToR) physical switches (2). ToR switches are cabled into the core network (3). Traffic enters/leaves the data center via edge routers (4). Additional core network services like firewalls and IDS/IPS devices are implemented in End of Row (EoR) configurations (5). This results in efficient cabling and good network designs.

Typically, hosts are segregated into VLAN/subnets, and VMs are restricted to deployment within hosts in their respective “silos”. First level security is achieved by hair-pinning traffic out of the VLAN and to the firewall/IPS service nodes.

This architecture worked well when servers were physical/static, with most of the traffic being “North-South” i.e. client-server traffic. With the virtualization of servers, server consolidation is accelerating, and the amount of North-South traffic has exploded. But more challenging to the architecture, is the fact that the new workloads are provisioned/de-provisioned more rapidly, there is more mobility of workloads across the hosts, and there is lot more “East-West” traffic driven by control traffic (e.g. vMotion, DRS, HBR) and access to shared services like storage and backup. When we begin to add notions of multi-tenancy and scale requirements to this new dynamic, fluid NS+EW mesh, the architecture really begins to show its age.

Some of the issues are:

• Host-centric physical and static segmentation based on VLANs/subnets, curtailing the ability for VI admins to have more flexibility in consolidating VMs across hosts.
• VLAN & switch TCAM limits, VLAN sprawl
• Onerous lock-step requirements between VM provisioning and network re-mapping
• Firewall rule explosion and static IP-based rules
• Hair-pinning of firewall rules, L4-7 services, resulting in choke points
• Shared edge services

In summary, the rigidity and static nature of current network architectures stand in the way of the agility, flexibility and dynamic requirements of modern workloads. Network re-mapping becomes an ongoing, onerous task.

A better approach is needed, one which separates the consumption of these network constructs from the underlying physical network. We need to un-tether VMs from the underlying physical network, much as we un-tethered OSes from the server hardware. The approach is in line with comments made in an earlier post.

From a tenant or org or app owner perspective, we need to abstract and simplify the underlying network/security architecture, and present consumable constructs such as logical networks, edges and zones, as shown below.

Specifically, the requirements are simply stated as:

•     VM workloads need to be optimally placed (manually or automagically) across the host cluster, untethered from the underlying network segmentation.

•     Each vApp (logical collection of VMs) is given its own logical network(s); each logical network represents an isolated L2 broadcast domain. “A” above represents this “vApp” scenario.

•     Additionally, each org (or tenant in public clouds) can opt for a logical edge, providing edge security & networking services e.g. firewall, NAT, VPN capabilities, and the ability to route between logical networks. “B” above represents this “VDC” scenario.

•     Furthermore, each tenant can further opt to partition its workspace into Trust Zones, with associated security policies. “C” above captures this scenario. Note such Trust Zones could either mirror virtual abstractions like VDCs, vApps, and PortGroups, or be fungibly abstracted based on identities, sensitive data, or administrative span of control concerns, for example.

In order to realize such a logical representation of networks, edges and zones, we need to work together across the industry (network/security/NIC vendors, virtualization providers, cloud admins) on the provider side of the equation. Let’s touch on the key areas undergoing change:

1. The Virtual Distributed Switch (VDS) needs to provide a homogenized “sea of ports” across the cluster of hosts; these can be grouped into “port groups”. These “port groups” are allocated on demand to each vApp, and presented as “logical networks”. Port Groups are ideally backed by isolated L2 broadcast domains, that span subnets, and operate in a tenant-specific namespace. There is room for innovation in delivering such “multi-tenant, L2 overlays”.
2. The Access tier, typically represented by Top of Rack switches, maps the logical “sea of ports” into the physical network infrastructure. Top of Rack switches are fast evolving to support higher bandwidth, lower latency, greater port density, convergence (FCoE), and ingress port to egress port one-hop routing. We can continue to see tighter linkages between the virtual switches and physical NICs above, and the network fabric below. For example, support for multi-tenancy and programmable elasticity:
   • Multi-tenancy refers to the need to have separate addressing namespaces for each tenant, to avoid MAC/IP/broadcast overlapping.
   • Programmable elasticity refers to the need to control creation of logical networks, and add/delete ports on demand.
3. To be able to meet the demands of such dynamic, fluid virtualized environments, where logical networks are allocated on demand, the network fabric continues to become “Fast, Fat and Flat”
   • Fast meaning the ongoing trend from 1Gbps to 10Gbps and more, to support the increased north-south and east-west traffic explosion.
   • Fat means getting away from Spanning Tree Protocols and moving to multi-pathing which uses inter-device links more efficiently.
   • Flat refers to the emerging trend of moving from 3-tier networks (Core/Aggregation/Access), to 2-tier (Spine/Leaf), or even 1-tier. Driven by low latency & simplified fabrics.
   • Note that the distinction between the Access tier and the Core/Aggregation tier is beginning to blur, so we can ultimately consider items 2 & 3 as a collective network fabric requirement.
4. The WAN Edge tier itself needs to get virtualized to support logical edges, available to each tenant/org on demand. Key drivers are the scale out  (versus scale up) architecture, the ability to have customizable (even self-service eventually) edge services on a pay-as-you-go basis, and the ability to provision such services on demand e.g. edge firewalls, VPNs or Load Balancers. Note that some capabilities e.g. DDOS detection & protection are better left at the physical edge as a first line of defense, and where cross-tenant context is useful.
5. Likewise, current service node architectures in the data center need to get logical. Today, firewalls, IDS/IPS, email spam filters, NAC devices, etc, are implemented as service nodes sitting in a “End of Row” configuration, with traffic steered to such a node via “hair pinning” i.e. traffic is forced to leave the VLAN, and steered towards the service node, where several functions are chained. With increased server consolidation, increased east-west traffic, logical networks, multi-tenancy, etc, such devices become potential choke points, we have firewall rule explosion, and VLAN depletion. There are already examples of such purpose-built services getting virtualized, and logically inserted into the virtual plane on a per-tenant basis.

SUMMARY:

We are entering a new phase of data center networking, driven by the needs of modern virtualized/cloud workloads. We need to transition from an era of static, host-centric, IP-centric, pre-segmented networks, to a modern, efficient programmable network fabric, that provides dynamically allocatable logical abstractions to the new workloads. An era that leverages:

LOGICAL NETWORKS, LOGICAL EDGES, LOGICAL ZONES

Let’s get logical!

/Allwyn