Local Storage and Desktop Virtualization

4

Why are so many people unwilling to challenge traditional buying patterns and instead use critical thinking to solve problems?

First a joke…but one that I see all to often and not just related to this discussion but in all areas of life.

Start with 5 monkeys locked in a cage.

Hang a banana from the roof on a string and place a set of stairs under it.

Before long the monkeys will go to the stairs and start to climb toward the banana.

As soon as the first monkey touches the stairs, hose the other monkeys with cold water.

After a while another monkey makes an attempt with the same result. All the others are sprayed with cold water.

Pretty soon, when another monkey tries to climb the stairs, the other monkeys will try to prevent it.

Now, put away the cold water. Remove one monkey from the cage and replace it with a new one. The new monkey sees the banana and goes to climb the stairs. To his surprise and horror, all of the other monkeys attack him. After another attempt and attack, he knows that if he tries to climb the stairs, he will be assaulted.

Next, remove another of the original five monkeys and replace it with a new one. The newcomer goes to the stairs and is attacked. The previous newcomer takes part in the punishment with enthusiasm!

Likewise, replace a third original monkey with a new one, then a fourth, then the fifth. Every time the newest monkey takes to the stairs, he is attacked.

Most of the monkeys that are beating him have no idea why they were not permitted to climb the stairs or why they are participating in the beating of the newest monkey.

After replacing all the original monkeys, none of the remaining monkeys have ever been sprayed with cold water.

Nevertheless, no monkey ever again approaches the stairs to try for the banana.

Why not?

Because as far as they know that’s the way it’s always been done around here. And that, my friends, is how company policy begins.

This survey that Chris Wolf (Gartner Analyst) did completely blew me away.

I’m shocked at the number of people when asked about their Virtual Desktop Storage Preferences answered that they would use Enterprise Storage array features (EMC, HDS, HP, NetApp). I suppose I shouldn’t be shocked after spending the last few years working on desktop virtualization solutions but it shows how little people still understand about the workload, requirements, and technologies that desktop virtualization has. It is these same people that are probably also shocked when they look at the costs of deploying a hosted virtual desktop solution.

How do you think Exxon would answer if asked whether you should use an electric car vs one with a combustion engine? That they would reply anything other than a combustion engine would be ludicrous (well until they own the electricity too). So it’s no surprise that when you ask a shared storage vendor what storage architecture they suggest what would be best for you is using their shared storage solution. Recognize that the answer they are giving you is because that is what they sell but may not be what is in the best interests of your company of storage solution for desktop virtualization. There is certainly a place for shared storage solutions in Desktop Virtualization, but it really shouldn’t be the defacto standard.

In my experience the solution that most IT departments are trying to move towards is one that involves a non-persistent virtual desktop, I’m a fan of persistent desktops too, but that in my experience has rarely been the first use case companies focus on to maximize their return in desktop virtualization technology investments.

A non-persistent virtual desktop means that changes to the operating system including OS updates, configuration changes, and applications installations are not retained. After a user logs off, the virtual desktop is refreshed and reverted back to its gold image state. Personalization of the operating system or applications that are desired are stored in the user profile can be maintained separately from the virtual desktop and therefore persisted (on shared storage). What a non-persistent virtual desktop really means for most companies is that if a user installs and application (provided they have the rights to do so) that application will not be persisted. In many cases this is the desired functionality that companies are seeking in deploying a hosted virtual desktop solution.

So what then are you getting by adding a shared storage solution? If the server the desktop is hosted on fails the session is lost and the user has to reconnect regardless of the storage architecture. If the storage on the server the desktop is hosted on fails the session is lost and the user reconnects…again regardless of the storage architecture. So why then are so many companies using shared storage for the VDI environment? I honestly don’t know. My best educated guess is because that’s just what they think IT departments should do.

So where are the gaps when using local disk for desktop virtualization? Since we are using local storage technologies such as live migration/vMotion/XenMotion can’t be used (OK, so shared nothing migrations exist, but you probably won’t use this in production…although if you have 10Gb and local SSD then this might be an option) and therefore load distribution by migrating VM’s to balance load across hypervisor hosts can’t be done. Add to that, the broker does not take in to account the load on the host hypervisor(s) when making placement decisions for a user that needs a non-persistent virtual desktop. Since the virtual machines are using local hard disks there isn’t any ability for the underlying hypervisor cluster to migrate virtual machines from one host to another. So the potential exists that you could be brokering users to a virtual desktop on a host that is saturated while other hosts in the cluster have available capacity.

When Citrix XenDesktop and VMware View make brokering decisions why is their no calculation of host utilization (hypervisor) used in that decision? We need a smarter broker in order to free ourselves of shared disk and therefore the bonds of 40% of our desktop virtualization project going to storage manufacturers. Dell/Quest vWorkspace does this, they call this Connection Time Load Balancing. From their admin guide: Connection-time load balancing distributes user connection requests to a managed computer on the least busy Hyper-V hypervisor. Kudo’s guys!

So there, something to think about, don’t do what everyone else does just because everyone else is doing it. I’m a firm believer that most people are stupid, the last thing I want to do is blindly make the same decision that person did. Shared storage might have a place in your desktop virtualization deployment, but it should not be assumed. Don’t be a f*cking monkey. :)

The vSphere CPU Scheduler and VDI part 2

1

Today I saw that an updated version of a VMware whitepaper discussing the CPU scheduler in VMware ESX/vSphere was published, The CPU Scheduler in VMware vSphere 5.1. I have used the previous whitepaper written for ESX 4.1 in a few presentations I’ve given and I frequently reference it when discussing VDI/Server Hosted Virtual Desktop solutions with customers. I wrote a blog post in 2011 discussing some of the key points to understand about the CPU scheduler and VDI.

I thought with this update of the whitepaper it would be a good time to once again focus on items that apply to all virtualized workloads but is critically important to understand when deploying desktop virtualization workloads in which user experience is of the highest importance and you have high over-commitment ratios. From the whitepaper published by VMware I think these paragraphs are the most important for desktop virtualization admins to understand:

When making scheduling decisions, the ratio of the consumed CPU resources to the entitlement is used as the priority of the world. If there is a world that has consumed less than its entitlement, the world is considered high priority and will likely be chosen to run next.

One way to understand prioritizing by the CPU scheduler is to compare it to the CPU scheduling that occurs in UNIX.The key difference between CPU scheduling in UNIX and ESXi involves how a priority is determined. In UNIX, a priority is arbitrarily chosen by the user. If one process is considered more important than others, it is given higher priority. Between two priorities, it is the relative order that matters, not the degree of the difference.

In ESXi, a priority is dynamically re-evaluated based on the consumption and the entitlement. The user controls the entitlement, but the consumption depends on many factors including scheduling, workload behavior, and system load. Also, the degree of the difference between two entitlements dictates how much CPU time should be allocated.

Those sentences are the ones all desktop virtualization administrators should reread until they understand what that means to them. Here’s my translation: “Shares being equal, the more CPU resources you consume (CPU time) the more likely another workload (world) will preempt yours.”

When looking at the workload being done by your users within their hosted virtual desktop it is important to understand the applications and ways in which they will use those apps, the real-time nature of any of the applications being used by them, etc. The main thing I look for are applications that rely on audio or video in order to be effectively used, or is their primary purpose.

These applications are negatively affected when the pCPU is not available and the vCPU must wait for it to become available. When you are overcommitting vCPU to pCPU in ratios like 8 to 1 there is a much higher chance that another vCPU will be waiting for the pCPU. Some interuption and waiting for the pCPU probably won’t be noticed, but if the other 7 vCPU’s are also trying to schedule audio and video you’re going to have serious contention on that pCPU and it will most likely degrade the user experience.

Remember, shares being equal and all vCPU’s having work to be done the CPU scheduler will equally distribute work between the vCPU’s, there is no priority of operating system thread that the vSphere CPU scheduler sees.

Commonly forgotten upgrade component, the Clients!

2

I work with many customers in the course of my job on Citrix upgrade projects. Common scenario, Presentation Server 4.5 to XenApp 6.5 upgrade. One thing I see often overlooked by the customer in their planning (beside app compatibility with Windows 2008 R2) is the clients that are connecting to this infrastructure. In typical server administrator fashion we focus on the datacenter and the backend infrastructure, you know, the cool shit (insert manly growl).

Well, that fancy upgrade isn’t going to be noticed by your users, except that they website probably looks different and Windows start button is gone, if you don’t take into account an upgrade plan for the client software on the endpoints. The Windows and Mac machines can be tricky, and functionality that used to exist like launching apps from the system tray might be gone, but these are all things that can be overcome with time and training.

The hard ones I’ve found are projects that didn’t take into account that the thin clients connecting to the environment might not support that latest version of Citrix Receiver, even worse they might not support a version of Citrix Online Plug-in that is tested or supported against XenApp 6.5. Now we’re going to have a tough conversation, because it’s no longer something we can overcome with training, email blasts, and time…you’re going to have to spend money.

The Citrix stance on support for client software is non-existent from what I can find, but I remember something about supporting the current version and 1 major version back. In the system requirements for Citrix XenApp 6.5 they mention they tested with Online Plug-in 12.1 but some features are not available. My general suggestion is you need to be thinking about how to get to version 12.x…if you can’t get there you need to think about buying new thin clients, changing your client strategy, or accepting the risk. I’ve seen ICA v7.x clients connect to a XenApp 6.5 farm :)

If you’re using thin clients, they don’t last forever, in fact their useful lifetime is probably not defined by the hardware but rather the software they support (see post on BrianMadden.com). Thin clients add complexity, it’s a world of choices and there is no perfect solution, just don’t forget them in your upgrade budgeting and planning.

Upgrade of vCenter and vSphere 5.1 to 5.1a strangeness

2

I recently installed 2 new vSphere 5.1 hosts and deployed a new instance of vCenter 5.1 in our lab environment. Inevitably shortly thereafter VMware released version 5.1a to address some upgrade issues and to support View 5.1 on top of vSphere 5.1. KB article on this upgrade here.

From VMware Update Manager I was able to quickly and with minimal effort remediate both new vSphere 5.1 hosts to 5.1a. Then on the vCenter Server…after downloading the new release I updated products in this order:

  • Single Sign On
  • Inventory Service
  • vCenter Server
  • Web Client

Everything except the vCenter upgrade went without issue. The vCenter Server upgrade failed during the upgrade of the Profile Driven Storage component. What appears to have happened is during a reboot of the vCenter server the Profile Driven Storage service failed to start because the vCenter Server service was not yet fully started. What happened next was unexpected, when I went looking for the Profile Driven Storage components (after the failed upgrade) in the filesystem it was gone…additionally the service was missing too. I then went to add/remove programs, did a repair of the 5.1 install (using the 5.1 media), the Profile Driven Storage service and files were back and the service was started. I then attempted an upgrade of the vCenter Server to 5.1a and it succeeded. Strange behavior, hope that helps someone.

UPDATE: great post I couldn’t find when I was writing this http://technodrone.blogspot.com/2012/10/going-over-vsphere-510a-release-notes.html

F11 key on a Mac

Perhaps you’ve tried installing vSphere/ESXi via a remote console session from your Mac and have been stuck at the screen where you have to click F11 to accept the licensing agreement to move on.

By default on my 10.8.2 (Mountain Lion) Macbook Air when I use fn + F11 it would show my desktop, not send F11 thru to the console session. In order to change the behavior of F11 you may need to do the following:

Go to System PreferencesMission Control

Change Show Desktop from F11 to some other key, I’ve used F13 in this example.

Now from your remote console session you should be able to use fn + F11.

Why VDI projects fail

3

First, I’m using VDI as a catch all term in this post for all Desktop Virtualization choices…I despise the term VDI, but I’m using it here because it’s the term I hear from customers most.

I wrote a post earlier this year that was quite popular, “How to Fail at VDI”. The first reason I listed was, “Start desktop virtualization not knowing the business problem you’re solving”. Now this list wasn’t necessarily in order of importance or frequency but in this case this is the #1 reason I see VDI projects fail. Failure comes in many ways, stalled projects, cost overruns, or all out failure delivering what users want and need. In my job I’m often brought in early on to discuss VDI or desktop virtualization initiatives with our customers and I’m quite frequently accused of hating on VDI in these meetings.

In order to learn about my customer’s motivations and expectations for VDI I frequently play devil’s advocate in my customer meetings to get to them to express the reasons they want to use and benefits they expect to receive from VDI. The reason for this is simple…many times the customer reasons for wanting to do VDI are bullshit, non-existant, of no substance…and are doomed to probable failure. In some cases what we all learn is that the customer doesn’t need VDI, they just need to virtualize some desktops on a hypervisor and use RDP to connect to them to solve the problem that they have.

So what questions do I usually ask to begin this conversation?

  • What benefits do (you, the business, the user, the financial decision maker) expect to receive from VDI?
  • What problems (you, the business, the user, the financial decision maker) do you expect VDI to solve?
  • How long do you expect this project to take? (pilot, user acceptance testing, production rollout)
  • What project costs have you budgeted for?

Then I just dig in to their answers and ask more questions, not to talk them out of VDI but rather to fully understand the problems that exist in their environment, the ones they’ve already identified VDI might help with and ones they might not even be aware of that could be solved or helped by a VDI solution. We typically talk about strategies other than VDI, whether that be continuing to do what they’ve been doing with a few minor tweaks, or other solutions in the stack like Application or Profile Management (depending on the customer). What hopefully happens is a discussion about all of the options that are available to them, a more complete understanding of the complexities of each solution, and ultimately a joint discussion about what our next steps should be.

Yeah, it doesn’t seem like VDI hating but in a world where vendors and VARs push product on customers it probably throws a few people off.

Recap – VDI projects fail because the there wasn’t a business reason to have a VDI project in the first place, not because there wasn’t one, but because nobody spent time figuring out what problem they were solving before they started.

The “Post-PC” Era is about the “Workspace”

4

I’ve heard a lot of discussion in the last two weeks since VMworld 2012 San Francisco about Horizon Suite, View, Horizon Data, etc… how they’ll be combined or replace each other, which one is the broker, blah blah blah..and what I can’t help but scream to myself is, “You’re missing the point!”. What I believe VMware showed at VMworld with their Horizon Suite is the new “Workspace”. Citrix CloudGateway is this same new “Workspace”. So let me explain a bit.

The Windows desktop (literally the desktop screen) has been our workspace since the days of Windows 95 and it hasn’t changed much in the time since, even with Windows 8 on my laptop the desktop is still my primary screen. The reason that the Windows desktop hasn’t changed much is because it works well for the average user. It’s where we put shortcuts to recently used applications, saved documents we were working on, and put pictures of our family or dream automobile. What has changed is that we now have multiple devices and therefore workspaces. We have smartphones, tablets, laptops, desktops, web browsers, etc…and each of these for many people is made by a different vendor with potentially a different operating system for each. This is the fundamental change that has happened in the industry, we’re no longer Windows desktop only users. Each of these new devices has a desktop/workspace that you use to launch applications, documents, pictures…you get the point, they all have a “desktop”. The post-PC era isn’t one without Windows, it’s an era of Windows, iOS, Android, OSX all at the same time and interchangeably depending on the user.

A Workspace is like your desk. It’s a place to stack up documents, books, to-do lists, bills, etc. It’s the base to which you return to do your job. If VMware and Citrix can replace the Windows desktop with a new desktop or “Workspace” as I’m referring to it, then they can become the base that users require to do their job. Citrix and VMware aren’t stopping here, they realize that the other fundamental change that has happened is people are increasingly using their own devices and so a new category of IT applications has emerged, those meant to manage application and data security on non-company owned devices. The big challenge in my mind isn’t the application and data management, this will come in time. It’s changing people’s behavior, changing them from using the native Workspace on each their devices to using a new Workspace provided by Citrix/VMware.

VMworld 2012 Recap

1

Last week I spent about 100 hours on the ground in San Francisco at VMware VMworld which had I’m told had over 20,000 attendees. Given my focus is on End-User Computing (EUC) I attended almost exclusively sessions from VMware’s End-User Computing track, my only deviation was for my last session I attended on Thursday which was an advanced esxtop session. I attended mostly breakout style sessions but on Wednesday I discovered Group Discussions. The Group Discussions I attended were lightly attended but the content and people there, both presenters and audience, were outstanding. Group Sessions were centered around VMware product architecture, development, and marketing personnel with HEAVY interaction from those attending the session. I made some great connections with key VMware employees in these sessions and in the future these will be my highest priority sessions to attend.

This year was also my first year attending VMware as a VMware vExpert. While at VMworld I attended a vExpert meeting where we talked about some of the goals of the program for the next year and networked and met of vExperts attending VMworld. Also courtesy of the vExpert program I was lucky enough to attend a CTO reception on Wednesday night before the main attendee party. The CTO reception was a great time to again meet with other vExperts and key VMware personnel…and as you can see below, get pictures and meet with people like the CEO and CTO of VMware.

Quite a lot of the value of VMworld for me is connecting with key industry and VMware employees. These connections are invaluable, these are the people who set direction, create product decision, and represent the End-User Computing industry on a worldwide basis.

What some people don’t yet recognize is that VMware is not just a hypervisor company, nor or they just a virtualization company, they clearly believe they will be (or already are) a top 3 software industry giant. They are competing and enabling every area of technology: Servers, Storage, Networking, End-User Computing, IAaS, PAaS…and the list goes on. Inevitably while this growth into every partner market continues they’re going to find themselves working as frenemies with many of their partners of today, there seems to be no sacred space not in the crosshairs of VMware ever expanding product suite(s).

The big takeaway I had from VMworld as it relates to my space (EUC) started during the keynote on Tuesday. I’ve attached a link to that general session here http://www.youtube.com/vmworldtv?x=us-_vmworldtv_5686_155. I highly recommend anyone interested in Desktop Virtualization or End-User Computing to watch this session. VMware covered use cases for their new product Mirage which was part of the Wanova acquisition. This product is a layering solution allowing administrators to manage the desktop, both physical and virtual, in layers (OS, Apps, User Data)…very cool stuff!

What VMware is doing with Horizon Suite is moving the “workspace” from the desktop OS to Horizon Suite. WIndow simply is the execution engine for “legacy” apps that run on Windows. From Horizon Suite you’ll be able to authenticate and access SAaS applications, ThinApp packaged applications, Windows Desktops/Applications, and Data. Those of you who have been following this space probably remember last year when VMware announced Project Octopus which was an internal project to create an enterprise Dropbox-like product. Project Octopus is now called Horizon Data. Also announced last year that received much attention was Project AppBlast which was delivery of a Windows Desktop via only a HTML5 browser. Project AppBlast is not referenced anymore but the technology has been included in the Horizon demo’s they showed. As to the timing and ultimate features Horizon will come to market with and when that day is still is unknown. VMware said Horizon Suite is currently an alpha product and they expect it to enter a beta status in Q4 of 2012.

My expectations of VMworld 2012 were high and they were fully exceeded by what I saw, the people I met, and the community of vExperts that I spent time with. Fantastic job VMware.

VMworld 2012 – Sunday’s sober thoughts

I’ve arrived in San Francisco for VMworld 2012, apparently a day later than many who are already here and attending HOL’s and sightseeing in the city, but I’m here now and I’m excited to spend a week neck deep in technology. I’m aware of a few upcoming announcements that VMware will be making this week and I’m hoping for a few more that might surprise me, especially if they relate to VMware View and EUC. The integration of Wanova into the VMware View product line is of my highest interest right now, physical pc management, great to see that VMware gets it…these things aren’t going away! I’m also interested to hear from customers attending how they will receive some of the changes that VMware is preparing to announce this week.

My schedule for tomorrow is completely biased with only End-User Computing (EUC) sessions, posted below. I’m also doing a quick interview with Colin at TechTarget in the morning before my 11AM session.

Monday:

  • EUC1190 – VMware View 5.1 Reference Architecture
  • EUC2792 – VMware View 5.1 Security Deep Dive
  • EUC2005 – Troubleshooting VMware View: Looking under the Hood
  • EUC1363 – Centralized Management with Local Execution: A Cost-effective Solution for Physical and Virtual Desktops with Mirage

Shortly I plan to go and register and pick up my Blogger pass for the conference (exciting). Tonight I’ll be at the Welcome Reception and then later at vmunderground. I hope to see a few people I know. If you’re looking for me I recently added a beard to my face, check my twitter profile for an updated picture. I’ve also got a GroupMe group going (VMworld Hijinks) if you want to be in the loop as to where myself and a bunch of others are, add yourself or tweet me your phone number and I’ll add you to the list, be warned…it will kill your phone battery fairly quickly after 8PM every night ;)

High Availability – Citrix Machine Creation Services vs Provisioning Services

4

My blog post from last week on Machine Creation Services, Provisioning Services, and vSphere Content-Based Read Cache caused quite a few tweets back and forth between those in the Citrix community and I thought I’d continue our discussion with a tweet I posted last week, “PVS less highly available than MCS“. So what do I mean by that comment? Well let’s take a look at the components required to deliver a highly available non-persistent image via Machine Creation Services (MCS) and Provisioning Services (PVS) in Citrix XenDesktop delivering a hosted virtual desktop.

Provisioning Services

  • 2 Windows Server instances with Citrix Provisioning Services installed using local, SAN, CIFS, or NFS storage (if you are going to use CIFS or NFS make sure you look at the articles I have posted under Virtualization Resources on my blog.
  • 2 XenDesktop delivery controllers using a highly available SQL database (of course you’d also need Web Interface or Storefront but for the purposes of this discussion and just “delivering” the image we’ll not focus on this)
  • Hypervisor and management infrastructure that PVS will call in to in order to power on/power off virtual machines
  • Storage infrastructure for the virtual machines, either local or shared
Diagram of XenDesktop using Provisioning Services
Screenshot of the storage as seen by the vSphere hypervisor

Machine Creation Services

  • 2 XenDesktop delivery controllers using a highly available SQL database (of course you’d also need Web Interface or Storefront but for the purposes of this discussion and just “delivering” the image we’ll not focus on this)
  • Hypervisor and management infrastructure that MCS will call in to in order to clone replica images, create linked clones, power on/off
  • Storage infrastructure for the virtual machines, either local or shared
Diagram of XenDesktop using Machine Creation Services
Screenshot of the storage as seen by the vSphere hypervisor

So perhaps you’ve noticed that Citrix Provisioning Services only adds additional infrastructure requirements in the way of 2 additional Windows Servers that are used to run the PVS components. While that isn’t necessarily a bad thing it does introduce some additional things that can fail that will affect your ability to deliver an image to your hypervisor infrastructure. The Windows Server OS could fail, in which case if properly configured the virtual machines would fail over to the other Provisioning Server. The failure that I’m ultimately far more concerned about though is one that affects the Citrix Streaming Service. It is this service that is responsible for “streaming” the requested image blocks across the network. If this service has a bug and crashes on one Provisioning Server and the virtual desktops fail over to the other Provisioning Server and if that same circumstance occurs which caused the virtual desktops to crash the streaming service on host 1, then it will probably crash the streaming service on host 2…and now you’re dead in the water. It is for this reason that I say Provisioning Services is less highly available than Machine Creation Services.

Machine Creation Services has one less dependent…Provisioning Servers (OS, Application, and most importantly Streaming Services bugs)