Local Storage and Desktop Virtualization

4

Why are so many people unwilling to challenge traditional buying patterns and instead use critical thinking to solve problems?

First a joke…but one that I see all to often and not just related to this discussion but in all areas of life.

Start with 5 monkeys locked in a cage.

Hang a banana from the roof on a string and place a set of stairs under it.

Before long the monkeys will go to the stairs and start to climb toward the banana.

As soon as the first monkey touches the stairs, hose the other monkeys with cold water.

After a while another monkey makes an attempt with the same result. All the others are sprayed with cold water.

Pretty soon, when another monkey tries to climb the stairs, the other monkeys will try to prevent it.

Now, put away the cold water. Remove one monkey from the cage and replace it with a new one. The new monkey sees the banana and goes to climb the stairs. To his surprise and horror, all of the other monkeys attack him. After another attempt and attack, he knows that if he tries to climb the stairs, he will be assaulted.

Next, remove another of the original five monkeys and replace it with a new one. The newcomer goes to the stairs and is attacked. The previous newcomer takes part in the punishment with enthusiasm!

Likewise, replace a third original monkey with a new one, then a fourth, then the fifth. Every time the newest monkey takes to the stairs, he is attacked.

Most of the monkeys that are beating him have no idea why they were not permitted to climb the stairs or why they are participating in the beating of the newest monkey.

After replacing all the original monkeys, none of the remaining monkeys have ever been sprayed with cold water.

Nevertheless, no monkey ever again approaches the stairs to try for the banana.

Why not?

Because as far as they know that’s the way it’s always been done around here. And that, my friends, is how company policy begins.

This survey that Chris Wolf (Gartner Analyst) did completely blew me away.

I’m shocked at the number of people when asked about their Virtual Desktop Storage Preferences answered that they would use Enterprise Storage array features (EMC, HDS, HP, NetApp). I suppose I shouldn’t be shocked after spending the last few years working on desktop virtualization solutions but it shows how little people still understand about the workload, requirements, and technologies that desktop virtualization has. It is these same people that are probably also shocked when they look at the costs of deploying a hosted virtual desktop solution.

How do you think Exxon would answer if asked whether you should use an electric car vs one with a combustion engine? That they would reply anything other than a combustion engine would be ludicrous (well until they own the electricity too). So it’s no surprise that when you ask a shared storage vendor what storage architecture they suggest what would be best for you is using their shared storage solution. Recognize that the answer they are giving you is because that is what they sell but may not be what is in the best interests of your company of storage solution for desktop virtualization. There is certainly a place for shared storage solutions in Desktop Virtualization, but it really shouldn’t be the defacto standard.

In my experience the solution that most IT departments are trying to move towards is one that involves a non-persistent virtual desktop, I’m a fan of persistent desktops too, but that in my experience has rarely been the first use case companies focus on to maximize their return in desktop virtualization technology investments.

A non-persistent virtual desktop means that changes to the operating system including OS updates, configuration changes, and applications installations are not retained. After a user logs off, the virtual desktop is refreshed and reverted back to its gold image state. Personalization of the operating system or applications that are desired are stored in the user profile can be maintained separately from the virtual desktop and therefore persisted (on shared storage). What a non-persistent virtual desktop really means for most companies is that if a user installs and application (provided they have the rights to do so) that application will not be persisted. In many cases this is the desired functionality that companies are seeking in deploying a hosted virtual desktop solution.

So what then are you getting by adding a shared storage solution? If the server the desktop is hosted on fails the session is lost and the user has to reconnect regardless of the storage architecture. If the storage on the server the desktop is hosted on fails the session is lost and the user reconnects…again regardless of the storage architecture. So why then are so many companies using shared storage for the VDI environment? I honestly don’t know. My best educated guess is because that’s just what they think IT departments should do.

So where are the gaps when using local disk for desktop virtualization? Since we are using local storage technologies such as live migration/vMotion/XenMotion can’t be used (OK, so shared nothing migrations exist, but you probably won’t use this in production…although if you have 10Gb and local SSD then this might be an option) and therefore load distribution by migrating VM’s to balance load across hypervisor hosts can’t be done. Add to that, the broker does not take in to account the load on the host hypervisor(s) when making placement decisions for a user that needs a non-persistent virtual desktop. Since the virtual machines are using local hard disks there isn’t any ability for the underlying hypervisor cluster to migrate virtual machines from one host to another. So the potential exists that you could be brokering users to a virtual desktop on a host that is saturated while other hosts in the cluster have available capacity.

When Citrix XenDesktop and VMware View make brokering decisions why is their no calculation of host utilization (hypervisor) used in that decision? We need a smarter broker in order to free ourselves of shared disk and therefore the bonds of 40% of our desktop virtualization project going to storage manufacturers. Dell/Quest vWorkspace does this, they call this Connection Time Load Balancing. From their admin guide: Connection-time load balancing distributes user connection requests to a managed computer on the least busy Hyper-V hypervisor. Kudo’s guys!

So there, something to think about, don’t do what everyone else does just because everyone else is doing it. I’m a firm believer that most people are stupid, the last thing I want to do is blindly make the same decision that person did. Shared storage might have a place in your desktop virtualization deployment, but it should not be assumed. Don’t be a f*cking monkey. :)

The vSphere CPU Scheduler and VDI part 2

1

Today I saw that an updated version of a VMware whitepaper discussing the CPU scheduler in VMware ESX/vSphere was published, The CPU Scheduler in VMware vSphere 5.1. I have used the previous whitepaper written for ESX 4.1 in a few presentations I’ve given and I frequently reference it when discussing VDI/Server Hosted Virtual Desktop solutions with customers. I wrote a blog post in 2011 discussing some of the key points to understand about the CPU scheduler and VDI.

I thought with this update of the whitepaper it would be a good time to once again focus on items that apply to all virtualized workloads but is critically important to understand when deploying desktop virtualization workloads in which user experience is of the highest importance and you have high over-commitment ratios. From the whitepaper published by VMware I think these paragraphs are the most important for desktop virtualization admins to understand:

When making scheduling decisions, the ratio of the consumed CPU resources to the entitlement is used as the priority of the world. If there is a world that has consumed less than its entitlement, the world is considered high priority and will likely be chosen to run next.

One way to understand prioritizing by the CPU scheduler is to compare it to the CPU scheduling that occurs in UNIX.The key difference between CPU scheduling in UNIX and ESXi involves how a priority is determined. In UNIX, a priority is arbitrarily chosen by the user. If one process is considered more important than others, it is given higher priority. Between two priorities, it is the relative order that matters, not the degree of the difference.

In ESXi, a priority is dynamically re-evaluated based on the consumption and the entitlement. The user controls the entitlement, but the consumption depends on many factors including scheduling, workload behavior, and system load. Also, the degree of the difference between two entitlements dictates how much CPU time should be allocated.

Those sentences are the ones all desktop virtualization administrators should reread until they understand what that means to them. Here’s my translation: “Shares being equal, the more CPU resources you consume (CPU time) the more likely another workload (world) will preempt yours.”

When looking at the workload being done by your users within their hosted virtual desktop it is important to understand the applications and ways in which they will use those apps, the real-time nature of any of the applications being used by them, etc. The main thing I look for are applications that rely on audio or video in order to be effectively used, or is their primary purpose.

These applications are negatively affected when the pCPU is not available and the vCPU must wait for it to become available. When you are overcommitting vCPU to pCPU in ratios like 8 to 1 there is a much higher chance that another vCPU will be waiting for the pCPU. Some interuption and waiting for the pCPU probably won’t be noticed, but if the other 7 vCPU’s are also trying to schedule audio and video you’re going to have serious contention on that pCPU and it will most likely degrade the user experience.

Remember, shares being equal and all vCPU’s having work to be done the CPU scheduler will equally distribute work between the vCPU’s, there is no priority of operating system thread that the vSphere CPU scheduler sees.

Launching Citrix Receiver subscribed apps using OS X Spotlight

2

If you are using a Mac and Citrix Receiver with Citrix CloudGateway Express/Enterprise and have subscribed to applications you will notice that these applications have a file placeholder in the directory ~/Users/username/Applications/

What this does is it allows Spotlight to index the files and then make them available to Spotlight or any other quick launch app that uses the index that Spotlight created. So now I can launch my Citrix applications without first launching Citrix Receiver. The subscribed applications also show up in LaunchPad.

The folder with my subscribed applications:

Citrix Receiver subscribed apps on filesystem

Citrix Receiver showing my subscribed applications:

Citrix Receiver subscribed applications

Below is a video showing me using Spotlight to launch subscribed applications, in this case I’m using it to launch both CloudGateway Enterprise web applications such as MyCitrix and Linkedin as well as published desktops, in this case a physical PC delivered via Citrix RemotePC.

Commonly forgotten upgrade component, the Clients!

2

I work with many customers in the course of my job on Citrix upgrade projects. Common scenario, Presentation Server 4.5 to XenApp 6.5 upgrade. One thing I see often overlooked by the customer in their planning (beside app compatibility with Windows 2008 R2) is the clients that are connecting to this infrastructure. In typical server administrator fashion we focus on the datacenter and the backend infrastructure, you know, the cool shit (insert manly growl).

Well, that fancy upgrade isn’t going to be noticed by your users, except that they website probably looks different and Windows start button is gone, if you don’t take into account an upgrade plan for the client software on the endpoints. The Windows and Mac machines can be tricky, and functionality that used to exist like launching apps from the system tray might be gone, but these are all things that can be overcome with time and training.

The hard ones I’ve found are projects that didn’t take into account that the thin clients connecting to the environment might not support that latest version of Citrix Receiver, even worse they might not support a version of Citrix Online Plug-in that is tested or supported against XenApp 6.5. Now we’re going to have a tough conversation, because it’s no longer something we can overcome with training, email blasts, and time…you’re going to have to spend money.

The Citrix stance on support for client software is non-existent from what I can find, but I remember something about supporting the current version and 1 major version back. In the system requirements for Citrix XenApp 6.5 they mention they tested with Online Plug-in 12.1 but some features are not available. My general suggestion is you need to be thinking about how to get to version 12.x…if you can’t get there you need to think about buying new thin clients, changing your client strategy, or accepting the risk. I’ve seen ICA v7.x clients connect to a XenApp 6.5 farm :)

If you’re using thin clients, they don’t last forever, in fact their useful lifetime is probably not defined by the hardware but rather the software they support (see post on BrianMadden.com). Thin clients add complexity, it’s a world of choices and there is no perfect solution, just don’t forget them in your upgrade budgeting and planning.

Windows 2008 R2 1-to-1 via XenDesktop

2

What many enterprises may not be aware of is that Citrix offers Windows 2008 R2 desktops on a 1-to-1 basis instead of Windows 7 desktops as an option for Citrix Service Providers. This feature was introduced with the Citrix Cloud Provider Pack back in March of 2012 and was featured in a blog post recently by Citrix. So why might an enterprise want to use Windows 2008 R2 instead of Windows 7? Licensing! Brian Madden wrote about this same concept a while back too here and here.

No worrying about Microsoft VDA licensing or companion device licensing in the future and you can buy unlimited copies of the server OS buying by the processor using Windows Datacenter licensing makes licensing much more straightforward. If you are using Windows 2008 R2 desktops on a 1-to-1 basis you do still need to purchase RDS cals for your users or devices connecting to these desktops but RDS licensing is much simpler when compared to VDA licensing.

Feature limitations of interest with this technology:

  • Machine Creation Services (MCS) provisioning is not supported, Provisioning Services (PVS) is
  • Personal vDisks are not supported
  • VM hosted apps not supported
  • Citrix WDDM driver is not used, therefore no Aero remoting

Windows 2008 R2 can be made to look like Windows 7 by turning on the Desktop Experience feature. Here are some resources to help you set this up:

So my only question is why is this delivery model not available to enterprises in addition to the Citrix Service Providers? I think most enterprises don’t yet realize how complex the VDA licensing is and are probably in violation of various components of it and maybe therefore haven’t evaluated delivery technologies purely on their ability to help them avoid VDA licenses.

Use memory for Citrix Provisioning Services cache location

5

With a new generation of server hardware hitting the market I think it’s time to revisit this topic. First some background.

Provisioning Services in conjunction with Citrix XenApp allows us to create non-persistent XenApp worker nodes on demand, all of which are identical because they are streamed from a read-only image. This allows us to use a single operating system (Windows 2008 R2) to provide a desktop or applications to multiple users and at a lower cost than using the 1-to-1 OS-to-User model of VDI. Furthermore these XenApp worker nodes maintain their consistency by rebooting them on a regular basis (daily/weekly) at which time they flush their write cache and upon reboot return to their original gold image state. When using Provisioning Services each XenApp worker node uses a cache location for the write IO that they generate. The location of the write cache when using provisioning services is the key item we’re discussing in this post.

The options are as follows as per Citrix documentation:

The focus of this post is that in servers that are shipping today that easily hold 192GB of RAM (at a reasonable cost) we now have the option to instead of using virtual hard disks attached to each worker node (what I hear people using most often) that we can now use memory as a cache location and therefore offload the write IO to memory instead of disk, thereby decreasing the overall cost of deploying this solution even more and undoubtedly improving responsiveness and performance.

So in a server with 192GB of RAM and 16 physical cores lets see how this breaks down. With 16 cores we’re probably looking at a base configuration of eight 4 vCPU vm’s based on the latest testing from Citrix done here. Let’s take the example of eight 4vCPU vm’s with 16GB of RAM each.

8 VM’s 16GB RAM 128GB RAM Total
8 VM’s 5GB RAM Cache 40GB RAM Total
  168GB RAM Total per host

As you might notice in the Citrix testing that Frank Anderson did was that min memory available was a little more than 50GB, we’re getting tight especially when you factor in the 6% threshold before hypervisors like vSphere will start inflating the balloon driver.

So that’s the math and we’re pretty close to maxing out 192GB of RAM, but with hosts holding even larger capacities this napkin math should hold up even better. Non-persistent Windows 2008 R2 XenApp desktops and apps with no disk IO impact…pretty sweet…so how come nobody I know is doing this?

Citrix receiver for web screenshot

Change default icon for published XenApp desktop

If you are publishing both XenApp desktops and XenDesktop Windows 7 desktops you may want to have the same icon for your XenApp published desktop as the default icon you have for Windows 7 desktops.

If you do follow these steps:

  1. go to your Citrix Delivery Controller and navigate to C:\Program Files\Citrix\Desktop Studio
  2. copy the console.ico file to your XenApp Controller server
  3. login to Citrix AppCenter on the XenApp Controller
  4. select the published desktop
  5. right click on it, select application properties
  6. select shortcut presentation, select change icon, browse for the console.ico file you just copied
  7. select OK

That’s it! Now all of your “Desktops” look the same.

VMware vSphere CBRC removes the need for Citrix Provisioning Services

9

I know this will be a sensitive topic among some Citrix folks…but I personally dislike the product (Citrix Provisioning Services). I dislike it because it’s not the most straightforward product, it’s had its share of challenges (E1000, vmxnet3 for example), and even with n+1 Provisioning Servers I’ve still seen bugs that crash the stream service, targets fail over to the other node, then promptly crash that stream service. There are some scenarios like streamed OS over the LAN for lab, campus, education environments where I can overlook any shortcoming because the product frankly is the best solution on the market for these types of environments…game changer for sure, no question. But in other environments where we use it to manage large XenApp farms and deploy hosted virtual desktops I think we’re starting to see advancements in the hypervisor that will reduce the need for Provisioning Services, that’s not a bad thing, just other technology catching up.

I’ve talked about the VMware vSphere Content-Based Read Cache (CBRC) in a previous post, namely how to enable it on your Machine Creation Services (MCS) provisioned virtual desktops. For those of you who are not aware, CBRC was introduced in VMware vSphere 5 to provide a method for providing a memory based read cache for frequently accessed portions of a vmdk virtual hard disk.  Text from the VMware blogs states:

When enabled for specific VMs, the host hypervisor scans the storage disk blocks to generate digests of the block contents. When these blocks are read into the hypervisor, they are cached in the host based CBRC. Subsequent reads of blocks with the same digest will be served from the in-memory cache directly.  This significantly improves the desktop performance, especially during boot storms or anti-virus scanning storms when a large number of blocks with identical contents are read. 

Now first a few caveats on my title of this post. Today you need and use Provisioning Services for the following:

  • Non-persistent image delivery for Citrix XenApp worker nodes, this is the best way to successfully manage large deployments of Citrix XenApp as you have single image management and non-persistent user nodes, it’s nearly bulletproof and provides a significantly reduced hardware cost model when compared to VDI/hosted virtual desktops
  • Image delivery for Windows XP/7 virtual desktops where you want to use the read cache of Provisioning Services and the Windows operating system to increase scalability and reduce read IOPs. Additionally the single image management, rather than using replica images on each datastore does reduce overall storage requirements.
  • Streamed OS delivery to physical PC’s over the LAN (this is the only future I see for PVS)

The major advantage to Citrix Provisioning Services (PVS) vs Machine Creation Services (MCS) is that PVS is a giant read cache for all of the images it serves up. While there is some storage savings too, it’s usually not a significant enough amount to drastically impact a project cost or complexity.

Now imagine a future world where Citrix can take advantage of the Content-Based Read Cache (CBRC) for Windows XP/7 virtual desktop delivery using MCS. Instead of your Citrix Provisioning Server/OS serving the read requests from memory the VMware vSphere hypervisor does this.

Imagine in that same future world what Citrix has already hinted publicly about (at Synergy) which is the combining of Citrix XenApp IMA architecture into the current XenDesktop FMA architecture, a world where provisioning XenApp worker nodes via MCS might be possible. Again, these images could then take advantage of the hypervisor read cache.

So I ask you…how long do you think Provisioning Services is going to matter for XenApp and XenDesktop hosted delivery? Now all we need is for Citrix to work with VMware to support CBRC with MCS provisioning…and while they are at it adding support for CSV cache on Windows 2012 would be nice too.

Glad to hear your comments on this…I wrote this rather quickly tonight and I’m sure I’ve overlooked an argument or two :)

Reasons to do Desktop Virtualization

3

There seem to be no lack of reasons not to do Desktop Virtualization and plenty of people on twitter can be found daily shooting holes in the solutions that make up this category so I thought I’d be different and talk about the reasons I’ve seen that organizations take on Desktop Virtualization, whether that be hosted virtual desktop (HVD) or hosted shared virtual desktop (RDSH).

  • Driving to work is expensive and time consuming
  • Using a Mac as your primary desktop is fucking awesome
  • 4G isn’t available everywhere, your apps won’t work with limited bandwidth
  • 4G is everywhere, why have apps installed locally
  • Tablets are a shitty laptop replacement, but your CEO wants Windows apps on this iPad so just do it already
  • Virtualization isn’t security, but keeping data in the datacenter doesn’t hurt your security strategy
  • Access to business apps means your workers can work more and from when/where they want
  • Highly managed Windows endpoints outside of your building are highly difficult

End the week on a positive note, post in the comments on your reasons to do Desktop Virtualization.

The CPU scheduler and VDI

3

Something for those of you considering VDI to chew on. First start off with a good read about the vSphere CPU scheduler http://www.vmware.com/resources/techresources/10131 and what you’ll learn is that CPU scheduling on a hypervisor is all about proportional share, not priority as is the case in unix/linux/windows. Quoting here from the whitepaper “when making scheduling decisions, the ratio of the consumed CPU resources to the entitlement is used as the priority of the world“. Basically, all shares equally distributed, the more compute resources a world consumes the more likely another world will preempt it.

So in a VDI world where there is a high ratio of vCPU’s to physical cores the hypervisor does a very good job of fairly distributing compute resources to requesting worlds. What the hypervisor is blind to is the type of workload that is being done and since compute resources are typically oversubscribed, latency sensitive workloads which require also tend to use more CPU than others…things like video and audio, rich multimedia applications tend to struggle as physical cores become oversubscribed. Take a reference made by Andre Leibovici in this article http://myvirtualcloud.net/?p=3371

“A 1vCPU desktop can deliver 720p@25fps without any hardware acceleration. In this scenario it is recommended a maximum of 2 desktops per core for concurrent playbacks at 720p@25fps. Most VDI deployments don’t cater for this kind playback conditions, but if this is the case of your deployment you should follow these guidelines to ensure the best user experience.”

This isn’t a dig against VMware View, Citrix XenDesktop or VDI…this is just the reality of the world of compute oversubscription. I’m not implying we shouldn’t oversubscribe CPU’s either, I doubt all of your employees are watching 720p video all day long. I do however think it is important to understand how and why these things happen. I think some people have an unrealistic view of the expected performance of 100 VM’s on a 12-16 core server.  I’d also contend that the days of a 1 vCPU desktop VM are drawing to a close. Application developers have been told for years now to multithread their apps to harness more of the power of the AMD and Intel cpu’s.

I’ll leave you with this if it helps drive home the point I’m making as it relates to compute oversubscription..

Traditional PC/laptop

Microsoft Remote Desktop Services/TS/XenApp

VDI