Local Storage and Desktop Virtualization

4

Why are so many people unwilling to challenge traditional buying patterns and instead use critical thinking to solve problems?

First a joke…but one that I see all to often and not just related to this discussion but in all areas of life.

Start with 5 monkeys locked in a cage.

Hang a banana from the roof on a string and place a set of stairs under it.

Before long the monkeys will go to the stairs and start to climb toward the banana.

As soon as the first monkey touches the stairs, hose the other monkeys with cold water.

After a while another monkey makes an attempt with the same result. All the others are sprayed with cold water.

Pretty soon, when another monkey tries to climb the stairs, the other monkeys will try to prevent it.

Now, put away the cold water. Remove one monkey from the cage and replace it with a new one. The new monkey sees the banana and goes to climb the stairs. To his surprise and horror, all of the other monkeys attack him. After another attempt and attack, he knows that if he tries to climb the stairs, he will be assaulted.

Next, remove another of the original five monkeys and replace it with a new one. The newcomer goes to the stairs and is attacked. The previous newcomer takes part in the punishment with enthusiasm!

Likewise, replace a third original monkey with a new one, then a fourth, then the fifth. Every time the newest monkey takes to the stairs, he is attacked.

Most of the monkeys that are beating him have no idea why they were not permitted to climb the stairs or why they are participating in the beating of the newest monkey.

After replacing all the original monkeys, none of the remaining monkeys have ever been sprayed with cold water.

Nevertheless, no monkey ever again approaches the stairs to try for the banana.

Why not?

Because as far as they know that’s the way it’s always been done around here. And that, my friends, is how company policy begins.

This survey that Chris Wolf (Gartner Analyst) did completely blew me away.

I’m shocked at the number of people when asked about their Virtual Desktop Storage Preferences answered that they would use Enterprise Storage array features (EMC, HDS, HP, NetApp). I suppose I shouldn’t be shocked after spending the last few years working on desktop virtualization solutions but it shows how little people still understand about the workload, requirements, and technologies that desktop virtualization has. It is these same people that are probably also shocked when they look at the costs of deploying a hosted virtual desktop solution.

How do you think Exxon would answer if asked whether you should use an electric car vs one with a combustion engine? That they would reply anything other than a combustion engine would be ludicrous (well until they own the electricity too). So it’s no surprise that when you ask a shared storage vendor what storage architecture they suggest what would be best for you is using their shared storage solution. Recognize that the answer they are giving you is because that is what they sell but may not be what is in the best interests of your company of storage solution for desktop virtualization. There is certainly a place for shared storage solutions in Desktop Virtualization, but it really shouldn’t be the defacto standard.

In my experience the solution that most IT departments are trying to move towards is one that involves a non-persistent virtual desktop, I’m a fan of persistent desktops too, but that in my experience has rarely been the first use case companies focus on to maximize their return in desktop virtualization technology investments.

A non-persistent virtual desktop means that changes to the operating system including OS updates, configuration changes, and applications installations are not retained. After a user logs off, the virtual desktop is refreshed and reverted back to its gold image state. Personalization of the operating system or applications that are desired are stored in the user profile can be maintained separately from the virtual desktop and therefore persisted (on shared storage). What a non-persistent virtual desktop really means for most companies is that if a user installs and application (provided they have the rights to do so) that application will not be persisted. In many cases this is the desired functionality that companies are seeking in deploying a hosted virtual desktop solution.

So what then are you getting by adding a shared storage solution? If the server the desktop is hosted on fails the session is lost and the user has to reconnect regardless of the storage architecture. If the storage on the server the desktop is hosted on fails the session is lost and the user reconnects…again regardless of the storage architecture. So why then are so many companies using shared storage for the VDI environment? I honestly don’t know. My best educated guess is because that’s just what they think IT departments should do.

So where are the gaps when using local disk for desktop virtualization? Since we are using local storage technologies such as live migration/vMotion/XenMotion can’t be used (OK, so shared nothing migrations exist, but you probably won’t use this in production…although if you have 10Gb and local SSD then this might be an option) and therefore load distribution by migrating VM’s to balance load across hypervisor hosts can’t be done. Add to that, the broker does not take in to account the load on the host hypervisor(s) when making placement decisions for a user that needs a non-persistent virtual desktop. Since the virtual machines are using local hard disks there isn’t any ability for the underlying hypervisor cluster to migrate virtual machines from one host to another. So the potential exists that you could be brokering users to a virtual desktop on a host that is saturated while other hosts in the cluster have available capacity.

When Citrix XenDesktop and VMware View make brokering decisions why is their no calculation of host utilization (hypervisor) used in that decision? We need a smarter broker in order to free ourselves of shared disk and therefore the bonds of 40% of our desktop virtualization project going to storage manufacturers. Dell/Quest vWorkspace does this, they call this Connection Time Load Balancing. From their admin guide: Connection-time load balancing distributes user connection requests to a managed computer on the least busy Hyper-V hypervisor. Kudo’s guys!

So there, something to think about, don’t do what everyone else does just because everyone else is doing it. I’m a firm believer that most people are stupid, the last thing I want to do is blindly make the same decision that person did. Shared storage might have a place in your desktop virtualization deployment, but it should not be assumed. Don’t be a f*cking monkey. :)

The vSphere CPU Scheduler and VDI part 2

1

Today I saw that an updated version of a VMware whitepaper discussing the CPU scheduler in VMware ESX/vSphere was published, The CPU Scheduler in VMware vSphere 5.1. I have used the previous whitepaper written for ESX 4.1 in a few presentations I’ve given and I frequently reference it when discussing VDI/Server Hosted Virtual Desktop solutions with customers. I wrote a blog post in 2011 discussing some of the key points to understand about the CPU scheduler and VDI.

I thought with this update of the whitepaper it would be a good time to once again focus on items that apply to all virtualized workloads but is critically important to understand when deploying desktop virtualization workloads in which user experience is of the highest importance and you have high over-commitment ratios. From the whitepaper published by VMware I think these paragraphs are the most important for desktop virtualization admins to understand:

When making scheduling decisions, the ratio of the consumed CPU resources to the entitlement is used as the priority of the world. If there is a world that has consumed less than its entitlement, the world is considered high priority and will likely be chosen to run next.

One way to understand prioritizing by the CPU scheduler is to compare it to the CPU scheduling that occurs in UNIX.The key difference between CPU scheduling in UNIX and ESXi involves how a priority is determined. In UNIX, a priority is arbitrarily chosen by the user. If one process is considered more important than others, it is given higher priority. Between two priorities, it is the relative order that matters, not the degree of the difference.

In ESXi, a priority is dynamically re-evaluated based on the consumption and the entitlement. The user controls the entitlement, but the consumption depends on many factors including scheduling, workload behavior, and system load. Also, the degree of the difference between two entitlements dictates how much CPU time should be allocated.

Those sentences are the ones all desktop virtualization administrators should reread until they understand what that means to them. Here’s my translation: “Shares being equal, the more CPU resources you consume (CPU time) the more likely another workload (world) will preempt yours.”

When looking at the workload being done by your users within their hosted virtual desktop it is important to understand the applications and ways in which they will use those apps, the real-time nature of any of the applications being used by them, etc. The main thing I look for are applications that rely on audio or video in order to be effectively used, or is their primary purpose.

These applications are negatively affected when the pCPU is not available and the vCPU must wait for it to become available. When you are overcommitting vCPU to pCPU in ratios like 8 to 1 there is a much higher chance that another vCPU will be waiting for the pCPU. Some interuption and waiting for the pCPU probably won’t be noticed, but if the other 7 vCPU’s are also trying to schedule audio and video you’re going to have serious contention on that pCPU and it will most likely degrade the user experience.

Remember, shares being equal and all vCPU’s having work to be done the CPU scheduler will equally distribute work between the vCPU’s, there is no priority of operating system thread that the vSphere CPU scheduler sees.

Launching Citrix Receiver subscribed apps using OS X Spotlight

2

If you are using a Mac and Citrix Receiver with Citrix CloudGateway Express/Enterprise and have subscribed to applications you will notice that these applications have a file placeholder in the directory ~/Users/username/Applications/

What this does is it allows Spotlight to index the files and then make them available to Spotlight or any other quick launch app that uses the index that Spotlight created. So now I can launch my Citrix applications without first launching Citrix Receiver. The subscribed applications also show up in LaunchPad.

The folder with my subscribed applications:

Citrix Receiver subscribed apps on filesystem

Citrix Receiver showing my subscribed applications:

Citrix Receiver subscribed applications

Below is a video showing me using Spotlight to launch subscribed applications, in this case I’m using it to launch both CloudGateway Enterprise web applications such as MyCitrix and Linkedin as well as published desktops, in this case a physical PC delivered via Citrix RemotePC.

Commonly forgotten upgrade component, the Clients!

2

I work with many customers in the course of my job on Citrix upgrade projects. Common scenario, Presentation Server 4.5 to XenApp 6.5 upgrade. One thing I see often overlooked by the customer in their planning (beside app compatibility with Windows 2008 R2) is the clients that are connecting to this infrastructure. In typical server administrator fashion we focus on the datacenter and the backend infrastructure, you know, the cool shit (insert manly growl).

Well, that fancy upgrade isn’t going to be noticed by your users, except that they website probably looks different and Windows start button is gone, if you don’t take into account an upgrade plan for the client software on the endpoints. The Windows and Mac machines can be tricky, and functionality that used to exist like launching apps from the system tray might be gone, but these are all things that can be overcome with time and training.

The hard ones I’ve found are projects that didn’t take into account that the thin clients connecting to the environment might not support that latest version of Citrix Receiver, even worse they might not support a version of Citrix Online Plug-in that is tested or supported against XenApp 6.5. Now we’re going to have a tough conversation, because it’s no longer something we can overcome with training, email blasts, and time…you’re going to have to spend money.

The Citrix stance on support for client software is non-existent from what I can find, but I remember something about supporting the current version and 1 major version back. In the system requirements for Citrix XenApp 6.5 they mention they tested with Online Plug-in 12.1 but some features are not available. My general suggestion is you need to be thinking about how to get to version 12.x…if you can’t get there you need to think about buying new thin clients, changing your client strategy, or accepting the risk. I’ve seen ICA v7.x clients connect to a XenApp 6.5 farm :)

If you’re using thin clients, they don’t last forever, in fact their useful lifetime is probably not defined by the hardware but rather the software they support (see post on BrianMadden.com). Thin clients add complexity, it’s a world of choices and there is no perfect solution, just don’t forget them in your upgrade budgeting and planning.

More articles on Citrix RemotePC

I’m a huge fan of RemotePC, not because it’s a Citrix product (some people think I’m biased…I’m pro-solution!) but because it is an extremely easy to implement technology that enables mobility without a major investment or architecture change to desktop/application delivery tools. I’ve written about this previously on What Would Dan Do. You don’t even need a Microsoft VDA license, which can be a huge barrier to companies implementing VDI as your just accessing your already licensed PC!

If you already have Citrix XenDesktop 5.6 Feature Pack 1 in your environment the only additional things you need to do are configure the RemotePC service via the configuration file and deploy the 5.6 FP1 VDA to a physical desktop in your environment. This gives you business control of who/when/what your users connect to and you can use technologies like SmartAccess to provide further granularity to the virtual channels you allow for your users accessing onsite or remotely based on conditions detected during an endpoint scan. It’s a corporate managed GotoMyPC with much more control for the enterprise market.

RemotePC articles on Citrix’s site:

Windows 2008 R2 1-to-1 via XenDesktop

2

What many enterprises may not be aware of is that Citrix offers Windows 2008 R2 desktops on a 1-to-1 basis instead of Windows 7 desktops as an option for Citrix Service Providers. This feature was introduced with the Citrix Cloud Provider Pack back in March of 2012 and was featured in a blog post recently by Citrix. So why might an enterprise want to use Windows 2008 R2 instead of Windows 7? Licensing! Brian Madden wrote about this same concept a while back too here and here.

No worrying about Microsoft VDA licensing or companion device licensing in the future and you can buy unlimited copies of the server OS buying by the processor using Windows Datacenter licensing makes licensing much more straightforward. If you are using Windows 2008 R2 desktops on a 1-to-1 basis you do still need to purchase RDS cals for your users or devices connecting to these desktops but RDS licensing is much simpler when compared to VDA licensing.

Feature limitations of interest with this technology:

  • Machine Creation Services (MCS) provisioning is not supported, Provisioning Services (PVS) is
  • Personal vDisks are not supported
  • VM hosted apps not supported
  • Citrix WDDM driver is not used, therefore no Aero remoting

Windows 2008 R2 can be made to look like Windows 7 by turning on the Desktop Experience feature. Here are some resources to help you set this up:

So my only question is why is this delivery model not available to enterprises in addition to the Citrix Service Providers? I think most enterprises don’t yet realize how complex the VDA licensing is and are probably in violation of various components of it and maybe therefore haven’t evaluated delivery technologies purely on their ability to help them avoid VDA licenses.

Use memory for Citrix Provisioning Services cache location

5

With a new generation of server hardware hitting the market I think it’s time to revisit this topic. First some background.

Provisioning Services in conjunction with Citrix XenApp allows us to create non-persistent XenApp worker nodes on demand, all of which are identical because they are streamed from a read-only image. This allows us to use a single operating system (Windows 2008 R2) to provide a desktop or applications to multiple users and at a lower cost than using the 1-to-1 OS-to-User model of VDI. Furthermore these XenApp worker nodes maintain their consistency by rebooting them on a regular basis (daily/weekly) at which time they flush their write cache and upon reboot return to their original gold image state. When using Provisioning Services each XenApp worker node uses a cache location for the write IO that they generate. The location of the write cache when using provisioning services is the key item we’re discussing in this post.

The options are as follows as per Citrix documentation:

The focus of this post is that in servers that are shipping today that easily hold 192GB of RAM (at a reasonable cost) we now have the option to instead of using virtual hard disks attached to each worker node (what I hear people using most often) that we can now use memory as a cache location and therefore offload the write IO to memory instead of disk, thereby decreasing the overall cost of deploying this solution even more and undoubtedly improving responsiveness and performance.

So in a server with 192GB of RAM and 16 physical cores lets see how this breaks down. With 16 cores we’re probably looking at a base configuration of eight 4 vCPU vm’s based on the latest testing from Citrix done here. Let’s take the example of eight 4vCPU vm’s with 16GB of RAM each.

8 VM’s 16GB RAM 128GB RAM Total
8 VM’s 5GB RAM Cache 40GB RAM Total
  168GB RAM Total per host

As you might notice in the Citrix testing that Frank Anderson did was that min memory available was a little more than 50GB, we’re getting tight especially when you factor in the 6% threshold before hypervisors like vSphere will start inflating the balloon driver.

So that’s the math and we’re pretty close to maxing out 192GB of RAM, but with hosts holding even larger capacities this napkin math should hold up even better. Non-persistent Windows 2008 R2 XenApp desktops and apps with no disk IO impact…pretty sweet…so how come nobody I know is doing this?

RemotePC solves mobility needs…and you can deploy it tomorrow

2

Citrix RemotePC was released as part of Citrix XenDesktop 5.6 Feature Pack 1. RemotePC is the secure brokering of a physical endpoint (desktop or laptop) that is in your office (typically) via Citrix HDX technology.

Much has been written already by some of my twitter friends:

Think of it as GotoMyPC but with the centralized control over virtual channels (printing, clipboard, local drives, etc), automated provisioning of PC and end users, and the high performance of Citrix HDX. I use Citrix GotoMyPC…and HDX/RemotePC blows it away.

Here is a quick diagram outlining the infrastructure required. In small environments you could combine the Delivery Controller(s) and StoreFront server(s) on the same VM’s to further reduce the number of virtual machines required while still providing high availability.

Funny story…I was talking to a prospect who was interested in allowing their users extended mobility options to their applications and data from personal laptops, tablets, and hotel kiosks. The purpose of the meeting was for me to explain to them how to use “VDI” to provide this type of access. Further questioning revealed their server virtualization infrastructure was non-existent, as in they had absolutely nothing virtualized and use all direct-attached disk. At this point I was not optimistic that they had much chance of accomplishing their goals. However things were looking up when they told me had 100 users requesting this type of access and all of them had physical desktops. RemotePC I exclaimed! I wish I could tell you this prospect was using RemotePC today but to my knowledge they haven’t moved forward on this yet, in my opinion because they are too fixated on a technology (VDI) and not on what they can do immediately to improve the life of their end users. Another barrier for them was that this solution did not provide any BC/DR advantages because if the building power was off, burned down, etc they PC’s would be unavailable. A valid argument…but I still think starting somewhere is better than doing nothing and they’ve got a long road.

Another thing nearly every Citrix XenApp engineer will tell you…”We publish RDP so users can connect to their desktops”. RemotePC! RemotePC! RemotePC!

So you want mobility and you’ve decided that you want to move to a hosted virtual desktop (HVD/VDI) solution so that you can connect to that desktop from anywhere and from any device. Well that’s just great but before you can do that you need to categorize your users, determine which applications they need, determine the server impact of those applications running when they’re sharing a few physical processors (highly overcommitted), buy hardware…and on and on and on….a year later and lots of dollars later you’re ready to roll this solution out. So you can do VDI in a year…or you can deploy RemotePC and broker the user’s applications, data, etc that already works (arguably well enough) tomorrow to any device, anywhere…while still allowing IT to control who, what, and when they can access it. Did I mention that you don’t need a Microsoft VDA license to use it? Boom!

So deploy it already, stop over-thinking it and just do it.

Additional info on RemotePC for your reading enjoyment:

Citrix receiver for web screenshot

Change default icon for published XenApp desktop

If you are publishing both XenApp desktops and XenDesktop Windows 7 desktops you may want to have the same icon for your XenApp published desktop as the default icon you have for Windows 7 desktops.

If you do follow these steps:

  1. go to your Citrix Delivery Controller and navigate to C:\Program Files\Citrix\Desktop Studio
  2. copy the console.ico file to your XenApp Controller server
  3. login to Citrix AppCenter on the XenApp Controller
  4. select the published desktop
  5. right click on it, select application properties
  6. select shortcut presentation, select change icon, browse for the console.ico file you just copied
  7. select OK

That’s it! Now all of your “Desktops” look the same.

High Availability – Citrix Machine Creation Services vs Provisioning Services

4

My blog post from last week on Machine Creation Services, Provisioning Services, and vSphere Content-Based Read Cache caused quite a few tweets back and forth between those in the Citrix community and I thought I’d continue our discussion with a tweet I posted last week, “PVS less highly available than MCS“. So what do I mean by that comment? Well let’s take a look at the components required to deliver a highly available non-persistent image via Machine Creation Services (MCS) and Provisioning Services (PVS) in Citrix XenDesktop delivering a hosted virtual desktop.

Provisioning Services

  • 2 Windows Server instances with Citrix Provisioning Services installed using local, SAN, CIFS, or NFS storage (if you are going to use CIFS or NFS make sure you look at the articles I have posted under Virtualization Resources on my blog.
  • 2 XenDesktop delivery controllers using a highly available SQL database (of course you’d also need Web Interface or Storefront but for the purposes of this discussion and just “delivering” the image we’ll not focus on this)
  • Hypervisor and management infrastructure that PVS will call in to in order to power on/power off virtual machines
  • Storage infrastructure for the virtual machines, either local or shared
Diagram of XenDesktop using Provisioning Services
Screenshot of the storage as seen by the vSphere hypervisor

Machine Creation Services

  • 2 XenDesktop delivery controllers using a highly available SQL database (of course you’d also need Web Interface or Storefront but for the purposes of this discussion and just “delivering” the image we’ll not focus on this)
  • Hypervisor and management infrastructure that MCS will call in to in order to clone replica images, create linked clones, power on/off
  • Storage infrastructure for the virtual machines, either local or shared
Diagram of XenDesktop using Machine Creation Services
Screenshot of the storage as seen by the vSphere hypervisor

So perhaps you’ve noticed that Citrix Provisioning Services only adds additional infrastructure requirements in the way of 2 additional Windows Servers that are used to run the PVS components. While that isn’t necessarily a bad thing it does introduce some additional things that can fail that will affect your ability to deliver an image to your hypervisor infrastructure. The Windows Server OS could fail, in which case if properly configured the virtual machines would fail over to the other Provisioning Server. The failure that I’m ultimately far more concerned about though is one that affects the Citrix Streaming Service. It is this service that is responsible for “streaming” the requested image blocks across the network. If this service has a bug and crashes on one Provisioning Server and the virtual desktops fail over to the other Provisioning Server and if that same circumstance occurs which caused the virtual desktops to crash the streaming service on host 1, then it will probably crash the streaming service on host 2…and now you’re dead in the water. It is for this reason that I say Provisioning Services is less highly available than Machine Creation Services.

Machine Creation Services has one less dependent…Provisioning Servers (OS, Application, and most importantly Streaming Services bugs)