Friday, May 21, 2010

Have you ever seen a running database server move LIVE between Data centers across Cities?

Disaster proof Zero down time solutions is the topic of today' blog and i will start it with last week EMC VPLEX announcement where EMC showcased their new solution which enables you to present LUN's accross a 100KM away remote Datacenter while the actual storage is not there but present in local Datacenter. Another capbility of their Metro VPLEX solution is Distributed coherent cache which makes long distance Live migration [Hyper-V] and VMotion [Vsphere] possible [see videos below]. Though EMC claims this is something new, HP CLX and HP SVC provides this functionality of long distance Live migration already. [see videos below]

The key new feature that VPLEX provides is that different nodes across different datacenters can simultaneously read and right Lun on shared bus which means you can enable Cluster shared volume [CSV] feature on those nodes and though the physical nodes are in different geographical location [limited to 100KM as of now] they can share the VHD files placed on a CSV Lun provsioned via EMC VPLEX. Till now this was only possible by using ISCSI based SAN like HP Lefthand, Dell Equalogic etc.

Another benefit of VPLEX is that it runs on a layer above storage so EMC and Non EMC storage array can be aggregated to create new Luns. Ok Fair enough, now you can have the pleasure of watching the videos where running database server move LIVE between Data centers across Cities!

EMC VPLEX - Simple, Scalable vMotion over distance

Hyper-V Long distance Live Migration with HP StorageWorks

The intention of the article was to give you a quick insight of new storage solutions including quick intro to VPLEX which can be leveraged for disaster proof datacenters. Hope you enjoyed the Videos!


Wednesday, May 19, 2010

If you ever thought of getting exposed to Citrix Virtualization solutions & VDI

Many of you might have worked on Vmware and Hyper-V virtualization solutions but if you have not tried Citrix Xen Virtualization stack you are missing some fun especially as i found Citrix Xen as an economical product giving lot of features in free edition as compared to Vmware, which can be leveraged by SMB customers.

You can very well see that it comes with FREE Live migration, Xencenter management, Convert features. [ofcourse you can also use Hyper-V 2008 R2 server with SCVMM workstation edition [to manage 3 Hyper-V hosts only] again free of cost which comes with Live migration and High availability ] Below you can see a quick comparison of Free edition from Vmware ESXi and Citrix Xenserver

Enough of comparison, Citrix is Pioneer in virtual desktop infrastructure [VDI] and the purpose of writing this blog was to share some information on Citrix Xen Desktop Lab which i came across last month and found useful as it gives you easy exposure to Citrix based VDI solution and that too in a virtual self paced lab provided here along with very nice product Introduction documentation here and if you want to see detailed architecture please check here

Citrix XenApp can be used in conjunction with Xen server and Xen desktop and gives you a complete Virtualized environment. Microsoft is collaborating with Citrix and recognizes their HDX technology which provides rich user experience along with plug and play usb and monitor support and at same time Microsoft is also coming up with their RemoteFX technology to be used in VDI solutions. And if this was not sufficient to excite you, recently Xen 4.0 hypervisor was launched with nice features like Fault Tolerance - Xen 4.0 now supports live transactional synchronization of VM states between physical servers as a basic component, enabling administrators to guarantee a high degree of service reliability without requiring additional software solutions. [Pretty similar to Vmware fault tolerance feature in Vsphere] and Memory Enhancements - New algorithms, such as Transcendent Memory [very similar to Vmware balloning but claims to be better than it as it covers its shortcomings] [and in contrast with Microsoft Dynamic memory as mentioned in my previous blog too] have been introduced in Xen 4.0 to enhance the performance and capabilities of the hypervisor memory operations.

A free self paced training covering Citrix XenServer 5.5 and Citrix Essentials for XenServer 5.5 is available here and Citrix Xen Desktop technology overview can be accessed here. Hope you find that Lab and training links interesting and this helped you in learning Citrix Virtualization poducts.


Thursday, May 6, 2010

What is MPIO and Best Practices of MPIO configuration

I got numerous requests to cover Multipath IO [MPIO] on my blog so I decided to write about how to configure and best practices of MPIO configuration. Before we jump on those I will just define MPIO for starters-- MPIO provides the logical facility for routing I/O over redundant hardware paths connecting server to storage. These redundant hardware paths are made up of components such as the cabling, Host Bus Adapters (HBA’s), Switches, and Storage Controllers and possibly even power. MPIO solutions logically manage these redundant connections so that I/O requests can be rerouted in the event that a component along one path fails. MPIO is a Microsoft-provided framework that allows storage providers to develop multipath solutions that contain the hardware specific information needed to optimize connectivity with their storage arrays. These modules are called Device Specific Modules (DSM’s). In windows 2008 and later release OS ships with an integrated MSDSM.sys which can be used if storage arrays support either: Active/Active or Asymmetric logical unit access (ALUA) configuration.

So let’s say I have 2 redundant paths for my lun…if I do not use MPIO software [enable MPIO role in server manager and configure MPIO ] OS will see 2 luns in device manager and disk management.msc though actually there is only one lun being provisioned but from 2 different storage paths. Now if I start writing on these 2 luns it will be a problem for NTFS file system and hence the need for MPIO software. The MPIO software prevents data corruption by ensuring correct handling of the driver associated with a single device that is visible to the operating system through multiple paths. Data corruption is likely to occur because when an operating system believes two separate paths lead to two separate storage volumes, it does not enforce any serialization or prevent any cache conflicts. Consider what would happen if NTFS tries to initialize its journal log twice on a single volume. The vendor based DSM [more functionality as the vendor knows its storage and has more control and features] or MS based DSM provides failover and load balancing capability for these redundant storage paths and make sure that OS only see one path i.e. a pseudo device.

So now when we know why we need MPIO..let’s take the next step..enable MPIO feature—reboot---go to start-administrative tools—MPIO—Discover Multi-Paths tab runs an algorithm for  every device instance that is present on the system and determines if multiple instances actually represent the same LUN (through different paths). For such devices found, their hardware ids are presented for the Admin for MPIO’ing (they’ll get MSDSM support though).----on MPIO devices tab—it shows the hardware ID’s of devices that will currently be managed by MPIO whenever they are present. The decision is based on their hardware id (ie. Vendor+Product string) matching one that is maintained by MPIO in its MPIOSupportedDeviceList (this is something that every DSM specifies in its INF at the time of installation).

Many storage arrays which are active/active and SPC-3 compliant will work using the Microsoft MPIO DSM.  Some storage array partners also provide their own DSMs to use with the Microsoft MPIO architecture. These DSMs may be installed using the DSM Install tab in the MPIO properties. When vendor package is run on Windows Server 2008, the operating system allows the vendor DSM to be installed, but prevents the MPIO binaries already present in Windows Server 2008 from being updated by the package. Thus the end result of running the vendor installation package is to preserve the core MPIO binaries already present in Windows Server 2008, but the 3rd party provided DSM is added. Remember that you do not need to install MSDSM driver.

Very let’s go to device manager and see the changes..

So you see for my local boot device manager it shows scsi device [no multi path enabled for this disk and nor it is being managed by DSM software] however for my other luns that I have assigned to server via multiple paths..i see them as multi path disk device…right click on anyone of them….go to MPIO tab…you will see the DSM this DSM is having handle on disk and i.e. MSDSM.sys….so this is an easy way to identify which DSM is managing the disk, if you have multiple DSM softwares on server especially in case when you have luns being assigned from different storage arrays. You can also see other details including MPIO policy which we will talk later.

PathVerifyEnabled: This flag enables path verification by MPIO on all paths every N seconds (where N depends on the value set in PathVerificationPeriod).

PathVerificationPeriod: This setting is used to indicate the periodicity (in seconds) with which MPIO has been requested to perform path verification. This field is only honored if PathVerifyEnabled is TRUE.

RetryCount: This setting specifies the number of times a failed IO if the DSM determines that a failing request must be retried. This is invoked when DsmInterpretError() returns Retry = TRUE.
The default setting is 3.
PDORemovePeriod: This setting controls the amount of time (in seconds) that the multipath pseudo-LUN will continue to remain in system memory, even after losing all paths to the device. 
So lets see what really happens @ MPIO stack and how device driver stack walks through in discovering, enumerating and grouping the physical devices, device paths into a logical set. (Assuming a scenario where a new device is being presented to the server)

  1. New device arrives.
  1. PnP manager detects the arrival of this device.
  1. MPIO driver stack is notified of this device arrival (it will take further action if it is a supported MPIO device).
  1. MPIO driver stack creates a pseudo device for this physical device.
  1. MPIO driver walks through all the available DSM’s to find out which vendor specific DSM can claim this device. After a DSM claims a device it is associated only with the DSM that claimed it.
  1. The MPIO driver, along with the DSM, makes sure the path to this device is connected, active, and ready for IO.
If a new path for this same device arrives, MPIO then works with the DSM to determine whether this device is the same as any other claimed device. It then groups this physical path for the same device into a logical set called multipath group. 

 MPIO has different failover and load balancing options and these may differ in case of MSDSM and vendor based DSM.

Failover, where no load balancing is performed. The DSM will specify a primary path and a set of standby paths. Primary path is used for processing device requests. If the primary path fails, one of the standby paths will be used.. Any one of the available paths could be used as primary path, and the remaining paths will be used as standby paths.

Note: With an array that supports ALUA, paths will typically be referred to as Active / Optimized and Active Unoptimized rather than only as a Primary path.

Failback is the ability to dedicate I/O to a designated preferred path whenever it is operational. If the preferred path fails, I/O will be directed to an alternate path, but will automatically switch back to the preferred path, with some DSM assistance when it becomes operational again.

Round Robin Load balancing where the DSM will use all available paths for I/O in a balanced, round robin fashion. This is the default policy chosen when the storage controller follows the true Active-Active model and the management application does not explicitly choose a load balancing policy.

Round Robin with subset of paths is load balancing wherein the application will specify a set of paths to be use in Round Robin fashion, and a set of standby paths. The DSM will use paths from primary pool of paths for processing requests as long as at least one of the paths is available. The DSM will use a standby path only when all the primary paths fail. For example, given 4 paths – A, B, C, and D,  A, B, and C are listed as primary paths and D is standby path. The DSM will choose a path from A, B, and C in round robin fashion as long as at least one of them is available. If all three fail, the DSM will start using D, the standby path. If A, B, or C become available, DSM will stop using D and switch to the available paths among A, B, and C.

Dynamic Least Queue Depth Load Balancing where the DSM will route I/O to the path with the least number of outstanding requests.

Weighted Path load balancing where a weight is assigned to each path; the weight indicates the relative priority of a given path. The larger the number the lower the priority. The DSM will choose a path, among the available paths, with least weight.

Not all errors result in failover to a new path. Some errors are temporary and can be recovered using a recovery routine in the DSM; if recovery is successful, MPIO is notified and path validity checked to verify that it can be used again to transmit I/O requests.  When a fatal error occurs, the path is invalidated and a new path is selected. The I/O is resubmitted on this new path without requiring the Application layer to resubmit the data.

Ok so now lets go back to MPIO settings see how a vendor or MS based DSM selects its device list. For a vendor based DSM software this is in the inf file which is used during the installation of the vendor dsm driver and all the supported devices present on the server will be seen in registry too. so for a server where I have luns assigned from multiple different storage arrays..i will have multiple vendor DSM’s installed and as soon as a new storage object arrives… MPIO driver walks through all the available DSM’s to find out which vendor specific DSM can claim this device… …[Microsoft DSM is contacted last in list of DSM providers ]….After a DSM claims a device it is associated only with the DSM that claimed it and problem is that lets say you have 2 DSM driver who will support the device….anyone of those can put a handle on disk and the other one will not even get a chance….and that’s why It is strongly recommended that the hardware ID for a specific disk device should be configured such that it is only associated with one DSM in the services key. This will help ensure that the device is only available to be claimed by the desired DSM, and will help avoid a situation where a disk is not claimed by the desired DSM, when multiple DSM’s have the ability to support a given device. This process may also be used to determine which DSM a disk drive could be associated with for troubleshooting purposes. Warning: Removing hardware ID’s from DSM’s that are not required should be performed by editing the registry keys directly, as removing a device from the MPIO GUI will result in the hardware ID being removed in both registry locations above and prevent the device from being claimed by the desired DSM.

When configuring a disk device to be managed by MPIO for multipath access, the hardware ID for the disk device is required to be present in two different locations in registry in order to be claimed by MPIO and the DSM managing connection to the device. These two locations are:


Last but not the least…the very nice feature of seeing all settings is configuration.log from MPIO-configuration snapshot tab

You can also configure MPIO with iscsi provisoned luns and inbox MSDSM  will take care of the luns if you have not installed any vendor based DSM from your iscsi storage array vendor. you can also use command line tool Mpioclaim for configuring especially when you are on server Core.

May be in future blog I will try to cover more on ISCSI boot capabilities. Hope you find this information useful and yes you can also capture MPIO trace but you will need to open support call with MS PSS to get it analyzed. Please refer MPIO step by step guide for more information and thanks for your time and hope you liked this article.


Saturday, May 1, 2010

Servicing a Windows Server 2008 R2 Hyper V Host based cluster using Virtual Machine Servicing Tool 3.0

Microsoft has just released the beta of Virtual Machine Servicing tool 3.0. which manages the workflow of updating large numbers of virtual machine templates, offline .vhd files, and failover clusters according to their individual needs. To do this, the tool works with Microsoft System Center Virtual Machine Manager (VMM 2008 or VMM 2008 R2) and with the following software update management systems:

• Windows Server Update Services [WSUS 3.0 Service Pack 1 (SP1) or SP2].

– Or –

• Microsoft System Center Configuration Manager 2007 SP1, 2007 R2, or 2007 SP2.

Version 3 of the Virtual Machine Servicing Tool adds features to service virtual machines on a host, virtual machine templates, .vhd files, and Hyper-V failover clusters running on Windows Server 2008 R2. This version also supports virtual machines running 64-bit operating systems.

Servicing Offline Virtual Machines in a VMM Library

Servicing Stopped Virtual Machines on a Host

Servicing Virtual Machine Templates

Servicing Offline VHDs

Servicing a Windows Server 2008 R2 Hyper V Host by Using Live Migration

In this blog we will see the workflow of Servicing a Windows Server 2008 R2 Hyper V Host leveraging Live Migration via virtual machine servicing tool. You can update Windows Server 2008 R2 Hyper-V hosts without any downtime of the virtual machines running on the host. This feature uses the maintenance mode of Virtual Machine Manager and the live migration feature of Hyper-V.

The servicing job for this feature does the following:

1. Selects the first Windows Server 2008 R2 Hyper-V host to service, and then starts maintenance mode on the Hyper-V host. This in turn starts the live migration of all virtual machines running on the host.

2. Waits for the Hyper-V host in maintenance mode to evacuate all the of the high availability virtual machines running on the host to another node of the cluster on the host without any downtime for the running virtual machines.

3. Invokes either Configuration Manager or WSUS to update the host.

4. Moves the original virtual machines back to the original host if you select this option in the servicing job for this feature in the tool.

5. Selects the next Windows Server 2008 R2 Hyper-V node in the cluster to repeat these steps as needed.

To see more on Virtual Machine Servicing tool join its beta here . Hope you find this information useful.