Different implementations for File Server Failover Clustering

I have got this question many times in last few days and also seen a couple of implementations where Hyper-V host clustering was configured alongside File Server failover clustering on same hosts. The intention of this blog is to cover some failover clustering scenarios and reiterating the fact that it is not recommended to install any role in Hyper-V parent partition and if you have configured Hyper-V host clustering to run highly available virtual machines, make sure you do not install any other role in the Hyper-V parent partition and this applies to file server role too. The reason you should not install such a configuration is as you load the hyper-V parent partition by adding more roles, it may affect the performance of the hyper-V host and all the virtual machines running on Host. Check here , here and here where it clearly mentions that Microsoft Hyper-V Server parent partition is specifically for system management.

Second interesting question that comes up in discussion is how to configure File server clustering?

• Host file server failover clustering.
• Configure file server in highly available virtual machine.
• Configure guest file server clustering.
• Configure guest file server clustering on HA Virtual machines running on Hyper-V host clustering.

 I am not going to reinvent the wheel as Failover Clustering program manager Elden has already shared good thoughts on this topic here and shared below.

• Host file server failover clustering.

If host goes down…shares failover to another node ..if host needs to be patched..shares can be failed over to other node. This scenario does not provide you the benefits of virtualization but at same time it is preferable if there is considerably high IO happening on file server.

• Configure file server in highly available virtual machine.

HA VM can live migrate or failover from one node to another…for patching purposes we can leverage live migration for moving file server HA VM from one host to another…however during failover the file server VM will take couple of seconds to come online on another node and then server service and SMB shares will come online. Host based failover clustering also monitors the health of the VMs, so if the virtual OS crashes, hangs, or blue screens, it can be automatically restarted, however the end user would need to wait a little longer while the VM state is loaded and started. While Host Clustering can monitor the state of the virtual OS, it does not monitor the health of applications inside the VMs.

• Configure guest file server clustering.

You set up a highly available file server failover cluster across two virtual machines which are running on Hyper-V hosts. If one of the Hyper-V host goes unexpectedly down the guest file server cluster VM will turn off and shares will failover to other guest file server cluster VM running on other hyper-V host. In this scenario if we have to patch the hyper-v host we have to failover the shares from one VM to another.

• Configure guest file server clustering on HA Virtual machines running on Hyper-V host clustering.

Host+Guest clustering provides both Virtual machine [VM] and application mobility. You will be able to move the guest file server cluster virtual machine from one host to another using live migration without any downtime and cluster service in guest clustering will keep checking the health of server service and SMB shares and will failover the shares from one VM to another when needed. Though you have to be careful while planning Host+Guest clustering and make sure that you configure the anti affinity property for the guest cluster file server workloads so that they don’t preferably exist on same Hyper-V host node. You should also check mark the “auto start” property for HA VM so that when one of the physical host goes down and VM fails over to another node, it starts itself automatically so that file shares become available again. While during the failover as the VM will go down…the cluster service in guest clustering will failover the SMB shares to another HA VM running on another hyper-V host and will reduce the downtime.

 

There are some nice blog posts by failover cluster team and Jose Bareto which are nice reference.

http://blogs.msdn.com/b/clustering/archive/2008/06/21/8628515.aspx

http://blogs.technet.com/b/josebda/archive/2008/06/17/windows-server-2008-hyper-v-failover-clustering-options.aspx
Hope this blog will answer some of the questions in reference to implementation of file server failover clustering and advantages and disadvantages of each implementation solution. Thanks for your time and hope this was a reading pleasure.

GAURAV ANAND

Recorded Dynamic Memory and its Architecture presentation

Here you go with the recorded version of Dynamic Memory and its Architecture presentation and its around 33 MB zip file. Though the Demo Gods were not very happy and I did fluttered during demo when task manager values were not giving me the right results but soon I realized that I should not trust task manager output when dynamic memory is enabled for those virtual machines. The same approach which I employed in demo works fine just after the virtual machine reboot as by then DM has not removed or added much memory. Though I am not very satisfied with demo but still hope this is informational and helps you in understanding dynamic memory.

You can download the Live meeting based recorded presentation from here

GAURAV ANAND

Server 2008 R2 SP1 Dynamic Memory & its Architecture Webcast

I will be presenting a Public community webcast on "Dynamic Memory Feature of Windows Server 2008 R2 Service Pack 1 [Beta]" on 18th August i.e. Wednesday at 12PM. The details of joining the session are mentioned below.

************************************************************************

Hyper-V Dynamic Memory is a memory management enhancement, designed for production use that enables customers to achieve higher consolidation/VM density ratios. In this session we will see how it works and its Architecture.

Session contains Demo of Dynamic Memory, Memory buffer and priority settings, Dynamic memory performance counters, best practices and considerations when implemented for Hyper-V & Hyper-V host clustering.

Gaurav Anand has invited you to present at an online meeting using Live Meeting.

When: Wednesday, August 18, 2010 12:00 PM-1:00 PM (GMT+05:30)

Where: Live Meeting

Join the meeting.
Audio Information
Computer Audio
To use computer audio, you need speakers and microphone, or a headset.
First Time Users:
To save time before the meeting, check your system to make sure it is ready to use Microsoft Office Live Meeting.

Notes :
Troubleshooting
Unable to join the meeting? Follow these steps:

1. Copy this address and paste it into your web browser:
https://www.livemeeting.com/cc/mvp/join

2. Copy and paste the required information:

Meeting ID: NDST7P

Entry Code: Q4>3-"Ct3

Location: https://www.livemeeting.com/cc/mvp
If you still cannot enter the meeting, contact support

Notice:
Microsoft Office Live Meeting can be used to record meetings. By participating in this meeting, you agree that your communications may be monitored or recorded at any time during the meeting.
**********************************************************************
GAURAV ANAND

The factors that influence “How many Cluster Shared Volumes [CSV] in a Cluster & VHD’s per CSV”

Cluster shared volumes provides many benefits over a traditional cluster physical disk resource however implementation and designing of cluster shared volumes [CSV] need significant planning to get the maximum benefits out of it. The quick questions that appear during planning is how many CSV Lun’s spanned across cluster nodes, how many VHD’s per CSV, which VHD’s to club together on same CSV and what is the optimum size of a CSV Lun. By the very nature of the CSV’s, if they are in redirected access mode [whether planned or unplanned], will bring down the performance considerably and we should make sure that SMB IO happens for the minimum possible time. In today’s blog I won’t be answering these questions as there is no “one size fits all” answer but we will touch all the points which help you in figuring out the right size for your IT environment.


1. You may want to place OS and Database/logs VHD’s on separate CSV luns. You may even want to place different databases eg. SQL database VHD’s and Exchange VHD’s on separate CSV luns. You can also use CSV in conjunction with pass through disk if you need as mentioned here.

2. How much IOPS your CSV Lun can handle. You may like to get an approximate IOPS estimate of all the VHD’s you are planning to place on a specific CSV Lun, so that you take an informed decision on number of VHD’s that CSV can handle.

3. You may like to make sure that average queue length and disk latency values of CSV Lun are under permissible range after placing the VHD’s. You may like to check with your SAN vendor leveraging storage performance monitoring tools and plan Raid configuration depending on the capabilities of the Storage.

4. While calculating IOPS we not only need to consider Applications generated IOPS but also the maintenance jobs like antivirus scanning, defrag, backup etc. you may also like to consider your backup strategy and whether your backup vendor uses software provider or hardware provider for VSS. To see why it is recommended to use VSS hardware provider along with CSV and the impact of using software shadow copy provider, check here and here.

5. While deciding the size of the CSV Lun you also need to consider the time chkdsk will take to finish. However in server 2008 R2, NTFS self healing thread and improvements in chkdsk and defrag improves the customer experience.To understand how chkdsk works and how much time it takes to run depends on size of volume, number of files and their size and corruption..check here [chkdsk in server 2008 and above is better than earlier OS but you still cannot predict time it will take]

6. While planning the size of CSV Lun, you may also need to plan for VHD’s snapshots. You may also need to plan how much free space should be left available on CSV after placing the VHD’s.

7. Cluster shared volume performance counters will also help you in planning/sizing along with other performance counters. you can see how much direct Read/Write IO happening from the nodes. In same fashion you can monitor the Metadata/Redirected IO for all the placed VM’s. You may also like to spread your CSV Luns uniformly across all cluster nodes assuming all nodes have same computing resources.

Hope this article helps you in planning your CSV design and enables you to reap the maximum benefits out of Cluster shared volumes and Hyper-V host clustering.

GAURAV ANAND

Blog is based on my Personal understanding of the Technologies mentioned above and information provided is AS IS.

How Dynamic Memory feature of Server 2008 R2 SP1 works

Microsoft first mentioned about Dynamic memory in 2008 PDC conference and it seemed that it would be part of server 2008 R2 but the feature got delayed and came in server 2008 R2 SP1 whose public beta was released just a few days back. Microsoft is planning to ship SP1 in 1st quarter of next year. Dynamic Memory allows for memory on a host machine to be pooled and dynamically distributed to virtual machines as necessary. Memory is dynamically added or removed based on current workloads, and is done so without service interruption. At a high level, Hyper-V Dynamic Memory is a memory management enhancement for Hyper-V designed for production use that enables customers to achieve higher consolidation/VM density ratios. We will enable dynamic memory today and dig deep into what it is and how it differs from VMware implementation of over commit feature.

lets get started by installing SP1,

Once it is done, you will see the build as 7601, Service Pack 1

Ok, however if you are trying to install SP1 on a server core machine then you may have to uninstall Chinese or other language packs as they are not part of the SP1 package and you will have to use Lpksetup.exe as mentioned here

Once you have installed SP1 and rebooted the host machine, you can see the dynamic memory tab in settings of the VM.

However though you can enable Dynamic memory by selecting the radio button but remember that till the time you install the windows 7 SP1 or windows server 2008 R2 SP1 or the Hyper-V integration components for these or earlier operating systems, your VM will only use the startup ram and will not increase dynamically. So lets say you have a windows server 2003 SP2 machine, till the time you do not install Hyper-V integration components [latest] you will not be able to Dynamic memory feature ...your VM will only get startup ram. . In other words, the Dynamic Memory settings for the virtual machine can be configured but they don't do anything—a virtual machine that doesn't have the latest Integration Components can only have a fixed amount of memory assigned to it.

Once that is done and VM's rebooted, you can use Dynamic memory feature and it will look like as below. Key point to remember is that once a virtual machine has been configured to use Dynamic Memory by installing the latest Integration Components on the guest operating system, the virtual machine will no longer work on pre-SP1 hosts and cannot be moved to such hosts. You have to be careful if the VM's are highly available as in that scenario you need to enable dynamic memory on all cluster hosts otherwise on non dynamic memory enabled hosts, VM's will have access to startup ram only.

To understand more about Hyper-V Dynamic memory and before reading ahead, please read this Microsoft whitepaper. [Recommended]

The two new groups of performance counters for monitoring Dynamic Memory are Hyper-V Dynamic Memory Balancer and Hyper-V Dynamic Memory VM. I enabled those and checked the results and at same time i opened Msinfo32 from host to see how much is memory available to host after the memory has been consumed by the VM's. you can very well see from perfmon counters that when we are choosing a buffer we are actually choosing a acceptable memory pressure value for our VM—if my buffer is 80% , pressure value will be around 20%. I also found that Domain Controller VM’s need more start up ram than recommended by Microsoft in above mentioned whitepaper [that’s what I found during my testing]. You may notice that the amount of ram reported by task Manager in the guest operating system does not decrease when a virtual machine uses less ram, so all the procedures/rules for rightly configuring a machine for capturing memory dumps will need to be modified  [http://support.microsoft.com/kb/969028 & http:/support.microsoft.com/kb/254649] as they are dependent on physical ram available.

Now lets try to understand how Dynamic memory works from a 100 FT view.

The host has a parent partition that can be configured to provide resources to guest operating systems executing in the child partitions by using virtualization service providers (VSPs). Broadly, the VSPs can be used to multiplex the interfaces to the hardware resources by way of virtualization service clients (VSCs).  A dynamic memory virtualization service provider (DMVSP) can be used to adjust the amount of memory accessible to a child partition. Broadly, the DMVSP can commit and de-commit memory to partitions using one or more techniques. The DMVSP can be associated with one or more virtualization service clients, namely dynamic memory virtualization service clients (DMVSCs). Broadly, the DMVSCs can provide information to the DMVSP . Each DMVSC can also help commit and de-commit memory from the partition it operates within. The DMVSCs and DMVSP communicate by way of a virtualization bus VMbus. VM worker process can work in conjunction with the a virtualization infrastructure driver (VID) which can allocate memory to a child partition. Each guest operating system includes a memory manager which can allocate memory to applications at their request and free the memory when it is no longer needed by the applications. The memory addresses that memory managers actually manipulate are guest physical addresses (GPAs) allocated to the guest operating systems by the VID . The guest physical address in turn can be backed by system physical addresses (SPAs), e.g., system memory addresses that are managed by the hypervisor . The GPAs and SPAs can be arranged into memory blocks. In operation, when a guest operating system stores data in GPA of block 1, the data may actually be stored in a different SPA such as block 6 on the system [see below].

Memory status for a guest operating system can be obtained, and memory status can identify how performance of the guest is affected by the amount of memory that is available. This can be calculated during the runtime of the guest operating system by, DMVSC. This information can then be sent to the DMVSP. The memory status information can include a series of values which identify level of memory pressure that the guest OS is experiencing. As the guest operating system becomes more stressed, i.e., as the amount of memory required to efficiently execute the current workload increases, the DMVSC can revise the value and communicate this information to the DMVSP. Based on the obtained memory status, an amount of guest physical addresses reported to a memory manager of the guest operating system can be adjusted. DMVSP can adjust the amount of guest physical addresses reported to the memory manager of guest operating system. That is, the DMVSP can adjust the amount of address spaces that are detected by the memory manager and can operate to commit or de-commit memory based on the memory pressure that the guest OS is experiencing, e.g., if guest operating system is stressed, memory can be committed.

The memory manager can be configured to support dynamic addition of memory to a running system. DMVSC can be configured to access a hot-add interface of the memory manager and the DMVSC can send a message to the operating system that describes the hot-added GPAs. The memory manager can then make the new memory available to the guest operating system, drivers, applications. For example, the DMVSC can receive the hot-added memory addresses from the DMVSP after the VID generates the relationships between GPAs and SPAs.

The way Microsoft implementation is different from VMware as Microsoft does not uses something similar like Vsphere host swap file and moreover the guest OS does the paging when it needs nor the hypervisor so guest OS knows very well which pages to swap and which not. Also the buffer option available in Dynamic memory is not available in VMware implementation. To Read more about it please check here

Another point that you need to keep in mind is that you may see zero or negative values of memory buffer in Hyper-V console and this means that VM does not have the needed memory and memory pressure is being faced by the guest OS.

By default when we enable dynamic memory some amount is also reserved for host OS.

I can see a approx difference of 1GB between Available memory [Hyper-V dynamic memory balancer] perfmon counter and host msinfo32 output.

Though the product is yet in beta and RTM version may take another 4-6 months, a lot of things may change in RTM version. Though I will definitely say that Microsoft has done a great job and this was most awaited feature and gives users an opportunity to leverage this feature to better optimize the physical resources and increase the VM density on the host. To see how dynamic memory compliments Failover Clustering check here.  Hope you enjoyed the blog and it gave you some insight into Microsoft Dynamic memory implementation and yes, your valuable time is highly appreciated.


GAURAV ANAND

Blog is based on my Personal understanding of the Technologies mentioned above and information provided is AS IS.

Demo of 2008 R2 SP1 features and cool insight of SCVMM 2011

There is a wave of cloud in same way as there was a wave of virtualization 2 years back. Many big players are coming up with their own cloud solutions and cloud management softwares. Microsoft gave some cool insight about its cloud vision in Microsoft Teched North America convention being held in New Orleans, Lousiana. The most interesting for me was the next version of system center Virtual machine manager i.e. SCVMM 2011. The next version will be cloud capable and you can combine your OS images and server apps [using APP-V] in a profile/template which can be used to deploy services along with new Service Designer feature. Microsoft also announced the availability of 2008 R2 Sp1 beta by month of July. The 2 cool features of sp1 are Dynamic memory and RemoteFX and you can see their video Demos below.

DYNAMIC MEMORY



REMOTE-FX


Hope you enjoyed the Vidoes and for the key note Video of Tech ED 2010 you can visit here but for all these videos you will need updated silverlight as all videos are based on silverlight.

GAURAV ANAND

Have you ever seen a running database server move LIVE between Data centers across Cities?

Disaster proof Zero down time solutions is the topic of today' blog and i will start it with last week EMC VPLEX announcement where EMC showcased their new solution which enables you to present LUN's accross a 100KM away remote Datacenter while the actual storage is not there but present in local Datacenter. Another capbility of their Metro VPLEX solution is Distributed coherent cache which makes long distance Live migration [Hyper-V] and VMotion [Vsphere] possible [see videos below]. Though EMC claims this is something new, HP CLX and HP SVC provides this functionality of long distance Live migration already. [see videos below]

The key new feature that VPLEX provides is that different nodes across different datacenters can simultaneously read and right Lun on shared bus which means you can enable Cluster shared volume [CSV] feature on those nodes and though the physical nodes are in different geographical location [limited to 100KM as of now] they can share the VHD files placed on a CSV Lun provsioned via EMC VPLEX. Till now this was only possible by using ISCSI based SAN like HP Lefthand, Dell Equalogic etc.

Another benefit of VPLEX is that it runs on a layer above storage so EMC and Non EMC storage array can be aggregated to create new Luns. Ok Fair enough, now you can have the pleasure of watching the videos where running database server move LIVE between Data centers across Cities!

EMC VPLEX - Simple, Scalable vMotion over distance


Hyper-V Long distance Live Migration with HP StorageWorks


The intention of the article was to give you a quick insight of new storage solutions including quick intro to VPLEX which can be leveraged for disaster proof datacenters. Hope you enjoyed the Videos!

GAURAV ANAND

If you ever thought of getting exposed to Citrix Virtualization solutions & VDI

Many of you might have worked on Vmware and Hyper-V virtualization solutions but if you have not tried Citrix Xen Virtualization stack you are missing some fun especially as i found Citrix Xen as an economical product giving lot of features in free edition as compared to Vmware, which can be leveraged by SMB customers.

You can very well see that it comes with FREE Live migration, Xencenter management, Convert features. [ofcourse you can also use Hyper-V 2008 R2 server with SCVMM workstation edition [to manage 3 Hyper-V hosts only] again free of cost which comes with Live migration and High availability ] Below you can see a quick comparison of Free edition from Vmware ESXi and Citrix Xenserver

Enough of comparison, Citrix is Pioneer in virtual desktop infrastructure [VDI] and the purpose of writing this blog was to share some information on Citrix Xen Desktop Lab which i came across last month and found useful as it gives you easy exposure to Citrix based VDI solution and that too in a virtual self paced lab provided here along with very nice product Introduction documentation here and if you want to see detailed architecture please check here

Citrix XenApp can be used in conjunction with Xen server and Xen desktop and gives you a complete Virtualized environment. Microsoft is collaborating with Citrix and recognizes their HDX technology which provides rich user experience along with plug and play usb and monitor support and at same time Microsoft is also coming up with their RemoteFX technology to be used in VDI solutions. And if this was not sufficient to excite you, recently Xen 4.0 hypervisor was launched with nice features like Fault Tolerance - Xen 4.0 now supports live transactional synchronization of VM states between physical servers as a basic component, enabling administrators to guarantee a high degree of service reliability without requiring additional software solutions. [Pretty similar to Vmware fault tolerance feature in Vsphere] and Memory Enhancements - New algorithms, such as Transcendent Memory [very similar to Vmware balloning but claims to be better than it as it covers its shortcomings] [and in contrast with Microsoft Dynamic memory as mentioned in my previous blog too] have been introduced in Xen 4.0 to enhance the performance and capabilities of the hypervisor memory operations.

A free self paced training covering Citrix XenServer 5.5 and Citrix Essentials for XenServer 5.5 is available here and Citrix Xen Desktop technology overview can be accessed here. Hope you find that Lab and training links interesting and this helped you in learning Citrix Virtualization poducts.

GAURAV ANAND

What is MPIO and Best Practices of MPIO configuration

I got numerous requests to cover Multipath IO [MPIO] on my blog so I decided to write about how to configure and best practices of MPIO configuration. Before we jump on those I will just define MPIO for starters-- MPIO provides the logical facility for routing I/O over redundant hardware paths connecting server to storage. These redundant hardware paths are made up of components such as the cabling, Host Bus Adapters (HBA’s), Switches, and Storage Controllers and possibly even power. MPIO solutions logically manage these redundant connections so that I/O requests can be rerouted in the event that a component along one path fails. MPIO is a Microsoft-provided framework that allows storage providers to develop multipath solutions that contain the hardware specific information needed to optimize connectivity with their storage arrays. These modules are called Device Specific Modules (DSM’s). In windows 2008 and later release OS ships with an integrated MSDSM.sys which can be used if storage arrays support either: Active/Active or Asymmetric logical unit access (ALUA) configuration.

So let’s say I have 2 redundant paths for my lun…if I do not use MPIO software [enable MPIO role in server manager and configure MPIO ] OS will see 2 luns in device manager and disk management.msc though actually there is only one lun being provisioned but from 2 different storage paths. Now if I start writing on these 2 luns it will be a problem for NTFS file system and hence the need for MPIO software. The MPIO software prevents data corruption by ensuring correct handling of the driver associated with a single device that is visible to the operating system through multiple paths. Data corruption is likely to occur because when an operating system believes two separate paths lead to two separate storage volumes, it does not enforce any serialization or prevent any cache conflicts. Consider what would happen if NTFS tries to initialize its journal log twice on a single volume. The vendor based DSM [more functionality as the vendor knows its storage and has more control and features] or MS based DSM provides failover and load balancing capability for these redundant storage paths and make sure that OS only see one path i.e. a pseudo device.

So now when we know why we need MPIO..let’s take the next step..enable MPIO feature—reboot---go to start-administrative tools—MPIO—Discover Multi-Paths tab runs an algorithm for  every device instance that is present on the system and determines if multiple instances actually represent the same LUN (through different paths). For such devices found, their hardware ids are presented for the Admin for MPIO’ing (they’ll get MSDSM support though).----on MPIO devices tab—it shows the hardware ID’s of devices that will currently be managed by MPIO whenever they are present. The decision is based on their hardware id (ie. Vendor+Product string) matching one that is maintained by MPIO in its MPIOSupportedDeviceList (this is something that every DSM specifies in its INF at the time of installation).

Many storage arrays which are active/active and SPC-3 compliant will work using the Microsoft MPIO DSM.  Some storage array partners also provide their own DSMs to use with the Microsoft MPIO architecture. These DSMs may be installed using the DSM Install tab in the MPIO properties. When vendor package is run on Windows Server 2008, the operating system allows the vendor DSM to be installed, but prevents the MPIO binaries already present in Windows Server 2008 from being updated by the package. Thus the end result of running the vendor installation package is to preserve the core MPIO binaries already present in Windows Server 2008, but the 3^rd party provided DSM is added. Remember that you do not need to install MSDSM driver.

Very cool..so let’s go to device manager and see the changes..

So you see for my local boot device..in device manager it shows scsi device [no multi path enabled for this disk and nor it is being managed by DSM software] however for my other luns that I have assigned to server via multiple paths..i see them as multi path disk device…right click on anyone of them….go to MPIO tab…you will see the DSM name..so this DSM is having handle on disk and i.e. MSDSM.sys….so this is an easy way to identify which DSM is managing the disk, if you have multiple DSM softwares on server especially in case when you have luns being assigned from different storage arrays. You can also see other details including MPIO policy which we will talk later.

PathVerifyEnabled: This flag enables path verification by MPIO on all paths every N seconds (where N depends on the value set in PathVerificationPeriod).

PathVerificationPeriod: This setting is used to indicate the periodicity (in seconds) with which MPIO has been requested to perform path verification. This field is only honored if PathVerifyEnabled is TRUE.

RetryCount: This setting specifies the number of times a failed IO if the DSM determines that a failing request must be retried. This is invoked when DsmInterpretError() returns Retry = TRUE.
The default setting is 3.

PDORemovePeriod: This setting controls the amount of time (in seconds) that the multipath pseudo-LUN will continue to remain in system memory, even after losing all paths to the device. 

So lets see what really happens @ MPIO stack and how device driver stack walks through in discovering, enumerating and grouping the physical devices, device paths into a logical set. (Assuming a scenario where a new device is being presented to the server)

1. New device arrives.

2. PnP manager detects the arrival of this device.

3. MPIO driver stack is notified of this device arrival (it will take further action if it is a supported MPIO device).

4. MPIO driver stack creates a pseudo device for this physical device.

5. MPIO driver walks through all the available DSM’s to find out which vendor specific DSM can claim this device. After a DSM claims a device it is associated only with the DSM that claimed it.

6. The MPIO driver, along with the DSM, makes sure the path to this device is connected, active, and ready for IO.

If a new path for this same device arrives, MPIO then works with the DSM to determine whether this device is the same as any other claimed device. It then groups this physical path for the same device into a logical set called multipath group. 

 MPIO has different failover and load balancing options and these may differ in case of MSDSM and vendor based DSM.

Failover, where no load balancing is performed. The DSM will specify a primary path and a set of standby paths. Primary path is used for processing device requests. If the primary path fails, one of the standby paths will be used.. Any one of the available paths could be used as primary path, and the remaining paths will be used as standby paths.

Note: With an array that supports ALUA, paths will typically be referred to as Active / Optimized and Active Unoptimized rather than only as a Primary path.

Failback is the ability to dedicate I/O to a designated preferred path whenever it is operational. If the preferred path fails, I/O will be directed to an alternate path, but will automatically switch back to the preferred path, with some DSM assistance when it becomes operational again.

Round Robin Load balancing where the DSM will use all available paths for I/O in a balanced, round robin fashion. This is the default policy chosen when the storage controller follows the true Active-Active model and the management application does not explicitly choose a load balancing policy.

Round Robin with subset of paths is load balancing wherein the application will specify a set of paths to be use in Round Robin fashion, and a set of standby paths. The DSM will use paths from primary pool of paths for processing requests as long as at least one of the paths is available. The DSM will use a standby path only when all the primary paths fail. For example, given 4 paths – A, B, C, and D,  A, B, and C are listed as primary paths and D is standby path. The DSM will choose a path from A, B, and C in round robin fashion as long as at least one of them is available. If all three fail, the DSM will start using D, the standby path. If A, B, or C become available, DSM will stop using D and switch to the available paths among A, B, and C.

Dynamic Least Queue Depth Load Balancing where the DSM will route I/O to the path with the least number of outstanding requests.

Weighted Path load balancing where a weight is assigned to each path; the weight indicates the relative priority of a given path. The larger the number the lower the priority. The DSM will choose a path, among the available paths, with least weight.

Not all errors result in failover to a new path. Some errors are temporary and can be recovered using a recovery routine in the DSM; if recovery is successful, MPIO is notified and path validity checked to verify that it can be used again to transmit I/O requests.  When a fatal error occurs, the path is invalidated and a new path is selected. The I/O is resubmitted on this new path without requiring the Application layer to resubmit the data.

Ok so now lets go back to MPIO settings see how a vendor or MS based DSM selects its device list. For a vendor based DSM software this is in the inf file which is used during the installation of the vendor dsm driver and all the supported devices present on the server will be seen in registry too. so for a server where I have luns assigned from multiple different storage arrays..i will have multiple vendor DSM’s installed and as soon as a new storage object arrives… MPIO driver walks through all the available DSM’s to find out which vendor specific DSM can claim this device… …[Microsoft DSM is contacted last in list of DSM providers ]….After a DSM claims a device it is associated only with the DSM that claimed it and problem is that lets say you have 2 DSM driver who will support the device….anyone of those can put a handle on disk and the other one will not even get a chance….and that’s why It is strongly recommended that the hardware ID for a specific disk device should be configured such that it is only associated with one DSM in the services key. This will help ensure that the device is only available to be claimed by the desired DSM, and will help avoid a situation where a disk is not claimed by the desired DSM, when multiple DSM’s have the ability to support a given device. This process may also be used to determine which DSM a disk drive could be associated with for troubleshooting purposes. Warning: Removing hardware ID’s from DSM’s that are not required should be performed by editing the registry keys directly, as removing a device from the MPIO GUI will result in the hardware ID being removed in both registry locations above and prevent the device from being claimed by the desired DSM.

When configuring a disk device to be managed by MPIO for multipath access, the hardware ID for the disk device is required to be present in two different locations in registry in order to be claimed by MPIO and the DSM managing connection to the device. These two locations are:

HKLM\System\CurrentControlSet\Control\MPDEV\MPIOSupportedDeviceList

AND

HKLM\System\CurrentControlSet\Services\\Parameters\DsmSupportedDeviceList

Last but not the least…the very nice feature of seeing all settings is configuration.log from MPIO-configuration snapshot tab

You can also configure MPIO with iscsi provisoned luns and inbox MSDSM  will take care of the luns if you have not installed any vendor based DSM from your iscsi storage array vendor. you can also use command line tool Mpioclaim for configuring especially when you are on server Core.

May be in future blog I will try to cover more on ISCSI boot capabilities. Hope you find this information useful and yes you can also capture MPIO trace but you will need to open support call with MS PSS to get it analyzed. Please refer MPIO step by step guide for more information and thanks for your time and hope you liked this article.

GAURAV ANAND

Servicing a Windows Server 2008 R2 Hyper V Host based cluster using Virtual Machine Servicing Tool 3.0

Microsoft has just released the beta of Virtual Machine Servicing tool 3.0. which manages the workflow of updating large numbers of virtual machine templates, offline .vhd files, and failover clusters according to their individual needs. To do this, the tool works with Microsoft System Center Virtual Machine Manager (VMM 2008 or VMM 2008 R2) and with the following software update management systems:

• Windows Server Update Services [WSUS 3.0 Service Pack 1 (SP1) or SP2].

– Or –

• Microsoft System Center Configuration Manager 2007 SP1, 2007 R2, or 2007 SP2.

Version 3 of the Virtual Machine Servicing Tool adds features to service virtual machines on a host, virtual machine templates, .vhd files, and Hyper-V failover clusters running on Windows Server 2008 R2. This version also supports virtual machines running 64-bit operating systems.

Servicing Offline Virtual Machines in a VMM Library

Servicing Stopped Virtual Machines on a Host

Servicing Virtual Machine Templates

Servicing Offline VHDs

Servicing a Windows Server 2008 R2 Hyper V Host by Using Live Migration

In this blog we will see the workflow of Servicing a Windows Server 2008 R2 Hyper V Host leveraging Live Migration via virtual machine servicing tool. You can update Windows Server 2008 R2 Hyper-V hosts without any downtime of the virtual machines running on the host. This feature uses the maintenance mode of Virtual Machine Manager and the live migration feature of Hyper-V.

The servicing job for this feature does the following:

1. Selects the first Windows Server 2008 R2 Hyper-V host to service, and then starts maintenance mode on the Hyper-V host. This in turn starts the live migration of all virtual machines running on the host.

2. Waits for the Hyper-V host in maintenance mode to evacuate all the of the high availability virtual machines running on the host to another node of the cluster on the host without any downtime for the running virtual machines.

3. Invokes either Configuration Manager or WSUS to update the host.

4. Moves the original virtual machines back to the original host if you select this option in the servicing job for this feature in the tool.

5. Selects the next Windows Server 2008 R2 Hyper-V node in the cluster to repeat these steps as needed.

To see more on Virtual Machine Servicing tool join its beta here . Hope you find this information useful.

GAURAV ANAND

How HP Insight Control suite for Microsoft System Center integrates with Microsoft System Center Suite

Customers who are using Microsoft System center suite which comprises mainly of four products i.e. System Center Configuration manager, System Center Operations manager, System Center Virtual machine manager and System Center Data protection manager can now leverage Hp insight control suite for system center to to deploy, monitor and control their server infrastructure from almost anywhere. With this first release of the HP Insight Control suite for System Center, HP delivers Operating System Deployment (OSD) capability, Performance and Resource Optimization (PRO), enhanced inventory, the HP Server Updates Catalog, an ILO Advanced license, licenses for HP Insight Power Manager and Performance Management Pack, and HP specific management packs for System Center Operations Manager to optimize deployment, virtualization and configuration of HP servers being managed by Microsoft System Center. However HP ICE for System center does not integrates or adds any functionality for system center Data protection manager.

HP ICE for System center provides proliant server deployment functionality which enables system center configuration manager [SCCM] administrator to configure proliant server bios settings, bios boot order and ILO configuration from the SCCM console using a task sequence and then same task sequence can be advertised with OS image and settings resulting into deployment of bare proliant server using a truly zero touch deployment method. That means you purchase the proliant box and place it in the datacenter and rest everything can be managed via system center consoles. HP ICE for System center adds HP specific capabilities for deployment of proliant servers.

Along with operating system image deployment you can also integrate driver packages or proliant support pack along with other 3rd party software packages or Microsoft software like Exchange 2007 or different operating system roles like DHCP/DNS etc. so that once server is deploys it’s up to date with all patches and software requirements. Within SCCM console you can run inventory reports which will capture hardware inventory from proliant servers.

You can directly manage and control your proliant and blade servers using system center operations manager by installing proliant server management pack and blade system management pack respectively. However at this point of time management of HP integrity servers and VMware host servers from SCOM is not supported.

Hope you find this information useful and check here to learn more on Insight control for Microsoft system center.
 
 
 
 
 
 
 
 
 

GAURAV ANAND

How cluster shared volumes Direct and Redirected IO works

Cluster shared volume [CSV] is a great feature offered with 2008 R2 failover clustering. Cluster shared volumes allows the different nodes of cluster to have concurrent access to the LUN where highly available virtual machine's VHD is stored. So in case of Live migration/quick migration or manual move/failover operation the LUN (which has been configured as a physical disk resource in cluster) is not dismounted and mounted which was the case till 2008 server. In server 2008 R2 both the nodes have simultaneous access to the LUN and both the nodes can read and write on the LUN where multiple VHD's of the highly available virtual machines are placed. This allows multiple VHD per LUN and removes the traditional one VHD per LUN issue. Earlier clustered virtual machines can only fail over independently if each virtual machine has its own LUN, which makes the management of LUNs and clustered virtual machines more difficult. Now that limitation is gone with CSV. From the last blog we know how persistent reservation works for cluster shared volumes and we know how both nodes are able to read and write at same time on the CSV disk resource. In this blog we will see how it works on file system level. But before we do that lets have a quick recap of cluster shared volumes below and here

To understand Direct IO, we will take a 2 node cluster scenario. Node A is the coordinator node as shown in the fig 1 and holds the CSV volume containing VHD files of VM 1 and VM 2.

                                                             FIG 1

Node A is directing read write IO and meta data IO [file renames, attribute changes, new file creation etc ] directly as shown by red and yellow lines simultaneously. At the same time I collected a perfmon and enabled new counters for cluster shared volumes. We can see in fig 1 that we have Read write IO along with metadata IO however as both the VM’s are running on coordinator node we do not pass any IO request to mini SMB redirector and none of the IO flows on network. Once you move the VM2 on node B the metadata IO goes via network [see fig 2]. CSV redirector passes over request to SMB mini director which passes the request from client side to server side and IO request flows to CSV redirector on coordinator i.e. node A. CSV redirector passes the IO to ntfs.sys and passes the information back to Node B via same channel. And this is the main reason why it is asked to enable SMB protocol on the CSV network.

                                                                  FIG 2

Cluster disk manger [DCM] is responsible for managing the CSV resources and manages the read write policy set on non coordinator nodes for access to CSV disk resources. You can look at cluster.log below and see that [DCM] is responsible for starting csvfilter.sys and creating CSV disk resources.

From cluster.log

01255 00000724.00000388::2009/10/08-18:01:35.592 INFO [DCM] Cluster Shared Volume Root is C:\ClusterStorage
01257 00000724.00000388::2009/10/08-18:01:35.607 INFO [DCM] service/driver CSVFilter started
01258 00000724.00000388::2009/10/08-18:01:35.607 INFO [DCM] short name is C:\CLUSTE~1
01259 00000724.00000388::2009/10/08-18:01:35.607 INFO [DCM] Filter.CfsSetRootFolder (P.id=1828, RootFolder=\ClusterStorage\)
01260 00000724.00000388::2009/10/08-18:01:35.607 INFO [DCM] SetRoot message sent
01261 00000724.00000388::2009/10/08-18:01:35.607 INFO [DCM] Pnp CfsFilter Launching Filter Listener
01269 00000724.00000388::2009/10/08-18:01:35.607 INFO [DCM] db.CreateDcmDisk 'SR' 8e62ea00-9763-4c21-86d1-4a05708be24a-----SR is name of my CSV disk resource
02000 00000724.00000928::2009/10/08-18:01:41.957 INFO [DCM] FsFilterCanUseDirectIO is called for file:///?\Volume{f98a57c7-9ec6-11de-ad77-806e6f6e6963}\
02001 00000724.00000928::2009/10/08-18:01:41.957 INFO [DCM] PostOnline. CanUseDirectIO for Volume1 => true
02008 00000724.00000928::2009/10/08-18:01:41.957 INFO [DCM] ClearVolumeStates: resource 'SR' states
02011 00000724.00000928::2009/10/08-18:01:41.957 INFO [DCM] Reservation.SetMembership(SR,(1 2))
02019 00000724.000008b4::2009/10/08-18:01:41.972 INFO [DCM] volume 'Volume1' is already paused
02020 00000724.000008b4::2009/10/08-18:01:41.972 INFO [DCM] CreateLink C:\ClusterStorage\Volume1 => \\?\Volume{f98a57c7-9ec6-11de-ad77-806e6f6e6963}\

The interesting part is that when a CSV resource is created it is put in its own group which has a unique GUID instead of name. when you run command cluster.exe group command these CSV GUID based group will not be displayed and the reason is Microsoft does not want users to mess with these. Also there are lot of limitations with these groups like you cannot add any resources in these and many more. However running a “cluster.exe res” will show you CSV group’s GUID name.

 
Now as we know that if you are accessing CSV disk resource in a “Redirected access” mode you are bound to get a performance impact as all the read/write and metadata IO is flowing over the network from all non coordinator nodes instead of direct IO to storage object. You can calculate the performance impact by capturing a baseline perfmon during direct IO and redirected IO.

 

In Redirected IO, we leverage SMB mini director and Csvfilter.sys and send all the read/write IO along with NTFS metadata over on network instead of storage path. CSV helps in increased fault tolerance as even if storage path of lun fails from a particular node, the Node will keep accessing the Lun via coordinator node and re direct all IO on network as seen above [though with performance constraints] . We will see more details in next part of this series on cluster shared volumes. We will also see what are options and best practices regarding backup of Vhd files on cluster shared volumes. Hope you find this information interesting and liked today's blog. Thanks for your time and good bye till next blog.