Vmware and the CBT issue
Yes I’m late to the game with this post, but I’m in the process of fixing it on my servers so I’m putting this post together with information I’ve found about the issue.
First what is the CBT issue, basically ESX versions 5 through 6 having an issue when a vdisk is expanded when its bigger than 150GB the cbt for backups can cause data corruption. Here’s the details from Veeam’s KB1940
Possible data corruption in backups after extending disk size past a 128GB boundary on VMWare.
CauseAccording to VMWare KB kb.vmware.com/kb/2090639, CBT may return an incorrect list of VM disk sectors when a software runs QueryChangedDiskAreas(“*”).
This issue occurs when expanding a virtual disk (vmdk) file with Change Block Tracking (CBT) enabled. This may cause CBT to incorrectly calculate the in-use blocks in the vmdk file.
Note: The amount of space the virtual disk is extended is not relevant, the increment of space by which a virtual disk is extended is not relevant. The virtual machine has this issue when its disk is grown past any 128GB boundary in absolute size. The issue is triggered at other sizes which are a power of 2 from 128GB up. For example: 256GB, 512GB, and 1024GB.
SolutionVMware has released patches for ESXi 5.0,5.1, and 5.5 to fix this issue, please see VMware KB article kb.vmware.com/kb/2090639. Their patch is NOT retroactive and CBT must still be reset, their patch will only prevent it from occurring moving forward after applying the Patch/Update.
First I wanted to find out which of my vms have cbt enabled to do this I downloaded and installed vmware powercli for powershell interface into vsphere and the hosts.
To add powercli to an existing powershell prompt you can run these commands
# Adds the base cmdlets
Add-PSSnapin VMware.VimAutomation.Core
# Add the following if you want to do things with Update Manager
Add-PSSnapin VMware.VumAutomation
One you have this, you need to connect to your VSphere Server, if your running your powershell prompt from an account that has access to VSphere run this command
Connect-VIServer Server
Make sure you use your server name in that command.
To get a list of your vms and if they have cbt enabled or not run.
Get-VM | Get-View | Sort Name | Select Name, @{N="ChangeTrackingStatus";E={$_.Config.ChangeTrackingEnabled}} > c:cbt.txt
If you want a list of machines that doesn’t have cbt enabled.
After you patch your ESX host, veeam recommends that you reset the cbt to make sure all your future backups does have corruption. They provide a script to disable cbt, take a snap and then let the backup software create a backup with cbt off.
You can find that script at their kb1940
I’ll be working through this next week and our weekend backup is going to take a lot longer because it won’t be able to use the cbt to speed up the incremental job. I hope this post helps someone else that hasn’t gone through this or knew about it.
These are the sites I’ve used to collect this information
http://kenumemoto.blogspot.com/2013/08/powercli-how-to-find-vms-with-change.html
https://www.vmware.com/support/developer/PowerCLI/PowerCLI51/html/Connect-VIServer.html
http://www.itsupportforum.net/topic/how-to-find-vms-with-change-block-tracking-cbt-enabled/