Lab Manager 4.0.3 Updated Best Practices & Design Considerations

Lab Manager is one of my favorite technologies in the market today, but before you install, beware of the limitations!

NEW IN 4.0.3 :

Windows 2008 (32-bit) support for LBM Server installation

Support for ESX & vSphere 4.1U1 & 4.0U3

End of Life for LBM

Since LBM 4 has announced and End of Life in May 1st 2013, this is the final major version of Lab Manager to be replaced to vCD (VMware Cloud Director), hence this will be my last best practices guide for it. All future guides from me will be about vCD, but as of today vCD 1.0 is primarily designed as a public cloud offering built around Multitenancy, LBM is a private cloud Software Dev/Test product, we will need the next version(s) of vCD to return many of the LBM test/dev features (which I’m assured they will). I will need those features before I make the jump in a test/dev environment to vCD from LBM.

i. 8 ESX hosts max to connect to VMFS3 Datastore (each LUN), you can use NFS to get around this, but for our use case, this is not performant enough. The VMFS limit has to do with Linked-Clones, normal ESX has a much higher limit (64).

ii. 2TB vmfs3 size limit, and don’t start there, we started at 1.2TB Luns so we could expand as needed. Avoid SSMOVE if at all possible (SSMOVE is slow and painful, but works well), if you fill up your Lun, create the extend and/or disable templates and move them to a new datastore. You can go up to 64TB with Extends, but I like smaller LUNS for performance and other reasons.

iii. Only available backups are SAN Snapshots (well the only realistic one for us), and for this to be useful, see #1 below

iv. Recommended to put vCenter & Lab Manager Servers on VM’s inside cluster on SAN with the guests (use resource pools to guarantee performance)

v. 4.0 vCenter limits

1. 32 bit has max of 2000 deployed machines and 3000 registered

2. 64 bit has max of 3000 deployed machines and 4500 registered

v. 4.1 vCenter limits (NEW!!)

64 bit has max of 10,000 deployed machines and 15,000 registered

20,000 ports per vDS (4,096 in vSphere 4.0)

(You are still limited to 3000 machines and 32 hosts per cluster, which is important for Host Spanning)

Best Practices & What we’ve learned

i. Make Luns 10x the size of your Template(s)

ii. Shared Storage is generally the first bottleneck. I used all Raid 1+0 since we observed this on our first LBM deployment and our application is database driven (disk I/O intensive). Tier your storage if possible, EMC FAST or similar technology.

iii. We have averaged between 80-120 VM’s per blade, so this means our LBM 4.0 environment should top out at approximately 80 hosts (5 full HP c7000’s) (one cluster, otherwise you lose the advantages of Host Spanning Transport Networks).

iv. LOTS of IP addresses, I recommend at least a /20 for LBM installs = 4096 IP’s, you do not want to have to re-IP lab manager guests, we’ve done that before.

v. Create a pivot Data Store, Many SAN technologies require that you present the same LUNS to a group of hosts, Think “Storage Group” from EMC. Because of this, you may want to move a VM from one storage group to another, there isn’t any way to accomplish that without having either another SAN, iSCSI, or NFS storage available that you can use for a transfer point to svmotion the VM/Template to, and then to the appropriate storage group.

vi. If using ESXi, keep one ESX per storage group for Exports, ESXi does not support SMB, so users can not export their VM’s without this.

vii. Create Gold Master Libraries for your users, helps prevent the 30 disk chain limit from being hit as often.

viii. Encourage Libraries, not snapshots

ix. Do not allow users to create\import templates, Export Only

x. Do not allow users to modify Disk, CPU or Memory on VM’s.

xi. Storage and Deployment leases are the best thing since sliced bread. Recommend between 14-28 days for both.

xii. Linked Clones are great, but Hardware Dedupe is better.

xiii. Train your users, we even went as far as to have two access levels, one for trained, one for untrained, so the untrained are less dangerous, and if they want the advanced features it forces them to get training.

After a recent power outage, one of the 1TB Sata drives in a Raid-5 died, not a big deal. However, one of the Hot-Spare 300GB FC drives took over for it. We are nearly full allocated space-wise, so this is a scary thing. We do have a 1TB identical Hot-spare in the chassis, so my rep told me to just yank the FC Hot-spare drive that was now part of the raid (and had been for 5 days). Luckily this worked perfectly, pulling the drive forced the Celerra/Clariion to Transfer the data to the correct Hot-spare and after the new drive arrived everything went back to how it should be.

I want to thank Colorado Springs Utilities for unexpectedly cutting our power for routine maintenance.

When trying to power our VM's back on, we got two different odd errors on separate ESX 4.0.0 hosts, both with the same resolution.

"An error occurred while communicating with the remote host."
and
"Detected an invalid snapshot configuration."

Both of which had us concerned about data corruption to say the least, however we restarted the VMW management service and they both came back to life.

service mgmt-vmware restart

Since you can’t modify unattend.xml or really do anything else to keep it enabled through sysprep (which auto disables it)
Do this:
Create %WINDIR%\Setup\Scripts\
Create SetupComplete.cmd (make sure no hidden extensions like SetupComplete.cmd.txt)
Edit that file, put this in there:
net user administrator /active:yes
(optional change password) net user administrator new_password
After Windows is installed, but before the logon screen appears, Windows Setup searches for the SetupComplete.cmd file in the %WINDIR%\Setup\Scripts\ directory. If a SetupComplete.cmd file is found, the file is executed. Otherwise, installation continues normally. Windows Setup logs the action in the Setupact.log file.
That should do it, now that Template should keep local admin enabled even after sysprep.

Somedays I can’t see the obvious.  I got this error when trying to reboot my vCenter 4.1 Lab server and the ESX 4.1 host it is on.  Due to a lack of Nics I went against my own best practice and put the vCenter’s nic on a distributed switch.  By doing this, in the reboot, obviously vCenter could not boot because it had no network attached to the nic (to be more specific, the distributed switch did not restart because vCenter was not running), it reported an error about not being able to connect to LDAP.

C:\ProgramData\VMware\VMware VirtualCenter\Logs\vpx*.log

[2011-02-08 13:11:32.972 02480 info 'App'] [LdapBackup] Making sure LDAP instance VMwareVCMSDS is running
[2011-02-08 13:11:32.972 02480 info 'App'] [LdapBackup] Attempting to start service ADAM_VMwareVCMSDS...
[2011-02-08 13:11:32.972 02480 info 'App'] [LdapBackup] Service stopped, starting
[2011-02-08 13:17:50.003 02480 error 'App'] [LdapBackup] Timed out waiting for service.
[2011-02-08 13:17:56.003 02480 error 'App'] [LDAP Client] Failed to connect to LDAP: 0x51 (Cannot contact the LDAP server.)
[2011-02-08 13:17:56.003 02480 error 'App'] [VpxdLdap] Failed to create LDAP client

because there was no network, the service could not start.  The easiest solution was to add another nic to ESX, create a local vSwitch, tie to to that nic, then move the vCenter VM over to use it, then was able to power things back up properly.

If you ever have a hard drive or USB key that has partitions you can't get rid of in Disk Manager, just use the command line tool diskpart.

Make sure to select your disk / partition / volume / vdisk (if necessary) and use the "clean command"

like Delpart, this is a VERY powerful tool that will wipe your drive clean in a second, so use with extreme caution. Make absolutely certain you have the right thing selected before you say "clean" The selected items will have "*" stars by them, see below.

DISKPART> list volume

Volume ### Ltr Label Fs Type Size Status Info
---------- --- ----------- ----- ---------- ------- --------- --------
Volume 0 H DVD-ROM 0 B No Media
Volume 1 DVD-ROM 0 B No Media
Volume 2 System Rese NTFS Partition 100 MB Healthy System
Volume 3 C NTFS Partition 297 GB Healthy Boot
* Volume 4 Removable 0 B Unusable



Happy Partitioning.

 

PowerShell 1.0

$Host.Version

Major  Minor  Build  Revision
-----  -----  -----  --------
1      0      0      0

PowerShell 2.0 (Windows 7 - 2008)

$Host.Version
Major  Minor  Build  Revision
-----  -----  -----  --------
2      0      -1      -1

This has been a frustrating error that started out of nowhere. Suddenly the backups of one of my VM's just stopped working. This VM does not do alot of transactions, not does it generate alot of disk activity. The error messages tend to suggest that. However, I tried powering down the VM to get the snapshot to work so the backup would complete, that worked as an interrum solution. Long term I moved the VM to other LUNS, I tried going from think to thick provisioned, that didn't help either. Finally got it fixed, I uninstalled the VMW tools, then reinstalled the tools with a 'custom' install and unchecked the option for VSS (from a suggestion I read). This seems to have fixed the issue and the online backup is running right now for the first time in a long time.

I have a VM with snapshots exported from lab manager, so I am using the vmware-vdiskmanager to consolidate my 14 or so linked clones/snapshots to one flat file.  the vmware-vdiskmanager can be found in server or workstation installations, it’s not in the path, you have to go to the directory to find it.

vmware-vdiskmanager -r 015495-2008R2.vmdk -t 2 2008R2.vmdk

What I found was that NTP has a default maximum correction of about 1000 seconds. I had this same issue, but if I set the clocks 3-4 minutes off, they would auto correct within about 15 minutes. Using ESXi 4.1, almost none of the ntpd commands gave me any output, but I was able to see the corrections happening in the /var/log/messages.