This morning I read the post by Gabe on his blog about ESXi on USB devices here. Coincidentally I’m taking a look at the ESXi installation of the HP BL460c G7 using kickstart. The installation process is explained in this post. This HP BL460c G7 blade server has two types of media to install ESX I to : SD or SSD. In this post I’ll try to explain what problems I ran into and my take on VMware not supporting the installation on SD / USB.
No scripted install to USB / SD device
Before getting into details : scripted install to USB (or SD which ESXi sees as a USB device) isn’t supported by VMware. This KB article by VMware explicitly says that you can’t install ESXi via scripted install.
“You cannot use Scripted Install to install ESXi Installable to a USB device”
Question however is : Why is this not supported? It’s not a technical issue. You can install ESXi to a USB / SD device when you install ESXi manually. And with kickstart you should be able to pinpoint the installation device to the USB / SD device. So why doesn’t VMware support the installation via scripted install to the USB / SD device?
I can only guess for the reason why. Haven’t found the answer to that question anywhere and would love to know. The only thing I can do is guess. My opinion about it is that is has something to do with embedded ESXi vs. installed ESXi.
Since it’s not technical issue, it must have some other reason. Just think when it would be possible to kickstart your ESXi installation to a USB / SD device, it would practically make the ESXi embedded solution obsolete. You just buy your servers with a non-installed USB device / SD card and install your own version of ESXi onto the USB / SD disk via kickstart. No need anymore for a the vendor specific ESXi embedded version.
Also taking into account the fact that most vendors are moving away from the ESXi embedded solution (for example HP discontinued ESXi embedded) , makes you wonder even more : Why can’t I install ESXi to a USB / SD device?
Making a choice : SD vs. SSD
So why, if VMware doesn’t support it, do you want to install to an SD device. Most server vendors currently install a SD slot into their server for the purpose of installing an OS onto a SD card. This gives you an installation method besides installing onto SSD disks.
From a cost perspective this is a huge benefit. For example an SD card for the HP BL460c G7 will cost you about 80 euros. An SSD disk in that same server will cost about 800 euros. Which in my opinion is a big difference in price if you take into account that the underlying technology (flash) is basically the same.
But from a redundancy design point of view it would be better to use the SSD disk. You will be able to put them in a RAID configuration. Which would prevent a single-point-of-failure in the server in case one of the SSD disks fails. Most server vendors don’t have a solution to make SD redundant. The only solution I know is the Dell Internal Dual SD module. So basically it’s an advantage of SSD over SD when looking at redundancy.
But you can also question that when you ask yourself what failure rate is to be expected of flash disks (SD or SSD) versus traditional HD disk. In general flash disks are more stable then traditional disks. More risk probably comes from a bad batch of SD or SSD disks than from a SD / SSD disk failing due to a technical error.
Then again if ESXi hosts fail, you always have VMware High Availability (HA) to kick in and save the day.
Conclusion
The choice of SD vs. SSD comes down to the same old design mantra : It depends.
The choice should be made based on the requirements that your vSphere design has to meet. It is based on the down time your company is willing to except. If it’s not a problem for VMs to go down for a couple of minutes then SD can be your preferred solution. If availability is key then you should go for the RAID SSD solution. Availability will cost you money as always.
But back to reality. SD isn’t supported with scripted install. So I had no choice. SSD it’s going to be. I do however regret that I don’t have the option to choose. I would like to see VMware supporting the kickstart scripted install for ESXi so I do have the option to install on USB / SD via scripted install.
Cloud technology is currently redefining IT infrastructure as we speak. Companies are presenting their cloud solutions at rapid speed and new cloud products are being announced every week. It seems that every vendor is on the cloud train and wants their customers to hop on too.
Question however is : Is your company ready for the cloud transformation?
As with all new technologies it also requires a different mindset. A new way of thinking about how to incorporate the cloud technology into your company. Cloud requires a new vision and strategy with regards to the IT infrastructure in your company. It’s a transformation process and the key is not in the technology, but in the organization adjustment.
The picture below (click to enlarge) gives a good representation of how the traditional, vertical IT management approach is being transformed to a cloud, policy driven approach.
More information about transforming and integrating cloud solutions with your IT infrastructure can be found in the whitepaper “Accelerate Hybrid Cloud Succes: Adjusting the IT mindset” by IDC here. This whitepaper was sponsored by VMware. See the press release here (Dutch).
VMware released an excellent whitepaper on troubleshooting performance problems in vSphere 4.1. It really is a great resource and start point for anyone who has performance issues in his / her vSphere infrastructure.
The steps discussed in the document use performance data and charts readily available in the vSphere Client and esxtop to aid the troubleshooting flows. Each performance troubleshooting flow has two parts:
1. How to identify the problem using specific performance counters.
2. Possible causes of the problem and solutions to solve it.
Quote for the Introduction of the Performance Troubleshooting for vSphere 4.1 whitepaper :
Performance problems can arise in any computing environment. Complex application behaviors, changing demands, and shared infrastructure can lead to problems arising in previously stable environments. Troubleshooting performance problems requires an understanding of the interactions between the software and hardware components of a computing environment. Moving to a virtualized computing environment adds new software layers and new types of interactions that must be considered when troubleshooting performance problems.
Proper performance troubleshooting requires starting with a broad view of the computing environment and systematically narrowing the scope of the investigation as possible sources of problems are eliminated. Troubleshooting efforts that start with a narrowly conceived idea of the source of a problem often get bogged down in detailed analysis of one component, when the actual source of problem is elsewhere in the infrastructure. In order to quickly isolate the source of performance problems, it is necessary to adhere to a logical troubleshooting methodology that avoids preconceptions about the source of the problems.
The document can be found here. Source is the blog post from the VMware VROOOM! Blog.
Install your new ESXi with your brand new installation process. Check! Verify that all your custom settings for ESXi are correct. Check! Install your vCenter server. Check! Configure vCenter and create a cluster. Check! Add ESXi host to vCenter. ERROR! *argh*
Troubleshooting. Always fun. You learn new stuff by exploring what you are doing wrong.
But OK. Then what went wrong?
Adding a host to vCenter
When you try to add your ESXi host to your vCenter server, the vCenter agent is installed to the ESXi host. It is installed locally on the ESXi host and is the agent that the vCenter server uses to communicate to the ESXi host.
During the installation of the vCenter agent the following steps are taken :
1. Upload vCenter agent to the ESXi host.
2. Install vCenter agent on the ESXi host and start the daemon.
3. Verify that the vCenter agent is running and vCenter is able to communicate with it.
4. Retrieve host configuration and set configuration settings (if necessary)
Everything went ok, until step 3. At that moment the process stalled and eventually the vCenter came back with the following error message :
Troubleshooting
Ok, so whenever I have a problem with vSphere I turn to my good old friend Google to solve all my problems. Cause, face it, the events in vCenter / ESXi aren’t always that clear about what’s going on. Ok, they give you a starting point. From that point on it’s either Google or deep dive the log files of vCenter or ESXi.
But Google returned no results that satisfied my requirements. Well then it’s of to the command line and view some logs.
But first enable the Remote Tech Support Mode to be able to log in via SSH. Read about it over here.
The logs for VMware on the ESXi host are located in : /var/log/vmware
There is the file located to install the vpxa daemon, vxp-iupgrade.log. Viewing the last 10 lines with the following command : tail –f vpx-iupgrade.log
This shows that the starting of the vpxa daemon fails after the installation has been completed. Which was also shown during the installation through vCenter. The daemon was installed, but could not be started.
Now turn to the vpxa.log in the /var/log/vmware/vpxa directory to find out what the problem is. There the following error is shown :
Bingo! So we now know that there is a problem with the custom certificate file which was uploaded during the installation of ESXi. Apparently there is something wrong with the certificate file.
Resolution
The problem is in fact the custom SSL files that ESXi and vCenter use to communicate to one another securely. For more information see my previous post here. My custom SSL certificate file was not recognized. So therefor I decided to re-generate the SSL certificate and private key.
Normally this is done during the installation process first boot. But you can also execute the bash script yourself from the command line:
/sbin/generate-certificates.sh
This will generate new SSL certificate files an put them in the default location /etc/vmware/ssl
Afterwards restart the host daemon to load the SSL certificates again :
/etc/init.d/hostd restart
Add the host to vCenter and you’ll see that the ESXi host will be added to your vCenter correctly.
Conclusion
The reason for this post isn’t only to give you a solution to what my problem is, but also a path how to troubleshoot your VMware problems. I think every VMware administrator should poses these skills and should be able to look beyond the events that are created in vCenter. As you can see in the post above, it only takes analytic skills, common sense and the internet / Google to solve your problems. And no the command line isn’t something to be afraid of, even for a Windows sysadmin
Some good resources on troubleshooting can be found here :
VMware announced that it is going to release the VMware vCloud Connector. With this connector you will be able to connect to public vCloud solutions that are provided by service providers like Bluelock and Colt and in the near future Verizon (currently in beta).
Over the last couple of months these service providers have been building public vClouds based on VMware vCloud technology. The VMware vCloud Connector is the missing piece of linking your private vCloud (a.k.a. vSphere) to one of the public vClouds of the service providers.
The following link gives a graphic representation on how you should visualize the creation of a hybrid vCloud using the VMware vCloud Connector.
The VMware vCloud Connector is a virtual appliance running in your own private vCloud. By using a plugin in your vSphere client you can use the vCloud Connector to connect to public vClouds that are provided by the service providers that have build vClouds that are accessible through the vCloud API.
By using your vSphere Client together with the vCloud Connector you create a “single pain glass” management console for managing both your private vCloud and public vCloud resources.
This creates a hybrid cloud management interface with the following capabilities :
– Visualize workloads and templates across vSphere and private/public vClouds
– Migrate workloads and templates between vSphere and vClouds
– Perform basic power and deployment operations on workloads and templates
– Access console of vApps in vClouds
For more information on the VMware vCloud Connector, have a look at the post created by VMware vCloud Architect Massimo Re Ferre’
So you have to deploy ESXi and think about automating the installation. Then the kickstart file is going to be your new best friend. It’s a think once, apply many concept to deploy ESXi to your hardware.
At the moment I’m building an vSphere 4.1 infrastructure with 300+ HP blades. All need to be migrated from ESX 3.5 to ESXi 4.1. This requires a new installation of ESXi. I’m using kickstart to automate the installation process and to get a consistent installation state when deploying ESXi.
ESXi deployment options
In general there are 2 options if you want to install ESXi : interactive and scripted. As you will probably understand, kickstart is the core of the scripted installation. The kickstart.cfg file is the file that contains the configuration settings for your ESXi installation and more.
The picture below shows the different options you have for installing ESXi onto your hardware.
Both installation method use either CD or PXE to install ESXi. The key difference is the kickstart file. The most commonly used scripted installation is the PXE method. With PXE you boot your server into a PXE image which deploys your ESXi from a media depot hosted through HTTP(S), FTP or NFS. This media in combination with your kickstart file will created a custom ESXi server.
Deployment tool
For the deployment I have been using HP Insight Control Server Deployment 6.2 (former HP Rapid Deployment Pack (RDP)) an OEM version of Altiris Server Deployment 6.9. As far as I know the only commercial tool at the moment that supports ESXi 4.1 deployment.
There are however also two freeware appliances available for PXE ESXi deployment. Ultimate Deployment Appliance (UDA) and ESX Deployment Appliance (EDA) can be downloaded from the VMware Virtual Appliance Marketplace. But you can also take a look at the ESXi Installable and vCenter Server Setup Guide which gives good leads on building your own PXE deployment solution.
HP Insight Control Server Deployment presents you with a workflow tool to deploy a server. The workflow can be created by adding scripts to installation jobs. HP creates specials deployment packs for OS installations. Also one for ESXi 4.1 which has the following 4 jobs in them :
1. Configure BIOS (Very handy to set those BIOS settings for virtualization!)
2. Deploy GRUB image
3. Configure GRUB image
4. Create kickstart file based on default kickstart script
This deployment pack generally gives you a basic installation of ESXi if you don’t edit the kickstart file. The only thing you have to do afterwards is assign the job to a specific HP server and it will install ESXi out-of-the-box.
Kickstart file
The kickstart file below is used in conjunction with HP Insight Control Server Deployment. This tool has a database with all the configuration variables per server object. The @@VARIABLES@@ in the kickstart script below are replaced each time the installation job is executed.
This creates a server specific kickstart file for each server in your server park.
The lines until %firstboot are created to install ESXi. All commands after that line will be executed after the ESXi host has booted for the first time.
For more information about the specific commands in the kickstart file :
#————————————————————————-
# Customer default Kickstart for ESXi 4.1.x
# Created by : Martijn Baecke
# Date : 26-01-2011
# Summary : This kickstart script is used for the installation of
# ESXi using HP Insight Control Server Deployment 6.2
#————————————————————————-
########### Start : Networking ###########
# Add vMotion portgroup to vSwitch0
esxcfg-vswitch -A “vMotion Network” vSwitch0
# Add vmnic3 to vSwitch0
esxcfg-vswitch -L vmnic3 vSwitch0
# Add IP address to vMotion vmk1
esxcfg-vmknic -a -i @@VMOTIONIP@@ -n @@VMOTIONNETMASK@@ -p “vMotion Network”
# Assign VLAN to vMotion Network portgroup
# esxcfg-vswitch -v XX -p “vMotion Network” vSwitch0
sleep 10
# Set vMotion to vmk1
vim-cmd hostsvc/vmotion/vnic_set vmk1
# Set security policy to reject on vSwitch0
vim-cmd hostsvc/net/vswitch_setpolicy –securepolicy-promisc=0 vSwitch0
vim-cmd hostsvc/net/vswitch_setpolicy –securepolicy-macchange=0 vSwitch0
vim-cmd hostsvc/net/vswitch_setpolicy –securepolicy-forgedxmit=0 vSwitch0
# Set NIC order policy for portgroups on vSwitch0
vim-cmd hostsvc/net/portgroup_set –nicorderpolicy-active=vmnic0 –nicorderpolicy-standby=vmnic3 vSwitch0 “Management Network”
vim-cmd hostsvc/net/portgroup_set –nicorderpolicy-active=vmnic3 –nicorderpolicy-standby=vmnic0 vSwitch0 “vMotion Network”
# Set failback to No for portgroups on vSwitch0
vim-cmd hostsvc/net/portgroup_set –nicteaming-rollingorder=1 vSwitch0 “Management Network”
vim-cmd hostsvc/net/portgroup_set –nicteaming-rollingorder=1 vSwitch0 “vMotion Network”
############ End : Networking ############
########### Start : Storage ###########
# Configure local datastore with different label
vim-cmd hostsvc/datastore/rename datastore1 “$(hostname -s)-local-storage”
# Set Round Robin (RR) as default PSP for VMW_SATP_SYMM (EMC devices)
esxcli nmp satp setdefaultpsp –satp VMW_SATP_SYMM –psp VMW_PSP_RR
# Set Round Robin (RR) for all EMC devices
EMC_DEVICES=`esxcli nmp device list | grep ‘EMC Fibre Channel Disk’| awk {‘print $NF’}| sed -e ‘s/[()]//g’`
for i in $EMC_DEVICES;do esxcli nmp device setpolicy -d $i -P VMW_PSP_RR; done;
# Set Round Robin (RR) for all HDS devices
HDS_DEVICES=`esxcli nmp device list | grep ‘HDS Fibre Channel Disk’| awk {‘print $NF’}| sed -e ‘s/[()]//g’`
for i in $HDS_DEVICES;do esxcli nmp device setpolicy -d $i -P VMW_PSP_RR; done;
# Determine if this is a cluster with EMC or HDS storage. Cluster with mixed storage not allowed! If mixed EMC settings apply!
if [ -n “$HDS_DEVICES” ]; then
# This is a cluster with HDS storage
# Queue depth for HDS remains default = 32
# If this needs to change unquote following lines. Replace YY with value
# esxcfg-module -s ql2xmaxqdepth=YY qla2xxx
# vim-cmd hostsvc/advopt/update Disk.SchedNumReqOutstanding long YY
# Enable VAAI for HDS
vim-cmd hostsvc/advopt/update DataMover.HardwareAcceleratedMove long 1
vim-cmd hostsvc/advopt/update DataMover.HardwareAcceleratedInit long 1
vim-cmd hostsvc/advopt/update VMFS3.HardwareAcceleratedLocking long 1
else
# This is a cluster with EMC storage
# Set queue depth to 64
esxcfg-module -s ql2xmaxqdepth=64 qla2xxx
vim-cmd hostsvc/advopt/update Disk.SchedNumReqOutstanding long 64
fi
############ End : Enable Management Traffic ############
########### Start : Enable SSH Tech Support Mode ###########
# Only enable this when doing testing with kickstart!
# Default = disable by using # in front of commands
Security is an important factor within your vSphere Infrastructure. This is why all traffic between the vCenter server and the ESXi hosts is encrypted using Transport Layer Security (TLS). ESXi support TLS by the use of SSL version 3 or TLS version 1 type certificates, generally referred to as SSL certificates. For more information on data encryption see Transport Layer Security on Wikipedia.
The use of SSL certificates within ESXi
SSL certificates are used within vSphere by default. Both vCenter and ESXi create their own SSL certificates during the installation process.During the creation a certificate and a private key are generated and are stored in the following location :
SSL certificate => /etc/vmware/ssl/rui.crt
SSL private key => /etc/vmware/ssl/rui.key
Both these keys are used to encrypt the data communication between the ESXi host and the vCenter client. And between the ESXi host and the vSphere client. This can be noticed when connect to either the vCenter server or ESXi host with a vSphere client. A warning message is then shown that the certificated used by the vCenter server or ESXi host could not be verified. Simply ignoring this will notice or installing the certificate on your local host will resolve this warning message.
To summarize : ESXi by default uses data communication encryption with self-generated SSL certificates.
Replacing the default SSL certificates ESXi
The ESXi default certificates that are created during the installation process are unique. However these certificates aren’t verifiable and aren’t checked against a central certificate authority (CA) that is known and trusted within the IT infrastructure.
Some companies require the use of a company CA server to validate the SSL certificates. Most commonly used is the Microsoft Active Directory Certificates server that validated certificates in a Public Key Infrastructure (PKI). You will need to generate your new SSL certificates for ESXi that are signed by the CA. After generating the new certificate and private key for the ESXi host(s) that have been signed by the CA the files need to be uploaded to your ESXi host.
Upload your files via the HTTPS PUT command to the following sites :
For SSL certificates : https://<hostname>/host/ssl_crt
For keys : https://<hostname>/host/ssl_key
The last can be achieved by using a program that is able to execute a HTTPS PUT command.
Powershell
To be able to upload the certificate and private key file I’ve created a script that is able to upload the SSL certificate and private key using the HTTPS PUT command and Powershell. The script can be downloaded via VMware DOC 14655.
There are several ways to add additional drivers to ESXi. This can sometimes be necessary when implementing a new type of servers. So if you have a device that isn’t supported by the default installation of ESXi a driver needs to be added after the installation.
This post will describe how to add an additional drivers to ESXi with update manager. It can also be done via the vCLI which is described by the guys at VMGuru.nl here.
The additional drivers that are available to ESXi can be downloaded from the vSphere download page under the tab Drivers & Tools. All downloads will be delivered in ISO format.
The ISO includes a zip file that contains the (offline-bundle) additional driver. The additional driver will be available in zip-file which can be imported under Configuration -> Patch Download Settings in VMware Update Manager. There is a note (see picture in yellow) with a link to “Import Patches”.
This will start a proces of importing the additional update into Update Manager. After this the update will be available in your repository. After which you can create a baseline and patch your ESXi hosts.
For more information on how to patch your ESXi host with baselines and remediation see the following video :
Virtualization of servers results in the fact that server are no longer depeded on the hardware on which it runs. This is one of the key selling points of virtualization, but it also results in the fact that you can’t connect external hardware devices to your virtual machine directly.
Especially for USB devices this can be a problem. There are still application vendors out there that have implemented some sort of USB dongle for licensing their software. To make it possible to connect your USB dongle to a virtual machines an implementation is needed of remote USB to connect the USB devices to your virtual machine.
The following picture gives a graphical representation of how this works.
Click to enlarge
The concept of remote USB is based on the client-server model. All USB devices will be centralized to one USB remote server. Through a software USB device driver (=client) in the virtual machine a connection is made with this USB remote server.
Through management on the USB remote server, USB devices can be allocated to a particular virtual machine. This allows virtual machines to make use of USB devices without connecting that USB device to the hardware on which it runs. All USB data is now send over the network from the virtual machine to the remote USB server and vice versa.
There are several vendors on the market with a remote USB solution. All have the same result : giving access to USB devices over the network. Only difference is that some have software based solution while others have a hardware based solution.
When using a software based solution a server is needed that acts as the remote USB host. The server must have enough USB slots available to connect all the USB devices. The hardware based solution is a network device which act as the end point in the remote USB solution. All USB devices will be connected to this network device.
Note : Remote USB isn’t needed for VDI implementations. Most VDI vendors have USB redirection incorporated into their solution. In case you are using thin clients, USB redirection can also be implemented on the thin client policy server.
