So you have to deploy ESXi and think about automating the installation. Then the kickstart file is going to be your new best friend. It’s a think once, apply many concept to deploy ESXi to your hardware.
At the moment I’m building an vSphere 4.1 infrastructure with 300+ HP blades. All need to be migrated from ESX 3.5 to ESXi 4.1. This requires a new installation of ESXi. I’m using kickstart to automate the installation process and to get a consistent installation state when deploying ESXi.
ESXi deployment options
In general there are 2 options if you want to install ESXi : interactive and scripted. As you will probably understand, kickstart is the core of the scripted installation. The kickstart.cfg file is the file that contains the configuration settings for your ESXi installation and more.
The picture below shows the different options you have for installing ESXi onto your hardware.
Both installation method use either CD or PXE to install ESXi. The key difference is the kickstart file. The most commonly used scripted installation is the PXE method. With PXE you boot your server into a PXE image which deploys your ESXi from a media depot hosted through HTTP(S), FTP or NFS. This media in combination with your kickstart file will created a custom ESXi server.
Deployment tool
For the deployment I have been using HP Insight Control Server Deployment 6.2 (former HP Rapid Deployment Pack (RDP)) an OEM version of Altiris Server Deployment 6.9. As far as I know the only commercial tool at the moment that supports ESXi 4.1 deployment.
There are however also two freeware appliances available for PXE ESXi deployment. Ultimate Deployment Appliance (UDA) and ESX Deployment Appliance (EDA) can be downloaded from the VMware Virtual Appliance Marketplace. But you can also take a look at the ESXi Installable and vCenter Server Setup Guide which gives good leads on building your own PXE deployment solution.
HP Insight Control Server Deployment presents you with a workflow tool to deploy a server. The workflow can be created by adding scripts to installation jobs. HP creates specials deployment packs for OS installations. Also one for ESXi 4.1 which has the following 4 jobs in them :
1. Configure BIOS (Very handy to set those BIOS settings for virtualization!)
2. Deploy GRUB image
3. Configure GRUB image
4. Create kickstart file based on default kickstart script
This deployment pack generally gives you a basic installation of ESXi if you don’t edit the kickstart file. The only thing you have to do afterwards is assign the job to a specific HP server and it will install ESXi out-of-the-box.
Kickstart file
The kickstart file below is used in conjunction with HP Insight Control Server Deployment. This tool has a database with all the configuration variables per server object. The @@VARIABLES@@ in the kickstart script below are replaced each time the installation job is executed.
This creates a server specific kickstart file for each server in your server park.
The lines until %firstboot are created to install ESXi. All commands after that line will be executed after the ESXi host has booted for the first time.
For more information about the specific commands in the kickstart file :
ESXi Installable and vCenter Server Setup Guide
vSphere Command-Line Interface Installation and Scripting Guide
#————————————————————————-
# Customer default Kickstart for ESXi 4.1.x
# Created by : Martijn Baecke
# Date : 26-01-2011
# Summary : This kickstart script is used for the installation of
# ESXi using HP Insight Control Server Deployment 6.2
#————————————————————————-
########### Start : Variables ###########
#HOSTNAME=@@HOSTNAME@@
#HOSTIP=@@HOSTIP@@
#HOSTNETMASK=@@HOSTNETMASK@@
#HOSTGATEWAY=@@HOSTGATEWAY@@
#VMOTIONIP=@@VMOTIONIP@@
#VMOTIONNETMASK=@@VMOTIONNETMASK@@
########### End : Variables ###########
rootpw vmware
keyboard Default
reboot
vmaccepteula
#Clear the local hard drive.
clearpart –initlabel –firstdisk=local
autopart –firstdisk=local –overwritevmfs
install url ftp://@@DSIPADDRESS@@/@@FTPFEATUREDIR@@/dist
network –bootproto=static –device=vmnic0 –ip=@@HOSTIP@@ –gateway=@@HOSTGATEWAY@@ –nameserver=10.1.1.64 –netmask=@@HOSTNETMASK@@ –hostname=@@HOSTNAME@@ –addvmportgroup=0
#–vlanid=2000
%firstboot –unsupported –interpreter=busybox –level=990
esxcfg-advcfg -s @@HOSTNAME@@ /Misc/HostName
########### Start : Date & Time ###########
# Add time server to configuration file
cat >> /etc/ntp.conf << EOF
server time.customer.corp
EOF
# Enable the NTP deamon to start during boot
chkconfig ntpd on
############ End : Date & Time ############
########### Start : DNS ###########
# Set DNS configuration
vim-cmd hostsvc/net/dns_set –hostname=@@HOSTNAME@@ –domainname=customer.corp –searchdomain=customer.corp –ip-addresses=10.0.0.1,10.0.0.2
############ End : DNS ############
########### Start : Networking ###########
# Add vMotion portgroup to vSwitch0
esxcfg-vswitch -A “vMotion Network” vSwitch0
# Add vmnic3 to vSwitch0
esxcfg-vswitch -L vmnic3 vSwitch0
# Add IP address to vMotion vmk1
esxcfg-vmknic -a -i @@VMOTIONIP@@ -n @@VMOTIONNETMASK@@ -p “vMotion Network”
# Assign VLAN to vMotion Network portgroup
# esxcfg-vswitch -v XX -p “vMotion Network” vSwitch0
sleep 10
# Set vMotion to vmk1
vim-cmd hostsvc/vmotion/vnic_set vmk1
# Set security policy to reject on vSwitch0
vim-cmd hostsvc/net/vswitch_setpolicy –securepolicy-promisc=0 vSwitch0
vim-cmd hostsvc/net/vswitch_setpolicy –securepolicy-macchange=0 vSwitch0
vim-cmd hostsvc/net/vswitch_setpolicy –securepolicy-forgedxmit=0 vSwitch0
# Set NIC order policy for portgroups on vSwitch0
vim-cmd hostsvc/net/portgroup_set –nicorderpolicy-active=vmnic0 –nicorderpolicy-standby=vmnic3 vSwitch0 “Management Network”
vim-cmd hostsvc/net/portgroup_set –nicorderpolicy-active=vmnic3 –nicorderpolicy-standby=vmnic0 vSwitch0 “vMotion Network”
# Set failback to No for portgroups on vSwitch0
vim-cmd hostsvc/net/portgroup_set –nicteaming-rollingorder=1 vSwitch0 “Management Network”
vim-cmd hostsvc/net/portgroup_set –nicteaming-rollingorder=1 vSwitch0 “vMotion Network”
############ End : Networking ############
########### Start : Storage ###########
# Configure local datastore with different label
vim-cmd hostsvc/datastore/rename datastore1 “$(hostname -s)-local-storage”
# Set Round Robin (RR) as default PSP for VMW_SATP_SYMM (EMC devices)
esxcli nmp satp setdefaultpsp –satp VMW_SATP_SYMM –psp VMW_PSP_RR
# Set Round Robin (RR) for all EMC devices
EMC_DEVICES=`esxcli nmp device list | grep ‘EMC Fibre Channel Disk’| awk {‘print $NF’}| sed -e ‘s/[()]//g’`
for i in $EMC_DEVICES;do esxcli nmp device setpolicy -d $i -P VMW_PSP_RR; done;
# Set Round Robin (RR) for all HDS devices
HDS_DEVICES=`esxcli nmp device list | grep ‘HDS Fibre Channel Disk’| awk {‘print $NF’}| sed -e ‘s/[()]//g’`
for i in $HDS_DEVICES;do esxcli nmp device setpolicy -d $i -P VMW_PSP_RR; done;
# Determine if this is a cluster with EMC or HDS storage. Cluster with mixed storage not allowed! If mixed EMC settings apply!
if [ -n “$HDS_DEVICES” ]; then
# This is a cluster with HDS storage
# Queue depth for HDS remains default = 32
# If this needs to change unquote following lines. Replace YY with value
# esxcfg-module -s ql2xmaxqdepth=YY qla2xxx
# vim-cmd hostsvc/advopt/update Disk.SchedNumReqOutstanding long YY# Enable VAAI for HDS
vim-cmd hostsvc/advopt/update DataMover.HardwareAcceleratedMove long 1
vim-cmd hostsvc/advopt/update DataMover.HardwareAcceleratedInit long 1
vim-cmd hostsvc/advopt/update VMFS3.HardwareAcceleratedLocking long 1else
# This is a cluster with EMC storage
# Set queue depth to 64
esxcfg-module -s ql2xmaxqdepth=64 qla2xxx
vim-cmd hostsvc/advopt/update Disk.SchedNumReqOutstanding long 64
fi
############ End : Storage ############
########### Start : Syslog server ###########
# Syslog server: syslog.customer.corp
vim-cmd hostsvc/advopt/update Syslog.Remote.Hostname string syslog.customer.corp
vim-cmd hostsvc/advopt/update Syslog.Remote.Port int 514
vim-cmd hostsvc/advopt/update Syslog.Local.DatastorePath string “[$(hostname -s)-local-storage] /syslog-$(hostname -s).log”
############ End : Syslog server ############
########### Start : Certificates ###########
cp /etc/vmware/ssl/rui.key /etc/vmware/ssl/rui.key.backup
cp /etc/vmware/ssl/rui.crt /etc/vmware/ssl/rui.crt.backup
wget ftp://@@DSIPADDRESS@@/@@FTPFEATUREDIR@@/certificates/@@HOSTNAME@@.key -O /etc/vmware/ssl/rui.key
wget ftp://@@DSIPADDRESS@@/@@FTPFEATUREDIR@@/certificates/@@HOSTNAME@@.crt -O /etc/vmware/ssl/rui.crt
############ End : Certificates ############
########### Start : Enable Management Traffic ###########
# Enable Management Traffic on vmk0
HOSTSVC_FILE=/etc/vmware/hostd/hostsvc.xml
cat > ${HOSTSVC_FILE} << __CREATE_HOST_SVC__
<configroot>
<mangementVnics>
<nic id=”0000″>vmk0</nic>
</mangementVnics>
<mode>normal</mode>
<service>
<ntpd>on</ntpd>
</service>
</configroot>
__CREATE_HOST_SVC__
/etc/init.d/hostd restart
vim-cmd hostsvc/net/refresh
############ End : Enable Management Traffic ############
########### Start : Enable SSH Tech Support Mode ###########
# Only enable this when doing testing with kickstart!
# Default = disable by using # in front of commands
vim-cmd hostsvc/enable_remote_tsm
vim-cmd hostsvc/start_remote_tsm
vim-cmd hostsvc/net/refresh
############ End : Enable SSH Tech Support Mode ############
# Write to ramdisk
esxcfg-boot -b
# Enter maintenance mode
vim-cmd hostsvc/maintenace_mode_enter
sleep 30
reboot
Other resources for kickstart
ZenHat : How To: Sample kickstart file for VMware ESXi 4.1
KendrickColeman.com : ESXi 4.1 Kickstart Install – WIP
VMware ESXi Chronicles : Scripted Install with ESXi
ESXi Installable and vCenter Server Setup Guide
Automating ESXi 4.1 Kickstart Tips & Tricks
Update : Have added the enabling of the Management Traffic. Thanks to @lamw.
Nice post! Your %firstboot script was extremely helpful! One thing to consider:
xCAT is another open source toolkit that supports PXE boot for ESXi. In addition, it has a cool feature that allows to PXE boot a diskless ESXi image. I’ve tested it on HP, Cisco, Dell, IBM, and other whiteboxes and works really well.
Thanks again for the post!
Hi,
Nice post. I was wondering how I could use this to deploy a number of hosts. The script “Configure Answer File” will set the variables like hostname and dsipaddress but what about the other variables like Management, vMotion.
Thanks
Hi virtualsj,
I’ve modified the “Configure Answer File” with the content below. As you can see I’m using more files in the Computer tabel of the database. Each field has a value which is configured in the kickstart file. For example : host ip address uses the field nw_pref _server.
# Configure Answer File
export basekickstartfile=kickstart-en_us.cfg
export kickstartfile=%ID%.cfg
export dsipaddress=%#*”select tcp_addr from aclient_prop where computer_id=0″%
export hostname=%#*”select replace([name],’ ‘,”) from computer where computer_id={ID}”%
export hostip=%#!computer@nw_pref_server%
export hostnetmask=%#!computer@nw_nds_context%
export hostgateway=%#!computer@nw_pref_tree%
export vmotionip=%#!computer@lic_os_user%
export vmotionnetmask=%#!computer@lic_os_key%
export vmotiongateway=%#!computer@lic_os_org%
/mnt/ds/hpfeatures/esxi41/bin/configurekickstart.sh
Hello,
Are you pre-populating the information manually to your rdp database? For example, in the “computer” table in the “lic_os_user” column, are you entering in the the vmotion IP and the ks script is picking it up during the install? I am trying to figure out a way to use our RDP server to deploy many ESXi hosts and not use a separate script for each individual host.
thanks.
Yes you will need to populate the tables in the database _per host_! When you PXE boot your new HP server it will fire up the Altiris client. This will contact the RDP server which will record an entry (with new ID) in the eXpress database.
You can then populate the variables in the Computer database which you want the “Configure Answer File” job to modify. I’ve added a couple of variables more, since my kickstart script needed to modify more variables in the default kickstart script.
Each time you deploy this job to a new server it will create a new, unique kickstart for the HP server that has been added to the database. This way Altiris / HP creates a unique kickstart script for each individual host based on your default kickstart script.
To conclude : default kickstart script + variables modified in “Configure Answer File” job = unique kickstart file for server.
Thank you that makes it much clearer.
Martijn,
Thanks for the script and particularly the explanation of how the elements of the HP Insight Control Deployment Server hang together.
Am I right in assuming that in addition to the details you gave about the answer file that you also have to manually edit the configurekickstart.sh script to include the sed commands for the variables that you detailed?
I have tried this but every time I update the configurekickstart.sh the task bombs out with ‘error 126 during script execution’ and the computerID script doesn’t get modified/created.
Hi Richard,
Yes you do need to create a link between the variables that you use in the kickstart script and the tables in the database. Currently don’t have access anymore to the HP deployment server so can’t really tell.
But walkthrough the scripts provided by HP. Try to understand what they do and how you can add your own custom field.
Best regards,
Martijn
Hi Martijn,
Found an answer shortly after posting last night. Somebody mentioned that I needed to edit the files using a Linux-aware text editor so as not to screw up the CR/LF. Talk about a rookie mistake! Sorted this morning and I’m off and running. It’s taken me the best part of a week to decipher all that goes on under the bonnet of HPICSD, but your article was certainly the most valuable pointer.
The only conundrum I’m left with now is how best to bulk update the eXpress.dbo.computer table from our spreadsheet. Out of interest did you wait for the computer objects to be created in HPICSD or did you populate the whole table beforehand?
Kind regards,
Richard