Upgrade Instruction from 4.4.x
This section will guide you from CloudStack 4.4.x to CloudStack 4.19.1.1.
Any steps that are hypervisor-specific will be called out with a note.
We recommend reading through this section once or twice before beginning your upgrade procedure, and working through it on a test system before working on a production system.
Note
The following upgrade instructions should be performed regardless of hypervisor type.
Upgrade Steps:
Backup CloudStack database (MySQL)
Install new systemvm Template
Add package repository for MySQL connector
Upgrade CloudStack management server(s)
Update hypervisors specific dependencies
Packages repository
Most users of CloudStack manage the installation and upgrades of CloudStack with one of Linux’s predominant package systems, RPM or APT. This guide assumes you’ll be using RPM and Yum (for Red Hat Enterprise Linux or CentOS), or APT and Debian packages (for Ubuntu).
Create RPM or Debian packages (as appropriate) and a repository from the 4.19.1.1 source, or check the Apache CloudStack downloads page at http://cloudstack.apache.org/downloads.html for package repositories supplied by community members. You will need them for Management Server on Ubuntu or Management Server on CentOS/RHEL and Hypervisor: KVM hosts upgrade.
Instructions for creating packages from the CloudStack source are in the CloudStack Installation Guide.
Update System VM Templates
Note
From ACS 4.16 onwards, CloudStack will support automatic registration of System VM Templates (when using noredist packages), if not done prior initiating upgrade. However, the usual upgrade process continues to be supported.
While running the existing 4.4.x system, log in to the UI as the root administrator.
In the left navigation bar, click Templates.
In Select view, click Templates.
Click Register Template. The Register Template dialog box is displayed.
To register the System VM Template do the following:
In the Register Template dialog box, specify the following values (do not change these):
Hypervisor
Description
XenServer
Name: systemvm-xenserver-4.19.1
Description: systemvm-xenserver-4.19.1
URL: http://download.cloudstack.org/systemvm/4.19/systemvmtemplate-4.19.1-xen.vhd.bz2
Zone: Choose the zone where this hypervisor is used
Hypervisor: XenServer
Format: VHD
OS Type: Other Linux (64-bit)
Extractable: no
Password Enabled: no
Public: no
Featured: no
Routing: no
KVM
Name: systemvm-kvm-4.19.1
Description: systemvm-kvm-4.19.1
URL: http://download.cloudstack.org/systemvm/4.19/systemvmtemplate-4.19.1-kvm.qcow2.bz2
Zone: Choose the zone where this hypervisor is used
Hypervisor: KVM
Format: QCOW2
OS Type: Debian GNU/Linux 7.0 (64-bit) (or the highest Debian release number available in the dropdown)
Extractable: no
Password Enabled: no
Public: no
Featured: no
Routing: no
VMware
Name: systemvm-vmware-4.19.1
Description: systemvm-vmware-4.19.1
URL: http://download.cloudstack.org/systemvm/4.19/systemvmtemplate-4.19.1-vmware.ova
Zone: Choose the zone where this hypervisor is used
Hypervisor: VMware
Format: OVA
OS Type: Other Linux (64-bit)
Extractable: no
Password Enabled: no
Public: no
Featured: no
Routing: no
HyperV
Name: systemvm-hyperv-4.19.1
Description: systemvm-hyperv-4.19.1
URL: http://download.cloudstack.org/systemvm/4.19/systemvmtemplate-4.19.1-hyperv.vhd.zip
Zone: Choose the zone where this hypervisor is used
Hypervisor: Hyper-V
Format: VHD
OS Type: Debian GNU/Linux 7.0 (64-bit) (or the highest Debian release number available in the dropdown)
Extractable: no
Password Enabled: no
Public: no
Featured: no
Routing: no
Watch the screen to be sure that the Template downloads successfully and enters the READY state. Do not proceed until this is successful.
Java Version Requirement
CloudStack 4.19 requires installation of Java 11 JRE for management server and the KVM agent. On installing or upgrading cloudstack-management and/or cloudstack-agent packages, please configure Java 11 as the default java version using:
$ sudo alternatives --config java
Note: For Ubuntu distributions where the openjdk-11 packages are not available from the main repositories, the JRE can be installed from an external PPA such as openjdk-r. The PPA can be added before installation/upgrade:
$ sudo add-apt-repository ppa:openjdk-r/ppa $ sudo apt-get update
Database Preparation
Backup current database
Stop your management server or servers. Run this on all management server hosts:
$ sudo service cloudstack-management stop
If you are running a usage server or usage servers, stop those as well:
$ sudo service cloudstack-usage stop
Make a backup of your MySQL database. If you run into any issues or need to roll back the upgrade, this will assist in debugging or restoring your existing environment. You’ll be prompted for your password.
$ mysqldump -u root -p cloud > cloud-backup_$(date +%Y-%m-%d-%H%M%S) $ mysqldump -u root -p cloud_usage > cloud_usage-backup_$(date +%Y-%m-%d-%H%M%S)
Management Server on Ubuntu
If you are using Ubuntu, follow this procedure to upgrade your packages. If not, skip to step Management Server on CentOS/RHEL.
Time zone requirements
As of CloudStack 4.14, you must explicitly configure time zone either in the MySQL server or JDBC driver (db.properties). In previous CloudStack versions, UTC time zone was assumed by default (all event’s have UTC time stamps), so the same UTC time zone should now be explicitly configured.
You can do this by editing the /etc/cloudstack/management/db.properties file and adding “serverTimezone=UTC” to the “db.cloud.url.params=” and “db.usage.url.params=” lines. Example lines, from a clean 4.14 installation, are given below:
db.cloud.url.params=prepStmtCacheSize=517&cachePrepStmts=true&sessionVariables=sql_mode='STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION'&serverTimezone=UTC
db.usage.url.params=serverTimezone=UTC
Note
Community Packages: This section assumes you’re using the community supplied packages for CloudStack. If you’ve created your own packages and APT repository, substitute your own URL for the ones used in these examples.
The first order of business will be to change the sources list for each system with CloudStack packages. This means all management servers, and any hosts that have the KVM agent. (No changes should be necessary for hosts that are running VMware or Xen.)
CloudStack apt repository
Start by opening
/etc/apt/sources.list.d/cloudstack.list
on any systems that have CloudStack packages installed.This file should have one line, which contains:
deb http://download.cloudstack.org/ubuntu precise 4.4We’ll change it to point to the new package repository:
deb http://download.cloudstack.org/ubuntu precise 4.9Setup the public key for the above repository:
wget -qO - http://download.cloudstack.org/release.asc | sudo apt-key add -If you’re using your own package repository, change this line to read as appropriate for your 4.19 repository.
Now update your apt package list:
$ sudo apt-get update
- Now that you have the repository configured, it’s time to upgrade
the
cloudstack-management
package.
$ sudo apt-get upgrade cloudstack-management
If you use CloudStack usage server
$ sudo apt-get upgrade cloudstack-usage
Management Server on CentOS/RHEL
If you are using CentOS or RHEL, follow this procedure to upgrade your packages. If not, skip to hypervisors section Hypervisor: XenServer.
Time zone requirements
As of CloudStack 4.14, you must explicitly configure time zone either in the MySQL server or JDBC driver (db.properties). In previous CloudStack versions, UTC time zone was assumed by default (all event’s have UTC time stamps), so the same UTC time zone should now be explicitly configured.
You can do this by editing the /etc/cloudstack/management/db.properties file and adding “serverTimezone=UTC” to the “db.cloud.url.params=” and “db.usage.url.params=” lines. Example lines, from a clean 4.14 installation, are given below:
db.cloud.url.params=prepStmtCacheSize=517&cachePrepStmts=true&sessionVariables=sql_mode='STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION'&serverTimezone=UTC
db.usage.url.params=serverTimezone=UTC
Note
Community Packages: This section assumes you’re using the community supplied packages for CloudStack. If you’ve created your own packages and yum repository, substitute your own URL for the ones used in these examples.
Install new MySQL connector
Starting with 4.9.0, cloudstack-management RPM’s now depend on the
mysql-connector-python
package. Therefore Apache CloudStack
4.19.1.1 requires the instalation of the MySQL connector on CentOS.
MySQL connector RPM repository
Add a new yum repo /etc/yum.repos.d/mysql.repo
:
[mysql-community]
name=MySQL Community connectors
baseurl=http://repo.mysql.com/yum/mysql-connectors-community/el/$releasever/$basearch/
enabled=1
gpgcheck=1
Import GPG public key from MySQL:
rpm --import http://repo.mysql.com/RPM-GPG-KEY-mysql
Install mysql-connector
yum install mysql-connector-python
CloudStack RPM repository
The first order of business will be to change the yum repository for each system with CloudStack packages. This means all management servers, and any hosts that have the KVM agent.
(No changes should be necessary for hosts that are running VMware or Xen.)
Start by opening /etc/yum.repos.d/cloudstack.repo
on any
systems that have CloudStack packages installed.
This file should have content similar to the following:
[apache-cloudstack]
name=Apache CloudStack
baseurl=http://download.cloudstack.org/rhel/4.4/
enabled=1
gpgcheck=0
If you are using the community provided package repository, change
the base url to http://download.cloudstack.org/centos/$releasever/4.9/
.
Setup the GPG public key if you wish to enable gpgcheck=1
:
rpm --import http://download.cloudstack.org/RPM-GPG-KEY
If you’re using your own package repository, change this line to read as appropriate for your 4.19 repository.
Remove the deprecated dependency for awsapi.
$ sudo rpm -e --nodeps cloudstack-awsapi
- Now that you have the repository configured, it’s time to upgrade the
cloudstack-management
.
$ sudo yum upgrade cloudstack-management
If you use CloudStack usage server
$ sudo yum upgrade cloudstack-usage
Hypervisor: XenServer
(XenServer only) Copy vhd-utils file on CloudStack management servers.
Copy the file vhd-utils
to /usr/share/cloudstack-common/scripts/vm/hypervisor/xenserver
.
wget -P /usr/share/cloudstack-common/scripts/vm/hypervisor/xenserver http://download.cloudstack.org/tools/vhd-util
As of Apache CloudStack 4.4, CloudStack is not responsible to promote a new pool master on a Citrix XenServer pool. In case of failure of the pool master host, the responsibility of electing a new pool master as been delegated back to the HA feature of XenServer. CloudStack remain responsible to honored HA capability for Compute Offerings of Instances. The XenServer HA feature must be enabled only for the pool master, not for virtual-machines.
Make sure XenServer has enabled HA on the pool.
To test if poolHA is currently turned on:
xe pool-list params=all | grep -E "ha-enabled|ha-config"
Output when poolHA is ON:
ha-enabled ( RO): true
ha-configuration ( RO): timeout: 180
Output when poolHA is OFF:
ha-enabled ( RO): false
ha-configuration ( RO):
To enable poolHA, use something like this:
xe pool-enable-ha heartbeat-sr-uuids={SR-UUID} ha-config:timeout=180
Please refer to the XenServer documentation, as there are multiple ways of configuring it either on NFS, iSCSI or Fibre Channel. Be aware though, that the timeout setting is not documented. The default is 30 seconds so you may want to bump that towards 120-180 seconds.
Hypervisor: VMware
Warning
For VMware hypervisor CloudStack management server packages must be build using “noredist”. Refer to Building Non-OSS
(VMware only) Additional steps are required for each VMware cluster. These steps will not affect running guests in the cloud. These steps are required only for clouds using VMware clusters:
Stop the Management Server:
$ sudo service cloudstack-management stop
Generate the encrypted equivalent of your vCenter password:
$ java -classpath /usr/share/cloudstack-common/lib/jasypt-1.9.2.jar org.jasypt.intf.cli.JasyptPBEStringEncryptionCLI encrypt.sh input="_your_vCenter_password_" password="cat /etc/cloudstack/management/key" verbose=false
Store the output from this step, we need to add this in cluster_details table and vmware_data_center tables in place of the plain text password
Find the ID of the row of cluster_details table that you have to update:
$ mysql -u <username> -p<password>
select * from cloud.cluster_details;
Update the plain text password with the encrypted one
update cloud.cluster_details set value = '_ciphertext_from_step_1_' where id = _id_from_step_2_;
Confirm that the table is updated:
select * from cloud.cluster_details;
- Find the ID of the correct row of vmware_data_center that you
want to update
select * from cloud.vmware_data_center;
update the plain text password with the encrypted one:
update cloud.vmware_data_center set password = '_ciphertext_from_step_1_' where id = _id_from_step_5_;
Confirm that the table is updated:
select * from cloud.vmware_data_center;
Hypervisor: KVM
(KVM only) Additional steps are required for each KVM host. These steps will not affect running guests in the cloud. These steps are required only for clouds using KVM as hosts and only on the KVM hosts.
Configure the APT repo as detailed above.
Stop the running agent.
$ sudo service cloudstack-agent stop
Update the agent software.
$ sudo apt-get upgrade cloudstack-agent
- Verify that the file
/etc/cloudstack/agent/environment.properties
has a line that reads:
paths.script=/usr/share/cloudstack-common
If not, add the line.
- Verify that the file
Start the agent.
$ sudo service cloudstack-agent start
For KVM hosts, upgrade the cloudstack-agent
package
Configure the CloudStack RPM repository as detailed above.
$ sudo yum upgrade cloudstack-agent
Verify that the file
/etc/cloudstack/agent/environment.properties
has a line that reads:paths.script=/usr/share/cloudstack-common
If not, add the line.
Restart the agent:
$ sudo service cloudstack-agent stop $ sudo service cloudstack-agent start
Restart management services
If upgrading fresh installation of 4.4.0
If you are upgrading fresh installation of CloudStack 4.4.0, the following MySQL command must be executed before restarting the management server. If the system was running pre 4.4 and then upgraded to 4.4.0, the MySQL command is not required. Refer to: CLOUDSTACK-7813
use cloud; ALTER TABLE `snapshot_policy` ADD `display` TINYINT( 1 ) NOT NULL DEFAULT '1';
Now it’s time to start the management server
$ sudo service cloudstack-management start
If you use it, start the usage server
$ sudo service cloudstack-usage start
System VMs and Virtual Routers
From Apache CloudStack version 4.17.0 onward, there is support to live patch System VMs, namely, SSVM, CPVM, Routers. Live patching provides support for zero-downtime upgrades, wherein, the System VM software is updated to the latest code version without having to destroy and recreate them / restart them.
With this feature, users will have a choice wherein they can use the existing System VM Template with the latest software by using the live patch feature, or can follow the usual workflow of restarting the system VM to use the latest System VM Template. Live Patching System VMs serves to be especially useful in cases when the code version has upgraded but the Template hasn’t. In such a scenario users will no longer need to restart the System VMs to use the latest code.
When one attempts to live-patch the System VMs, it pretty much mimics the patching process that happens when booting up the System VMs but without having to shut down the System VMs. This will update the software packages, which were previously bundled in the systemvm.iso i.e., agent.zip and cloud-scripts.tgz and restart the services that are present in the /var/cache/cloud/enabled_svcs file in the System VMs.
Note
The following services will be restarted once a system VM is live patched:
System VM
Services
SSVM
cloud, apache2, portmap
CPVM
cloud
VRs
haproxy, apache2, dnsmasq
With respect to VRs, a Network restart without cleanup is initiated to during live patching to ensure all rules are re-applied.
NOTE: In cases where upgrading the system VM Template is necessary due to availability of security patches or updated packages in the Template, or in case live-patch fails for system VMs and virtual routers due to any issues or limitations (such as VPC Networks without any Network tiers) then please follow the traditional method of upgrading system VMs and virtual routers by restarting or recreating the system VMs and virtual routers (including restarting the Network with/without cleanup as required), which could mean some downtime.
Following matrix lists the versions of CloudStack that support live patching.
ACS Version
Upgrade Version
Live Patching Support
Reason / Comment
<=4.13
4.17+
No
Update in the openJDK version
4.14
4.17+
Yes
May notice some issue with remove access VPN due to older version of Strongswan
>=4.15
4.17+
Yes
N/A
In addition to the support for live patching, users still have the facility to follow the legacy workflow of restarting the system VMs once the packages on the management servers have been upgraded. Here you’ll need to restart the system VMs in order for those VMs to be rebuilt from the new system VM Template version.
Note
Restarting system VMs can be done in different ways. You can use script “cloudstack-sysvmadm” which is provided with CloudStack, or do a manual restart of system VMs or do it by using third-party tools such as Ansible. Below we are giving instructions for using the “cloudstack-sysvmadm” script.
Ensure that the admin port is set to 8096 by using the “integration.api.port” global parameter. This port is used by the cloudstack-sysvmadm script at the end of the upgrade procedure. For information about how to set this parameter, see configuration parameters Changing this parameter will require a management server restart.
If you run the cloudstack-sysvmadm script from outside the management server, make sure port 8096 is open in your local host firewall.
Warning
Never allow access to port 8096 from the public internet! The management server accepts API calls without authentication on this port, which can pose a serious security risk.
There is a script that will do this for you, all you need to do is run the script and supply the IP address for your MySQL instance and your MySQL credentials:
# nohup cloudstack-sysvmadm -d IPaddress -u cloud -p password -a > sysvm.log 2>&1 &
You can monitor the log for progress. The process of restarting the system VMs can take an hour or more.
# tail -f sysvm.log
The output to sysvm.log
will look something like this:
Stopping and starting 1 secondary storage vm(s)...
Done stopping and starting secondary storage vm(s)
Stopping and starting 1 console proxy vm(s)...
Done stopping and starting console proxy vm(s).
Stopping and starting 4 running routing vm(s)...
Done restarting router(s).
After the upgrade process is complete, you can disable unauthenticated API access again by setting “integration.api.port” to 0. Don’t forget to restart the management server afterwards.