Upgrade Instruction from 4.16.x

This section will show you how to upgrade from CloudStack 4.16.x to latest CloudStack 4.19.1.3.

Any steps that are hypervisor-specific will be called out with a note.

We recommend reading through this section once or twice before beginning your upgrade procedure, and working through it on a test system before working on a production system.

Note

The following upgrade instructions should be performed regardless of hypervisor type.

Overview of Upgrade Steps:

  1. Check any customisations and integrations

  2. Upload the 4.19.1 System VM Template if not already using it.

  3. Stop all running management servers

  4. Backup CloudStack database (MySQL)

  5. Upgrade 1st CloudStack management server

  6. Update hypervisors specific dependencies

  7. Restart 1st management server

  8. Check that your upgraded environment works as expected

  9. Upgrade and restart the remaining management servers

CloudStack Customisations

There are a number of ways in which administrators can customise CloudStack. During an upgrade, a number of these could be overridden. Therefore steps should be taken to ensure that they can be restored after the upgrade is completed.

Guest OS mappings

A new CloudStack release often brings compatibility with new hypervisors, and therefore new Guest OS mappings. An API is provided to manually add guest OSes and the relevant hypervisor mappings, however, there is a high probability that manually added guest OSes and/or mappings would conflict with guest OSes and/or mappings added as part of a version upgrade.

It is therefore essential to remove any Guest OS mappings that were manually added in order to ensure a successful upgrade. If need be, any custom Guest OS mappings still ‘missing’ after an upgrade can be re-added after the upgrade. That means that any custom added rows in the guest_os, guest_os_hypervisor, guest_os_details and guest_os_category database tables, should be removed prior to the upgrade, and added later if needed.

Warning

Manually added guest OS mappings can cause the upgrade process to fail.

Customised CSS

If you have altered the CSS files in order to customise the appearance of the CloudStack UI, you should make a backup copy as the installed CSS files are likely to be overwritten during any upgrade.

You should inspect a ‘diff’ of your customised css files and the new versions, and then reapply your changes to the new files as the new versions may contain changes to better display existing elements or have new entries to support new UI elements.

Plugins

If you have 3rd party plugins installed, you should backup your plugins directories and the plugins.js file. While the plugins directories should remain untouched, the plugins.js file is likely to be overwritten.

3rd Party Integrations

CloudStack is put through extensive regression testing during a release cycle, however the numerous 3rd party integrations which are available cannot all be tested by the community nor indeed may the community know about many of them. Therefore it is essential that you verify that your integrations will continue to work after an upgrade through thorough testing and checking with the vendor/supplier of your integrations.

Warning

If you are not already using the 4.19.1 System VM Template you will need to upgrade your System VM Template prior to performing the upgrade of the CloudStack packages.

Update System VM Templates

Note

From ACS 4.16 onwards, CloudStack will support automatic registration of System VM Templates (when using noredist packages), if not done prior initiating upgrade. However, the usual upgrade process continues to be supported.

  1. While running the existing 4.16.x system, log in to the UI as the root administrator.

  2. In the left navigation bar, click Templates.

  3. In Select view, click Templates.

  4. Click Register Template. The Register Template dialog box is displayed.

  5. To register the System VM Template do the following:

    In the Register Template dialog box, specify the following values (do not change these):

    Hypervisor

    Description

    XenServer

    Name: systemvm-xenserver-4.19.1

    Description: systemvm-xenserver-4.19.1

    URL: http://download.cloudstack.org/systemvm/4.19/systemvmtemplate-4.19.1-xen.vhd.bz2

    Zone: Choose the zone where this hypervisor is used

    Hypervisor: XenServer

    Format: VHD

    OS Type: Other Linux (64-bit)

    Extractable: no

    Password Enabled: no

    Public: no

    Featured: no

    Routing: no

    KVM

    Name: systemvm-kvm-4.19.1

    Description: systemvm-kvm-4.19.1

    URL: http://download.cloudstack.org/systemvm/4.19/systemvmtemplate-4.19.1-kvm.qcow2.bz2

    Zone: Choose the zone where this hypervisor is used

    Hypervisor: KVM

    Format: QCOW2

    OS Type: Debian GNU/Linux 7.0 (64-bit) (or the highest Debian release number available in the dropdown)

    Extractable: no

    Password Enabled: no

    Public: no

    Featured: no

    Routing: no

    VMware

    Name: systemvm-vmware-4.19.1

    Description: systemvm-vmware-4.19.1

    URL: http://download.cloudstack.org/systemvm/4.19/systemvmtemplate-4.19.1-vmware.ova

    Zone: Choose the zone where this hypervisor is used

    Hypervisor: VMware

    Format: OVA

    OS Type: Other Linux (64-bit)

    Extractable: no

    Password Enabled: no

    Public: no

    Featured: no

    Routing: no

    HyperV

    Name: systemvm-hyperv-4.19.1

    Description: systemvm-hyperv-4.19.1

    URL: http://download.cloudstack.org/systemvm/4.19/systemvmtemplate-4.19.1-hyperv.vhd.zip

    Zone: Choose the zone where this hypervisor is used

    Hypervisor: Hyper-V

    Format: VHD

    OS Type: Debian GNU/Linux 7.0 (64-bit) (or the highest Debian release number available in the dropdown)

    Extractable: no

    Password Enabled: no

    Public: no

    Featured: no

    Routing: no

  6. Watch the screen to be sure that the Template downloads successfully and enters the READY state. Do not proceed until this is successful.

Packages repository

Most users of CloudStack manage the installation and upgrades of CloudStack with one of Linux’s predominant package systems, RPM or APT. This guide assumes you’ll be using RPM and Yum (for Red Hat Enterprise Linux or CentOS), or APT and Debian packages (for Ubuntu).

Create RPM or Debian packages (as appropriate) and a repository from the 4.19.1.3 source, or check the Apache CloudStack downloads page at http://cloudstack.apache.org/downloads.html for package repositories supplied by community members. You will need them for Management Server or Hypervisor: KVM hosts upgrade.

Instructions for creating packages from the CloudStack source are in the CloudStack Installation Guide.

Database Preparation

Backup current database

  1. Stop your management server or servers. Run this on all management server hosts:

    $ sudo service cloudstack-management stop
    
  2. If you are running a usage server or usage servers, stop those as well:

    $ sudo service cloudstack-usage stop
    
  3. Make a backup of your MySQL database. If you run into any issues or need to roll back the upgrade, this will assist in debugging or restoring your existing environment. You’ll be prompted for your password.

    $ mysqldump -u root -p -R cloud > cloud-backup_$(date +%Y-%m-%d-%H%M%S)
    $ mysqldump -u root -p cloud_usage > cloud_usage-backup_$(date +%Y-%m-%d-%H%M%S)
    

Management Server

Ubuntu

If you are using Ubuntu, follow this procedure to upgrade your packages. If not, skip to step CentOS/RHEL.

Note

Community Packages: This section assumes you’re using the community supplied packages for CloudStack. If you’ve created your own packages and APT repository, substitute your own URL for the ones used in these examples.

The first order of business will be to change the sources list for each system with CloudStack packages. This means all management servers, and any hosts that have the KVM agent (no changes should be necessary for hosts that are running VMware or Xen.)

Edit your /etc/apt/sources.list.d/cloudstack.list file on any systems that have CloudStack packages installed to points to version 4.19

This file should have one line, which contains:

deb http://download.cloudstack.org/ubuntu bionic 4.19

Setup the public key for the above repository:

wget -qO - http://download.cloudstack.org/release.asc | sudo apt-key add -
  1. Now update your apt package list:

    $ sudo apt-get update
    
  2. Now that you have the repository configured, it’s time to upgrade the cloudstack-management package.

    $ sudo apt-get upgrade cloudstack-management
    
  3. If you use CloudStack usage server

    $ sudo apt-get upgrade cloudstack-usage
    

CentOS/RHEL

If you are using CentOS or RHEL, follow this procedure to upgrade your packages. If not, skip to hypervisors section Upgrade Hypervisors.

Note

Community Packages: This section assumes you’re using the community supplied packages for CloudStack. If you’ve created your own packages and yum repository, substitute your own URL for the ones used in these examples.

The first order of business will be to change the yum repository for each system with CloudStack packages. This means all management servers, and any hosts that have the KVM agent (no changes should be necessary for hosts that are running VMware or Xen.)

Change your /etc/yum.repos.d/cloudstack.repo file on any systems that have CloudStack packages installed to points to version 4.19.

This file should have content similar to the following:

[apache-cloudstack]
name=Apache CloudStack
baseurl=http://download.cloudstack.org/centos/$releasever/4.19/
enabled=1
gpgcheck=0

Setup the GPG public key if you wish to enable gpgcheck=1:

rpm --import http://download.cloudstack.org/RPM-GPG-KEY
  1. Now that you have the repository configured, it’s time to upgrade the cloudstack-management.

    $ sudo yum upgrade cloudstack-management
    
  2. If you use CloudStack usage server

    $ sudo yum upgrade cloudstack-usage
    

Upgrade Hypervisors

Hypervisor: XenServer

No additional steps are required for XenServer Hypervisor for this upgrade.

Hypervisor: VMware

Warning

For VMware hypervisor, CloudStack management server packages must be built using “noredist”. Refer to Building Non-OSS.

No additional steps are required for the VMware Hypervisor for this upgrade.

Hypervisor: KVM

KVM on Ubuntu

(KVM only) Additional steps are required for each KVM host. These steps will not affect running guests in the cloud. These steps are required only for clouds using KVM as hosts and only on the KVM hosts.

  1. Configure the APT repo as detailed above.

  2. Stop the running agent.

    $ sudo service cloudstack-agent stop
    
  3. Update the agent software.

    $ sudo apt-get upgrade cloudstack-agent
    
  4. Start the agent.

    $ sudo service cloudstack-agent start
    

KVM on CentOS/RHEL

For KVM hosts, upgrade the cloudstack-agent package

  1. Configure the CentOS/RHEL as detailed above.

    $ sudo yum install -y epel-release
    $ sudo yum upgrade cloudstack-agent
    
  2. Restart the agent:

    $ sudo service cloudstack-agent stop
    $ sudo service cloudstack-agent start
    

Restart management services

  1. Now it’s time to start the management server

    $ sudo service cloudstack-management start
    
  2. If you use it, start the usage server

    $ sudo service cloudstack-usage start
    

System VMs and Virtual Routers

From Apache CloudStack version 4.17.0 onward, there is support to live patch System VMs, namely, SSVM, CPVM, Routers. Live patching provides support for zero-downtime upgrades, wherein, the System VM software is updated to the latest code version without having to destroy and recreate them / restart them.

With this feature, users will have a choice wherein they can use the existing System VM Template with the latest software by using the live patch feature, or can follow the usual workflow of restarting the system VM to use the latest System VM Template. Live Patching System VMs serves to be especially useful in cases when the code version has upgraded but the Template hasn’t. In such a scenario users will no longer need to restart the System VMs to use the latest code.

When one attempts to live-patch the System VMs, it pretty much mimics the patching process that happens when booting up the System VMs but without having to shut down the System VMs. This will update the software packages, which were previously bundled in the systemvm.iso i.e., agent.zip and cloud-scripts.tgz and restart the services that are present in the /var/cache/cloud/enabled_svcs file in the System VMs.

Note

The following services will be restarted once a system VM is live patched:

System VM

Services

SSVM

cloud, apache2, portmap

CPVM

cloud

VRs

haproxy, apache2, dnsmasq

With respect to VRs, a Network restart without cleanup is initiated to during live patching to ensure all rules are re-applied.

NOTE: In cases where upgrading the system VM Template is necessary due to availability of security patches or updated packages in the Template, or in case live-patch fails for system VMs and virtual routers due to any issues or limitations (such as VPC Networks without any Network tiers) then please follow the traditional method of upgrading system VMs and virtual routers by restarting or recreating the system VMs and virtual routers (including restarting the Network with/without cleanup as required), which could mean some downtime.

Following matrix lists the versions of CloudStack that support live patching.

ACS Version

Upgrade Version

Live Patching Support

Reason / Comment

<=4.13

4.17+

No

Update in the openJDK version

4.14

4.17+

Yes

May notice some issue with remove access VPN due to older version of Strongswan

>=4.15

4.17+

Yes

N/A

In addition to the support for live patching, users still have the facility to follow the legacy workflow of restarting the system VMs once the packages on the management servers have been upgraded. Here you’ll need to restart the system VMs in order for those VMs to be rebuilt from the new system VM Template version.

Note

Restarting system VMs can be done in different ways. You can use script “cloudstack-sysvmadm” which is provided with CloudStack, or do a manual restart of system VMs or do it by using third-party tools such as Ansible. Below we are giving instructions for using the “cloudstack-sysvmadm” script.

Ensure that the admin port is set to 8096 by using the “integration.api.port” global parameter. This port is used by the cloudstack-sysvmadm script at the end of the upgrade procedure. For information about how to set this parameter, see configuration parameters Changing this parameter will require a management server restart.

If you run the cloudstack-sysvmadm script from outside the management server, make sure port 8096 is open in your local host firewall.

Warning

Never allow access to port 8096 from the public internet! The management server accepts API calls without authentication on this port, which can pose a serious security risk.

There is a script that will do this for you, all you need to do is run the script and supply the IP address for your MySQL instance and your MySQL credentials:

# nohup cloudstack-sysvmadm -d IPaddress -u cloud -p password -a > sysvm.log 2>&1 &

You can monitor the log for progress. The process of restarting the system VMs can take an hour or more.

# tail -f sysvm.log

The output to sysvm.log will look something like this:

Stopping and starting 1 secondary storage vm(s)...
Done stopping and starting secondary storage vm(s)
Stopping and starting 1 console proxy vm(s)...
Done stopping and starting console proxy vm(s).
Stopping and starting 4 running routing vm(s)...
Done restarting router(s).

After the upgrade process is complete, you can disable unauthenticated API access again by setting “integration.api.port” to 0. Don’t forget to restart the management server afterwards.