Implement upgradeing method of containers with graceful shutdown

In Kolla, upgrades of OpenStack processes are basically conducted by replacing containers, which mainly have three steps: pull new container images, stop existing containers then launch new containers. When upgrading, termination of a container becomes safer by using graceful shutdown implemented in each processes. Using graceful shutdown, requests from OpenStack API users (e.g. end-users, other OpenStack services) will be guaranteed to be finished rather than killed forcibly.

However, during graceful shutdown, new requests to the upgrading containers are not executed for a few hours at worst if running requests need long execution time (e.g. VM snapshots, volume snapshots and VM migration). Because of this, some operations such as deleting/updating some resources, getting resource statuses and auto-scaling become unusable. This is especially fatal for containerized processes responsible for CRUD of resources since requests to them would be sticked to their hosts.

In order to solve this problem, I propose a new method for upgrading containers. In This method, launch new containers at first before terminating exsisting containers. After checking new containers are launched, then gracefully terminating existing containers. With this method, running requests in existing containers are guaranteed to be finished safely by graceful shutdown and any OpenStack APIs are usable since new requests are handled by new containers during graceful shutdown.

I first implement this method into upgrades of Nova services, since Nova is an advanced component in easy upgrading, which has features such as graceful shutdown, RPC version capping and online DB scheme migration. Easy upgrading has been discussed in OpenStack community, and implementation of required features are progressing in components other than Nova. After these features are implemented, I will make available the upgrading method proposed in this BP to other components.