Tempest test resource cleanup

Registered by Andrea Frittoli

The cleanup/release of test resources created/allocated in the class level
test fixtures is invoked in the class level tearDown fixure.
However tearDownClass is invoked by the unittest framework only in case
setUpClass is successful. This is causing resources being leaked when:

- a skip exception is raised after resources (typically test accounts)
  have already been allocated
- there is a temporary failure in the system under test which causes the
  setUpClass to fail

The test-accounts bp introduces the possibility to run parallel tests
using a configured list of pre-provisioned test accounts. Test accounts
are allocated and released by each test class, and a failure to release
leads to exhaustion of test accounts.

Blueprint information

Matthew Treinish
Andrea Frittoli
Andrea Frittoli
Series goal:
Proposed for kilo
Milestone target:
milestone icon kilo-rc
Started by
Andrea Frittoli
Completed by
Andrea Frittoli


Gerrit topic: https://review.openstack.org/#q,topic:bp/resource-cleanup,n,z

Addressed by: https://review.openstack.org/118352
    add bp:resource-cleanup

Addressed by: https://review.openstack.org/121537
    Framework for resource safe class level fixtures

Addressed by: https://review.openstack.org/121538
    Migrate computev2 API tests to resource_* fixtures

Addressed by: https://review.openstack.org/121539
    Migrate computev3 API tests to resource_* fixtures

Addressed by: https://review.openstack.org/121540
    Migrate identity API tests to resource_* fixtures

Addressed by: https://review.openstack.org/121541
    Migrate network API tests to resource_* fixtures

Addressed by: https://review.openstack.org/121542
    Migrate baremetal API tests to resource_* fixtures

Addressed by: https://review.openstack.org/121543
    Migrate database API tests to resource_* fixtures

Addressed by: https://review.openstack.org/121544
    Migrate data_processing API tests to resource_* fixtures

Addressed by: https://review.openstack.org/121545
    Migrate image API tests to resource_* fixtures

Addressed by: https://review.openstack.org/121546
    Migrate object_storage API tests to resource_* fixtures

Addressed by: https://review.openstack.org/121547
    Migrate orchestration API tests to resource_* fixtures

Addressed by: https://review.openstack.org/121548
    Migrate queuing API tests to resource_* fixtures

Addressed by: https://review.openstack.org/121549
    Migrate telemetry API tests to resource_* fixtures

Addressed by: https://review.openstack.org/121550
    Migrate volume API tests to resource_* fixtures

Addressed by: https://review.openstack.org/121551
    Migrate scenario API tests to resource_* fixtures

Addressed by: https://review.openstack.org/121552
    Migrate thirdparty API tests to resource_* fixtures

Addressed by: https://review.openstack.org/121553
    Migrate cli API tests to resource_* fixtures

Addressed by: https://review.openstack.org/121554
    Hacking rule to forbid resource unsafe fixtures

Addressed by: https://review.openstack.org/122531
    Drop unused safe_setup decorator

Addressed by: https://review.openstack.org/124896
    orchestration: check for attributes in resource_cleanup

Addressed by: https://review.openstack.org/126134
    Remove setUpClass added after cleanup

Addressed by: https://review.openstack.org/115353
    Framework for staged setup
- https://etherpad.openstack.org/p/resource-cleanup-tempest-2

Addressed by: https://review.openstack.org/141183
    Split resource_setup for volumes tests

Addressed by: https://review.openstack.org/141424
    Update compute base test to split up resource_setup

Addressed by: https://review.openstack.org/142584
    Reorder setup and cleanup functions for readability

Addressed by: https://review.openstack.org/142595
    Split resource_setup for servers tests

Addressed by: https://review.openstack.org/142909
    Split resource_setup for security-group tests

Addressed by: https://review.openstack.org/143160
    Split resource_setup for limit tests

Addressed by: https://review.openstack.org/150312
    Split resource_setup for network tests

Addressed by: https://review.openstack.org/151610
    Split resource_setup for volume tests

Addressed by: https://review.openstack.org/152105
    Split resource_setup for object_storage tests

Addressed by: https://review.openstack.org/152412
    Split resource_setup for data_processing tests

Addressed by: https://review.openstack.org/152417
    Split resource_setup for baremetal tests

Addressed by: https://review.openstack.org/152878
    Split resource_setup for compute tests

Addressed by: https://review.openstack.org/153158
    Split resource_setup for database tests

Addressed by: https://review.openstack.org/153205
    Split resource_setup for identity tests

Addressed by: https://review.openstack.org/153215
    Split resource_setup for image tests

Addressed by: https://review.openstack.org/153621
    Drop the legacy and un-used _interface

Addressed by: https://review.openstack.org/153681
    Initial class creds creation in test base class

Addressed by: https://review.openstack.org/153764
    Split resource_setup for messaging tests

Addressed by: https://review.openstack.org/153765
    Split resource_setup for orchestration tests

Addressed by: https://review.openstack.org/153766
    Split resource_setup for telemetry tests

Addressed by: https://review.openstack.org/153896
    Split creds and resources in tenant isolation

Addressed by: https://review.openstack.org/154693
    Use cls instead of self in classmethod

Addressed by: https://review.openstack.org/159256
    Split resource_setup for scenario tests

Addressed by: https://review.openstack.org/159496
    Split resource_setup for cli tests

Addressed by: https://review.openstack.org/159894
    Split resource_setup for thirdparty tests

Addressed by: https://review.openstack.org/162721
    Make scenario tests requiring admin skip if no admin creds

Addressed by: https://review.openstack.org/165019
    Remove redundant calls to clear_isolated_creds

Gerrit topic: https://review.openstack.org/#q,topic:bp/resource-cleanup-closure,n,z

Addressed by: https://review.openstack.org/171980
    Extend credentials to support roles

Addressed by: https://review.openstack.org/168366
    Remove unused methods

Addressed by: https://review.openstack.org/168367
    Extend get_client_manager to support roles

Addressed by: https://review.openstack.org/173338
    Move resource-cleanup to implemented

Addressed by: https://review.openstack.org/182355
    Introduce creds_provider in test.py

Addressed by: https://review.openstack.org/182356
    Move identity_version to class level


Work Items

Work items:
Define base fixtures: DONE
Migrate base classes and tests to use the new framework (multiple patches): DONE
Hacking rule to prevent overriding of setUpClass and tearDownClass: DONE
Split resource_setup in multiple steps in base class: DONE
Migrate resource_setups to multiple steps (multiple patches): DONE

This blueprint contains Public information 
Everyone can see this information.