autopkgtest/proposed-migration work

Nested KVM
=======
 ubiquity testing docs: http://bazaar.launchpad.net/~ubuntu-installer/ubiquity/trunk/view/head:/autopilot/README.md

 ✓ Our ScalingStack instances already use "-cpu host" or something similarly sane. /proc/cpuinfo does not show the "QEMU emulated CPU", but a proper "Intel Xeon". /dev/kvm in the instances works.

 ✓ I constructed a QEMU testbed (adt-buildvm-ubuntu-cloud) and ran autopkgtests using the QEMU runner for udisks2 (munging around a lot with kernel modules and low-level driver ops), kcoreaddons (C++ package build, CPU/memory intense) and ubiquity (autopilot, Xvfb). This works fine in both a local QEMU and a ScalingStack instance.

 ✓ I ran the graphical autopilot installer tests [1] successfully locally in nested QEMU.

 ⚠ Running the graphical autopilot installer tests in ScalingStack fails because they can't checkout lp:ubiquity. This needs to happen via proxy; this can be done with adding a file autopilot/ubiquity-autopilot-runner/custom-installation/iso-override/etc/environment with the proxy settings (I wasn't yet able to make this work with
   the --override option, but I'm probably just missing something trivial there).

 ⚠ The host kernel spews out an awful amount of messages like

    WARNING: CPU: 6 PID: 5998 at /build/linux-jJX2Dm/linux-4.2.0/arch/x86/kvm/svm.c:516 skip_emulated_instruction+0xd3/0x150 [kvm_
+amd]()

   together with kernel stack traces. This causes /var/log/syslog and /var/log/kern.log to grow to several dozen GB in a few minutes. I worked around that by stopping syslog.service and syslog.socket before starting the test.

 ⚠ Once the two issues above have been dealt with, ubiquity tests run, but fail various autopilot checks. They work fine in local nested KVM, so that does smell like an environment issue.

armhf autopkgtests in Scalingstack
=====================
Need to launch containers with

  --config raw.lxc=lxc.aa_profile=unconfined # tests like pbuilder mount /proc

and

  --config "raw.lxc=lxc.seccomp=" # work around bug 1522026

Can't set them both with this, this needs one unified command:

printf "lxc.aa_profile=unconfined\nlxc.seccomp=" | lxc profile set default raw.lxc -

Also need to lower the MTU to match the host:

lxc profile device remove default eth0
lxc profile device add default eth0 nic nictype=bridged parent=lxcbr0 mtu=1400

Work items for ubuntu-15.11:
autopkgtest: set up apt pinning to minimize packages from -proposed (#1503150): DONE
britney: request individual test runs for every trigger (#1503150): DONE
set up ppc64el autopkgtesting on ScalingStack (RT #84975 (done), #85847 (worked around) or #85915 (fallback)): DONE
once ppc64el tests got moved, notify Michael Hohnbaum about decommissioning wolfe: DONE
learn how to set up local MaaS installation in LXC/QEMU: DONE
develop ssh setup script for MaaS: DONE
link administration wiki docs from autopkgtest.u.c.: DONE
test if nested KVM works in ScalingStack in general (yes, tested several autopkgtests): DONE
provide latest autopkgtest in trusty-backports, get permanent approval for that (#1513992): DONE
turn http://autopkgtest.ubuntu.com/running.shtml prototype into something robust, less ugly, better navigatable, more info, integrated with debci: DONE
debci: add list of packages with tests, with links to results and .dsc: DONE
figure out how to expose currently queued tests in AMQP, expose on running.shtml: DONE
add worker arguments and setup commands for multiple PPA apt sources: DONE
britney: Add passthrough option for "ppas" AMQP test requests, for CI train: DONE

Work items for ubuntu-15.12:
autopkgtest worker: generalize backend configuration to also apply to QEMU and MaaS: DONE
figure out firewall changes with IS for ProdStack <-> kernel team MaaS (RT#86742): DONE
set up LXC based autopkgtesting on s390, once hardware is available: DONE
test armhf containers in arm64 scalingstack instances (RT#86895): DONE
worker: run tests from bzr or git branch, expose in AMQP params (can't do checkouts on Prodstack due to Prodstack firewall policy, needs indirection via adt-run): DONE
autopkgtest: Add running tests directly from git: DONE
worker: param for passing additional environment vars to tests: DONE
britney: Directly link test result to log on swift: DONE
britney: with ADT_PPAS config option, search autopkgtest logs in appropriate swift container: DONE

Work items for ubuntu-16.01:
[canonical-is-public] deployment: add enough capacity to DevOps environment (RT#87310): DONE
add SSL cert/conf to autopkgtest.ubuntu.com (prerequisite for web-based retry): DONE
integrate autopkgtest-retrier into debci: DONE
add retry links to excuses.html (needs firewall change, RT#88162): DONE

Work items for ubuntu-16.02:
autopkgtest-retrier: add retries for PPAs: DONE
autopkgtest-retrier: support multiple triggers: DONE
britney: add new hint for package/arch (for all versions): DONE
britney: support multiple hints for different versions/arches: DONE
britney: show overridden regressions more prominently: DONE
give ~canonical-kernel-distro-team the ability to (re)start autopkgtests via CGI: DONE

Work items for ubuntu-16.04:
demonstrate and automate armhf testing in ScalingStack through running lxd containers in an arm64 instances: DONE
britney: don't show tested package version (which is mostly wrong) on excuse lines with only "in progress" tests: DONE
britney: produce update_output_notest.txt with a dry run ignoring test results, for easier transitions: DONE

Work items for ubuntu-16.08:
dpkg: record test dependencies in .dsc (#1491145): DONE
britney: trigger tests for Testsuite-Triggers: DONE
move running.shtml from "polls every 10s" daemon to on-demand cgi-bin: DONE

Work items for ubuntu-16.09:
autopkgtest-web: add back package browsing to title page: DONE
autopkgtest-web: add on-the-fly queues.json: DONE
autopkgtest-web: add back test list: DONE
autopkgtest-web: add back statistics (per-release statistics available now, more upon request): DONE
autopkgtest-web: hide browse.cgi URL part so that /packages URLs don't change: DONE
update documentation for debci → browse.cgi move: DONE
add "all-proposed" flag to request.cgi: DONE

Work items for ubuntu-16.10:
enable armhf testing in Scalingstack: DONE
change retry-autopkgtest-regressions from ssh-snakefruit to request.cgi: DONE
provide request.cgi based script for github tests: DONE