Automated Server testing

Feedback mdz 2009-11-26: Running the upstream test suites during the build is good low-hanging fruit in terms of finding regressions, but this alone is not sufficient for functional testing. The test suite needs to be run regularly, even when the package is not updated, in order to test the complete stack (e.g. find a regression in a dependency). Can we aim to integrate at least some of the upstream tests into checkbox or similar so they can run outside of the build?
Feedback jdstrand 2009-11-30: While the integration work is for QA, the ubuntu-security team can help with adjusting scripts to better integrate into checkbox. As time allows and as is appropriate for our team, we're also happy to help with fixing tests for stable releases, fixing tests that are not as robust as they could be, and adding/updating tests moving forward.
Review mdz 2009-11-30: Looks good to me, setting to Pending Approval for Marjo's review
Re-targeted to beta -2

Status:
on track

Work items for lucid-alpha-2:
 Bug #488708: Fix Puppet test suite: DONE
 Make libvirt run its test suite at build time: DONE
 Implement live step file editing: DONE
 Create step file for kvm-autotest for "Default install" test case: DONE
 Create step file for kvm-autotest for "Bind9" test case: DONE
 Create step file for kvm-autotest for "LAMP" test case: DONE
 Create step file for kvm-autotest for "Mail" test case: DONE
 Create step file for kvm-autotest for "PostgreSQL" test case: DONE
 Create step file for kvm-autotest for "Default installation: LVM partitioning" test case: DONE
 Find a suitable place to run the daily PPA builds: DONE
 Set up a launchpad user to upload PPA builds: DONE
 Create PPA for daily regression test builds: DONE
 Set up cron job to do the daily regression test builds: DONE
 Integrate qa-regression-tests clamav tests into checkbox: POSTPONED
 Integrate qa-regression-tests cups tests into checkbox: POSTPONED
 Integrate qa-regression-tests cyrus-sasl2 tests into checkbox: POSTPONED
 Integrate qa-regression-tests fetchmail tests into checkbox: POSTPONED
 Integrate qa-regression-tests libvirt tests into checkbox: POSTPONED
 Integrate qa-regression-tests mysql tests into checkbox: POSTPONED
 [jdstrand] create a postgresql script for qa-regression-tests: DONE

Work items for lucid-alpha-3:
 [sbeattie] Integrate qa-regression-tests cups tests into checkbox: DONE
 [sbeattie] Integrate qa-regression-tests cyrus-sasl2 tests into checkbox: DONE
 [sbeattie] Integrate qa-regression-tests mysql tests into checkbox: DONE
 [sbeattie] Integrate qa-regression-tests php5 tests into checkbox: DONE
 [sbeattie] Integrate qa-regression-tests shadow tests into checkbox: DONE
 Integrate qa-regression-tests fetchmail tests into checkbox: DONE
 [sbeattie] Integrate qa-regression-tests avahi tests into checkbox: DONE
 [sbeattie] Integrate qa-regression-tests bind9 tests into checkbox: DONE
 [sbeattie] Integrate qa-regression-tests cron tests into checkbox: DONE
 [sbeattie] Integrate qa-regression-tests dovecot tests into checkbox: DONE
 [sbeattie] Integrate qa-regression-tests moin tests into checkbox: DONE
 [sbeattie] Integrate qa-regression-tests openldap tests into checkbox: DONE
 [sbeattie] Integrate qa-regression-tests openssl tests into checkbox: DONE
 [sbeattie] Integrate qa-regression-tests perl tests into checkbox: DONE
 [sbeattie] Integrate qa-regression-tests postfix tests into checkbox: DONE
 [sbeattie] Integrate qa-regression-tests postgresql tests into checkbox: DONE
 [sbeattie] Integrate qa-regression-tests tomcat6 tests into checkbox: DONE

Later:
 Create step file for kvm-autotest for "Live CD session" test case: TODO
 Integrate qa-regression-tests libvirt tests into checkbox: TODO
 Integrate qa-regression-tests clamav tests into checkbox: TODO
 Package http://imapwiki.org/ImapTest: TODO
 Integrate qa-regression-tests net-snmp tests into checkbox: TODO
 Integrate qa-regression-tests apache2 tests into checkbox: TODO
 Integrate qa-regression-tests dhcp3 tests into checkbox: TODO
 Integrate qa-regression-tests dnsmasq tests into checkbox: TODO
 Integrate qa-regression-tests freeradius tests into checkbox: TODO
 Integrate qa-regression-tests ipsec-tools tests into checkbox: TODO
 Integrate qa-regression-tests httpd tests into checkbox: TODO
 Integrate qa-regression-tests libnet-dns-perl tests into checkbox: TODO
 Integrate qa-regression-tests lighttpd tests into checkbox: TODO
 Integrate qa-regression-tests nagios3 tests into checkbox: TODO
 Integrate qa-regression-tests nfs-utils tests into checkbox: TODO
 Integrate qa-regression-tests ntp tests into checkbox: TODO
 Integrate qa-regression-tests nut tests into checkbox: TODO
 Integrate qa-regression-tests pptpd tests into checkbox: TODO
 Integrate qa-regression-tests python tests into checkbox: TODO
 Integrate qa-regression-tests ruby tests into checkbox: TODO
 Integrate qa-regression-tests samba tests into checkbox: TODO
 Integrate qa-regression-tests squid tests into checkbox: TODO
 Package TCK (http://libvirt.org/git/?p=libvirt-tck.git): TODO
 Package TCK's perl dependy #1: TODO
 Package TCK's perl dependy #2: TODO
 Package TCK's perl dependy #3: TODO
 Package TCK's perl dependy #4: TODO
 Create step file for kvm-autotest for "Openssh server" test case: TODO
 Create step file for kvm-autotest for "Print server" test case: TODO
 Create step file for kvm-autotest for "Samba server" test case: TODO
 Create step file for kvm-autotest for "Tomcat server" test case: TODO
 Investigate whether Samba's test suite (or parts thereof) can run without --enable-socket-wrapper: TODO