There are a couple ways to get involved in writing automated tests. The quality team is involved in writing autopkg and autopilot tests for ubuntu packages. Autopilot tests are functional tests, intended to simulate user interaction. Autopkg tests on the other hand are run at build time and are intended to be integration and lower level tests. See below for more information on each kind of testing.


Autopilot


What is autopilot test?

Autopilot is a testing tool capable of introspecting applications and generating keyboard and mouse events. Autopilot tests are utilized for functional testing, including GUI testing and simulating end user interaction. The tests are written in python and can be user executed or scripted via a test runner such as UTAH or jenkins to run and publish results automatically. The quality team maintains a project repository for autopilot testcases.

Writing an autopilot testcase

STEP 1: Choose an application

Choose an application you wish to write the testcase for. An excellent place to begin is by looking at the needed tests from the list. Branch the current testcases by issuing bzr branch lp:ubuntu-autopilot-tests.

STEP 2: Familiarize yourself with Autopilot

Take a look at the basic walk-through, parts one and two. Next, learn more about using the autopilot launch and vis tools to introspect your application by reading last part of the tutorial here. Also be sure to reference the official documentation and tutorial.

STEP 3: Write the test

Run the application you’ve chosen and pick a few of the primary features of the application. Document each feature you’ve chosen and write down step by step instructions in order to utilize the feature. You should template out your planned feature tests, and document each step as a comment. For example, if I want to ensure the bookmark manager in firefox works properly, I would write tests to test the ability to open, save, delete, edit and order a bookmark. These step by step instructions will be transformed into automated actions to be performed by your code.

Run through the steps you wrote down to ensure they exercise the feature you targeted. As you step through your instructions, record what happens for each step so you can add them to the test case. These will become your assertions in the testcase. For example, If I press Ctrl+o while in gedit, the expected result might be that an open window will appear.

Utilize autopilot’s mouse, keyboard, and data manipulation to perform the same steps. Write each feature test as a separate test function. Run your code and ensure all your intended actions are performed.

Add in the assertions your made about the state of the application as the code executes your testcase. Every line doesn’t necessarily need an assertion, provided the main actions of the testcase have assertions.

STEP 4: Contribute your new test!

Once your test is ready, commit your changes and push them to a branch in launchpad. Then submit a merge request that includes your new testcase to the project.


I want to help, but I got lost somewhere!

I’d encourage you to visit our wiki pages on contributing testcases for more in-depth instructions and help. In addition, let us know about your work! Send an email to the QA Community Coordinator and/or the mailing list, ubuntu-quality@lists.ubuntu.com. They are happy to help you!

What testcases need to be written?

Testcases for any package in the ubuntu archive is welcome! If you are looking for ideas, here is a list of current work items. Pick an item from the list and have at it!

How do I contribute an autopilot test?

Contributions can be submitted by branching the code, bzr branch lp:ubuntu-autopilot-tests, and submitting merge requests.

What happens to my contribution?

Once accepted and merged, your test will be available to run on the jenkins QA instance. This will allow the test to run against the new builds of the package as they become available.

How do I get more information?

The autopilot team has documentation with further information on writing an autopilot testcase, including examples of tests and a walkthrough on writing testcases.


Autopkg


What’s an autopkg test?

Autopkg tests are run at build time automatically by the buildbots for the package. The goal of these tests is to provide system and integration testing to guarantee basic functionality. Autopackage tests are written and submitted as additions to an individual package. They are low-level tests that verify functionality and are run during package build time. Autopkg tests can be written for any ubuntu package. The tests follow the DEP 8 specification for including tests as part of a deb package.

Writing an autopkg testcase

STEP 1: Choose an application

Choose an application you wish to write the testcase for. An excellent place to begin is by looking at the needed tests from the list.

STEP 2: Familiarize yourself with Autopkg

Take a look at the basic walk-through at contributing an autopkg testcase, including examples of tests to understand how autopkg tests work and function.

STEP 3: Write the test

Start by branch the package you wish to add the test for; bzr branch ubuntu:<packagename>

You’ll need to perform some basic modifications to ready the package for autopkg.

  • Add a source section in debian/control called XS-Testsuite: autopkgtest
  • Add a debian/tests/control which specifies the requirements for the testbed.

Next, add the tests to debian/tests/ folder. A test can be written in a myriad of languages. Common examples are C, bash, python and perl. The tests focus on ensuring low-level compatibility with the basic features of the package. For example, an autopkg test for a pkzip library may check to ensure the library starts without crashing, and can perform zip and unzip features.

STEP 4: Contribute your new test!

Once your test is ready, commit your changes and push them to a branch in launchpad. Then submit a merge request that includes your new testcase to package.


I want to help, but I got lost somewhere!

I’d encourage you to visit our wiki pages on contributing testcases for more in-depth instructions and help. In addition, let us know about your work! Send an email to the QA Community Coordinator and/or the mailing list, ubuntu-quality@lists.ubuntu.com. They are happy to help you!

What testcases need to be written?

Testcases for any package in the ubuntu archive is welcome! If you are looking for ideas, here is a list of current work items. Pick an item from the list and have at it!

How do I contribute an autopkg test?

Getting the test into ubuntu follows the normal ubuntu developer process. In short, you

  1. Branch the source of the package you wish to add a test
  2. Edit the debian/control and debian/tests/control file to enable the tests
  3. Add the test(s) to debian/tests folder
  4. Commit your changes and propose a merge

What happens to my contribution?

Once accepted and merged, your test will be run everytime that package or it’s dependencies are built. You can see the live jenkins output of all the tests that have been contributed and are currently being automatically run here.

How do I get more information?

The ubuntu developer documentation has further information on contributing an autopkg testcase, including examples of tests and a walkthrough on writing testcases.