Process notes: QA

QA process works towards the following goals:

1. Provide a qualitative metric of product quality (number of open and fixed/verified bugs)
2. Provide qualitative metric of functional requirements fulfillment (implemented and verified test cases or user stories against a target total)
3. Ensure stable product quality over time (regression testing)

QA process is based on product requirements and works with artifacts (labeled builds) produced by release management process. Below you will find some key points for the QA process to meet its goals:

• QA activities are applied to each official [labeled] build (the set of applied tests might differ, though)
• Product quality is evaluated against a set of test-cases. Test cases are usually derived into groups (by functional area; by priority) – this allows more flexibility in applying test efforts.
• Each produced [labeled] build should have an associated test report which specifies the tests being executed and a result of execution (positive/negative).
• Each build that’s publicly deployed (i.e. uat/staging/production) should be tested at least against a short-list of critical test cases (usually referred to as ‘smoke test’). If a build does not pass this ‘smoke test’ it can’t (won’t) be deployed anywhere.
• When adding new functionality or updating existing features, a list of test cases is updated to reflect these changes (new features usually result in new test cases being added to this list).
• Having a consistent track of test results provides an immediate insight on:
• Development progress (new test cases mean new functionality being added)
• Stability of each build we produced (number of bugs and their priority) – which greatly simplifies a task of choosing a ‘good enough’ build for urgent delivery
• Amount of work that lies on our plate (blocked test cases + open bugs)

When maintaining automated [selenium] tests with iterative development process, the practice is usually as follows:

1. Assuming we have a sufficient test coverage for the previous iteration
2. During current iteration, QA team does the following:
1. Preparing a list of test cases to cover new functionality (during current sprint there test cases are usually executed manually)
2. Implementing [automated] tests for test cases that were implemented/delivered during previous sprint
3. [optional] backing up critical bugs by automated tests in order to ensure proper regression testing

The reason for automated tests to be one iteration behind development is that in order to develop them, the functionality has to be in place, which is not the case during ‘current’ sprint. With this approach to testing (i.e. using automates tests) a sign of a healthy project is an increase in number of [passing] automated tests after each iteration.

HTH,
AlexS

Process notes: release management

Recently I had to spend some time to produce comments around different aspects of software development. Publishing these notes here is a good way of saving it for future use.

 

Basic definitions and requirements

Artifact – a set of files that we work with or produce. This can refer to a source code drop or to a binaries produced by building the application from this sources.

Process – a repeatable and traceable set of actions that are performed on a given artifact(s).

In scope of release management process we work with the following artifacts:

• Source code – a complete set of project source code together with required configuration files.
• Build – a labeled set of binary files that is sufficient for deployment to any environment we have.

Processes we need to have in place:

1. Build process – takes a source code and produces a labeled build [package].
2. Deployment process – takes a build and deploys it to a specified environment.

Process requirements:

1. Repeatability – we need to be able to repeat any process and be sure that for a given input it will produce exactly the same output.
2. Traceability – we need to be able to trace when a process was initiated, who initiated a process, what were the inputs and what were the outputs/results or process execution.

Security considerations:

1. We’d like to control who can trigger certain processes (like deployment to production).
2. We’d like to control who have influence on the artifacts used to produce the build.

Implementation notes

The crucial aspect of organizing release management process is a definition and labelling of artefacts. Here’s what might help us improve our process:

• Having a dedicated release branch which is used as an input for build process (helps with both repeatability and security)
• Establishing a consistent build labeling mechanism and using the same labels across all the environments we run.
  • Build label should consist of major and minor release number (might be based on sprint number)
  • Should include source control revision number in some form
  • Should be stamped on each binary we produce during the builds – usually embedded in each assembly via AssemblyVersion attribute
  • Should be surfaced somewhere on the web-site – does not necessary visible to end user, can be a hidden field or a hint
    • Simplifies support and solving technical issues
    • Helps in development and testing by making it 100% clear which version is being deployed/tested
    • Allows easy and reliable tracking of what was fixed when and where it is deployed to

Bottom-line:

1. Define consistent build artifacts labeling mechanism
2. Define how to build official (i.e. labeled) builds (release branch or some other strategy)
3. Define responsible personnel and limit access to corresponding artifacts/processes
4. Define how we ensure repeatability (the simplest and most common way is to store official builds somewhere and allowing deployment to be triggered for a given set of [build] artifacts)

HTH,
AlexS