Since the last decade or two, there have been tremendous changes in the software development methodologies. Varieties of tools besides agile development, extreme programming, and test-driven development have changed the philosophy and are still modifying how the software development takes place. Needless to say, there is a constant need for change in developing and owning a software development. During the course of software development project, however, there is a need for continuous maintenance and growth of the software product as it can be hit by an unexpected and unplanned change.
Build Verification Test (BVT) or the Build Acceptance Test is a set of tests that help the testing team to go a step back for a step ahead. These tests are run on Builds- each one, and newly created one to ensure that a given module is ready for testing. These tests are the core functionality test cases that ensure that the application is stable and can be tested thoroughly. Typically, the entire process is automated. The test helps subsequent stages and team side-step half-baked work and builds that are too early for testing. When these tests give a green signal, they lead the build to verification stages and bug-fixing tables. In simpler words, BVTs are run on daily builds. If the BVT fails, the build is rejected and a new build is released after the bugs are fixed.
The build tests address core functionality test cases, application stability and are thorough in their focus. This can entail main functions of the module, data-and-format alignment, and addition of files or improvements, new and improved files and culminate into overall project efficiency by making room for discovery and correction of defects before delivery.
Keep in mind that the success of BVT depends on which test cases are included in BVT. For that to happen, one has to consider the following points:
- Include only critical test cases in BVT
- Write effective test case scripts
- Include right file formats
- Include useful information for yes/no diagnosis
- Peek into, and flags around failure cause, choice and degree of automation, insertion of break-build penalties, the presence of all new modified files etc.
- All the test cases that are included should have known expected results
- Ensure that all included critical functionality test cases are sufficient for application test coverage
Only critical test cases
The most important point to consider is the choice of test cases, given its regression-test flavour. There is a lot of thrust on inclusion and exclusion. Hence, picking only critical test cases and including test coverage focus is a good approach to start with. This can facilitate a new critical case with the consistency of different configuration check. This helps avoid a build failure that is prompted by modules and test cases which are unstable or new.
It is worth knowing that tests take time, resources and server muscles and so it is pertinent to run those tests first where propensity of spotting failures is more or the improved time to fault detection comes in. This is helpful to shave away necessary time, cost, resources and completeness of a build along with ease for testing teams. Criticality and application test coverage should be balanced well in the selection of tests. Also, important functions of the module, clarity on expected results, verification-strength or implementable-readiness for application functions is vital to be considered.
Tests should be stable
These tests should devote adequate time to making cases and go for stable test cases. It is less probable to weed out critical bugs or software-breakers if test cases are not formulated well. That is the reason why a build has to be stable for strong and gainful execution of test cases in the ensuing testing process. Likewise, careful design along with the coverage of basic functionality is immensely valuable.
Presently, Agile is one of the most commonly used methodologies in software development life cycle, with its pre-defined short iteration cycles for 2–3 weeks that delivers a quality product. QA teams work with developers to achieve the goal of BVT. Under BVT, the developers set up or create the VMs/Machines for testing, set up the environment build deployment, verify the build and inform the QA team to start further testing.
If this entire process is automated, the BVT process will easily save around 3–4 man days on each iteration besides being reliable and without manual errors. To ensure the success, following steps will be followed:
1. Set up Bamboo or Jenkins in a way such that the nightly test builds would start at the end of every sprint or manually. This pulls the latest code from the GIT/SVN, executes unit test cases, generates the build and uploads the build on Artifactory. Once the build is uploaded; it will start the automated BVT plan.
2. The VMs or machines can be generated automatically using Amazon Web Service, Dockers or VMWare ESXi Server which have their own required software already installed. APIs can be called using Python, Java or even shell script, whichever is applicable.
3. After the VMs are available, ensure that the web application server and database software has been installed successfully. To automate this option, Ansible can be considered.
4. With the latest build from Artifactory, deploy it on Web Application server with the help of Ansible. Once it is deployed successfully, check if it can access a web page.
5. After the build is deployed, start executing the pre-defined test cases which encapsulate the core functionality of the product. Automate those test cases as per the requirement using any language or tool such as Java, Python, shell script, selenium etc.
6. Add the test step results into Jira directly from the Test script to avoid manual dependencies.
7. Send the final status email to the manual QA team that the build is stable/not stable for further testing.
The above is one of the examples to automate build verification test process in Agile.
It is quite understandable that the automated acceptance tests play the key role in software development process. Automated BVT is very effective for daily as well as long-term builds and helps save time, cost, and resources of the company. Given the nature of the build acceptance tests, their size and scope, an assortment of crucial functionality areas would be apt that would refine the coverage of these tests for the overall goal of empowering the actual testing that follows.