Jenkins setup for testers
Jenkins is commonly used CI/CD tool in Quality Assurance field. It can be configured in various ways depending on purpose. Different setup might be done if it’s needed for regression tests running before production release and different to support feature testing process. Numerous amount of Jenkins plugins can support your custom setup. In this article I will present what kind of setup and which plugging help my team to utilise Jenkins features in most productive way.
Jobs setup
Flexible usage
To be able to use Jenkins setup in optimised way, your test automation should be built in compliance with major principles:
1. Avoid hardcoded URLs with fixed environment indicator. Make your environment a variable, so that tests can run on multiple environments by just passing parameter to script. The same can be done with domain, if your project uses more than one and with country if scope of tests include multiple shops.
For example having a function which determines env:
one test case can run on any environment with any country you want by just calling you script with parameters:
robot -v env:test03 -v domain:shopxyz -v country:ch DetermineFrontendUrl.robot
2. Avoid hardcoded browser setup. For all UI tests it should be possible to execute test on multiple browsers by passing browser name as parameter.
Once having your test automation setup as above, it is possible to configure parameterised Jenkins job. When creating new job in ‘General’ section there is a checkbox called ‘This project is parameterised’. By ticking this checkbox you can add custom parameters:
Default value can be pre-filled or left empty. Once left empty, when running your job you will have to put parameter manually unless job inherits parameters from other build. By having default value, it will run your job with given parameter unless you provide a different one. Once building your job you’ll still have a chance to run it with different parameter than default one:
Concurrent execution
Normally Jenkins will execute one build and after it’s finished, new build can start. Since our job is configured with parameters, it might be the case that multiple builds of the same job should run in parallel with different parameters. The most common use case is to run same test scenario agains different browsers. To allow Jenkins to start multiple builds simultaneously tick checkbox ‘Execute concurrent builds if necessary’ in General tab:
Execution sequence
Even though one of the most important test automation principle says your test cases should be independent, in reality it’s not always truth. In order to complete E2E testing, test cases or even whole test suites might depend on each other. In such cases tests should be run in organised sequence and every next build should depend on previous build execution. Jenkins provides suitable setup in ‘Build Triggers’ section. By marking ‘Build after other projects are built’ checkbox you can specify after which build your current build should be triggered. Moreover you are able to define to trigger current job only if previous job passed or regardless it failed. Once you need specific jobs to run in sequence, and in case of first job failure there is no point to execute any further tests ‘Trigger only if build is stable’ radio button might be particularly useful.
Always up-to-date
Regardless which Version Control System do you use, your Jenkins jobs should pull directly from VCS. With such setup you’ll make sure changes made by other team members will always be included into build run. In my organisation we use GitLab and Jenkins is setup to pull from GitLab before starting any job. In ‘Source Code Management’ tab you need to specify repository and branch. By default master branch is set but it can be changed to pull from any branch you want.
Useful plugins
Jenkins provides a numerous amount of plugins which allow to customise your project and fulfil specific requirements. I will not go through all of them as this can be reviewed on official Jenkins website: https://plugins.jenkins.io/, but instead I will give you a list of these which I use in my organisation.
This plugin allows to re-run failed jobs automatically. It happens randomly, especially for UI tests, that job fails without a valid reason e.g browser unexpectedly crushes. Build fails then, but just to be sure that failure is not caused by bug in application it is recommended to automatically re-run test after a failure. If job runs twice and first build fails but second passes, it gives more confidence to make a statement that application is bug free and first failure was caused by unrelated circumstances. Once Naginator plugin is installed, go to ‘Post-build Actions’ tab and choose ‘Retry build after failure’ option from dropdown.
You can set a maximum number of retries, but I would limit it to 1 retry. If job fails more than once in a row it already can be considered suspicious and worth checking the logs from test execution.
This plugin allows to to prioritise jobs. Let’s assume you have a set of jobs which you want to execute as regression tests, but before you want to run sanity check. You can create 2 different views and include dedicated tests into list tests.
Let’s say that ‘Sanity check’ list contains job A and job B and Regression test C and test D. Once creating additional job which will trigger all tests, you can set priorities. After installing ‘Parameterised Trigger’ plugin additional entry in Dashboard menu appears — ‘Job Priorities’. Once setting a priority for jobs in ‘Sanity check’ list, these items will always run as first.
This plugin allows to inherit parameters from other jobs. In case you want to run set of jobs with the same parameter, you can create additional job, set parameters for this specific job and include other existing job. It will not matter what kind of values will these included jobs have set as parameters, because as long as these jobs will be triggered by your new item, they will inherit parameters from this item.
Once the plugin is installed, go to ‘Build’ tab and add new build step. From dropdown choose ‘Trigger/call builds on other projects’ option and add build triggers. What is important and very useful, in ‘Projects to build’ field you can specify not only which jobs should be run but also the sequence of execution. Jobs will not run in alphabetical order, but in sequence which you will provide. By adding ‘Current build parameters’ option, you will force all jobs included to run with parameters inherited from the parent job.
In my company Slack is used as a communicator within entire organisation. Among numerous integration possibilities with wide range of apps, Slack provides integration with Jenkins. This is very convenient way of checking test execution output for regression runs. In my company we release to production on a daily basis. Before every release there is a nightly scheduled run of regression tests. Quality Assurance Engineers need to verify builds execution by just checking Slack messages. Slack can be notified on all events related to build, when it starts, is aborted, succeeds etc. However I find it most handy to set up Slack notifications in case build fails.
Directly from Slack channel QA Engineer can review logs related to build execution and verify test failure:
Jenkins can be utilised different ways. Apart from nightly regression, we use Jenkins to support our feature testing process. By running regression or some specific suites we are able to identify bugs on an early stage. For such purpose there is no need to send messages to Slack. But it’s still ok to populate among QA team the same Jenkins setup as for nightly regression tests, because as long as your QA Engineers will not use ‘Slack Notification’ plugin on their local machines, this job setup will be ignored.
Summary
Jenkins can be utilised many different ways. Setup presented in this article works very well for me and my team, but this is my subjective evaluation. You might not agree with it and that’s fine. Jenkins gives a freedom to customise your jobs in a way it suits you and your team best. There is no golden rule. Learning by doing led me to have my perfect setup. The best way to find your own is to play with different configurations and plugins.
