Setup virtualenv for Jenkins Python CI Task

When setting up CI for python projects, virtualenv could help to separate the test/deployment environment. This post will provide a template for Jenkins to use virtualenv for Jenkins tasks with Jenkins example.

Common Test Script Template

Here is the script I used on Jenkins

#!/usr/bin/env bash
########################
# Step 1: Setup virtualenv
# This step is only for Jenkins. Travis and CircleCI will ignore this step.
########################

if [ -n "${WORKSPACE:+1}" ]; then
    # Path to virtualenv cmd installed by pip
    # /usr/local/bin/virtualenv
    PATH=$WORKSPACE/venv/bin:/usr/local/bin:$PATH
    if [ ! -d "venv" ]; then
            virtualenv venv
    fi
    . venv/bin/activate
else
    # Alternatively, $TRAVIS_REPO_SLUG could be utilized here to provide name.
    export JOB_NAME="ci-stat"
fi
pip install -r requirements.txt -r test/test_requirements.txt --cache-dir /tmp/$JOB_NAME

########################
# Step 2: Execute Test
########################
nosetests --with-xunit --all-modules --traverse-namespace --with-xcoverage --cover-package=me.maxwu --cover-inclusive --logging-level=INFO --debug=me.maxwu -s -v --xunit-file ci-stat_nose_xunit.xml --cover-html ./test

########################
# Step 3: PyLint
########################
cd src
pylint -f parseable -d I0011,R0801 me.maxwu | tee ../pylint.out
cd ..
#EOF

There are two requirements files in my repo: requirements.txt and test/test_requirements.txt. The later one defines the prerequisites for test using as told by name and location. Special attention is recommended to take for nose, coverage, pylint and codecov (if using travis).

The test_requirements.txt has below contents.

nose>=1.3.7
nosexcover>=1.0.11
pylint>=1.6.5
coverage>=4.3.4
# codecov for post-build Travis CI step.
codecov==2.0.5

Configure Jenkins with above test

Build Steps

To utilize above bash script, we can add one shell execution step on Jenkins task management portal.

bash -x test/nosetests_ci-stat.sh

Sometimes the build-step is divided to several steps including virtualenv, nose test, and, lint scoring. Explicit "shell exeuction" steps with Jenkins make the logic clear to maintain and reuse.

Some other times, the test is based on PyPi package and virtualenv could offer a separate environment to install real packages for test using.

Environmental Variables

It is a critical security concern to protect environmental variables if there is configuration items applied through them. For example, I have local config.yaml file to specify my Travis and Circle CI API token, but this config.yaml is in gitignore file. To read such secrets, I configure Jenkins secret as in menu "Build Environment -> Binding". For environmental variables on Travis, it has protection not to show plain text contents in logs so which is secured automatically.

Post Build Steps

Three popular but fundamental plugins will be introduced: - JUnit Test Result Publishment. - Violation Report: The lint scan results. - Cobertura Coverage Report: the coverage data (for given Python package).

+U Maxout!