Setting up Vault on Jenkins Pipelines for Production

2021-08-22

Jenkins has been providing top notch CI/CD systems for many years, and integrating Vault API into Jenkins is not really hard. But getting it done for production might be complicated depending on your requirements and the amount of security you might be needing in your company.

Using the Jenkins Hashicorp Vault is the most straightforward way of using Vault credentials inside your Jenkins pipeline. The usage is pretty much straightforward if you are a Groovy master. However, if you really need to keep your Jenkinsfile neat, without adding all that groovy stuff, or if you want more power on the vault credentials, adding something like post bash processing, or if you want to use jq to parse the vault output, then we will be going over to another interesting way of setting up Vault with Jenkins, primarily, on the Jenkins worker node.

So let’s get started! 😎

Setting up the Jenkins worker node⌗

On your worker node, make sure you have vault tool pre-installed, and is accessible from the terminal. Check out the Vault Installation guidelines for your linux distro. If you have a stateful VM as a worker node, things should be as simple as just installing the vault command line tool, on the worker node using apt, dnf or pacman, but if you use something more complicated like, a docker instance template for Jenkins worker node, make sure you add vault to the dockerfile, and it is accessible from $PATH.

Creating an access policy⌗

While setting up Vault, it would be wise to create access policies for the worker node, so that any programming error does not cause a huge damage, even if the secret key-value store is versioned. Its just some extra work, that could be prevented, if we could write good policies. I do not write credentials to Vault using Jenkins, so if you don’t write to Vault using Jenkins, it would be safe to just remove write permissions in the access policy.

path "secrets/data/*" {
   capabilities = ["read"]
}

Save it as jenkins.hcl. It is suggested that you store this in a version control.

export VAULT_ADDR="https://10.142.0.45:8200"
export VAULT_SKIP_VERIFY=1

vault policy write jenkins ./jenkins.hcl

Now, you will have a jenkins policy ready on Vault, ready to use.

For writing advanced policies, you will need to consider Vault’s documentation on Policies

Creating a App Role⌗

There are a few different methods of authenticating to the Vault host. The two important ways are:

  1. Token based authentication
  2. App Role based authentication

Token based authentication is one of the most simplest way of authenticating with Vault. The concept is simple: you provide a time-to-live, maybe something like 15 minutes, and associate the token with a policy which we created earlier.

vault token create -policy=jenkins

This will give us a token, which, by default, is valid for 768 hours.

If we would want to create a token which has a shorter TTL, we would use -period parameter

vault token create -policy=jenkins -period=30m

This will give us a token, which is valid for 30 minutes only.

It is recommended to have tokens with short TTLs. This would help us prevent exposure, in case a token is compromised. It is also possible to limit the number of uses of a token.

App Role based authentication is the recommended way of assigning machines access to Vault. Approle auth method provides a secret-id and a role-id. The secret ID, is, by definition… supposed to be secret. Duh! 😎

But, the secret ID and the role ID by themselves do not give access to any of the credentials. That is the neat part. By combining both the secret ID and the role ID, it is possible to retrieve a Vault token, which then can be used to retrieve the credentials. Generally, tokens have a lesser TTL when compared to secret ID. The secret IDs too have TTls.

To enable AppRole Auth Method:

vault auth enable approle

$ vault write auth/approle/role/jenkins-role \
    secret_id_ttl=24h \
    token_num_uses=0 \
    token_ttl=30m \
    token_max_ttl=2h \
    secret_id_num_uses=40 \
    policies="jenkins" 

role_id         fc4e66f8-eadb-437f-8eff-36d7014f6497

This will give a uuid. We can now get a secret ID from that role.

$ vault write -f auth/approle/role/jenkins-role/secret-id

secret_id       28930ad6-5269-47ee-a619-73a6bfef75b7

This will return a secret_id which is also a uuid. We can now use both the secret_id and the role_id to fetch a token. We can get the token by:

vault write auth/approle/login \
  role_id=fc4e66f8-eadb-437f-8eff-36d7014f6497 \
  secret_id=28930ad6-5269-47ee-a619-73a6bfef75b7

and, that would give the token:

Key                Value
---                -----
token              6d8cf699-f9cd-4c4b-8b6c-9fd7cbe8b253

Comparing the AppRole authentication method and the Token auth method, it is very natural to feel that the Token based authentication method is wayy more simpler. And, you might also be confused on how you would combat the TTL requirements on Jenkins. But we will shortly figure that out! 🥳

Adding credentials to Jenkins⌗

The next step would be adding the required credentials to the Jenkins instance. I prefer keeping the Vault address and the Role ID in the Jenkins credentials, if they are used across multiple pipelins.

Creating a bot user on Jenkins⌗

We need to access the Jenkins API, and we will require the credentials for the same. It would be best to create a bot account on Jenkins, and fetch its API key.

Creating Secret ID⌗

Create a root token first:

vault token create

On a VM, where you only have access to, or if you host Vault in a VM, then log on to the VM, and login to Vault.

vault login

And, here is the script!

#!/bin/bash

set -euxo pipefail

export CREDENTIAL_PATH="/home/vault/credentials"
export VAULT_ADDR="https://10.0.0.1:8200"

export JENKINS_USERNAME="vaultbot"
export JENKINS_TOKEN="86bce8f799ba458fb3969bc364b170ef"
export JENKINS_URL="jenkins.example.com"

# fetch and parse the secret ID from Vault
export VAULT_SECRET_ID="$(vault write -format=json -f auth/approle/role/jenkins-role/secret-id | jq -r '.data.secret_id')"

# write the credential XML to a file, which we will upload to Jenkins later 
cat > $CREDENTIAL_PATH/credential.xml <<EOF
<org.jenkinsci.plugins.plaincredentials.impl.StringCredentialsImpl>
  <scope>GLOBAL</scope>
  <id>vault-secret-id</id>
  <secret>$VAULT_SECRET_ID</secret>
</org.jenkinsci.plugins.plaincredentials.impl.StringCredentialsImpl>
EOF

# send the credential to Jenkins using the Jenkins API endpoint
echo "Settting $JENKINS_URL vault-secret-id"
curl -X POST -H content-type:application/xml -d @$CREDENTIAL_PATH/credential.xml "https://$JENKINS_USERNAME:$JENKINS_TOKEN@$JENKINS_URL/credentials/store/system/domain/_/credential/vault-secret-id/config.xml"

Try executing the above script, and check if you can see vault-secret-id in the Jenkins credential manager.

Automating using systemd-timer⌗

If the previous step worked well, we can now automate this bit, using systemd-timer. I like systemd-timer and there is no particular reason why I like it when compared to cron jobs.

Add a systemd-timer,

$ cat /etc/systemd/system/jenkins_script_renew.timer
[Unit]
Description=Renew secrets on Jenkins

[Timer]
OnUnitActiveSec=6h

[Install]
WantedBy=timers.target

$ cat /etc/systemd/system/jenkins_script_renew.service
[Unit]
Description=Renew secrets on Jenkins

[Service]
Type=oneshot
ExecStart=/home/vault/jenkins-vault-secret.sh
User=vault
Group=vault

And drumrolls, 🥁

sudo systemctl enable --now jenkins_script_renew.timer

And you are done!

You can check the status of the service,

sudo systemctl status jenkins_script_renew.service
● jenkins_script_renew.service - Renew secrets on Jenkins
     Loaded: loaded (/etc/systemd/system/jenkins_script_renew.service; static; vendor preset: enab>
     Active: inactive (dead) since Sat 2021-09-18 16:40:59 UTC; 5h 20min ago
TriggeredBy: ● jenkins_script_renew.timer
    Process: 1069147 ExecStart=/home/vault/jenkins-vault-secret.sh (code=exited, status=0/SUCCESS)
   Main PID: 1069147 (code=exited, status=0/SUCCESS)

Now, you may use vault in your pipelines. Just invoke the vault command line tool in the pipelines, and use jq to parse the results.

In the Jenkinsfile

pipeline {
    environment {
        VAULT_SECRET_ID = credentials('vault-secret-id')
        VAULT_ROLE_ID = credentials('vault-role-id')
        VAULT_ADDR = credentials('vault-address')
    }
    stages {
      ...
    }
}

and …

vault kv get -format=json secrets/my-super-secret

PS: Let me know if you face any problems following this tutorial, or if you have a better implementation, let’s have a chat! ☕

Read other posts