Setting up Vault on Jenkins Pipelines for Production
Jenkins has been providing top notch CI/CD systems for many years, and integrating Vault API into Jenkins is not really hard. But getting it done for production might be complicated depending on your requirements and the amount of security you might be needing in your company.
Using the Jenkins Hashicorp Vault is the most straightforward
way of using Vault credentials inside your Jenkins pipeline. The usage is pretty
much straightforward if you are a Groovy master.
However, if you really need to keep your Jenkinsfile
neat, without adding all that groovy stuff, or if you want more power on the vault credentials, adding something like post bash processing, or if you want to use jq
to parse the vault output, then we will be going over to another interesting way of setting up Vault with Jenkins, primarily, on the Jenkins worker node.
So let’s get started! 😎
Setting up the Jenkins worker node⌗
On your worker node, make sure you have vault
tool pre-installed, and
is accessible from the terminal. Check out the
Vault Installation guidelines for your linux distro.
If you have a stateful VM as a worker node, things should be as simple as
just installing the vault
command line tool, on the worker node using
apt
, dnf
or pacman
, but if you use something more complicated like,
a docker instance template for Jenkins worker node, make sure you add
vault to the dockerfile, and it is accessible from $PATH
.
Creating an access policy⌗
While setting up Vault, it would be wise to create access policies
for the worker node, so that any programming error does not cause
a huge damage, even if the secret key-value store is versioned. Its just
some extra work, that could be prevented, if we could write good policies.
I do not write
credentials to Vault using Jenkins, so if you don’t write
to Vault using Jenkins, it would be safe to just remove write permissions
in the access policy.
path "secrets/data/*" {
capabilities = ["read"]
}
Save it as jenkins.hcl
. It is suggested that you store this in a version control.
export VAULT_ADDR="https://10.142.0.45:8200"
export VAULT_SKIP_VERIFY=1
vault policy write jenkins ./jenkins.hcl
Now, you will have a jenkins
policy ready on Vault, ready to use.
For writing advanced policies, you will need to consider Vault’s documentation on Policies
Creating a App Role⌗
There are a few different methods of authenticating to the Vault host. The two important ways are:
Token based authentication is one of the most simplest way of authenticating with Vault. The concept is simple: you provide a time-to-live, maybe something like 15 minutes, and associate the token with a policy which we created earlier.
vault token create -policy=jenkins
This will give us a token, which, by default, is valid for 768 hours.
If we would want to create a token which has a shorter TTL, we would
use -period
parameter
vault token create -policy=jenkins -period=30m
This will give us a token, which is valid for 30 minutes only.
It is recommended to have tokens with short TTLs. This would help us prevent exposure, in case a token is compromised. It is also possible to limit the number of uses of a token.
App Role based authentication is the recommended way of assigning machines
access to Vault. Approle auth method provides a secret-id
and a role-id
.
The secret ID, is, by definition… supposed to be secret. Duh! 😎
But, the secret ID and the role ID by themselves do not give access to any of the credentials. That is the neat part. By combining both the secret ID and the role ID, it is possible to retrieve a Vault token, which then can be used to retrieve the credentials. Generally, tokens have a lesser TTL when compared to secret ID. The secret IDs too have TTls.
To enable AppRole Auth Method:
vault auth enable approle
$ vault write auth/approle/role/jenkins-role \
secret_id_ttl=24h \
token_num_uses=0 \
token_ttl=30m \
token_max_ttl=2h \
secret_id_num_uses=40 \
policies="jenkins"
role_id fc4e66f8-eadb-437f-8eff-36d7014f6497
This will give a uuid
. We can now get a secret ID from that role.
$ vault write -f auth/approle/role/jenkins-role/secret-id
secret_id 28930ad6-5269-47ee-a619-73a6bfef75b7
This will return a secret_id
which is also a uuid. We can now use both the secret_id
and the role_id
to fetch a token. We can get the token by:
vault write auth/approle/login \
role_id=fc4e66f8-eadb-437f-8eff-36d7014f6497 \
secret_id=28930ad6-5269-47ee-a619-73a6bfef75b7
and, that would give the token:
Key Value
--- -----
token 6d8cf699-f9cd-4c4b-8b6c-9fd7cbe8b253
Comparing the AppRole authentication method and the Token auth method, it is very natural to feel that the Token based authentication method is wayy more simpler. And, you might also be confused on how you would combat the TTL requirements on Jenkins. But we will shortly figure that out! 🥳
Adding credentials to Jenkins⌗
The next step would be adding the required credentials to the Jenkins instance. I prefer keeping the Vault address and the Role ID in the Jenkins credentials, if they are used across multiple pipelins.
Creating a bot user on Jenkins⌗
We need to access the Jenkins API, and we will require the credentials for the same. It would be best to create a bot account on Jenkins, and fetch its API key.
Creating Secret ID⌗
Create a root token first:
vault token create
On a VM, where you only have access to, or if you host Vault in a VM, then log on to the VM, and login to Vault.
vault login
And, here is the script!
#!/bin/bash
set -euxo pipefail
export CREDENTIAL_PATH="/home/vault/credentials"
export VAULT_ADDR="https://10.0.0.1:8200"
export JENKINS_USERNAME="vaultbot"
export JENKINS_TOKEN="86bce8f799ba458fb3969bc364b170ef"
export JENKINS_URL="jenkins.example.com"
# fetch and parse the secret ID from Vault
export VAULT_SECRET_ID="$(vault write -format=json -f auth/approle/role/jenkins-role/secret-id | jq -r '.data.secret_id')"
# write the credential XML to a file, which we will upload to Jenkins later
cat > $CREDENTIAL_PATH/credential.xml <<EOF
<org.jenkinsci.plugins.plaincredentials.impl.StringCredentialsImpl>
<scope>GLOBAL</scope>
<id>vault-secret-id</id>
<secret>$VAULT_SECRET_ID</secret>
</org.jenkinsci.plugins.plaincredentials.impl.StringCredentialsImpl>
EOF
# send the credential to Jenkins using the Jenkins API endpoint
echo "Settting $JENKINS_URL vault-secret-id"
curl -X POST -H content-type:application/xml -d @$CREDENTIAL_PATH/credential.xml "https://$JENKINS_USERNAME:$JENKINS_TOKEN@$JENKINS_URL/credentials/store/system/domain/_/credential/vault-secret-id/config.xml"
Try executing the above script, and check if you can see vault-secret-id
in the Jenkins
credential manager.
Automating using systemd-timer
⌗
If the previous step worked well, we can now automate this bit, using systemd-timer
.
I like systemd-timer
and there is no particular reason why I like it when compared to
cron jobs.
Add a systemd-timer
,
$ cat /etc/systemd/system/jenkins_script_renew.timer
[Unit]
Description=Renew secrets on Jenkins
[Timer]
OnUnitActiveSec=6h
[Install]
WantedBy=timers.target
$ cat /etc/systemd/system/jenkins_script_renew.service
[Unit]
Description=Renew secrets on Jenkins
[Service]
Type=oneshot
ExecStart=/home/vault/jenkins-vault-secret.sh
User=vault
Group=vault
And drumrolls, 🥁
sudo systemctl enable --now jenkins_script_renew.timer
And you are done!
You can check the status of the service,
sudo systemctl status jenkins_script_renew.service
● jenkins_script_renew.service - Renew secrets on Jenkins
Loaded: loaded (/etc/systemd/system/jenkins_script_renew.service; static; vendor preset: enab>
Active: inactive (dead) since Sat 2021-09-18 16:40:59 UTC; 5h 20min ago
TriggeredBy: ● jenkins_script_renew.timer
Process: 1069147 ExecStart=/home/vault/jenkins-vault-secret.sh (code=exited, status=0/SUCCESS)
Main PID: 1069147 (code=exited, status=0/SUCCESS)
Now, you may use vault in your pipelines. Just invoke the vault
command line tool in the
pipelines, and use jq
to parse the results.
In the Jenkinsfile
pipeline {
environment {
VAULT_SECRET_ID = credentials('vault-secret-id')
VAULT_ROLE_ID = credentials('vault-role-id')
VAULT_ADDR = credentials('vault-address')
}
stages {
...
}
}
and …
vault kv get -format=json secrets/my-super-secret
PS: Let me know if you face any problems following this tutorial, or if you have a better implementation, let’s have a chat! ☕
Read other posts