Automating getting CI/CD variables from gitlab

Tips & Tricks - Public
, ,
1

Whenever I have to setup private.yml when working setting up a grove cluster repository locally, it’s annoying to have to manually copy CI/CD variables into the file. I decided to automate it a bit.

First of all, we can get variables using GitLab’s API (gitlab docs):

curl --header "PRIVATE-TOKEN: <your access token with api access>" "https://gitlab.com/api/v4/projects/<project_id>/variables"

Then you would want to parse JSON. jq is a great command for that, and you can get all the variables in the form <name>: <value> you can do the following (assumes you’ve saved the output of the previous command to ci-cd-variables.json):

jq '.[] | .key + ": " + .value' ci-cd-variables.json --raw-output

We can go even further by using sed to automatically put values into the private.yml.

:warning: This will make changes to your private.yml.

jq '.[] | .key + ": .*" + "," + .key + ": " + .value' ci-cd-variables.json --raw-output | xargs -i sed -i -e 's,{},' private.yml

You can double check that the change are correct using:

diff private.yml private.yml.example
7 Likes
2

Adding my script that I use with 1Password to populate the Grove’s private.yml based on @maxim’s tip.

#!/bin/bash

ONEPASSWORD_SECRET_NAME="change-my-value"
PROJECT_ID="change-my-value"

# Get the Gitlab token from 1Password
echo "Getting the Gitlab Access Token from 1Password"
GITLAB_TOKEN=$(op item get "$ONEPASSWORD_SECRET_NAME" --format json --fields token | jq ".value" --raw-output)

# Create a new private.yml from template
cp private.yml.example private.yml

# Saving the API Response to ci-cd-variables.json
echo "Getting the CI/CD vars from Gitlab"
curl --header "PRIVATE-TOKEN: $GITLAB_TOKEN" "https://gitlab.com/api/v4/projects/$PROJECT_ID/variables" -o ci-cd-variables.json

# Update private.yml
echo "Updating private.yml"
jq '.[] | .key + ": .*" + "," + .key + ": " + .value' ci-cd-variables.json --raw-output | xargs -i sed -i -e 's,{},' private.yml
# Add the gitlab token
sed -i -e 's/GITLAB_PASSWORD: setme/GITLAB_PASSWORD: '$GITLAB_TOKEN'/' private.yml

This throws a sed error and messes up the NEW_RELIC_MONITORING_EMAILS: value. I didn’t spend time debugging it. Just edited it manually.

2 Likes