Documentation

Configure Pipelines

To preserve your Jenkins® pipelines and automate their recreation in case of instance failures, we strongly recommend using configuration as code files to set up pipelines.

Operator Service for Jenkins uses job-dsl and kubernetes-credentials-provider plugins for configuring jobs and deploy keys.

We have to start by preparing pipeline and job definitions in your project’s GitHub repository main folder using the following structure:

1
2
3
4
5
cicd/
├── jobs
│ └── build.jenkins
├── pipelines
│   └── build.jenkins

Jenkins will always check those configurations directly from those files in your repository so you don’t have to update the configuration every time you change the pipeline code itself.

Seed Job Definition

A seed job represents Jenkins job that creates one or more pipelines in Jenkins. It uses pipeline configuration files from your GitHub cicd folder. Let’s create a job configuration file and store it at https://github.com/your-project-repo/cicd/jobs/build.jenkins:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
#!/usr/bin/env groovy

pipelineJob('build-operator-service-for-jenkins'){
    displayName('Build Operator Service for Jenkins')

    definition {
        cpsScm {
            scm {
                git {
                    remote {
                        url('https://github.com/your-account/your-repo.git')
                    }
                    branches('*/master')
                }
            }
            scriptPath('cicd/pipelines/build.jenkins')
        }
    }
}
  • pipelineJob – name of pipeline resource that we will create in a while
  • displayName – name of seed job that will be displayed in Jenkins UI
  • remote – here you specify the url of GitHub repository of your project and username you are using in Operator Service for Jenkins, in order to access this repository
  • branches – branches from which you want to access the repo
  • scriptPath – the path in the above repo in which you store pipeline files from which you want to create Jenkins pipelines during this job run

Pipeline Definition

Now we can create the pipeline configuration file and store it at https://github.com/your-project-repo/cicd/pipelines/build.jenkins. This file will create pod with containers to run commands on each step (stage) in your pipeline.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
#!/usr/bin/env groovy

def label ="jenkins-example-${UUID.randomUUID().toString()}"

podTemplate(label: label,
        containers: [
                containerTemplate(name: 'jnlp', image: 'jenkins/inbound-agent:alpine'),
        ],
        ){
    node(label){
        stage('Init'){
            timeout(time: 3, unit: 'MINUTES'){
                checkout scm
            }
        }
        stage('Dep'){
            echo 'Hello from Dep stage'
        }
        stage('Test'){
            echo 'Hello from Test stage'
        }
        stage('Build'){
            echo 'Hello from Build stage'
        }
    }
}
  • label – here specify the name of the pipeline (pipelineJob value from Seed Job)
  • podTemplate – a pod that will be created during this pipeline run, to execute necessary steps
  • containers – containers in pod that will be used to run necessary steps. One jnlp container is always needed. All the others need to use images of programmes needed for the pipeline’s stages. Full list of possible functionalities can be found here. If you need to run kubectl commands you need to use container with image that incorporates kubectl, because it is not provided by default
  • stage – at each stage you specify stage name, scripts, commands and container which needs to run them

Update Jenkins Custom Resource

When you create a seed job and pipeline files, you need to reference the seed jobs in the Jenkins Custom Resources under spec: seed jobs. Jenkins will then create Jenkins jobs from seed job files from your git repository and if you run it in the Jenkins UI, they will create the necessary pipelines. Details about custom resource customization can be found in Customize Operator Service for Jenkins on Azure section of this guide.

1
2
3
4
5
6
7
8
9
10
11
apiVersion: jenkins.io/v1alpha2
kind
: Jenkins
metadata
:
  name
: example
spec
:
  seedJobs
:
  - id
: pipeline-example
    targets
: "cicd/jobs/*.jenkins"
    description
: "Example of seedJob and pipeline definition"
    repositoryBranch
: master
    repositoryUrl
: https://github.com/your-github-account/your-github-repository.git

Operator Service for Jenkins will then automatically discover and configure all the seed jobs.
You can verify if deploy keys were successfully configured in the Jenkins Credentials tab.

Authentication

If your GitHub repository is private or you need to authenticate to any other applications, you have to configure SSH or username/password authentication.

Using SSH Keys is a more secure option, while username/password method is good enough with solutions incorporating central location redirecting users for authentication or with multistep authentication.

Generate SSH Keys

To generate a private/public pair of keys we need to type the following command into our repo

1
2
$ ssh-keygen -t rsa -b 2048 -C "[email protected]"
Generating public/private rsa key pair.

Next, you will be asked for a file name. Use the default path:

1
Enter file in which to save the key (/Users/johndoe/.ssh/id_rsa):

Now you can optionally set a password. In that case leave the password empty.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /Users/johndoe/.ssh/id_rsa.
Your public key has been saved in /Users/johndoe/.ssh/id_rsa.pub.
The key fingerprint is:
SHA256:M0HppoJgPAhw2NYS0SVuYpyllpA7MbFfu+U0F0y8EDA [email protected]
The key's randomart image is:
+---[RSA 2048]----+
|      o. .   o   |
|     o .o + +    |
|   ...+o . + .   |
|  E .ooo .. .    |
|   o  + S . .o   |
|  .  o   . o. +  |
| .    .   .o.=   |
|  . .... [email protected]  |
| .oo.o [email protected]*=   |
+----[SHA256]-----+

We need PEM format:

1
$ssh-keygen -p -f /Users/johndoe/.ssh/id_rsa -m pem
1
2
3
4
Key has comment '[email protected]'
Enter new passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved with the new passphrase.

Now we have to copy the content of the public key file

1
pbcopy < /Users/johndoe/.ssh/id_rsa.pub

and add it to our GitHub repository.

In your project repository enter Settings > Deploy keys and click “Add deploy key”. Give it some title and paste our copied key.

We can see the key was added.

Configure SSH authentication

First, create a Secret file with your GitHub username and generated SSH private key.

Copy contents of the id_rsa file (not id_rsa.pub):

1
pbcopy < /Users/johndoe/.ssh/id_rsa

And paste it into privateKey field like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
apiVersion: v1
kind
: Secret
metadata
:
  name
: k8s-ssh
  labels
:
    "jenkins.io/credentials-type"
: "basicSSHUserPrivateKey"
  annotations
:
    "jenkins.io/credentials-description"
: "ssh github.com:your-account/your-repository"
stringData
:
  privateKey
: |
   -----BEGIN PRIVATE KEY-----
    MIIJKAIBAAKCAgEAxxDpleJjMCN5nusfW/AtBAZhx8UVVlhhhIKXvQ+dFODQIdzO
    oDXybs1zVHWOj31zqbbJnsfsVZ9Uf3p9k6xpJ3WFY9b85WasqTDN1xmSd6swD4N8
    ...

  username
: github_user_name

Second, create a Kubernetes Secret resource from this Secret config file.

1
kubectl -n jenkins apply -f mySecret.yaml

In the seedJob you added to your Jenkins Custom Resource file you can specify basicSSHUserPrivateKey as credentialType and add the name of the Secret, with your GitHub username and SSH key, in credentialID field’s value.

1
2
3
4
5
6
7
8
9
10
11
12
13
apiVersion: jenkins.io/v1alpha2
kind
: Jenkins
metadata
:
  name
: example
spec
:
  seedJobs
:
  - id
: jenkins-operator-ssh
    credentialType
: basicSSHUserPrivateKey
    credentialID
: k8s-ssh
    targets
: "cicd/jobs/*.jenkins"
    description
: "Jenkins Operator repository"
    repositoryBranch
: master
    repositoryUrl
: [email protected]:your-account/your-repository.git

Now we need to specify newly created credentials in your seedJob definition file. They will be used to connect to specified github repository. Don’t forget to also change the url for SSH:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
#!/usr/bin/env groovy

pipelineJob('build-operator-service-for-jenkins'){
    displayName('Build Operator Service for Jenkins')

    definition {
        cpsScm {
            scm {
                git {
                    remote {
                        url('[email protected]:your-account/your-repo.git')
                        credentials('k8s-sh')
                    }
                    branches('*/master')
                }
            }
            scriptPath('cicd/pipelines/build.jenkins')
        }
    }
}

Username & password authentication

First, create a Secret file with your GitHub credentials.

1
2
3
4
5
6
7
apiVersion: v1
kind
: Secret
metadata
:
  name
: k8s-user-pass
stringData
:
  username
: github_user_name
  password
: password_or_token

Second, create a Kubernetes Secret resource from this file.

1
kubectl -n jenkins apply -f mySecret.yaml

In the seedJob you added to your Jenkins Custom Resource file you can specify usernamePassword as credentialType and add the name of the Secret, with your GitHub credentials, in the credentialID field’s value.

1
2
3
4
5
6
7
8
9
10
11
12
13
apiVersion: jenkins.io/v1alpha2
kind
: Jenkins
metadata
:
  name
: example
spec
:
  seedJobs
:
  - id
: operator-service-user-pass
    credentialType
: usernamePassword
    credentialID
: k8s-user-pass
    targets
: "cicd/jobs/*.jenkins"
    description
: "Operator Service for Jenkins repository"
    repositoryBranch
: master
    repositoryUrl
: https://github.com/your-github-account/your-github-repository.git