The goal of this article is to show a reference architecture on how to run Jenkins in Kubernetes. Also, the most important aspects of managing Jenkins will be outlined in-depth to give you a better understanding of where they can be automated using the operator pattern.  

This content should be relevant to Developers, Architects, and Managers working in the area of cloud-native CI/CD tooling. Many of the topics covered will be technical in nature. It’s advised any readers have at least a basic understanding of the following concepts:

Introduction

When it comes to running a stateful application in a cloud-native environment like Kubernetes, it usually requires significant refactoring or additional automation handling lifecycle-management activities to make configuration fully operational. 

Kubernetes brings declarative configuration, autoscaling, self-healing, and other capabilities natively supported by the platform that can be leveraged by the application. However, with this power and flexibility, there comes a need for a little bit of an extra engineering effort.

Jenkins is a stateful application that heavily relies on XML configuration files, and is not designed to run in scalable environments without integration with the Kubernetes runtime.  It makes it difficult for users to navigate and use this technology, especially at scale within large organizations. This often leads to some manual work and likely duplication of effort within your organization.

Recommended Architecture

The diagram below shows the recommended architecture for running Jenkins on Kubernetes.

Please be aware of the following aspects of the design:

  • Both the Jenkins master and agent processes have a corresponding Role-Based Authorization Strategy (RBAC) model
  • Kubernetes Plugin is used to run and scale Jenkins agents dynamically
  • There is one dedicated Seed Job Agent for creating Jenkins jobs
  • A backup process is running in a sidecar container
  • Backup files are uploaded to external storage, such as Azure Storage Blob or AWS S3
  • Authn & Authz are based on GitHub OAuth
  • DSL Pipelines and Job Definitions are versioned in GitHub following GitOps model
  • Secure access to the Jenkins instance is provided via VPN or Bastion Host
  • Lifecycle management and initial configuration are handled by the Kubernetes Operator

Lifecycle Management

It’s a common use case when CI/CD systems are a critical part of the production environment in terms of deploying new artefacts, provisioning infrastructure or configuration management. Managing Jenkins is a complex process that usually consists of a number of activities:

  • Installing/upgrading Jenkins
  • Installing/upgrading plugins
  • Backing up the job history and/or artifacts
  • Using external services, e.g., Azure Storage Account or AWS S3 Bucket for storing backups

It’s important to understand that a change of plugin, authorization mechanism or others might require the Jenkins process to be restarted. Jenkins can’t be run in master-master mode, so this process causes downtime.

In any case, the Jenkins instance should be immutable: all changes should be made using Groovy or other configuration scripts that are part of the Jenkins boot process. All changes done manually should be ephemeral and disappear after the restart.

For more automation, using the Jenkins Operator is a recommended approach.

Configuration as Code

Most of the configuration steps are part of the Jenkins startup process. Jenkins executes Groovy Scripts located in the init.d folder in alphanumerical order. Any failure at this stage results in termination of the Jenkins process, with a corresponding error message in the log file.

After all scripts in the init.d folder are executed successfully, the Configuration as Code (CasC) plugin loads declarative YAML code, which may contain additional Jenkins configuration steps.

The initial configuration phase splits into two parts:

  • Base configuration – install and configure the core functionalities of Jenkins, such as authorization, core plugins, and security hardening
  • User’s configuration – perform additional configuration specified by the user such as plugins, and Groovy or CasC scripts

The recommended best approach is always to separate the core and user’s configuration. In this context, the user’s configuration might be an additional list of plugins, custom authentication mechanism or any other Configuration as Code/Groovy script.

Plugins

There are a large number of community plugins that make Jenkins customizable and modular, and they’re a key way to make Jenkins more extensible. However, in some deployments the user should be aware of:

  • Plugin dependency incompatibilities
  • Potential security risks introduced by third-party plugins
  • Unexpected side effects of plugin configurations, which might affect overall Jenkins status

The recommended solutions for these problems is to always verify the source of plugins, make sure configuration scripts don’t cause any side effects, and use a fixed version of the plugin.

This is our list of recommended plugins for Jenkins in the Kubernetes environment:

  • Configuration-as-code – enables CASC capabilities
  • DSL-Job-plugin – pipelines as code
  • Kubernetes – can be used to create pods as Jenkins agents
  • Notification – lets you send JSON or XML notifications

Authentication / Authorization

The default authentication mechanism is based on username/password, which is a good starting point, but not suitable in production CI/CD infrastructure. Jenkins supports other SAML and OAuth based identity providers that can be configured using specific plugins.

There is no one generic solution for OAuth; different OAuth providers require separate solutions (plugins). Example plugins include:

To manage users permissions and add authorization capabilities, use the Matrix Authorization Strategy plugin.
SAML is a generic solution and it requires only the SAML 2.0 Plugin to work with built-in authentication/authorization features.

Security

Due to the legacy nature of Jenkins and the high-security risk introduced by the plugins ecosystem, security is a very important and serious concern. After a fresh installation of Jenkins, at least the following aspects should be addressed:

  • Use Mode.EXCLUSIVE in basic settings – Jobs must specify that they want to run on the master node 
  • Enable CSRF protection
  • Disable usage statistics
  • Enable master access control
  • Disable old JNLP protocols – JNLP3-connect, JNLP2-connect and JNLP-connect should be disabled
  • Disable CLI – CLI access of the /cli URL should be disabled
  • Disable the script console
  • Ensure secure configuration of kubernetes-plugin

Also, an additional layer of security can be configured on top of Kubernetes:

  • Running the Seed Job Agent as a separate pod/deployment
  • Running Jenkins and Jenkins Agents in separate namespaces
  • Defining granular Kubernetes RBAC

The recommended way to protect Jenkins is to deploy it in a private network with VPN or bastion host access. The Kubernetes port-forward mechanism is also an option.

Pipelines as Code and the GitOps Model

GitOps is a methodology that uses Git as a single source of truth: a git repository that contains configuration files that describe the desired state of the application and its infrastructure components. An additional automation mechanism is responsible for the continuous deployment of such changes without human interaction.

The Job DSL plugin provides a declarative pipeline as code, which falls perfectly into the GitOps principles. This plugin also integrates with Kubernetes natively, so the pipeline steps can be executed on a Kubernetes pod, with any number of custom container images as shown below:

  • Seed Jobs Repository – consists of job and pipeline definitions which are automatically imported by Jenkins using the dsl-job-plugin
  • Jenkins uses ssh keys to check out the repository (this applies to private repositories only)
  • Jenkins uses kubernetes-plugins to programmatically create pipelines running as ephemeral Kubernetes Pods
  • Jenkins maintains one system agent for running seed jobs and other internal commands which should be separated from the user’s space

Key Takeaways

In our experience, key points about Jenkins and Kubernetes integration include:

  • Running Jenkins on top of Kubernetes is not a trivial task. It requires additional automation to handle lifecycle management.
  • Run Jenkins Master and Jenkins Agents in separate Kubernetes namespaces.
  • Use Configuration as Code or Groovy Scripts for configuration. Always separate the core and user’s configuration. 
  • The Jenkins instance should be immutable. Use the Sidecar Container Pattern to back up and restore Jenkins (job history and artifacts only).
  • Don’t expose Jenkins to the public internet; use VPN or Bastion Host instead.
  • The default authentication mechanism is not ready for production. Configure a SAML/OAuth based authentication mechanism instead.
  • Use plugins to extend the Jenkins capabilities.
  • For GitOps and Pipelines as Code, use the Job DSL plugin.
  • A fresh installation of Jenkins requires additional security hardening.

Final Words

Running Jenkins in a cloud-native environment is possible, but it is not a piece of cake. Kubernetes is designed to run stateless software, where Jenkins is a stateful application. 

This document covers the best practices for deploying and managing Jenkins in Kubernetes. These activities are ongoing tasks that take significant time, especially at scale within a large organization, and are usually handled by the dedicated operations teams. 

For an automated solution, check out Operator Service for Jenkins — the smart way of running scalable, immutable, and secure Jenkins in your organization.

Solve business problems with more efficient engineering

We have broad experience with the Cloud-Native landscape, ranging from building platforms to large-scale cloud transformations. Our team consists of Azure and Kubernetes certified experts, who actively participate in the Jenkins community.