Ops Tutorial: SSL Setup for Jenkins

As I was digging around the internet encountering conflicting configurations and advice about how to setup SSL for Jenkins, I decided that I should really split this bit out of the Jenkins Post Mortem (still in progress).

Into the Trenches

It took a few iterations of failure before finding a working configuration. Here's what I tried that didn't work, along with the error behavior, to hopefully aid others in their troubleshooting.

Using Java's cacerts store

I started by basically doing what anyone else would do: googling / ducking for variations of "how to setup ssl for Jenkins". I started by trying to follow the instructions provided by CloudBees. These instructions worked after a fashion, and I've included them below for a quick reference:

sudo mkdir $JENKINS_HOME/.keystore

sudo chown jenkins:jenkins $JENKINS_HOME/.keystore

cp $JAVA_HOME/jre/lib/security/cacerts $JENKINS_HOME/.keystore

$JAVA_HOME/bin/keytool -keystore $JENKINS_HOME/.keystore/cacerts -import -alias <YOUR_ALIAS_HERE> -file <YOUR_CA_FILE>

<YOUR_ALIAS_HERE\> should be a helpful name, e.g. jenkins-wildcard if you're using a wildcard cert, and <YOUR_CA_FILE> would be the name of your x509 cert, e.g. wildcard-example-com-x509.crt. The crt you need to download from your certificate provider.

After setting all that up, I ran into a little snag:

→  sudo service jenkins start
Starting Jenkins                                           [  OK  ]

→  sudo service jenkins status
jenkins dead but pid file exists

Why is Jenkins dead? Well, the instructions I linked / copied above neglect to tell you to disable the HTTP port and set the HTTPS port, like so:

#disable HTTP
JENKINS_PORT="-1"

#enable HTTPS
JENKINS_HTTPS_PORT="8443"

Source: IconArchive

Once that's all working, though, you'll still end up with...

Self Signed Cert Error / Warning

For the curious, the browser plugins are 1Password, Momentum, AdBlocker Plus, Amazon, and Chromecast in that order.

I suspected part of that was perhaps it wasn't able to pull my specific cert out of the store, and I was a bit curious about what all was in there, so I took a look:

→  $JAVA_HOME/bin/keytool -list -keystore $JAVA_HOME/lib/security/cacerts
Enter keystore password:
Keystore type: JKS
Keystore provider: SUN

Your keystore contains 167 entries
{{ snip }}

Well if nothing else there are 167 entries in there and that's an unwiedly beast to troubleshoot to know for sure. Instead, I tried to create my own store that would only house my wildcard cert. Things got a bit unfortunate here, and I spun my wheels for quite a bit.

Trying to make my cert store

This was a bit difficult because it seems like some of the errors it threw were red herrings. I followed the instructions here to setup a keystore and they were as follows, my errors included:

→  openssl pkcs12 -export -out jenkins_keystore.p12 -passout 'pass:changeit' -inkey wildcard.example.com.key -in wildcard.example.com.crt -certfile ca-bundle.crt -name wildcard-example

→  $JAVA_HOME/bin/keytool -importkeystore -srckeystore jenkins_keystore.p12 -srcstorepass 'changeit' -srcstoretype PKCS12 -srcalias wildcard.example -deststoretype JKS -destkeystore jenkins_keystore.jks -deststorepass 'changeit' -destalias wildcard.example
Importing keystore jenkins_keystore.p12 to jenkins_keystore.jks...
keytool error: java.lang.Exception: Alias <wildcard.example> does not exist

→  $JAVA_HOME/bin/keytool -list -keystore jenkins_keystore.p12
Enter keystore password:
Keystore type: JKS
Keystore provider: SUN

Your keystore contains 1 entry

wildcard-example, Feb 8, 2018, PrivateKeyEntry,
Certificate fingerprint (SHA1): ██:██:██:██:██:██:██:██:██:██:██:██:██:██:██:██:██:██:██:██

→  $JAVA_HOME/bin/keytool -importkeystore -srckeystore jenkins_keystore.p12 -srcstorepass 'changeit' -srcstoretype PKCS12 -srcalias wildcard-example -deststoretype JKS -destkeystore jenkins_keystore.jks -deststorepass 'changeit' -destalias wildcard-example
Importing keystore jenkins_keystore.p12 to jenkins_keystore.jks...

Warning:
The JKS keystore uses a proprietary format. It is recommended to migrate to PKCS12 which is an industry standard format using "keytool -importkeystore -srckeystore jenkins_keystore.jks -destkeystore jenkins_keystore.jks -deststoretype pkcs12".

→  sudo cp jenkins_keystore.jks $JENKINS_HOME/.keystore/

→  sudo chown jenkins:jenkins $JENKINS_HOME/.keystore/jenkins_keystore.jks

→  sudo chmod 600 !$
sudo chmod 600 $JENKINS_HOME/.keystore/jenkins_keystore.jks


→  sudo vim /etc/sysconfig/jenkins

→  sudo service jenkins restart
Shutting down Jenkins                                      [FAILED]
Starting Jenkins                                           [  OK  ]

→  ^restart^status
sudo service jenkins status
jenkins dead but pid file exists

And what was in the log?

→  sudo tail -n 25 /var/log/jenkins/jenkins.log
        at Main._main(Main.java:294)
        at Main.main(Main.java:132)
Caused by: winstone.WinstoneException: No SSL key store found at /etc/jenkins/jenkins_keystore.jks
        at winstone.AbstractSecuredConnectorFactory.configureSsl(AbstractSecuredConnectorFactory.java:64)
        at winstone.HttpsConnectorFactory.start(HttpsConnectorFactory.java:41)
        at winstone.Launcher.spawnListener(Launcher.java:207)
        ... 8 more
Feb 08, 2018 9:12:28 PM winstone.Logger logInternal
SEVERE: Container startup failed
java.io.IOException: Failed to start a listener: winstone.HttpsConnectorFactory
        at winstone.Launcher.spawnListener(Launcher.java:209)
        at winstone.Launcher.<init>(Launcher.java:150)
        at winstone.Launcher.main(Launcher.java:354)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at Main._main(Main.java:294)
        at Main.main(Main.java:132)
Caused by: winstone.WinstoneException: No SSL key store found at /etc/jenkins/jenkins_keystore.jks
        at winstone.AbstractSecuredConnectorFactory.configureSsl(AbstractSecuredConnectorFactory.java:64)
        at winstone.HttpsConnectorFactory.start(HttpsConnectorFactory.java:41)
        at winstone.Launcher.spawnListener(Launcher.java:207)
        ... 8 more

So a couple of things. You can see at the top there I neglected my alias and typoed it, but I left the mistake so you could see the command to retreive the alias from the store.

Also, JENKINS_HOME is /var/lib/jenkins. This is the standard configuration. You may notice that the Winstone log is looking in /etc/jenkins, which I didn't set, so it appears on the surface at least that this is defaulted somewhere. Noticing the path descrepancy I moved the keystore there, but it unfortunately still threw the same error. I took back to the internet and found this, which I took as a sign that maybe, just maybe, I needed to pursue a different solution.

Simple Setup: Jenkins + Nginx Reverse Proxy

Jenkins

To start, you'll need to set the Jenkins variables in /etc/sysconfig/jenkins:

→  sudo cat /etc/sysconfig/jenkins | grep -v "\#"
JENKINS_HOME="/var/lib/jenkins"

JENKINS_JAVA_CMD=""

JENKINS_USER="jenkins"


JENKINS_JAVA_OPTIONS="-Djava.awt.headless=true"

JENKINS_PORT="8080"

JENKINS_LISTEN_ADDRESS="127.0.0.1"

JENKINS_HTTPS_PORT=""

JENKINS_HTTPS_KEYSTORE=""

JENKINS_HTTPS_KEYSTORE_PASSWORD=""

JENKINS_HTTPS_LISTEN_ADDRESS=""


JENKINS_DEBUG_LEVEL="5"

JENKINS_ENABLE_ACCESS_LOG="no"

JENKINS_HANDLER_MAX="100"

JENKINS_HANDLER_IDLE="20"

JENKINS_ARGS=""

This is a minimal configuration, so you may have modified some values like JENKINS_ARGS if you have an existing Jenkins setup. This is fine. The main values to focus on here are JENKINS_PORT and JENKINS_LISTEN_ADDRESS. You should not set any of the JENKINS_HTTPS_* variables as the HTTPS configuration, if you choose it, will be handled by nginx.

Nginx

Setting up the Nginx proxy was so simple that I wish I had started there, but hey you live and learn. To set this up, install nginx:

sudo yum install nginx -y

I don't want to use the default nginx configurations as it does a bunch of wizarding that I don't want to inherit or troubleshoot. The latter being the usual case.

→  cd /etc/nginx
→  sudo mkdir nginx_defaults
→  sudo mv * nginx_defaults/
mv: cannot move ‘nginx_defaults’ to a subdirectory of itself, ‘nginx_defaults/nginx_defaults’
→  sudo mv nginx_defaults/mime.types .
→  sudo mkdir certs
→  tree
.
├── certs
├── mime.types
└── nginx_defaults
    ├── conf.d
    │   └── virtual.conf
    ├── default.d
    ├── fastcgi.conf
    ├── fastcgi.conf.default
    ├── fastcgi_params
    ├── fastcgi_params.default
    ├── koi-utf
    ├── koi-win
    ├── mime.types.default
    ├── nginx.conf
    ├── nginx.conf.default
    ├── scgi_params
    ├── scgi_params.default
    ├── uwsgi_params
    ├── uwsgi_params.default
    └── win-utf

We're going to be using mime.types, so definitely keep that one in the parent directory. Also, in this case ignore the cannot move error since that is expected.

HTTP Configuration

The following is a complete nginx.conf file for the HTTP proxy configuration, just make sure to change jenkins.example.com to your Jenkins URL.

The bulk of this configuration is taken from the Jenkins wiki page with the relevant information added in from the default nginx.conf.

user nginx;
worker_processes auto;
error_log /var/log/nginx/error.log;
pid /var/run/nginx.pid;

events {
    worker_connections 1024;
}

http {
    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    access_log  /var/log/nginx/access.log  main;

    sendfile            on;
    tcp_nopush          on;
    tcp_nodelay         on;
    keepalive_timeout   65;
    types_hash_max_size 2048;

    include             /etc/nginx/mime.types;
    default_type        application/octet-stream;

    index   index.html index.htm;

    server {
      listen          80;       # Listen on port 80 for IPv4 requests

      server_name     jenkins.example.com;  ## FIXME: This should be your URL

      #this is the jenkins web root directory (mentioned in the /etc/default/jenkins file)
      root            /var/run/jenkins/war/;

      access_log      /var/log/nginx/access.log;
      error_log       /var/log/nginx/error.log;
      ignore_invalid_headers off; #pass through headers from Jenkins which are considered invalid by Nginx server.

      location ~ "^/static/[0-9a-fA-F]{8}\/(.*)__aSyNcId_<_gvgxiJC___quot; {
        #rewrite all static files into requests to the root
        #E.g /static/12345678/css/something.css will become /css/something.css
        rewrite "^/static/[0-9a-fA-F]{8}\/(.*)" /$1 last;
      }

      location /userContent {
        #have nginx handle all the static requests to the userContent folder files
        #note : This is the $JENKINS_HOME dir
        root /var/lib/jenkins/;
        if (!-f $request_filename){
          #this file does not exist, might be a directory or a /**view** url
          rewrite (.*) /$1 last;
          break;
        }
        sendfile on;
      }

      location @jenkins {
          sendfile off;
          proxy_pass         http://127.0.0.1:8080;
          proxy_redirect     default;
          proxy_http_version 1.1;

          proxy_set_header   Host              $host;
          proxy_set_header   X-Real-IP         $remote_addr;
          proxy_set_header   X-Forwarded-For   $proxy_add_x_forwarded_for;
          proxy_set_header   X-Forwarded-Proto $scheme;
          proxy_max_temp_file_size 0;

          #this is the maximum upload size
          client_max_body_size       10m;
          client_body_buffer_size    128k;

          proxy_connect_timeout      90;
          proxy_send_timeout         90;
          proxy_read_timeout         90;
          proxy_request_buffering    off; # Required for HTTP CLI commands in Jenkins > 2.54
      }

      location / {
        # Optional configuration to detect and redirect iPhones
        if ($http_user_agent ~* '(iPhone|iPod)') {
          rewrite ^/$ /view/iphone/ redirect;
        }

        try_files $uri @jenkins;
      }
    }
}

HTTPS Configuration

The HTTPS configuration is almost identical, save a few changes:

• Changing the listen port from 80 to 443
• Adding the section I've labeled # HTTP redirect, so that http://jenkins.example.com is redirected to https://jenkins.example.com
• Adding the block that I've noted as # SSL in the nginx.conf which provides the path to the cert files
  • Make sure to update the paths / filenames to match your setup
• Updating the proxy_redirect value

Quick cert detour

For your certs: you'll need the key file and the x509 cert. Depending on your provider, these may be hard to identify so to hopefully help you I'm going to show how I inspected the certs I downloaded from our cert provider:

→  sha1sum *
f1c██████████████████████████████████c71  wildcard.example.com.apache.crt
819██████████████████████████████████444  wildcard.example.com.ee_x509.crt
304██████████████████████████████████992  wildcard.example.com.i1_issuer.crt
1a8██████████████████████████████████c4c  wildcard.example.com.pkcs7.p7s
f1c██████████████████████████████████c71  wildcard.example.com.plesk.crt

From this you can see that the top and bottom files are the same, but the middle three are different from each other.

Inspecting further:

→  openssl x509 -noout -text -in wildcard.example.com.apache.crt
Certificate:
    Data:
        Version: █████
        Serial Number: ████████
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: C=US, O=GeoTrust Inc., CN=GeoTrust Global CA
        Validity
            Not Before: Aug ██ 21:39:32 ████ GMT
            Not After : May ██ 21:39:32 ████ GMT
        Subject: C=US, O=GeoTrust Inc., CN=RapidSSL SHA256 CA - G3
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (2048 bit)
                Modulus:
{{{ SNIP }}}

This wildcard.example.com.apache.crt is the intermediary cert issued by "GeoTrust Global CA" (the root) to "GeoTrust RapidSSL SHA256 CA - G3" (the intermediary). An intermediary cert sits between the wildcard cert issued to us/me and the root cert. Usually when you download a bunch of certificate files from your provider this a file like this is included because not all intermediary CAs are trusted by all sources, so in some cases you may need to provide the intermediate cert. To see if your intermediate cert is trusted by your browser you can check that browser's certificate store.

Quick example for how to check a cert store in Firefox: go to about:preferences in the address bar and scroll to the bottom of the page. Here, you can see that "GeoTrust's RapidSSL SHA256 CA - 3" is trusted by Firefox:

Click image to view full size.

→  openssl x509 -noout -text -in wildcard.example.com.ee_x509.crt
Certificate:
    Data:
        Version: █████
        Serial Number: ████████
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: C=US, O=GeoTrust Inc., CN=RapidSSL SHA256 CA - G3
        Validity
            Not Before: Dec ██ 19:21:43 ████ GMT
            Not After : Jan ██ 16:01:28 ████ GMT
        Subject: CN=*.example.com
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (2048 bit)
                Modulus:
{{{ SNIP }}}

The Subject: CN=*.example.com tells us that this is the file we want to use as the ssl_certificate in the nginx configuration. This is actually a type of pem file, so I'm actually going to change the extension when I move this file and the key file to /etc/nginx/certs:

→  sudo cp wildcard.example.com.ee_x509.crt /etc/nginx/certs/wildcard.example.com.pem
→  sudo cp wildcard.example.com.key /etc/nginx/certs/

The HTTPS Configuration

As before, the complete HTTPS configuration is below, making sure you change jenkins.example.com and the cert paths to match your configuration:

user nginx;
worker_processes auto;
error_log /var/log/nginx/error.log;
pid /var/run/nginx.pid;

events {
    worker_connections 1024;
}

http {
    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    access_log  /var/log/nginx/access.log  main;

    sendfile            on;
    tcp_nopush          on;
    tcp_nodelay         on;
    keepalive_timeout   65;
    types_hash_max_size 2048;

    include             /etc/nginx/mime.types;
    default_type        application/octet-stream;

    index   index.html index.htm;

    # HTTP redirect
    server {
        listen 80;
        server_name jenkins.example.com;
        return 301 https://$host$request_uri;
    }

    server {
      listen          443;       # Listen on port 443 for IPv4 requests

      server_name     jenkins.example.com;

      # SSL
      ssl on;
      ssl_certificate            /etc/nginx/certs/wildcard.example.com.pem;
      ssl_certificate_key        /etc/nginx/certs/wildcard.example.com.key;
      ssl_protocols              TLSv1.2;
      ssl_ciphers                'EECDH+AESGCM:EDH+AESGCM';
      ssl_prefer_server_ciphers  on;
      ssl_session_cache          shared:SSL:10m;

      #this is the jenkins web root directory (mentioned in the /etc/default/jenkins file)
      root            /var/run/jenkins/war/;

      access_log      /var/log/nginx/access.log;
      error_log       /var/log/nginx/error.log;
      ignore_invalid_headers off; #pass through headers from Jenkins which are considered invalid by Nginx server.

      location ~ "^/static/[0-9a-fA-F]{8}\/(.*)__aSyNcId_<_gvgxiJC___quot; {
        #rewrite all static files into requests to the root
        #E.g /static/12345678/css/something.css will become /css/something.css
        rewrite "^/static/[0-9a-fA-F]{8}\/(.*)" /$1 last;
      }

      location /userContent {
        #have nginx handle all the static requests to the userContent folder files
        #note : This is the $JENKINS_HOME dir
        root /var/lib/jenkins/;
        if (!-f $request_filename){
          #this file does not exist, might be a directory or a /**view** url
          rewrite (.*) /$1 last;
          break;
        }
        sendfile on;
      }

      location @jenkins {
          sendfile off;
          proxy_pass         http://127.0.0.1:8080;
          proxy_redirect     http://localhost:8080 $scheme://jenkins.example.com;
          proxy_http_version 1.1;

          proxy_set_header   Host              $host;
          proxy_set_header   X-Real-IP         $remote_addr;
          proxy_set_header   X-Forwarded-For   $proxy_add_x_forwarded_for;
          proxy_set_header   X-Forwarded-Proto $scheme;
          proxy_max_temp_file_size 0;

          #this is the maximum upload size
          client_max_body_size       10m;
          client_body_buffer_size    128k;

          proxy_connect_timeout      90;
          proxy_send_timeout         90;
          proxy_read_timeout         90;
          proxy_request_buffering    off; # Required for HTTP CLI commands in Jenkins > 2.54
      }

      location / {
        # Optional configuration to detect and redirect iPhones
        if ($http_user_agent ~* '(iPhone|iPod)') {
          rewrite ^/$ /view/iphone/ redirect;
        }

        try_files $uri @jenkins;
      }
    }
}

Cleaning up the loose ends

Fixing the Reverse Proxy Error

Once you have your reverse proxy setup in either case, you'll most likely encounter this error:

To resolve this, go to Manage Jenkins -> Configure System, or go to https://${YOUR_JENKINS_URL}/configure, and update your configuration to use your new HTTP or HTTPS address, as can be seen in the side by side configuration below:

Fixing Github Oauth

If you have Github oauth configured you might have a mild heart attack when auth fails to redirect because of your proxy. Not to fear, this is another quick fix. Log into your Github account / Github org and go to where you've configured oauth:

And then go into the configuation and update both the Homepage URL and the Authorization Callback URL to your new HTTP or HTTPS address.

That's it! You can now auth with Github normally.

Updating Github webhooks

If you are using Github webhooks, you'll need to update them from something like:

http://jenkins.example.com:8080/github-webhook/

To either your new HTTP or HTTPS URL, e.g.:

http://jenkins.example.com/github-webhook/
https://jenkins.example.com/github-webhook/

Make sure you scroll to the bottom and save your configuration!

Fixing SSL Webhooks - "Peer certificate cannot be authenticated"

If you are using SSL, as you likely are if you're reading this, you may encounter the following after updating your webhooks:

Opening up the most recent one to take a look:

What does this mean?

If you recall, earlier I checked the Firefox cert store to see if it had "RapidSSL SHA256 CA - G3" in it - this is the intermediary that signed the certificate that I'm using. I also mentioned that not all intermediate sources are trusted. Here, it appears that while Firefox does trust my intermediate source, Github does not.

How to fix this? Certificate bundles are actually just a series of PEM files in a single file, so we need to either pre-pend or append the intermediary to the cert file that nginx is using.

I quickly tried pre-pending first, and encountered this error when I restarted nginx:

→  sudo service nginx restart
nginx: [emerg] SSL_CTX_use_PrivateKey_file("/etc/nginx/certs/wildcard.example.com.key") failed (SSL: error:0B080074:x509 certificate routines:X509_check_private_key:key values mismatch)
nginx: configuration file /etc/nginx/nginx.conf test failed

So I reversed the cert order and violá, nginx started and the page loaded. Next, I tried resending that last payload to see if SSL verification passed:

Excellent.

For completeness, I checked back in after letting a few builds run to verify that the webhook history was green and healthy once more, and indeed it was/is:

Header image: Word Cloud drawn by Word Clouds Generator