As I was digging around the internet encountering conflicting configurations and advice about how to setup SSL for Jenkins, I decided that I should really split this bit out of the Jenkins Post Mortem (still in progress).
Into the Trenches
It took a few iterations of failure before finding a working configuration. Here's what I tried that didn't work, along with the error behavior, to hopefully aid others in their troubleshooting.
Using Java's cacerts store
I started by basically doing what anyone else would do: googling / ducking for variations of "how to setup ssl for Jenkins". I started by trying to follow the instructions provided by CloudBees. These instructions worked after a fashion, and I've included them below for a quick reference:
sudo mkdir $JENKINS_HOME/.keystore
sudo chown jenkins:jenkins $JENKINS_HOME/.keystore
cp $JAVA_HOME/jre/lib/security/cacerts $JENKINS_HOME/.keystore
$JAVA_HOME/bin/keytool -keystore $JENKINS_HOME/.keystore/cacerts -import -alias <YOUR_ALIAS_HERE> -file <YOUR_CA_FILE>
<YOUR_ALIAS_HERE\> should be a helpful name, e.g. jenkins-wildcard if you're using a wildcard cert, and <YOUR_CA_FILE> would be the name of your x509 cert, e.g. wildcard-example-com-x509.crt. The crt you need to download from your certificate provider.
After setting all that up, I ran into a little snag:
→ sudo service jenkins start
Starting Jenkins [ OK ]
→ sudo service jenkins status
jenkins dead but pid file exists
Why is Jenkins dead? Well, the instructions I linked / copied above neglect to tell you to disable the HTTP port and set the HTTPS port, like so:
#disable HTTP
JENKINS_PORT="-1"
#enable HTTPS
JENKINS_HTTPS_PORT="8443"
Source: IconArchive
Once that's all working, though, you'll still end up with...
Self Signed Cert Error / Warning
For the curious, the browser plugins are 1Password, Momentum, AdBlocker Plus, Amazon, and Chromecast in that order.
I suspected part of that was perhaps it wasn't able to pull my specific cert out of the store, and I was a bit curious about what all was in there, so I took a look:
→ $JAVA_HOME/bin/keytool -list -keystore $JAVA_HOME/lib/security/cacerts
Enter keystore password:
Keystore type: JKS
Keystore provider: SUN
Your keystore contains 167 entries
{{ snip }}
Well if nothing else there are 167 entries in there and that's an unwiedly beast to troubleshoot to know for sure. Instead, I tried to create my own store that would only house my wildcard cert. Things got a bit unfortunate here, and I spun my wheels for quite a bit.
Trying to make my cert store
This was a bit difficult because it seems like some of the errors it threw were red herrings. I followed the instructions here to setup a keystore and they were as follows, my errors included:
→ openssl pkcs12 -export -out jenkins_keystore.p12 -passout 'pass:changeit' -inkey wildcard.example.com.key -in wildcard.example.com.crt -certfile ca-bundle.crt -name wildcard-example
→ $JAVA_HOME/bin/keytool -importkeystore -srckeystore jenkins_keystore.p12 -srcstorepass 'changeit' -srcstoretype PKCS12 -srcalias wildcard.example -deststoretype JKS -destkeystore jenkins_keystore.jks -deststorepass 'changeit' -destalias wildcard.example
Importing keystore jenkins_keystore.p12 to jenkins_keystore.jks...
keytool error: java.lang.Exception: Alias <wildcard.example> does not exist
→ $JAVA_HOME/bin/keytool -list -keystore jenkins_keystore.p12
Enter keystore password:
Keystore type: JKS
Keystore provider: SUN
Your keystore contains 1 entry
wildcard-example, Feb 8, 2018, PrivateKeyEntry,
Certificate fingerprint (SHA1): ██:██:██:██:██:██:██:██:██:██:██:██:██:██:██:██:██:██:██:██
→ $JAVA_HOME/bin/keytool -importkeystore -srckeystore jenkins_keystore.p12 -srcstorepass 'changeit' -srcstoretype PKCS12 -srcalias wildcard-example -deststoretype JKS -destkeystore jenkins_keystore.jks -deststorepass 'changeit' -destalias wildcard-example
Importing keystore jenkins_keystore.p12 to jenkins_keystore.jks...
Warning:
The JKS keystore uses a proprietary format. It is recommended to migrate to PKCS12 which is an industry standard format using "keytool -importkeystore -srckeystore jenkins_keystore.jks -destkeystore jenkins_keystore.jks -deststoretype pkcs12".
→ sudo cp jenkins_keystore.jks $JENKINS_HOME/.keystore/
→ sudo chown jenkins:jenkins $JENKINS_HOME/.keystore/jenkins_keystore.jks
→ sudo chmod 600 !$
sudo chmod 600 $JENKINS_HOME/.keystore/jenkins_keystore.jks
→ sudo vim /etc/sysconfig/jenkins
→ sudo service jenkins restart
Shutting down Jenkins [FAILED]
Starting Jenkins [ OK ]
→ ^restart^status
sudo service jenkins status
jenkins dead but pid file exists
And what was in the log?
→ sudo tail -n 25 /var/log/jenkins/jenkins.log
at Main._main(Main.java:294)
at Main.main(Main.java:132)
Caused by: winstone.WinstoneException: No SSL key store found at /etc/jenkins/jenkins_keystore.jks
at winstone.AbstractSecuredConnectorFactory.configureSsl(AbstractSecuredConnectorFactory.java:64)
at winstone.HttpsConnectorFactory.start(HttpsConnectorFactory.java:41)
at winstone.Launcher.spawnListener(Launcher.java:207)
... 8 more
Feb 08, 2018 9:12:28 PM winstone.Logger logInternal
SEVERE: Container startup failed
java.io.IOException: Failed to start a listener: winstone.HttpsConnectorFactory
at winstone.Launcher.spawnListener(Launcher.java:209)
at winstone.Launcher.<init>(Launcher.java:150)
at winstone.Launcher.main(Launcher.java:354)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at Main._main(Main.java:294)
at Main.main(Main.java:132)
Caused by: winstone.WinstoneException: No SSL key store found at /etc/jenkins/jenkins_keystore.jks
at winstone.AbstractSecuredConnectorFactory.configureSsl(AbstractSecuredConnectorFactory.java:64)
at winstone.HttpsConnectorFactory.start(HttpsConnectorFactory.java:41)
at winstone.Launcher.spawnListener(Launcher.java:207)
... 8 more
So a couple of things. You can see at the top there I neglected my alias and typoed it, but I left the mistake so you could see the command to retreive the alias from the store.
Also, JENKINS_HOME is /var/lib/jenkins. This is the standard configuration. You may notice that the Winstone log is looking in /etc/jenkins, which I didn't set, so it appears on the surface at least that this is defaulted somewhere. Noticing the path descrepancy I moved the keystore there, but it unfortunately still threw the same error. I took back to the internet and found this, which I took as a sign that maybe, just maybe, I needed to pursue a different solution.
Simple Setup: Jenkins + Nginx Reverse Proxy
Jenkins
To start, you'll need to set the Jenkins variables in /etc/sysconfig/jenkins:
→ sudo cat /etc/sysconfig/jenkins | grep -v "\#"
JENKINS_HOME="/var/lib/jenkins"
JENKINS_JAVA_CMD=""
JENKINS_USER="jenkins"
JENKINS_JAVA_OPTIONS="-Djava.awt.headless=true"
JENKINS_PORT="8080"
JENKINS_LISTEN_ADDRESS="127.0.0.1"
JENKINS_HTTPS_PORT=""
JENKINS_HTTPS_KEYSTORE=""
JENKINS_HTTPS_KEYSTORE_PASSWORD=""
JENKINS_HTTPS_LISTEN_ADDRESS=""
JENKINS_DEBUG_LEVEL="5"
JENKINS_ENABLE_ACCESS_LOG="no"
JENKINS_HANDLER_MAX="100"
JENKINS_HANDLER_IDLE="20"
JENKINS_ARGS=""
This is a minimal configuration, so you may have modified some values like JENKINS_ARGS if you have an existing Jenkins setup. This is fine. The main values to focus on here are JENKINS_PORT and JENKINS_LISTEN_ADDRESS. You should not set any of the JENKINS_HTTPS_* variables as the HTTPS configuration, if you choose it, will be handled by nginx.
Nginx
Setting up the Nginx proxy was so simple that I wish I had started there, but hey you live and learn. To set this up, install nginx:
sudo yum install nginx -y
I don't want to use the default nginx configurations as it does a bunch of wizarding that I don't want to inherit or troubleshoot. The latter being the usual case.
→ cd /etc/nginx
→ sudo mkdir nginx_defaults
→ sudo mv * nginx_defaults/
mv: cannot move ‘nginx_defaults’ to a subdirectory of itself, ‘nginx_defaults/nginx_defaults’
→ sudo mv nginx_defaults/mime.types .
→ sudo mkdir certs
→ tree
.
├── certs
├── mime.types
└── nginx_defaults
├── conf.d
│ └── virtual.conf
├── default.d
├── fastcgi.conf
├── fastcgi.conf.default
├── fastcgi_params
├── fastcgi_params.default
├── koi-utf
├── koi-win
├── mime.types.default
├── nginx.conf
├── nginx.conf.default
├── scgi_params
├── scgi_params.default
├── uwsgi_params
├── uwsgi_params.default
└── win-utf
We're going to be using mime.types, so definitely keep that one in the parent directory. Also, in this case ignore the cannot move error since that is expected.
HTTP Configuration
The following is a complete nginx.conf file for the HTTP proxy configuration, just make sure to change jenkins.example.com to your Jenkins URL.
The bulk of this configuration is taken from the Jenkins wiki page with the relevant information added in from the default nginx.conf.
user nginx;
worker_processes auto;
error_log /var/log/nginx/error.log;
pid /var/run/nginx.pid;
events {
worker_connections 1024;
}
http {
log_format main '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for"';
access_log /var/log/nginx/access.log main;
sendfile on;
tcp_nopush on;
tcp_nodelay on;
keepalive_timeout 65;
types_hash_max_size 2048;
include /etc/nginx/mime.types;
default_type application/octet-stream;
index index.html index.htm;
server {
listen 80; # Listen on port 80 for IPv4 requests
server_name jenkins.example.com; ## FIXME: This should be your URL
#this is the jenkins web root directory (mentioned in the /etc/default/jenkins file)
root /var/run/jenkins/war/;
access_log /var/log/nginx/access.log;
error_log /var/log/nginx/error.log;
ignore_invalid_headers off; #pass through headers from Jenkins which are considered invalid by Nginx server.
location ~ "^/static/[0-9a-fA-F]{8}\/(.*)$" {
#rewrite all static files into requests to the root
#E.g /static/12345678/css/something.css will become /css/something.css
rewrite "^/static/[0-9a-fA-F]{8}\/(.*)" /$1 last;
}
location /userContent {
#have nginx handle all the static requests to the userContent folder files
#note : This is the $JENKINS_HOME dir
root /var/lib/jenkins/;
if (!-f $request_filename){
#this file does not exist, might be a directory or a /**view** url
rewrite (.*) /$1 last;
break;
}
sendfile on;
}
location @jenkins {
sendfile off;
proxy_pass http://127.0.0.1:8080;
proxy_redirect default;
proxy_http_version 1.1;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_max_temp_file_size 0;
#this is the maximum upload size
client_max_body_size 10m;
client_body_buffer_size 128k;
proxy_connect_timeout 90;
proxy_send_timeout 90;
proxy_read_timeout 90;
proxy_request_buffering off; # Required for HTTP CLI commands in Jenkins > 2.54
}
location / {
# Optional configuration to detect and redirect iPhones
if ($http_user_agent ~* '(iPhone|iPod)') {
rewrite ^/$ /view/iphone/ redirect;
}
try_files $uri @jenkins;
}
}
}
HTTPS Configuration
The HTTPS configuration is almost identical, save a few changes:
- Changing the listen port from
80to443 - Adding the section I've labeled
# HTTP redirect, so thathttp://jenkins.example.comis redirected tohttps://jenkins.example.com - Adding the block that I've noted as
# SSLin thenginx.confwhich provides the path to the cert files- Make sure to update the paths / filenames to match your setup
- Updating the
proxy_redirectvalue
Quick cert detour
For your certs: you'll need the key file and the x509 cert. Depending on your provider, these may be hard to identify so to hopefully help you I'm going to show how I inspected the certs I downloaded from our cert provider:
→ sha1sum *
f1c██████████████████████████████████c71 wildcard.example.com.apache.crt
819██████████████████████████████████444 wildcard.example.com.ee_x509.crt
304██████████████████████████████████992 wildcard.example.com.i1_issuer.crt
1a8██████████████████████████████████c4c wildcard.example.com.pkcs7.p7s
f1c██████████████████████████████████c71 wildcard.example.com.plesk.crt
From this you can see that the top and bottom files are the same, but the middle three are different from each other.
Inspecting further:
→ openssl x509 -noout -text -in wildcard.example.com.apache.crt
Certificate:
Data:
Version: █████
Serial Number: ████████
Signature Algorithm: sha256WithRSAEncryption
Issuer: C=US, O=GeoTrust Inc., CN=GeoTrust Global CA
Validity
Not Before: Aug ██ 21:39:32 ████ GMT
Not After : May ██ 21:39:32 ████ GMT
Subject: C=US, O=GeoTrust Inc., CN=RapidSSL SHA256 CA - G3
Subject Public Key Info:
Public Key Algorithm: rsaEncryption
Public-Key: (2048 bit)
Modulus:
{{{ SNIP }}}
This wildcard.example.com.apache.crt is the intermediary cert issued by "GeoTrust Global CA" (the root) to "GeoTrust RapidSSL SHA256 CA - G3" (the intermediary). An intermediary cert sits between the wildcard cert issued to us/me and the root cert. Usually when you download a bunch of certificate files from your provider this a file like this is included because not all intermediary CAs are trusted by all sources, so in some cases you may need to provide the intermediate cert. To see if your intermediate cert is trusted by your browser you can check that browser's certificate store.
Quick example for how to check a cert store in Firefox: go to about:preferences in the address bar and scroll to the bottom of the page. Here, you can see that "GeoTrust's RapidSSL SHA256 CA - 3" is trusted by Firefox:
Click image to view full size.
→ openssl x509 -noout -text -in wildcard.example.com.ee_x509.crt
Certificate:
Data:
Version: █████
Serial Number: ████████
Signature Algorithm: sha256WithRSAEncryption
Issuer: C=US, O=GeoTrust Inc., CN=RapidSSL SHA256 CA - G3
Validity
Not Before: Dec ██ 19:21:43 ████ GMT
Not After : Jan ██ 16:01:28 ████ GMT
Subject: CN=*.example.com
Subject Public Key Info:
Public Key Algorithm: rsaEncryption
Public-Key: (2048 bit)
Modulus:
{{{ SNIP }}}
The Subject: CN=*.example.com tells us that this is the file we want to use as the ssl_certificate in the nginx configuration. This is actually a type of pem file, so I'm actually going to change the extension when I move this file and the key file to /etc/nginx/certs:
→ sudo cp wildcard.example.com.ee_x509.crt /etc/nginx/certs/wildcard.example.com.pem
→ sudo cp wildcard.example.com.key /etc/nginx/certs/
The HTTPS Configuration
As before, the complete HTTPS configuration is below, making sure you change jenkins.example.com and the cert paths to match your configuration:
user nginx;
worker_processes auto;
error_log /var/log/nginx/error.log;
pid /var/run/nginx.pid;
events {
worker_connections 1024;
}
http {
log_format main '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for"';
access_log /var/log/nginx/access.log main;
sendfile on;
tcp_nopush on;
tcp_nodelay on;
keepalive_timeout 65;
types_hash_max_size 2048;
include /etc/nginx/mime.types;
default_type application/octet-stream;
index index.html index.htm;
# HTTP redirect
server {
listen 80;
server_name jenkins.example.com;
return 301 https://$host$request_uri;
}
server {
listen 443; # Listen on port 443 for IPv4 requests
server_name jenkins.example.com;
# SSL
ssl on;
ssl_certificate /etc/nginx/certs/wildcard.example.com.pem;
ssl_certificate_key /etc/nginx/certs/wildcard.example.com.key;
ssl_protocols TLSv1.2;
ssl_ciphers 'EECDH+AESGCM:EDH+AESGCM';
ssl_prefer_server_ciphers on;
ssl_session_cache shared:SSL:10m;
#this is the jenkins web root directory (mentioned in the /etc/default/jenkins file)
root /var/run/jenkins/war/;
access_log /var/log/nginx/access.log;
error_log /var/log/nginx/error.log;
ignore_invalid_headers off; #pass through headers from Jenkins which are considered invalid by Nginx server.
location ~ "^/static/[0-9a-fA-F]{8}\/(.*)$" {
#rewrite all static files into requests to the root
#E.g /static/12345678/css/something.css will become /css/something.css
rewrite "^/static/[0-9a-fA-F]{8}\/(.*)" /$1 last;
}
location /userContent {
#have nginx handle all the static requests to the userContent folder files
#note : This is the $JENKINS_HOME dir
root /var/lib/jenkins/;
if (!-f $request_filename){
#this file does not exist, might be a directory or a /**view** url
rewrite (.*) /$1 last;
break;
}
sendfile on;
}
location @jenkins {
sendfile off;
proxy_pass http://127.0.0.1:8080;
proxy_redirect http://localhost:8080 $scheme://jenkins.example.com;
proxy_http_version 1.1;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_max_temp_file_size 0;
#this is the maximum upload size
client_max_body_size 10m;
client_body_buffer_size 128k;
proxy_connect_timeout 90;
proxy_send_timeout 90;
proxy_read_timeout 90;
proxy_request_buffering off; # Required for HTTP CLI commands in Jenkins > 2.54
}
location / {
# Optional configuration to detect and redirect iPhones
if ($http_user_agent ~* '(iPhone|iPod)') {
rewrite ^/$ /view/iphone/ redirect;
}
try_files $uri @jenkins;
}
}
}
Cleaning up the loose ends
Fixing the Reverse Proxy Error
Once you have your reverse proxy setup in either case, you'll most likely encounter this error:
To resolve this, go to Manage Jenkins -> Configure System, or go to https://${YOUR_JENKINS_URL}/configure, and update your configuration to use your new HTTP or HTTPS address, as can be seen in the side by side configuration below:
Fixing Github Oauth
If you have Github oauth configured you might have a mild heart attack when auth fails to redirect because of your proxy. Not to fear, this is another quick fix. Log into your Github account / Github org and go to where you've configured oauth:
And then go into the configuation and update both the Homepage URL and the Authorization Callback URL to your new HTTP or HTTPS address.
That's it! You can now auth with Github normally.
Updating Github webhooks
If you are using Github webhooks, you'll need to update them from something like:
http://jenkins.example.com:8080/github-webhook/
To either your new HTTP or HTTPS URL, e.g.:
http://jenkins.example.com/github-webhook/
https://jenkins.example.com/github-webhook/
Make sure you scroll to the bottom and save your configuration!
Fixing SSL Webhooks - "Peer certificate cannot be authenticated"
If you are using SSL, as you likely are if you're reading this, you may encounter the following after updating your webhooks:
Opening up the most recent one to take a look:
What does this mean?
If you recall, earlier I checked the Firefox cert store to see if it had "RapidSSL SHA256 CA - G3" in it - this is the intermediary that signed the certificate that I'm using. I also mentioned that not all intermediate sources are trusted. Here, it appears that while Firefox does trust my intermediate source, Github does not.
How to fix this? Certificate bundles are actually just a series of PEM files in a single file, so we need to either pre-pend or append the intermediary to the cert file that nginx is using.
I quickly tried pre-pending first, and encountered this error when I restarted nginx:
→ sudo service nginx restart
nginx: [emerg] SSL_CTX_use_PrivateKey_file("/etc/nginx/certs/wildcard.example.com.key") failed (SSL: error:0B080074:x509 certificate routines:X509_check_private_key:key values mismatch)
nginx: configuration file /etc/nginx/nginx.conf test failed
So I reversed the cert order and violá, nginx started and the page loaded. Next, I tried resending that last payload to see if SSL verification passed:
Excellent.
For completeness, I checked back in after letting a few builds run to verify that the webhook history was green and healthy once more, and indeed it was/is:
Header image: Word Cloud drawn by Word Clouds Generator