Top level note: while exhausting, this is nothing compared to the injustices and level of violence that some citizens of our country are exposed to. So while you read, and hopefully enjoy, this process malfunction piece please keep in mind it's not the biggest problem we're facing.

And now to the post.

I recently had an issue where I needed to call the IRS, multiple times, about the status of both my 2018 and 2019 tax returns. It was ... a trainwreck, to put it mildly. Yes, there is a pandemic and yes, their offices have been closed for a long time - as the representative on the line reminded me. But these issues aren't confined to the pandemic.

Allow me to begin.

It started when I tried to e-file my 2019 taxes, same as I have been doing for years. Filing was rejected with an error code I couldn't easily look up. I contacted my e-filing provider, which took awhile as this was after everything closed down here in the States. In the interim, I tried just filing again - and was unsurprisingly met with the same error code.

I finally heard back from my e-filing provider and apparently the error was that I didn't enter "my PIN". I don't have a PIN, "you should have a PIN, check your mail". I don't have a PIN, "have you moved recently?" Not for years. "Ok, paper file and call the IRS."

Ok.

Calling the IRS is no joke. Especially when everything was closed. I tried calling a local office when I failed to get a human on the main line, but oh right that's closed. (It should be, mind you, people shouldn't be exposed to COVID because I should have a PIN that I don't have.) Then I Duck for "how to reach a human at the IRS" and find some instructions. Doing the straightforward method about "questions about my tax return" resulted in me verifying my SSN, amount of my 2019 return, etc., only to have the phone tree terminate with a "we cannot access your return at this time" message.

Ok. (Again.)

So I manage to reach a human, and was bounced over to Taxpayer Protection (TPP) as apparently there was a fraud alert on my account. Great. I have an abusive ex and lots of accounts locked down as a result, so my anxiety spikes wondering if she could have done something.

But who needs an abusive ex when you have the IRS.

I kid. Kind of.

I finally reach another human at TPP. The summary of the next 20ish minutes is:

Rep: You should have a PIN.
Me: I don't have a PIN.
Rep: Did you check your mail?
Me: ... yes.
Rep: Did you move recently?
Me: ... no.
Rep: I'll check your account again, please hold for 5-7 minutes.
Me: Ok.
Rep: (Returns) Oh. We flagged your account but never sent you a PIN. Sorry about that. Did you paper file?
Me: I did.
Rep: Great, that resolves that. But there's an issue with your 2018 return.
Me: ... what? E-File Provider says that the return was filed successfully, etc.?

Then there was a long session of me Verifying I Am Who I AM, as you might intuit when you're speaking to TPP. What's your mother's full maiden name, what's your father's full name, what's your date of birth, city of birth, and so on.

Ok. (I keep say it's ok for my own emotional state. This will not be the last Ok in this post, I promise.)

Then they tell me they mailed me something that I didn't answer, describing it as "it might have looked like spam but it wasn't spam".

Great.

When was that sent out? No exact date. But last year. Sometime.

Ok.

There's confusion about whether or not I received my 2018 refund / return, which I'd think I'd notice, so I leave to check paperwork and get off the multihour phone call, looking for anything on the federal (not state) 2018 return.

This isn't even the end. It's only Mayish still.

The lead time for a paperfiled return is about 45 days. So after hearing nothing on the 2019 return, I call again in July, but before the new tax deadline, to try to verify that my paper return was at least received if not processed. The rep at that point seemd to indicate that it was received, but not processed. (Why I phrased it that way will become clear in a moment.)

I wait another month. At this point I'm due for a change of address, for the first time in years, and need to make sure that the "totally should have arrived by now" returns will go to the correct place.

The joke's on me.

I call the IRS. I had forgotten, somehow, in the intervening weeks that the phone tree terminates in a hangup and no human when I select the options that make sense for questions about about the 1040. I Duck, again, for reaching a human and follow the instructions. I spent 2 hours and 16 minutes on the phone. Most of that on hold.

Results?

They still don't know about my 2018 return. They're sending me a letter. They cannot change my address, but the letter should forward if it doesn't arrive before my move out date so I can address it (i.e. call again) at that time.

Great.

Despite the previous rep seeming to be able to verify that my paper return had at least been received somewhere, the current rep could not. That conversation can be summarized as "your paper return would be stored at our processing facility, this is the call center, don't you know those are separate systems?" As someone who works in systems, all I could think of the metaphorically broken, stripped gears that were failing to interlock for someone on the phone not to be able to get that kind of data. Don't know which of the multiple filing centers across the US would even have my return to directly call, the received returns cannot be scanned or otherwise logged into a system so that people who answer the phones can actually answer questions. No wonder the phone tree terminates - that said, it should terminate before having me verify my SSN, return amount, and filing status as apparently it can't even be known. Just put that "Cannot know" response at the top of the tree.

Anyway.

They cannot update my address in their system as it'll revert whenever they process my 2019 return. No they cannot override it. Federal documents, like paper returns, do not forward. I point out that means the new owner of my current residence will then have access to my refund. The rep said she "knows" but that's how it's setup. It cannot forward. I asked how I would know whether or not my return processed, if it won't forward. Apparently the answer is that when my tax return is returned to them, that will kick off a letter to be sent to the same address that my tax return was returned to sender. That letter will forward. At which point I can then call them, they can update my address, and then resend my tax return. Oh, and don't re-file electronically in the interim, that'll flag my account and create More Problems. Or as the rep said, "I cannot advise you to do that at this time."

As someone who lives in processes and systems, this is just So Much for my brain.

First of all, there doesn't seem to be a way to input that paper filed returns are received. Can't print a barcode to put on the envelope, even if the envelope isn't opened, to be scanned on arrival at a facility? This should be randomized so someone can't (easily) get the tax information from the envelope, literally just a tracking number.

To the rep's point that "this is the call center, that's the processing facility" - since the users need these systems to talk to each other, they should. Put some tracking and confirmation information and make sure it reports into a centralized source of truth that is accessible to the people in the call center so they can answer questions like "did you even receive my return?" In fact, track the "most common questions" that need to be answered by a human, i.e that you cannot put a canned resonse in your phone tree for, and make sure that those working the call center are able to actually pull the data they need to answer those caller-specific questions.

Be able to change someone's address. People move all the time, and with the current global crisis that's likely only to become more often not less. Since we're required to verify our identity on the call, allow the rep to override the address with the caller's data with a flag that the override was requested and approved by the person who corresponds with the SSN/taxpayer ID.

Allow mail forwarding for all documents. When I had to temporarily live out of my home with the craziness of abusive ex, I set up a PO Box. I had initially tried to set up a mail forward to the PO Box, then set up a mail hold for anything that didn't forward. Unfortunately, as a different problem, you cannot have both a mail hold and a forward in place. Since there's currently no way to forward certain documents, then people should be able to place a hold on anything that cannot forward. Let me show my photo ID to pick up my mail: license, passport - whatever. Even better, if you can verify my identity for fraud protection use the same level of protection for forwarding sensitive documents. "Regular mail forward" can just be requested, or "all mail forward" requires a photo ID and whatever to verify the requestor is the person who is supposed to be receiving the documents. Because honestly, what's the point of a mail forward that doesn't forward?

Anyway, that's my rant. Thank you for reading this far, and maybe if we defund the police we can use a subset of the funds to redesigning the process flows that are so horribly broken.

Header image: Photo by Cristian Palmer on Unsplash

Not the greatest joke in the world, granted, but the reason I haven't been posting is because I've embarked on the great and wonderful journey called "career switching". In my particular case, from "infrastructure engineer" (a.k.a. Site Reliability Engineer / SRE, Cloud Engineer, Ops, etc.) to "Developer Advocate". It's been a lovely, but busy, journey.

I've also picked up some Hebrew along the way:
!שלום

(Note that regardless of the medium, blog or Slack, nothing handles changing carriage returns particularly well. Especially if you're trying to change direction on the same line, rather than a new one. Joy ;) )

In any event it's been a busy yearish. I'll be posting some things to come, but to start I had a funny idea wherein which I'll post a quick stub about a domain and/or handle idea that I had, that someone else had first and therefore is either in use or, regrettably, just poached. Here's some quick ones:

Twitter: @quintessence
This is pretty straightforward, as it's just my forename, but what "grinds my gears" about this one is that Twitter, unlike Github, does not have a "no parking" policy. Someone made two tweets in 2007 (twelve years ago) and now the handle is indefinitely in use :(

Twitter: @blackpajamas
I had an idea recently that I wanted to start learning how to cook Indian food like some of my snazzy international friends. This involves a lot of tasty and unforgiving spices (stains!), so I had the idea that I should cook in black clothes. In black pajamas, specifically. I was also writing up a blog post on Chef at the time, which means that I had the doublethink that it'd be funny to rename everything (even the blog ... 😅) to "I Code in Black Pajamas". Additional fun to be had when working from home, because work pajamas are definitely a thing.

URL: quintessence.dev
When the new .dev TLD was about to be released I was waiting on the trigger. I was finally going to have something in my name. It spurred a new era of hope: if I was able to secure "my" Twitter handle, that would mean consistent branding across Twitter, Github, and domain. Trifecta! Alas I was always a poor gambler and didn't want to to pay so I waited until it was opened up to the general public. The last cost tier was $125, so non-trivial even for a URL. Alas someone else managed to snag it, I blame bots, and now it is... parked.

Variations of "my (fore)name as a domain" that others put to use first:
quintessence.is -> Redirects to a Tumblr owned by a possibly current (?) Director of Content for Parachute Home, but hasn't been updated since 2015.
quintessence.com -> professional sound equiment. From the looks of it, exceeding even "prosumer grade". Cool.
quintessence.me -> Parked
quintessence.space -> Parked, but "for sale" for $450 USD. 😅

Anyway, y'all get the picture on the domains. I might reveal some of the other iterations of my name that I tried for things over time, if they're funny.

Banner source: "I'm not dead" from the greatly missed Hyperbole & A Half and some shameless backdrop by yours truly.

So as recently as only a few minutes ago, I had a minor panic attack. See, normally when I cd around in shell I tab to auto-complete like a fiend. Just now, I actually typed out a directory and was met with an error: it didn't exist. So then I tried to tab auto-complete. It still. didn't. exist. After suffering a minor panic attach I realized: I cloned the repo on my other laptop. No big. Since it was a repo for my own software projects, I should move the cloned repo to be a subdirectory of a shared iCloud directory and then iCloud magic will save me from future self. Right?

Wrong.

→  git co my-branch
fatal: unable to read tree ████████████████████████████████████████

Oh, no problem. I'll just pull the files back down.

Wrong. Again.

→  git pull
error: refs/heads/gh-pages does not point to a valid object!
error: refs/remotes/origin/HEAD does not point to a valid object!
error: refs/remotes/origin/gh-pages does not point to a valid object!
error: refs/heads/gh-pages does not point to a valid object!
error: refs/remotes/origin/HEAD does not point to a valid object!
error: refs/remotes/origin/gh-pages does not point to a valid object!
error: refs/heads/gh-pages does not point to a valid object!
error: refs/remotes/origin/HEAD does not point to a valid object!
error: refs/remotes/origin/gh-pages does not point to a valid object!
error: Could not read ████████████████████████████████████████
error: Could not read ████████████████████████████████████████
error: refs/heads/gh-pages does not point to a valid object!
error: refs/remotes/origin/HEAD does not point to a valid object!
error: refs/remotes/origin/gh-pages does not point to a valid object!
remote: Total 0 (delta 0), reused 0 (delta 0), pack-reused 0
fatal: bad object ████████████████████████████████████████
error: github.com:<ORG>/<REPO>.git did not send all necessary objects

What's going on?

In short? Optimization. See, iCloud storage has a "neat" optimization feature so that "unnecessary" files aren't downloaded locally. The problem with this, as you may have guessed already if you're here, is that git uses a lot of artifacts that get left out of the party and a lack of them causes issues like what I'm seeing above.

What to do?

Option 1: Disable iCloud Optimization

After some research there's no way to exempt a directory from Optimization like you can with Time Machine. Since optimization is either "on" or "off", it's up to you to see if you'd like to turn off optimization overall just for git in particular. As a one-off, if you don't want to turn off optimization, you can try to trigger a file to download by recusively trying to touch and/or open individual files. Caveat: an initial test of this method was not fruitful.

Option 2: Just push the changes

The alternative of course is to just use git as the tool is intended to be used and regularly push your changes. In terms of coolness I do wish I had found a relatively quick-and-easy way to force the iCloud sync situation to work, but in hindsight this would just encourage me to not push my changes as frequently as I so obviously should. And since I have access to my own repos, there's nothing stopping me from using a shell script to clone / pull changes from repos in my org every time I switch between my work and personal computers.

Banner sources: "Like an adult" image shamelessly stolen from the greatly missed Hyperbole & a Half comic. Also iCloud and Github logos.

As I was digging around the internet encountering conflicting configurations and advice about how to setup SSL for Jenkins, I decided that I should really split this bit out of the Jenkins Post Mortem (still in progress).

Into the Trenches

It took a few iterations of failure before finding a working configuration. Here's what I tried that didn't work, along with the error behavior, to hopefully aid others in their troubleshooting.

Using Java's cacerts store

I started by basically doing what anyone else would do: googling / ducking for variations of "how to setup ssl for Jenkins". I started by trying to follow the instructions provided by CloudBees. These instructions worked after a fashion, and I've included them below for a quick reference:

sudo mkdir $JENKINS_HOME/.keystore

sudo chown jenkins:jenkins $JENKINS_HOME/.keystore

cp $JAVA_HOME/jre/lib/security/cacerts $JENKINS_HOME/.keystore

$JAVA_HOME/bin/keytool -keystore $JENKINS_HOME/.keystore/cacerts -import -alias <YOUR_ALIAS_HERE> -file <YOUR_CA_FILE>

<YOUR_ALIAS_HERE\> should be a helpful name, e.g. jenkins-wildcard if you're using a wildcard cert, and <YOUR_CA_FILE> would be the name of your x509 cert, e.g. wildcard-example-com-x509.crt. The crt you need to download from your certificate provider.

After setting all that up, I ran into a little snag:

→  sudo service jenkins start
Starting Jenkins                                           [  OK  ]

→  sudo service jenkins status
jenkins dead but pid file exists

Why is Jenkins dead? Well, the instructions I linked / copied above neglect to tell you to disable the HTTP port and set the HTTPS port, like so:

#disable HTTP
JENKINS_PORT="-1"

#enable HTTPS
JENKINS_HTTPS_PORT="8443"

Source: IconArchive

Once that's all working, though, you'll still end up with...

Self Signed Cert Error / Warning

For the curious, the browser plugins are 1Password, Momentum, AdBlocker Plus, Amazon, and Chromecast in that order.

I suspected part of that was perhaps it wasn't able to pull my specific cert out of the store, and I was a bit curious about what all was in there, so I took a look:

→  $JAVA_HOME/bin/keytool -list -keystore $JAVA_HOME/lib/security/cacerts
Enter keystore password:
Keystore type: JKS
Keystore provider: SUN

Your keystore contains 167 entries
{{ snip }}

Well if nothing else there are 167 entries in there and that's an unwiedly beast to troubleshoot to know for sure. Instead, I tried to create my own store that would only house my wildcard cert. Things got a bit unfortunate here, and I spun my wheels for quite a bit.

Trying to make my cert store

This was a bit difficult because it seems like some of the errors it threw were red herrings. I followed the instructions here to setup a keystore and they were as follows, my errors included:

→  openssl pkcs12 -export -out jenkins_keystore.p12 -passout 'pass:changeit' -inkey wildcard.example.com.key -in wildcard.example.com.crt -certfile ca-bundle.crt -name wildcard-example

→  $JAVA_HOME/bin/keytool -importkeystore -srckeystore jenkins_keystore.p12 -srcstorepass 'changeit' -srcstoretype PKCS12 -srcalias wildcard.example -deststoretype JKS -destkeystore jenkins_keystore.jks -deststorepass 'changeit' -destalias wildcard.example
Importing keystore jenkins_keystore.p12 to jenkins_keystore.jks...
keytool error: java.lang.Exception: Alias <wildcard.example> does not exist

→  $JAVA_HOME/bin/keytool -list -keystore jenkins_keystore.p12
Enter keystore password:
Keystore type: JKS
Keystore provider: SUN

Your keystore contains 1 entry

wildcard-example, Feb 8, 2018, PrivateKeyEntry,
Certificate fingerprint (SHA1): ██:██:██:██:██:██:██:██:██:██:██:██:██:██:██:██:██:██:██:██

→  $JAVA_HOME/bin/keytool -importkeystore -srckeystore jenkins_keystore.p12 -srcstorepass 'changeit' -srcstoretype PKCS12 -srcalias wildcard-example -deststoretype JKS -destkeystore jenkins_keystore.jks -deststorepass 'changeit' -destalias wildcard-example
Importing keystore jenkins_keystore.p12 to jenkins_keystore.jks...

Warning:
The JKS keystore uses a proprietary format. It is recommended to migrate to PKCS12 which is an industry standard format using "keytool -importkeystore -srckeystore jenkins_keystore.jks -destkeystore jenkins_keystore.jks -deststoretype pkcs12".

→  sudo cp jenkins_keystore.jks $JENKINS_HOME/.keystore/

→  sudo chown jenkins:jenkins $JENKINS_HOME/.keystore/jenkins_keystore.jks

→  sudo chmod 600 !$
sudo chmod 600 $JENKINS_HOME/.keystore/jenkins_keystore.jks


→  sudo vim /etc/sysconfig/jenkins

→  sudo service jenkins restart
Shutting down Jenkins                                      [FAILED]
Starting Jenkins                                           [  OK  ]

→  ^restart^status
sudo service jenkins status
jenkins dead but pid file exists

And what was in the log?

→  sudo tail -n 25 /var/log/jenkins/jenkins.log
        at Main._main(Main.java:294)
        at Main.main(Main.java:132)
Caused by: winstone.WinstoneException: No SSL key store found at /etc/jenkins/jenkins_keystore.jks
        at winstone.AbstractSecuredConnectorFactory.configureSsl(AbstractSecuredConnectorFactory.java:64)
        at winstone.HttpsConnectorFactory.start(HttpsConnectorFactory.java:41)
        at winstone.Launcher.spawnListener(Launcher.java:207)
        ... 8 more
Feb 08, 2018 9:12:28 PM winstone.Logger logInternal
SEVERE: Container startup failed
java.io.IOException: Failed to start a listener: winstone.HttpsConnectorFactory
        at winstone.Launcher.spawnListener(Launcher.java:209)
        at winstone.Launcher.<init>(Launcher.java:150)
        at winstone.Launcher.main(Launcher.java:354)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at Main._main(Main.java:294)
        at Main.main(Main.java:132)
Caused by: winstone.WinstoneException: No SSL key store found at /etc/jenkins/jenkins_keystore.jks
        at winstone.AbstractSecuredConnectorFactory.configureSsl(AbstractSecuredConnectorFactory.java:64)
        at winstone.HttpsConnectorFactory.start(HttpsConnectorFactory.java:41)
        at winstone.Launcher.spawnListener(Launcher.java:207)
        ... 8 more

So a couple of things. You can see at the top there I neglected my alias and typoed it, but I left the mistake so you could see the command to retreive the alias from the store.

Also, JENKINS_HOME is /var/lib/jenkins. This is the standard configuration. You may notice that the Winstone log is looking in /etc/jenkins, which I didn't set, so it appears on the surface at least that this is defaulted somewhere. Noticing the path descrepancy I moved the keystore there, but it unfortunately still threw the same error. I took back to the internet and found this, which I took as a sign that maybe, just maybe, I needed to pursue a different solution.

Simple Setup: Jenkins + Nginx Reverse Proxy

Jenkins

To start, you'll need to set the Jenkins variables in /etc/sysconfig/jenkins:

→  sudo cat /etc/sysconfig/jenkins | grep -v "\#"
JENKINS_HOME="/var/lib/jenkins"

JENKINS_JAVA_CMD=""

JENKINS_USER="jenkins"


JENKINS_JAVA_OPTIONS="-Djava.awt.headless=true"

JENKINS_PORT="8080"

JENKINS_LISTEN_ADDRESS="127.0.0.1"

JENKINS_HTTPS_PORT=""

JENKINS_HTTPS_KEYSTORE=""

JENKINS_HTTPS_KEYSTORE_PASSWORD=""

JENKINS_HTTPS_LISTEN_ADDRESS=""


JENKINS_DEBUG_LEVEL="5"

JENKINS_ENABLE_ACCESS_LOG="no"

JENKINS_HANDLER_MAX="100"

JENKINS_HANDLER_IDLE="20"

JENKINS_ARGS=""

This is a minimal configuration, so you may have modified some values like JENKINS_ARGS if you have an existing Jenkins setup. This is fine. The main values to focus on here are JENKINS_PORT and JENKINS_LISTEN_ADDRESS. You should not set any of the JENKINS_HTTPS_* variables as the HTTPS configuration, if you choose it, will be handled by nginx.

Nginx

Setting up the Nginx proxy was so simple that I wish I had started there, but hey you live and learn. To set this up, install nginx:

sudo yum install nginx -y

I don't want to use the default nginx configurations as it does a bunch of wizarding that I don't want to inherit or troubleshoot. The latter being the usual case.

→  cd /etc/nginx
→  sudo mkdir nginx_defaults
→  sudo mv * nginx_defaults/
mv: cannot move ‘nginx_defaults’ to a subdirectory of itself, ‘nginx_defaults/nginx_defaults’
→  sudo mv nginx_defaults/mime.types .
→  sudo mkdir certs
→  tree
.
├── certs
├── mime.types
└── nginx_defaults
    ├── conf.d
    │   └── virtual.conf
    ├── default.d
    ├── fastcgi.conf
    ├── fastcgi.conf.default
    ├── fastcgi_params
    ├── fastcgi_params.default
    ├── koi-utf
    ├── koi-win
    ├── mime.types.default
    ├── nginx.conf
    ├── nginx.conf.default
    ├── scgi_params
    ├── scgi_params.default
    ├── uwsgi_params
    ├── uwsgi_params.default
    └── win-utf

We're going to be using mime.types, so definitely keep that one in the parent directory. Also, in this case ignore the cannot move error since that is expected.

HTTP Configuration

The following is a complete nginx.conf file for the HTTP proxy configuration, just make sure to change jenkins.example.com to your Jenkins URL.

The bulk of this configuration is taken from the Jenkins wiki page with the relevant information added in from the default nginx.conf.

user nginx;
worker_processes auto;
error_log /var/log/nginx/error.log;
pid /var/run/nginx.pid;

events {
    worker_connections 1024;
}

http {
    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    access_log  /var/log/nginx/access.log  main;

    sendfile            on;
    tcp_nopush          on;
    tcp_nodelay         on;
    keepalive_timeout   65;
    types_hash_max_size 2048;

    include             /etc/nginx/mime.types;
    default_type        application/octet-stream;

    index   index.html index.htm;

    server {
      listen          80;       # Listen on port 80 for IPv4 requests

      server_name     jenkins.example.com;  ## FIXME: This should be your URL

      #this is the jenkins web root directory (mentioned in the /etc/default/jenkins file)
      root            /var/run/jenkins/war/;

      access_log      /var/log/nginx/access.log;
      error_log       /var/log/nginx/error.log;
      ignore_invalid_headers off; #pass through headers from Jenkins which are considered invalid by Nginx server.

      location ~ "^/static/[0-9a-fA-F]{8}\/(.*)$" {
        #rewrite all static files into requests to the root
        #E.g /static/12345678/css/something.css will become /css/something.css
        rewrite "^/static/[0-9a-fA-F]{8}\/(.*)" /$1 last;
      }

      location /userContent {
        #have nginx handle all the static requests to the userContent folder files
        #note : This is the $JENKINS_HOME dir
        root /var/lib/jenkins/;
        if (!-f $request_filename){
          #this file does not exist, might be a directory or a /**view** url
          rewrite (.*) /$1 last;
          break;
        }
        sendfile on;
      }

      location @jenkins {
          sendfile off;
          proxy_pass         http://127.0.0.1:8080;
          proxy_redirect     default;
          proxy_http_version 1.1;

          proxy_set_header   Host              $host;
          proxy_set_header   X-Real-IP         $remote_addr;
          proxy_set_header   X-Forwarded-For   $proxy_add_x_forwarded_for;
          proxy_set_header   X-Forwarded-Proto $scheme;
          proxy_max_temp_file_size 0;

          #this is the maximum upload size
          client_max_body_size       10m;
          client_body_buffer_size    128k;

          proxy_connect_timeout      90;
          proxy_send_timeout         90;
          proxy_read_timeout         90;
          proxy_request_buffering    off; # Required for HTTP CLI commands in Jenkins > 2.54
      }

      location / {
        # Optional configuration to detect and redirect iPhones
        if ($http_user_agent ~* '(iPhone|iPod)') {
          rewrite ^/$ /view/iphone/ redirect;
        }

        try_files $uri @jenkins;
      }
    }
}

HTTPS Configuration

The HTTPS configuration is almost identical, save a few changes:

• Changing the listen port from 80 to 443
• Adding the section I've labeled # HTTP redirect, so that http://jenkins.example.com is redirected to https://jenkins.example.com
• Adding the block that I've noted as # SSL in the nginx.conf which provides the path to the cert files
  • Make sure to update the paths / filenames to match your setup
• Updating the proxy_redirect value

Quick cert detour

For your certs: you'll need the key file and the x509 cert. Depending on your provider, these may be hard to identify so to hopefully help you I'm going to show how I inspected the certs I downloaded from our cert provider:

→  sha1sum *
f1c██████████████████████████████████c71  wildcard.example.com.apache.crt
819██████████████████████████████████444  wildcard.example.com.ee_x509.crt
304██████████████████████████████████992  wildcard.example.com.i1_issuer.crt
1a8██████████████████████████████████c4c  wildcard.example.com.pkcs7.p7s
f1c██████████████████████████████████c71  wildcard.example.com.plesk.crt

From this you can see that the top and bottom files are the same, but the middle three are different from each other.

Inspecting further:

→  openssl x509 -noout -text -in wildcard.example.com.apache.crt
Certificate:
    Data:
        Version: █████
        Serial Number: ████████
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: C=US, O=GeoTrust Inc., CN=GeoTrust Global CA
        Validity
            Not Before: Aug ██ 21:39:32 ████ GMT
            Not After : May ██ 21:39:32 ████ GMT
        Subject: C=US, O=GeoTrust Inc., CN=RapidSSL SHA256 CA - G3
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (2048 bit)
                Modulus:
{{{ SNIP }}}

This wildcard.example.com.apache.crt is the intermediary cert issued by "GeoTrust Global CA" (the root) to "GeoTrust RapidSSL SHA256 CA - G3" (the intermediary). An intermediary cert sits between the wildcard cert issued to us/me and the root cert. Usually when you download a bunch of certificate files from your provider this a file like this is included because not all intermediary CAs are trusted by all sources, so in some cases you may need to provide the intermediate cert. To see if your intermediate cert is trusted by your browser you can check that browser's certificate store.

Quick example for how to check a cert store in Firefox: go to about:preferences in the address bar and scroll to the bottom of the page. Here, you can see that "GeoTrust's RapidSSL SHA256 CA - 3" is trusted by Firefox:

Click image to view full size.

→  openssl x509 -noout -text -in wildcard.example.com.ee_x509.crt
Certificate:
    Data:
        Version: █████
        Serial Number: ████████
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: C=US, O=GeoTrust Inc., CN=RapidSSL SHA256 CA - G3
        Validity
            Not Before: Dec ██ 19:21:43 ████ GMT
            Not After : Jan ██ 16:01:28 ████ GMT
        Subject: CN=*.example.com
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (2048 bit)
                Modulus:
{{{ SNIP }}}

The Subject: CN=*.example.com tells us that this is the file we want to use as the ssl_certificate in the nginx configuration. This is actually a type of pem file, so I'm actually going to change the extension when I move this file and the key file to /etc/nginx/certs:

→  sudo cp wildcard.example.com.ee_x509.crt /etc/nginx/certs/wildcard.example.com.pem
→  sudo cp wildcard.example.com.key /etc/nginx/certs/

The HTTPS Configuration

As before, the complete HTTPS configuration is below, making sure you change jenkins.example.com and the cert paths to match your configuration:

user nginx;
worker_processes auto;
error_log /var/log/nginx/error.log;
pid /var/run/nginx.pid;

events {
    worker_connections 1024;
}

http {
    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    access_log  /var/log/nginx/access.log  main;

    sendfile            on;
    tcp_nopush          on;
    tcp_nodelay         on;
    keepalive_timeout   65;
    types_hash_max_size 2048;

    include             /etc/nginx/mime.types;
    default_type        application/octet-stream;

    index   index.html index.htm;

    # HTTP redirect
    server {
        listen 80;
        server_name jenkins.example.com;
        return 301 https://$host$request_uri;
    }

    server {
      listen          443;       # Listen on port 443 for IPv4 requests

      server_name     jenkins.example.com;

      # SSL
      ssl on;
      ssl_certificate            /etc/nginx/certs/wildcard.example.com.pem;
      ssl_certificate_key        /etc/nginx/certs/wildcard.example.com.key;
      ssl_protocols              TLSv1.2;
      ssl_ciphers                'EECDH+AESGCM:EDH+AESGCM';
      ssl_prefer_server_ciphers  on;
      ssl_session_cache          shared:SSL:10m;

      #this is the jenkins web root directory (mentioned in the /etc/default/jenkins file)
      root            /var/run/jenkins/war/;

      access_log      /var/log/nginx/access.log;
      error_log       /var/log/nginx/error.log;
      ignore_invalid_headers off; #pass through headers from Jenkins which are considered invalid by Nginx server.

      location ~ "^/static/[0-9a-fA-F]{8}\/(.*)$" {
        #rewrite all static files into requests to the root
        #E.g /static/12345678/css/something.css will become /css/something.css
        rewrite "^/static/[0-9a-fA-F]{8}\/(.*)" /$1 last;
      }

      location /userContent {
        #have nginx handle all the static requests to the userContent folder files
        #note : This is the $JENKINS_HOME dir
        root /var/lib/jenkins/;
        if (!-f $request_filename){
          #this file does not exist, might be a directory or a /**view** url
          rewrite (.*) /$1 last;
          break;
        }
        sendfile on;
      }

      location @jenkins {
          sendfile off;
          proxy_pass         http://127.0.0.1:8080;
          proxy_redirect     http://localhost:8080 $scheme://jenkins.example.com;
          proxy_http_version 1.1;

          proxy_set_header   Host              $host;
          proxy_set_header   X-Real-IP         $remote_addr;
          proxy_set_header   X-Forwarded-For   $proxy_add_x_forwarded_for;
          proxy_set_header   X-Forwarded-Proto $scheme;
          proxy_max_temp_file_size 0;

          #this is the maximum upload size
          client_max_body_size       10m;
          client_body_buffer_size    128k;

          proxy_connect_timeout      90;
          proxy_send_timeout         90;
          proxy_read_timeout         90;
          proxy_request_buffering    off; # Required for HTTP CLI commands in Jenkins > 2.54
      }

      location / {
        # Optional configuration to detect and redirect iPhones
        if ($http_user_agent ~* '(iPhone|iPod)') {
          rewrite ^/$ /view/iphone/ redirect;
        }

        try_files $uri @jenkins;
      }
    }
}

Cleaning up the loose ends

Fixing the Reverse Proxy Error

Once you have your reverse proxy setup in either case, you'll most likely encounter this error:

To resolve this, go to Manage Jenkins -> Configure System, or go to https://${YOUR_JENKINS_URL}/configure, and update your configuration to use your new HTTP or HTTPS address, as can be seen in the side by side configuration below:

Fixing Github Oauth

If you have Github oauth configured you might have a mild heart attack when auth fails to redirect because of your proxy. Not to fear, this is another quick fix. Log into your Github account / Github org and go to where you've configured oauth:

And then go into the configuation and update both the Homepage URL and the Authorization Callback URL to your new HTTP or HTTPS address.

That's it! You can now auth with Github normally.

Updating Github webhooks

If you are using Github webhooks, you'll need to update them from something like:

http://jenkins.example.com:8080/github-webhook/

To either your new HTTP or HTTPS URL, e.g.:

http://jenkins.example.com/github-webhook/
https://jenkins.example.com/github-webhook/

Make sure you scroll to the bottom and save your configuration!

Fixing SSL Webhooks - "Peer certificate cannot be authenticated"

If you are using SSL, as you likely are if you're reading this, you may encounter the following after updating your webhooks:

Opening up the most recent one to take a look:

What does this mean?

If you recall, earlier I checked the Firefox cert store to see if it had "RapidSSL SHA256 CA - G3" in it - this is the intermediary that signed the certificate that I'm using. I also mentioned that not all intermediate sources are trusted. Here, it appears that while Firefox does trust my intermediate source, Github does not.

How to fix this? Certificate bundles are actually just a series of PEM files in a single file, so we need to either pre-pend or append the intermediary to the cert file that nginx is using.

I quickly tried pre-pending first, and encountered this error when I restarted nginx:

→  sudo service nginx restart
nginx: [emerg] SSL_CTX_use_PrivateKey_file("/etc/nginx/certs/wildcard.example.com.key") failed (SSL: error:0B080074:x509 certificate routines:X509_check_private_key:key values mismatch)
nginx: configuration file /etc/nginx/nginx.conf test failed

So I reversed the cert order and violá, nginx started and the page loaded. Next, I tried resending that last payload to see if SSL verification passed:

Excellent.

For completeness, I checked back in after letting a few builds run to verify that the webhook history was green and healthy once more, and indeed it was/is:

Header image: Word Cloud drawn by Word Clouds Generator

Hold on to your seats

Cause this is going to be a bit of a ride. This is a tale of explosions, defeat, perserverance, and ultimate victory.

I'm giving you a spoiler for the ultimate victory because, like all situations of prolonged pain, there were several points that I didn't think we were going to get there.

Hooked? Here's how it began

We were having an issue where some of our Jenkins jobs were hanging on the git clone. This is something relatively new that started happening this week, so after manually killing and kicking a couple of jobs until they Just Worked (already forgetting the HTML/PEM file lesson) I decided to do a rollback.

This wasn't too huge of a deal, really. We use the weekly Jenkins builds, and I keep the previous week's build handy, so:

sudo service jenkins stop
sudo cp ~/backup/jenkins-v2.103.war /usr/lib/jenkins/jenkins.war
sudo service jenkins start

Everything came up normally, except for the fact that the jobs still hung on git clone. Oh well. Rinse repeat the above, replace jenkins-v2.103.war with jenkins-v2.104.war. Everything came back live, nothing to see here.

A few of the plugins were upgraded this week as well. Since one of them was the Github plugin, and the issue was with git clone, I figured I'd try to roll back that first.

Feel free to click so you can view this insanity in all its glory.

Source: Giphy, Firefly

But wait, what?

It's worth now noting what fired through my brain in rapid succession:

1. We use Github Oauth
2. This appears to be something with Github core
3. How could could a plugin rollback do this
4. I knew this box was fragile, I should have snapshotted it before
5. Is there a snapshot?
6. ...Of course the last one is from freaking December.

What to do: Act 1, Scenes 1-3

My first instinct, as any battle hardened person will tell you, was

to

Google

everything.

Filtering out the seemingly unending volumes of advice about how to roll plugin versions forward and backward, using the UI thank you very much for that, I found some advice about how to install plugins using their CLI.

YIL that Jenkins has a CLI.

I went to find where to download it, but lo: you need a working version of Jenkins to download the CLI. And the version of the CLI is dependent on the Jenkins release version as well. And you can only download it from using your actual Jenkins install, e.g.

wget -O /desired/path/to/jenkins-cli.war https://${JENKINS_URL}/jnlpJars/jenkins-cli.jar

You can also go to the latter path in your browser if your installation is up and running.

Which mine was not.

Oh! I know! I'll make a fresh Jenkins box with the same version and download the CLI from there!

I'm not going to detail this part for you (yet), but stay tuned. I got the CLI, but primary Jenkins was so hosed that pointing the CLI at it threw a Java exception. Even if this hadn't happened, though, I was reminded when I created a sandbox Jenkins that I probably still would have run into difficulties without the exception since Github auth was still enabled and I would have needed to disable it or create a token for the CLI. Which I would have needed access to the UI to do. (Jenkins is really reliant on having that UI up and running.)

It's worth mentioning that while all that was going on I had concurrently attempted to spin up a new instance using an AMI I had made from the December image, but when I tried to start Jenkins on that instance it also died and threw exceptions. Not in the web browser, in the browser I was just presented with a lovely site unreachable error. I sshed into the instance to see Jenkins' logs (/var/log/jenkins/jenkins.log) and there were several Java exceptions everywhere. Also, importantly, errors referencing missing jobs.

Source: Gunnerkrigg Court. It's an awesome web comic.

It was at this point I practiced some deep breathing.

What to do: Act 2

So I was half hoping that at least some of the exceptions could be handled by copying over the jobs from the dying dying dead Jenkins to the Dec Jenkins. I did this by shutting off the dead one's EC2 instance, detaching the volume, and attaching it to the December Jenkins instance. Amazon has how to attach a volume in the console documented very well. It's worth noting you use essentially the same process, when the instances in question are powered off, to detach the volume.

I think it's important to clarify again, since you may encounter instructions of other ways to unmount disk volumes while a system is powered on, e.g. this EBS doc by Amazon, that in this case those instructions will not work as you can't (safely) detach the root and only volume from a running system.

Anywho, after the volume is attached then just create the mount point, get volume list, mount volume:

→ sudo mkdir /jenkins-defunct

→ lsblk
NAME    MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
xvda    202:0    0  512G  0 disk
└─xvda1 202:1    0  512G  0 part /
xvdf    202:2    0  512G  0 disk
└─xvdf1 202:3    0  512G  0 part /

→ sudo mount /dev/xvdf1 /jenkins-defunct

Breathing. Ok.

Now it's time to rsync.

→ sudo mv /var/lib/jenkins/jobs{,--bkp}
→ sudo rsync -a /jenkins-defunct/var/lib/jenkins/jobs /var/lib/jenkins/jobs

And now we wait.

And wait.

*** Skipping any contents from this failed directory ***
rsync: recv_generator: mkdir "/var/lib/jenkins/jobs/jobs/${SOME_PLACEHOLDER_JOB}/workspace" failed: No space left on device (28)
*** Skipping any contents from this failed directory ***
rsync: mkstemp "/var/lib/jenkins/jobs/jobs/${SOME_PLACEHOLDER_JOB}/.config.xml.dovJOF" failed: No space left on device (28)
rsync: mkstemp "/var/lib/jenkins/jobs/jobs/${SOME_PLACEHOLDER_JOB}/.disk-usage.xml.xnYWQ8" failed: No space left on device (28)
rsync: mkstemp "/var/lib/jenkins/jobs/jobs/${SOME_PLACEHOLDER_JOB}/.github-polling.log.RVYdTB" failed: No space left on device (28)
rsync: mkstemp "/var/lib/jenkins/jobs/jobs/${SOME_PLACEHOLDER_JOB}/.nextBuildNumber.EtLAV4" failed: No space left on device (28)
rsync: recv_generator: mkdir "/var/lib/jenkins/jobs/jobs/${SOME_PLACEHOLDER_JOB}/workspace@tmp" failed: No space left on device (28)
*** Skipping any contents from this failed directory ***
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1039) [sender=3.0.6]

After a couple hours of seemingly silent, copying bliss I was abruptedly introduced to an unhumanly countable number of lines like that.

But wait, I filled up the whole drive? With jobs?

$ df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/xvda1      504G  276G  229G  55% /
devtmpfs        7.9G   64K  7.9G   1% /dev
tmpfs           7.9G     0  7.9G   0% /dev/shm
/dev/xvdf1      504G  258G  247G  52% /jenkins-defunct

I am using... half the drive. What.

One of my coworker's offered to ssh in at this point to see if he saw anything.

But he could not, he was given an out of disk error.

I looked at CloudWatch metrics for the alarm and thankfully df didn't lie there: using half disk.

What gives?

inodes

The short version, since they aren't the focus here, is that inodes store file and directory metadata on *nix filesystems. To the passerby, they become important when you have a ton of tiny files, as Jenkins clearly does, and you run out:

$ df -ih
Filesystem     Inodes IUsed IFree IUse% Mounted on
/dev/xvda1        32M   32M     0  100% /
devtmpfs         2.0M   477  2.0M    1% /dev
tmpfs            2.0M     1  2.0M    1% /dev/shm
/dev/xvdf1        32M   27M  5.7M   83% /jenkins-defunct

(If you'd like to read more on what inodes are, please check out this Linux Magazine article.)

Of course I ran out of inodes.

The quickest way to address this problem is to resize the disk. This is the option I went with, since this is production Jenkins and we're now several hours into this outage.

Since this is a short term solution, I stopped the instance, resized the root volume to 2 TB, rebooted, and restarted the rsync.

By the way, here is an example of why to rsync rather than cp: the latter would just completely start over and write over what it had already done as needed, taking more time, whereas rsync will pick up where it left off.

That said, it still took another hour or two for the sync to complete.

While this was going on, I was pondering my next move. Also: getting really tired as it was late now.

What to do: Act 3, all the climatic scenes

I read a bit more on Jenkins admin. Since everything should be hypothetically recoverable from JENKINS_HOME, I decided to try re-installing all the plugins on a new Jenkins instance (don't ask me how many I have running now), and the copy that's instances /var/lib/jenkins/plugins directory back to the original.

This ultimately ended in failure, but I ended up with a very useful shell script that allowed me to install plugins very quickly.

Another spoiler: vim tricks are super handy.

Spinning up Jenkins on a Linux AMI

I'm going to go a little detailed here for those new to installing Jenkins.

The instance configuration:

• Instance Type: m5.xlarge
• Security: security group with ports 22, 80, 8080, 443, and 4443 are open / accessible on our VPC and via VPN.
• EBS type: 512 GB GP2
  • In hindsight, though, I recommend using IO1. Cost is similar and would help speed up rsync later.
• AMI: Amazon Linux AMI 2017.09.1 (HVM)

ssh into the instance and:

• Update
• Install some basic tools
• Create a group
• Use visudo to enable passwordless sudo
• Create your user
• Add your user to the group
• Switch to the user account
• Install a handy prompt and useful rc files

Here we go.

$ sudo yum upgrade -y
$ sudo yum install git tmux tree htop ack unzip -y
$ sudo groupadd admin
$ sudo useradd quintessence
$ sudo usermod -aG admin quintessence
$ sudo EDITOR=vim visudo
$ sudo su - quintessence
Last login: Fri Feb  2 16:49:01 UTC 2018 on pts/0
[quintessence@ip-███-███-███-███ ~]$ sudo ls /etc/
acpi               blkid                      csh.cshrc
{{{ snip }}}

[quintessence@ip-███-███-███-███ ~]$ git clone https://github.com/jhunt/env
Cloning into 'env'...
remote: Counting objects: 713, done.
remote: Total 713 (delta 0), reused 0 (delta 0), pack-reused 713
Receiving objects: 100% (713/713), 128.96 KiB | 18.42 MiB/s, done.
Resolving deltas: 100% (419/419), done.
[quintessence@ip-███-███-███-███ ~]$ cd env/
[quintessence@ip-███-███-███-███ env]$ ./install
setting up dot files in ~
configuring vim...
copying in ~/bin scripts...
  installing jq...
  installing spruce (v1.8.2)...
configuring git...
setting up ~/.bashrc...
hostname: No address associated with name
[quintessence@ip-███-███-███-███ env]$ cd ..
[quintessence@ip-███-███-███-███ ~]$ vim .host
[quintessence@ip-███-███-███-███ ~]$ source ~/.bashrc
+033+16:52:54:8:0 ███.███.███.███/20 quintessence@jenkins ~
→  

For visudo, here's the magic line to allow admin group passwordless sudo:

%admin        ALL=(ALL)       NOPASSWD: ALL

As a quick aside: the .host file is used by the prompt to display the hostname if none is set, which is the case for this dev instance. Right now it just has jenkins in it and that's what you see in the prompt above. For the remainder of the blog post when you see →, that's just part of my prompt.

Now for the Jenkins install

I'm going to use yum to install the last stable release of Jenkins to make sure that loads. Here's the needful, spaced out with output:

→  sudo wget -O /etc/yum.repos.d/jenkins.repo http://pkg.jenkins-ci.org/redhat-stable/jenkins.repo
--2018-02-02 16:53:25--  http://pkg.jenkins-ci.org/redhat-stable/jenkins.repo
Resolving pkg.jenkins-ci.org (pkg.jenkins-ci.org)... 52.202.51.185
Connecting to pkg.jenkins-ci.org (pkg.jenkins-ci.org)|52.202.51.185|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 85
Saving to: ‘/etc/yum.repos.d/jenkins.repo’

/etc/yum.repos.d/jenkins.repo                                       100%[===================================================================================================================================================================>]      85  --.-KB/s    in 0s

2018-02-02 16:53:26 (30.9 MB/s) - ‘/etc/yum.repos.d/jenkins.repo’ saved [85/85]



→  sudo rpm --import http://pkg.jenkins-ci.org/redhat-stable/jenkins-ci.org.key



→  sudo yum install jenkins -y
Loaded plugins: priorities, update-motd, upgrade-helper
amzn-main                                                                                                                                                                                                                                                | 2.1 kB  00:00:00
amzn-updates                                                                                                                                                                                                                                             | 2.5 kB  00:00:00
jenkins                                                                                                                                                                                                                                                  | 2.9 kB  00:00:00
jenkins/primary_db                                                                                                                                                                                                                                       |  23 kB  00:00:00
Resolving Dependencies
--> Running transaction check
---> Package jenkins.noarch 0:2.89.3-1.1 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

================================================================================================================================================================================================================================================================================
 Package                                                          Arch                                                            Version                                                                Repository                                                        Size
================================================================================================================================================================================================================================================================================
Installing:
 jenkins                                                          noarch                                                          2.89.3-1.1                                                             jenkins                                                           71 M

Transaction Summary
================================================================================================================================================================================================================================================================================
Install  1 Package

Total download size: 71 M
Installed size: 71 M
Downloading packages:
jenkins-2.89.3-1.1.noarch.rpm                                                                                                                                                                                                                            |  71 MB  00:00:01
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Installing : jenkins-2.89.3-1.1.noarch                                                                                                                                                                                                                                    1/1
  Verifying  : jenkins-2.89.3-1.1.noarch                                                                                                                                                                                                                                    1/1

Installed:
  jenkins.noarch 0:2.89.3-1.1

Complete!

When I ran sudo service jenkins start to start Jenkins, I received the following because Jenkins needs Java 8 and apparently Amazon Linux is shipping with Java 7:

→  sudo service jenkins start
Starting Jenkins Jenkins requires Java8 or later, but you are running 1.7.0_161-mockbuild_2017_12_19_23_46-b00 from /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.161.x86_64/jre
java.lang.UnsupportedClassVersionError: 51.0
        at Main.main(Main.java:124)
                                                           [  OK  ]

I'm going to remove Java 7 and install Java 8 (output not included for the yum commands):

→  sudo yum remove java-1.7.0-openjdk -y
→  sudo yum install java-1.8.0 -y
→  sudo service jenkins start
Starting Jenkins                                           [  OK  ]

I'm also going to add the jenkins service to start on boot:

→  sudo chkconfig jenkins on

Ok, now that I have all of that: does bare Jenkins load?

Yes.

This the first moment of relief that I've had in hours.

Source: IconArchive

As a result, I breezed through the next bit:

• Unlocked grabbing the initial admin password as indicated on the splash page
• Selected to have Jenkins Install the Recommended Plugins
• Set up my Jenkins username + password (also part of the setup wizard)
  • Made sure my Jenkins username matched my Github username to prevent redundancy when hooking up Github Oauth
• Made sure Jenkins loaded

You can click on this image for full size.

This is the happy time. So happy, I'm gonna use that emoji again:

Ok, now to stop the service, update to the latest weekly build so it matches the desired environment, and then restart the bare Jenkins. Not anticipating any issues since this is still a bare environment. Skipping the step where I download the war file:

→  ls
bin  code  env  jenkins-v2.104.war

→  sudo mv /usr/lib/jenkins/jenkins.war{,_old}

→  sudo cp jenkins-v2.104.war /usr/lib/jenkins/jenkins.war

→  sudo ls /usr/lib/jenkins/
jenkins.war  jenkins.war_old

→  sudo service jenkins restart

On restart this looks mostly the same:

You can click on this image for full size.

With a crucial difference:

At this point, I think we can upgrade to a full on smile:

Source: IconArchive

Plugin installs

I discovered very quickly that installing these plugins via the UI was going to be a nightmare because their search feature is a challenge. And not in the "what doesn't kill you makes you stronger" way. It was more like this:

Source: This LifeHacker AUS article. I'll admit to not verifying the quote because it fits my needs here.

I'll show you what I mean, and then I'll show you how I used the Jenkins CLI (remember that?) to get around it.

Searching ... Searching ... Searching ...

You can see the names of the plugins, as Jenkins understands them, in /var/lib/jenkins/plugins. One of the plugins we have is cloudbees-folder so let's try to find that with the UI and install it.

Searching for "folder" by itself was really generic, so I tried to search for "cloudbees". It was also unhelpful. I did happen to notice, though, the URL for the plugins is actually linked to the download directory:

You can click on this image for full size.

This is somewhat helpful as it shows me to go to http://updates.jenkins-ci.org/download/plugins/ for the download list. Here, the plugins appear the same as they do once installed rather than the "friendly" or "long" names that they are given.

Which is great and all, and sure this is a lot easier to manage than that hot mess of a search, but how to install them?

Source: Emojipedia

Jenkins CLI to save the day

It is at this point that I remember the only thing stopping me from using the CLI before was that Jenkins was so hosed it wouldn't even talk to it.

That isn't the case now though. So:

→  wget -O ~/jenkins-cli.jar  https://${JENKINS_PUBLIC_URL}/jnlpJars/jenkins-cli.jar

Swap out your Jenkins instance's route or public IP for ${JENKINS_PUBLIC_URL} and you're in business.

Once I have the CLI, I run it against the local Jenkins and just supply help to see if it has a help page:

→  java -jar jenkins-cli.jar -s http://127.0.0.1:8080/ help

ERROR: You must authenticate to access this Jenkins.
Jenkins CLI
Usage: java -jar jenkins-cli.jar [-s URL] command [opts...] args...
Options:
...

Ah, ok. I need to auth. This Jenkins doesn't have Github Oauth enabled, so I'm still using a username and password. This actually makes my CLI life easy, so I'm going to leave that alone and just run it with my username and password like so:

→  java -jar jenkins-cli.jar --username quintessence --password █████████████ -s http://127.0.0.1:8080/ help
Neither -s nor the JENKINS_URL env var is specified.
Jenkins CLI
Usage: java -jar jenkins-cli.jar [-s URL] command [opts...] args...
Options:
-s URL       : the server URL (defaults to the JENKINS_URL env var)
{{{ SNIP }}}

→  export JENKINS_URL=http://127.0.0.1:8080/

→  java -jar jenkins-cli.jar help --username quintessence --password █████████████
  add-job-to-view
    Adds jobs to view.
  build
    Builds a job, and optionally waits until its completion.
  cancel-quiet-down
    Cancel the effect of the "quiet-down" command.
  clear-queue
  {{{ SNIP }}}

Now to test with cloudbees-folder:

→  java -jar jenkins-cli.jar install-plugin cloudbees-folder --username quintessence --password █████████████
Installing cloudbees-folder from update center

→  sudo service jenkins restart
Shutting down Jenkins                                      [  OK  ]
Starting Jenkins                                           [  OK  ]

→  sudo ls /var/lib/jenkins/plugins/cloud*
/var/lib/jenkins/plugins/cloudbees-folder.jpi

/var/lib/jenkins/plugins/cloudbees-folder:
images  META-INF  WEB-INF

→  sudo cat /var/lib/jenkins/plugins/cloudbees-folder/META-INF/MANIFEST.MF
Manifest-Version: 1.0
Archiver-Version: Plexus Archiver
Created-By: Apache Maven
Built-By: jglick
Build-Jdk: 1.8.0_151
Extension-Name: cloudbees-folder
Specification-Title: This plugin allows users to create "folders" to o
 rganize jobs. Users can define custom taxonomies (like
     by project type, organization type etc). Folders are nestable and
  you can define views within folders. Maintained by CloudBees, Inc.
Implementation-Title: cloudbees-folder
Implementation-Version: 6.3
Group-Id: org.jenkins-ci.plugins
Short-Name: cloudbees-folder
Long-Name: Folders Plugin
Url: https://wiki.jenkins.io/display/JENKINS/CloudBees+Folders+Plugin
Compatible-Since-Version: 5.2
Plugin-Version: 6.3
Hudson-Version: 2.60.3
Jenkins-Version: 2.60.3
Plugin-Dependencies: credentials:2.1.11;resolution:=optional
Plugin-Developers:

You can click the image to view full size, note that the path is now to the plugin doc page.

Here we learn a few things:

• Plugin is installed
• "cloudbees-folder" is what is called the "short name"
• "Folders Plugin" is what's called the "long name", which is what the search feature uses. Not helpful if you only know this name after you find your plugin though.

It works! Hooray!

Now for for the rest.

First get the complete list of plugins from the /var/lib/jenkins/plugins directory of the defunct Jenkins instance.

→  cd /defunct-jenkins/var/lib/jenkins/plugins
→  ls -d */
ace-editor/                          blueocean-github-pipeline/          cloudbees-folder/         git-changelog/               jenkins-multijob-plugin/      parameterized-trigger/             resource-disposer/  warnings/
amazon-ecr/                          blueocean-git-pipeline/             cobertura/                git-client/                  jira/                         performance/                       run-condition/      windows-slaves/
analysis-core/                       blueocean-i18n/                     codedeploy/               github/                      jira-ext/                     phabricator-plugin/                runscope/           workflow-aggregator/
ansicolor/                           blueocean-jira/                     command-launcher/         github-api/                  jquery-detached/              pipeline-build-step/               saferestart/        workflow-api/
ant/                                 blueocean-jwt/                      conditional-buildstep/    github-branch-source/        jsch/                         pipeline-github-lib/               sauce-ondemand/     workflow-basic-steps/
antisamy-markup-formatter/           blueocean-personalization/          credentials/              github-oauth/                junit/                        pipeline-graph-analysis/           schedule-build/     workflow-cps/
apache-httpcomponents-client-4-api/  blueocean-pipeline-api-impl/        credentials-binding/      github-organization-folder/  ldap/                         pipeline-input-step/               scm-api/            workflow-cps-global-lib/
authentication-tokens/               blueocean-pipeline-editor/          cvs/                      github-pr-comment-build/     liquibase-runner/             pipeline-milestone-step/           script-security/    workflow-durable-task-step/
aws-credentials/                     blueocean-pipeline-scm-api/         disk-usage/               github-pullrequest/          mailer/                       pipeline-model-api/                slack/              workflow-job/
aws-java-sdk/                        blueocean-rest/                     display-url-api/          git-server/                  mapdb-api/                    pipeline-model-declarative-agent/  sse-gateway/        workflow-multibranch/
BlazeMeterJenkinsPlugin/             blueocean-rest-impl/                docker-commons/           git-userContent/             matrix-auth/                  pipeline-model-definition/         ssh/                workflow-scm-step/
blueocean/                           blueocean-web/                      docker-workflow/          greenballs/                  matrix-project/               pipeline-model-extensions/         ssh-agent/          workflow-step-api/
blueocean-autofavorite/              bouncycastle-api/                   durable-task/             handlebars/                  maven-plugin/                 pipeline-rest-api/                 ssh-credentials/    workflow-support/
blueocean-bitbucket-pipeline/        branch-api/                         envinject/                handy-uri-templates-2-api/   memegen/                      pipeline-stage-step/               ssh-slaves/         ws-cleanup/
blueocean-commons/                   build-environment/                  envinject-api/            htmlpublisher/               mercurial/                    pipeline-stage-tags-metadata/      structs/
blueocean-config/                    build-monitor-plugin/               external-monitor-job/     icon-shim/                   metrics/                      pipeline-stage-view/               subversion/
blueocean-core-js/                   build-timeout/                      favorite/                 jackson2-api/                momentjs/                     plain-credentials/                 token-macro/
blueocean-dashboard/                 built-on-column/                    feature-branch-notifier/  jacoco/                      multi-branch-project-plugin/  port-allocator/                    translation/
blueocean-display-url/               chucknorris/                        ghprb/                    javadoc/                     multiple-scms/                postbuild-task/                    variant/
blueocean-events/                    cloudbees-bitbucket-branch-source/  git/                      jenkins-design-language/     pam-auth/                     pubsub-light/                      violations/

Don't forget to scroll sideways...

At this point you may also realize "Oh great, all I need to do is run this":

java -jar jenkins-cli.jar install-plugin ${PLUGIN_NAME} --username quintessence --password █████████████

....for all of these?

Source: IconArchive

Itty Bitty Shell Script Saves the Day

Well, as they say: play to your strengths.

I don't know about you, but one of my strengths is using shell scripts of any size to save my sanity. Even better in this case as minimal typing required and, true to magical fashion, only a little vim wizardry is required.

To get started, copy all of the plugins on the above list into a new file in vim and run three commands:

:%s/\//\r/g
:%s/^\s\+//e
:g/^$/d

These commands will:

• Change all of the trailing / characters to new lines (\r is carriage return)
• Replace all the whitespaces (\s) at the beginning of each line (^) with nothing, eliminating them.
• Replace all the lines that only have a start and end of line with nothing, eliminating them.

If you used my plugin list to practice your vim fu you, like I at this point, realize that I have 154 plugins to install. 153, if you don't count the one that's already there.

Keep that file open and run these three commands:

:%s/^/"/
:%s/$/" /
:%s/\n//

These commands will:

• Add a " to the beginning of each line
• Add a " to the end of each line (don't neglect the space here!)
  • But if you do, run :%s/$/ /, which will add a space to the end of each line
• This last one deletes the newline character, so you get a nice handy blob

You can use that blob to make an array, and then loop through that array like follows:

#!/bin/bash

PLUGIN_LIST=( "ace-editor" "blueocean-github-pipeline" "cloudbees-folder" "git-changelog" "jenkins-multijob-plugin" "parameterized-trigger" "resource-disposer" "warnings" "amazon-ecr" "blueocean-git-pipeline" "cobertura" "git-client" "jira" "performance" "run-condition" "windows-slaves" "analysis-core" "blueocean-i18n" "codedeploy" "github" "jira-ext" "phabricator-plugin" "runscope" "workflow-aggregator" "ansicolor" "blueocean-jira" "command-launcher" "github-api" "jquery-detached" "pipeline-build-step" "saferestart" "workflow-api" "ant" "blueocean-jwt" "conditional-buildstep" "github-branch-source" "jsch" "pipeline-github-lib" "sauce-ondemand" "workflow-basic-steps" "antisamy-markup-formatter" "blueocean-personalization" "credentials" "github-oauth" "junit" "pipeline-graph-analysis" "schedule-build" "workflow-cps" "apache-httpcomponents-client-4-api" "blueocean-pipeline-api-impl" "credentials-binding" "github-organization-folder" "ldap" "pipeline-input-step" "scm-api" "workflow-cps-global-lib" "authentication-tokens" "blueocean-pipeline-editor" "cvs" "github-pr-comment-build" "liquibase-runner" "pipeline-milestone-step" "script-security" "workflow-durable-task-step" "aws-credentials" "blueocean-pipeline-scm-api" "disk-usage" "github-pullrequest" "mailer" "pipeline-model-api" "slack" "workflow-job" "aws-java-sdk" "blueocean-rest" "display-url-api" "git-server" "mapdb-api" "pipeline-model-declarative-agent" "sse-gateway" "workflow-multibranch" "BlazeMeterJenkinsPlugin" "blueocean-rest-impl" "docker-commons" "git-userContent" "matrix-auth" "pipeline-model-definition" "ssh" "workflow-scm-step" "blueocean" "blueocean-web" "docker-workflow" "greenballs" "matrix-project" "pipeline-model-extensions" "ssh-agent" "workflow-step-api" "blueocean-autofavorite" "bouncycastle-api" "durable-task" "handlebars" "maven-plugin" "pipeline-rest-api" "ssh-credentials" "workflow-support" "blueocean-bitbucket-pipeline" "branch-api" "envinject" "handy-uri-templates-2-api" "memegen" "pipeline-stage-step" "ssh-slaves" "ws-cleanup" "blueocean-commons" "build-environment" "envinject-api" "htmlpublisher" "mercurial" "pipeline-stage-tags-metadata" "structs" "blueocean-config" "build-monitor-plugin" "external-monitor-job" "icon-shim" "metrics" "pipeline-stage-view" "subversion" "blueocean-core-js" "build-timeout" "favorite" "jackson2-api" "momentjs" "plain-credentials" "token-macro" "blueocean-dashboard" "built-on-column" "feature-branch-notifier" "jacoco" "multi-branch-project-plugin" "port-allocator" "translation" "blueocean-display-url" "chucknorris" "ghprb" "javadoc" "multiple-scms" "postbuild-task" "variant" "blueocean-events" "cloudbees-bitbucket-branch-source" "git" "jenkins-design-language" "pam-auth" "pubsub-light" "violations" )

for PLUGIN in "${PLUGIN_LIST[@]}"
do
#    echo "Plugin name: ${PLUGIN}"
    java -jar jenkins-cli.jar install-plugin ${PLUGIN} --username quintessence --password █████████████
done

exit 0

Note that I only needed to add a few lines around the big blob of text, most of our savings here are vim manipulations to change a directory list to a useful blob. The echo line is for you to test printing out the plugin names if you would like - just comment out the jenkins-cli line.

Now for the mass plugin install. Fingers crossed.

→  chmod +x plugin-install.sh

→  ./plugin-install.sh
Installing ace-editor from update center
Installing blueocean-github-pipeline from update center
Installing cloudbees-folder from update center
Installing git-changelog from update center
Installing jenkins-multijob-plugin from update center
Installing parameterized-trigger from update center
Installing resource-disposer from update center
Installing warnings from update center
Installing amazon-ecr from update center
Installing blueocean-git-pipeline from update center
Installing cobertura from update center
...

Source: IconArchive

The Moment of Truth

As you recall me mentioning more than once, rsyncing these directories was a time consuming affair. On the order of hours. But I've had a moment of inspiration: what if I just put the jobs and workspaces directories in to the working Jenkins instead? Since copying the plugins directory into borked Jenkins didn't de-bork it.

To test this a little faster and saner, since I have the jenkins-defunct volume attached and mounted to the new Jenkins, I decided to test this by creating symlinks to the defunct Jenkins' jobs and workspaces directories.

Important note: This is not production ready, please do not do this in production. This is a drill.

Now to continue: I'm going to both backup the empty jobs directory of the bare Jenkins install as well as its whole HOME directory so, if all else fails, I can swiftly get the bare install back. Then I'm going to make the symlinks.

→  sudo mkdir JENKINS_BARE_v2.104_BKP_WITH_PLUGINS

→  sudo cp -r /var/lib/jenkins JENKINS_BARE_v2.104_BKP_WITH_PLUGINS/

→  sudo service jenkins stop
Shutting down Jenkins                                      [  OK  ]

→  sudo mv /var/lib/jenkins/jobs{,--bkp}

→  sudo ln -s /jenkins-ci-old/var/lib/jenkins/jobs /var/lib/jenkins/jobs

→  sudo ls -lh /var/lib/jenkins/job*
lrwxrwxrwx 1 root    root      36 Feb  2 19:21 /var/lib/jenkins/jobs -> /jenkins-defunct/var/lib/jenkins/jobs

/var/lib/jenkins/jobs--bkp:
total 0

→  sudo ln -s /jenkins-defunct/var/lib/jenkins/workspace /var/lib/jenkins/workspace

→  sudo ls -lh /var/lib/jenkins/work*
lrwxrwxrwx 1 root    root      41 Feb  2 19:22 /var/lib/jenkins/workspace -> /jenkins-defunct/var/lib/jenkins/workspace

Note: there was no existing workspaces directory as that involves a plugin that we use / that was just installed.

Now.

to.

Restart.

Jenkins.

Source: IconArchive

MOMENT OF TRUTH

JOBS JOBS JOBS

Source: IconArchive

As a bit of a throwback: those two hanging jobs are what, if you still remember the top of this post, inspired the rollback and caused all this drama.

What to do: Resolution

Of course since this works that means that I need to unlink the symlinks and rsync the actual data where it belongs. This is a short section by word count, but it took it's 2-3 hours to do. To unlink:

→  sudo unlink /var/lib/jenkins/jobs

→  sudo unlink /var/lib/jenkins/workspace

And now for the rsync. I didn't mention it directly before, but the reason I was able to get up and walk away, open other sessions with ease, etc. is because I was using tmux sessions. You may have noticed that was one of the "packages I like to install" above. This is why.

→  tmux new -s rsync

Here's a tmux cheatsheet If you're new to tmux. If you're using the env I cloned above, there is a tmux configuration in there and to create new windows you'll use control+a+c. If you're using the default / not that env, then I believe the default is control+b+c.

In one window:

sudo rsync -a /jenkins-defunct/var/lib/jenkins/jobs /var/lib/jenkins/jobs

And in the other:

sudo rsync -a /jenkins-defunct/var/lib/jenkins/workspace /var/lib/jenkins/workspace

And then you wait.

As a quick tip: I mentioned above that I had initially made this instance with a GP2 type SSD in AWS. In hindsight, it would have been nice to have had IO1 and then it would have made more sense to up the instance type to something beefier, at least just for the transfer, so it'd have been less slow. There's no way to change a volume from GP2 to IO1, though, so I would have needed to snapshot and recreate with a new instance. Alas.

I can verify that after the rsync completed that Jenkins booted up successfully. Let's take another look at that sweet, sweet image.

JOBS JOBS JOBS

Source: IconArchive

Github Oauth Note

When I was flipping the routes around to swap the new Jenkins to production, I noticed it kept trying to preserve the old route. There are a couple of ways to add the Jenkins route. One is in the UI, if it's working. To do that go to Manage Jenkins -> Configure System and scroll to the Jenkins location section:

The other place to do it is by editing the /var/lib/jenkins/jenkins.model.JenkinsLocationConfiguration.xml.

I verified that the URL was correctly set in both of these places; however, Jenkins kept bouncing back to the jenkins-dev.example.com route I had made for it and it also popped a notification that I had a broken reverse proxy. So what gives?

Apparently the culprit was Github Oauth. When you configure the Oauth app in Github it looks like this:

The fields indicated with arrows were using the old route, so Jenkins kept redirecting due to auth.

Post Mortem: Adding Resiliency and What Not

So one of the reasons I ended up in this mess is the lack of backups for a single point of Jenkins shaped failure. There were also some undocumented dependencies and a few other pain points that were uncovered as we did the first round of jobs. There are actually enough points here that I'm splitting this portion into it's own post, which will be released shortly.

Documented on my frequently used assets page.

Sources for header: Jenkins logo and Jenkins art: Fire from Jenkins site, Health Potion by adorabless @ DeviantArt, and a curved arrow from FreePik. Fiery background is from Shutterstock user Bernatskaya Oxana.

Over the weekend we consolidated some of our databases onto a single RDS instance, as a dual effort to reduce costs as well as to allow for cross database joins.

Our primary RDS instance has a read only replica, so imagine my surprise when the next morning I see this:

primadmin@mysql_prod > show databases;
+--------------------+
| Database           |
+--------------------+
| information_schema |
| prod_db_1          |
| prod_db_2          |
| innodb             |
| mysql              |
| performance_schema |
| sys                |
| tmp                |
+--------------------+
8 rows in set (0.01 sec)

---

readadmin@mysql_prod-ro ]> show databases;
+--------------------+
| Database           |
+--------------------+
| information_schema |
| prod_db_1          |
+--------------------+
2 rows in set (0.00 sec)

Looks like something, somewhere is missing from how we added prod_db_2 to mysql_prod last night. I took a look at the read only replica properties to verify its parameter group and noticed this:

Hrm 🤔

Since it's the middle of the day, I needed to schedule a reboot rather than just rebooting the instance. So we're going to cycle back around to this later.

While I await my reboot window, I decide to rule out two other possibilities and see if there are any "suprises" for how AWS implements MySQL v5.7.19. Specifically I want to know if the read only replica will pick up a second database in both of the following scenarios, as opposed to only grabbing the primary database:

1. Create the read only replica before creating a second database
2. Create the read only replica after creating a second database

As a quick note about the instance creation process itself: I noticed I couldn't create a new instance with the same version of MySQL that our production instance is running:

Note the lack of instance types available.

And here are all the instance types.

So to even start my test, I needed to create an instance with 5.7.16, then upgrade to 5.7.19. Lovely.

Skipping past that part of today's mini-🔥, after spinning up the test instance I went through the above test cases. For brevity, the steps for test case #2 are shown below:

primary mysql> show databases;
+--------------------+
| Database           |
+--------------------+
| information_schema |
| innodb             |
| mysql              |
| performance_schema |
| sys                |
| testingonetwothree |
+--------------------+
6 rows in set (0.00 sec)

primary mysql> create database iambluedabudee;
Query OK, 1 row affected (0.01 sec)

primary mysql> show databases;
+--------------------+
| Database           |
+--------------------+
| information_schema |
| iambluedabudee     |
| innodb             |
| mysql              |
| performance_schema |
| sys                |
| testingonetwothree |
+--------------------+
7 rows in set (0.00 sec)

Moment of truth: did the read only replica pick up the second database?

replica mysql> show databases;
+--------------------+
| Database           |
+--------------------+
| information_schema |
| iambluedabudee     |
| innodb             |
| mysql              |
| performance_schema |
| sys                |
| testingonetwothree |
+--------------------+
7 rows in set (0.00 sec)

Brilliant 😄 It turns out that in both cases (#1 not being documented above) that the replica picked up the second database.

Upon the arrival of the reboot window I was able to reboot the production replica without an issue and found that both databases were now in the replica:

readadmin@mysql_prod-ro ]> show databases;
+--------------------+
| Database           |
+--------------------+
| information_schema |
| prod_db_1          |
| prod_db_2          |
+--------------------+
3 rows in set (0.00 sec)

Something else that occurred to me to check, that I neglected to run before the reboot, is checking the version in console like so:

readadmin@mysql_prod-ro ]> show variables like '%version%';
+-------------------------+------------------------------+
| Variable_name           | Value                        |
+-------------------------+------------------------------+
| innodb_version          | 5.7.19                       |
| protocol_version        | 10                           |
| slave_type_conversions  |                              |
| tls_version             | TLSv1,TLSv1.1                |
| version                 | 5.7.19-log                   |
| version_comment         | MySQL Community Server (GPL) |
| version_compile_machine | x86_64                       |
| version_compile_os      | Linux                        |
+-------------------------+------------------------------+
8 rows in set (0.00 sec)

It's entirely probable, nay likely, that the version being reported in the AWS console (below) was incorrect:

Pinocchio versioning.

The actual version was probably a prior minor release, the upgrade being applied only on reboot.

Documented on my frequently used assets page.

Source for header: Cloud database image from Iconfinder, firey background from burnt embers created by Shutterstock user Bernatskaya Oxana.

Happy Santa Lucia Day to those who celebrate!

Source: Table wreath from an eBay blog post, here.

Desserts Can Save the Day

Although Santa Lucia Day is typically celebrated with lussekatter, i.e. rolls of sweet deliciousness, what is going to help us today is actually pie.

That's right, pie.

I recently ran into multiple issues where I had invisible characters in a CSV file. Notably, issues with carriage returns and a feff at the beginning of each line, which is a zero break no space. You may recall from that post I mentioned that there was always more than one error - and here we are.

In a subsequent CSV file, I noticed even. more. invisicharacters. Visually, it looked something like this:

12345  ,"some text"

And I thought: oh, some white spaces. Instead of going right for the kill, my recent invisiperience taught me caution. I moved the cursor over the character and hit x.

Source: X Marks the Spot map, here.

Except it doesn't.

This one was a little harder to troubleshoot. I use my friend's vimrc configuration, which means that I could use ctrl-H to view the hexidecimal characters. Turns out, they weren't white spaces, in fact:

00000000: 31 32 33 34 35 c2 a0 2c 22 73 6f 6d 65 20 74 65  12345..,"some te$

c2 a0 in UTF-8 translates to 00A0, which is a non-breaking space.

!@#$ invisicharacters.

Of course right now it isn't UTF-8, which is why the hex is c2 a0, which means that using perl -CSD in my replacement doesn't work like it did for feff, which was UTF-8.

Why doesn't it work?

perl -CSD is shorthand for -CIOEio, which breaks down as:

C          Command Switch
I          stdin is UTF-8
O          stdout is UTF-8
E          stderr is UTF-8
i          Perl input stream is UTF-8
o          Perl output stream is UTF-8

You can read more about this on the Perldoc.

So, if CSD can't help in this case, what will?

-pie:

p          Loops throgh args similar to sed
i          Edit in place
e          Perl command line expression

To fix this specifically:

perl -pie `s/\x{c2}\x{a0}//` $FILE

This tells perl to replace the first (and only, in this case) instance of c2 a0 with nothing, thus removing it from the line, for the file $FILE.

Header source: Pie vector from VectEezy and ASCII conversion from picascii.

Well Holy Hannah, I received my first Twitter DM over a blog post I've written. I'm officially a pro now 🧐

The DM was from a user that was having some issues with the instructions I provided on a post I wrote in 2015 for S&W here about how to run the then-current release of macOS inside a Vagrant box.

So I thought, what the heck that blog post is 2 yrs old and in computer time that's pratically middle aged. And Trump's been president for essentially a full year now and let's not even talk about what that's done to the flow of time.

Source: Space.com article, here. Article credits NASA/JPL-CalTech.

tldr - it's worth revisiting, Me Said To Me

Laptop Specs

• 15" 2015 Macbook Pro
  • 16 GB of RAM
  • 2.8 GHz CPU
  • 1 TB SSD
• macOS High Sierra, 10.13.2

Downloads

• macOS High Sierra from the App Store
  • Image is approximately 5.2 GB
• latest Vagrant from Hashicorp
  • As of this post the latest release is 2.0.1
  • Image is approximately 70 MB
• latest release of VirtualBox from Oracle
  • As of this post the latest release is 5.2.2
  • Image is approximately 93 MB
• Homebrew - a package installer for macOS. Not required but recommended / encouraged.
• iTerm - an alternative to the Terminal app included in macOS. Again, not required but recommended. For the rest of this post "terminal" can be used interchangeably for iTerm or Terminal, whichever you chose.

Protip: you can install Vagrant and VirtualBox using Homebrew:

• Open terminal
• Install Homebrew (brew) using the command provided on the link above
• Install Vagrant
  brew cask install vagrant
• Install VirtualBox
  brew cask install virtualbox

The main benefit of using a package installer like this is that your packages, like Vagrant and VirtualBox, can now be kept updated with brew update rather than manually downloading / updating package installers from various websites.

For the VirtualBox install you will be prompted to open System Preferences → Security & Privacy at some point. If you don't do this "fast enough" (by the system's pre-programmed determination) then the install may error out. Fear not! It caches the install like so:

The incomplete download is cached at /Users/quintessence/Library/Caches/Homebrew/Cask/virtualbox--5.2.2-119230.dmg.incomplete

So if the install halts for this reason, or because Connection reset by peer (typically that means slow internet connection), or for any other reason, then you can just hit the up arrow to re-run the last command and hit Enter. Just make sure you fix whatever it asks you to first 😉

What this will essentially look like in terminal:

→  brew cask install vagrant
==> Satisfying dependencies
==> Downloading https://releases.hashicorp.com/vagrant/2.0.1/vagrant_2.0.1_x86_64.dmg
######################################################################## 100.0%
==> Verifying checksum for Cask vagrant
==> Installing Cask vagrant
==> Running installer for vagrant; your password may be necessary.
==> Package installers may write to any location; options such as --appdir are ignored.
Password: ██████████████████████████
==> installer: Package name is Vagrant
==> installer: Installing at base path /
==> installer: The install was successful.
🍺  vagrant was successfully installed!

→  brew cask install virtualbox
==> Tapping caskroom/cask
Cloning into '/usr/local/Homebrew/Library/Taps/caskroom/homebrew-cask'...
remote: Counting objects: 3962, done.
remote: Compressing objects: 100% (3928/3928), done.
remote: Total 3962 (delta 38), reused 921 (delta 30), pack-reused 0
Receiving objects: 100% (3962/3962), 1.35 MiB | 9.10 MiB/s, done.
Resolving deltas: 100% (38/38), done.
Tapped 0 formulae (3,971 files, 4.2MB)
==> Creating Caskroom at /usr/local/Caskroom
==> We'll set permissions properly so we won't need sudo in the future
Password:
==> Caveats
To install and/or use virtualbox you may need to enable their kernel extension in

  System Preferences → Security & Privacy → General

For more information refer to vendor documentation or the Apple Technical Note:

  https://developer.apple.com/library/content/technotes/tn2459/_index.html

==> Satisfying dependencies
==> Downloading http://download.virtualbox.org/virtualbox/5.2.2/VirtualBox-5.2.2-119230-OSX.dmg
######################################################################## 100.0%
==> Verifying checksum for Cask virtualbox
==> Installing Cask virtualbox
==> Running installer for virtualbox; your password may be necessary.
==> Package installers may write to any location; options such as --appdir are ignored.
Password: ██████████████████████████
==> installer: Package name is Oracle VM VirtualBox
==> installer: Installing at base path /
==> installer: The install failed (The Installer encountered an error that caused the installation to fail. Contact the software manufacturer for assistance.)
==> Purging files for version 5.2.2-119230 of Cask virtualbox
==> installer: Package name is Oracle VM VirtualBox
==> installer: Installing at base path /
==> installer: The install was successful.
🍺  virtualbox was successfully installed!

Documented on my frequently used assets page.

The Gritty Bits

Ok, so now that we have All the Things we need to do ... quite a few steps, actually.

Creating your ISO File

First, when you download High Sierra, or any other macOS release to date, you are downloading the file as a DMG, which is a proprietary Apple disk image format. In order to run macOS in VirtualBox, we'll need to convert the DMG file to a more general format such as an ISO. To do this, we're going to start by converting the installer DMG that we downloaded from the App Store into an image of the full OS, store that in a DMG, and then use a utility called PowerISO to do the final conversion.

If that just read like a paragraph of panic, don't worry! We're going through it step by step.

hdiutil attach /Applications/Install\ macOS\ High\ Sierra.app/Contents/SharedSupport/InstallESD.dmg -noverify -mountpoint /Volumes/HighSierraInstall

hdiutil create -o /tmp/HighSierra.cdr -size 5130m -layout SPUD -fs HFS+J
created: /tmp/HighSierra.cdr.dmg

hdiutil attach /tmp/HighSierra.cdr.dmg -noverify -mountpoint /Volumes/InstallBuild

sudo /Applications/Install\ macOS\ High\ Sierra.app/Contents/Resources/createinstallmedia --volume /Volumes/InstallBuild

mv /tmp/HighSierra.cdr.dmg ~/Desktop/HighSierraToMakeISO.dmg

The output will look like this:

→  hdiutil attach /Applications/Install\ macOS\ High\ Sierra.app/Contents/SharedSupport/InstallESD.dmg -noverify -mountpoint /Volumes/HighSierraInstall
/dev/disk2              GUID_partition_scheme
/dev/disk2s1            EFI
/dev/disk2s2            Apple_HFS                       /Volumes/HighSierraInstall

→  hdiutil create -o /tmp/HighSierra.cdr -size 5130m -layout SPUD -fs HFS+J
created: /tmp/HighSierra.cdr.dmg

→  hdiutil attach /tmp/HighSierra.cdr.dmg -noverify -mountpoint /Volumes/InstallBuild
/dev/disk3              Apple_partition_scheme
/dev/disk3s1            Apple_partition_map
/dev/disk3s2            Apple_HFS                       /Volumes/InstallBuild

→  sudo /Applications/Install\ macOS\ High\ Sierra.app/Contents/Resources/createinstallmedia --volume /Volumes/InstallBuild
Password:
Ready to start.
To continue we need to erase the volume at /Volumes/InstallBuild.
If you wish to continue type (Y) then press return: Y
Erasing Disk: 0%... 10%... 20%... 30%...100%...
Copying installer files to disk...
Copy complete.
Making disk bootable...
Copying boot files...
Failed to copy kernelcache, “prelinkedkernel” couldn’t be copied to “.IABootFiles”.
Done.

→  mv /tmp/HighSierra.cdr.dmg ~/Desktop/HighSierraToMakeISO.dmg

Troubleshooting

What to do if you see the following error:

→  sudo hdiutil attach /Applications/Install\ macOS\ High\ Sierra.app/Contents/SharedSupport/InstallESD.dmg -noverify -mountpoint /Volumes/HighSierraInstall
Password:
hdiutil: attach failed - Resource busy

This most likely means that you've already mounted the InstallESD image. Go into Disk Utility and unmount it:

This likely happened when the High Sierra download completed and immediately opened the installer.

Using your VHD file with VirualBox

Open VirtualBox:

Click New:

Choose a name, Type is "Mac OS X", and Version is "macOS 10.13 High Sierra (64-bit)". Then click "Continue":

The default memory is 2 GB, I recommend upping it to 4 GB though if you can:

Choose "Create a Virtual Hard Disk File now" and "Create":

For the curious
The image manipulations above, with arrow / blurring of text / etc., were done with the Skitch app. You only need an account to save in Evernote - account-less you can just locally save / export to whatever image file format you prefer.

Quickest Vagrant Tutoral Ever

Add the Vagrant box you want to use. We'll use Ubuntu 12.04 for the following example.

$ vagrant box add precise64 http://files.vagrantup.com/precise64.box
You can find more boxes at Vagrant Cloud

Now create a test directory and cd into the test directory. Then we'll initialize the vagrant machine.

$ vagrant init precise64
Now lets start the machine using the following command.

$ vagrant up
You can ssh into the machine now.

$ vagrant ssh
Halt the vagrant machine now.

$ vagrant halt
Other useful commands are suspend, destroy etc.

So I decided to upgrade my Pi3 this morning, a task that I thought would be be pretty straightforward. It's worth noting that I haven't upgraded my Pi since I did the initial package installs when I set up the OS and pihole, so ... it was more than due for at least a cursory upgrade.

I logged into my Pi:

self@GreenScreen:~$  ssh pi
Welcome to Ubuntu 16.04.2 LTS (GNU/Linux 4.4.0-1009-raspi2 armv7l)

 * Documentation:  https://help.ubuntu.com
 * Management:     https://landscape.canonical.com
 * Support:        https://ubuntu.com/advantage

  Get cloud support with Ubuntu Advantage Cloud Guest:
    http://www.ubuntu.com/business/services/cloud

106 packages can be updated.
1 update is a security update.


*** System restart required ***
Last login: Mon Mar  6 01:11:08 2017 from 192.168.█.█
ubuntu@ubuntu:~$ sudo apt-get update && sudo apt-get upgrade
Hit:1 http://ppa.launchpad.net/ubuntu-raspi2/ppa-rpi3/ubuntu xenial InRelease
Hit:2 http://ports.ubuntu.com/ubuntu-ports xenial InRelease
Hit:3 http://ports.ubuntu.com/ubuntu-ports xenial-updates InRelease
Hit:4 http://ports.ubuntu.com/ubuntu-ports xenial-backports InRelease
Hit:5 http://ports.ubuntu.com/ubuntu-ports xenial-security InRelease
E: Could not get lock /var/lib/dpkg/lock - open (11: Resource temporarily unavailable)
E: Unable to lock the administration directory (/var/lib/dpkg/), is another process using it?
ubuntu@ubuntu:~$ sudo reboot now

Since it was brunch time, I got up and grabbed my peanut butter latte, and sat back down. More than enough time for a reboot.

Or so you would think.

self@GreenScreen:~$  ssh pi -v
OpenSSH_7.5p1, LibreSSL 2.5.4
debug1: Reading configuration data /Users/quintessence/.ssh/config
debug1: /Users/quintessence/.ssh/config line 41: Applying options for pi
debug1: /Users/quintessence/.ssh/config line 138: Applying options for *
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 52: Applying options for *
debug1: Connecting to 192.168.█.█ [192.168.█.█] port 22.
^C

self@GreenScreen:~$  ssh ubuntu@192.168.█.█ -v
OpenSSH_7.5p1, LibreSSL 2.5.4
debug1: Reading configuration data /Users/quintessence/.ssh/config
debug1: /Users/quintessence/.ssh/config line 138: Applying options for *
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 52: Applying options for *
debug1: Connecting to 192.168.█.█ [192.168.█.█] port 22.
debug1: connect to address 192.168.█.█ port 22: Operation timed out
ssh: connect to host 192.168.█.█ port 22: Operation timed out

Oh.

Good.

Fishing out an HDMI cable, I connect the Pi to my TV and see that U-Boot is trying to PXE boot like so:

(( snip ))
Waiting for Ethernet connection...done
*** ERROR: `serverip' is not set
missing environment variable: bootfile
Retrieving file: pxelinux.cfg/default-arm
Waiting for Ethernet connection...done
*** ERROR: `serverip' is not set
missing environment variable: bootfile
Retrieving file: pxelinux.cfg/default
Waiting for Ethernet connection...done
*** ERROR: `serverip' is not set
(( snip ))

It looped around like this until I got a prompt, but I wasn't in bash or sh, I was in U-Boot - the bootloader. That became obvious when commands like ls and cd did not exist and the output of the help command looked like this:

?      - alias for 'help'
base    - print or set address offset
bdinfo  - print Board Info structure
boot    - boot default, i.e., run 'bootcmd'
bootd  - boot default, i.e., run 'bootcmd'
bootelf - Boot from an ELF image in memory
bootm  - boot application image from memory
bootp  - boot image via network using BOOTP/TFTP protocol
bootvx  - Boot vxWorks from an ELF image
(( snip ))

Why was I in U-Boot? For some reason the SD card was not being recognized as a boot device and so the Pi tried the next thing it knew: PXE boot. This failed because I don't have a TFTP server config for it to find, so it's trying and failing to find PXE config files that don't exist.

As for how I ended up in this mess: I suspect that I didn't notice the system restart required message at some earlier point and therefore not only did the current upgrade fail, something in the initial batch did as well. The combination, well...

In any event, I'm lucky in that since this Pi was really just for running the pihole software, I don't have to waste too much (more) time trying to dig around in there. So I just wiped it clean and re-wrote a new image on it.

After reading up on Ubuntu a bit, and the fact that the image for the Pi is "unofficial" and has some "upgrade issues" (no, really?) I opted for Raspbian, the official Pi distro.

From the instructions, I decided to download Etcher to write the image to disk, like so:

Since I only have one removable drive connected, it automatically found my SD card which was appreciated. The timer also lets me know that I can have another snack, which with how long this is taking why not...

Awesome. At this point the SD card is no longer mounted as there is a setting in Etcher to auto-unmount on success. After a successful boot I enable ssh:

sudo systemctl enable ssh
sudo systemctl start ssh

I need to remove the 192.168.█.█ line from my ~/.ssh/known_hosts file since on my first attempt to ssh I receive an error about a potential man in the middle attack due to the changed fingerprint. This is expected, though, since there is a new image on the Pi at the same IP address.

Moving along I add my pubkey to ~/.ssh/authorized_keys on the Pi and sudo passwd pi to change the password from the default raspberry:

pi@raspberrypi:~ $ mkdir .ssh
pi@raspberrypi:~ $ vim ~/.ssh/authorized_keys
-bash: vim: command not found
pi@raspberrypi:~ $ vi !$
vi ~/.ssh/authorized_hosts
pi@raspberrypi:~ $ sudo passwd pi
Enter new UNIX password:
Retype new UNIX password:
passwd: password updated successfully

Ran all the package updates, installed the pihole package which basically involved a curl and then navigating through prompts:

pi@raspberrypi:~ $ curl -sSL https://install.pi-hole.net | bash
::: system networking, it requires elevated rights. Please check the contents of the script for
::: any concerns with this requirement. Please be sure to download this script from a trusted source.
:::
::: Detecting the presence of the sudo utility for continuation of this install...
::: Utility sudo located.

        .;;,.
        .ccccc:,.
         :cccclll:.      ..,,
          :ccccclll.   ;ooodc
           'ccll:;ll .oooodc
             .;cll.;;looo:.
                 .. ','.
                .',,,,,,'.
              .',,,,,,,,,,.
            .',,,,,,,,,,,,....
          ....''',,,,,,,'.......
        .........  ....  .........
        ..........      ..........
        ..........      ..........
        .........  ....  .........
          ........,,,,,,,'......
            ....',,,,,,,,,,,,.
               .',,,,,,,,,'.
                .',,,,,,'.
                  ..'''.

:::
::: You are root.
::: Verifying free disk space...
:::
::: Updating local cache of available packages...
(( snip ))

Navigated to a few known ad-heavy sites like Forbes and took a look at my dashboard:

And now I'm off to a safer, more secure browsing experience.

Updates

After 1 hr of use.

After 24 hrs of use.

Source for header: firey background from burnt embers created by Shutterstock user Bernatskaya Oxana and the Raspberry Pi logo.