Opsfire: Downgrading from Purgatory to Limbo

After fixing nginx woes earlier you might be feeling great and decide: let's fix this across the board (in dev only because we're not confident in prod yet 😅).

Uh, oh wait.

There's a monitoring alarm that an instance healthcheck failed?

Oh, good. What's running on port 80 and 443?

$ sudo netstat -tlnp | grep '443\|80'

Oh, good.

First: let's try downgrading:

$ sudo yum downgrade nginx
Loaded plugins: priorities, update-motd, upgrade-helper
amzn-main                                                                                                                                                                                                                                                | 2.1 kB  00:00:00
amzn-updates                                                                                                                                                                                                                                             | 2.5 kB  00:00:00
Nothing to do

🙃

Did you know that you can undo yum history? Yeppers! Let's hop on that ASAP.

$ sudo yum history
Loaded plugins: priorities, update-motd, upgrade-helper
ID     | Login user               | Date and time    | Action(s)      | Altered
-------------------------------------------------------------------------------
   107 | EC2 ... <ec2-user>       | 2017-12-05 19:42 | Install        |    1
   106 | EC2 ... <ec2-user>       | 2017-12-05 19:42 | Erase          |    1 EE
   105 | EC2 ... <ec2-user>       | 2017-12-05 15:36 | Update         |    1 EE
   104 | EC2 ... <ec2-user>       | 2017-11-21 12:47 | E, I, U        |   10 EE
   103 | EC2 ... <ec2-user>       | 2017-11-21 12:47 | Erase          |    7
   102 | root <root>              | 2017-11-08 18:21 | Update         |    1 EE
   101 | EC2 ... <ec2-user>       | 2017-11-07 15:04 | E, I, O, U     |   76 EE
   100 | System <unset>           | 2017-11-06 17:49 | Update         |    1 EE
    99 | System <unset>           | 2017-11-02 18:13 | Update         |    1 EE
    98 | System <unset>           | 2017-10-31 19:57 | Update         |    1 EE
    97 | System <unset>           | 2017-10-13 13:10 | Update         |    1 EE
    96 | System <unset>           | 2017-10-10 18:54 | Update         |    1 EE
    95 | EC2 ... <ec2-user>       | 2017-09-19 15:30 | E, I, U        |   27 EE
    94 | System <unset>           | 2017-09-18 16:40 | Update         |    1 EE
    93 | System <unset>           | 2017-09-12 15:54 | Update         |    1 EE
    92 | System <unset>           | 2017-08-28 13:45 | Update         |    1 EE
    91 | EC2 ... <ec2-user>       | 2017-08-15 22:26 | E, I, U        |   47 EE
    90 | EC2 ... <ec2-user>       | 2017-07-28 11:46 | Erase          |    2 EE
    89 | System <unset>           | 2017-07-26 14:55 | Update         |    1 EE
    88 | EC2 ... <ec2-user>       | 2017-07-18 13:53 | I, U           |  183 EE
history list


$ sudo yum history undo 107
Loaded plugins: priorities, update-motd, upgrade-helper
Undoing transaction 107, from Tue Dec  5 19:42:31 2017
    Install nginx-1:1.12.1-1.33.amzn1.x86_64 @amzn-main
Resolving Dependencies
--> Running transaction check
---> Package nginx.x86_64 1:1.12.1-1.33.amzn1 will be erased
--> Finished Dependency Resolution

Dependencies Resolved

================================================================================================================================================================================================================================================================================
 Package                                                      Arch                                                          Version                                                                     Repository                                                         Size
================================================================================================================================================================================================================================================================================
Removing:
 nginx                                                        x86_64                                                        1:1.12.1-1.33.amzn1                                                         @amzn-main                                                        1.4 M

Transaction Summary
================================================================================================================================================================================================================================================================================
Remove  1 Package

Installed size: 1.4 M
Is this ok [y/N]: y
Downloading packages:
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Erasing    : 1:nginx-1.12.1-1.33.amzn1.x86_64                                                                                                                                                                                                                             1/1
  Verifying  : 1:nginx-1.12.1-1.33.amzn1.x86_64                                                                                                                                                                                                                             1/1

Removed:
  nginx.x86_64 1:1.12.1-1.33.amzn1

Complete!

$ sudo yum history undo 106
Loaded plugins: priorities, update-motd, upgrade-helper
Undoing transaction 106, from Tue Dec  5 19:42:27 2017
    Erase nginx-1:1.8.0-10.25.amzn1.x86_64 @amzn-main
Error: No package(s) available to install

Oh right, we're no longer using our internal repo.

$ sudo vim /etc/yum.repos.d/internal.repo

$ sudo yum history undo 106
Loaded plugins: priorities, update-motd, upgrade-helper
Undoing transaction 106, from Tue Dec  5 19:42:27 2017
    Erase nginx-1:1.8.0-10.25.amzn1.x86_64 @amzn-main
internal-aws-arched                                                                                                                                                                                                                                      | 2.9 kB  00:00:00
internal-aws-noarch                                                                                                                                                                                                                                      | 2.9 kB  00:00:00
1 packages excluded due to repository priority protections
Error: No package(s) available to install

😐

Since the internal repo is re-enabled, let's just to a regular install:

 sudo yum install nginx
Loaded plugins: priorities, update-motd, upgrade-helper
internal-aws-arched                                                                                                                                                                                                                                      | 2.9 kB  00:00:00
internal-aws-noarch                                                                                                                                                                                                                                      | 2.9 kB  00:00:00
1 packages excluded due to repository priority protections
Package nginx is obsoleted by nginx-all-modules, trying to install 1:nginx-all-modules-1.12.1-1.33.amzn1.x86_64 instead
Resolving Dependencies
--> Running transaction check
---> Package nginx-all-modules.x86_64 1:1.12.1-1.33.amzn1 will be installed
--> Processing Dependency: nginx-mod-stream(x86-64) = 1:1.12.1-1.33.amzn1 for package: 1:nginx-all-modules-1.12.1-1.33.amzn1.x86_64
--> Processing Dependency: nginx-mod-mail(x86-64) = 1:1.12.1-1.33.amzn1 for package: 1:nginx-all-modules-1.12.1-1.33.amzn1.x86_64
--> Processing Dependency: nginx-mod-http-xslt-filter(x86-64) = 1:1.12.1-1.33.amzn1 for package: 1:nginx-all-modules-1.12.1-1.33.amzn1.x86_64
--> Processing Dependency: nginx-mod-http-perl(x86-64) = 1:1.12.1-1.33.amzn1 for package: 1:nginx-all-modules-1.12.1-1.33.amzn1.x86_64
--> Processing Dependency: nginx-mod-http-image-filter(x86-64) = 1:1.12.1-1.33.amzn1 for package: 1:nginx-all-modules-1.12.1-1.33.amzn1.x86_64
--> Processing Dependency: nginx-mod-http-geoip(x86-64) = 1:1.12.1-1.33.amzn1 for package: 1:nginx-all-modules-1.12.1-1.33.amzn1.x86_64
--> Running transaction check
{{ snip -> you remember this bit from the earlier fire, right? }}

🤔

Googling for the package, nginx-1.8.0-10.25.amzn1.x86_64, yields a result on DynaTrace. Great!

$ wget http://packages.eu-west-1.amazonaws.com/2015.09/main/201509419456/x86_64/Packages/nginx-1.8.0-10.25.amzn1.x86_64.rpm
--2017-12-05 20:03:23--  http://packages.eu-west-1.amazonaws.com/2015.09/main/201509419456/x86_64/Packages/nginx-1.8.0-10.25.amzn1.x86_64.rpm
Resolving packages.eu-west-1.amazonaws.com (packages.eu-west-1.amazonaws.com)... 52.218.20.193
Connecting to packages.eu-west-1.amazonaws.com (packages.eu-west-1.amazonaws.com)|52.218.20.193|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 568699 (555K) [binary/octet-stream]
Saving to: ‘nginx-1.8.0-10.25.amzn1.x86_64.rpm’

nginx-1.8.0-10.25.amzn1.x86_64.rpm                                  100%[===================================================================================================================================================================>] 555.37K   491KB/s    in 1.1s

2017-12-05 20:03:25 (491 KB/s) - ‘nginx-1.8.0-10.25.amzn1.x86_64.rpm’ saved [568699/568699]

$ sudo rpm -ivh nginx-1.8.0-10.25.amzn1.x86_64.rpm
Preparing...                          ################################# [100%]
Updating / installing...
   1:nginx-1:1.8.0-10.25.amzn1        ################################# [100%]

Now to start the service and see how it's doing:

$ sudo service nginx start
Starting nginx:                                            [  OK  ]

$ nginx -v
nginx version: nginx/1.8.0

$ sudo ps aux | grep nginx
root     29442  0.0  0.1 110644  4344 ?        Ss   20:05   0:00 nginx: master process /usr/sbin/nginx -c /etc/nginx/nginx.conf
nginx    29443  0.0  0.1 110648  7408 ?        S    20:05   0:00 nginx: worker process
nginx    29444  0.0  0.1 110648  5844 ?        S    20:05   0:00 nginx: worker process
ec2-user 29484  0.0  0.0 110472  2112 pts/0    S+   20:05   0:00 grep --color=auto nginx

$ sudo netstat -tlnp | grep '443\|80'
tcp        0      0 0.0.0.0:443                 0.0.0.0:*                   LISTEN      29442/nginx
tcp        0      0 0.0.0.0:80                  0.0.0.0:*                   LISTEN      29442/nginx

😅

And that heartbeat monitor should be recovering now too.

Mischief managed.

Quick Addendum

You may have noticed that I googled for the package name nginx-1.8.0-10.25.amzn1.x86_64 rather than nginx-1:1.8.0-10.25.amzn1.x86_64, the latter being how the package is identified in the error msgs above. The reason for this is the 1 in 1: is what is called the Epoch. The Epoch allows you to reset a version if you change your versioning scheme. As a quick example, if you wrote a package called memnommer and provided a release as memnommer-0.12345.6 but then provided the subsequent release as memnommer-0.2.1 your package installer would not be able to clearly determine which is the upgrade since 12345 > 2. In this case, you would bump your Epoch number, e.g. memnommer-0:0.12345.6 to memnommer-1:0.2.1. Now when you try to upgrade with yum upgrade memnommer, yum knows what package to install as the epoch takes precedence. Note that since this value is to be used by package manager, it is usually not in the file name of the release itself which is why it wasn't in my search.

Documented on my frequently used assets page.