Opsfire: Downgrading from Purgatory to Limbo
After fixing nginx woes earlier you might be feeling great and decide: let's fix this across the board (in dev only because we're not confident in prod yet 😅).
Uh, oh wait.
There's a monitoring alarm that an instance healthcheck failed?
Oh, good. What's running on port 80 and 443?
$ sudo netstat -tlnp | grep '443\|80'
Oh, good.
First: let's try downgrading:
$ sudo yum downgrade nginx
Loaded plugins: priorities, update-motd, upgrade-helper
amzn-main | 2.1 kB 00:00:00
amzn-updates | 2.5 kB 00:00:00
Nothing to do
🙃
Did you know that you can undo yum history? Yeppers! Let's hop on that ASAP.
$ sudo yum history
Loaded plugins: priorities, update-motd, upgrade-helper
ID | Login user | Date and time | Action(s) | Altered
-------------------------------------------------------------------------------
107 | EC2 ... <ec2-user> | 2017-12-05 19:42 | Install | 1
106 | EC2 ... <ec2-user> | 2017-12-05 19:42 | Erase | 1 EE
105 | EC2 ... <ec2-user> | 2017-12-05 15:36 | Update | 1 EE
104 | EC2 ... <ec2-user> | 2017-11-21 12:47 | E, I, U | 10 EE
103 | EC2 ... <ec2-user> | 2017-11-21 12:47 | Erase | 7
102 | root <root> | 2017-11-08 18:21 | Update | 1 EE
101 | EC2 ... <ec2-user> | 2017-11-07 15:04 | E, I, O, U | 76 EE
100 | System <unset> | 2017-11-06 17:49 | Update | 1 EE
99 | System <unset> | 2017-11-02 18:13 | Update | 1 EE
98 | System <unset> | 2017-10-31 19:57 | Update | 1 EE
97 | System <unset> | 2017-10-13 13:10 | Update | 1 EE
96 | System <unset> | 2017-10-10 18:54 | Update | 1 EE
95 | EC2 ... <ec2-user> | 2017-09-19 15:30 | E, I, U | 27 EE
94 | System <unset> | 2017-09-18 16:40 | Update | 1 EE
93 | System <unset> | 2017-09-12 15:54 | Update | 1 EE
92 | System <unset> | 2017-08-28 13:45 | Update | 1 EE
91 | EC2 ... <ec2-user> | 2017-08-15 22:26 | E, I, U | 47 EE
90 | EC2 ... <ec2-user> | 2017-07-28 11:46 | Erase | 2 EE
89 | System <unset> | 2017-07-26 14:55 | Update | 1 EE
88 | EC2 ... <ec2-user> | 2017-07-18 13:53 | I, U | 183 EE
history list
$ sudo yum history undo 107
Loaded plugins: priorities, update-motd, upgrade-helper
Undoing transaction 107, from Tue Dec 5 19:42:31 2017
Install nginx-1:1.12.1-1.33.amzn1.x86_64 @amzn-main
Resolving Dependencies
--> Running transaction check
---> Package nginx.x86_64 1:1.12.1-1.33.amzn1 will be erased
--> Finished Dependency Resolution
Dependencies Resolved
================================================================================================================================================================================================================================================================================
Package Arch Version Repository Size
================================================================================================================================================================================================================================================================================
Removing:
nginx x86_64 1:1.12.1-1.33.amzn1 @amzn-main 1.4 M
Transaction Summary
================================================================================================================================================================================================================================================================================
Remove 1 Package
Installed size: 1.4 M
Is this ok [y/N]: y
Downloading packages:
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
Erasing : 1:nginx-1.12.1-1.33.amzn1.x86_64 1/1
Verifying : 1:nginx-1.12.1-1.33.amzn1.x86_64 1/1
Removed:
nginx.x86_64 1:1.12.1-1.33.amzn1
Complete!
$ sudo yum history undo 106
Loaded plugins: priorities, update-motd, upgrade-helper
Undoing transaction 106, from Tue Dec 5 19:42:27 2017
Erase nginx-1:1.8.0-10.25.amzn1.x86_64 @amzn-main
Error: No package(s) available to install
Oh right, we're no longer using our internal repo.
$ sudo vim /etc/yum.repos.d/internal.repo
$ sudo yum history undo 106
Loaded plugins: priorities, update-motd, upgrade-helper
Undoing transaction 106, from Tue Dec 5 19:42:27 2017
Erase nginx-1:1.8.0-10.25.amzn1.x86_64 @amzn-main
internal-aws-arched | 2.9 kB 00:00:00
internal-aws-noarch | 2.9 kB 00:00:00
1 packages excluded due to repository priority protections
Error: No package(s) available to install
😐
Since the internal repo is re-enabled, let's just to a regular install:
sudo yum install nginx
Loaded plugins: priorities, update-motd, upgrade-helper
internal-aws-arched | 2.9 kB 00:00:00
internal-aws-noarch | 2.9 kB 00:00:00
1 packages excluded due to repository priority protections
Package nginx is obsoleted by nginx-all-modules, trying to install 1:nginx-all-modules-1.12.1-1.33.amzn1.x86_64 instead
Resolving Dependencies
--> Running transaction check
---> Package nginx-all-modules.x86_64 1:1.12.1-1.33.amzn1 will be installed
--> Processing Dependency: nginx-mod-stream(x86-64) = 1:1.12.1-1.33.amzn1 for package: 1:nginx-all-modules-1.12.1-1.33.amzn1.x86_64
--> Processing Dependency: nginx-mod-mail(x86-64) = 1:1.12.1-1.33.amzn1 for package: 1:nginx-all-modules-1.12.1-1.33.amzn1.x86_64
--> Processing Dependency: nginx-mod-http-xslt-filter(x86-64) = 1:1.12.1-1.33.amzn1 for package: 1:nginx-all-modules-1.12.1-1.33.amzn1.x86_64
--> Processing Dependency: nginx-mod-http-perl(x86-64) = 1:1.12.1-1.33.amzn1 for package: 1:nginx-all-modules-1.12.1-1.33.amzn1.x86_64
--> Processing Dependency: nginx-mod-http-image-filter(x86-64) = 1:1.12.1-1.33.amzn1 for package: 1:nginx-all-modules-1.12.1-1.33.amzn1.x86_64
--> Processing Dependency: nginx-mod-http-geoip(x86-64) = 1:1.12.1-1.33.amzn1 for package: 1:nginx-all-modules-1.12.1-1.33.amzn1.x86_64
--> Running transaction check
{{ snip -> you remember this bit from the earlier fire, right? }}
🤔
Googling for the package, nginx-1.8.0-10.25.amzn1.x86_64, yields a result on DynaTrace. Great!
$ wget http://packages.eu-west-1.amazonaws.com/2015.09/main/201509419456/x86_64/Packages/nginx-1.8.0-10.25.amzn1.x86_64.rpm
--2017-12-05 20:03:23-- http://packages.eu-west-1.amazonaws.com/2015.09/main/201509419456/x86_64/Packages/nginx-1.8.0-10.25.amzn1.x86_64.rpm
Resolving packages.eu-west-1.amazonaws.com (packages.eu-west-1.amazonaws.com)... 52.218.20.193
Connecting to packages.eu-west-1.amazonaws.com (packages.eu-west-1.amazonaws.com)|52.218.20.193|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 568699 (555K) [binary/octet-stream]
Saving to: ‘nginx-1.8.0-10.25.amzn1.x86_64.rpm’
nginx-1.8.0-10.25.amzn1.x86_64.rpm 100%[===================================================================================================================================================================>] 555.37K 491KB/s in 1.1s
2017-12-05 20:03:25 (491 KB/s) - ‘nginx-1.8.0-10.25.amzn1.x86_64.rpm’ saved [568699/568699]
$ sudo rpm -ivh nginx-1.8.0-10.25.amzn1.x86_64.rpm
Preparing... ################################# [100%]
Updating / installing...
1:nginx-1:1.8.0-10.25.amzn1 ################################# [100%]
Now to start the service and see how it's doing:
$ sudo service nginx start
Starting nginx: [ OK ]
$ nginx -v
nginx version: nginx/1.8.0
$ sudo ps aux | grep nginx
root 29442 0.0 0.1 110644 4344 ? Ss 20:05 0:00 nginx: master process /usr/sbin/nginx -c /etc/nginx/nginx.conf
nginx 29443 0.0 0.1 110648 7408 ? S 20:05 0:00 nginx: worker process
nginx 29444 0.0 0.1 110648 5844 ? S 20:05 0:00 nginx: worker process
ec2-user 29484 0.0 0.0 110472 2112 pts/0 S+ 20:05 0:00 grep --color=auto nginx
$ sudo netstat -tlnp | grep '443\|80'
tcp 0 0 0.0.0.0:443 0.0.0.0:* LISTEN 29442/nginx
tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN 29442/nginx
😅
And that heartbeat monitor should be recovering now too.
Mischief managed.
Quick Addendum
You may have noticed that I googled for the package name nginx-1.8.0-10.25.amzn1.x86_64 rather than nginx-1:1.8.0-10.25.amzn1.x86_64, the latter being how the package is identified in the error msgs above. The reason for this is the 1 in 1: is what is called the Epoch. The Epoch allows you to reset a version if you change your versioning scheme. As a quick example, if you wrote a package called memnommer and provided a release as memnommer-0.12345.6 but then provided the subsequent release as memnommer-0.2.1 your package installer would not be able to clearly determine which is the upgrade since 12345 > 2. In this case, you would bump your Epoch number, e.g. memnommer-0:0.12345.6 to memnommer-1:0.2.1. Now when you try to upgrade with yum upgrade memnommer, yum knows what package to install as the epoch takes precedence. Note that since this value is to be used by package manager, it is usually not in the file name of the release itself which is why it wasn't in my search.
Documented on my frequently used assets page.