Opsfire: In Dependency Purgatory with Nginx

Recently ran into this issue on an a few instances in an environment I'm working in:

$ sudo yum upgrade
Loaded plugins: priorities, update-motd, upgrade-helper
1 packages excluded due to repository priority protections
Resolving Dependencies
--> Running transaction check
---> Package nginx.x86_64 1:1.8.1-1.26.amzn1 will be obsoleted
---> Package nginx-all-modules.x86_64 1:1.12.1-1.33.amzn1 will be obsoleting
--> Processing Dependency: nginx-mod-stream(x86-64) = 1:1.12.1-1.33.amzn1 for package: 1:nginx-all-modules-1.12.1-1.33.amzn1.x86_64
--> Processing Dependency: nginx-mod-mail(x86-64) = 1:1.12.1-1.33.amzn1 for package: 1:nginx-all-modules-1.12.1-1.33.amzn1.x86_64
--> Processing Dependency: nginx-mod-http-xslt-filter(x86-64) = 1:1.12.1-1.33.amzn1 for package: 1:nginx-all-modules-1.12.1-1.33.amzn1.x86_64
--> Processing Dependency: nginx-mod-http-perl(x86-64) = 1:1.12.1-1.33.amzn1 for package: 1:nginx-all-modules-1.12.1-1.33.amzn1.x86_64
--> Processing Dependency: nginx-mod-http-image-filter(x86-64) = 1:1.12.1-1.33.amzn1 for package: 1:nginx-all-modules-1.12.1-1.33.amzn1.x86_64
--> Processing Dependency: nginx-mod-http-geoip(x86-64) = 1:1.12.1-1.33.amzn1 for package: 1:nginx-all-modules-1.12.1-1.33.amzn1.x86_64
--> Running transaction check
---> Package nginx-mod-http-geoip.x86_64 1:1.12.1-1.33.amzn1 will be installed
--> Processing Dependency: nginx(x86-64) = 1:1.12.1-1.33.amzn1 for package: 1:nginx-mod-http-geoip-1.12.1-1.33.amzn1.x86_64
---> Package nginx-mod-http-image-filter.x86_64 1:1.12.1-1.33.amzn1 will be installed
--> Processing Dependency: nginx(x86-64) = 1:1.12.1-1.33.amzn1 for package: 1:nginx-mod-http-image-filter-1.12.1-1.33.amzn1.x86_64
---> Package nginx-mod-http-perl.x86_64 1:1.12.1-1.33.amzn1 will be installed
--> Processing Dependency: nginx(x86-64) = 1:1.12.1-1.33.amzn1 for package: 1:nginx-mod-http-perl-1.12.1-1.33.amzn1.x86_64
---> Package nginx-mod-http-xslt-filter.x86_64 1:1.12.1-1.33.amzn1 will be installed
--> Processing Dependency: nginx(x86-64) = 1:1.12.1-1.33.amzn1 for package: 1:nginx-mod-http-xslt-filter-1.12.1-1.33.amzn1.x86_64
---> Package nginx-mod-mail.x86_64 1:1.12.1-1.33.amzn1 will be installed
--> Processing Dependency: nginx(x86-64) = 1:1.12.1-1.33.amzn1 for package: 1:nginx-mod-mail-1.12.1-1.33.amzn1.x86_64
---> Package nginx-mod-stream.x86_64 1:1.12.1-1.33.amzn1 will be installed
--> Processing Dependency: nginx(x86-64) = 1:1.12.1-1.33.amzn1 for package: 1:nginx-mod-stream-1.12.1-1.33.amzn1.x86_64
--> Finished Dependency Resolution
Error: Package: 1:nginx-mod-http-geoip-1.12.1-1.33.amzn1.x86_64 (amzn-main)
           Requires: nginx(x86-64) = 1:1.12.1-1.33.amzn1
           Removing: 1:nginx-1.8.1-1.26.amzn1.x86_64 (@amzn-main)
               nginx(x86-64) = 1:1.8.1-1.26.amzn1
           Obsoleted By: 1:nginx-all-modules-1.12.1-1.33.amzn1.x86_64 (amzn-main)
               Not found
           Available: nginx-1.9.2-1.x86_64 (internal-aws-arched)
               nginx(x86-64) = 1.9.2-1
Error: Package: 1:nginx-mod-http-xslt-filter-1.12.1-1.33.amzn1.x86_64 (amzn-main)
           Requires: nginx(x86-64) = 1:1.12.1-1.33.amzn1
           Removing: 1:nginx-1.8.1-1.26.amzn1.x86_64 (@amzn-main)
               nginx(x86-64) = 1:1.8.1-1.26.amzn1
           Obsoleted By: 1:nginx-all-modules-1.12.1-1.33.amzn1.x86_64 (amzn-main)
               Not found
           Available: nginx-1.9.2-1.x86_64 (internal-aws-arched)
               nginx(x86-64) = 1.9.2-1
Error: Package: 1:nginx-mod-http-perl-1.12.1-1.33.amzn1.x86_64 (amzn-main)
           Requires: nginx(x86-64) = 1:1.12.1-1.33.amzn1
           Removing: 1:nginx-1.8.1-1.26.amzn1.x86_64 (@amzn-main)
               nginx(x86-64) = 1:1.8.1-1.26.amzn1
           Obsoleted By: 1:nginx-all-modules-1.12.1-1.33.amzn1.x86_64 (amzn-main)
               Not found
           Available: nginx-1.9.2-1.x86_64 (internal-aws-arched)
               nginx(x86-64) = 1.9.2-1
Error: Package: 1:nginx-mod-http-image-filter-1.12.1-1.33.amzn1.x86_64 (amzn-main)
           Requires: nginx(x86-64) = 1:1.12.1-1.33.amzn1
           Removing: 1:nginx-1.8.1-1.26.amzn1.x86_64 (@amzn-main)
               nginx(x86-64) = 1:1.8.1-1.26.amzn1
           Obsoleted By: 1:nginx-all-modules-1.12.1-1.33.amzn1.x86_64 (amzn-main)
               Not found
           Available: nginx-1.9.2-1.x86_64 (internal-aws-arched)
               nginx(x86-64) = 1.9.2-1
Error: Package: 1:nginx-mod-stream-1.12.1-1.33.amzn1.x86_64 (amzn-main)
           Requires: nginx(x86-64) = 1:1.12.1-1.33.amzn1
           Removing: 1:nginx-1.8.1-1.26.amzn1.x86_64 (@amzn-main)
               nginx(x86-64) = 1:1.8.1-1.26.amzn1
           Obsoleted By: 1:nginx-all-modules-1.12.1-1.33.amzn1.x86_64 (amzn-main)
               Not found
           Available: nginx-1.9.2-1.x86_64 (internal-aws-arched)
               nginx(x86-64) = 1.9.2-1
Error: Package: 1:nginx-mod-mail-1.12.1-1.33.amzn1.x86_64 (amzn-main)
           Requires: nginx(x86-64) = 1:1.12.1-1.33.amzn1
           Removing: 1:nginx-1.8.1-1.26.amzn1.x86_64 (@amzn-main)
               nginx(x86-64) = 1:1.8.1-1.26.amzn1
           Obsoleted By: 1:nginx-all-modules-1.12.1-1.33.amzn1.x86_64 (amzn-main)
               Not found
           Available: nginx-1.9.2-1.x86_64 (internal-aws-arched)
               nginx(x86-64) = 1.9.2-1
 You could try using --skip-broken to work around the problem
 You could try running: rpm -Va --nofiles --nodigest

Initially tried the suggested --skip-broken to work around the problem, but alas:

$ sudo yum update --skip-broken
Loaded plugins: priorities, update-motd, upgrade-helper
amzn-main                                                                                                                                                                                                                                                | 2.1 kB  00:00:00
amzn-updates                                                                                                                                                                                                                                             | 2.5 kB  00:00:00
internal-aws-arched                                                                                                                                                                                                                                      | 2.9 kB  00:00:00
internal-aws-noarch                                                                                                                                                                                                                                      | 2.9 kB  00:00:00
1 packages excluded due to repository priority protections
Resolving Dependencies
--> Running transaction check
---> Package nginx.x86_64 1:1.8.1-1.26.amzn1 will be obsoleted
---> Package nginx-all-modules.x86_64 1:1.12.1-1.33.amzn1 will be obsoleting
--> Processing Dependency: nginx-mod-stream(x86-64) = 1:1.12.1-1.33.amzn1 for package: 1:nginx-all-modules-1.12.1-1.33.amzn1.x86_64
--> Processing Dependency: nginx-mod-mail(x86-64) = 1:1.12.1-1.33.amzn1 for package: 1:nginx-all-modules-1.12.1-1.33.amzn1.x86_64
--> Processing Dependency: nginx-mod-http-xslt-filter(x86-64) = 1:1.12.1-1.33.amzn1 for package: 1:nginx-all-modules-1.12.1-1.33.amzn1.x86_64
--> Processing Dependency: nginx-mod-http-perl(x86-64) = 1:1.12.1-1.33.amzn1 for package: 1:nginx-all-modules-1.12.1-1.33.amzn1.x86_64
--> Processing Dependency: nginx-mod-http-image-filter(x86-64) = 1:1.12.1-1.33.amzn1 for package: 1:nginx-all-modules-1.12.1-1.33.amzn1.x86_64
--> Processing Dependency: nginx-mod-http-geoip(x86-64) = 1:1.12.1-1.33.amzn1 for package: 1:nginx-all-modules-1.12.1-1.33.amzn1.x86_64
--> Running transaction check
---> Package nginx-mod-http-geoip.x86_64 1:1.12.1-1.33.amzn1 will be installed
--> Processing Dependency: nginx(x86-64) = 1:1.12.1-1.33.amzn1 for package: 1:nginx-mod-http-geoip-1.12.1-1.33.amzn1.x86_64
---> Package nginx-mod-http-image-filter.x86_64 1:1.12.1-1.33.amzn1 will be installed
--> Processing Dependency: nginx(x86-64) = 1:1.12.1-1.33.amzn1 for package: 1:nginx-mod-http-image-filter-1.12.1-1.33.amzn1.x86_64
---> Package nginx-mod-http-perl.x86_64 1:1.12.1-1.33.amzn1 will be installed
--> Processing Dependency: nginx(x86-64) = 1:1.12.1-1.33.amzn1 for package: 1:nginx-mod-http-perl-1.12.1-1.33.amzn1.x86_64
---> Package nginx-mod-http-xslt-filter.x86_64 1:1.12.1-1.33.amzn1 will be installed
--> Processing Dependency: nginx(x86-64) = 1:1.12.1-1.33.amzn1 for package: 1:nginx-mod-http-xslt-filter-1.12.1-1.33.amzn1.x86_64
---> Package nginx-mod-mail.x86_64 1:1.12.1-1.33.amzn1 will be installed
--> Processing Dependency: nginx(x86-64) = 1:1.12.1-1.33.amzn1 for package: 1:nginx-mod-mail-1.12.1-1.33.amzn1.x86_64
---> Package nginx-mod-stream.x86_64 1:1.12.1-1.33.amzn1 will be installed
--> Processing Dependency: nginx(x86-64) = 1:1.12.1-1.33.amzn1 for package: 1:nginx-mod-stream-1.12.1-1.33.amzn1.x86_64
--> Running transaction check
---> Package nginx.x86_64 1:1.8.1-1.26.amzn1 will be obsoleted

Packages skipped because of dependency problems:
    1:nginx-all-modules-1.12.1-1.33.amzn1.x86_64 from amzn-main
    1:nginx-mod-http-geoip-1.12.1-1.33.amzn1.x86_64 from amzn-main
    1:nginx-mod-http-image-filter-1.12.1-1.33.amzn1.x86_64 from amzn-main
    1:nginx-mod-http-perl-1.12.1-1.33.amzn1.x86_64 from amzn-main
    1:nginx-mod-http-xslt-filter-1.12.1-1.33.amzn1.x86_64 from amzn-main
    1:nginx-mod-mail-1.12.1-1.33.amzn1.x86_64 from amzn-main
    1:nginx-mod-stream-1.12.1-1.33.amzn1.x86_64 from amzn-main

While not eliminating the problem it certainly makes it easier to find the source of my woe:

---> Package nginx.x86_64 1:1.8.1-1.26.amzn1 will be obsoleted
---> Package nginx-all-modules.x86_64 1:1.12.1-1.33.amzn1 will be obsoleting

It looks like when AWS moved from 1.8 they, at some point, moved from separate packages to a monolith. In our testing environment I took a look at one of the impacted instances and removed nginx hoping a fresh install would resolve the issue:

$ sudo yum remove nginx
Loaded plugins: priorities, update-motd, upgrade-helper
Resolving Dependencies
--> Running transaction check
---> Package nginx.x86_64 1:1.8.1-1.26.amzn1 will be erased
--> Finished Dependency Resolution

Dependencies Resolved

================================================================================================================================================================================================================================================================================
 Package                                                      Arch                                                          Version                                                                     Repository                                                         Size
================================================================================================================================================================================================================================================================================
Removing:
 nginx                                                        x86_64                                                        1:1.8.1-1.26.amzn1                                                          @amzn-main                                                        1.3 M

Transaction Summary
================================================================================================================================================================================================================================================================================
Remove  1 Package

Installed size: 1.3 M
Is this ok [y/N]: y
Downloading packages:
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Erasing    : 1:nginx-1.8.1-1.26.amzn1.x86_64                                                                                                                                                                                                                              1/1
warning: /etc/nginx/nginx.conf saved as /etc/nginx/nginx.conf.rpmsave
warning: /etc/logrotate.d/nginx saved as /etc/logrotate.d/nginx.rpmsave
  Verifying  : 1:nginx-1.8.1-1.26.amzn1.x86_64                                                                                                                                                                                                                              1/1

Removed:
  nginx.x86_64 1:1.8.1-1.26.amzn1

Complete!


$ sudo yum install nginx
Loaded plugins: priorities, update-motd, upgrade-helper
1 packages excluded due to repository priority protections
Package nginx is obsoleted by nginx-all-modules, trying to install 1:nginx-all-modules-1.12.1-1.33.amzn1.x86_64 instead
Resolving Dependencies
--> Running transaction check
---> Package nginx-all-modules.x86_64 1:1.12.1-1.33.amzn1 will be installed
--> Processing Dependency: nginx-mod-stream(x86-64) = 1:1.12.1-1.33.amzn1 for package: 1:nginx-all-modules-1.12.1-1.33.amzn1.x86_64
--> Processing Dependency: nginx-mod-mail(x86-64) = 1:1.12.1-1.33.amzn1 for package: 1:nginx-all-modules-1.12.1-1.33.amzn1.x86_64
--> Processing Dependency: nginx-mod-http-xslt-filter(x86-64) = 1:1.12.1-1.33.amzn1 for package: 1:nginx-all-modules-1.12.1-1.33.amzn1.x86_64
--> Processing Dependency: nginx-mod-http-perl(x86-64) = 1:1.12.1-1.33.amzn1 for package: 1:nginx-all-modules-1.12.1-1.33.amzn1.x86_64
--> Processing Dependency: nginx-mod-http-image-filter(x86-64) = 1:1.12.1-1.33.amzn1 for package: 1:nginx-all-modules-1.12.1-1.33.amzn1.x86_64
--> Processing Dependency: nginx-mod-http-geoip(x86-64) = 1:1.12.1-1.33.amzn1 for package: 1:nginx-all-modules-1.12.1-1.33.amzn1.x86_64
--> Running transaction check
---> Package nginx-mod-http-geoip.x86_64 1:1.12.1-1.33.amzn1 will be installed
--> Processing Dependency: nginx(x86-64) = 1:1.12.1-1.33.amzn1 for package: 1:nginx-mod-http-geoip-1.12.1-1.33.amzn1.x86_64
---> Package nginx-mod-http-image-filter.x86_64 1:1.12.1-1.33.amzn1 will be installed
--> Processing Dependency: nginx(x86-64) = 1:1.12.1-1.33.amzn1 for package: 1:nginx-mod-http-image-filter-1.12.1-1.33.amzn1.x86_64
---> Package nginx-mod-http-perl.x86_64 1:1.12.1-1.33.amzn1 will be installed
--> Processing Dependency: nginx(x86-64) = 1:1.12.1-1.33.amzn1 for package: 1:nginx-mod-http-perl-1.12.1-1.33.amzn1.x86_64
---> Package nginx-mod-http-xslt-filter.x86_64 1:1.12.1-1.33.amzn1 will be installed
--> Processing Dependency: nginx(x86-64) = 1:1.12.1-1.33.amzn1 for package: 1:nginx-mod-http-xslt-filter-1.12.1-1.33.amzn1.x86_64
---> Package nginx-mod-mail.x86_64 1:1.12.1-1.33.amzn1 will be installed
--> Processing Dependency: nginx(x86-64) = 1:1.12.1-1.33.amzn1 for package: 1:nginx-mod-mail-1.12.1-1.33.amzn1.x86_64
---> Package nginx-mod-stream.x86_64 1:1.12.1-1.33.amzn1 will be installed
--> Processing Dependency: nginx(x86-64) = 1:1.12.1-1.33.amzn1 for package: 1:nginx-mod-stream-1.12.1-1.33.amzn1.x86_64
--> Finished Dependency Resolution
Error: Package: 1:nginx-mod-http-xslt-filter-1.12.1-1.33.amzn1.x86_64 (amzn-main)
           Requires: nginx(x86-64) = 1:1.12.1-1.33.amzn1
           Available: nginx-1.9.2-1.x86_64 (internal-aws-arched)
               nginx(x86-64) = 1.9.2-1
Error: Package: 1:nginx-mod-http-image-filter-1.12.1-1.33.amzn1.x86_64 (amzn-main)
           Requires: nginx(x86-64) = 1:1.12.1-1.33.amzn1
           Available: nginx-1.9.2-1.x86_64 (internal-aws-arched)
               nginx(x86-64) = 1.9.2-1
Error: Package: 1:nginx-mod-mail-1.12.1-1.33.amzn1.x86_64 (amzn-main)
           Requires: nginx(x86-64) = 1:1.12.1-1.33.amzn1
           Available: nginx-1.9.2-1.x86_64 (internal-aws-arched)
               nginx(x86-64) = 1.9.2-1
Error: Package: 1:nginx-mod-http-perl-1.12.1-1.33.amzn1.x86_64 (amzn-main)
           Requires: nginx(x86-64) = 1:1.12.1-1.33.amzn1
           Available: nginx-1.9.2-1.x86_64 (internal-aws-arched)
               nginx(x86-64) = 1.9.2-1
Error: Package: 1:nginx-mod-stream-1.12.1-1.33.amzn1.x86_64 (amzn-main)
           Requires: nginx(x86-64) = 1:1.12.1-1.33.amzn1
           Available: nginx-1.9.2-1.x86_64 (internal-aws-arched)
               nginx(x86-64) = 1.9.2-1
Error: Package: 1:nginx-mod-http-geoip-1.12.1-1.33.amzn1.x86_64 (amzn-main)
           Requires: nginx(x86-64) = 1:1.12.1-1.33.amzn1
           Available: nginx-1.9.2-1.x86_64 (internal-aws-arched)
               nginx(x86-64) = 1.9.2-1
 You could try using --skip-broken to work around the problem
 You could try running: rpm -Va --nofiles --nodigest

Oh, good. It looks like the package was renamed from nginx to nginx-all-modules, but that's the least of my woes. My main woe is this pair of lines that appear on repeat:

Requires: nginx(x86-64) = 1:1.12.1-1.33.amzn1
Available: nginx-1.9.2-1.x86_64 (internal-aws-arched)

So, so the nginx-all-modules package that is the new and shiny version requires version 1.12, but it can only find 1.9. What gives?

Assumptions are bad but you gotta start somewhere.

Assumption 1: The package installer needs 1.12, but is only finding 1.9. There are probably conflicting repos.

Assumption 2: It is unlikely that AWS has repos that are so broken that this is an AWS problem, because nginx is commonly used and there'd probably be a ton of bug submissions over this.

Assumption 3: There is another repo, somewhere, with an older version of nginx wrecking havoc on my life.

Running cat /etc/yum.repos.d/* I get ... a ton of output. With some focus:

$ cat /etc/yum.repos.d/* | grep 'example\.com'
baseurl=http://yum.int.example.com/yum/noarch/
baseurl=http://yum.int.example.com/yum/x86_64/
$ ack 'example\.com' /etc/yum.repos.d/
/etc/yum.repos.d/internal.repo
3:baseurl=http://yum.int.example.com/yum/noarch/
11:baseurl=http://yum.int.example.com/yum/x86_64/

Oh, look at that. Internal repos. If I open the file and change enabled to 0:

[internal-aws-noarch]
name=Private Internal Repo (noarch)
baseurl=http://yum.int.example.com/yum/noarch/
enabled=0
gpgcheck=0
priority=1
metadata_expire=1200

[internal-aws-arched]
name=Private Internal Repo (arched)
baseurl=http://yum.int.example.com/yum/x86_64/
enabled=0
gpgcheck=0
priority=1
metadata_expire=1200

Now let's see when I try to install nginx:

$ sudo yum install nginx
Loaded plugins: priorities, update-motd, upgrade-helper
Resolving Dependencies
--> Running transaction check
---> Package nginx.x86_64 1:1.12.1-1.33.amzn1 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

================================================================================================================================================================================================================================================================================
 Package                                                      Arch                                                          Version                                                                      Repository                                                        Size
================================================================================================================================================================================================================================================================================
Installing:
 nginx                                                        x86_64                                                        1:1.12.1-1.33.amzn1                                                          amzn-main                                                        561 k

Transaction Summary
================================================================================================================================================================================================================================================================================
Install  1 Package

Total download size: 561 k
Installed size: 1.4 M
Is this ok [y/d/N]: y
Downloading packages:
nginx-1.12.1-1.33.amzn1.x86_64.rpm                                                                                                                                                                                                                       | 561 kB  00:00:00
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Installing : 1:nginx-1.12.1-1.33.amzn1.x86_64                                                                                                                                                                                                                             1/1
  Verifying  : 1:nginx-1.12.1-1.33.amzn1.x86_64                                                                                                                                                                                                                             1/1

Installed:
  nginx.x86_64 1:1.12.1-1.33.amzn1

Complete!

Bam!

Documented on my frequently used assets page.