Opsfire: In Dependency Purgatory with Nginx
Recently ran into this issue on an a few instances in an environment I'm working in:
$ sudo yum upgrade
Loaded plugins: priorities, update-motd, upgrade-helper
1 packages excluded due to repository priority protections
Resolving Dependencies
--> Running transaction check
---> Package nginx.x86_64 1:1.8.1-1.26.amzn1 will be obsoleted
---> Package nginx-all-modules.x86_64 1:1.12.1-1.33.amzn1 will be obsoleting
--> Processing Dependency: nginx-mod-stream(x86-64) = 1:1.12.1-1.33.amzn1 for package: 1:nginx-all-modules-1.12.1-1.33.amzn1.x86_64
--> Processing Dependency: nginx-mod-mail(x86-64) = 1:1.12.1-1.33.amzn1 for package: 1:nginx-all-modules-1.12.1-1.33.amzn1.x86_64
--> Processing Dependency: nginx-mod-http-xslt-filter(x86-64) = 1:1.12.1-1.33.amzn1 for package: 1:nginx-all-modules-1.12.1-1.33.amzn1.x86_64
--> Processing Dependency: nginx-mod-http-perl(x86-64) = 1:1.12.1-1.33.amzn1 for package: 1:nginx-all-modules-1.12.1-1.33.amzn1.x86_64
--> Processing Dependency: nginx-mod-http-image-filter(x86-64) = 1:1.12.1-1.33.amzn1 for package: 1:nginx-all-modules-1.12.1-1.33.amzn1.x86_64
--> Processing Dependency: nginx-mod-http-geoip(x86-64) = 1:1.12.1-1.33.amzn1 for package: 1:nginx-all-modules-1.12.1-1.33.amzn1.x86_64
--> Running transaction check
---> Package nginx-mod-http-geoip.x86_64 1:1.12.1-1.33.amzn1 will be installed
--> Processing Dependency: nginx(x86-64) = 1:1.12.1-1.33.amzn1 for package: 1:nginx-mod-http-geoip-1.12.1-1.33.amzn1.x86_64
---> Package nginx-mod-http-image-filter.x86_64 1:1.12.1-1.33.amzn1 will be installed
--> Processing Dependency: nginx(x86-64) = 1:1.12.1-1.33.amzn1 for package: 1:nginx-mod-http-image-filter-1.12.1-1.33.amzn1.x86_64
---> Package nginx-mod-http-perl.x86_64 1:1.12.1-1.33.amzn1 will be installed
--> Processing Dependency: nginx(x86-64) = 1:1.12.1-1.33.amzn1 for package: 1:nginx-mod-http-perl-1.12.1-1.33.amzn1.x86_64
---> Package nginx-mod-http-xslt-filter.x86_64 1:1.12.1-1.33.amzn1 will be installed
--> Processing Dependency: nginx(x86-64) = 1:1.12.1-1.33.amzn1 for package: 1:nginx-mod-http-xslt-filter-1.12.1-1.33.amzn1.x86_64
---> Package nginx-mod-mail.x86_64 1:1.12.1-1.33.amzn1 will be installed
--> Processing Dependency: nginx(x86-64) = 1:1.12.1-1.33.amzn1 for package: 1:nginx-mod-mail-1.12.1-1.33.amzn1.x86_64
---> Package nginx-mod-stream.x86_64 1:1.12.1-1.33.amzn1 will be installed
--> Processing Dependency: nginx(x86-64) = 1:1.12.1-1.33.amzn1 for package: 1:nginx-mod-stream-1.12.1-1.33.amzn1.x86_64
--> Finished Dependency Resolution
Error: Package: 1:nginx-mod-http-geoip-1.12.1-1.33.amzn1.x86_64 (amzn-main)
Requires: nginx(x86-64) = 1:1.12.1-1.33.amzn1
Removing: 1:nginx-1.8.1-1.26.amzn1.x86_64 (@amzn-main)
nginx(x86-64) = 1:1.8.1-1.26.amzn1
Obsoleted By: 1:nginx-all-modules-1.12.1-1.33.amzn1.x86_64 (amzn-main)
Not found
Available: nginx-1.9.2-1.x86_64 (internal-aws-arched)
nginx(x86-64) = 1.9.2-1
Error: Package: 1:nginx-mod-http-xslt-filter-1.12.1-1.33.amzn1.x86_64 (amzn-main)
Requires: nginx(x86-64) = 1:1.12.1-1.33.amzn1
Removing: 1:nginx-1.8.1-1.26.amzn1.x86_64 (@amzn-main)
nginx(x86-64) = 1:1.8.1-1.26.amzn1
Obsoleted By: 1:nginx-all-modules-1.12.1-1.33.amzn1.x86_64 (amzn-main)
Not found
Available: nginx-1.9.2-1.x86_64 (internal-aws-arched)
nginx(x86-64) = 1.9.2-1
Error: Package: 1:nginx-mod-http-perl-1.12.1-1.33.amzn1.x86_64 (amzn-main)
Requires: nginx(x86-64) = 1:1.12.1-1.33.amzn1
Removing: 1:nginx-1.8.1-1.26.amzn1.x86_64 (@amzn-main)
nginx(x86-64) = 1:1.8.1-1.26.amzn1
Obsoleted By: 1:nginx-all-modules-1.12.1-1.33.amzn1.x86_64 (amzn-main)
Not found
Available: nginx-1.9.2-1.x86_64 (internal-aws-arched)
nginx(x86-64) = 1.9.2-1
Error: Package: 1:nginx-mod-http-image-filter-1.12.1-1.33.amzn1.x86_64 (amzn-main)
Requires: nginx(x86-64) = 1:1.12.1-1.33.amzn1
Removing: 1:nginx-1.8.1-1.26.amzn1.x86_64 (@amzn-main)
nginx(x86-64) = 1:1.8.1-1.26.amzn1
Obsoleted By: 1:nginx-all-modules-1.12.1-1.33.amzn1.x86_64 (amzn-main)
Not found
Available: nginx-1.9.2-1.x86_64 (internal-aws-arched)
nginx(x86-64) = 1.9.2-1
Error: Package: 1:nginx-mod-stream-1.12.1-1.33.amzn1.x86_64 (amzn-main)
Requires: nginx(x86-64) = 1:1.12.1-1.33.amzn1
Removing: 1:nginx-1.8.1-1.26.amzn1.x86_64 (@amzn-main)
nginx(x86-64) = 1:1.8.1-1.26.amzn1
Obsoleted By: 1:nginx-all-modules-1.12.1-1.33.amzn1.x86_64 (amzn-main)
Not found
Available: nginx-1.9.2-1.x86_64 (internal-aws-arched)
nginx(x86-64) = 1.9.2-1
Error: Package: 1:nginx-mod-mail-1.12.1-1.33.amzn1.x86_64 (amzn-main)
Requires: nginx(x86-64) = 1:1.12.1-1.33.amzn1
Removing: 1:nginx-1.8.1-1.26.amzn1.x86_64 (@amzn-main)
nginx(x86-64) = 1:1.8.1-1.26.amzn1
Obsoleted By: 1:nginx-all-modules-1.12.1-1.33.amzn1.x86_64 (amzn-main)
Not found
Available: nginx-1.9.2-1.x86_64 (internal-aws-arched)
nginx(x86-64) = 1.9.2-1
You could try using --skip-broken to work around the problem
You could try running: rpm -Va --nofiles --nodigest
Initially tried the suggested --skip-broken to work around the problem, but alas:
$ sudo yum update --skip-broken
Loaded plugins: priorities, update-motd, upgrade-helper
amzn-main | 2.1 kB 00:00:00
amzn-updates | 2.5 kB 00:00:00
internal-aws-arched | 2.9 kB 00:00:00
internal-aws-noarch | 2.9 kB 00:00:00
1 packages excluded due to repository priority protections
Resolving Dependencies
--> Running transaction check
---> Package nginx.x86_64 1:1.8.1-1.26.amzn1 will be obsoleted
---> Package nginx-all-modules.x86_64 1:1.12.1-1.33.amzn1 will be obsoleting
--> Processing Dependency: nginx-mod-stream(x86-64) = 1:1.12.1-1.33.amzn1 for package: 1:nginx-all-modules-1.12.1-1.33.amzn1.x86_64
--> Processing Dependency: nginx-mod-mail(x86-64) = 1:1.12.1-1.33.amzn1 for package: 1:nginx-all-modules-1.12.1-1.33.amzn1.x86_64
--> Processing Dependency: nginx-mod-http-xslt-filter(x86-64) = 1:1.12.1-1.33.amzn1 for package: 1:nginx-all-modules-1.12.1-1.33.amzn1.x86_64
--> Processing Dependency: nginx-mod-http-perl(x86-64) = 1:1.12.1-1.33.amzn1 for package: 1:nginx-all-modules-1.12.1-1.33.amzn1.x86_64
--> Processing Dependency: nginx-mod-http-image-filter(x86-64) = 1:1.12.1-1.33.amzn1 for package: 1:nginx-all-modules-1.12.1-1.33.amzn1.x86_64
--> Processing Dependency: nginx-mod-http-geoip(x86-64) = 1:1.12.1-1.33.amzn1 for package: 1:nginx-all-modules-1.12.1-1.33.amzn1.x86_64
--> Running transaction check
---> Package nginx-mod-http-geoip.x86_64 1:1.12.1-1.33.amzn1 will be installed
--> Processing Dependency: nginx(x86-64) = 1:1.12.1-1.33.amzn1 for package: 1:nginx-mod-http-geoip-1.12.1-1.33.amzn1.x86_64
---> Package nginx-mod-http-image-filter.x86_64 1:1.12.1-1.33.amzn1 will be installed
--> Processing Dependency: nginx(x86-64) = 1:1.12.1-1.33.amzn1 for package: 1:nginx-mod-http-image-filter-1.12.1-1.33.amzn1.x86_64
---> Package nginx-mod-http-perl.x86_64 1:1.12.1-1.33.amzn1 will be installed
--> Processing Dependency: nginx(x86-64) = 1:1.12.1-1.33.amzn1 for package: 1:nginx-mod-http-perl-1.12.1-1.33.amzn1.x86_64
---> Package nginx-mod-http-xslt-filter.x86_64 1:1.12.1-1.33.amzn1 will be installed
--> Processing Dependency: nginx(x86-64) = 1:1.12.1-1.33.amzn1 for package: 1:nginx-mod-http-xslt-filter-1.12.1-1.33.amzn1.x86_64
---> Package nginx-mod-mail.x86_64 1:1.12.1-1.33.amzn1 will be installed
--> Processing Dependency: nginx(x86-64) = 1:1.12.1-1.33.amzn1 for package: 1:nginx-mod-mail-1.12.1-1.33.amzn1.x86_64
---> Package nginx-mod-stream.x86_64 1:1.12.1-1.33.amzn1 will be installed
--> Processing Dependency: nginx(x86-64) = 1:1.12.1-1.33.amzn1 for package: 1:nginx-mod-stream-1.12.1-1.33.amzn1.x86_64
--> Running transaction check
---> Package nginx.x86_64 1:1.8.1-1.26.amzn1 will be obsoleted
Packages skipped because of dependency problems:
1:nginx-all-modules-1.12.1-1.33.amzn1.x86_64 from amzn-main
1:nginx-mod-http-geoip-1.12.1-1.33.amzn1.x86_64 from amzn-main
1:nginx-mod-http-image-filter-1.12.1-1.33.amzn1.x86_64 from amzn-main
1:nginx-mod-http-perl-1.12.1-1.33.amzn1.x86_64 from amzn-main
1:nginx-mod-http-xslt-filter-1.12.1-1.33.amzn1.x86_64 from amzn-main
1:nginx-mod-mail-1.12.1-1.33.amzn1.x86_64 from amzn-main
1:nginx-mod-stream-1.12.1-1.33.amzn1.x86_64 from amzn-main
While not eliminating the problem it certainly makes it easier to find the source of my woe:
---> Package nginx.x86_64 1:1.8.1-1.26.amzn1 will be obsoleted
---> Package nginx-all-modules.x86_64 1:1.12.1-1.33.amzn1 will be obsoleting
It looks like when AWS moved from 1.8 they, at some point, moved from separate packages to a monolith. In our testing environment I took a look at one of the impacted instances and removed nginx hoping a fresh install would resolve the issue:
$ sudo yum remove nginx
Loaded plugins: priorities, update-motd, upgrade-helper
Resolving Dependencies
--> Running transaction check
---> Package nginx.x86_64 1:1.8.1-1.26.amzn1 will be erased
--> Finished Dependency Resolution
Dependencies Resolved
================================================================================================================================================================================================================================================================================
Package Arch Version Repository Size
================================================================================================================================================================================================================================================================================
Removing:
nginx x86_64 1:1.8.1-1.26.amzn1 @amzn-main 1.3 M
Transaction Summary
================================================================================================================================================================================================================================================================================
Remove 1 Package
Installed size: 1.3 M
Is this ok [y/N]: y
Downloading packages:
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
Erasing : 1:nginx-1.8.1-1.26.amzn1.x86_64 1/1
warning: /etc/nginx/nginx.conf saved as /etc/nginx/nginx.conf.rpmsave
warning: /etc/logrotate.d/nginx saved as /etc/logrotate.d/nginx.rpmsave
Verifying : 1:nginx-1.8.1-1.26.amzn1.x86_64 1/1
Removed:
nginx.x86_64 1:1.8.1-1.26.amzn1
Complete!
$ sudo yum install nginx
Loaded plugins: priorities, update-motd, upgrade-helper
1 packages excluded due to repository priority protections
Package nginx is obsoleted by nginx-all-modules, trying to install 1:nginx-all-modules-1.12.1-1.33.amzn1.x86_64 instead
Resolving Dependencies
--> Running transaction check
---> Package nginx-all-modules.x86_64 1:1.12.1-1.33.amzn1 will be installed
--> Processing Dependency: nginx-mod-stream(x86-64) = 1:1.12.1-1.33.amzn1 for package: 1:nginx-all-modules-1.12.1-1.33.amzn1.x86_64
--> Processing Dependency: nginx-mod-mail(x86-64) = 1:1.12.1-1.33.amzn1 for package: 1:nginx-all-modules-1.12.1-1.33.amzn1.x86_64
--> Processing Dependency: nginx-mod-http-xslt-filter(x86-64) = 1:1.12.1-1.33.amzn1 for package: 1:nginx-all-modules-1.12.1-1.33.amzn1.x86_64
--> Processing Dependency: nginx-mod-http-perl(x86-64) = 1:1.12.1-1.33.amzn1 for package: 1:nginx-all-modules-1.12.1-1.33.amzn1.x86_64
--> Processing Dependency: nginx-mod-http-image-filter(x86-64) = 1:1.12.1-1.33.amzn1 for package: 1:nginx-all-modules-1.12.1-1.33.amzn1.x86_64
--> Processing Dependency: nginx-mod-http-geoip(x86-64) = 1:1.12.1-1.33.amzn1 for package: 1:nginx-all-modules-1.12.1-1.33.amzn1.x86_64
--> Running transaction check
---> Package nginx-mod-http-geoip.x86_64 1:1.12.1-1.33.amzn1 will be installed
--> Processing Dependency: nginx(x86-64) = 1:1.12.1-1.33.amzn1 for package: 1:nginx-mod-http-geoip-1.12.1-1.33.amzn1.x86_64
---> Package nginx-mod-http-image-filter.x86_64 1:1.12.1-1.33.amzn1 will be installed
--> Processing Dependency: nginx(x86-64) = 1:1.12.1-1.33.amzn1 for package: 1:nginx-mod-http-image-filter-1.12.1-1.33.amzn1.x86_64
---> Package nginx-mod-http-perl.x86_64 1:1.12.1-1.33.amzn1 will be installed
--> Processing Dependency: nginx(x86-64) = 1:1.12.1-1.33.amzn1 for package: 1:nginx-mod-http-perl-1.12.1-1.33.amzn1.x86_64
---> Package nginx-mod-http-xslt-filter.x86_64 1:1.12.1-1.33.amzn1 will be installed
--> Processing Dependency: nginx(x86-64) = 1:1.12.1-1.33.amzn1 for package: 1:nginx-mod-http-xslt-filter-1.12.1-1.33.amzn1.x86_64
---> Package nginx-mod-mail.x86_64 1:1.12.1-1.33.amzn1 will be installed
--> Processing Dependency: nginx(x86-64) = 1:1.12.1-1.33.amzn1 for package: 1:nginx-mod-mail-1.12.1-1.33.amzn1.x86_64
---> Package nginx-mod-stream.x86_64 1:1.12.1-1.33.amzn1 will be installed
--> Processing Dependency: nginx(x86-64) = 1:1.12.1-1.33.amzn1 for package: 1:nginx-mod-stream-1.12.1-1.33.amzn1.x86_64
--> Finished Dependency Resolution
Error: Package: 1:nginx-mod-http-xslt-filter-1.12.1-1.33.amzn1.x86_64 (amzn-main)
Requires: nginx(x86-64) = 1:1.12.1-1.33.amzn1
Available: nginx-1.9.2-1.x86_64 (internal-aws-arched)
nginx(x86-64) = 1.9.2-1
Error: Package: 1:nginx-mod-http-image-filter-1.12.1-1.33.amzn1.x86_64 (amzn-main)
Requires: nginx(x86-64) = 1:1.12.1-1.33.amzn1
Available: nginx-1.9.2-1.x86_64 (internal-aws-arched)
nginx(x86-64) = 1.9.2-1
Error: Package: 1:nginx-mod-mail-1.12.1-1.33.amzn1.x86_64 (amzn-main)
Requires: nginx(x86-64) = 1:1.12.1-1.33.amzn1
Available: nginx-1.9.2-1.x86_64 (internal-aws-arched)
nginx(x86-64) = 1.9.2-1
Error: Package: 1:nginx-mod-http-perl-1.12.1-1.33.amzn1.x86_64 (amzn-main)
Requires: nginx(x86-64) = 1:1.12.1-1.33.amzn1
Available: nginx-1.9.2-1.x86_64 (internal-aws-arched)
nginx(x86-64) = 1.9.2-1
Error: Package: 1:nginx-mod-stream-1.12.1-1.33.amzn1.x86_64 (amzn-main)
Requires: nginx(x86-64) = 1:1.12.1-1.33.amzn1
Available: nginx-1.9.2-1.x86_64 (internal-aws-arched)
nginx(x86-64) = 1.9.2-1
Error: Package: 1:nginx-mod-http-geoip-1.12.1-1.33.amzn1.x86_64 (amzn-main)
Requires: nginx(x86-64) = 1:1.12.1-1.33.amzn1
Available: nginx-1.9.2-1.x86_64 (internal-aws-arched)
nginx(x86-64) = 1.9.2-1
You could try using --skip-broken to work around the problem
You could try running: rpm -Va --nofiles --nodigest
Oh, good. It looks like the package was renamed from nginx to nginx-all-modules, but that's the least of my woes. My main woe is this pair of lines that appear on repeat:
Requires: nginx(x86-64) = 1:1.12.1-1.33.amzn1
Available: nginx-1.9.2-1.x86_64 (internal-aws-arched)
So, so the nginx-all-modules package that is the new and shiny version requires version 1.12, but it can only find 1.9. What gives?
Assumptions are bad but you gotta start somewhere.
Assumption 1: The package installer needs 1.12, but is only finding 1.9. There are probably conflicting repos.
Assumption 2: It is unlikely that AWS has repos that are so broken that this is an AWS problem, because nginx is commonly used and there'd probably be a ton of bug submissions over this.
Assumption 3: There is another repo, somewhere, with an older version of nginx wrecking havoc on my life.
Running cat /etc/yum.repos.d/* I get ... a ton of output. With some focus:
$ cat /etc/yum.repos.d/* | grep 'example\.com'
baseurl=http://yum.int.example.com/yum/noarch/
baseurl=http://yum.int.example.com/yum/x86_64/
$ ack 'example\.com' /etc/yum.repos.d/
/etc/yum.repos.d/internal.repo
3:baseurl=http://yum.int.example.com/yum/noarch/
11:baseurl=http://yum.int.example.com/yum/x86_64/
Oh, look at that. Internal repos. If I open the file and change enabled to 0:
[internal-aws-noarch]
name=Private Internal Repo (noarch)
baseurl=http://yum.int.example.com/yum/noarch/
enabled=0
gpgcheck=0
priority=1
metadata_expire=1200
[internal-aws-arched]
name=Private Internal Repo (arched)
baseurl=http://yum.int.example.com/yum/x86_64/
enabled=0
gpgcheck=0
priority=1
metadata_expire=1200
Now let's see when I try to install nginx:
$ sudo yum install nginx
Loaded plugins: priorities, update-motd, upgrade-helper
Resolving Dependencies
--> Running transaction check
---> Package nginx.x86_64 1:1.12.1-1.33.amzn1 will be installed
--> Finished Dependency Resolution
Dependencies Resolved
================================================================================================================================================================================================================================================================================
Package Arch Version Repository Size
================================================================================================================================================================================================================================================================================
Installing:
nginx x86_64 1:1.12.1-1.33.amzn1 amzn-main 561 k
Transaction Summary
================================================================================================================================================================================================================================================================================
Install 1 Package
Total download size: 561 k
Installed size: 1.4 M
Is this ok [y/d/N]: y
Downloading packages:
nginx-1.12.1-1.33.amzn1.x86_64.rpm | 561 kB 00:00:00
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
Installing : 1:nginx-1.12.1-1.33.amzn1.x86_64 1/1
Verifying : 1:nginx-1.12.1-1.33.amzn1.x86_64 1/1
Installed:
nginx.x86_64 1:1.12.1-1.33.amzn1
Complete!
Bam!
Documented on my frequently used assets page.