Today I ran into an interesting problem. As a bit of context, what I typically work with on a daily basis is a decent number of instances running Amazon Linux. Today, one of my coworkers, who I'll call Jayne Cobb, said that he needed access to another coworker's, who I'll call George Parley, dev box for testing. With George's permission, of course.
Seems like it'd be a straightforward thing: ssh
onto instance, useradd ${USER}
, etc.
$ sudo useradd jayne
useradd: user 'jayne' already exists
Well that's odd. Jayne just asked for access now, but maybe George already made his user and just didn't tell him?
$ sudo su - jayne
org.freedesktop.DBus.Error.FileNotFound: Failed to connect to socket /var/run/dbus/system_bus_socket: No such file or directory
Last login: Thu Oct 5 19:18:56 UTC 2017 on pts/0
su: warning: cannot change directory to /home/jayne: No such file or directory
-bash-4.2$
What.
Well let's see if this user legit exists, shall we? First stop: vipw
, which checks /etc/passwd
. File contents look a little something like this:
root:x:0:0:root:/root:/bin/bash
bin:x:1:1:bin:/bin:/sbin/nologin
{{ snip }}
Nothing in the file about a user named jayne
though. 🤔
Next stop: /etc/shadow
. /etc/shadow
, amongst other things, stores encrypted passwords for user accounts amongst other attributes. So it would make sense that if George created a user for Jayne, that the account would be in here. Alas, jayne
was not.
Final stop: getent passwd jayne
. Output?
$ getent passwd jayne
jayne:*:1618601212:1618600513:Jayne Cobb:/home/jayne:/bin/bash
Wait. What.
That is a long as... long as f... long user ID. And group ID. Also, where the heck is this user coming from and why is there a home directory there that doesn't appear to really exist?
There are two more files I'm going to check: /etc/passwd-
and /etc/shadow-
, which are backups of the respective files above. Fortunately / unfortunately there is no indication of jayne
. Again.
Time to dig deep.
$ strace getent passwd jayne &> getent.strace
(Protip: You'll definitely want to redirect your output to file. You're welcome.)
strace
is a diagnostic tool that will show how what processes are being called when, at least in this case, I run getent passwd jayne
. I'm not going to post the whole output here for a variety of reasons, but if you want to see the output for yourself you can easily run the above command and replace jayne
with any actual user on your system.
Glancing through the output though, this happens:
{{ snip }}
open("/etc/passwd", O_RDONLY|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=1718, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f3b921a2000
read(3, "root:x:0:0:root:/root:/bin/bash\n"..., 4096) = 1718
read(3, "", 4096) = 0
close(3) = 0
munmap(0x7f3b921a2000, 4096) = 0
open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=38848, ...}) = 0
mmap(NULL, 38848, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f3b92199000
close(3) = 0
open("/usr/lib64/libnss_sss.so.2", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\0\30\0\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=33440, ...}) = 0
{{ snip }}
Hrm, what's this? Well it look like getent
tried /etc/passwd
and, failing that, moved onto sssd
(the libnss_sss.so.2
line). This points to George using sssd
, the System Security Services Daemon, which is a way of managing remote directories / authentication. Basically, it looks like George is using an external mechanism for user accounts on his instance. Let's take a look at his /etc/sssd/sssd.conf
file:
[sssd]
domains = ad.example.com
config_file_version = 2
services = nss, pam, ssh
[domain/ad.example.com]
{{snip}}
Ah, so he has linked his Linux instance to our Windows AD server, which is why the user exists, but does so outside the realm of /etc/passwd
and /etc/shadow
. It also explains the atypically high (for Linux) user and group IDs for Jayne.
Some additional, useful, info to have
What is getent passwd ${USER}
?
Straight from the man
page:
The
getent
command displays entries from databases supported by the Name Service Switch libraries, which are configured in/etc/nsswitch.conf
.
In somewhat plainer English: gentent
looks through a specified text based database, like passwd
, for the value provided (${USER}
in this case). So when I ran getent passwd jayne
it looked through all the password databases for that string.
Note that I said databases, plural. /etc/passwd
and /etc/shadow
are two text file databases, but those are only local databases. Since getent
searches local and remote databases, getent
was still able to find jayne
even though jayne
wasn't created locally.
What is /etc/nsswitch.conf
?
Recall that getent
uses the Name Service Switch libraries from the man
page? Well the NSS is configured in /etc/nsswitch.conf
. A relevant snippet from George's /etc/nsswitch.conf
file:
# Example:
#passwd: db files nisplus nis
#shadow: db files nisplus nis
#group: db files nisplus nis
passwd: files sss
shadow: files sss
group: files sss
#hosts: db files nisplus nis dns
hosts: files dns
#
are comment lines, but you can see in this nsswitch.conf
file that NSS is checking files
first, sss
second. That means when I ran getent passwd jayne
, that getent
first looked in the local file databases, e.g. /etc/passwd
and /etc/shadow
, before looking for remote databases, e.g. the remote database specified /etc/sssd/sssd.conf
.
What does the strace
output look like if the user is not using sssd
?
Excellent question! The process actually terminates after looking in /etc/passwd
, e.g. looking up my user on a different Linux instance:
{{ snip }}
open("/etc/passwd", O_RDONLY|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=1413, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f8cc3fc6000
read(3, "root:x:0:0:root:/root:/bin/bash\n"..., 4096) = 1413
close(3) = 0
munmap(0x7f8cc3fc6000, 4096) = 0
fstat(1, {st_mode=S_IFREG|0644, st_size=4021, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f8cc3fc6000
write(1, "quintessence:x:501:502:Quintesse"..., 65quintessence:x:501:502:Quintessence:/home/quintessence:/bin/bash
) = 65
exit_group(0) = ?
+++ exited with 0 +++
As you can see, getent
opened /etc/passwd
, found the desired user, and closed. And just to further clarify, if the user didn't exist, like iamnotauser
:
open("/etc/passwd", O_RDONLY|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=1413, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f8a64b1f000
read(3, "root:x:0:0:root:/root:/bin/bash\n"..., 4096) = 1413
read(3, "", 4096) = 0
close(3) = 0
munmap(0x7f8a64b1f000, 4096) = 0
open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=20465, ...}) = 0
mmap(NULL, 20465, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f8a64b1b000
close(3) = 0
open("/lib64/tls/x86_64/libnss_sss.so.2", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat("/lib64/tls/x86_64", 0x7ffea7f89360) = -1 ENOENT (No such file or directory)
open("/lib64/tls/libnss_sss.so.2", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat("/lib64/tls", {st_mode=S_IFDIR|0555, st_size=4096, ...}) = 0
open("/lib64/x86_64/libnss_sss.so.2", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat("/lib64/x86_64", 0x7ffea7f89360) = -1 ENOENT (No such file or directory)
open("/lib64/libnss_sss.so.2", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat("/lib64", {st_mode=S_IFDIR|0555, st_size=12288, ...}) = 0
open("/usr/lib64/tls/x86_64/libnss_sss.so.2", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat("/usr/lib64/tls/x86_64", 0x7ffea7f89360) = -1 ENOENT (No such file or directory)
open("/usr/lib64/tls/libnss_sss.so.2", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat("/usr/lib64/tls", {st_mode=S_IFDIR|0555, st_size=4096, ...}) = 0
open("/usr/lib64/x86_64/libnss_sss.so.2", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat("/usr/lib64/x86_64", 0x7ffea7f89360) = -1 ENOENT (No such file or directory)
open("/usr/lib64/libnss_sss.so.2", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat("/usr/lib64", {st_mode=S_IFDIR|0555, st_size=20480, ...}) = 0
munmap(0x7f8a64b1b000, 20465) = 0
exit_group(2) = ?
+++ exited with 2 +++
The fastest way to see the user doesn't exist is by noting that the exit code is non-zero, but to further investigate if sssd
is present you can that when getent
tries to open the sssd
socket that a No such file or directory
error is returned. No sssd
on this instance :)
Documented on my frequently used assets page.