OpsFire: Where's That User?

by Quintessence

Today I ran into an interesting problem. As a bit of context, what I typically work with on a daily basis is a decent number of instances running Amazon Linux. Today, one of my coworkers, who I'll call Jayne Cobb, said that he needed access to another coworker's, who I'll call George Parley, dev box for testing. With George's permission, of course.

Seems like it'd be a straightforward thing: ssh onto instance, useradd ${USER}, etc.

$  sudo useradd jayne
useradd: user 'jayne' already exists

Well that's odd. Jayne just asked for access now, but maybe George already made his user and just didn't tell him?

$  sudo su - jayne
org.freedesktop.DBus.Error.FileNotFound: Failed to connect to socket /var/run/dbus/system_bus_socket: No such file or directory
Last login: Thu Oct  5 19:18:56 UTC 2017 on pts/0
su: warning: cannot change directory to /home/jayne: No such file or directory
-bash-4.2$

What.

Well let's see if this user legit exists, shall we? First stop: vipw, which checks /etc/passwd. File contents look a little something like this:

root:x:0:0:root:/root:/bin/bash
bin:x:1:1:bin:/bin:/sbin/nologin
{{ snip }}

Nothing in the file about a user named jayne though. 🤔

Next stop: /etc/shadow. /etc/shadow, amongst other things, stores encrypted passwords for user accounts amongst other attributes. So it would make sense that if George created a user for Jayne, that the account would be in here. Alas, jayne was not.

Final stop: getent passwd jayne. Output?

$  getent passwd jayne
jayne:*:1618601212:1618600513:Jayne Cobb:/home/jayne:/bin/bash

Wait. What.

That is a long as... long as f... long user ID. And group ID. Also, where the heck is this user coming from and why is there a home directory there that doesn't appear to really exist?

There are two more files I'm going to check: /etc/passwd- and /etc/shadow-, which are backups of the respective files above. Fortunately / unfortunately there is no indication of jayne. Again.

Time to dig deep.

$  strace getent passwd jayne &> getent.strace

(Protip: You'll definitely want to redirect your output to file. You're welcome.)

strace is a diagnostic tool that will show how what processes are being called when, at least in this case, I run getent passwd jayne. I'm not going to post the whole output here for a variety of reasons, but if you want to see the output for yourself you can easily run the above command and replace jayne with any actual user on your system.

Glancing through the output though, this happens:

{{ snip }}
open("/etc/passwd", O_RDONLY|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=1718, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f3b921a2000
read(3, "root:x:0:0:root:/root:/bin/bash\n"..., 4096) = 1718
read(3, "", 4096)                       = 0
close(3)                                = 0
munmap(0x7f3b921a2000, 4096)            = 0
open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=38848, ...}) = 0
mmap(NULL, 38848, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f3b92199000
close(3)                                = 0
open("/usr/lib64/libnss_sss.so.2", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\0\30\0\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=33440, ...}) = 0
{{ snip }}

Hrm, what's this? Well it look like getent tried /etc/passwd and, failing that, moved onto sssd (the libnss_sss.so.2 line). This points to George using sssd, the System Security Services Daemon, which is a way of managing remote directories / authentication. Basically, it looks like George is using an external mechanism for user accounts on his instance. Let's take a look at his /etc/sssd/sssd.conf file:

[sssd]
domains = ad.example.com
config_file_version = 2
services = nss, pam, ssh

[domain/ad.example.com]
{{snip}}

Ah, so he has linked his Linux instance to our Windows AD server, which is why the user exists, but does so outside the realm of /etc/passwd and /etc/shadow. It also explains the atypically high (for Linux) user and group IDs for Jayne.

Some additional, useful, info to have

What is getent passwd ${USER}?

Straight from the man page:

The getent command displays entries from databases supported by the Name Service Switch libraries, which are configured in /etc/nsswitch.conf.

In somewhat plainer English: gentent looks through a specified text based database, like passwd, for the value provided (${USER} in this case). So when I ran getent passwd jayne it looked through all the password databases for that string.

Note that I said databases, plural. /etc/passwd and /etc/shadow are two text file databases, but those are only local databases. Since getent searches local and remote databases, getent was still able to find jayne even though jayne wasn't created locally.

What is /etc/nsswitch.conf?

Recall that getent uses the Name Service Switch libraries from the man page? Well the NSS is configured in /etc/nsswitch.conf. A relevant snippet from George's /etc/nsswitch.conf file:

# Example:
#passwd:    db files nisplus nis
#shadow:    db files nisplus nis
#group:     db files nisplus nis
passwd:     files sss
shadow:     files sss
group:      files sss
#hosts:     db files nisplus nis dns
hosts:      files dns

# are comment lines, but you can see in this nsswitch.conf file that NSS is checking files first, sss second. That means when I ran getent passwd jayne, that getent first looked in the local file databases, e.g. /etc/passwd and /etc/shadow, before looking for remote databases, e.g. the remote database specified /etc/sssd/sssd.conf.

What does the strace output look like if the user is not using sssd?

Excellent question! The process actually terminates after looking in /etc/passwd, e.g. looking up my user on a different Linux instance:

{{ snip }}
open("/etc/passwd", O_RDONLY|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=1413, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f8cc3fc6000
read(3, "root:x:0:0:root:/root:/bin/bash\n"..., 4096) = 1413
close(3)                                = 0
munmap(0x7f8cc3fc6000, 4096)            = 0
fstat(1, {st_mode=S_IFREG|0644, st_size=4021, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f8cc3fc6000
write(1, "quintessence:x:501:502:Quintesse"..., 65quintessence:x:501:502:Quintessence:/home/quintessence:/bin/bash
) = 65
exit_group(0)                           = ?
+++ exited with 0 +++

As you can see, getent opened /etc/passwd, found the desired user, and closed. And just to further clarify, if the user didn't exist, like iamnotauser:

open("/etc/passwd", O_RDONLY|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=1413, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f8a64b1f000
read(3, "root:x:0:0:root:/root:/bin/bash\n"..., 4096) = 1413
read(3, "", 4096)                       = 0
close(3)                                = 0
munmap(0x7f8a64b1f000, 4096)            = 0
open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=20465, ...}) = 0
mmap(NULL, 20465, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f8a64b1b000
close(3)                                = 0
open("/lib64/tls/x86_64/libnss_sss.so.2", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat("/lib64/tls/x86_64", 0x7ffea7f89360) = -1 ENOENT (No such file or directory)
open("/lib64/tls/libnss_sss.so.2", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat("/lib64/tls", {st_mode=S_IFDIR|0555, st_size=4096, ...}) = 0
open("/lib64/x86_64/libnss_sss.so.2", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat("/lib64/x86_64", 0x7ffea7f89360)   = -1 ENOENT (No such file or directory)
open("/lib64/libnss_sss.so.2", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat("/lib64", {st_mode=S_IFDIR|0555, st_size=12288, ...}) = 0
open("/usr/lib64/tls/x86_64/libnss_sss.so.2", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat("/usr/lib64/tls/x86_64", 0x7ffea7f89360) = -1 ENOENT (No such file or directory)
open("/usr/lib64/tls/libnss_sss.so.2", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat("/usr/lib64/tls", {st_mode=S_IFDIR|0555, st_size=4096, ...}) = 0
open("/usr/lib64/x86_64/libnss_sss.so.2", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat("/usr/lib64/x86_64", 0x7ffea7f89360) = -1 ENOENT (No such file or directory)
open("/usr/lib64/libnss_sss.so.2", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat("/usr/lib64", {st_mode=S_IFDIR|0555, st_size=20480, ...}) = 0
munmap(0x7f8a64b1b000, 20465)           = 0
exit_group(2)                           = ?
+++ exited with 2 +++

The fastest way to see the user doesn't exist is by noting that the exit code is non-zero, but to further investigate if sssd is present you can that when getent tries to open the sssd socket that a No such file or directory error is returned. No sssd on this instance :)

Documented on my frequently used assets page.