OpsFire: The case of the HTML PEM file

by Quintessence

A tale as old as time: always check your assumptions.

I was contacted a couple weeks ago by a team member who said they could not ssh into an EC2 instance using what should have been the appropriate key, which I'll refer to as aws.pem. I asked them to try and ssh into the instance and send me the output, so they sent me this:

$ ssh -i aws.pem georgeparley@dev.example.com
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@         WARNING: UNPROTECTED PRIVATE KEY FILE!          @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
Permissions 0644 for 'aws.pem' are too open.
It is required that your private key files are NOT accessible by others.
This private key will be ignored.
Load key "aws.pem": bad permissions
georgeparley@dev.example.com: Permission denied (publickey).

Seemed easy enough, change the permissions to 0400 and the problem will resolve, right?

Wrong.

$ ssh -i aws.pem georgeparley@dev.example.com
Load key "aws.pem": invalid format
georgeparley@dev.example.com: Permission denied (publickey).

Wait, what?

Since the user was on a Mac, I asked them to send me the contents of their pem file using pbcopy so I could be sure the whole file would be picked up:

cat /path/to/aws.pem | pbcopy

I'll paste the first couple of lines here:

<!DOCTYPE html>
<html lang="en" >

<head>

Well that's a wholly unusual pem key, isn't it? It turns out that the actual contents of what would have been the pem key were buried alllll the way in a lengthy HTML doc. Not sure how that happened. The user claimed they got the key from someone else and it "works for them". 🤔

In any event, I had them rip out the HTML bits and keep only the relevant piece, i.e.:

-----BEGIN RSA PRIVATE KEY-----
{{{{ Key Contents }}}}
-----END RSA PRIVATE KEY-----

That worked, so hooray.

Why am I telling you this now? Well today I was contacted by a different team member who was having trouble sshing into an instance with aws.pem.

Suspicions. Raised.

$ ssh traceysmith@dev.example.com
Load key "/home/traceysmith/.ssh/aws.pem": invalid format
Permission denied (publickey).

I again addressed the permission denied first: indeed the file had 777 and yeah, ssh didn't like that at all. Changed the key to 400 and asked for the first few lines of the file.

$ head -n3 .ssh/aws.pem 
<!DOCTYPE html>
<html lang="en" >

Oh, look, another HTML pem file. What?

I asked him to strip out the HTML bits, and lo the key worked, but he swore up and down that the key was working before. And that he got the key from the same source user, so I decided to search that guy down.

I asked him for the same as above, and yes he had the inception HTML pem file. I asked him if he could ssh to instances and he said yes, so I asked him to try to ssh and send me the output:

→  ssh -i ~/.ssh/aws.pem jaynecobb@dev.example.com
Last login: Mon Jan 29 15:22:10 2018 from █.█.█.█
{{snip}}

Wait. WHAT?!

So then I asked him to append -vv. And behold, the truth will out. As the muggles say.

$ ssh -i ~/.ssh/aws.pem jaynecobb@dev.example.com -vv
Warning: Identity file /Users/jaynecobb/.ssh/aws.pem not accessible: No such file or directory.
OpenSSH_6.9p1, LibreSSL 2.1.8
debug1: Reading configuration data /Users/jaynecobb/.ssh/config
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 21: Applying options for *
debug2: ssh_connect: needpriv 0
debug1: Connecting to dev.example.com [█.█.█.█] port 22.
debug1: Connection established.
debug1: identity file /Users/jaynecobb/.ssh/id_rsa type 1
debug1: key_load_public: No such file or directory
debug1: identity file /Users/jaynecobb/.ssh/id_rsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /Users/jaynecobb/.ssh/id_dsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /Users/jaynecobb/.ssh/id_dsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /Users/jaynecobb/.ssh/id_ecdsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /Users/jaynecobb/.ssh/id_ecdsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /Users/jaynecobb/.ssh/id_ed25519 type -1
debug1: key_load_public: No such file or directory
debug1: identity file /Users/jaynecobb/.ssh/id_ed25519-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_6.9
debug1: Remote protocol version 2.0, remote software version OpenSSH_7.4
debug1: match: OpenSSH_7.4 pat OpenSSH* compat 0x04000000
debug2: fd 3 setting O_NONBLOCK
debug1: Authenticating to dev.example.com:22 as 'jaynecobb'
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug2: kex_parse_kexinit: curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
debug2: kex_parse_kexinit: ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com,ssh-dss-cert-v01@openssh.com,ssh-rsa-cert-v00@openssh.com,ssh-dss-cert-v00@openssh.com,ssh-ed25519,ssh-rsa,ssh-dss
debug2: kex_parse_kexinit: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
debug2: kex_parse_kexinit: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
debug2: kex_parse_kexinit: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1,hmac-md5-etm@openssh.com,hmac-ripemd160-etm@openssh.com,hmac-sha1-96-etm@openssh.com,hmac-md5-96-etm@openssh.com,hmac-md5,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1,hmac-md5-etm@openssh.com,hmac-ripemd160-etm@openssh.com,hmac-sha1-96-etm@openssh.com,hmac-md5-96-etm@openssh.com,hmac-md5,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: none,zlib@openssh.com,zlib
debug2: kex_parse_kexinit: none,zlib@openssh.com,zlib
debug2: kex_parse_kexinit: 
debug2: kex_parse_kexinit: 
debug2: kex_parse_kexinit: first_kex_follows 0 
debug2: kex_parse_kexinit: reserved 0 
debug2: kex_parse_kexinit: curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha256,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
debug2: kex_parse_kexinit: ssh-rsa,rsa-sha2-512,rsa-sha2-256,ssh-dss,ecdsa-sha2-nistp256,ssh-ed25519
debug2: kex_parse_kexinit: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-cbc,aes192-cbc,aes256-cbc,blowfish-cbc,cast128-cbc,3des-cbc
debug2: kex_parse_kexinit: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-cbc,aes192-cbc,aes256-cbc,blowfish-cbc,cast128-cbc,3des-cbc
debug2: kex_parse_kexinit: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: kex_parse_kexinit: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: kex_parse_kexinit: none,zlib@openssh.com
debug2: kex_parse_kexinit: none,zlib@openssh.com
debug2: kex_parse_kexinit: 
debug2: kex_parse_kexinit: 
debug2: kex_parse_kexinit: first_kex_follows 0 
debug2: kex_parse_kexinit: reserved 0 
debug1: kex: server->client chacha20-poly1305@openssh.com <implicit> none
debug1: kex: client->server chacha20-poly1305@openssh.com <implicit> none
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: ecdsa-sha2-nistp256 SHA256:███████████████████████████████████████████
debug1: Host 'dev.example.com' is known and matches the ECDSA host key.
debug1: Found key in /Users/jaynecobb/.ssh/known_hosts:9
debug2: set_newkeys: mode 1
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug2: set_newkeys: mode 0
debug1: SSH2_MSG_NEWKEYS received
debug1: SSH2_MSG_SERVICE_REQUEST sent
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug2: key: /Users/jaynecobb/.ssh/id_rsa (0x████████████),
debug2: key: /Users/jaynecobb/.ssh/id_dsa (0x█),
debug2: key: /Users/jaynecobb/.ssh/id_ecdsa (0x█),
debug2: key: /Users/jaynecobb/.ssh/id_ed25519 (0x█),
debug1: Authentications that can continue: publickey
debug1: Next authentication method: publickey
debug1: Offering RSA public key: /Users/jaynecobb/.ssh/id_rsa
debug2: we sent a publickey packet, wait for reply
debug1: Server accepts key: pkalg ssh-rsa blen 535
debug2: input_userauth_pk_ok: fp SHA256:███████████████████████████████████████████
debug1: Authentication succeeded (publickey).
Authenticated to dev.example.com ([█.█.█.█]:22).
debug1: channel 0: new [client-session]
debug2: channel 0: send open
debug1: Requesting no-more-sessions@openssh.com
debug1: Entering interactive session.
debug1: client_input_global_request: rtype hostkeys-00@openssh.com want_reply 0
debug2: callback start
debug2: fd 3 setting TCP_NODELAY
debug2: client_session2_setup: id 0
debug2: channel 0: request pty-req confirm 1
debug1: Sending environment.
debug1: Sending env LANG = en_US.UTF-8
debug2: channel 0: request env confirm 0
debug2: channel 0: request shell confirm 1
debug2: callback done
debug2: channel 0: open confirm rwindow 0 rmax 32768
debug2: channel_input_status_confirm: type 99 id 0
debug2: PTY allocation request accepted on channel 0
debug2: channel 0: rcvd adjust 2097152
debug2: channel_input_status_confirm: type 99 id 0
debug2: shell request accepted on channel 0
Last login: Mon Jan 29 15:34:04 2018 from █.█.█.█

To recap: aws.pem was not working, in fact he had it in ~/Downloads/aws.pem not ~/.ssh/aws.pem; however, his personal key was on the instance in question, so when aws.pem was rejected ssh picked up his id_rsa key that was hanging out in his keychain and that succeeded. And since the ssh command wasn't run verbosely, it did so silently and appeared to Just Work.

Things never Just Work though, always check your assumptions.

Documented on my frequently used assets page.