Sstrhsrtj

I see my own ssh logins to an AWS EC2 Ubuntu Linux instance as

Nov 18 07:32:21 ip-10-20-30-40 sshd[9487]: Accepted publickey for ubuntu from 15.26.37.48 port 46273 ssh2: RSA SHA256:e37A/qiEdkHHNpeksdPO

and when I run a few commands to seek vulnerabilities

ubuntu@ip-12-34-56-78:~$ grep "Failed password" /var/log/auth.log

ubuntu@ip-12-34-56-78:~$ egrep "Failed|Failure" /var/log/auth.log

ubuntu@ip-12-34-56-78:~$ grep "Failed password" /var/log/auth.log | awk '{print $11}' | uniq -c | sort -nr

ubuntu@ip-12-34-56-78:~$ journalctl _SYSTEMD_UNIT=ssh.service | egrep "Failed|Failure"

ubuntu@ip-12-34-56-78:~$ journalctl _SYSTEMD_UNIT=sshd.service | grep "failure"

nothing comes up. So far so good.

But when I browse the log file I see two suspicious issues. First, I see "authentication failure":

ubuntu@ip-12-34-56-78:~$  grep "authentication failure" /var/log/auth.log



Nov 14 09:23:19 ip-12-34-56-78 sshd[9711]: Disconnecting authenticating user root 98.76.54.32 port 36745: Too many authentication failures [preauth]

Nov 14 09:23:22 ip-12-34-56-78 sshd[9713]: Disconnecting authenticating user root 98.76.54.32 port 36754: Too many authentication failures [preauth]

Nov 16 11:45:29 ip-12-34-56-78 sshd[20710]: Disconnecting invalid user admin 15.48.27.78 port 51289: Too many authentication failures [preauth]

Second, even though I ssh through ssh -i myKeyPair.pem ubuntu@55.89.144.233, lines such as the following appear.

Nov 17 17:17:01 ip-12-34-56-78 CRON[948]: pam_unix(cron:session): session opened for user root by (uid=0)

Nov 17 17:17:01 ip-12-34-56-78 CRON[948]: pam_unix(cron:session): session closed for user root



Nov 17 18:23:25 ip-12-34-56-78 sshd[1004]: Invalid user stuart from 34.55.89.144 port 32946

Nov 17 18:23:25 ip-12-34-56-78 sshd[1004]: Connection closed by invalid user stuart 34.55.89.144 port 32946 [preauth]



Nov 17 07:29:47 ip-12-34-56-78 sshd[32620]: Invalid user ubnt from 21.34.55.89 port 56171

Nov 17 07:29:47 ip-12-34-56-78 sshd[32620]: error: Received disconnect from 21.34.55.89 port 56171:14: Unable to connect using the available authentication methods [preauth]

Nov 17 07:29:47 ip-12-34-56-78 sshd[32620]: Disconnected from invalid user ubnt 21.34.55.89 port 56171 [preauth]



Nov 17 12:39:00 ip-12-34-56-78 sshd[32695]: Did not receive identification string from 233.55.89.13 port 54959



Nov 17 23:51:58 ip-12-34-56-78 sshd[1396]: Received disconnect from 144.233.55.89 port 42056:11: Normal Shutdown, Thank you for playing [preauth]

Nov 17 23:51:58 ip-12-34-56-78 sshd[1396]: Disconnected from invalid user sinusbot 144.233.55.89 port 42056 [preauth]

Nov 17 23:52:24 ip-12-34-56-78 sshd[1398]: Invalid user sinusbot from 144.233.55.89 port 57180

I am unsure about a bunch of issues:

1. What kind of joke is this "Thank you for playing"? That's not an errant game server; is it?


2. There are many users such as "stuart" that look like automated attempts to "get lucky". Is that what this is, just bots?


3. Is there a configuration that will stop these attempts from even appearing or being processed?


4. In the past I used to generate via ssh-keygen a file that I copy/pasted to the remote server, but now I'm using AWS's ssh -i myKeyPair.pem ubuntu@11.22.33.44. Both are equal in safety, since anyone who grabs either the pem file or the private key can get in unhindered. Am I getting this right?


5. 
   Most importantly: I recall I chose public-key ssh as my only authentication method, but I've also dabbled with this and that on the AWS settings. What is a test that I can run to confirm, now and continuously, that neither I nor someone else have opened anything other than public key in ssh?

Short Answer for 1 through 3

For a less cluttered analysis, I find last -i and lastb -i to be indispensable.

• The preauth notation indicates that the login attempt has failed.




• The Thank you for playing notation is a bit of admin humor.




• All failed attempts are logged.




• The "gate" as you put it, is open to the Internet... so, every host in the world has "made it to the gate" unless you configure port knocking.

In general, if public key is the only authentication method for ssh your configuration is fairly unassailable from the scripting bots. However, you can expect several thousand attempts per month, maybe more.

If you are concerned about such attempts, you can install and configure fail2ban.

Longer answer and Welcome to the Matrix

Bots and Low Hanging Fruit

There is probably a moment of panic for most less experienced administrators who begin to watch the log files. I know when I had mine, I started asking many of the same questions you've posted. It's a common response to the seemingly endless flood of hack attempts on your tiny little corner of the Internet.

My tech friends and colleagues assured me that what I was seeing was not a targeted attack and was mostly automated scripts checking for low hanging fruit. And it is....

There are lists upon lists of usernames and passwords that can be bought and sold on various less reputable places. An individual running a scripting bot purchases one or many of these lists and uses it to try to gain access to any machine within a particular subnet or maybe even crawl through the entire IPv4 address space.

That's why you'll see a whole series of a hundred or so names like: Fred, Sally, George ... etc... You may even see your name, or something like it. The same is true for other services you may be running, like email. If you look in the /var/log/mail.log file, you'll see the same sort of username and password combination hack attempts.

These attacks are all bots. If you've configured sane security measures on your server, you are unlikely to be compromised through an ssh hack attempt.

SSH keys

SSH key pairs have a public component and a private component. The private component should never be distributed. The public component should placed in the target server's ~/.ssh/authorized_keys file.

During a public key authentication, neither the private key nor the passphrase for that key are transmitted across the connection. Once authentication has occurred, the ssh session is encrypted using the agreed upon symmetric session key. An attacker who has stolen your private key would still need to figure out your passphrase. And that attacker should not be able to sniff that passphrase without compromising the machine on which your private key is stored.

SSH keys are cryptographic objects and therefore are subject to algorithm sunset dates and/or vulnerabilities as discovered throughout the lifetime of any given algorithm. So, it's important to make sure that your server's software is updated frequently and your user keys checked against any newly identified problems.

The standard algorithm, as of this writing, seems to still be RSA 2048. The AWS key script generates RSA 2048 keys acccording to AWS documentation. However, I've changed all my keys to Curve 25519 since those keys are much shorter and the algorithm is purported to be developed independently from any State agencies or possibly compromised entities.

Server Auditing and Record Keeping

Since you've indicated that you enabled ssh public key authentication in the beginning and also disabled password authentication, there is no reason to suspect that public key authentication has been disabled by a remote attacker. That it is, unless your AWS account itself has been compromised. However, you would probably get several notifications regarding unexpected IP addresses used to access your AWS account.

If you'd like a longer log record, you can extend the length of time log files are kept by logrotate and/or install auditd.

Levels of Sanity for your configuration

Sane ssh security measures for small/personal servers

• Public key authentication and only public key authentication


• Disable ssh access for the root user


• Only enable sudo for users that require root permissions


• Use RSA keys of at least 2048 bits, or Curve 25519 keys


• Never place private key material on the server


• Disable X11 forwarding unless absolutely needed

For most new installations, the only settings that should be changed are enabling public key authentication and disabling X11.

/etc/ssh/shhd_config

...

PermitRootLogin no

...

PubkeyAuthentication yes

PasswordAuthentication no

X11Forwarding no

...

Insane ssh security

In addition to the above, enable port knocking, limit host keys to a single algorithm such as Curve 25519, place your server's host key fingerprint in your domain's DNS record, and secure that record using DNSSEC.

• To limit the server's host keys, simply uncomment the algorithm you will be using and restart the sshd server. Please note, that the host keys will not match once this is done and you attempt to connect again. Follow the directions upon connection attempt to replace the locally stored host key fingerprint.

/etc/ssh/sshd_config

...

#HostKey /etc/ssh/ssh_host_rsa_key

#HostKey /etc/ssh/ssh_host_dsa_key

#HostKey /etc/ssh/ssh_host_ecdsa_key

HostKey /etc/ssh/ssh_host_ed25519_key

...

• To generate an sshfp DNS record for the selected host key:

ssh-keygen -r domain.tld -f /etc/ssh/ssh_host_ed25519_key

The resulting record can be placed in your domain's DNS zone file and ssh connections will check that fingerprint. This protects against Man-In-The-Middle attacks for ssh upon first connection or when the server's host keys are changed.

• Install knockd
  

And configure as per the documentation:

knockd is a port-knock server.  It listens to all traffic on an ethernet (or PPP) interface, looking for special "knock" sequences of port-

hits.  A client makes these port-hits by sending a TCP (or UDP) packet to a port on the server.  This port need not be open -- since knockd

listens  at the link-layer level, it sees all traffic even if it's destined for a closed port.  When the server detects a specific sequence

of port-hits, it runs a command defined in its configuration file.  This can be used to open up holes in a firewall for quick access.



....



[options]

     logfile = /var/log/knockd.log



[openSSH]

     sequence    = 7000,8000,9000

     seq_timeout = 10

     tcpflags    = syn

     command     = /usr/sbin/iptables -A INPUT -s %IP% --dport 22 -j ACCEPT



[closeSSH]

     sequence    = 9000,8000,7000

     seq_timeout = 10

     tcpflags    = syn

     command     = /usr/sbin/iptables -D INPUT -s %IP% --dport 22 -j ACCEPT

• 
  DNSSEC configuration will not be detailed here. However, it's important to remember that DNS spoofing is a real problem which DNSSEC solves.