Difference between revisions of "SSH - loose notes"

From Helpful
Jump to: navigation, search
m (Having to keep a shell open from a computer that isn't yours)
m (Both of the above)
Line 574: Line 574:
For most of these, see {{inlinecode|man ssh_config}} and {{inlinecode|man sshd_config}}
For most of these, see {{inlinecode|man ssh_config}} and {{inlinecode|man sshd_config}}
===Both of the above===
The problem in the combination is that locking a terminal makes it silent.
Actually, the example above,
ssh user@example.com "vmstat 30"
more or less does what you want: Once you break that command, the SSH connection is also torn down.
The command itself won't allow anything, and it sends a packet every now and then.
===Keys not being picked up===
===Keys not being picked up===

Revision as of 17:18, 9 October 2021

Security related stuff.


Theory / unsorted

how to do a login system badly
how to do encryption badly
Disk and file encryption notes
This article/section is a stub — probably a pile of half-sorted notes, is not well-checked so may have incorrect bits. (Feel free to ignore, fix, or tell me)

Auth in scripts

On speed

On bandwidth, and high-bandwidth networks

Single-connection speed limit

Various SSH implementations have historically been too conservative about the size of some internal buffers - the effect of which is essentially the same as having a small TCP window size: only so many packets can be in flight, so on links with high latency you see single connections being slow even if the link itself can go much faster.

This is a relatively simple BDP (Bandwidth-Delay Product) thing, and as such e.g. doesn't matter on sub-millisecond-latency LANs, but very much does matter on non-local, dozen-millisecond-or-more links like much of the internet, where you may get just 2MByte/s on a link that bandwidthwise can easily do ten times that.

Since this is a per-connection issue, and usually only bothersome when transferring files, one workaround is using multiple copy commands and therefore multiple connections. The necessary data split isn't always easy to do, though.

Encryption load

Encryption takes CPU, and most encryption is single-threaded for simplicity and security. You are likely to max out a core before maxing out a single network link.

This is most noticeable in high-throughput, low-latency network links, such as ≥gigabit LAN.

Again, multiple copies solve this, now because multiple independent transfers will be scheduled to different cores.

Note that when using ssh for interoperable convenience more than for strongest security, you can just use a simpler, faster cipher for a particular connection.

For example, when I was testing on a host-to-host gBit line:

  • ...where I can get 116MB/s from an unencrypted TCP transfer
  • ...I get ~34MB/s using scp with its default (apparently aes128-cbc)
  • ...I get ~54MB/s when using
    -c arcfour
  • ...I get ~95MB/s peaks with two arcfour transfers peaks
  • ...I get ~95MB/s consistently with three arcfour transfers

People having run some tests at some time seem to prefer arcfour. Don't treat this as a constant - in particular, CPUs with AES instructions (various post-2010, see [1]) will make in particular AES-GCM options faster.

Note that you can prefer different ciphers via config, see #Ciphers

SSH updates

Apparently OpenSSH improved the buffer size part of this around version 4.7 [2] (~2007). Note that both ends need to be updated.

Further optimizations were experimented with, see e.g. HPN. It can offer more speed in many situations, but requires both sides to be patched. Changes:

  • has larger buffers. This is often the largest improvement in higher-BDP situations
  • has multi-threaded cipher implementations, so you can use more than one CPU core for a single connection
  • allows disabling encryption (without disabling integrity checks)


When link speed or encryption overhead is your bottleneck, and your data is compressible, compression is quite useful.

When link as fast ask your disks (single disk + gigabit LAN) or your data is not very compressible, compression only slows things down a little, due to overhead at both ends. The effect isn't huge, but is noticeable.

To control per connection:

  • -C
    means 'compress everything'
  • -o Compression=no
    -o Compression=delayed

You can set it in config, but may not want to.

On setup latency

This article/section is a stub — probably a pile of half-sorted notes, is not well-checked so may have incorrect bits. (Feel free to ignore, fix, or tell me)

Tips and tricks

Go to directory on remote host

- I've worked on a cluster, where you sometimes want to go to a node or a fileserver -- and would like to start in the same (network-mounted) directory you were in before.

I came up with:

function fileserver  {
  local gotodir=`pwd`
  ssh -Y -t fileserver "cd $gotodir ; bash -l"
  # Actually, I use the following to attempt to minimize IO stolen from our NFS server
  #ssh -Y -t fileserver "cd $gotodir ; ionice -c3 bash -l"

Copying through tar and ssh

This article/section is a stub — probably a pile of half-sorted notes, is not well-checked so may have incorrect bits. (Feel free to ignore, fix, or tell me)

That is, (ab)using ssh's communication of stdio for a data pipe,and using it to carry tar data (instead of running a shell).

Doing this instead of using file-aware copying can be

  • faster when you deal with many small files, largely because a continuous TCP stream will spend less time in overhead than many individual transfers (scp, sftp)
  • more convenient (and less IO) because you stream the archive out as you create it, instead of storing it on disk first


tar czf - sourcedir/ | ssh user@example.com "tar xzf - -C ~/gohere"
tar czf - sourcedir/ | ssh user@example.com "(cd ~/gohere; tar xzf - )"

In other words: Compress to stdout, pipe that to ssh, and on the other end make ssh hand this pipe to a tar that decompresses from its stdin -- in a particular directory.


  • You want to watch the directory structure you're packing, and where exactly it will go on the remote end.
  • -C is 'change to this directory', so the above two examples are basically the same.
  • In some cases it makes sense to add a mkdir, but it may be easier to make the created directory part of the tar, doing the work on the sending side
  • This example uses compression via tar.
    • z for gzip. You might want j for bzip2, but only when the tars on both ends support it
    • for uncompressable data, compression will just make things slower
    • if you use ssh with compression anyway, you probably don't want tar to compress too
  • you can preserve ownership information (to the degree it is communicable)
    • use -p on the receiving end's tar

Other config notes

On security

Check whether people brute-force you

To see the brute force attempts, look at your sshd logs.

A vague attempt at shell-fu that does automatic summaries: (location of your logs can vary. It may e.g. be /var/log/auth.log and you may need to consider log rotation)

Amount per source IP:

cat /var/log/sshd/* /var/log/auth /var/log/auth.log | grep Invalid | \
 sed -r 's/([^\ ]+ [^\ ]+) .*(nvalid user|for) (.*) from ([0-9.]+) port.*/\1\t\3\t\4/' | \
 cut -f 3 | sort | uniq -c | sort -n | less

Amount per day:

cat /var/log/sshd/* /var/log/auth /var/log/auth.log | grep Invalid | \
 sed -r 's/([^\ ]+ [^\ ]+) .*(nvalid user|for) (.*) from ([0-9.]+) port.*/\1\t\3\t\4/' | \
 cut -f 1 | sort | uniq -c | sort -n | less

The usernames they try:

cat /var/log/sshd/* /var/log/auth /var/log/auth.log | grep Invalid | \
 sed -r 's/([^\ ]+ [^\ ]+) .*(nvalid user|for) (.*) from ([0-9.]+) port.*/\1\t\3\t\4/' | \
 cut -f 2 | sort | uniq -c | sort -n | less

The source IPs of accepted logins, to see if there are any that may not be you:

cat /var/log/sshd/* /var/log/auth /var/log/auth.log | \
 egrep 'Accepted (keyboar|publi|pass)[^\ ]+ for' | \
 sed -r 's/(.*from[\ ])([0-9.]+)([\ ]port.*)/\2'/ | sort | uniq -c | sort -n

The same IPs, without count, but with hostname lookup:

cat /var/log/sshd/* /var/log/auth /var/log/auth.log | \
 egrep 'Accepted (keyboar|publi|pass)[^\ ]+ for' | \
 sed -r 's/(.*from[\ ])([0-9.]+)([\ ]port.*)/\2'/ | sort | uniq | xargs -n 1 host

Preventive policies

At network level

You can try a firewall with supports simple state, such as iptables with recent - you can use that to create an automatic blacklist for those that open many new connections to a port in a very short period of time.

Not very effective against botnet brute-forcing.


Things like denyhosts and fail2ban will look at auth logs for failed remote logins, and block any source which is trying a lot of them (by putting them into hosts.deny or iptables).

Somewhat like the recent thing above, but may be easier to set up, and has longer memory.

Not very effective against botnet brute-forcing.

At SSH level

possible policies include:

Whitelist users with AllowUsers in sshd_config, to avoid access via some random guest accounts you didn't create or really know about (though system accounts tend to be passwordless and homedirless so not valid logins anyway, but on servers it can be easy on admins to create accounts for specific people and/or services.

  • Sysadmin gets bothered for each case
  • Potential users usually need to go through slow official channels, typically discouraging use.

Whitelist hosts with a firewall,

  • People need to bother the sysadmin
  • ...for each IP, meaning you block their ability to work from even slightly unusual places

Don't allow root logins via SSH (often the default)

  • While it doesn't prevent much, it generally makes sense to use a regular account for admin (preferably not called something obvious like 'admin'), and use sudo or sudo su when you want something superuser-done

Allow only key-based logins

  • Bothersome/impossible to log in from unusual places.
  • Only really safe if users understand the importance of the security of their key
  • ...but still, it does block brute forcing.

fail2ban and denyhosts

Fail2ban looks at logs for bad things, and updates the firewall to block.

By default it looks just at ssh (repeated failed login attempts),

Plugins like webserver, mail are disabled.

You can make it look at further services, and make it take additional actions.

See also:

Denyhosts is similar, specific to just SSH.

It allows you to sync with a centralized list of blocked servers.

See also:


In fail2ban you can whitelist in /etc/fail2ban/jail.conf, editing the line like (default section), then restarting (you can also do it online, yes)

ignoreip =

In denyhosts, it seems whitelisting is impossible, and removing things is somwhat manual.

See denyhosts faq: "How can I remove an IP address that DenyHosts blocked?", where WORK_DIR is often /var/lib/denyhosts/

And helpers like https://binfalse.de/2015/02/12/denyhosts-remove/


Dropped idle connections

Some modems, routers, and access points will drop TCP connections that have not sent anything for a few minutes - sometimes shorter (1 or 2 minutes), sometimes much longer (say, an hour).

Modems/routers/APs do this to not have to keep track of ever-increasing forgotten things (in their fixed amount of memory), which makes sense, particularly since for most uses of TCP connections, minutes of absolutely 0 bytes is an eternity and probably a forgotten connection. So this sort of cleanup makes sense.

Yet a SSH login sitting at a prompt is silent, so will be disconnected quickly, which is annoying if you are doing work or admin stuff on it.

To avoid this, you can ensure the connection is never silent for that long.

(There are also a few reasons you might not want overly quick checking)

Manual / quick and dirty

There are hackish workarounds that are enough whenever the situation is an exception rather than the rule.

Like running a talkative command, such as
watch -n 15 date

More structural

For permanently configured, SSH-based solutions, see the rest of this section:

SSH v2 can poll via the encrypted channel - and drop the connection itself if it gets no response.

You can configure both sides to do this. Both are disabled by default.

This feature allows a few polls to go without answer, to be flexible about routes that can be expected to be down or congested for a few seconds. They default to 3, which is usually more than enough.

From the server side

The SSH server can ask the client to do this, typically be editing sshd_config to have:

ClientAliveInterval 60
ClientAliveCountMax 3

You want this polling to happen faster than the timeout of the connection-dropping culprit. A value on the order of 60 covers most cases. Slower is often fine too.

From the client side

Each client can ask a server to do this, by using

ServerAliveInterval 60
ServerAliveCountMax 3

in the client config. It's probably easiest to put it under Host * to make it apply to all.

You can also hand it in on the command line, e.g.

ssh -o ServerAliveInterval=15 ssh.example.com


If you tell the sshd config:

TCPKeepAlive yes

...it will occasionally use each connection, at TCP level.

When idle connections have disappeared at the remote end without mentioning it (e.g. if they crashed), this behaviour ensures will be noticed sooner rather than later (or never), and lets us reclaim local resources.

For most of these, see
man ssh_config
man sshd_config

Keys not being picked up

In theory, ssh will pick up identities from your ~/.ssh/ directory.

It may be that the .ssh-related file permissions are not secure enough for ssh's liking.

If you're not comfortable with permission details and want to check whether this is your problem, you can adding
StrictModes No
to your sshd_config and restarting sshd.

If that fixes it, the proper fix is to disable that again, and set your permissions properly.

Non-shell uses of ssh

....such as use of cron.

Consider what user it's running as -- cron, root, the user the crontab belongs to (not all crons have per-user crontabs)

Since you want things to work without interaction you will need to set up a keypair login, and either have it be phraseless, or perhaps use an agent, to get a one-interaction-per-bootup deal.

Shell-less use of SSH seems to not pick up the keys from the user's ~/.ssh/ directory (why? when?), in which case it will probably fall back to interactive login and fail because there is no terminal. If you suspect this, you can check it using ssh -v in the same place.

You can explicitly hand in a key via

Slow login

This article/section is a stub — probably a pile of half-sorted notes, is not well-checked so may have incorrect bits. (Feel free to ignore, fix, or tell me)

Logging in with a -v (or -vv or even -vvv) may help explain what it's doing. You may e.g. discover that

  • making config more specific to hosts helps
  • GSSAPI is a good idea when you need it - but when not used, or not correctly set up, it may stall while adding nothing
so if you don't use it for a specific host, or at all,
then tell your client GSSAPIAuthentication no on specific hosts (or even all)

Slow to ask for password (i.e. time between ssh command and getting a login prompt)

  • misconfigured DNS on client side (meaning slow connect at IP level to SSH server)
To check, see if the name resolves quickly (and keep in mind caches may not make that test repeatable. It randomly being slow and fast may be a hint towards this issue for the same reason)
It can help to see what part is parses on in a ssh -vvv
  • misconfigured IPv6
To avoid IPv6, use
AddressFamily inet
in config, or
on a command

Slow to show prompt (i.e. between entereing passphrase and having the shell load)

  • If only sometimes, the server may be heavily loaded.
  • initial login needing work, e.g.
quota on NFS


  • SSH daemon doing reverse DNS lookups
it's used for
hostname-based host auth (and not even key-based host auth)
matchin hostnames in from= entries in authorized_keys
basically, unless your security guys / you know why you need this, you don't need this.
also, most clients will not have reverse DNS entry
so usually you might as well disable this with UseDNS no
may be enabled or disabled by default on your install (and changed in openssh versions)

Immediate session close

I had a case where I was logging in successfully, but was immediately disconnected. The client side saw:

session closed for user username

and increasing client verbosity didn't add anything.

The problem was that the account I was logging into has /bin/false for a shell (/bin/false was the default for newly adduser'd accounts), which ssh ran without error, but which of course meant immediate termination.


"wrong authorisation protocol attempted"

In my case, my ~/.Xauthority file had somehow become owned by root.

The simplest fix is to
rm ~/.Xauthorit*
and log in again.


Moderately generic (OpenSSH?) error, frequently related to about PuTTY and key exchange (possibly not understanding the offering of newer keys?).

A newer PuTTY should fix it. Tweaking key exchange details also seems to work.

refused connect from IP

This seems to be sshd adhering to hosts.deny (e.g. due to denyhosts / fail2ban).

Connection closed by IP [preauth]

...in server logs. Seems to mean the client realized it has run out of authentication options. (verify)

e.g. seen when auth methods require a password/passphrase but are non-interactive, or where a keypair is rejected.

Unable to negotiate with hostname port 22: no matching cipher found

There is no match between the cipher the server offers and the client allows

Can be caused by:

  • updates disabling weaker ciphers (usually only a problem if one side haven't been updated in years)
  • You've configured specific ciphers via the Cipher configuration
in my case I had tried to force faster ciphers, which were since disabled

"protocol version mismatch -- is your shell clean?"

This article/section is a stub — probably a pile of half-sorted notes, is not well-checked so may have incorrect bits. (Feel free to ignore, fix, or tell me)

Can mean:

  • attempting to rsync over a shell-restricted SSH -- because it can't run rsyncd on it

"Too many authentication failures for username"

This article/section is a stub — probably a pile of half-sorted notes, is not well-checked so may have incorrect bits. (Feel free to ignore, fix, or tell me)

MaxAuthTries was exceeded, within a single login attempt.

In practice a common cause is offering many keys before offering the one that actually applies.

IdentitiesOnly yes
  • ...on the CLI means only use the ones I specified here, none from config or agent (verify)
you may actually need it when using -i
  • ...in the config file means only use the ones in the relevant host section, and none from agent (verify)

Not using it at all means all identities from agent and config (even if in host sections}} will be sent, without much control of the order (beyond that agent entries go first).

So yes, if you use keys a lot, you will often want IdentitesOnly.

If you e.g. have some general identities and it'd run some hosts out of attempts to go interactive, you may want:

PreferredAuthentications keyboard-interactive,password

See also:

X11 connection rejected because of wrong authentication


  • couldn't write to ~/.Xauthority can e.g. be
bad permissions on homedir
bad permissions on ~/.Xauthority (removing it is easiest)
full disk (or quota)
  • not being the user you log in as (verify)
e.g. a su, because it does not fully switch you to the other user (verify)
  • X11Forwarding not enabled (verify)

rexec line something: Deprecated option something

These are complains about your sshd_config

In theory you should go through them and see if your configuration actually does something different now

...but in most cases you can often just remove them - because most of these will have appeared around 2016, when openssh dropped SSH1 support, and all options that only applied to SSH1. (apparently including KeyRegenerationInterval, ServerKeyBits, RSAAuthentication, RhostsRSAAuthentication, and UsePrivilegeSeparation) (verify)




SFTP is a file transfer protocol designed as an extension of SSH.

It's often convenient exactly because it inherits encryption and authentication from an existing SSH setup.

(In theory it could be used outside of SSH, though that's rare because it only makes sense over other sorts of encryption (might as well use FTP if you don't care about security), and most SFTP clients assume it's over SSH in the first place)

You may care to know that OpenSSH can itself do a chrooted file transfer - see SSH_jail#SFTP_copying_only

SFTP is not FTP over SSH, and is unrelated to SFTP (plain FTP over TLS/SSL).

SFTP has some more features than SCP (Secure Copy, similarly an extension on SSH), including more filesystem operations (list, remove) and download resuming. Enough that there is software to mount SFTP as if it were a local drive.


man ssh_config
for the Ciphers entry.

The cipher that gets used is negotiatd, because it depends on which are supported and enabled, locally and remotely.

Which will change over time. For example, openssh recently started to consider most -cbc deprecated in favour of -ctr variants, as well as arcfour and blowfish

While years ago arcfour was the fastest due to simplicity, AES acceleration in CPUs (in particular in servers) should blow it out of the water.

Listing SSH connections

The simplest estimation is probably noticing entries with IPs/hostnames in:


Or perhaps:

last | grep 'still logged in'

If you want to see port numbers and tunnels and such, look at: (note difference between ssh and sshd entries)

sudo lsof -i -n | egrep 'ssh'

See also