SSH - loose notes

From Helpful
(Redirected from SSH)
Jump to: navigation, search
Security related stuff.
website security notes · some attack notes · not-so-formal notes · unsorted
This article/section is a stub — probably a pile of half-sorted notes, is not well-checked so may have incorrect bits. (Feel free to ignore, fix, or tell me)

On speed

On bandwidth, and high-bandwidth networks

Single-connection speed limit

Various SSH implementations have historically been dumb about the size of some internal buffers - the effect of which is basically the same as having a small TCP window size: only so many packets can be in flight, so on links with high latency you see single connections being slow even if the link itself can go much faster.

This doesn't matter on sub-millisecond-latency LANs, but very much does matter on non-local, dozen-millisecond-or-more links, where you may get just 2 or 3MByte/s per connection on a link that bandwidthwise could easily do ten times that.

Since this is a per-connection issue, one workaround is using multiple copy commands and therefore multiple connections. The necessary data split isn't always easy to do, though.

SSH updates

Apparently OpenSSH became smarter abour this around version 4.7 [1]. Note that both ends need to be that new to see much benefit.

There are optimised variants of SSH that address the buffer problem and CPU overhead.

In particular, read up on HPN. It can offer more speed in many situations, but requires both sides to be patched. Changes:

  • has larger buffers. This is often the largest improvement in higher-BDP situations
  • has multi-threaded cipher implementations, so you can use more than one CPU core for a single connection
  • allows disabling encryption (without disabling integrity checks)

Encryption load

Encryption takes CPU. You are likely to max out a core before maxing out network throughput. This is most noticeable in high-throughput, low-latency network links, such as ≥gigabit LAN.

Multiple independent transfers will be scheduled to different cores. Also, when using ssh for interoperable convenience more than for strong security, you can just use a simpler, faster cipher for a particular connection.

For example, when I was testing on a host-to-host gBit line:

  • ...where I can get 116MB/s from an unencrypted TCP transfer
  • ...I get ~34MB/s using scp with its default (apparently aes128-cbc)
  • ...I get ~54MB/s when using
    -c arcfour
  • ...I get ~95MB/s peaks with two arcfour transfers peaks
  • ...I get ~95MB/s consistently with three arcfour transfers

The favourite simpler cipher out there is arcfour, as it comes out on top in most benchmarks. Don't treat this as a constant - in particular, CPUs with AES instructions (various post-2010, see [2]) will make in particular AES-GCM options faster.

Note that you can prefer different ciphers via config, see #Ciphers


When link speed is your bottleneck and your data is compressible, compression is quite useful.

When link as fast ask your disks (single disk + gigabit LAN) or your data is not very compressible, compression probably just slows things down due to overhead at both ends. The effect should never be large, but may well be noticeable.

To control per connection:

  • -C
    means 'compress everything'
  • -o Compression=no
    -o Compression=delayed

To set a global default

In /etc/ssh/ssh_config:

Compression no
# yes, no, or delayed
# delayed means "Yes, but only start after authentication",
#           meaning zlib bugs couldn't exploit auth things

CompressionLevel 3
# 1 to 9. Default is (?)

# You could put these in Host sections,
# e.g. disabling it for cluster nodes but enabling it for internet connections

On setup latency

This article/section is a stub — probably a pile of half-sorted notes, is not well-checked so may have incorrect bits. (Feel free to ignore, fix, or tell me)

Tips and tricks

Go to directory on remote host

- I've worked on a cluster, where you sometimes want to go to a node or a fileserver -- and would like to start in the same (network-mounted) directory you were in before.

I came up with:

function fileserver  {
  local gotodir=`pwd`
  ssh -Y -t fileserver "cd $gotodir ; bash -l"
  # Actually, I use the following to attempt to minimize IO stolen from our NFS server
  #ssh -Y -t fileserver "cd $gotodir ; ionice -c3 bash -l"

Copying through tar and ssh

This article/section is a stub — probably a pile of half-sorted notes, is not well-checked so may have incorrect bits. (Feel free to ignore, fix, or tell me)

That is, (ab)using ssh's communication of stdio for a data pipe,and using it to carry tar data.

Doing this instead of using file-aware copying can be

  • more convenient (and less IO) because you stream the archive out as you create it, instead of storing it on disk first
  • faster when you deal with many small files, largely because a continuous TCP stream will spend less time in overhead than many individual transfers.


tar czf - sourcedir/ | ssh "tar xzf - -C ~/gohere"
tar czf - sourcedir/ | ssh "(cd ~/gohere; tar xzf - )"

In other words: Compress to stdout, pipe that to ssh, and on the other end make ssh hand this pipe to a tar that decompresses from its stdin -- in a particular directory.


  • You want to watch the directory structure you're packing, and where exactly it will go on the remote end.
  • -C is 'change to this directory', so the above two examples are basically the same.
  • In some cases it makes sense to add a mkdir, but it may be easier to make the created directory part of the tar, doing the work on the sending side
  • This example uses compression via tar.
    • z for gzip. You might want j for bzip2, but only when the tars on both ends support it
    • for uncompressable data, compression will just make things slower
    • if you use ssh with compression anyway, you probably don't want tar to compress too
  • you can preserve ownership information (to the degree it is communicable)
    • use -p on the receiving end's tar

Other config notes

On security

Check whether people brute-force you

To see the brute force attempts, look at your sshd logs.

A vague attempt at shell-fu that does automatic summaries: (location of your logs can vary. It may e.g. be /var/log/auth.log and you may need to consider log rotation)

Amount per source IP:

cat /var/log/sshd/* /var/log/auth | grep 'Invalid'|rev|cut -d ' ' -f 1 | rev | sort | uniq -c | sort -n | less

Amount per day:

cat /var/log/sshd/* | grep 'Invalid' | tr -s ' ' | cut -d ' ' -f 1-2 | sort | uniq -c | less

The usernames they try:

cat /var/log/sshd/* | grep 'Invalid'|rev|cut -d ' ' -f 3| rev | sort | uniq -c | sort -r -n | less

The source IPs of accepted logins, to see if there are any that may not be you:

egrep 'Accepted (keyboard|publi)[^\ ]+ for' /var/log/sshd/* | \
  sed -r 's/(.*from[\ ])([0-9.]+)([\ ]port.*)/\2'/ | sort | uniq -c | sort -n

The same IPs, without count, but with hostnames:

egrep 'Accepted (keyboard|publi)[^\ ]+ for' /var/log/sshd/* | \
  sed -r 's/(.*from[\ ])([0-9.]+)([\ ]port.*)/\2'/ | sort | uniq | xargs -n 1 host

Preventive policies

At network level

You can try a firewall with supports simple state, such as iptables with recent - you can use that to create an automatic blacklist for those that open many new connections to a port in a very short period of time.

Not very effective against botnet brute-forcing.


Things like denyhosts and fail2ban will look at auth logs for failed remote logins, and block any source which is trying a lot of them (by putting them into hosts.deny or iptables).

Somewhat like the recent thing above, but may be easier to set up, and has longer memory.

Not very effective against botnet brute-forcing.

At SSH level

possible policies include:

Whitelist users with AllowUsers in sshd_config, to avoid access via some random guest accounts you didn't create or really know about (though system accounts tend to be passwordless and homedirless so not valid logins anyway, but on servers it can be easy on admins to create accounts for specific people and/or services.

  • Sysadmin gets bothered for each case
  • Potential users usually need to go through slow official channels, typically discouraging use.

Whitelist hosts with a firewall,

  • People need to bother the sysadmin
  • ...for each IP, meaning you block their ability to work from even slightly unusual places

Don't allow root logins via SSH (often the default)

  • While it doesn't prevent much, it generally makes sense to use a regular account for admin (preferably not called something obvious like 'admin'), and use sudo or sudo su when you want something superuser-done

Allow only key-based logins

  • Bothersome/impossible to log in from unusual places.
  • Only really safe if users understand the importance of the security of their key
  • ...but still, it does block brute forcing.

fail2ban and denyhosts

Fail2ban looks at logs (ssh, mail, web), notices repeated login attempts, and updates the firewall to block It is modular, so you can tell it about other logs, make it do different things (e.g. mail you).

Denyhosts is similar, though more specific to just SSH, and allows you to sync with a centralized list of blocked servers.

See also:

Whitelisting / removing blocks


denyhosts faq: "How can I remove an IP address that DenyHosts blocked?"
where WORK_DIR is something like /var/lib/denyhosts/


  • open /etc/fail2ban/jail.conf, find the [DEFAULT] section and edit(/add) a line like:
ignoreip =
  • restart fail2ban

Alternatively, you an do it online

fail2ban-client set ssh addignoreip
# where ssh is the specific jail; list them via  fail2ban-client status


Dropped idle connections

Some modems, routers, and access points will drop network connections that are silent after a few minutes of not having communicated anything - sometimes shorter (1 or 2 minutes), sometimes much longer (say, an hour).

A SSH login sitting at a prompt is silent, so these connections are candidates for disconnection (at least with default ssh settings). To avoid this, you can ensure the connection is never silent for that long. You can configure SSH to send a packet every now and then.

If you have a laptop that sleeps quickly, or if you have a network that is intermittently bad, then you may actually want less or even no interaction, because the same thing that makes the network stack notice the connection is active will here make it notice it is gone (even when one side sleeps, even when one side has completely dropped off the earth).

Manual / quick and dirty
When the situation is an exception rather than the rule, you can run a talkative command, such as
watch -n 15 date

For permanently configured, SSH-based solutions, see the rest of this section:

SSH v2 can poll via the encrypted channel - and drop the connection itself if it gets no response.

You can configure both sides to do this. Both are disabled by default.

This feature allows a few polls to go without answer, to be flexible about routes that can be expected to be down or congested for a few seconds. They default to 3, which is usually more than enough.

From the server side

The SSH server can ask the client to do this, typically be editing sshd_config to have:

ClientAliveInterval 60
ClientAliveCountMax 3

You wat this polling to happen faster than the timeout of the connection-dropping culprit. A value on the order of 60 covers most cases. Less often is often fine too.

From the client side

Each client can ask a server to do this, by using

ServerAliveInterval 60
ServerAliveCountMax 3

in the client config. It's probably easiest to put it under Host * to make it apply to all.

You can also hand it in on the command line, e.g.

ssh -o ServerAliveInterval=15


If you tell the sshd config:

TCPKeepAlive yes will occasionally use each connection, at TCP level.

When idle connections have disappeared at the remote end without mentioning it (e.g. if they crashed), this behaviour ensures will be noticed sooner rather than later (or never), and lets us reclaim local resources.

For most of these, see
man ssh_config
man sshd_config

Having to keep a shell open from a computer that isn't yours

This article/section is a stub — probably a pile of half-sorted notes, is not well-checked so may have incorrect bits. (Feel free to ignore, fix, or tell me)

...usually for tunnels.

If it's purely temporary, are logging into an account with a password and type that in at the time of connecting,

...then you can use ssh's command argument to run something harmless instead of a shell, for example:

ssh "vmstat 30"

That command will run until broken off (at which the SSH connection is closed), and doesn't allow any input.

...another solution is to log into a shell, and then lock it, for example with vlock, or screen's locking (Ctrl-a x). Doesn't feel quite as safe, but probably good against all but hackers so good enough for a temporary tunnel.

For more generic solutions of restricting what a connection can do (including cases where you use passwordless keypairs), see SSH - SSH jails.

Both of the above

The problem in the combination is that locking a terminal makes it silent.

Actually, the example above,

ssh "vmstat 30"

more or less does what you want: Once you break that command, the SSH connection is also torn down. The command itself won't allow anything, and it sends a packet every now and then.

Keys not being picked up

In theory, ssh will pick up identities from your ~/.ssh/ directory.

It may be that the .ssh-related file permissions are not secure enough for ssh's liking.

If you're not comfortable with permission details and want to check whether this is your problem, you can adding
StrictModes No
to your sshd_config and restarting sshd.

If that fixes it, the proper fix is to disable that again, and set your permissions properly.

Non-shell uses of ssh

....such as use of cron.

Consider what user it's running as -- cron, root, the user the crontab belongs to (not all crons have per-user crontabs)

Since you want things to work without interaction you will need to set up a keypair login, and either have it be phraseless, or perhaps use an agent, to get a one-interaction-per-bootup deal.

Shell-less use of SSH seems to not pick up the keys from the user's ~/.ssh/ directory (why? when?), in which case it will probably fall back to interactive login and fail because there is no terminal. If you suspect this, you can check it using ssh -v in the same place.

You can explicitly hand in a key via

Slow login

This article/section is a stub — probably a pile of half-sorted notes, is not well-checked so may have incorrect bits. (Feel free to ignore, fix, or tell me)

Immediate session close

CLient side saw:

session closed for user username

I was I was logging in successfully, but was immediately disconnected, and client verbosity has nothing interesting to say.

The problem was that the account I was logging into has /bin/false for a shell (/bin/false was the default for newly adduser'd accounts), which ssh ran without error, but which of course meant immediate termination.


"wrong authorisation protocol attempted"

In my case, my ~/.Xauthority file had somehow become owned by root.

The simplest fix is to
rm ~/.Xauthorit*
and log in again.


Moderately generic (OpenSSH?) error, frequently related to about PuTTY and key exchange (possibly not understanding the offering of newer keys?).

A newer PuTTY should fix it. Tweaking key exchange details also seems to work.

refused connect from IP

This seems to be sshd adhering to hosts.deny (e.g. due to denyhosts / fail2ban).

Connection closed by IP [preauth] server logs. Seems to mean the client realized it has run out of authentication options. (verify)

e.g. seen when auth methods require a password/passphrase but are non-interactive, or where a keypair is rejected.

Unable to negotiate with hostname port 22: no matching cipher found

There is no match between the cipher the server offers and the client allows

Can be caused by:

  • updates disabling weaker ciphers (usually only a problem if one side haven't been updated in years)
  • You've configured specific ciphers via the Cipher configuration
in my case I had tried to force faster ciphers, which were since disabled

"protocol version mismatch -- is your shell clean?"

This article/section is a stub — probably a pile of half-sorted notes, is not well-checked so may have incorrect bits. (Feel free to ignore, fix, or tell me)

Can mean:

  • attempting to rsync over a shell-restricted SSH -- because it can't run rsyncd on it

"Too many authentication failures for username"

This article/section is a stub — probably a pile of half-sorted notes, is not well-checked so may have incorrect bits. (Feel free to ignore, fix, or tell me)

MaxAuthTries was exceeded, within a single login attempt.

In practice a common cause is offering many keys before offering the one that actually applies.

IdentitiesOnly yes
  • ...on the CLI means only use the ones I specified here, none from config or agent (verify)
you may actually need it when using -i
  • the config file means only use the ones in the relevant host section, and none from agent (verify)

Not using it at all means all identities from agent and config (even if in host sections}} will be sent, without much control of the order (beyond that agent entries go first).

So yes, if you use keys a lot, you will often want IdentitesOnly.

If you e.g. have some general identities and it'd run some hosts out of attempts to go interactive, you may want:

PreferredAuthentications keyboard-interactive,password

See also:

X11 connection rejected because of wrong authentication


  • couldn't write to ~/.Xauthority can e.g. be
bad permissions on homedir
bad permissions on ~/.Xauthority (removing it is easiest)
full disk (or quota)
  • not being the user you log in as (verify)
e.g. a su, because it does not fully switch you to the other user (verify)
  • X11Forwarding not enabled (verify)



SFTP is a file transfer protocol designed as an extension of SSH.

It's often convenient exactly because it inherits encryption and authentication from an existing SSH setup.

(In theory it could be used outside of SSH, though that's rare because it only makes sense over other sorts of encryption (might as well use FTP if you don't care about security), and most SFTP clients assume it's over SSH in the first place)

You may care to know that OpenSSH can itself do a chrooted file transfer - see SSH_jail#SFTP_copying_only

SFTP is not FTP over SSH, and is unrelated to SFTP (plain FTP over TLS/SSL).

SFTP has some more features than SCP (Secure Copy, similarly an extension on SSH), including more filesystem operations (list, remove) and download resuming. Enough that there is software to mount SFTP as if it were a local drive.


man ssh_config
for the Ciphers entry.

The cipher that gets used is negotiatd, because it depends on which are supported and enabled, locally and remotely.

Which will change over time. For example, openssh recently started to consider most -cbc deprecated in favour of -ctr variants, as well as arcfour and blowfish

While years ago arcfour was the fastest due to simplicity, AES acceleration in CPUs (in particular in servers) should blow it out of the water.

Listing SSH connections

The simplest estimation is probably noticing entries with IPs/hostnames in:


Or perhaps:

last | grep 'still logged in'

If you want to see port numbers and tunnels and such, look at: (note difference between ssh and sshd entries)

sudo lsof -i -n | egrep 'ssh'

See also