SSH - tunnels, X forwarding
Security related stuff.
Securing services
Unsorted - · Anonymization notes · website security notes · integrated security hardware · Glossary · unsorted |
📃 These are primarily notes, intended to be a collection of useful fragments, that will probably never be complete in any sense. |
TCP tunnels
Basic description
SSH tunnels mean that, on top of the SSH connection you're establishing always,
- you make one side listen to a specified port (you choose which side, and which port)
- -L is "once the SSH connection is established, listen from the connecting client's side"
- -R is "once the SSH connection is established, listen at remote/server side"
- when a regular TCP connection is made to that listening port
- it will be carried through that existing SSH connection to the other side...
- ...and from that other side there will be a new, non-secured (!) TCP connection, to the a host and port you also specified.
Why is that useful?
Introduction by example
Service on nearby computer
Say I have a database-exploring GUI on my laptop (pgadmin, dbeaver, etc.), and a database server in the corner of my room that has SSH open but is firewalled to everything else
- on that laptop I might do (e.g. for postgresql, so port 5432):
ssh -L 5432:localhost:5432 dbserver.example.com
- now, when that GUI connects to my laptop's port 5432, it actually gets connected to what on the the SSH server's networking side is localhost:5432 - itself.
- As far as the dbserver is concerned, it's seeing a connection from its localhost. Which also means
- my laptop's doesn't need to be trusted by the server (the database never even sees its IP)
- the database server can stay firewalled to that port, and the database daemon doesn't even need to listen to its LAN interfaces
I sometimes want to remote-desktop to an Windows server at work.
- And have an internet-exposed linux server at work, on the same (V)LAN as that windows server (if you don't, see the next section)
- different to the last mostly in that I now tell that connection not to go to localhost, but to go to another host (private 192.168.something IP)
ssh -L 3389:192.168.1.200:3389 ssh.work.example.com
- note that on the work side, that's still an private network - unroutable beyond that subnet
Considering direction of SSH connection and/or tunnel
I have some monitoring at work (exposed as a web-app) that I sometimes want to check from home, but it's intentionally firewalled beyond that work (V)LAN
- We could use the above, but to get a HTTP connection through with that method requires multiple steps, so is more awkward and fragile
- Instead...
- I have a ssh server on my home broadband, so when I think to plan, I have a simpler option: While I'm still at work, on that work host: (note: 5000 is just the one my app used)
ssh -R 5000:localhost:5000 ssh.home.example.com
- functionally this works out very similar to a home-to-work connection with -L
- in that both end up with a LISTENing port at home
- practically, however, the SSH connection is easier to set up in this direction
Similarly, before going home from work, I've used
ssh -R localhost:2222:localhost:22 ssh.home.example.com
so that once home, I can SSH to my home server's localhost:2222 which actually logs in to the SSH server at work I did that on.
- point being that my home server has SSH open, and work often does not (though it may have a bastion host running a SSH proxy)
- I'd run this ssh in tmux to
- have that SSH and tunnel connection live longer (see notes below)
- and stay connected without me having to leave a graphical terminal open with an shell to my home server
- ...and yes, I'm betting on that connection not dropping on my commute home. There are better ways to deal with that.
- I'd run this ssh in tmux to
Similarly, basically exactly that on a sensor on a respberry pi, installed at a remote site, connecting to my own server
- with extra attention paid to
- keypair logins so that it doesn't need interactivity to establish the connection
- Restricting what command can be run on my server so that's not a security issue
- Automatic re-establishing when disconnects happen, because I couldn't do that manually
Syntax and some security notes
The typical syntax is
listenPort:toHost:toPort
The full form is
listenIP:listenPort:toHost:toPort
and the first was a shorthand for
0.0.0.0:listenPort:toHost:toPort
i.e. to listen on all interfaces.
Security-wise, you might want to get used to
localhost:listenPort:toHost:toPort
so that even if you didn't have firewalling, only people on that SSH host can connect to that tunnel. (that still only helps security when you trust all users on that host, but multitenancy on a single host is less common now, particular when containers get involved).
(More recently, you can also listen and connect to a named socket, by its path, which is similarly host-only, but has very similar security implications in that there is nothing to shield other users on that host from connecting.)
In other cases, you may specifically want e.g. a coworker to use the same tunnel.
Even then, it may be worth noting that "all interfaces" could include things like VPNs, so to keep thinking about what you bind to.
The fact that the tunneled connection does not appear as a separate connection between the hosts is sometimes convenient and more secure, and sometimes convenient and less secure, depending on what exactly you do with it.
The convenience often comes from the fact that each host's firewall can be closed to everything but SSH, and this will still work.
This makes firewalling simpler, means you don't need to bother network admins (which they will probably not want to for good security reasons), you won't need exceptions, won't be able to have forgotten exceptions.
However, keep in mind that if you do it wrong enough (listening port is on a public IP), you might unintentionally create a bit of a backdoor.
There are also some practical footnotes, though, like that you'd have to keep the SSH connection open, which by default implies a shell on the other host. You can work around that, though, see notes below.
Limitations to use
- while the data over the tunnel is encrypted, the extra connection made is not
- this doesn't matter too much if it's a connection to localhost, but can matter if not.
- that connection is TCP (no UDP)
- sending over your encrypted channel isn't very high-bandwidth, and adds a bit of latency
- Forwards a single fixed port at at a time.
- You can do multiple, but it's certainly not as flexible or user-friendly as VPN, and other generic tunneling (...once you've set those up, that is)
- only one of these unsecured connections per port(verify)
- You need to keep the SSH connection open, so need to keep the shell open to keep the tunnel open - or use a way around that (see below)
- tunneled connections drop when SSH connection drops, for any reason, and don't get re-established unless you've set up something to do that
- In some situations this can be prohibitively annoying/fragile
- you probably can't listen to ports under 1024 (unless you're root)
- ...which is intentional, and usually a good thing
As such, it's most useful for quick, one-time, temporary use.
Practicalities to the open SSH connection
Mostly "Keeping the connection going" and "Ensuring nothing can get executed at the remote end"
Avoiding having a shell open on it
If you typed in a SSH command just to set up a tunnel,
the terminal that contains that SSH command now needs to stay open.
Depending on where you did that, this terminal might be found by other people, which may be a security issue for you, in that that is logged into another host.
You can hide the ssh command in the backgrounds somewhere (see e.g. screen/tmux).
Ideally, even if that terminal is found, there is nothing they can do on the other side.
For a one-time thing that lives until the next disconnect
SSH only defaults to a shell. You can tell it to not run anything at all, or run a command like watch date. In either case, a Ctrl-C will drop the connection, not drop to a remote shell.
And if you set up the connection one time, and needed to type in a pass-phrase, another person can't do that later.
(Something like vlock, or screen's locking (Ctrl-a x) may also work against casual abuse - but it's not good security practice to assume such session lockers cannot be defeated)
For more generic solutions of restricting what a connection can do (including cases where you use passwordless keypairs), see SSH - SSH jails.
For automatically established connections
This is potentially worse, in that
- you have some mechanism where people don't need to type in a passphrase (passphraseless key, or and agent), and in some cases that might mean control.
- so the client asking to not run a command isn't good enough, as another person won't ask that.
So if the hosts connects automatically, the safest bet is to have the remote server ensure that no possible client request gets a shell.
It turns out this is possible, probably most easily by tyinga specific command to a specific authorized_keys entry.
This may be a nice way of doing a more permanent tunnel anyway, so see SSH_jails#via_authorized_keys.
And perhaps SSH jails in general.
Trying to avoid disconnects
Firewalls and modems may drop entirely-idle connections. You should probably assume that they will.
For tunnels this is unhandy, as the extra connection will also need to be re-established, so it helps to do your best to never be idle.
The above example of watch date is one way to do that.
A more structural fix is configuring ssh/sshd's keepalive, which ensures it occasionally sends a do-nothing packet.
You can configure either or both sides to do that:
- For the client-side tweak, look for ServerAliveInterval.
- for the server-side configuration, look for ClientAliveInterval
While fewer is handier, you should accept that disconnects will sometimes happen, and you care about...
Automatic re-establishing when disconnects happen anyway
There may be various reasons for the SSH connection to break anyway - and you often want it to reconnect without assistance.
On linux, OSX
Look at autossh, (and/)or abuse your service manager's features.
You'll likely also want a keypair, probably both:
- a passphraseless one so you won't need human interaction at reconnection time
- and one specific for only this tunnel-supporting connection, because you can configure the server to always run a do-nothing command for a specific key (see also ssh jail).
- If you want a tunnel at bootup
Keep in mind that modern service management (systemd, upstart, etc) can watch and respawn things, meaning you don't even really need autossh, but it may be a little easier to still use it
- Otherwise you probably want to look at autossh (or perhaps its inspiration, rstunnel).
Via service management
using systemd
[Unit] Description=Temporary tunnel for remote access After=network-online.target [Service] User=worker ExecStart=/usr/bin/ssh -o "ServerAliveInterval 60" -o "ServerAliveCountMax 3" -o "StrictHostKeyChecking=no" -o "BatchMode=yes" -o "ExitOnForwardFailure=yes" -nNT -i /etc/temp_tunnel_key -R localhost:2221:localhost:22 tunneler@example.com Restart=always RestartSec=1min #StandardInput=tty [Install] WantedBy=multi-user.target
This combination of arguments:
-n stdin is /dev/null -N no remote command -T no pseudo-terminal allocation
...seemed to be easier than trying to get it to allocate a terminal, but TODO: explore this more.
On windows
Look at things like tunnelier.
Or perhaps ssh / autossh via cygwin.
Autossh notes
Autossh can be seen as a command that keeps re-executing ssh when it needs to.
The simplest autossh command is
autossh -M 0 user@example.com
Notes:
- -M (monitoring) is a required argument. When you want no monitoring, use -M 0
- -f tells autossh to sit in the background (forks to init?)
- can be useful if you put this in startup scripts, rather than a service
- options not understood are passed through to the ssh command, so you can add any of your usual options, say,
-L 3306:localhost:3306 -o "ServerAliveInterval 60″ -o "ServerAliveCountMax 3″
See also:
autossh imitation
If you can't install autossh, then a quick-and-dirty imitation of autossh could be based off commands / script like:
while true
do
ssh user@hostname # plus whatever further options you need
sleep 5 # possibly longer if the server has denyhosts/fail2ban
done
Options you may want on a tunnel
Avoid the idle-disconnect problem:
-o "ServerAliveInterval 60″ -o "ServerAliveCountMax 3″
By default, a tunnel failing to establish won't prevent a SSH connection from being made. But if the only point of this connection is that tunnel, then you probably do want it to fail if a tunnel fails:
-o "ExitOnForwardFailure yes"
Use a specific keyfile, rather than relying on things implied by account:
-i SSH_KEY_FILE_PATH
Fail if the login isn't passwordless (that is, if you get a password question, don't hang waiting for timeout - useful for background jobs):
-o BatchMode=yes
SOCKS
channel 2: open failed: administratively prohibited: open failed
Typically means AllowTCPForwarding is not enabled in the server's /etc/sshd_config
If that system isn't yours, then the sysadmin may never have enabled it, or may have specifically disabled it.
To sysadmins: you can conditionally enable this, e.g. for specific users only.
Authentication refused: bad ownership or modes for directory /home/someone
...in your logs, and keypairs not working.
Though relatively overzealous, the following should fix it (but this is a security thing you should probably know a few things about, so read up on the details):
chmod go-w ~/ chmod 700 ~/.ssh chmod 600 ~/.ssh/authorized_keys
Pseudo-terminal will not be allocated because stdin is not a terminal
I mostly run into this when I run ssh from things other than a shell (e.g. ssh/scp from cron, a service, and such), so a (pseudo-)terminal does not apply.
You can either disable or force pty allocation.
Which of the two is handiest depends on your case.
It seems the options for ssh are:
- -T no pty allocation
- -t force pty
- -t -t (-tt is the same) force pty even if there is no local tty
I ran into this in a more interesting case, where I was trying to automate going to a firewalled host in two ssh steps by trying another ssh command as the command to the first like:
ssh firsthost ssh secondhost
This basically means the inner ssh doesn't need (or get) a pty at all. In short, I needed:
ssh -t firsthost ssh secondhost
Error opening terminal: unknown
Causes seem to include
- programs that don't want to work on dumb terminals, like editors or curses.
- Setting TERM will probably work.
- ssh that does not have a (local) terminal to work with (same cause, really, but different fix)
remote port forwarding failed for listen port
...plus the actual port number.
Often seems to mean there is something already listening on that port.
Which may be something else, but for me was usually just another copy of my own SSH trickery,
or a TIME_WAIT thing.
In my case, stopping the autossh service for a few minutes solved the issue.
X11 forwarding
SSH clients such as the linux one can forward X windowing, so that you can get remote X clients to display on your SSH client side (...yeah, the X terminology makes that sentence a little confusing).
Basically, from another *nix host you can do: (or from windows, e.g. with putty and Xming)
ssh -X otherhost.com
roughly amounts to:
- set up a SSH tunnel
- setting the DISPLAY variable in the shell to point to that tunnel
- setting up X authentication for the connection
Now, in that shell, you can run any X client, and it'll come to you.
The quickest test of whether it works is often xclock or xeyes as they're likely to be installed, and lightweight to draw.
If X forwarding was not enabled
For this to work, the ssh server (and client) must have X support, and the server must allow it - which may be off by default for security reasons.
Find the server configuration (probably at /etc/ssh2/sshd_config or /etc/ssh2/sshd2_config) and see if there's a:
AllowX11Forwarding yes
As an admin, you may like to restrict it to a few users, e.g.
Match User myusername X11Forwarding yes AllowTcpForwarding yes
Notes:
- Don't forget to restart sshd.
- SSH may not allow this for root, and even if you coerce it, this may not be ideal security-wise
- ...and you often don't want to su, because you'll get an auth error "MIT-MAGIC-COOKIE-1 data did not match"
- ...so set it up for the user you want the first time
- On security and broken programs
Apparently, X clients (=GUI programs) don't always deal with authentication properly, which means that when X11 does do authentication properly they may break.(verify)
This seems to be the reason that in addition to ssh -X, which adheres to X11 SECURITY extensions, there is also ssh -Y (and its configuration counterpart, ForwardX11Trusted.
-Y is basically an overly trusting version of -X,
- so while more programs work with -Y,
- -Y also means clients are allowed to do more potentially bad things,
- so it's good habit to start with -X and move to -Y only when necessary.
- (also, because a few things have problems with -Y but not -X)
If you don't trust the remote host, don't use -Y.
- ...or, frankly, any X forwarding at all.
See also X_notes#X_Authentication
X11 forwarding request failed on channel 0
possible causes include
- xauth not installed, e.g. because it's a server with a minimal install withouth X's basis
- sshd_config says X11Forwarding off (also check user/group specific rules)
- a specific client doesn't understand the localhost trick, in which case you need(verify)
X11UseLocalhost no
- IPv6 weirdness, in which case try forcing SSH to do IPv4 only with
AddressFamily inet
Some issues are easily diagnosed using -v on the ssh client, you may see a message like
debug1: Remote: No xauth program; cannot forward with spoofing
In other cases not. Even -vvv said nothing useful when my issue was the IPv6ness.
PuTTY X11 proxy: wrong authorisation protocol attempted
I got this when the ssh server's system disk was full, which meant the negotiated secret couldn't be stored in .Xauthority.
There are other reasons.