SSH - tunnels, X forwarding

From Helpful
Jump to navigation Jump to search
Security related stuff.


Linux - PAM notes · SELinux

Securing services


A little more practical


More techincal waffling

Message signing notes · Hashing notes ·
Auth - identity and auth notes
Encryption - Encryption notes · public key encryption notes · data-at-rest encryption ·pre-boot authentication · encrypted connections

Unsorted - · Anonymization notes · website security notes · integrated security hardware · Glossary · unsorted

These are primarily notes
It won't be complete in any sense.
It exists to contain fragments of useful information.


TCP tunnels

Basic description

SSH tunnels mean that, on top of the SSH connection you're establishing always,

  • you make one side listen to a specified port (you choose which side, and which port)
-L is "once the SSH connection is established, listen from the connecting client's side"
-R is "once the SSH connection is established, listen at remote/server side"
  • when a regular TCP connection is made to that listening port
it will be carried through that existing SSH connection to the other side...
...and from that other side there will be a new, non-secured (!) TCP connection, to the a host and port you also specified.


Why is that useful?

Introduction by example

Service on nearby computer

Say I have a database-exploring GUI on my laptop (pgadmin, dbeaver, etc.), and a database server in the corner of my room that has SSH open but is firewalled to everything else

on that laptop I might do (e.g. for postgresql, so port 5432):
ssh -L 5432:localhost:5432 dbserver.example.com
so now, when something (like pgadmin or dBeaver) connects to my laptop's port 5432, it actually gets sent to what on the the SSH server's networking side is localhost:5432
As far as the dbserver is concerned, it's seeing a connection from its localhost. Which also means
my laptop's doesn't need to be trusted by the server (the database never even sees its IP)
the database server can stay tightly firewalled - the database daemon doesn't even need to listen to its LAN interfaces at all for this to work


I sometimes want to remote-desktop to an Windows server at work.

And have an internet-exposed linux server at work, on the same (V)LAN as that windows server (if you don't, see the next section)
different to the last mostly in that I now tell that connection not to go to localhost, but to go to another host (private 192.168.something IP)
ssh -L 3389:192.168.1.200:3389 ssh.work.example.com
note that on the work side, that's still an private network - unroutable beyond that subnet

Considering direction of SSH connection and/or tunnel

I have some monitoring at work (exposed as a web-app) that I sometimes want to check from home, but it's intentionally firewalled beyond that work (V)LAN

We could use the above, but to get a HTTP connection through with that method requires multiple steps, so is more awkward and fragile
Instead...
I have a ssh server on my home broadband, so when I think to plan, I have a simpler option: While I'm still at work, on that work host: (note: 5000 is just the one my app used)
ssh -R 5000:localhost:5000 ssh.home.example.com
functionally this works out very similar to a home-to-work connection with -L
in that both end up with a LISTENing port at home
practically, however, the SSH connection is easier to set up in this direction



Similarly, before going home from work, I've used

ssh -R localhost:2222:localhost:22 ssh.home.example.com

so that once home, I can SSH to my home server's localhost:2222 which actually logs in to the SSH server at work I did that on.

point being that my home server has SSH open, and work often does not (though it may have a bastion host running a SSH proxy)
I'd run this ssh in tmux to
have that SSH and tunnel connection live longer (see notes below)
and stay connected without me having to leave a graphical terminal open with an shell to my home server
...and yes, I'm betting on that connection not dropping on my commute home. There are better ways to deal with that.



Similarly, basically exactly that on a sensor on a respberry pi, installed at a remote site, connecting to my own server

with extra attention paid to
keypair logins so that it doesn't need interactivity to establish the connection
Restricting what command can be run on my server so that's not a security issue
Automatic re-establishing when disconnects happen, because I couldn't do that manually



Syntax and some security notes

The typical syntax is

listenPort:toHost:toPort

The full form is

listenIP:listenPort:toHost:toPort

and the first was a shorthand for

0.0.0.0:listenPort:toHost:toPort

i.e. to listen on all interfaces.


Security-wise, you might want to get used to

localhost:listenPort:toHost:toPort

so that even if you didn't have firewalling, only people on that SSH host can connect to that tunnel. (that still only helps security when you trust all users on that host, but multitenancy on a single host is less common now, particular when containers get involved).

(More recently, you can also listen and connect to a named socket, by its path, which is similarly host-only, but has very similar security implications in that there is nothing to shield other users on that host from connecting.)


In other cases, you may specifically want e.g. a coworker to use the same tunnel.

Even then, it may be worth noting that "all interfaces" could include things like VPNs, so to keep thinking about what you bind to.


The fact that the tunneled connection does not appear as a separate connection between the hosts is sometimes convenient and more secure, and sometimes convenient and less secure, depending on what exactly you do with it.

The convenience often comes from the fact that each host's firewall can be closed to everything but SSH, and this will still work.

This makes firewalling simpler, means you don't need to bother network admins (which they will probably not want to for good security reasons), you won't need exceptions, won't be able to have forgotten exceptions.

However, keep in mind that if you do it wrong enough (listening port is on a public IP), you might unintentionally create a bit of a backdoor.


There are also some practical footnotes, though, like that you'd have to keep the SSH connection open, which by default implies a shell on the other host. You can work around that, though, see notes below.

Limitations to use

  • while the data over the tunnel is encrypted, the extra connection made is not
this doesn't matter too much if it's a connection to localhost, but can matter if not.
  • that connection is TCP (no UDP)
  • sending over your encrypted channel isn't very high-bandwidth, and adds a bit of latency
  • Forwards a single fixed port at at a time.
You can do multiple, but it's certainly not as flexible or user-friendly as VPN, and other generic tunneling (...once you've set those up, that is)
  • only one of these unsecured connections per port(verify)
  • You need to keep the SSH connection open, so need to keep the shell open to keep the tunnel open - or use a way around that (see below)
  • tunneled connections drop when SSH connection drops, for any reason, and don't get re-established unless you've set up something to do that
In some situations this can be prohibitively annoying/fragile
  • you probably can't listen to ports under 1024 (unless you're root)
...which is intentional, and usually a good thing


As such, it's most useful for quick, one-time, temporary use.

Practicalities to the open SSH connection

Mostly "Keeping the connection going" and "Ensuring nothing can get executed at the remote end"


Avoiding having a shell open on it

This article/section is a stub — probably a pile of half-sorted notes and is probably a first version, is not well-checked, so may have incorrect bits. (Feel free to ignore, or tell me)


If you typed in a SSH command just to set up a tunnel, the terminal that contains that SSH command now needs to stay open.


Depending on where you did that, this terminal might be found by other people, which may be a security issue for you, in that that is logged into another host.

You can hide the ssh command in the backgrounds somewhere (see e.g. screen/tmux).

Ideally, even if that terminal is found, there is nothing they can do on the other side.




For a one-time thing that lives until the next disconnect

SSH only defaults to a shell. You can tell it to not run anything at all, or run a command like watch date. In either case, a Ctrl-C will drop the connection, not drop to a remote shell.

And if you set up the connection one time, and needed to type in a pass-phrase, another person can't do that later.

(Something like vlock, or screen's locking (Ctrl-a x) may also work against casual abuse - but it's not good security practice to assume such session lockers cannot be defeated)


For more generic solutions of restricting what a connection can do (including cases where you use passwordless keypairs), see SSH - SSH jails.



For automatically established connections

This is potentially worse, in that

  • you have some mechanism where people don't need to type in a passphrase (passphraseless key, or and agent), and in some cases that might mean control.
  • so the client asking to not run a command isn't good enough, as another person won't ask that.


So if the hosts connects automatically, the safest bet is to have the remote server ensure that no possible client request gets a shell.

It turns out this is possible, probably most easily by tyinga specific command to a specific authorized_keys entry.

This may be a nice way of doing a more permanent tunnel anyway, so see SSH_jails#via_authorized_keys.

And perhaps SSH jails in general.

Fewer disconnects

You should assume that firewalls and modems will drop idle connections.

For tunnels this is unhandy, as the extra connection will also need to be re-established, so it helps to do your best to never be idle.


The above example of watch date is one way to do that.

A more structural fix is configuring ssh/sshd's keepalive, which ensures it occasionally sends a do-nothing packet.

You can configure either or both sides to do that:

For the client-side tweak, look for ServerAliveInterval.
for the server-side configuration, look for ClientAliveInterval


While fewer is handier, you should accept that disconnects will sometimes happen, and you care about...

Automatic re-establishing when disconnects happen anyway

There may be various reasons for the SSH connection to break anyway - and you often want it to reconnect without assistance.


On linux, OSX

Look at autossh, (and/)or abuse your service manager's features.


You'll likely also want a keypair, probably both:

a passphraseless one so you won't need human interaction at reconnection time
and one specific for only this tunnel-supporting connection, because you can configure the server to always run a do-nothing command for a specific key (see also ssh jail).


If you want a tunnel at bootup

Keep in mind that modern service management (systemd, upstart, etc) can watch and respawn things, meaning you don't even really need autossh, but it may be a little easier to still use it

Otherwise you probably want to look at autossh (or perhaps its inspiration, rstunnel).
Via service management
This article/section is a stub — probably a pile of half-sorted notes and is probably a first version, is not well-checked, so may have incorrect bits. (Feel free to ignore, or tell me)


using systemd

[Unit]
Description=Temporary tunnel for remote access
After=network-online.target

[Service]
User=worker
ExecStart=/usr/bin/ssh -o "ServerAliveInterval 60" -o "ServerAliveCountMax 3" -o "StrictHostKeyChecking=no" -o "BatchMode=yes" -o "ExitOnForwardFailure=yes" -nNT -i /etc/temp_tunnel_key -R localhost:2221:localhost:22 tunneler@example.com
Restart=always
RestartSec=1min
#StandardInput=tty

[Install]
WantedBy=multi-user.target

This combination of arguments:

-n  stdin is /dev/null
-N  no remote command
-T  no pseudo-terminal allocation 

...seemed to be easier than trying to get it to allocate a terminal, but TODO: explore this more.



On windows

Look at things like tunnelier.

Or perhaps ssh / autossh via cygwin.


Autossh notes

Autossh can be seen as a command that keeps re-executing ssh when it needs to.


The simplest autossh command is

autossh -M 0 user@example.com

Notes:

  • -M (monitoring) is a required argument. When you want no monitoring, use -M 0
  • -f tells autossh to sit in the background (forks to init?)
can be useful if you put this in startup scripts, rather than a service
  • options not understood are passed through to the ssh command, so you can add any of your usual options, say,
-L 3306:localhost:3306 
-o "ServerAliveInterval 60″ -o "ServerAliveCountMax 3″ 


See also:


autossh imitation

If you can't install autossh, then a quick-and-dirty imitation of autossh could be based off commands / script like:

while true
do
  ssh user@hostname  # plus whatever further options you need
  sleep 5            # possibly longer if the server has denyhosts/fail2ban
done

Options you may want on a tunnel

Avoid the idle-disconnect problem:

-o "ServerAliveInterval 60″
-o "ServerAliveCountMax 3″ 


By default, a tunnel failing to establish won't prevent a SSH connection from being made. But if the only point of this connection is that tunnel, then you probably do want it to fail if a tunnel fails:

-o "ExitOnForwardFailure yes"


Use a specific keyfile, rather than relying on things implied by account:

-i SSH_KEY_FILE_PATH

Fail if the login isn't passwordless (that is, if you get a password question, don't hang waiting for timeout - useful for background jobs):

-o BatchMode=yes

Errors related to TCP tunnels (or general SSH)

channel 2: open failed: administratively prohibited: open failed

Typically means AllowTCPForwarding is not enabled in the server's /etc/sshd_config

If that system isn't yours, then the sysadmin may never have enabled it, or may have specifically disabled it.

To sysadmins: you can conditionally enable this, e.g. for specific users only.


Authentication refused: bad ownership or modes for directory /home/someone

...in your logs, and keypairs not working.


Though relatively overzealous, the following should fix it (but this is a security thing you should probably know a few things about, so read up on the details):

chmod go-w ~/
chmod 700 ~/.ssh
chmod 600 ~/.ssh/authorized_keys

Pseudo-terminal will not be allocated because stdin is not a terminal

This article/section is a stub — probably a pile of half-sorted notes and is probably a first version, is not well-checked, so may have incorrect bits. (Feel free to ignore, or tell me)

I mostly run into this when I run ssh from things other than a shell (e.g. ssh/scp from cron, a service, and such), so a (pseudo-)terminal does not apply.


You can either disable or force pty allocation. Which of the two is handiest depends on your case.

It seems the options for ssh are:

  • -T no pty allocation
  • -t force pty
  • -t -t (-tt is the same) force pty even if there is no local tty


I ran into this in a more interesting case, where I was trying to automate going to a firewalled host in two ssh steps by trying another ssh command as the command to the first like:

ssh firsthost ssh secondhost

This basically means the inner ssh doesn't need (or get) a pty at all. In short, I needed:

ssh -t firsthost ssh secondhost


Error opening terminal: unknown

This article/section is a stub — probably a pile of half-sorted notes and is probably a first version, is not well-checked, so may have incorrect bits. (Feel free to ignore, or tell me)

Causes seem to include

  • programs that don't want to work on dumb terminals, like editors or curses.
Setting TERM will probably work.
  • ssh that does not have a (local) terminal to work with (same cause, really, but different fix)


remote port forwarding failed for listen port

This article/section is a stub — probably a pile of half-sorted notes and is probably a first version, is not well-checked, so may have incorrect bits. (Feel free to ignore, or tell me)

...plus the actual port number.

Often seems to mean there is something already listening on that port.


Which may be something else, but for me was usually just another copy of my own SSH trickery, or a TIME_WAIT thing.

In my case, stopping the autossh service for a few minutes solved the issue.



X11 forwarding

SSH clients such as the linux one can forward X windowing, so that you can get remote X clients to display on your SSH client side (...yeah, the X terminology makes that sentence a little confusing).

Basically, from another *nix host you can do: (or from windows, e.g. with putty and Xming)

ssh -X otherhost.com

roughly amounts to:

set up a SSH tunnel
setting the DISPLAY variable in the shell to point to that tunnel
setting up X authentication for the connection


Now, in that shell, you can run any X client, and it'll come to you. The quickest test of whether it works is often xclock or xeyes as they're likely to be installed, and lightweight to draw.


If X forwarding was not enabled

For this to work, the ssh server (and client) must have X support, and the server must allow it - which may be off by default for security reasons.

Find the server configuration (probably at /etc/ssh2/sshd_config or /etc/ssh2/sshd2_config) and see if there's a:

AllowX11Forwarding        yes


As an admin, you may like to restrict it to a few users, e.g.

Match User myusername
  X11Forwarding yes
  AllowTcpForwarding yes

Notes:

  • Don't forget to restart sshd.
  • SSH may not allow this for root, and even if you coerce it, this may not be ideal security-wise
...and you often don't want to su, because you'll get an auth error "MIT-MAGIC-COOKIE-1 data did not match"
...so set it up for the user you want the first time




On security and broken programs

Apparently, X clients (=GUI programs) don't always deal with authentication properly, which means that when X11 does do authentication properly they may break.(verify)

This seems to be the reason that in addition to ssh -X, which adheres to X11 SECURITY extensions, there is also ssh -Y (and its configuration counterpart, ForwardX11Trusted.

-Y is basically an overly trusting version of -X,

so while more programs work with -Y,
-Y also means clients are allowed to do more potentially bad things,
so it's good habit to start with -X and move to -Y only when necessary.
(also, because a few things have problems with -Y but not -X)

If you don't trust the remote host, don't use -Y.

...or, frankly, any X forwarding at all.


See also X_notes#X_Authentication



Errors related to X11 forwarding

X11 forwarding request failed on channel 0

possible causes include

  • xauth not installed, e.g. because it's a server with a minimal install withouth X's basis
  • sshd_config says X11Forwarding off (also check user/group specific rules)
  • a specific client doesn't understand the localhost trick, in which case you need(verify)
X11UseLocalhost no
  • IPv6 weirdness, in which case try forcing SSH to do IPv4 only with
AddressFamily inet


Some issues are easily diagnosed using -v on the ssh client, you may see a message like

debug1: Remote: No xauth program; cannot forward with spoofing

In other cases not. Even -vvv said nothing useful when my issue was the IPv6ness.

PuTTY X11 proxy: wrong authorisation protocol attempted

I got this when the ssh server's system disk was full, which meant the negotiated secret couldn't be stored in .Xauthority.

There are other reasons.