SSH notes

From Noah.org
Jump to: navigation, search


SSH Config

See SSH config for information on configuring the ssh client and sshd server.

ERROR: sudo: no tty present and no askpass program specified

Sometimes sudo demand that the invoking process have a TTY. The solution is to always tell SSH to allocate a pseudo TTY to use for automation scripts. I am aware of no instances when having SSH allocate a TTY would cause problems. I am also not aware of any security issues with this configuration. My only philosophical concern with doing this is that it is a blanket fix, and blanket fixes can mask other bugs. Generally I prefer to make accommodations only as needed.

When you see this inlog files:

  + ssh www.example.com sudo /bin/bash -c '"./add_admin_remote' 'remote"'
  sudo: no tty present and no askpass program specified
  Sorry, try again.

Then you need to add "-t -t" (note twice) to your SSH commands, For example:

  ssh -t -t www.example.com 'ls -latr /etc'

Public keys are really public locks

How did the terminology come about that we have public keys' and private keys? It is very non-intuitive, yet analogies with the locks and keys we are already familiar with exist. When trying to explain public and private key pairs to my mother I told her to forget about public keys. That's a misnomer. They should have been called public locks. Imagine I have a padlock with a key. Now it turns out that I can order as many of these locks as I like. I give a few to my mother and tell her to keep them for whenever she wants to mail me priceless artwork or legal documents or anything of a sensitive or valuable nature that she should pack it in a sturdy wooden crate and then lock it with one of the padlocks I have given her. She can then ship me the package and be feel confident nobody can open the package. I get the padlocks for free. I can give away as many padlocks as I like to anybody. I don't have to know them very well. I'm just giving away the locks, not the key. Someone might take the padlock home and secretly try to disassemble the lock to figure out what key would open the lock. If someone could reproduce the key then they could open any package mailed to me. Every lock I had given would be in danger. To protect against this I have very finely made padlocks manufactured such that it is nearly impossible to disassemble, cut open, or even x-ray. I had thin I crystal tubes of acid embedded in the lock. Any attempt to break open the lock will cause the acid tubes to break open and dissolve the internal mechanism of the lock before anyone could figure out how to make key to fit it. I feel pretty safe giving away as many of these locks I like. I can have as many public locks as I like out in the open. I just have to be sure I keep my private key safe. If someone really wanted to open my padlocks they would not bother with figuring out the padlocks. They would try to bypass them by getting a copy of my private key somehow.

Regenerate public key from private key (create id_rsa.pub file from a id_rsa file)

You can regenerate a public key from a private key. This can be handy when transferring credentials from one server to another. You only need to copy the private key. The -y option of ssh-keygen will output the public key of a private key file.

ssh-keygen -y -f ~/.ssh/id_rsa > ~/.ssh/id_rsa.pub

Note that OpenSSH public keys have a comment field. That's the third field which often contains the username or email address of the owner of the public key. Regenerating the public key from a private key cannot recover this comment field. This does not matter for OpenSSH authentication, but some applications use this field, so beware. In the example below the address noah@noah.org is the comment field.

ssh-rsa AAAAB3NzaC1yc2EAXAADL8lc0Y4yP51/z5OPiE8p0ThbfKGZjEuzfJjuAdBlNuw/g6NJf+t+CdGdQKfQAgkcnUbgy5jUkKkrie8RNFKMu3JwuI9gKUgUfj1DfQLJrrqMgvRMlBCo51p/OvqzDisUJ0YbFw5d4wt/czwadnqV109huoLTBbH5AQABAAABAQC1RZeWyzs0GHcoDWrxnWNRQQBYNYDMLkTzANVQrnPoV4Rj9Ffxgib48YWha+ki5Vvwcq0MDmV4jyYlXJ6y3VP3VL6VpobaBq9O1FTkXc5BAUlfknwxFgFOPJENShJwxC4JiD16vG1BHFNSkgESYdBJ66bcwExRPi3gESGU/OFbfdL/1H8p+Z0B noah@noah.org

PEM file

The official way to convert an id_rsa private key file to PEM format is to use openssl

openssl rsa -in ~/.ssh/id_rsa -outform pem > id_rsa.pem

But as far as I can tell this always yields the same result as if you had simply run this:

cat ~/.ssh/id_rsa > id_rsa.pem

or this, for that matter:

cp ~/.ssh/id_rsa id_rsa.pem

Is there more to it than that? I believe the cause of the confusion is that openssl uses PEM format by default. The -outform pem option is intended for use with other formats when the key is not already in PEM format.

print the fingerprint of an SSH public key

ssh-keygen -lf id_rsa.pub

RSA versus DSA keys

There are even newer key algorithms allowed now, but each of these has issues that mean that RSA may still be the best choice for now. It's at least not a bad choice. So as of 2015-05-04, I am sticking with RSA.

Use RSA

  1. There are no current patents for either RSA or DSA.
  2. RSA is the default for OpenSSH.
  3. The default RSA key length of 2048 bits is much longer than the maximum key length of 1024 bits for DSA. Note that 1024 bits for DSA was a specified limitation by the NIST. OpenSSH will actually let you generate DSA keys longer than 1024 bits. Also, the 1024 bit limitation was later dropped. OpenSSH hasn't updated their docs yet.

SSH Chaining

SSH Chaining allows you to create a chain SSH logins through an intermediate gateway. You can do it all in one line if you specify the -t option. The following example connects to a server at 192.168.1.100, which is accessable only from example.com.

ssh -t noah@example.com "ssh noah@192.168.1.100"

SCP Chaining

Copy from local to a target host going through a gateway host.

scp -o ProxyCommand="ssh ${GATEWAY_HOST} -W ${TARGET_HOST}:22" ${LOCAL_DIR}/${FILENAME} ${TARGET_HOST}:${TARGET_DIR}/${FILENAME}

Copy from a source host to a target host going through a gateway host, initiated by a localhost.

scp -o ProxyCommand="ssh ${GATEWAY_HOST} -W ${TARGET_HOST}:22" ${SOURCE_HOST}:${SOURE_DIR}/${FILENAME} ${TARGET_HOST}:${TARGET_DIR}/${FILENAME}

share an existing open SSH session with other SSH tools

This speeds up additional connections to the same host because the client does not have to create a new socket connection and it does not have authenticate to the remote host.

WARNING! There is a dangerous side effect to using this. If you 'exit' from the first shell (master) it will hang until all the other shells have also exited. If you force the shell to exit by killing it with `kill -9` or by closing its xterm window then it will kill every other shell that was using the shared connection.

Add these lines to your local client ~/.ssh/config to enable connection sharing:

Host *
ControlMaster auto
ControlPath ~/.ssh/master-%r@%h:%p

Share X11 desktops

You can share a single mouse and keyboard seamlessly between two X11 servers. You must have `x2x` installed on the remote machine. This example lets you access a monitor located to the left (west) of your main monitor. When you move the mouse to the left edge of the main screen if will show up on the right edge of the remote X11 server.

ssh -f -Y noah@server.example.org x2x -west -to :0.0

You may need to specify the full path to x2x as /usr/bin/x2x or /usr/bin/X11/x2x.

Using `dd` and `ssh` to copy a disk image over a network

dd if=/dev/sdb0 | ssh noah@server.example.org "dd of=ops-tools.img"

Using `ssh` to copy files through a DMZ gateway to a machine behind a firewall

This example copies a directory, pictures/ to a tarball on the remote server.

tar c pictures/ | gzip -c - | ssh user@192.168.1.100 "dd of=pictures.tar.gz"

This makes use of `ssh` chaining to copy a file. This doesn't require a tunnel, but it requires that keys be setup between the DMZ gateway and the server on the remote LAN.

This example copies a file called pictures.tar.gz and requires a key between 208.77.188.166 and 192.168.1.100:

dd if=pictures.tar.gz | ssh user@208.77.188.166 "ssh user@192.168.1.100 \"dd of=pictures.tar.gz\""

You can't use the '-t' option for chaining `ssh` commands because this will cause one of the `ssh` commands to attempt to read the password from the output of `dd`. As far as I can tell there is no way to handle this situation with password authentication. Only public keys work in this situation. For example, this will not work:

dd if=pictures.tar.gz | ssh -t user@208.77.188.166 "ssh user@192.168.1.100 \"dd of=pictures.tar.gz\""

VPN over SSH

OpenSSH versions starting with v4.3 have a built-in VPN feature.

# cat /proc/sys/net/ipv4/ip_forward
0
# echo 1 > /proc/sys/net/ipv4/ip_forward
# cat /proc/sys/net/ipv4/ip_forward
1


remote server sshd_config

Check the settings on the remote server in /etc/ssh/sshd_config. If you change any settings be sure to reload the config settings by running `/etc/init.d/sshd reload`. The sshd_config file should have these settings:

# Allow either layer 2 or layer 3 (point-to-point ''tun'' or ethernet ''tap'').
PermitTunnel yes

remote server bridge setup

# tunctl -b -t tap_vpn_0
tap_vpn_0
# brctl addbr br_vpn_0
# brctl addif br_vpn_0 eth0
# brctl addif br_vpn_0 tap_vpn_0
# brctl show
bridge name     bridge id               STP enabled     interfaces
br_vpn_0        8000.0016366db84d       no              eth0
                                                        tap_vpn_0
# ifconfig tap_vpn_0 up
# ifconfig br_vpn_0 up

local client bridge setup

# tunctl -b -t tap_vpn_c0
tap_vpn_c0
# brctl addbr br_vpn_c0
# brctl addif br_vpn_c0 eth0
# brctl addif br_vpn_c0 tap_vpn_c0
# brctl show
bridge name     bridge id               STP enabled     interfaces
br_vpn_c0       8000.0016cbaa352a       no              eth0
                                                        tap_vpn_c0
# ifconfig tap_vpn_c0 up
# ifconfig br_vpn_c0 up

unfinished notes

Here are some additional notes and a script I found on the topic. Also mentions setting up tun/tap drivers for Mac OS X.

[1-sshvpn]
#!/bin/bash

# # prereqs:
# # remote host's sshd_config must have "PermitRootLogin=no", "AllowUsers user", and "PermitTunnel=yes"
# # "tunctl", in debians it is found in uml-utils, redhats another (dont remember but "yum provides tunctl" must tell)
# # remote user must be able to sudo-as-root
# # can opt by routing as in this case or soft bridge with brctl and you get full remote ethernet segment membership :D
# # that last i think i'll implement later as an option
# # other stuff to do is error checking, etcetc, this is just as came from the oven
# # juako
#
# http://ubuntuforums.org/showthread.php?t=926435&p=7587049#post7587049
#
# Sorry for the thread bump but the "Disabled Privacy Extensions" is probably because you haven't added
#     Tunnel yes
#     TunnelDevice any:any
# to /etc/ssh/ssh_config
#
# http://ubuntuforums.org/showthread.php?t=926435&page=3&p=7886699#post7886699
#
# Ok, I run across so many of these unfinished threads involving OSX and SSH
# interface tunneling (VPNs) that I decided to start suggesting the solution I
# use. For ease of use, I'll avoid describing port and proxy forwarding. Feedback
# is welcome!
#
# Drivers
#
# First, OSX doesn't come with tun/tap drivers. Download them at SourceForge's
# TUN/TAP for OSX project. You'll also need the drivers loaded on any other OS,
# but most UNIX/Linux flavors have them standard.
#
# Privileges
#
# Both the ssh client and the remote login user need enough privileges to
# configure the tun/tap interfaces. This means running ssh as root and loging in
# remotely as root. I will forever be looking for a means around this (setting
# read/write permissions on the tun/tap device files doesn't cut it), but it's
# really not a security threat. Just don't leave a root terminal open and you'll
# be fine.
#
# Server Configuration
#
# You must be able to log in as root. So far, the only user I can get to
# configure tun/tap interfaces is the root user. As far as SSH is concerned,
# PermitRootLogin should default to yes, but make sure it is indeed set to yes.
#

userhost='user@host'
sshflags='-Ap 2020 -i /path/to/some/authkey'
vpn='10.0.0.0/24'
rnet=192.168.40.0/24

# START VPN
if [ "$1" == "start" ]; then
  echo setting up local tap ...
  ltap=$(tunctl -b)
  ifconfig $ltap ${vpn%%?/*}2/${vpn##*/} up

  echo setting remote configuration and enabling root login ...
  rtap="ssh $sshflags $userhost sudo 'bash -c \"rtap=\\\$(tunctl -b); echo \\\$rtap; ifconfig \\\$rtap ${vpn%%?/*}1/${vpn##*/} up; iptables -A FORWARD -i \\\$rtap -j ACCEPT; iptables -A FORWARD -o \\\$rtap -j ACCEPT; iptables -t nat -A POSTROUTING -o eth0 -s ${vpn%%?/*}2 -j SNAT --to \\\$(ip r | grep $rnet | sed \\\"s/^.*src \\\(.*\\\$\\\)/\1/g\\\"); sed -i -e \\\"s/\\\(PermitRootLogin\\\).*\\\$/\1 without-password/g\\\" -e \\\"s/\\\(AllowUsers.*\\\)\\\$/\1 root/g\\\" /etc/ssh/sshd_config; /usr/sbin/sshd -t\"'"
  rtap=$(sh -c "$rtap")

  echo setting up local routes ...
  # since my ISP sucks with transparent filters (i can't opt for another where i live), i'll just use my work net as gateway
  ip r a $(ip r | grep default | sed "s/default/${userhost##*@}/")
  ip r c default via ${vpn%%?/*}1 dev $ltap

  echo bringing up the tunnel and disabling root login ...
  ssh $sshflags -f -w ${ltap##tap}:${rtap##tap} -o Tunnel=ethernet -o ControlMaster=yes -o ControlPath=/root/.ssh/vpn-$userhost-l$ltap-r$rtap root@${userhost##*@} bash -c "\"sed -i -e 's/\(PermitRootLogin\).*\$/\1 no/g' -e 's/\(AllowUsers.*\) root\$/\1/g' /etc/ssh/sshd_config; /usr/sbin/sshd -t\""

  echo connected.
# STOP VPN
elif [ "$1" == "stop" ]; then
  echo searching control socket and determining configuration ...
  controlpath=$(echo /root/.ssh/vpn-$userhost*)
  ltap=${controlpath%%-rtap*} && ltap=tap${ltap##*-ltap}
  rtap=${controlpath##*rtap} && rtap=tap${rtap%%-*}

  echo bringing the tunnel down ...
  ssh $sshflags -o ControlPath=$controlpath -O exit $userhost

  echo restoring local routes ...
  ip r c default $(ip r | grep ${userhost##*@} | sed "s/${userhost##*@}\(.*$\)/\1/g")
  ip r d ${userhost##*@}

  echo restoring remote configuration ...
  sh -c "ssh $sshflags $userhost sudo 'bash -c \"tunctl -d $rtap; iptables -D FORWARD -i $rtap -j ACCEPT; iptables -D FORWARD -o $rtap -j ACCEPT; iptables -t nat -D POSTROUTING -o eth0 -s ${vpn%%?/*}2 -j SNAT --to \$(ip r | grep $rnet | sed \"s/^.*src \(.*\$\)/\1/g\")\"'"

  echo deleting local tap ...
  tunctl -d $ltap

  echo disconnected.
fi

Remote Server Security Enhancement with SSH Keys

You can make port forwarding even more secure by limiting what a privileged account can do. When you add a key to authorized_key you may pass parameters to fine tune the connection. This can be used to restrict what the client is allowed to do. On the remote server, add the following to ~username/.ssh/authorized_keys:

from="192.168.1.69",command="/bin/false",no-pty,no-X11-forwarding,no-agent-forwarding,no-port-forwarding,permitopen="localhost:143"
ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA8XIr8LEXdvc4VZEvNenWkJrerTzNhqTT7QvCD+Y2EjCUPQwfBcSnvhY3oasNigNonghQFqm7/HqWBLpcN+4mqDUrXrEdj6HQmHvCV6WozNUVb5jjiyQ/JF4hqcQd6oelCkVw8wD32I2jlYqydpqOGY4xqakWDAfm3SOx5il3Kl49mKCg5B3GQPexhTujaTT3y/Q1eeT3zGpHE9Mp7k20X8rMxSjp5ncLAmdf42fRh05HY5f1GrupQIEdi0/TDcPNWL1ml89zttrDOLgDnwny7P0x2jmcX41cSxL/8svER7BAk2sroyQe6L21pJ7o2MYz1IwnsQgji/GjJoaA7hTNCQ== username@client.example.com
  • from="192.168.1.69": accept connection only from the given IP address
  • command="/bin/false": forces this command to be run no matter what is passed via ssh from the client
  • no-pty: never allocate a PTY for interactivity
  • no-X11-forwarding: No X11
  • no-agent-forwarding: we don't want or need ssh-agent
  • no-port-forwarding: prevent ssh -R ...
  • permitopen="localhost:143": allow only localhost connections to port 143 for `ssh -L` requests

Turn off remote host key identity checks (ignore known_hosts)
WARNING: this makes your connection less secure

WARNING! This defeats detection of man-in-the-middle attacks.

The short answer... ... for the command-line (will ask for your password if necessary):

ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o LogLevel=ERROR noah@example.org "uptime"

... for scripts (will not ask for a password, or otherwise block waiting for user input).

ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o LogLevel=ERROR -o BatchMode=yes noah@example.org "uptime"

... additionally, for scripts (will timeout quickly to skip past down hosts).

ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o LogLevel=ERROR -o BatchMode=yes -o ConnectTimeout=10 noah@example.org "uptime"

You don't want to be bothered by messages like this one:

The authenticity of host 'www.example.org (192.168.0.2)' can't be established.
RSA key fingerprint is 7c:05:58:a2:fb:ca:b5:2b:e3:37:3c:c4:ff:55:48:b1.
Are you sure you want to continue connecting (yes/no)?

...or this one:

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!

...or this other one, either:

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@       WARNING: POSSIBLE DNS SPOOFING DETECTED!          @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@

There are use cases where you want to connect to a remote host and you feel it is a reasonable security risk to allow the connection even when the host identity key is not set or has changed. This note shows how to automatically ignore the identity checks that the SSH client normally enforces.

One situation where identity checks will make your life miserable is when connecting to a host that is running multiple SSH services on different ports. SSH only stores the remote host's name and identity key in ~/.ssh/known_hosts. It does not store which port number an identity key is associated with. SSH will complain about a remote host's identification changing even though there might be multiple SSH services running on ports other than 22.

Another situation where identity key checks become tiresome is when working with virtual machines that are destroyed and rebuilt frequently for testing. You can cache and restore keys, but that isn't always practical for ad hoc tasks.

There are several options that control host key checks, StrictHostKeyChecking, UserKnownHostsFile, and BatchMode. The StrictHostKeyChecking option controls whether ssh will ask if you want to continue connecting when the authenticity of the host can't be established (a new host not found in known_hosts). It also controls whether ssh will allow connections to a known host where the identity key does not match. By setting this option to no you tell ssh to always connect. This creates a new problem because ssh will still store the key in your known_hosts file, which can dirty this file with lots of useless keys that may never be seen again. The UserKnownHostsFile option tells ssh which known_hosts file to use. By setting this option to /dev/null you tell ssh read and write the host keys in /dev/null, which is as good as not saving them. This will result in the following annoying message for every connection, Warning: Permanently added 'www.example.org,192.168.0.2' (RSA) to the list of known hosts.. This happens because ssh still thinks it's storing the host key in a known hosts file even though that file is /dev/null. You can get rid of this message by setting the LogLevel option to ERROR, which will silence any message with the severity level of "WARNING" or below. The final option you may want to use in unattended scripts is BatchMode, which will prevent SSH from ever blocking for user input, such as asking for a password. Instead of block SSH will exit with an error. This is useful in unattended scripts where blocking forever on user input is worse than exiting with an error.

The following examples show how to connect to a remote host while disabling host identity key checks:

ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o LogLevel=ERROR noah@example.org
ssh -p 2222 -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o LogLevel=ERROR noah@example.org

If you intend to use these settings to script SSH to run remote commands, then you may want to add the BatchMode=yes option. This will disable requests for a password so that the SSH command won't just block, waiting for a password. Instead, the command will fail right away so that your script can more quickly detect that something is misconfigured. For example, this checks the load of a remote server without checking that the host identity key is valid and without asking for your password if it turns out that your id key does not match the one cached on the remote (usually ~/.ssh/authorized_keys).

ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o LogLevel=ERROR -o BatchMode=yes noah@example.org "uptime"

If you want to live dangerously then you can actually put these options in your SSH client config, ~/.ssh/config. You may want to put these options under a Host section so that these options are only applied to specific hosts that match the given IP pattern. In the following example I assume that the 192.168.100.0/24 network is an isolated network where I build and test virtual machines. I never care about confirming the host identity for these VMs.

Host 192.168.100.*
StrictHostKeyChecking no
UserKnownHostsFile /dev/null
LogLevel ERROR

Finally, I didn't tell you to actually do any of this, and I tried to warn you -- don't say I didn't warn you!

scp warning: "stty: standard input: Invalid argument"

While using scp to copy a file to or from a remote server you may see the following warning message:

stty: standard input: Invalid argument

The problem is in the remote host's .bashrc or /etc/bashrc (and other places). When scp connects to the remote host the server on the remote end will source the .bashrc file. Most .bashrc file have some logic at the top to prevent running if the connection does not have a terminal. There are two common variants:

[ ! -t 0 ] && return

The other, more common style;

# If not running interactively, don't do anything
[ -z "$PS1" ] && return

If that logic is not there and the .bashrc attempts any interaction with stdin then you are likely to get a warning message. It seems counterintuitive that scp would invoke the remote's .bashrc.

To fix this you need to add one of those two lines to the remote's .bashrc file. This will get rid of the warnings.

regain terminal control from a locked SSH session

Press Enter then type '~.'. In other words:

<ENTER>~.

SSH for Windows

Putty is the best free SSH client for Windows. You can also use Cygwin, but the ANSI terminal emulation isn't as good unless you start the X11 server and run xterm.

MindTerm SSH client Java Applet

MindTerm_2.1 (non-commercial). This was the last free version of MindTerm.

   mindterm.jar

Put this applet on a web page and point the <applet> "archive" attribute to the URL of the JAR file:

 <applet archive="mindterm.jar" code="com.mindbright.application.MindTerm.class" width="620" height="440"> 
     <param name="te" value="xterm-color"> <!-- "vt102" -->
     <param name="fs" value="18">
     <param name="gm" value="80x32+0+0">
     <param name="port" value="22">
     <param name="cipher" value="blowfish"> <!-- "des" -->
     <param name="usrname" value="">
     <param name="sepframe" value="false">
     <param name="quiet" value="false">
     <param name="cmdsh" value="false">
     <param name="verbose" value="true">
     <pa ram name="autoprops" value="none">
     <param name="idhost" value="false">
     <param name="alive" value="10">
     <param name="appletbg" value="white">
 </applet>

<include iframe src="http://www.noah.org/engineering/mindterm/mindterm_iframe.html" width="660" height="480px" />