Fortunately, the GNU Screen program can share that command line session of yours with other users with accounts on the same Unix host. Screen comes with a multi-user mode that allows them to connect to it. Each user can be granted permissions to each window to view, interact, or issue other commands to Screen. Here is a .screenrc file I have used to implement three groups of users: read-only users that can view all windows, read-write users that can also type in all windows, and an administrative user (the account that runs screen in my case) that retains all permissions within Screen.

multiuser on

aclchg READERS +r-wx "#"
aclchg READERS -rwx "?"
aclchg READERS +x colon,license,windows,next,prev,select,other,detach,suspend,redisplay,lastmsg,windowlist,help,stuff

aclchg WRITERS +rw-x "#"
aclchg WRITERS -rwx "?"
aclchg WRITERS +x colon,license,windows,next,prev,select,other,detach,suspend,redisplay,lastmsg,windowlist,help,stuff

aclgrp watcher1 READERS
aclgrp watcher2 READERS
aclgrp trusted1 WRITERS
aclgrp trusted2 WRITERS

aclumask "*"-rwx cosine+rwx READERS+r-wx WRITERS+rw-x "?"-rwx "??"-rwx

In this example, the administrative user is cosine. The read-only users watcher1 and watcher2 are in the READERS group. Likewise, the read-write users trusted1 and trusted2 are in the WRITERS group. Users are placed in groups with the aclgrp command.

The other commands to set this up can be understood with a careful read of Screen’s man page. The short of it is that for READERS we give read access but no write or execute (+r-wx) for all windows (#), then we do not allow any access (-rwx) to any Screen commands. Next we do enable access (+x) to a reasonable set of Screen commands necessary for navigation and comfortable viewing—otherwise Screen would not allow the user to change windows, exit his session, or a few other commands Screen users take for granted. The aclumask command provides a similar level of coverage to windows that are yet to be created. The WRITERS group is given similar permissions, except users in that group are also allowed to write (i.e. type things) into all windows.

If we wanted to create an ADMINS group that had full control of screen (able to create new windows and execute arbitrary screen commands), we could do so by replacing the aclumask command with another that replaces the mention of the cosine user with an ADMINS group, adding a pair of aclchg commands to grant this group access to everything, and then adding some users to the group:

aclchg ADMINS +rwx "#"
aclchg ADMINS +rwx "?"

aclgrp cosine ADMINS
aclgrp admin1 ADMINS
aclgrp admin2 ADMINS

aclumask "*"-rwx ADMINS+rwx READERS+r-wx WRITERS+rw-x "?"-rwx "??"-rwx

If you begin experimenting with these “acl” commands in Screen, do make sure you test your configurations before using them. Screen did not make these commands particularly user-friendly, and they do not always behave how you might think they do. They feel like they were bolted onto the program as an afterthought, and if you have been reading my blog for very long you know how I feel about that. I would not use Screen’s multi-user mode for any high-security application.

My solution is not very complicated or long at 57 characters (9 of them spaces, 6 of them optional), but it is less than the ideal of the shell providing better support for pipe redirection. I would like to see zsh incorporate the Plan 9 redirection syntax to make this easier. I would have suggested it to them, but I feel like such a suggestion should have a patch attached.

Anyway, go ahead and post answers or potential solutions to the puzzle below in the comments. If you do not want to register for an account you can email your answers or questions to cosine at cosine dot org. I will post noteworthy comments sent to my email below.

; {puzzle2000 | tee out.log} |[2] tee err.log
This is standard output
This is standard error
; cat out.log
This is standard output
; cat err.log
This is standard error

As seen above, the puzzle2000 script outputs two lines. It outputs “This is standard output” to standard output and “This is standard error” to standard error. The challenge is repeating the above Plan 9 shell functionality using the Bourne shell instead. Back in 2000 it took me about two weeks of throwing ideas around to work out a solution. This is non-trivial because the Bourne shell’s pipe syntax can only connect the pipe from standard output of the first process to standard input of the second. So can you figure out how to extend this functionality? Give it a try!

Please do not post answers or spoilers in the comments of this post. I have set up an additional post to discuss possible answers and to avoid spoiling it for those that want to work out the challenge. I have also provided a little bit of information about my own solution there, and will eventually update that post with the solution itself. Feel free to post your answers in the comments there (do not read them if you do not want to be spoiled!). Also, if you do not want to register for an account you can email your answers or questions to cosine at cosine dot org, and I will post them if they are good (that is, if they would contribute to a positive discussion) even if they are not correct.

Note that I will delete comments below if they are a spoiler or if they discuss possible answers, so please limit the discussion here to clarification about what the challenge is or other general comments.

To help understand what ends up being sent as command line arguments to a command, we can use a little Perl script I wrote to see exactly what the arguments are and how they were broken up by the shell. It does nothing more that print each argument out on a line of its own, but just that little service is enough to learn the finer points of command line argument processing. I called this script sargs, and here is the code:

#!/usr/bin/env perl
foreach $arg (@ARGV) {
  print("$arg\n");
}

Let us take a look at it in action in a few different circumstances. First with three arguments:

% sargs 1 2 3
1
2
3

Now with the same three arguments combined as one:

% sargs '1 2 3'
1 2 3

We all know that quoting an argument allows us to put spaces and other special characters within them, and with sargs we can see it first hand. With one more simple example, let us see shell wildcards in action:

% sargs /*
/bin
/boot
/cdrom
/dev
/etc
/home
/lib
/lost+found
/mnt
/opt
/proc
/root
/sbin
/service
/sys
/tmp
/usr
/var

This is the evidence that it is the shell, not the command, that interprets the wildcard asterisk (*) and expands it into many arguments, one for each file with a matching name.

Now let us use sargs for something more useful. What happens to command line arguments in a command issued with SSH? SSH mangles the command line when used to issue shell commands. How? We can see exactly how. Three arguments apart are seen as normal:

% ssh localhost sargs 1 2 3
1
2
3

But if I group them inside quotes, something does not work right:

% ssh localhost sargs '1 2 3'
1
2
3

Those were supposed to be one argument! What SSH does is it joins all the command line arguments it receives and passes the string to a shell for interpretation, which in this case includes parsing the one argument as separate arguments. Thus, to keep those arguments together we need to make sure that the quotes are seen by SSH by escaping them:

% ssh localhost sargs "'1 2 3'"
1 2 3

That is what we wanted. Since SSH joins all the arguments together, we can even send them to SSH as separate arguments as long as the quotes are escaped:

% ssh localhost sargs \'1 2 3\'
1 2 3

So next time you have trouble with running commands over SSH remember that the shell gets to see the SSH command and run it and that it is not run directly by the sshd process. Let us take a quick look at sudo to see if it behaves similarly:

% sudo sargs '1 2 3'
1 2 3

There is no reprocessing by the shell here. This is good most of the time when you want to run sudo for a single command because there are no shell double-interpreting surprises. This is bad if you want to run two commands with one invocation of sudo, like this:

% sudo sargs 1 2 '&&' sargs 3 4
1
2
&&
sargs
3
4

Oops! That second command got processed as arguments of the first! Normally it is recommended that you invoke sudo twice and not quote the double-ampersand like so:

% sudo sargs 1 2 && sudo sargs 3 4
1
2
3
4

But sometimes you cannot do that, such as if the first command is sleep and you cannot allow the second sudo command to request your password again because you will be at lunch (sleep 900; shutdown -r now). Sure, you could use at, but that can be messy in its own way for some tasks. So how can we invoke sudo so that it can run two commands?

The answer is you cannot, but you can invoke sudo to run a command that runs two commands. What command might that be? A shell invoked as a command, of course! All of the shells that I know of use the -c flag for this. If you provide a -c flag, the next argument is expanded as a shell command and you can use the special shell characters to break the command up into multiple commands:

% sudo zsh -c 'sargs 1 2 && sargs 3 4'
1
2
3
4

It is not as pretty as one might hope, but it is reasonably easy to do.

I hope that if you had trouble in the past with figuring out how arguments are handled by different programs that you take this chance to learn using sargs. It can really come in handy some day when you are programming that script that you need “an hour ago” but cannot quite figure out why it does not work right. If that happens to you, just whip out sargs and verify that your commands are seeing their arguments exactly as they are supposed to see them, and if not then escape or unescape the offending special characters appropriately!

The Threats

Let us first start with what is wrong with allowing SSH through a firewall. Then we will address each problem one by one. The problems are:

1. Buffer overflow bugs have led to system compromise.
2. Shell access can often be obtained when none is necessary.
3. SSH supports weak forms of authentication—like passwords.
4. SSH allows the forwarding of connections between the client and the server.
5. SSH helps facilitate X11 forwarding, allowing graphical programs to run remotely.

Buffer Overflows

This is your greatest threat to giving up root access to your server, but fortunately there is something you can do to reduce the risk of break-in to negligible levels. That something is to turn on privilege separation. The only SSH daemon that supports privilege separation is OpenSSH and its more recent derivatives. Fortunately, OpenSSH is a full featured SSH server so even if you are not using it, you should be able to switch to it without much difficulty.

Note that despite Sun SSH being a derivative of OpenSSH, Sun has chosen not to implement privilege separation and thus you would need to replace it. I have badgered people at Sun about this. I can tell you they had a reasonable excuse for omitting it from Solaris 9, but they really dropped the ball with Solaris 10. I hope they can do better for Solaris 11. Your mileage may vary on other operating systems, but the free ones generally use OpenSSH by default.

The reason privilege separation is so good is that it causes the majority of the SSH server code to run with no privileges in a chrooted jail. The little bit of SSH that runs with full root privilege is scrutinized much more closely by the OpenBSD team and security researchers, so it is considered very safe. I do not know of any exploitable vulnerabilities ever found in the privileged code after OpenSSH introduced privilege separation a few years ago (though I may have missed one).

To activate it, create an sshd user (the unprivileged account) and a /var/empty directory (the jail). Then set the UsePrivilegeSeparation option to “yes” in sshd_config and restart your server. You can test if privilege separation is active by looking at your process list after you connect. When privilege separation is in use there will always be two extra processes for each SSH connection, one privileged owned by root, and one unprivileged owned by a regular account (authenticated user) or the sshd account (unauthenticated user). Without privilege separation there is always just one extra process per connection.

Since you need to be using OpenSSH or a derivative to use privilege separation, I will assume you are using it for the rest of this post.

Shell Access When None is Necessary

Sometimes you only want to give someone the ability to transfer files via SCP or SFTP, two protocols that use SSH. Traditionally you have to give full access to a shell to allow only file transfers with SSH.

One solution that has been around for quite some time is a program called scponly. You use scponly as the default shell for “file transfer only” accounts, and it disallows command line and command access. This program has previously had a few security problems that has occasionally allowed an authenticated user to “break out” into a shell, but overall has been very good. The maintainers take its security seriously.

Password Authentication

Password authentication is great if you—oh who am I kidding!? Password authentication is bad, and if you are trying to keep things apart by a firewall and there is a good chance that an attacker will start trying out passwords against your open port, then you better have something better than password authentication protecting the accounts.

SSH can be configured to disallow password authentication and require stronger credentials. Just set the PasswordAuthentication option to “no” and those pesky little weak passwords will not be a problem. Of course you will need to set up something else, such as SSH keys, Kerberos, or a challenge-response mechanism. If you are in a high-security situation, you should consider using some form of two-factor authentication.

Support for Forwarding Connections

SSH can be used to forward connections, effectively bridging two networks to a limited extent. If you are scrutinizing the security of an SSH server on the other side of a firewall, you will want to turn this off. Do so by setting AllowTcpForwarding to “no”.

However, perhaps there is one port in particular that you do want to forward with SSH through your firewall. This could be used to encrypt an insecure channel as it travels over an untrusted network, or it can allow an “outsider” to connect to a service inside your firewall. Whatever the reason, you have a means of limiting what is allowed through at the server side. This is done with the PermitOpen option on newer versions of OpenSSH—it can be used to limit what target systems and ports the server allows clients to forward to. Even if you have full control of the client system, it always helps to add a layer of security on the server side, too.

Support for Forwarding X11 Displays

When it was released in the mid-90s, one of the biggest problems facing Unix users was how to run their X11 graphical applications securely. The security of an X program was usually dependent on weak host-level security, and as a result anyone with a login on the system with the X client (the program) could also run an X client to snoop or annoy the user. There was also a cookie-based authentication available, but it was difficult to teach users how to use it.

Along came SSH, and not only did it replace Telnet by providing a secure shell connection, it also provided a means to automatically configure the cookie-based authentication for X11 and forwarded the traffic over the encrypted channel. Good for you when connecting to your company server internally. Bad for you when you do not fully trust a user or an account that needs access to the system through a firewall.

This one is easy to disable, too. Just set the X11Forwarding option to “no”.

Summary

Do not be afraid to allow SSH through your firewall, but take reasonable precautions and know your SSH server. Know how to disable the features you do not need. Get the SSH client to use the strongest form of authentication available. Read the man pages for ssh_config and sshd_config and be familiar with many of their options. Someday, when you need to boost the security of a connection somewhere, you will be glad you did.

For those that have never heard of /usr/bin/yes, it repeatedly outputs the letter ‘y’ followed by a newline. It used to be used to answer ‘y’ to questions posed by programs asking things like “should I proceed?” and “are you sure?” If such commands do not have a mode to force them to be non-interactive and you want them to just perform the task without question, /usr/bin/yes is your tool. Run ‘yes’ and pipe the output to the interrogative command! As always, be careful to not run something that could cause unknown harm when using this technique!

let D=1; while (( D <= 31 )); do
  DD=$(printf "%02d" $D)
  for M in server-01 server-02 server-03 ; do
    gzip -cd $M/weblog.200708$DD.gz | \\
        perl -anle 'print "'$M' $F[0] $F[3]"
            if $F[0] =~ /^10\.1\.2\.[34]$/'
  done
  let D=D+1
done > ~/log_report

There are a few things I want to point out about this mini-script:

1. Iteration is performed on an index value ($D), tested using Kornshell’s double parentheses syntax, and incremented using a let statement.
2. It makes use of a Perl one-liner to rearrange the data on each line and only print that data that is relevant to the task at hand.
3. I send the output of the entire thing to a file by redirecting it after the final “done” statement.

None of these things is really a big deal to me anymore, but I have noticed many people do not think or know you can do these things. I think that if you do realize it then you can come up with some more powerful mini-scripts from your command line.

I noted before that I use the Z shell, a Bourne compatible shell. While the above mini-script will not work in the Bourne shell itself, it should also work in the Kornshell and the Bourne Again shell, though your mileage may vary.

Incrementing an Index

The generic script code for iterating over an increasing numeric index is:

let I=0; while (( I < 10 )); do
  ... your code here ...
  let I=I+1
done

If your shell is a traditional Bourne shell that does not support let-statements and double parentheses notation, then you may do this:

I=0; while [ $I -lt 10 ]; do
  ... your code here ...
  I=`echo $I + 1 | bc`
done

If you have a small set of numbers to iterate over then it is often still faster to just type them all out in a for-statement, but that stops being true when you have a large set of numbers.

Invoking Perl and/or Ruby

This technique should not be alien to someone that writes shell scripts. Even if you never thought to invoke Perl and/or Ruby in your scripts, you have probably used sed and awk. In the above example I could have used awk instead of Perl, but Perl seemed like a better choice at the time. Ruby and Perl have flags that help them be more useful on the command line, and in fact they can both become more powerful replacements for sed and awk. Some flags are:

-n: I call this the awk-mode flag. It causes both Ruby and Perl to loop on each line of the input, putting that line of input into $_.

-p: I call this the sed-mode flag. It causes both Ruby and Perl to loop on each line of input like -n, but also prints the value of $_ at the end of the loop (presumibly you have modified it).

-a: This is often used with -n, though it could also be used with -p. As each line of input is read, it is split and the resulting array stored in @F (Perl) or $F (Ruby). I used this feature in the above mini-script to limit which fields are output to the report I generated.

-l: This flag causes lines output to be terminated with a newline. It also causes lines of data input via -n and -p to be auto-chomped.

-e: The most important flag for one-liners! This causes Perl and Ruby to take their commands from the argument given after -e. It may be specified multiple times to issue more code from the command line.

-i: If you want to edit a file in-place you can use this flag to tell Perl or Ruby to edit each file of input instead of leaving those files untouched. You can also opt to provide an extension to be used for a backup of the input. Note that because of the syntax for the optional extension use, you should check the man page for usage.

Redirecting Output

You can redirect output from any part of a script. Most people focus on redirection of individual commands, but that causes one to have to keep appending data to an output file. One does not need to do that! Just redirect your output after the “done” and you will capture all of the output from all of the commands inside the loop. Spiffy, eh?

Note that I use the Z Shell (zsh), which is a Bourne compatible shell. These examples should also work in the Bourne shell (sh), the Kornshell (ksh), and the Bourne Again shell (bash). They will not work in the C Shell (csh) or the Tenex C Shell (tcsh). Those shells use a different syntax to accomplish the same thing.

If you want to run a program on one remote computer, you can with ssh like so:

workstation% ssh server-01 /usr/local/bin/locate ruby

Now you want to run the same thing on ten hosts (or twenty-five, or perhaps hundreds). You could start by listing them all in a for-loop (note that the ellipses below are not literal—substitute those with the actual system names):

workstation% for M in server-01 server-02 ... server-10; do
workstation for% ssh $M /usr/local/bin/locate ruby
workstation for% done

And there you go, and all done from the command line! You can build things up even higher. Now let us say you put your hosts into a file called HOSTLIST:

workstation% for M in `cat HOSTLIST`; do
workstation for% ssh $M /usr/local/bin/locate ruby
workstation for% done

You can add more logic in the middle of your loop. This mini-script generates a list of systems that do not have their Ruby installed at a particular location:

workstation% for M in `cat HOSTLIST`; do
workstation for% LOCATION=`ssh $M /usr/local/bin/locate ruby`
workstation for% if [ $LOCATION != /usr/local/bin/ruby ]; then
workstation for if% echo "$M: Ruby at non-standard location: $LOCATION"
workstation for if% fi
workstation for% done

Somewhere along the line here you might decide to move this script into a file of its own, but that is up to you to decide. Whether or not I do so depends on how likely it is that I will use the ad hoc script again and how long it is. In the example above, how often do you need to check that a particular program is at a particular path? For me it is not very often, so I end up doing it with this kind of mini-script. Perhaps you are always checking paths as part of regular audits, though, and thus would move the script to its own file to make it available for repeated use. You will figure out the right thing to do in your environment.

Firstly, check your routing tables and your basic network configuration. More than one Unix administrator on a rampage of righteous indignation against their network team has found himself swallowing his own pride when it was discovered that he had errors in his routing table. So check your IP address, your netmask, and your default router first. All good? Now check if you need or have any other routes in your routing table. All good here, too? Excellent! If you are new to Unix and need help finding this information, read the man pages for ifconfig and netstat to get started. I would love to give some examples but it differs very much from Unix variant to Unix variant. In general, though, use “netstat -nr” for viewing the routing table and “ifconfig -a” for seeing all of the network interface configuration information.

Now that you IP configuration is confirmed, let us move on to the connection in question. Firstly, you have to identify the port and protocol of the connection. If you are running a standard protocol, such as HTTP, then you can likely assume port 80, but perhaps you do not know? We have a couple options.

There are a myriad of tools to see what connections and proto-connections exist on the system. I like lsof because I can view the connections for each individual process. So let us say I have that HTTP server I mentioned above running on my Unix server, but I am not sure if it is running on port 80 or some other port. I can start the web server process, note what its process ID is with ps, and then use lsof to view its listening sockets to see what port the clients should connect to.

I am using Mac OS X today, which has a BSD style ps command:

% ps auxw | grep httpd
root     13888   0.0 -0.3    41732   5348  ??  Ss   Tue05PM   0:15.24 /usr/sbin/httpd
www      14549   0.0 -0.0    32636    796  ??  S    Wed09AM   0:00.00 /usr/sbin/httpd

When using a System V style ps I use “ps -ef” instead of “ps auxw”. The process ID here for httpd is 13888 (I could also use 14549 because they share the port, in a sense, but that may not always be the case). So let us run lsof to see what ports are open:

% sudo lsof -Pnp 13888 | egrep 'TCP|UDP'
httpd   13888 root   16u    IPv4 0x76fe710      0t0       TCP *:80 (LISTEN)

It is port 80! And in particular it is TCP port 80. Note that this is opposed to UDP port 80, which is another beast altogether that we will address below. For now let us move on to testing this connection.

Testing TCP Connectivity

The most widespread tool for checking if there is a firewall between two systems blocking a particular TCP port is telnet. It is not the best utility, but it does the job and is ubiquitous; you will find it on almost all Unix and Windows systems, and sometimes even VMS and mainframes.

To use telnet, just get to a command line and type “telnet” followed by the target system and port number:

telnet my-web-server.example.com 80

If all is well and you can reach the server, you should see something like this:

% telnet my-web-server.example.com 80
Trying 192.168.1.10...
Connected to my-web-server.example.com.
Escape character is '^]'.

If the service is down, but there is no firewall intervention, you will see something along the lines of “connection refused” like below:

% telnet my-web-server.example.com 80
Trying 192.168.1.10...
telnet: connect to address 192.168.1.10: Connection refused
telnet: Unable to connect to remote host

But what if there is a firewall in between dropping all the packets? You would see the message attempting to make the connection, and then telnet will seem to just freeze for many seconds. After some time it will report a failure, but if you see it freezing for more than ten seconds then you can terminate the connection attempt with control-C. Here is what it looks like:

% telnet my-web-server.example.com 80
Trying 192.168.1.10...
(At this point telnet will stop and wait for a few minutes.)
telnet: connect to address 192.168.1.10: Operation timed out

Note that even if it looks like a firewall is dropping the packets, there may be any number of other causes:

1. Host-based firewall on source host blocking outbound traffic.
2. Host-based firewall on destination host blocking inbound traffic.
3. Target host is down.
4. Either host has an error in its routing table.

Testing UDP Connectivity

Sometimes applications use UDP instead of TCP for their communications. This can be tricky to handle if trying to determine if you have a network issue between nodes. The reason is that UDP does not require the other host to reply; if you send a packet that is received you see the same thing as if you send a packet that never reaches its destination. Note that if you send a UDP packet that arrives to a system not accepting packets then you do get a different behavior that you can measure, because in that case the system will notify you that it is not accepting packets there.

So first make a little extra effort to ensure both hosts are up and that the routing table works between them. You can do this by probing any TCP port (open or closed) or with an ICMP ping. As long as those packets can get through you know your routing tables are okay (note this does not rule out host-based firewalls as a problem as specific ports may be filtered).

We can try to send a generic packet to the other host with either ping or traceroute—those two are widely available—or netcat if you have it. Netcat is the best if it is available:

% nc -vvuz my-cifs-server.example.com 137
my-cifs-server.example.com [192.168.1.20] 137 (netbios-ns) open
 sent 0, rcvd 0

The means of getting ping or traceroute to send a single UDP packet to a specific port varies per system and is even impossible in some (all?) versions of Windows. Because of these differences, I recommend installing netcat and using it if you need to debug UDP connections (it is also better than telnet for debugging TCP connections).

Since we really only know that we do not have a network issue if the port shows up as “closed” we have to do more research to confirm a network problem. Note that if you have the luxury of being able to shutdown the UDP service on the destination host then you should do so for testing so you can observe the closed state.

The next tool you need to fully debug a broken UDP connection is a packet sniffer. I recommend Wireshark if you are on Windows, or tcpdump if you are on any non-Solaris Unix system. On Solaris, use the built-in “snoop” command. Each of these has a somewhat different usage:

tcpdump -i en1 'host my-cifs-server.example.com and udp and port 137'
snoop -d ce0 'host my-cifs-server.example.com and udp and port 137'
Wireshark is usually run in a GUI, particularly on Windows.

Get your packet sniffer running and start talking to your port. If you do not see your packets going out, you are doing something wrong. Go back and figure out if you are sniffing the wrong interface or have an error in your usage. If you do see your packets going out but none returning, then it is still inconclusive, but at least you know you are sniffing the right thing. If you do see return packets, your connection is open and you can proceed with troubleshooting your application instead of the network.

The key here is to get the other server to send packets back in response to packets sent. If you can do that then you know the connection works. Unfortunately, many servers just ignore stuff they consider garbage, like the packets we were sending with netcat.

In our case, we are trying to connect to a NetBIOS name service running on UDP port 137. Some such servers reply to garbage with garbage (success for us!) and some just drop the packets, leaving us confused. To get these guys stirred up we need to generate some real NetBIOS name service traffic from our workstations or servers. In this case, we would log into our Windows workstation and try to connect to a file share on the server; we could also run the nbtstat command instead. In either case, with our sniffer armed and ready we will hope it is enough to generate a reply. If there is truly a NetBIOS name service running on the remote port, we should see evidence of it in the sniffer output. If not, then it is at this point that we can turn towards the network and in good conscience assume all is well from the server side (because you did check your host-based firewall settings, right?).

Debugging network connectivity issues does not have to be a nightmare. With the right tools, knowledge, and experience doing so you can be a pro at determining if an issue needs resolution in your application, on the OS it is running on, or the network. Be the hero that knows instead of guesses what is happening out there, because the resolution often follows.

However, in the rush to address the needs to secure file transmissions we ended up with multiple protocols. And even among those protocols there are multiple standards regarding how to operate them. To connect a client to a server both sides have to select the same technology, and simply saying “we want/do secure FTP” just leaves implementors in a quandary about which kind of secure FTP.

SFTP — FTP over SSH

SFTP (S before FTP) is the protocol that describes an FTP-like protocol that is secured over an SSH channel. SSH’s standard port is 22, and so SFTP’s standard port is also 22. SFTP is very “firewall friendly” because all communications run over a single connection. On the down side, most implementations of SFTP carry with them a full implementation of SSH, and SSH is traditionally used for shell access. For a typical business connection, allowing shell access is highly undesirable. The most recent versions of SSH servers are beginning to realize an “SFTP only” configuration is a good feature to support, so this situation is fast improving.

FTPS — FTP over SSL

FTPS (S after FTP) is a different protocol entirely. FTPS is a super-set of the same FTP protocol that the Internet has relied upon for years and years, but it allows for encryption of the connection over an SSL or TLS encrypted socket. This protocol is harder to allow through firewalls because it follows the same process of opening additional connections from server to client or from client to server as the unencrypted version of the FTP protocol.

Implicit FTPS

To further complicate things a little, the FTPS protocol can be run in an “implicit” mode or an “explicit” mode. The implicit mode was used originally; it is an SSL encrypting socket wrapped around the entire communication starting at the point of initial connection. To differentiate this original FTPS from unencrypted FTP, the encrypted FTPS was assigned a standard port of 990. If you see an FTPS server running on port 990, that is almost certainly an implicit FTPS server. It is called “implicit” because the directive to encrypt the connection is implied by using port 990. Note that this mode is far less common than the explicit mode.

Explicit FTPS

Soon after FTPS was in use some smart people decided it would be best if we could have an FTP server that could support unencrypted as well as encrypted connections, and do it all over the same port. To accommodate this the “explicit” FTPS protocol connection begins as a normal unencrypted FTP session over FTP’s standard port 21. The client then explicitly informs the server that it wants to encrypt the connection by sending an “AUTH TLS” command to the server. At that point the FTPS-enabled server and the client begin the SSL or TLS handshake and further communications happen encrypted. Note that most (if not all) explicit FTPS servers can be optionally configured to require encryption, so it will deny clients that attempt to transfer data unencrypted. Often this can be configured on a user by user basis.

Of these three “secure FTP” protocols, only explicit FTPS is defined by an RFC. In this case it is RFC 4217, released in 2005, even though all these protocols had been in use for years prior.

By the way, I have not discussed what encryption algorithms each of these modes use. They all can use AES if you want, and they often have other good options, too. Do not worry about the algorithm when simply selecting which protocol to use; they are all good from this standpoint.

Comparison

Which kind of secure FTP should you use? It really depends on your business, software availability, and network firewall situation. I personally have no preference for or against any of these protocols on their base merits. Here is a table recapping some of the differences:

Firewall Implications

I have been mentioning some firewall implications for the FTP over SSL variants, but let us go into more detail. The FTP protocol initially opens one connection to authenticate the user and for the user to issue commands to the server. This is all well and good until the user wants to view the contents of a directory or transfer a file. Once either of those activities are requested, there is another connection made. To make things even more fun, the connection can be either from client to server (called passive mode) or from server to client (called active mode). I will just simply call these passive and active data connections, respectively, based on the name of the mode of transfer.

When FTP is always unencrypted, firewalls have been using a little trick to allow the active and passive data connections. They can snoop your connection and see what ports are being decided upon for these connections, and then they quickly allow the connection to get through once. That option is not available when the connection is encrypted. Thus, firewalls need another way to figure things out.

The active data connections (from server back to client) turn out to be relatively easy to firewall from the server side. They always use port 20 (for explicit FTPS) or port 989 (for implicit FTPS) as the source port of the connection (this can be configured to be a different port in many servers). A firewall protecting an FTPS server can just confirm that there exists at least one FTPS connection to that server, and assume that outbound connections with the right source port are okay. Most firewalls just end up being configured to always allow these connections based on the source port, and that is usually okay, too.

The passive data connections, by default on most servers, could be anything inbound between 49152 and 65535 (or sometimes between 1024 and 65535, depending on OS and server). Most organizations do not want to allow this many ports through the firewall to the server. Fortunately, the better FTPS servers also allow this range to be configured. Select a fixed range large enough to handle your traffic load and configure your FTPS server to only use that range for its passive ports. Then allow those ports through the firewall in addition to the main connection at 21 or 990.

If you want to serve the greatest number of clients with your FTPS server, you will want to support both active and passive mode connections. Some clients can figure out how to get one type but not the other through their firewalls. You can never guess which one it will be, though passive mode is more likely because all they have to do is allow the connections out from their network. If for some reason your organization decides to only support one mode and not the other, make sure your clients have some means of being notified! Even better if you can configure your FTP server to simply deny an attempt to open data connections using the mode your firewalls do not permit.

Note that even if you are not using SFTP or FTPS, there are other means of securing FTP connections defined in RFC 2228. Someone could, in theory, refer to a “secure FTP” connection that uses another means besides SSH or SSL to secure the connection via the extensions to the FTP protocol defined by that RFC. In practice (excepting Kerberized FTP), I have seen only the types of secure FTP I described above.

I wrote this article to help you decide which type of secure FTP to use. You need to know all these issues to make the best decision for your company: SFTP versus FTPS, implicit versus explicit, and active versus passive. Perhaps even more important is that you need to know how to correctly communicate your decisions to other technical implementers, and you need to understand the right questions to ask of others intending to implement (or already have implemented) “secure FTP” but who do not know about the differences between these forms.

You can use RCS to version control your files. RCS is able to keep a history of previous revisions, and it provides a log for people to note why they made their change. Let us work through an example! A common administrative file that can benefit from RCS is /etc/sudoers. This file is not changed automatically and should only be altered by the Unix administrators. Let us start with a basic sudoers file:

%unixadmins     ALL = (ALL) ALL

We want to add another line for our Oracle administrators, but we want to save the original copy of this file. We first use RCS’s ci (check in) program to create the initial revision of the file:

i_am_root# cd /etc
i_am_root# ci -l sudoers
sudoers,v  <--  sudoers
enter description, terminated with single '.' or end of file:
NOTE: This is NOT the log message!
>> sudo uses this file to determine permissions
>> .
initial revision: 1.1
done

This creates the revision history file, sudoers,v, in the /etc directory right with the sudoers file. If a directory named RCS exists, then the revision history file will be located there. You can use a symbolic link named RCS to store your revision history files anywhere you want (I put all mine in a centralized location).

Creating your revision history file leaves the sudoers file in a “checked out” state. This is necessary because creating the revision history file without also checking out the file will cause the original file to be deleted (oops). Therefore, you may begin editing the file now, or you might want to just check it in so someone else can edit it, too. The command to check files in, ci, can be used with the -u flag instead of -l to unlock the file instead of locking it:

i_am_root# ci -u sudoers
sudoers,v  <--  sudoers
file is unchanged; reverting to previous revision 1.1
done

Let us check the file back out for editing now. We use RCS’s co (check out) command to perform the task, and co takes -l as an option when locking is requested:

i_am_root# co -l sudoers
sudoers,v  -->  sudoers
revision 1.1 (locked)
done

Now edit the file to add the permissions for the Oracle DBAs. I do not care what your editor religion is as long as it is vi or emacs.

%unixadmins     ALL = (ALL) ALL
%oracledbas     ALL = (oracle) ALL

We can use RCS’s rcsdiff command to see what we have changed:

i_am_root# rcsdiff sudoers
===================================================================
RCS file: sudoers,v
retrieving revision 1.1
diff -r1.1 sudoers
1a2
> %oracledbas     ALL = (oracle) ALL

Beautiful! It does not seem so useful for so small a change, but if you are making a lot of changes to a file then it is more useful. You can also use rcsdiff to view differences between revisions, which is extremely useful if you are trying to figure out when a particular line in the file was introduced (e.g. rcsdiff -r1.1 -r1.3 sudoers). If the standard diff format output is undesired, then try -c or -u (the latter not available on all systems) to change up the output format. As with everything, read the man pages for more details. If your man pages are lacking, use the man pages at OpenBSD because they are the best (they describe OpenRCS, an RCS clone, whereas most systems will have the official RCS; but the basic usage is the same).

We are not quite done, but we are almost there. We have to check in our change.

i_am_root# ci -u sudoers
sudoers,v  <--  sudoers
new revision: 1.2; previous revision: 1.1
enter log message, terminated with single '.' or end of file:
>> added permissions for Oracle DBAs
>> .
done

There are other revision control programs out there, and some of them have their place, too, but RCS is still the best for individual system file revision control. Also be careful in how you use it with ever-changing system files, such as /etc/passwd and /etc/shadow. For files such as those a cron job taking a daily backup by using ci -l will suffice. So get rid of all those extra backup files! The only extra copy you ever need has a suffix of ,v.

As it turns out, Ruby includes an excellent module for parsing dates. It can handle several common formats, too! All you need to do is require ‘parsedate’ and you have access to it:

(Text in bold is what you would type.)

require 'parsedate'
user_input = ARGF.readline
seconds = Time.local(*ParseDate.parsedate(user_input)).to_i
puts seconds

% ruby parsedate_example
August 10, 2007
1186722000

If you are in a hurry, try a one-liner:

% ruby -rparsedate -nle \
      'print Time.local(*ParseDate.parsedate($_)).to_i'
August 10, 2007
1186722000
2006-09-21
1158814800
2001 sep 8
999925200

If you deal with /etc/shadow a lot you will find this useful for figuring out what today is in terms of the password last changed field by dividing the given number by 86400 (the number of seconds in each day):

% ruby -rparsedate -nle \
      'print Time.local(*ParseDate.parsedate($_)).to_i / 86400'
August 10, 2007
13735

Or do it in reverse with Time.at:

% env TZ=GMT ruby -nle 'print Time.at($_.to_i * 86400)'
13735
Fri Aug 10 00:00:00 +0000 2007

With Ruby you never need to venture far to manage your dates! Let me know if you find this useful or if you have questions about similar should-be-easy problems you encounter!

Capistrano has no facility for continuing tasks upon failure. In a large Unix environment, stuff needs to happen even if only 98% of the systems are up. What about the 2% not available? They can get the update later when they’re back. We needed a tool that can just do the work and get it done on the 98% and report on who failed. Later, when the failed systems are back online, the tool should automatically update them on the next run. Then we can go about fixing the issues causing those 2% of systems to fail and rest assured that all our changes will be pushed soon.

I intend to fill this gap. Jamis Buck, the maintainer of Capistrano, also wrote the Net::SSH module for Ruby. It is a spectacular module. With it, I plan to write an agentless tool that runs from a central server and uses SSH to log into each system for whatever task necessary. Couple it with some tools to manage users, groups, system configurations, and software installations, and a new tool to conquer the world will be born.

If you want to conquer the world with me, let me know!

I have written a similar tool before. Back when I worked at Abbott I wrote a little tool we called confmgt. I used Perl because when I started the project around 1999 or 2000 that was the best language I knew. Additionally, Perl was installed on every Research and Development Unix system in the company, so I could count on it being present. The original version ran on a central server and used rdist to push files to target hosts after performing some non-trivial logic on which files to include in and exclude from the transfer.

Eventually the tool grew and features were added, and by version 3.0 (the last major version when I left Abbott) the program used rsync by default and could be run either from central server or from the host being updated. More complicated configurations required features to include more customization in how distribution areas were laid out. Despite still being written in Perl, I was very proud of the 3.0 release of confmgt. It was a masterpiece of Perl and of programming in general. With this release other Unix administrators were no longer coming to me and asking for new features; instead they came, asked, and then left knowing how to do what they were requesting. I have rarely felt greater pride in my work.

So now I am going to repeat myself, but this time in Ruby. Instead of rsync I will use Net::SFTP. The configuration file will likely be similar to Capistrano’s so that people familiar with one can use the other easily (as it should be!). It is time to change the world.