Cosine Jeremiah and his Musings

How command line arguments are processed in Unix is important to know when scripting. That which should be a simple script is often troublesome to debug if the arguments to some of the commands get mangled unexpectedly.

To help understand what ends up being sent as command line arguments to a command, we can use a little Perl script I wrote to see exactly what the arguments are and how they were broken up by the shell. It does nothing more that print each argument out on a line of its own, but just that little service is enough to learn the finer points of command line argument processing. I called this script sargs, and here is the code:

#!/usr/bin/env perl
foreach $arg (@ARGV) {
  print("$arg\n");
}

Let us take a look at it in action in a few different circumstances. First with three arguments:

% sargs 1 2 3
1
2
3

Now with the same three arguments combined as one:

% sargs '1 2 3'
1 2 3

We all know that quoting an argument allows us to put spaces and other special characters within them, and with sargs we can see it first hand. With one more simple example, let us see shell wildcards in action:

% sargs /*
/bin
/boot
/cdrom
/dev
/etc
/home
/lib
/lost+found
/mnt
/opt
/proc
/root
/sbin
/service
/sys
/tmp
/usr
/var

This is the evidence that it is the shell, not the command, that interprets the wildcard asterisk (*) and expands it into many arguments, one for each file with a matching name.

Now let us use sargs for something more useful. What happens to command line arguments in a command issued with SSH? SSH mangles the command line when used to issue shell commands. How? We can see exactly how. Three arguments apart are seen as normal:

% ssh localhost sargs 1 2 3
1
2
3

But if I group them inside quotes, something does not work right:

% ssh localhost sargs '1 2 3'
1
2
3

Those were supposed to be one argument! What SSH does is it joins all the command line arguments it receives and passes the string to a shell for interpretation, which in this case includes parsing the one argument as separate arguments. Thus, to keep those arguments together we need to make sure that the quotes are seen by SSH by escaping them:

% ssh localhost sargs "'1 2 3'"
1 2 3

That is what we wanted. Since SSH joins all the arguments together, we can even send them to SSH as separate arguments as long as the quotes are escaped:

% ssh localhost sargs \'1 2 3\'
1 2 3

So next time you have trouble with running commands over SSH remember that the shell gets to see the SSH command and run it and that it is not run directly by the sshd process. Let us take a quick look at sudo to see if it behaves similarly:

% sudo sargs '1 2 3'
1 2 3

There is no reprocessing by the shell here. This is good most of the time when you want to run sudo for a single command because there are no shell double-interpreting surprises. This is bad if you want to run two commands with one invocation of sudo, like this:

% sudo sargs 1 2 '&&' sargs 3 4
1
2
&&
sargs
3
4

Oops! That second command got processed as arguments of the first! Normally it is recommended that you invoke sudo twice and not quote the double-ampersand like so:

% sudo sargs 1 2 && sudo sargs 3 4
1
2
3
4

But sometimes you cannot do that, such as if the first command is sleep and you cannot allow the second sudo command to request your password again because you will be at lunch (sleep 900; shutdown -r now). Sure, you could use at, but that can be messy in its own way for some tasks. So how can we invoke sudo so that it can run two commands?

The answer is you cannot, but you can invoke sudo to run a command that runs two commands. What command might that be? A shell invoked as a command, of course! All of the shells that I know of use the -c flag for this. If you provide a -c flag, the next argument is expanded as a shell command and you can use the special shell characters to break the command up into multiple commands:

% sudo zsh -c 'sargs 1 2 && sargs 3 4'
1
2
3
4

It is not as pretty as one might hope, but it is reasonably easy to do.

I hope that if you had trouble in the past with figuring out how arguments are handled by different programs that you take this chance to learn using sargs. It can really come in handy some day when you are programming that script that you need “an hour ago” but cannot quite figure out why it does not work right. If that happens to you, just whip out sargs and verify that your commands are seeing their arguments exactly as they are supposed to see them, and if not then escape or unescape the offending special characters appropriately!

In my previous post titled Bolting on Security, I mentioned that port 22 is generally not scrutinized as much as 445 when being allowed through a firewall. Obviously the situation varies from incident to incident, but I wanted to say that port 22 really should be looked at more closely. All is not always roses and peaches when you allow port 22, but I am going to explain some of the saving graces of SSH that can make it less threatening in some hairy situations.

The Threats

Let us first start with what is wrong with allowing SSH through a firewall. Then we will address each problem one by one. The problems are:

1. Buffer overflow bugs have led to system compromise.
2. Shell access can often be obtained when none is necessary.
3. SSH supports weak forms of authentication—like passwords.
4. SSH allows the forwarding of connections between the client and the server.
5. SSH helps facilitate X11 forwarding, allowing graphical programs to run remotely.

Buffer Overflows

This is your greatest threat to giving up root access to your server, but fortunately there is something you can do to reduce the risk of break-in to negligible levels. That something is to turn on privilege separation. The only SSH daemon that supports privilege separation is OpenSSH and its more recent derivatives. Fortunately, OpenSSH is a full featured SSH server so even if you are not using it, you should be able to switch to it without much difficulty.

Note that despite Sun SSH being a derivative of OpenSSH, Sun has chosen not to implement privilege separation and thus you would need to replace it. I have badgered people at Sun about this. I can tell you they had a reasonable excuse for omitting it from Solaris 9, but they really dropped the ball with Solaris 10. I hope they can do better for Solaris 11. Your mileage may vary on other operating systems, but the free ones generally use OpenSSH by default.

The reason privilege separation is so good is that it causes the majority of the SSH server code to run with no privileges in a chrooted jail. The little bit of SSH that runs with full root privilege is scrutinized much more closely by the OpenBSD team and security researchers, so it is considered very safe. I do not know of any exploitable vulnerabilities ever found in the privileged code after OpenSSH introduced privilege separation a few years ago (though I may have missed one).

To activate it, create an sshd user (the unprivileged account) and a /var/empty directory (the jail). Then set the UsePrivilegeSeparation option to “yes” in sshd_config and restart your server. You can test if privilege separation is active by looking at your process list after you connect. When privilege separation is in use there will always be two extra processes for each SSH connection, one privileged owned by root, and one unprivileged owned by a regular account (authenticated user) or the sshd account (unauthenticated user). Without privilege separation there is always just one extra process per connection.

Since you need to be using OpenSSH or a derivative to use privilege separation, I will assume you are using it for the rest of this post.

Shell Access When None is Necessary

Sometimes you only want to give someone the ability to transfer files via SCP or SFTP, two protocols that use SSH. Traditionally you have to give full access to a shell to allow only file transfers with SSH.

One solution that has been around for quite some time is a program called scponly. You use scponly as the default shell for “file transfer only” accounts, and it disallows command line and command access. This program has previously had a few security problems that has occasionally allowed an authenticated user to “break out” into a shell, but overall has been very good. The maintainers take its security seriously.

Password Authentication

Password authentication is great if you—oh who am I kidding!? Password authentication is bad, and if you are trying to keep things apart by a firewall and there is a good chance that an attacker will start trying out passwords against your open port, then you better have something better than password authentication protecting the accounts.

SSH can be configured to disallow password authentication and require stronger credentials. Just set the PasswordAuthentication option to “no” and those pesky little weak passwords will not be a problem. Of course you will need to set up something else, such as SSH keys, Kerberos, or a challenge-response mechanism. If you are in a high-security situation, you should consider using some form of two-factor authentication.

Support for Forwarding Connections

SSH can be used to forward connections, effectively bridging two networks to a limited extent. If you are scrutinizing the security of an SSH server on the other side of a firewall, you will want to turn this off. Do so by setting AllowTcpForwarding to “no”.

However, perhaps there is one port in particular that you do want to forward with SSH through your firewall. This could be used to encrypt an insecure channel as it travels over an untrusted network, or it can allow an “outsider” to connect to a service inside your firewall. Whatever the reason, you have a means of limiting what is allowed through at the server side. This is done with the PermitOpen option on newer versions of OpenSSH—it can be used to limit what target systems and ports the server allows clients to forward to. Even if you have full control of the client system, it always helps to add a layer of security on the server side, too.

Support for Forwarding X11 Displays

When it was released in the mid-90s, one of the biggest problems facing Unix users was how to run their X11 graphical applications securely. The security of an X program was usually dependent on weak host-level security, and as a result anyone with a login on the system with the X client (the program) could also run an X client to snoop or annoy the user. There was also a cookie-based authentication available, but it was difficult to teach users how to use it.

Along came SSH, and not only did it replace Telnet by providing a secure shell connection, it also provided a means to automatically configure the cookie-based authentication for X11 and forwarded the traffic over the encrypted channel. Good for you when connecting to your company server internally. Bad for you when you do not fully trust a user or an account that needs access to the system through a firewall.

This one is easy to disable, too. Just set the X11Forwarding option to “no”.

Summary

Do not be afraid to allow SSH through your firewall, but take reasonable precautions and know your SSH server. Know how to disable the features you do not need. Get the SSH client to use the strongest form of authentication available. Read the man pages for ssh_config and sshd_config and be familiar with many of their options. Someday, when you need to boost the security of a connection somewhere, you will be glad you did.

To build security in your project, you need to make it a consideration from the start. Bolting it on afterward is always a recipe for disaster, and unless you start over with a complete application rewrite you are likely to fail to secure your application in some subtle manner or another. If you are in a hurry to get Version 1.0 out to meet a limited opportunity or to meet a costly deadline to miss, it may be acceptable to a business to provide incomplete security in an application to get the product out the door. However, the application needs to have been designed for its security profile from the start or its security can never be completed.

For example, let us say I have an application that needs strong password controls to be secure, but the developers are running out of time to ship. For Version 1.0, I, as a decision maker for the business, might want to drop the password controls on the application. However, I must be certain that the password controls remain in the architecture design or else the application will end up deployed and used in a manner that would defeat ever trying to add password controls later. In particular, if I need a mechanism in the application that allows it to facilitate a user changing his expired password but I then deploy the application to users without any authentication at all, then I will likely find myself completely unable to add the strong password controls later. Firstly, I would need to somehow find out what users I have. Since none have ever authenticated, I have no idea who is even using the system! And now I am expected to add strong password controls?

This kind of thing happens all the time, but often more subtly. Many of the Microsoft protocols suffered security issues for years because they had to provide backward compatibility with older Windows desktops. I am not even certain to what degree the Windows 2003 servers protocols still suffer from many of those same issues, but I know that allowing traffic through a firewall on port 445 between Windows hosts is still something to be eyed with great suspicion. It is somewhat because that one port allows so many services: file sharing, printing, and authentication are the most common. But one can accomplish many of the same things on a Unix system over port 22 with SSH, and that port is not one that needs to be scrutinized nearly as much. Perhaps it is because the SSH protocol is much better documented than the Microsoft protocols, or at least much better understood by many more people. Perhaps it is because Windows systems are more often the targets of viruses.

Anyway, while I wish that every application could build in security in every release, I understand that a security-starved Version 1.0 will occasionally need to see the light of day. When you are involved in such a Version 1.0, just remember to keep security in your architecture. Allow it manifest in Version 1.1 or even Version 2.0 if necessary by keeping the goal in mind. Just do not expect that you can cut it out of your architecture completely and then bolt it on later—you cannot; the smartest people in the world have already tried, and we know how your story will end. Security cannot be bolted on as an afterthought.

I was talking with one of my Unix administrator friends yesterday that had never heard of /usr/bin/yes before. I have to admit I was a little surprised at first, but when you think about it, /usr/bin/yes is hardly ever used these days. To a younger administrator used to having a -y or -f flag on commands that cause an “always answer yes” or “force the action” effect, why would one ever need a command like /usr/bin/yes?

For those that have never heard of /usr/bin/yes, it repeatedly outputs the letter ‘y’ followed by a newline. It used to be used to answer ‘y’ to questions posed by programs asking things like “should I proceed?” and “are you sure?” If such commands do not have a mode to force them to be non-interactive and you want them to just perform the task without question, /usr/bin/yes is your tool. Run ‘yes’ and pipe the output to the interrogative command! As always, be careful to not run something that could cause unknown harm when using this technique!

After my previous post on ad hoc scripting I had an example come up in my own work. I needed to search through some web logs for all hits in August to three servers from a pair of IP addresses. Below is the actual command line script I used, except I changed some identifying information and extended the script from a single line to multiple lines for clarity. This script reduced the logs to a file that details which web server was it, which IP address the request came from, and the date and time of the request:

let D=1; while (( D <= 31 )); do
  DD=$(printf "%02d" $D)
  for M in server-01 server-02 server-03 ; do
    gzip -cd $M/weblog.200708$DD.gz | \\
        perl -anle 'print "'$M' $F[0] $F[3]"
            if $F[0] =~ /^10\.1\.2\.[34]$/'
  done
  let D=D+1
done > ~/log_report

There are a few things I want to point out about this mini-script:

1. Iteration is performed on an index value ($D), tested using Kornshell’s double parentheses syntax, and incremented using a let statement.
2. It makes use of a Perl one-liner to rearrange the data on each line and only print that data that is relevant to the task at hand.
3. I send the output of the entire thing to a file by redirecting it after the final “done” statement.

None of these things is really a big deal to me anymore, but I have noticed many people do not think or know you can do these things. I think that if you do realize it then you can come up with some more powerful mini-scripts from your command line.

I noted before that I use the Z shell, a Bourne compatible shell. While the above mini-script will not work in the Bourne shell itself, it should also work in the Kornshell and the Bourne Again shell, though your mileage may vary.

Incrementing an Index

The generic script code for iterating over an increasing numeric index is:

let I=0; while (( I < 10 )); do
  ... your code here ...
  let I=I+1
done

If your shell is a traditional Bourne shell that does not support let-statements and double parentheses notation, then you may do this:

I=0; while [ $I -lt 10 ]; do
  ... your code here ...
  I=`echo $I + 1 | bc`
done

If you have a small set of numbers to iterate over then it is often still faster to just type them all out in a for-statement, but that stops being true when you have a large set of numbers.

Invoking Perl and/or Ruby

This technique should not be alien to someone that writes shell scripts. Even if you never thought to invoke Perl and/or Ruby in your scripts, you have probably used sed and awk. In the above example I could have used awk instead of Perl, but Perl seemed like a better choice at the time. Ruby and Perl have flags that help them be more useful on the command line, and in fact they can both become more powerful replacements for sed and awk. Some flags are:

-n: I call this the awk-mode flag. It causes both Ruby and Perl to loop on each line of the input, putting that line of input into $_.

-p: I call this the sed-mode flag. It causes both Ruby and Perl to loop on each line of input like -n, but also prints the value of $_ at the end of the loop (presumibly you have modified it).

-a: This is often used with -n, though it could also be used with -p. As each line of input is read, it is split and the resulting array stored in @F (Perl) or $F (Ruby). I used this feature in the above mini-script to limit which fields are output to the report I generated.

-l: This flag causes lines output to be terminated with a newline. It also causes lines of data input via -n and -p to be auto-chomped.

-e: The most important flag for one-liners! This causes Perl and Ruby to take their commands from the argument given after -e. It may be specified multiple times to issue more code from the command line.

-i: If you want to edit a file in-place you can use this flag to tell Perl or Ruby to edit each file of input instead of leaving those files untouched. You can also opt to provide an extension to be used for a backup of the input. Note that because of the syntax for the optional extension use, you should check the man page for usage.

Redirecting Output

You can redirect output from any part of a script. Most people focus on redirection of individual commands, but that causes one to have to keep appending data to an output file. One does not need to do that! Just redirect your output after the “done” and you will capture all of the output from all of the commands inside the loop. Spiffy, eh?