Okay, fine—sort of. I have a beef with this.

Good security practice is about implementing layers of security, and this flies in the face of that. If certain other vulnerabilities are present, the most obvious one being HTTP response splitting, then CSRF is once again possible. Any vulnerability that could allow custom headers to be inserted in a CSRF request now makes them possible because the authenticity token is bypassed by a custom header.

In my opinion, if a site has data worth protecting with an anti-CSRF authenticity token, there’s no reason not to use it for all requests that can alter that data. Doing the bare minimum is what I call Just-Barely-Enough Security.

Note that Rails 3.0 handles CSRF protection differently (within Rack), and I have not examined it; the above comments apply only to Rails 2.x.

So it turns out that Rails is not the only web framework out there where the CSRF protection was in the just-barely-enough category. I have also been working on a Django 1.1 project recently, and in doing so I have found another category of just-barely-enough security (fixed in Django 1.2). Django 1.1 generates the authenticity token in a somewhat weak manner, but it is not weak enough that it is exploitable by itself. The weakness is that the authenticity token is the MD5 hash of a site-wide secret concatenated by the session ID (in contrast, Rails 2.3 and Django 1.2 generate a completely random token for each session). Because of how MD5 hashes are computed, an attacker could potentially learn the state of the MD5 computation up to the point of hashing the site-wide secret and himself compute any authenticity token given a session ID without further consulting the Django server. However, this is not an issue by default because Django’s session fixation protection prevents it from responding with session IDs that it did not generate, so the attacker cannot build up the necessary information to perform the attack. But if something were to be configured wrong on the Django server and a session fixation attack were to be possible, then the attacker would have more avenues for exploitation than just using session fixation on the victim directly.

Even worse would be a situation where the session fixation issue was found and repaired, but not before an attacker gleaned the necessary information to generate authenticity tokens for the site. The site administrator would think he closed the security hole, but his site may still be exploitable based on previously leaked information. That would be rather unfortunate.

My morning at work went relatively fast. I had very few scheduled tasks because of meetings dotting my schedule and my vice president’s group was all going out to play Whirlyball for the afternoon. Additionally, due to my company’s sales team scoring a big contract, the company was serving free Lou Malnati’s pizza for lunch.

Whirlyball was probably the most fun of the day, but it is what happened afterward that is the most interesting. I arrived at Houlihan’s at 6:30, preparing to socialize with other security professionals at Chisec 16 starting at 7:00. I did not wait long. I had hardly sat down when Maniac showed up, always full of interesting conversation and even an Asus Eee PC, something I had never seen before. It was not long before the room was full of other security professionals from all over the Chicago area.

With 20–30 people in a room, you do not get to talk to everyone. I primarily spoke with a couple guys from the University of Chicago and some consultants from no less than three different firms. It was Thomas Ptacek that gave me the biggest surprise of the evening.

Tom told me that it is getting increasingly rare to find computer people that know C. I had never thought about it, but I could see why this would be a problem. I have been using C since 1994, and I simply consider it a staple of my computer abilities. It is like part of the foundation. It is through C that I know how a shell interacts with an operating system, or how any program interacts with other components of the system. My knowledge of C is how I learned about the general structure of a running process in memory, and from that I understand how things like buffer overflow attacks actually work. It is through C that I even know how Ruby handles its garbage collection, at a low level. In regard to modern computer architecture, if you do not know C, then I would be incredulous if you told me that you really understand computer architecture. I am not sure that knowledge of C++ can really convey the same understanding, except that someone could do so by paying close attention to the subset of C++ that is C.

I wonder… do you know C? If I were to compile a list of important languages that all computer programmers should learn, C would be high on the list, if not the top language. Certainly there are other important languages out there that expand ones mind around advanced programming topics, such as Ruby, Lisp, and ML, but down at the linker level C is the language that all other languages communicate with the operating system or the hardware—application binary interfaces (ABIs) are designed around how C compilers generate object files. It is that important.

http://download.oracle.com/technology/tech/soa/soa_best_practices_1013x_drop3.pdf

I was going to review it for its security best practices and WS-Security recommendations… but there are not any. Take that to mean what you will.

1. SSL is ubiquitous and most everyone knows how to use it, whereas WS-Security is largely an “unknown” to most organizations.

2. Both SSL and WS-Security are easy to “drop in” to a typical web service architecture, though SSL is somewhat easier due to the previous observation.

Both SSL and WS-Security have programs or libraries that make adding them to an existing web service a reasonable task.

Adding SSL to a web service can be as easy as putting Stunnel or other SSL termination software in front of the service. However, to integrate SSL’s authentication features one would need to have the results of the SSL handshake passed into the application somehow. More often a server ends up using weaker credentials, such as username and password, inside the SSL connection to establish the identity of its users.

With WS-Security, implementation can be done by adding a few calls to an appropriate WS-Security library, such as WSS4J (Java) or WSS4R (Ruby).

3. SSL provides security for an entire connection; WS-Security secures each message one by one.

If an application keeps a connection open to make multiple requests of a web service, then SSL would provide better performance than WS-Security. Exactly how much better may depend on your servers’ average load. The difference in computation is due to SSL using mostly symmetric key encryption after the initial handshake at the start of the connection, whereas WS-Security needs to perform asymmetric key encryption on each message (one computation for handling encryption and one computation for handling the signature). Symmetric key encryption is a lot faster than asymmetric key encryption, and while both technologies switch to using symmetric key encryption as fast as possible to provide the best performance, WS-Security is unable to avoid those asymmetric key computations on every message.

On the other hand, since security is applied on a message-by-message basis with WS-Security, an application is not susceptible to possible channel hijacking or HTTP splitting attacks that could occur with SSL. If either of these things were to happen when WS-Security was in use then its security features would still protect the message and the data.

4. SSL requires an immediate communication with the endpoint to securely transmit data; WS-Security can protect a message that is queued up or stored on intermediate servers without additional concerns.

With SSL, the SOAP messages themselves remain unencrypted but are transmitted through a secure tunnel (the SSL link). If that message ends up being stored on a disk (queued) prior to use, then it is unprotected from being read or tampered with by anyone who can obtain access to that disk. If you think this is unlikely then you should know that attackers are often able to leverage poorly configured web, FTP, Citrix, remote desktop, and Windows file sharing servers to access disk. This is one of several reasons that encryption of data at rest has become more important recently, and “at rest” refers to any time data hits a disk or a tape, not just when it resides within a database.

On the other hand, WS-Security imposes its protections upon the message itself, so if the message were to be copied by an adversary he could not read it. If the attacker could find a way to get write access to the disk containing the WS-Security protected message waiting in queue, any attempt to tamper with the message is detected by WS-Security’s integrity protection features.

Comparison Summary

SSL (pros):

1. SSL is ubiquitous—it is easy to find people with experience using it.
2. Easy to “drop in” to a typical web service architecture.
3. Provides encryption for an entire connection.
4. Client may authenticate the server during SSL handshake.
5. Server may authenticate the client during SSL handshake.

SSL (cons):

1. Connections must be active to transport data and responses. In other words, the endpoints must have a direct channel to each other.
2. Once authenticated with the SSL handshake, any data transported on the connection is trusted. This might leave a channel open that can be hijacked and used in similar fashion to a cross-site request forgery attack.
3. Data is vulnerable to capture and tampering if left unencrypted on a disk by queuing software; extra precautions may be necessary to protect it.

WS-Security (pros):

1. Easy to “drop in” to a typical web service architecture.
2. Provides encryption for a single message.
3. Client may be assured the server is the message recipient by virtue of the public key used to encrypt the message. Likewise, the server is assured that the client is the recipient of the reply by virtue of the public key used to encrypt the reply.
4. Server may be assured the client is the message sender by virtue of the public key used to decrypt the message signature. Likewise, the client is assured that the server is the sender of the reply by virtue of the public key used to decrypt the reply signature.
5. Messages may be time limited to prevent expired messages from being processed.
6. Messages are innately protected if left on disk and queued during transmission to the consumer.

WS-Security (cons):

1. WS-Security is not ubiquitous. You will likely need to train your development staff how to use it.
2. Each individual message and its response requires processor expensive asymmetric key encryption to occur even if a communications channel is maintained. This is negligible on small loads, but systems with hundreds, thousands, or yet more messages per second may become loaded.

After reviewing the differences I suggest that WS-Security should be the preferred choice of most enterprises. Do note that I suggest you do both SSL and WS-Security (for defense in depth and to provide some security by obscurity against those that would sniff your packets), but if you were to limit it to one or the other then I would prefer WS-Security. The issue that convinced me was the security of messages left in queue on disks. This situation happens rather frequently (swap files, anyone?) and is often done unbeknown to the original application architect as the system grows and the message delivery mechanism is changed. WS-Security simply provides security from end to end without worrying that every step of the way is secured appropriately as well; SSL only promises security between the endpoints of the SSL tunnel.

However, today I will do as many other blogs do and point to some interesting content from elsewhere that I recently came across. There are two recent articles by Nate Lawson that are very interesting; you may find them here and here, and I suggest you read them. They discuss the security of internal system components—hardware security.

As security practitioners continue to struggle with networks and applications, we still have the same issues affecting components of our hardware. The exact same issues, in fact: authorization, authentication, and security versus performance to name a few. I smile at the thought of a conference room somewhere in the bowels of Intel or AMD where engineers and managers discuss threat models and risk equations and make decisions on what to implement and how to implement PC hardware features based on costs and risks in a manner no different than how a web application’s security is discussed at a software company.

The Threats

Let us first start with what is wrong with allowing SSH through a firewall. Then we will address each problem one by one. The problems are:

1. Buffer overflow bugs have led to system compromise.
2. Shell access can often be obtained when none is necessary.
3. SSH supports weak forms of authentication—like passwords.
4. SSH allows the forwarding of connections between the client and the server.
5. SSH helps facilitate X11 forwarding, allowing graphical programs to run remotely.

Buffer Overflows

This is your greatest threat to giving up root access to your server, but fortunately there is something you can do to reduce the risk of break-in to negligible levels. That something is to turn on privilege separation. The only SSH daemon that supports privilege separation is OpenSSH and its more recent derivatives. Fortunately, OpenSSH is a full featured SSH server so even if you are not using it, you should be able to switch to it without much difficulty.

Note that despite Sun SSH being a derivative of OpenSSH, Sun has chosen not to implement privilege separation and thus you would need to replace it. I have badgered people at Sun about this. I can tell you they had a reasonable excuse for omitting it from Solaris 9, but they really dropped the ball with Solaris 10. I hope they can do better for Solaris 11. Your mileage may vary on other operating systems, but the free ones generally use OpenSSH by default.

The reason privilege separation is so good is that it causes the majority of the SSH server code to run with no privileges in a chrooted jail. The little bit of SSH that runs with full root privilege is scrutinized much more closely by the OpenBSD team and security researchers, so it is considered very safe. I do not know of any exploitable vulnerabilities ever found in the privileged code after OpenSSH introduced privilege separation a few years ago (though I may have missed one).

To activate it, create an sshd user (the unprivileged account) and a /var/empty directory (the jail). Then set the UsePrivilegeSeparation option to “yes” in sshd_config and restart your server. You can test if privilege separation is active by looking at your process list after you connect. When privilege separation is in use there will always be two extra processes for each SSH connection, one privileged owned by root, and one unprivileged owned by a regular account (authenticated user) or the sshd account (unauthenticated user). Without privilege separation there is always just one extra process per connection.

Since you need to be using OpenSSH or a derivative to use privilege separation, I will assume you are using it for the rest of this post.

Shell Access When None is Necessary

Sometimes you only want to give someone the ability to transfer files via SCP or SFTP, two protocols that use SSH. Traditionally you have to give full access to a shell to allow only file transfers with SSH.

One solution that has been around for quite some time is a program called scponly. You use scponly as the default shell for “file transfer only” accounts, and it disallows command line and command access. This program has previously had a few security problems that has occasionally allowed an authenticated user to “break out” into a shell, but overall has been very good. The maintainers take its security seriously.

Password Authentication

Password authentication is great if you—oh who am I kidding!? Password authentication is bad, and if you are trying to keep things apart by a firewall and there is a good chance that an attacker will start trying out passwords against your open port, then you better have something better than password authentication protecting the accounts.

SSH can be configured to disallow password authentication and require stronger credentials. Just set the PasswordAuthentication option to “no” and those pesky little weak passwords will not be a problem. Of course you will need to set up something else, such as SSH keys, Kerberos, or a challenge-response mechanism. If you are in a high-security situation, you should consider using some form of two-factor authentication.

Support for Forwarding Connections

SSH can be used to forward connections, effectively bridging two networks to a limited extent. If you are scrutinizing the security of an SSH server on the other side of a firewall, you will want to turn this off. Do so by setting AllowTcpForwarding to “no”.

However, perhaps there is one port in particular that you do want to forward with SSH through your firewall. This could be used to encrypt an insecure channel as it travels over an untrusted network, or it can allow an “outsider” to connect to a service inside your firewall. Whatever the reason, you have a means of limiting what is allowed through at the server side. This is done with the PermitOpen option on newer versions of OpenSSH—it can be used to limit what target systems and ports the server allows clients to forward to. Even if you have full control of the client system, it always helps to add a layer of security on the server side, too.

Support for Forwarding X11 Displays

When it was released in the mid-90s, one of the biggest problems facing Unix users was how to run their X11 graphical applications securely. The security of an X program was usually dependent on weak host-level security, and as a result anyone with a login on the system with the X client (the program) could also run an X client to snoop or annoy the user. There was also a cookie-based authentication available, but it was difficult to teach users how to use it.

Along came SSH, and not only did it replace Telnet by providing a secure shell connection, it also provided a means to automatically configure the cookie-based authentication for X11 and forwarded the traffic over the encrypted channel. Good for you when connecting to your company server internally. Bad for you when you do not fully trust a user or an account that needs access to the system through a firewall.

This one is easy to disable, too. Just set the X11Forwarding option to “no”.

Summary

Do not be afraid to allow SSH through your firewall, but take reasonable precautions and know your SSH server. Know how to disable the features you do not need. Get the SSH client to use the strongest form of authentication available. Read the man pages for ssh_config and sshd_config and be familiar with many of their options. Someday, when you need to boost the security of a connection somewhere, you will be glad you did.

For example, let us say I have an application that needs strong password controls to be secure, but the developers are running out of time to ship. For Version 1.0, I, as a decision maker for the business, might want to drop the password controls on the application. However, I must be certain that the password controls remain in the architecture design or else the application will end up deployed and used in a manner that would defeat ever trying to add password controls later. In particular, if I need a mechanism in the application that allows it to facilitate a user changing his expired password but I then deploy the application to users without any authentication at all, then I will likely find myself completely unable to add the strong password controls later. Firstly, I would need to somehow find out what users I have. Since none have ever authenticated, I have no idea who is even using the system! And now I am expected to add strong password controls?

This kind of thing happens all the time, but often more subtly. Many of the Microsoft protocols suffered security issues for years because they had to provide backward compatibility with older Windows desktops. I am not even certain to what degree the Windows 2003 servers protocols still suffer from many of those same issues, but I know that allowing traffic through a firewall on port 445 between Windows hosts is still something to be eyed with great suspicion. It is somewhat because that one port allows so many services: file sharing, printing, and authentication are the most common. But one can accomplish many of the same things on a Unix system over port 22 with SSH, and that port is not one that needs to be scrutinized nearly as much. Perhaps it is because the SSH protocol is much better documented than the Microsoft protocols, or at least much better understood by many more people. Perhaps it is because Windows systems are more often the targets of viruses.

Anyway, while I wish that every application could build in security in every release, I understand that a security-starved Version 1.0 will occasionally need to see the light of day. When you are involved in such a Version 1.0, just remember to keep security in your architecture. Allow it manifest in Version 1.1 or even Version 2.0 if necessary by keeping the goal in mind. Just do not expect that you can cut it out of your architecture completely and then bolt it on later—you cannot; the smartest people in the world have already tried, and we know how your story will end. Security cannot be bolted on as an afterthought.

Recently I had started using Microsoft’s DREAD framework for performing written security risk assessments. I liked it because at the end of my assessment I can deliver a number to my management. Low numbers are good, and high numbers are bad. For about two months and about six written assessments (not many, I know) I was very happy with the numbers the DREAD framework was giving me when I used it. I felt the numbers were adequately expressing the risk to the organization.

Then, earlier this month, I did one more for a project that I felt held insignificant risk to the organization, but I did not give it the usual rubber stamp approval because the project ignored some good practices that I wanted to insist upon (when I say ignored I mean thumbed their nose up at me when I told them they should do it). I thought that if they were going to go against my recommendations that we should all have an accurate risk assessment of what that meant. However, the resulting score out of DREAD was high, and as I said before, does not accurately represent the true risk of this project. The reason DREAD failed here was that its score is the result of an addition instead of a multiplication of the various risk factors (most models I have seen use multiplication), so one low score means little in the face of some other high factors.

I scratched my head, delivered my DREAD report along with my thoughts that the score was higher than the true risk, and moved along to other projects. Today, however, I came across this article by Richard Bejtlich. In it he provides some criticism of FAIR, another security risk assessment framework currently being promoted by its originator(s). What really caught my attention is that he essentially argues that risk assessment is the wrong path for us in the security industry. I have to admit it took me a long time to climb on board with the whole risk assessment bandwagon myself, but I have never seen someone with this much experience argue against it. As I read I realized that I never really believed in the whole risk assessment model, and that I really only use it as a means of selling security to my management—an important task, no doubt, but is it perhaps the wrong approach? Certainly managers need this to be able to reduce risk into dollars and cents for the business?

The problem is that risk assessments are based on guesses. Not just any guesses, though; they are based on wild-ass guesses. We are supposed to say “do X instead of Y for COST dollars against our budget or else we can expect hackers to come in and cost us GUESS.” But maybe they will not know we are here and hack someone else instead. The reality is that we have no idea what the odds of being breached are, and we have no idea what the cost will be if we are breached. We have no idea what the motivation of our attacker(s) might be, and we have no idea what that motivation will spur them to do. What are the odds of being breached in some manner we did not think of? After all, if we had thought of it we probably would have closed the door on that mechanism, right? So what good is a risk assessment on a problem we already know about?

We really need to be focusing on best practices and engineering the systems to be as paranoid of everything as possible. To encrypt or not to encrypt? You encrypt! To trust or not to trust? If you have to ask, don’t trust! Why can we not build systems like this every time? For an organization that lives and breathes encryption, the incremental cost of encrypting data and connections dwindles toward zero over time. You just do encryption exactly like you just drink coffee. Does anyone lament that we no longer use Telnet now? There was a fixed cost for moving people to SSH, and now the cost to keep everyone using SSH is negligible. The costs of security should be focused on new threats instead of ignoring them until we can finish securing our browsers from the latest Javascript abuse (and whatever else plagues us this week). And best of all, those pesky auditors can become your friends, and you can answer questions about your security practices without trying to dance around the truth. Wouldn’t that be nice?

Firstly, check your routing tables and your basic network configuration. More than one Unix administrator on a rampage of righteous indignation against their network team has found himself swallowing his own pride when it was discovered that he had errors in his routing table. So check your IP address, your netmask, and your default router first. All good? Now check if you need or have any other routes in your routing table. All good here, too? Excellent! If you are new to Unix and need help finding this information, read the man pages for ifconfig and netstat to get started. I would love to give some examples but it differs very much from Unix variant to Unix variant. In general, though, use “netstat -nr” for viewing the routing table and “ifconfig -a” for seeing all of the network interface configuration information.

Now that you IP configuration is confirmed, let us move on to the connection in question. Firstly, you have to identify the port and protocol of the connection. If you are running a standard protocol, such as HTTP, then you can likely assume port 80, but perhaps you do not know? We have a couple options.

There are a myriad of tools to see what connections and proto-connections exist on the system. I like lsof because I can view the connections for each individual process. So let us say I have that HTTP server I mentioned above running on my Unix server, but I am not sure if it is running on port 80 or some other port. I can start the web server process, note what its process ID is with ps, and then use lsof to view its listening sockets to see what port the clients should connect to.

I am using Mac OS X today, which has a BSD style ps command:

% ps auxw | grep httpd
root     13888   0.0 -0.3    41732   5348  ??  Ss   Tue05PM   0:15.24 /usr/sbin/httpd
www      14549   0.0 -0.0    32636    796  ??  S    Wed09AM   0:00.00 /usr/sbin/httpd

When using a System V style ps I use “ps -ef” instead of “ps auxw”. The process ID here for httpd is 13888 (I could also use 14549 because they share the port, in a sense, but that may not always be the case). So let us run lsof to see what ports are open:

% sudo lsof -Pnp 13888 | egrep 'TCP|UDP'
httpd   13888 root   16u    IPv4 0x76fe710      0t0       TCP *:80 (LISTEN)

It is port 80! And in particular it is TCP port 80. Note that this is opposed to UDP port 80, which is another beast altogether that we will address below. For now let us move on to testing this connection.

Testing TCP Connectivity

The most widespread tool for checking if there is a firewall between two systems blocking a particular TCP port is telnet. It is not the best utility, but it does the job and is ubiquitous; you will find it on almost all Unix and Windows systems, and sometimes even VMS and mainframes.

To use telnet, just get to a command line and type “telnet” followed by the target system and port number:

telnet my-web-server.example.com 80

If all is well and you can reach the server, you should see something like this:

% telnet my-web-server.example.com 80
Trying 192.168.1.10...
Connected to my-web-server.example.com.
Escape character is '^]'.

If the service is down, but there is no firewall intervention, you will see something along the lines of “connection refused” like below:

% telnet my-web-server.example.com 80
Trying 192.168.1.10...
telnet: connect to address 192.168.1.10: Connection refused
telnet: Unable to connect to remote host

But what if there is a firewall in between dropping all the packets? You would see the message attempting to make the connection, and then telnet will seem to just freeze for many seconds. After some time it will report a failure, but if you see it freezing for more than ten seconds then you can terminate the connection attempt with control-C. Here is what it looks like:

% telnet my-web-server.example.com 80
Trying 192.168.1.10...
(At this point telnet will stop and wait for a few minutes.)
telnet: connect to address 192.168.1.10: Operation timed out

Note that even if it looks like a firewall is dropping the packets, there may be any number of other causes:

1. Host-based firewall on source host blocking outbound traffic.
2. Host-based firewall on destination host blocking inbound traffic.
3. Target host is down.
4. Either host has an error in its routing table.

Testing UDP Connectivity

Sometimes applications use UDP instead of TCP for their communications. This can be tricky to handle if trying to determine if you have a network issue between nodes. The reason is that UDP does not require the other host to reply; if you send a packet that is received you see the same thing as if you send a packet that never reaches its destination. Note that if you send a UDP packet that arrives to a system not accepting packets then you do get a different behavior that you can measure, because in that case the system will notify you that it is not accepting packets there.

So first make a little extra effort to ensure both hosts are up and that the routing table works between them. You can do this by probing any TCP port (open or closed) or with an ICMP ping. As long as those packets can get through you know your routing tables are okay (note this does not rule out host-based firewalls as a problem as specific ports may be filtered).

We can try to send a generic packet to the other host with either ping or traceroute—those two are widely available—or netcat if you have it. Netcat is the best if it is available:

% nc -vvuz my-cifs-server.example.com 137
my-cifs-server.example.com [192.168.1.20] 137 (netbios-ns) open
 sent 0, rcvd 0

The means of getting ping or traceroute to send a single UDP packet to a specific port varies per system and is even impossible in some (all?) versions of Windows. Because of these differences, I recommend installing netcat and using it if you need to debug UDP connections (it is also better than telnet for debugging TCP connections).

Since we really only know that we do not have a network issue if the port shows up as “closed” we have to do more research to confirm a network problem. Note that if you have the luxury of being able to shutdown the UDP service on the destination host then you should do so for testing so you can observe the closed state.

The next tool you need to fully debug a broken UDP connection is a packet sniffer. I recommend Wireshark if you are on Windows, or tcpdump if you are on any non-Solaris Unix system. On Solaris, use the built-in “snoop” command. Each of these has a somewhat different usage:

tcpdump -i en1 'host my-cifs-server.example.com and udp and port 137'
snoop -d ce0 'host my-cifs-server.example.com and udp and port 137'
Wireshark is usually run in a GUI, particularly on Windows.

Get your packet sniffer running and start talking to your port. If you do not see your packets going out, you are doing something wrong. Go back and figure out if you are sniffing the wrong interface or have an error in your usage. If you do see your packets going out but none returning, then it is still inconclusive, but at least you know you are sniffing the right thing. If you do see return packets, your connection is open and you can proceed with troubleshooting your application instead of the network.

The key here is to get the other server to send packets back in response to packets sent. If you can do that then you know the connection works. Unfortunately, many servers just ignore stuff they consider garbage, like the packets we were sending with netcat.

In our case, we are trying to connect to a NetBIOS name service running on UDP port 137. Some such servers reply to garbage with garbage (success for us!) and some just drop the packets, leaving us confused. To get these guys stirred up we need to generate some real NetBIOS name service traffic from our workstations or servers. In this case, we would log into our Windows workstation and try to connect to a file share on the server; we could also run the nbtstat command instead. In either case, with our sniffer armed and ready we will hope it is enough to generate a reply. If there is truly a NetBIOS name service running on the remote port, we should see evidence of it in the sniffer output. If not, then it is at this point that we can turn towards the network and in good conscience assume all is well from the server side (because you did check your host-based firewall settings, right?).

Debugging network connectivity issues does not have to be a nightmare. With the right tools, knowledge, and experience doing so you can be a pro at determining if an issue needs resolution in your application, on the OS it is running on, or the network. Be the hero that knows instead of guesses what is happening out there, because the resolution often follows.

However, in the rush to address the needs to secure file transmissions we ended up with multiple protocols. And even among those protocols there are multiple standards regarding how to operate them. To connect a client to a server both sides have to select the same technology, and simply saying “we want/do secure FTP” just leaves implementors in a quandary about which kind of secure FTP.

SFTP — FTP over SSH

SFTP (S before FTP) is the protocol that describes an FTP-like protocol that is secured over an SSH channel. SSH’s standard port is 22, and so SFTP’s standard port is also 22. SFTP is very “firewall friendly” because all communications run over a single connection. On the down side, most implementations of SFTP carry with them a full implementation of SSH, and SSH is traditionally used for shell access. For a typical business connection, allowing shell access is highly undesirable. The most recent versions of SSH servers are beginning to realize an “SFTP only” configuration is a good feature to support, so this situation is fast improving.

FTPS — FTP over SSL

FTPS (S after FTP) is a different protocol entirely. FTPS is a super-set of the same FTP protocol that the Internet has relied upon for years and years, but it allows for encryption of the connection over an SSL or TLS encrypted socket. This protocol is harder to allow through firewalls because it follows the same process of opening additional connections from server to client or from client to server as the unencrypted version of the FTP protocol.

Implicit FTPS

To further complicate things a little, the FTPS protocol can be run in an “implicit” mode or an “explicit” mode. The implicit mode was used originally; it is an SSL encrypting socket wrapped around the entire communication starting at the point of initial connection. To differentiate this original FTPS from unencrypted FTP, the encrypted FTPS was assigned a standard port of 990. If you see an FTPS server running on port 990, that is almost certainly an implicit FTPS server. It is called “implicit” because the directive to encrypt the connection is implied by using port 990. Note that this mode is far less common than the explicit mode.

Explicit FTPS

Soon after FTPS was in use some smart people decided it would be best if we could have an FTP server that could support unencrypted as well as encrypted connections, and do it all over the same port. To accommodate this the “explicit” FTPS protocol connection begins as a normal unencrypted FTP session over FTP’s standard port 21. The client then explicitly informs the server that it wants to encrypt the connection by sending an “AUTH TLS” command to the server. At that point the FTPS-enabled server and the client begin the SSL or TLS handshake and further communications happen encrypted. Note that most (if not all) explicit FTPS servers can be optionally configured to require encryption, so it will deny clients that attempt to transfer data unencrypted. Often this can be configured on a user by user basis.

Of these three “secure FTP” protocols, only explicit FTPS is defined by an RFC. In this case it is RFC 4217, released in 2005, even though all these protocols had been in use for years prior.

By the way, I have not discussed what encryption algorithms each of these modes use. They all can use AES if you want, and they often have other good options, too. Do not worry about the algorithm when simply selecting which protocol to use; they are all good from this standpoint.

Comparison

Which kind of secure FTP should you use? It really depends on your business, software availability, and network firewall situation. I personally have no preference for or against any of these protocols on their base merits. Here is a table recapping some of the differences:

Firewall Implications

I have been mentioning some firewall implications for the FTP over SSL variants, but let us go into more detail. The FTP protocol initially opens one connection to authenticate the user and for the user to issue commands to the server. This is all well and good until the user wants to view the contents of a directory or transfer a file. Once either of those activities are requested, there is another connection made. To make things even more fun, the connection can be either from client to server (called passive mode) or from server to client (called active mode). I will just simply call these passive and active data connections, respectively, based on the name of the mode of transfer.

When FTP is always unencrypted, firewalls have been using a little trick to allow the active and passive data connections. They can snoop your connection and see what ports are being decided upon for these connections, and then they quickly allow the connection to get through once. That option is not available when the connection is encrypted. Thus, firewalls need another way to figure things out.

The active data connections (from server back to client) turn out to be relatively easy to firewall from the server side. They always use port 20 (for explicit FTPS) or port 989 (for implicit FTPS) as the source port of the connection (this can be configured to be a different port in many servers). A firewall protecting an FTPS server can just confirm that there exists at least one FTPS connection to that server, and assume that outbound connections with the right source port are okay. Most firewalls just end up being configured to always allow these connections based on the source port, and that is usually okay, too.

The passive data connections, by default on most servers, could be anything inbound between 49152 and 65535 (or sometimes between 1024 and 65535, depending on OS and server). Most organizations do not want to allow this many ports through the firewall to the server. Fortunately, the better FTPS servers also allow this range to be configured. Select a fixed range large enough to handle your traffic load and configure your FTPS server to only use that range for its passive ports. Then allow those ports through the firewall in addition to the main connection at 21 or 990.

If you want to serve the greatest number of clients with your FTPS server, you will want to support both active and passive mode connections. Some clients can figure out how to get one type but not the other through their firewalls. You can never guess which one it will be, though passive mode is more likely because all they have to do is allow the connections out from their network. If for some reason your organization decides to only support one mode and not the other, make sure your clients have some means of being notified! Even better if you can configure your FTP server to simply deny an attempt to open data connections using the mode your firewalls do not permit.

Note that even if you are not using SFTP or FTPS, there are other means of securing FTP connections defined in RFC 2228. Someone could, in theory, refer to a “secure FTP” connection that uses another means besides SSH or SSL to secure the connection via the extensions to the FTP protocol defined by that RFC. In practice (excepting Kerberized FTP), I have seen only the types of secure FTP I described above.

I wrote this article to help you decide which type of secure FTP to use. You need to know all these issues to make the best decision for your company: SFTP versus FTPS, implicit versus explicit, and active versus passive. Perhaps even more important is that you need to know how to correctly communicate your decisions to other technical implementers, and you need to understand the right questions to ask of others intending to implement (or already have implemented) “secure FTP” but who do not know about the differences between these forms.

Just remember: people tend to resist that which is forced upon them. People tend to support that which they help to create. —Vince Pfaff

Policies fail because they are forced upon people without any input to address their needs and concerns. This is counter to a culture of excellence, a mindset where people are enabled to perform their best and do so because they love what they do.

Policies were born out of a need to prohibit undesired activity. We have names for the desired activities we want instead: best business practices, security controls, change management, and others. These are good things indeed, but it does not change that policies exist to prohibit activity. In the great attempt to stop bad things from happening, policies usually end up just preventing excellence from blooming in their “squish activities first, ask questions later” manner.

Policies are often like a law in a corporation and are enforced by the culture of the organization, if not by the threat of disciplinary action. In any case, if someone has a need counter to a policy and does so by direct violation then he will find just how much force the organization can muster to defend the policy, though it may not come immediately if the violation is swept under the table. If you have oppressive policies in your organization, you can be certain there are violations happening out of sight. One way or another, those violations will cause problems of their own in time.

In a natural state, an individual will strive for excellence. People all have something they are very good at, and they want to do it. People that match their careers up to their desires have an awesome potential. This is why new businesses are able to flourish and even show up the established behemoths. A startup is highly focused and filled with people with lots of passion and drive for the product they make. These people have little, if any, policy to stand in the way of their excellence, and when we see them in the news their excellence has already flowered into something beautiful.

Of course, not all startups succeed, but due to those that do we know it happens. This point is important because if the large corporation that is being challenged was really a center of excellence then the startup should have no chance of ever succeeding. An established company has a customer base already, and presumably some credentials in the field. The established company already has the same focus as the startup—let us say they both make Widgets—and it has a lot more money and resources it can devote to producing those Widgets than the startup. However, as the startup proceeds people notice that the new company’s Widgets are better or more convenient or have whatever it is people want in a Widget. Why did not that innovation come from the established company? The difference in the Widgets is due to the difference in individual excellence displayed by employees at each company. Some people say that luck has something to do with it, too, but that is really an excuse for someone that does not work hard to make their own luck.

Policies are not the only cause of the erosion of excellence. Actually, bad policies are more a symptom of bad management, which itself will erode excellence more than policies can. However, the policies are often the visible agent of the oppression of excellence when bad management presides over an organization. The real story behind bad policies is the bad management behind them, and the subject of bad management takes us back to Pfaff’s quote above. Bad managers force orders upon their employees. Good managers become co-creators with their employees.

The roles of policies under bad management are enforcement and control. The roles of policies under good management is facilitating excellence and documenting foregone conclusions. These are policies that are documentation of decided practices instead of being the ruler to slap the back of the hand; these policies might be used to remind people of their own decisions (and referred to just like a checklist) instead of telling them what someone else already decided for them. What do the policies at your company do?

Only one way to find out! I downloaded a fresh copy of the source code for Ruby 1.8.6 and started poking around in random.c, where the function in question lives. My findings are mixed, and I have good news and bad news to report!

The good news is that if the Unix character device /dev/urandom is available then the random number generator is seeded with a value based from a read from that device. The /dev/urandom device is a “secure” pseudo-random number generator device, so the resulting random numbers are going to be pretty good. Note that you are still at the mercy of how well your operating system implements the algorithm for /dev/urandom and how that number generator gets seeded, but generally it works pretty well.

The bad news is that if /dev/urandom does not exist then your random numbers come from a podge of the current time, process id, and a sequence. That will get you a random number good enough for shuffling your deck of cards in a friendly game of Euchre, but you do not want to depend on it for your bank account’s security.

Should you use Kernel#rand on a system without /dev/urandom? That depends. How secure do you want your application to be? If there is any level of money or privacy involved, then I think you better make sure Ruby can find your /dev/urandom device. You also better verify that your operating system’s /dev/urandom algorithm is implemented well.

What if there are large amounts of money involved? All I have spoken of so far is the security of the seed to the random number generator. How good is the random number generator itself? For high security applications, does it matter how good our seed value is if we fail to use a good algorithm to crunch it?

Once again there is good news and bad news on that front. Ruby uses the Mersenne Twister random number algorithm developed in 1997 by Makoto Matsumoto. Wouldn’t you know it that a Japanese algorithm is in use in a programming language from the same islands? The bad news is that it is not an algorithm that is naturally suited for secure applications—speed of number generation is its primary focus. After being seeded the number generator will fall into a pattern. The good news is that you need to generate lots of numbers before that pattern emerges, and it turns out that it can be used in secure applications with some limitations. Please note that I am not a cryptographer, so I am gleaning this information from an external source or two. Use it at your own risk.

But before you run out and use Kernel#rand on your new banking application, heed one more note of caution. High security applications should not be depending on /dev/urandom for seeding another random number generator. If you are going to be feeding a random number generator, why not get a real random number? Systems that have /dev/urandom also have a /dev/random. This latter device generates random numbers that do not get filtered through a random number generator. Assuming your system has a good means of obtaining random bits to regurgitate to /dev/random and your application does not consume them too quickly (reads on /dev/random will block if it runs out of bits), you should seed your random number generator directly from /dev/random for your highest security applications.

It is easy for them [policy makers] to ignore the side effects, such as wasted bureaucratic efforts, delays to real progress, and other general dysfunctions. The side effects are outside of their purview, and highly difficult to measure. These two factors lead to application of the “What isn’t easily measurable, doesn’t exist” rule.

I did some brainstorming about what kinds of changes an upper manager could make to the organization that would provide some relief to the situation without going through the tedious and expensive task of identifying costs that generally go unaccounted for. I aimed for something simple enough that a brave manager can try it out with little effort but could really distinguish his company from most others (and hopefully in a positive way).

The idea is that the policy management team is made somewhat subservient to the teams affected by policy. There is a policy committee that is comprised of the policy team and representatives from various affected areas. The representatives each have one vote on policy decisions, and the policy team members have no vote but exist to facilitate creation, management, and follow-through of the resulting policies. In communicating to the policy committee, management is able to specify what policies must exist but cannot specify any content for those policies. Finally, any team that finds a particular policy too much of a burden may release their own version to address the issue. If the team version cannot be reconciled with the official version within four weeks (times may vary by organization) then that team is no longer bound by the official policy and instead uses their version.

How did I arrive at this? I focused first and foremost on keeping power in the hands of the affected teams. If a company cannot measure the pain induced by policies upon its employees, then simply giving those employees power to correct the issue—even at the cost of potentially losing conformity—seems like sufficient counterbalance to me. People do not want to stand out and be different. They do not want to be tasked with managing their own version of a policy. They will conform with the official version until it becomes cost prohibitive to do so, and they will likely put in healthy efforts at having the existing policy updated first. Only when they feel their attempts at compromise are insufficiently met will they want to branch their policy.

Would this approach work? I recognize that it might bring problems of its own. A company may end up with a situation where every department ends up with its own version of almost every policy. But then again, is that really a bad thing? I do not think so.

I do not mean to belittle all those policies out there, but I would be very surprised if more than 10% of policies in medium-to-large companies provide a net positive return on investment.

Return on Investment

The core of the problem is properly identifying the return on investment by quantifying both positive and negative influences of each policy and official procedure. Most security organizations and managers do not do this well. Instead of considering the benefits of a policy and contrasting them to the costs, they ignore the costs and narrow their focus on an illusionary idea, often false, of the benefit.

Why does this happen? Why is it that people are unable to adequately assess the value of the policy? It is that most people cannot quantify costs. Time equals money, but many people do not know how to perform calculations across that equation. Time is the largest cost ignored by policy makers.

Identifying Costs

Some costs of policies and procedures are:

Additional capital and expense costs accrued directly against the budget: a change control procedure might require tracking tools that are bought and paid for directly against the budget. These are generally noticed and the only costs reported in a typical situation.

Additional time to follow procedure: if a policy requires that all documentation be in the same format, such as particular settings for font, margins, headers, footers, and standard sections (pseudo-justification of which is that everything looks neat and organized for auditors), then everyone that needs to create or modify a policy must be trained on those rules. Additionally, if templates are not provided then everyone must spend time manually formatting the documents to fit the standard. If one is lucky enough to have templates available, then one must not ignore the costs of creating and maintaining those.

Delays imposed: sometimes one must wait until certain times for updates due to over-reaching policies. If one has many sequential tasks to perform, he could be significantly delayed—even by weeks—by trivial work.

Necessary changes never implemented: I regularly see situations where necessary work to prevent future losses is ignored because the policies make it too much of a hassle to get it done. Management is simply not informed of the undone deed except perhaps in passing. No one follows up—to do so would add yet more burdensome tasks to an already overworked employee (overworked because their management thinks the process overhead on their activities is a fraction of what it actually is).

Minimize Policies

All I can suggest for now is to be very vigilant of all your policies. Can you trim them back? Can any be merged together? Do they cover too much? Do they present too much of a burden? Can you lighten known burdens with minimal trimming of the policy’s benefit? If you want to empower your IT staff, be their hero—not their agitator.

Firstly, my recent computer research efforts have been to learn yet more about application security. I know what I know very well, but I also know when I feel a weakness in my knowledge—and I need to do more with application security to advance my knowledge there. I have all the basics now; the next step is application.

Additionally, I need to do more with Rails. I know Ruby very well, and it is absolutely the best programming language I have ever used. I have dabbled with Rails know the basics, but I need to do something useful with it so I have an ongoing Rails project to care for. Without something like that my knowledge in Rails has not broken a mediocre level, and I have fallen behind in recent advances that the Rails core team has delivered recently. An update to this very website into a Rails powered site is very likely!

In summary, the two areas I want to target my development is application security and Rails. Tune in to see what I decide to do with each of these. Drop me a line if you want to chat about it! I love talking about these subjects, so I will love to talk.