Every time I hit upon this problem, I curse the person who decided this.
If you're not sure of what I am talking about, it's the rule in Linux (and other related OSes) that prevents a non-root user from binding (i.e. setting up a server) on a TCP or UDP port below 1024.
Say I'm writing a Java app that wants to listen on a port below 1024 - I either have to run it as root (yuk), or start it as root, and use JNI to drop privs, or redirect ports in iptables ( from 80 to 8070, and run the Java app listening on 8070). All of these are just annoying and unnecessary work-arounds.
Now, every sane application (such as Apache, which needs to bind on port 80 usually, or Postfix - port 25) will drop the rights of root irrevocably once it has bound as root. The reason for this is security.
A quick lesson on how to suck eggs.
Most daemons (services) have their own user account in Linux - Apache runs as apache, Postfix as postfix - well, I'm sure you can work the rest out. The thinking behind this is that if a hacker is able to utilise a buffer overflow and end up with a shell on the system, they'll only end up with a shell running as the user that the process was running as. If the daemon was running as root - well, they've immediately got the keys to the whole kingdom. You'd better hope they're nice and don't trash your box, and you'd better make sure you reinstall, and restore all your data.
With a user account that's only used for 1 service, the only things they have access to  are usually the config files, spool directories, and binaries that the service uses. And they've already broken your service, so there's not much problem there. They are unable to:
- Delete files belonging to other users.
- Kill processes belonging to other users.
So, we're all agreed. It's a pretty good idea to run each service (daemon) as a different user.
( I'm going to use Apache for the rest of the examples. But any server that binds to a port, and handles requests is the same. )
Now, what happens currently with Apache?
Well, it has to be started as root (due to this archaic rule of <1024), but the moment it's bound, it switches user to "apache".
So, one might ask: What's the point of that first bit.
Indeed. It seems like an unnecessary, and pointless step.
If you speak to a crusty old Unix admin (and the arguments to my suggestion will mainly be based on the lines of tradition, that's how it always has been), they'll tell you that that rule is in place to stop someone binding something to a "well known port" on a server without being allowed to.
One thing to mention though - if someone is bound to port 80, no-one else can bind to it until that first program exits. So, if I'm a malicious user, on a box, with this <1024 rule not in place, I can still only start my webserver on port 80 if the "proper" port 80 server crashes.
Well yes, that's all very well. I can envisage a few server situations, and I'll explain why none of them require that you have this ridiculous rule.
Servers where it would be embarrassing/costly to have the wrong webpages served on port 80:
- Multi-user server, with untrusted users.
If you don't trust your users, and you're letting people run commands on your server, you're probably running a public shell server.
And if you're doing that, you should be (if you're not already) using some RBAC like GRSec, or SELinux to control things like what programs can bind to what ports, what users can run what programs, etc. The 1024 rule is redundant in this example then.
 With RBAC, you can specify that only /usr/sbin/apache2 may bind to tcp/80 as user apache, for example.
- Multi-user server, with trusted users.
So, perhaps you're running a build farm for your company, and company users are allowed to SSH on, and compile stuff? In this case, you think a user would be trying to mess around, and crash your Apache, and then bind their own one to port 80 before you've noticed? I think in that case, your users aren't that trusted. Or, if you're worried that a user account could be hacked, look into RBAC.
- A single purpose server
This might be something like just an Web/DB/SMTP server for a company. If that's the case, there's probably only a few admin users who are meant to access the box for maintenance. And, as they've probably got root privs anyway, if they're going to start messing around, you've probably got more problems anyway.
Servers where it would just be a small inconvenience to have the wrong webpages served on port 80:
- A dev box/home PC
If the <1024 rule didn't exist on my home PC, and I found that someone had managed to crash my copy of Apache, and start their own Apache server there instead, I'd a: be impressed (crashing Apache totally is hard), b: be annoyed, and c: kill off their process, disable their account, and restart Apache. But it's not that important, so I wouldn't worry a lot.
Basically, I don't think it's necessary in these days. Sure, when running a Unix box was hard, and you had lots of people sharing 1 box, maybe. But these days where anyone can install a copy of Linux, it's an anachronism.
It hinders security, rather than helping it.
If you really need to make sure that the right people can't do anything other than what you want them to, then look into a proper solution for it - RBAC.
The <1024 = root rule is a really blunt tool.
So please, can we get this silly, outdated, almost useless restriction removed, or at least made into an option in the kernel config?
[*] Restrict root to binding to <1024