The Node

Archive for June, 2011

BuildBrighton is organising a Mini Maker Faire at the Brighton Dome on the 3rd September 2011 from 10:00am to 5:00pm. Tickets are free for visitors. A reader from Wired magazine described the event thusly: -

A Maker Faire is the place to come for anyone who makes things themselves, or wants to have fun trying. You can expect to see all kinds of things at a Maker Faire. Sometimes useful, often toys, usually surprising, always fun. From electronics, soldering and robots, to knitting, metal and woodwork, if you can make it, it will be there.

This Mini Maker Faire is being organised independently by the Brighton Maker community to showcase all the best local creative talent and energy.

It’s a great day out for kids, families, or anyone who wants to learn how to make things, to be inspired or to meet other makers. There’ll be exhibitions and demonstrations, workshops where you can learn new skills, and make things to take away.

Things like this are usually organised by MAKE magazine and have been events held mostly in the US. However, recently with the popularity of things like Linux, Arduino and developing mobile applications and devices, such events have seen a rise in popularity in the UK. I’ll be attending to take part, I hope you will too :-)

Please see here for more details.

PodGrab, my Python open source command-line downloader has been updated to version 1.1.0. This will probably be the last version aside from minor bug fixes. New in this version is import/export support for OPML files and more media formats supported than just MP3. It now supports pretty much every type of downloadable audio/video media podcast format out there, aside from pure text feeds. The full list is MP3, M4V, OGG, FLV, MP4, MPG/MPEG, WMA, WMV and WEBM. It currently uses SQLite for it’s subscription database, but if I get enough requests or I want it for myself I’ll add MySQL database support.

You can grab the new version here.

Fedora 15 is the first Linux distribution to ship with the new GNOME 3.0 user interface. This is what most Linux distributions barring Ubuntu will no doubt be using in the near future so it’s pretty important.

I’ve used Fedora 15 on the desktop for a couple of weeks now. I was expecting to love it, mainly because although I wasn’t all that taken with Ubuntu’s new Unity interface, I didn’t hate it as passionately as some. And hey….different isn’t necessarily bad, right?

Right?

This is so frustrating. I’ve been using Fedora since version 6 and I much prefer the actual “Linux” side of it to Debian-based distros. You’re reading this right now on a Fedora 15 server, for example. The GUI side has always been pretty “meh” but Canonical really showed how damned old everything looked. It was high time the GUI was overhauled and Ubuntu – regardless of how you feel about the Unity interface – pretty much paved the way. GNOME was conceived when Windows 95 was the height of UI design, and simple subtraction will tell you that was a long time ago.
So great, GNOME 3.0 with a completely – some might say radically – new interface.
The good is that it’s Fedora Linux. That is to say, it’s Red Hat underneath and I’ve always used Red Hat since version 6.0 so I’m comfortable with it. As a server it’s still as fine as it ever was. The only jolt so far being the new systemd init daemon rather than the more traditional sysvinit. This will take some getting used to, but for now the compatibility layer (which I assume is there) shields you from most of the shock. It’s still disconcerting to not have a ‘/etc/inittab’ file, however.
But none of this really matters – Linux is a proven technology on the server and is likely to remain so. So that leaves the desktop which frankly, I’ve not used since Fedora 12.
Now, if Fedora 15 was actually bad, I wouldn’t be so frustrated. The reason being that GNOME 3.0 is great. It’s slick and the new interface paradigm is also great.

Or at least it would be if the Fedora team had finished it.

That’s my beef here. Fedora 15 has shipped with a broken interface. The sweet, gooey command-line centre is fine, but GNOME 3.0 is unfinished….visibly unfinished. For example, the main interface bar at the top is a tasteful dark colour. The dialog boxes and windows on the other hand are lifted straight from GNOME 2.3, complete with light-grey colours and big, blocky icons.
I really harshed on the Linux Action Show guys for slating Fedora 15 from the angle they did and, while I don’t share their ire with the Fedora distribution in general (Debian fanboys, natch :-)), they were spot on about GNOME 3.0.
Dialog border colours don’t match, things are labelled inconsistently or badly. Lots of little things that in isolation wouldn’t be a big deal but they mount up quickly. Using Fedora 15 as a desktop system is a little like being flogged to death by silk bootlaces. You can set terminal windows to be transparent, but it doesn’t seem to work because of the new desktop. The design team didn’t seem to know where to display mounted volumes. Streaming a video under VLC from a sever is smooth on Ubuntu 10.10 but buffers frequently on Fedora 15. The list goes on.

The really annoying thing is what does work in GNOME 3.0 works really well. The overall philosophy is much more integrated than Unity is – indeed it makes it seem positively bright and clunky like spit-covered Duplo blocks. The “log off” dialog box looks really cool and you wish the rest of the dialog boxes looked that way, yet they don’t. The way the desktop zooms out rather than the use of work-spaces is brilliant, especially as all the windows are still non-modal and visibly running. Why bother re-sizing windows when you can do that? Inspired.

It’s bewildering that some things are so great and others so clearly not. In using it, you sometimes catch glimpses of the future and I imagine Fedora 16 will be great. The question is, why release this? It’s open source, has no stockholders to answer to and doesn’t need to turn a profit. The mystery isn’t why Fedora 15 is broken, I get that…they just didn’t have time to finish. The real mystery is why release something so obviously and clearly unfinished at all? Not broken as such but certainly not ready to hit the FTP servers. If I had been in charge and I had used this, I would have said “not for another six months, fellas”. Why stick to a firm six-month release cycle if you’re just going to put out occasionally dubious releases? Why not wait a couple of months. I’d have been just as happy to use Fedora 15 in September.

If you’re using Fedora on a server – go for it. The new dynamic firewall looks interesting and systemd appears to be far more versatile than sysvinit and you’ll get all the regular package upgrades and it’s stable as it ever was. But on the desktop? Not yet.

But probably soon :-) Say, in another six months. As they say, one to watch.

ClamAV antivirus software is in most Linux distribution repositories. The problem is, ClamAV gets updated faster than the new versions can be added to the respositories, so every time you run a freshclam virus database update, ClamAV will inform you that it’s not the latest version. Not ideal, especially when you’re talking about security.
The only way to get the latest latest version is to download the stable source code release direct from ClamAV’s website and install it manually. Which is what we’re going to do :-)

First, you’ll need to grab the source code. You can download the tar.gz file here for the latest stable release version. As of the time of writing, the latest version is 0.97.1. I’m just going to copy the download URL and download it with wget.

wget http://downloads.sourceforge.net/clamav/clamav-0.97.1.tar.gz

Then you’ll need to unpack the archive.


gunzip http://downloads.sourceforge.net/clamav/clamav-0.97.1.tar.gz
tar -xf clamav-0.97.1.tar
cd clamav-0.97.1


Okay, you should now be in the ClamAV directory. I want to install my new version of ClamAV in ‘/usr/local/clamav-0.97.1′. So I’ll need to configure it to install to that directory. If you run into dependency problems, you’ll probably need to install GCC/Make if it isn’t already. If you do need to do this, you can do it under Fedora with: -

yum install gcc make

…and under Debian/Ubuntu with: -

sudo apt-get install gcc make

Once you’ve got all the prerequisites installed, you can run the configure script. Make sure you’re still in the sub-directory where you unpackged the archive and run: -

./configure --prefix=/usr/local/clamav-0.97.1

Once this is complete and you have no errors, you can run: -


make
make install


This will install the new version of ClamAV to the directory you specified in the configure script. If you have the version of ClamAV installed from your distribution’s repositories, you have two choices. You can either keep the repository version and use aliasing to run the version you want or you can uninstall the repo version and add append the path to the system $PATH environment variable. I’ll show you both.

Assuming you want to keep the version of ClamAV you have in case it ever gets updated via the repositories, you’ll need to copy your ‘/etc/freshclam.conf’ to the new location.

cp /etc/freshclam.conf /usr/local/clamav-0.97.1/etc

However, if you try to run ClamAV from anywhere but the ‘/usr/local/clamav-0.97.1/bin’ directory where the program executables are, you’ll still get the warning that ClamAV is out of date. This is because the system $PATH variable finds the old version first as it’s part of the system path. We want to override this and run our new manually installed version. Since we’re only going to be running ClamAV with the root user so that we have permissions to scan the entire file system, we’ll add a new alias.

vi ~/.bashrc

Then add the following lines:-


alias clamscan='/usr/local/clamav-0.97.1/bin/clamscan'
alias freshclam='/usr/local/clamav-0.97.1/bin/freshclam'
alias clamd='/usr/local/clamav-0.97.1/sbin/clamd'


Aliases are very handy. Basically, when the alias is typed as a command, it points to the command we specify, regardless of what is in the system $PATH environment variable. Save this file and logout. The new settings for the ‘.bashrc’ file are only picked up on login of that user. Once you login again you should be running the latest version of ClamAV.

If you want to remove the repo version of ClamAV, you’ll need to uninstall it. Under Fedora, use: -

yum remove clamav-*

…and under Debian/Ubuntu use: -

sudo apt-get remove clamav-*

Once this is done, you’ll need to add the new ClamAV to the system path. Under Fedora, this is: -

vi /etc/profile

Under Debian/Ubuntu, this is under: -

vi /etc/environment

Find the following section, or something that looks similar: -


export PATH USER LOGNAME MAIL HOSTNAME HISTSIZE HISTCONTROL


Above this line add the line: -

export PATH=$PATH:/usr/local/clamav-0.97.1/bin

This appends the path to the binary directory of our new ClamAV to the system path. Again, for this to take effect, you’ll need to logout and log back in again. You can verify the path by typing: -

echo $PATH

When you simply run freshclam to update, it should find the configuration file under it’s own /etc directory and you should now be able to scan the filesystem using ‘clamscan’ without having to type in the absolute path to the new binary executable.

One thing you might want to be aware of is that quite a lot of distributions have pre-set up cron to periodically update the version installed from the distribution repositories. But of course now, even though you have the newer version of ClamAV installed in /usr/local, it will keep issuing system log or e-mail alerts saying it is out of date. This is all the fault of the default cron task that runs it’s own absolute path to the repo-installed version of ClamAV. You can deal with this in two ways, either simply remove the ClamAV that is already installed with: -

yum remove clamav-*

or disable the cron task. This is a small script file which resides in ‘/etc/cron.d/clamupdate’ (at least in Fedora). Either delete this file or commend out it’s contents.

With residential broadband, you only get one IP address typically and that isn’t even usually static – pretty unsuitable for servers at home. You can get round this problem with dynamic DNS services which allow you to update your DNS record each time your ISP-assigned IP address changes. But this still means that the only machine visible to the Internet at large is that server and only that server. All your other machines behind your router have DHCP-assigned IP addresses which are private to that internal network, like 192.168.1.43. These machines are not visible to the Internet, only the router is.
A router is really just a specialised computer server that directs Internet traffic to the right destination. A Linux server itself might be the gateway server. Indeed, to get dynamic DNS to work, you also have to configure traffic to route the traffic from the outside to your Linux server.
For the purposes of this article, I’ll assume the router and gateway are the same machine. If not, the router will be configured in such a way that it routes all traffic to the gateway server so that, in practical terms, they are the same.

So what happens if you want to have a web server on a separate machine on your network? Well, technically you can’t – it only has a private (or ‘local’) IP address. But you can cheat using a method called port forwarding. This allows the gateway server to change the destination and source of the data packets based on certain criteria dynamically which allows packets destined for the gateway server to be rewritten for the new web server with it’s private IP address. The gateway server also needs to be able to send packets back out of the local network to the originating client because the internal web server will only be talking to the gateway server so it’s packets source address needs to be rewritten.

So we need a mechanism that that distinguish between traffic meant for the gateway server and traffic

So if Internet traffic is hitting the gateway server with it’s static IP address (or at least host name and DynDNS), that traffic is destined for the gateway server. It doesn’t know anything about the web server on the internal network. The gateway server’s firewall has to decide which packets are for itself and which packets should be passed to the web server which only the gateway server knows about. The easiest way to distinguish the traffic is send the web server traffic to another non-standard port. I’ll use 7070.

Think about two web servers in this scenario, the gateway and internal. Normal web traffic will hit the gateway server (public IP address of 89.232.33.123 and therefore visible to the Internet) on the standard port 80. Web traffic for the internal web server will hit port 7070 on the same gateway server and the it knows to re-route the destination for each packet to the internal private IP address of 192.168.1.43 because of the non-standard port.

So if a client requests 89.232.33.123:80 they’ll get a reply from the gateway server as normal. If a client requests 89.232.33.123:7070 (note the same IP address of the gateway) the client will get a reply that appears to be from 89.232.33.123 but is actually from 192.168.1.43, the internal web server.

If you want more than one machine visible to the outside world but only have the one static IP address, you can configure the gateway Linux server to to route traffic to anywhere you like using iptables. While iptables is primarily a firewall, the rules you can set up to deny or allow a specific packet are flexible enough to allow traffic routing as well.

On the gateway server which is accessible from the Internet, open the iptables configuration file.

vi /etc/sysconfig/iptables-config

Add the following line: -

IPTABLES_MODULES="iptable_nat"

This adds the Network Address Translation (NAT) kernel module to the system so that iptables can use NAT. This will allow iptables to hold the packets before comparing them to the NAT table we will create.

Open a new file under /etc/sysconfig called iptables.NAT-RULES.

vi /etc/sysconfig/iptables.NAT-RULES

This will hold all our iptables configuration rules. First thing we need to do define our network address translation rules. The first rules will be to reset everything so that nothing gets in.


*nat
:PREROUTING DROP [0:0]
:OUTPUT DROP [0:0]
:POSTROUTING DROP [0:0]


Next, we’ll switch on masquerading. This is known as “many-to-one NAT”. In other words, traffic from all devices on one or more protected networks will appear as if it originated from a single IP address on the Internet side of the firewall. The masquerade IP address always defaults to the IP address of the firewall’s main interface. The advantage of this is that you never have to specify the NAT IP address. This makes it much easier to configure iptables NAT with DHCP, which we are doing as the internal web server has a DHCP-assigned address rather than a static address.


-A POSTROUTING -o eth0 -j MASQUERADE
-A PREROUTING -i eth0 -p tcp --dport 7070 -j DNAT --to-destination 192.168.1.43:80
COMMIT


The next line is the real “meat” of the NAT table. If the gateway gets a data packet on port 7070 of the main ethernet card (eth0), it should send it on to 192.168.1.43 on the internal network on it’s port 80 (HTTP traffic). These rules are then committed to the NAT table. Next, iptables deals the filters – the main firewall configuration which will allow or deny data packets.


*filter
:INPUT DROP [0:0]
:FORWARD DROP [0:0]
:OUTPUT DROP [0:0]


Again, you can see the configuration is reset first. Next, comes the first section of filter rules which are the INPUT rules, or the rules governing what the firewall does with incoming packets.

-A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
-A INPUT -p icmp -j ACCEPT


The first line allows data packets through to wherever they want to go assuming that the connection has already been established. The next line simply states that we will accept ICMP or (‘ping’) packets used for network diagnostics. The next section deals with the actual ports that the firewall will accept.


-A INPUT -m state --state NEW -m tcp -p tcp --dport 22 -j ACCEPT
-A INPUT -m state --state NEW -m tcp -p tcp --dport 7070 -j ACCEPT


The first line above appends a new rule to the rule-set accepting new connections on TCP port 22 (which is the port for remote access SSH). The next line does the same for our custom 7070 port which will handle web traffic from the internal web server. Next we come to the FORWARDing rules for the firewall, what it does with the packets it’s accepting.


-A FORWARD -m state --state ESTABLISHED,RELATED -j ACCEPT
-A FORWARD -p icmp -j ACCEPT
-A FORWARD -i lo -j ACCEPT
-A FORWARD -o eth0 -j ACCEPT
-A FORWARD -p 7070 -m state --state NEW -m tcp -p tcp -d 192.168.1.43 --dport 80 -j ACCEPT


Again, it’ll accept established connections for forwarding, these are default settings for the most part. The last line is the most important, where it forwards the port 7070 to the internal web server.

Here is the completed iptables rule-set for NAT and filtering.


*nat
:PREROUTING DROP [0:0]
:OUTPUT DROP [0:0]
:POSTROUTING DROP [0:0]
-A POSTROUTING -o eth0 -j MASQUERADE
-A PREROUTING -i eth0 -p tcp --dport 7070 -j DNAT --to-destination 192.168.1.43:80
COMMIT
*filter
:INPUT DROP [0:0]
:FORWARD DROP [0:0]
:OUTPUT DROP [0:0]
-A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
-A INPUT -p icmp -j ACCEPT
-A INPUT -i lo -j ACCEPT
-A INPUT -m state --state NEW -m tcp -p tcp --dport 22 -j ACCEPT
-A INPUT -m state --state NEW -m tcp -p tcp --dport 7070 -j ACCEPT
-A INPUT -j REJECT --reject-with icmp-host-prohibited
-A FORWARD -m state --state ESTABLISHED,RELATED -j ACCEPT
-A FORWARD -p icmp -j ACCEPT
-A FORWARD -i lo -j ACCEPT
-A FORWARD -p 7070 -m state --state NEW -m tcp -p tcp -d 192.168.1.43 --dport 80 -j ACCEPT
-A FORWARD -j REJECT --reject-with icmp-host-prohibited
COMMIT


This concludes the quick n’ dirty guide to port forwarding with iptables :-)

The ‘dd’ command in Linux is pretty mysterious and to most casual users has no real use. ‘dd’ stands for ‘Data Description’ and is essentially a very low level utility for the low-level copying and conversion of raw data.
As you should know, Linux treats everything as a file. If you keep this in mind when looking over the dd command’s functionality, you can probably imagine some of the neat things you can do, especially in terms of bash scripting :-)

If you do an ‘ls’ under /dev you’ll see lots of files. Same for /proc. The operating system creates them on the fly if you try to read them and you’ll notice that their last modifed date is always up to date. Very basically, the /proc directory gives you a view into the running kernel, which can be useful for monitoring performance, discovering system information, finding out how the system is configured, and changing that configuration. Everything under /dev is a hardware device attached to the system.

The dd command is different from other Linux programs in that it doesn’t follow the standard input/output/error streams of Linux. What does that mean? Under UNIX, all programs that run are given three open files when they are started by a shell: STDIN, STDOUT and STDERR, standard in, out and error. Standard out and error are usually set to the terminal you’re actually using at the time so you can see the output and standard input is usually the command line arguments to any command. Simply put, input redirection means supplying input to some program from a file which would otherwise take it from standard input device, such as a keyboard. Output redirection means redirecting the output of some program to a file rather than letting the program output it to standard output such as a monitor.

Consider this command: -

ls -l > /path/to/some/file.txt

This will redirect output from the ls command to a text file. Or…

grep -i Bob < /etc/passwd

This will perform a grep (search) for the user Bob taking the /etc/passwd file as it's standard input and print the result. A slightly more complex command would be: -

grep -i Bob < /etc/passwd > /tmp/users

This will take /etc/passd as the STDIN for the grep command and redirection the results (STDOUT) to the file /tmp/users. If you think about it, this makes it possible to create long chains of simple commands which can pass their results to another program as it’s input. Many complex tasks under Linux (and indeed UNIX) can be performed in this way, especially if you’re using it in conjunction with bash scripting. This, to me, is the real power behind Linux.
Keep in mind that redirecting output ‘>’ to a file will destroy the file if it already exists. If you want to append output to an existing file, use ‘>>’.

Piping is very much like IO redirection, except it acts on processes. This way the output of one program can act as the input to another. Consider: -

ls -la /etc | grep passwd

will print the file details the contents of /etc and then pipe (|) the standard out as an input to grep, which will filter out and display just the file details of the ‘passwd’ file to standard out, the terminal display. You can use IO redirection and pipes together to perform complex tasks. So you could print the file type details of the above command to a file with: -

ls -la /etc | grep passwd > /tmp/passwd-file-details.txt

So the contents of /tmp/passwd-file-details.txt would be: -

-rw-r--r-- 1 root root 1754 Jun 8 11:56 passwd

This makes the Linux command line an extremely powerful tool to manage and automate complex tasks without having to resort to full-on programming.

We can use this sort of power with the DD command. Unlike usual *nix with IO redirection and pipes as above, dd has distinct parameters.

dd if=[INPUT] io=[OUTPUT] bs=[BYTE_SIZE] conv=[CONVERSION_TYPE]

As you can see, the syntax is not what you’d expect from Linux, based on what we’ve discussed above. We can use special “files” under /dev for use with the dd command.

• /dev/null – Any output redirected to /dev/null will be destroyed. It’s like a black hole, information can go in but nothing gets out. It’s useful if you want to delete standard output to suppress warning messages for example. If you use /dev/null for input, say to a big file, it will empty that file.
• /dev/urandom and /dev/random – This is the interface to the system kernel’s random number generator and will produce an endless series of random numbers.
• /dev/zero – This produces nothing but zeros.

Now we have the overview of the dd command and some of the special devices under /dev, we can do some useful things with dd. Such as…

1. Wiping a hard-drive clean.

dd if=/dev/zero of=/dev/sdb bs=1M

This will use the dd command to use /dev/zero which produces only zeros and outputs this to a disk partition with a byte-size of 1MB chunks, effectively replacing all data on the drive with zeroes. If you want to securely wipe a drive before selling it, for example, you could use…

2. Securely wipe a hard-drive clean.

dd if=/dev/urandom of=/dev/sda bs=1M

This does the same as above, but replaces all data on the drive with random data from /dev/random. You can simply securely wipe a single partition by appending the partition number to the hard drive device: -

dd if=/dev/urandom of=/dev/sda2 bs=1M

If you corrupted your Master Boot Record (MBR) on a drive, you can…

3. Delete the Master Boot Record (MBR).

dd if=/dev/zero of=/dev/sda bs=446 count=1

This deletes the first 443 bytes of drive /dev/sda, which is the size and location of the master boot record.

4. Make a CD .iso file from a disk partition.

dd if=/dev/sdb1 of=/home/bob/my-partition.iso bs=2048 conv=notrunc

CD sectors are 2048 bytes long, so this copies the partition sector by sector, although you’d have to use a tool such as Brasero to actually burn the ISO image to CD.

5. Restore a CD ISO file to a partition.

dd if=/home/bob/my-partition.iso of=/dev/sdb2 bs=2048 conv=notrunc

5. Copy the contents of RAM to a file.

dd if=/dev/mem of=/root/memory-dump.bin bs=1024

6. Make a file of 1024 random bytes.

dd if=/dev/urandom of=/home/bob/random-bytes.bin bs=1 count=1024

7. Duplicate a disk partition to another.

dd if=/dev/sda1 of=/dev/sdb1 bs=4096 conv=noerror

8. Save a partition to a file.

dd if=/dev/sdb1 of=/home/bob/my-partition.img bs=4096 conv=notrunc,noerror

9. Restore a partition from a file.

dd if=/home/bob/partition.img of=/dev/sdb1 bs=4096 conv=notrunc,noerror

8. Make an ISO image of a CD.

dd if=/dev/cdrom of=/home/bob/my-music-CD.iso bs=2048 conv=sync

There are many things you can do with the dd command against /dev files (devices). You can do similar things with /proc which is the real-time interface to the system kernel.

1. View your virtual memory.

dd if=/proc/kcore | hexdump -C | less

This takes the /proc/kcore “file” and pipes it to hex output which is then piped to the ‘less’ command for page-by-page tabbing.

2. Have a look at your interrupt table.

dd if=/proc/interrupts | hexdump -C | less

This does the same as above, but for your system’s interrupt table.

You can even have the dd command act against files for data conversion. Here, we convert a file from lower-case to upper-case.

dd if=/home/bob/data-lower.txt of=/home/bob/data-uppper.txt conv=ucase

Pretty neat eh? Well, I hope this has given you some ideas for bash scripting using IO redirection, pipes and the dd command.

Firstly, be warned – this is a long article. LDAP is non-trivial, but if you’ve mastered NIS/NFS, this is the next step :-)

The Lightweight Directory Access Protocol, better known as LDAP is what is known as a directory service, much like Windows’ Active Directory. Indeed, Active Directory is the Windows implementation of LDAP so you’ve probably heard to LDAP without even being aware of it :-) Like Active Directory, you can store all sorts of information in an LDAP database and because it’s standards-based and cross-platform, you can read this information securely from anywhere using any LDAP-aware application. Strictly speaking, LDAP isn’t a database system but a protocol used to access information stored in an information directory also known as an LDAP directory. LDAP is particularly useful for storing information that you wish to read from many locations, but update infrequently.

While you can store all sorts of information in an LDAP directory, we’re going to use it to store Linux user login credentials and access the LDAP directory whenever we want to login to any machine on our network rather than the local user credentials. Assuming you’ve been wanting to set up something like this, you may be asking why you don’t just use NIS/NFS. In the past encrypted passwords contained in ‘/etc/passwd’ or ‘/etc/shadow’ were distributed by NIS, thus making it act as an authentication as well as authorisation server. Nowadays, this technique is not so suitable because it involves moving the password over an open and insecure network where a sniffer could capture it and, with the processing power currently available, unencrypt it in a few machine hours. NIS is fine for simple solutions on an trusted intranet but unsuitable as an enterprise solution. Also, unlike NIS, LDAP is expandable, which means that the type of data it can represent is not established beforehand by the protocol, but can be changed by the administrator by adding schemas. NIS operates on “flat” domains and is therefore unsuitable for large organisations which due to their nature may be organised hierarchically in a “tree” structure.

Fedora ships its own LDAP based server but we will be using the OpenLDAP implementation with Berkley Database (bdb) as the database backend. Data is entered into the LDAP server via plain text LDIF (LDAP Data Interchange Format) files.

This exercise assumes that you’ve got a networked Fedora install working and that you’ve assigned it a host name that is reachable via DNS. If you don’t have a static IP address that you can assign a host name, you can set one up using Dynamic DNS. Make note of the IP address of the server – for the purposes of this, I’ll assume it’s ’192.168.1.2′ internally as we’re using this on a home network.

If you want to test that your server which you’re going to use as a LDAP server on your network has a fully qualified domain name (FQDN), you can run: -

dig [your_host_name]

Configuring the LDAP server

Obviously, you’ll also need to be root for this :-) Okay, let’s get started. First you’ll need to install the OpenLDAP packages.

yum install openldap-servers migrationtools openldap openldap-clients openldap-devel nss_ldap

Next, you’ll need to create an LDAP root user – this is similar to a regular Linux root user in that it has entire control over your LDAP directory service. Do so with: -

slappasswd

This will ask you to enter a password and return the SHA-encrypted hash of the password. It will be something like: -

{SSHA}MP0BeMJzmCoCi5olBhwcRDYJaGBFgN5D

Make a note of this string, as you will need it later on.

First you need to set up an LDAP user. Enter the following commands: -

useradd ldap.ldap

and set a password for it.

passwd ldap

Next, you need to set up a directory for your database and give is the right access permissions. We’re going to use the domain ‘ldap.test.lan’

mkdir /var/lib/ldap/ldap.test.lan

…and similarly give our LDAP users access permissions: -

chown -R ldap.ldap /var/lib/ldap/ldap.test.lan

The base configuration

Now we need to set up our base configuration. You’ll need to delete the /etc/openldap/slapd.d directory or rename it as else Fedora will ignore our configuration and use the one under slapd.d. I’m just going to rename it.

mv /etc/openldap/slapd.d /etc/openldap/slapd.d.DEFAULT

Then we’ll create a configuration file. This is actually the “old” way of doing things, but it’s far simpler than running through the mess of auto-generated configuration files under the slapd.d sub-directory.

vi /etc/openldap/slapd.conf

Add the following, changing the example host for yours.


include /etc/openldap/schema/core.schema
include /etc/openldap/schema/cosine.schema
include /etc/openldap/schema/inetorgperson.schema
include /etc/openldap/schema/nis.schema

database bdb
suffix "dc=ldap,dc=test,dc=lan"
rootdn "cn=admin,dc=ldap,dc=test,dc=lan"
rootpw {SSHA}MP0BeMJzmCoCi5olBhwcRDYJaGBFgN5D
directory /var/lib/ldap/ldap.test.lan

TLSCipherSuite HIGH:MEDIUM:+SSLv2:+SSLv3:RSA
TLSCACertificateFile /etc/openldap/ssl/ldap-server.pem
TLSCertificateFile /etc/openldap/ssl/ldap-server.pem
TLSCertificateKeyFile /etc/openldap/ssl/ldap-server.pem
TLSVerifyClient allow


This creates database of the default type ‘bdb’ (Berkeley Database) using the domain suffix ldap.test.lan made up of domain components (DCs) ‘ldap’, ‘test’ and ‘lan’. It then creates the root user with a common name (CN) or nickname of ‘admin’ who, as expected, is part of the ‘ldap’, ‘test’ and ‘lan’ DCs. Lastly, it defines the encrypted version of the LDAP root password you created earlier as well as the location of the LDAP database within the file system and tell LDAP where to find the certificate and key files for encryption.

Because we are using LDAP for authentication across a network, we’ll need to encrypt the traffic, which is what the encryption key entries above are all about and is one of the key differences with something like NIS. We need to tell Fedora how to start the secure LDAP daemon which is done by editing the sysconfig entry for ldap: -

vi /etc/sysconfig/ldap

Add the following line or uncomment the one that’s there and set it to ‘yes’.

SLAPD_LDAPS=yes

Save this and return to the command line. We can now run LDAP on default port of 389 with TLS. Of course, we’ll have to create some encryption keys for this. Run the following which will create encryption keys in the location we specified in the ‘/etc/openldap/slapd.conf’ file:-

openssl req -newkey rsa:1024 -x509 -nodes -out /etc/openldap/ssl/ldap-server.pem -keyout /etc/openldap/ssl/ldap-server.pem -days 365

Make sure the following sections are filled in with your host details.


Common Name (eg, your name or your server's hostname) []: ldap.test.lan
Email Address []: user@test.lan


You also need to set these directories to something that the ldap user can read: -


chown -rf root.ldap /etc/openldap/ssl
chmod -rf 750 /etc/openldap/ssl


Check that the ‘/etc/openldap/ssl/ldap-server.pem’ private key file exists and is of the right format: -

cat /etc/openldap/ssl/ldap-server.pem

It should look something like: -


-----BEGIN RSA PRIVATE KEY-----
MIICXAIBAAKBgQDj64XGJe1uA1Ybr/1kWTsQcxktU7W9i29OkbmFwI1hc8qYXuO5
qAAGCFHHupInzy9uoXJVvGW3yEw0gasLR6hzyC2+1b8vfG3Eb0yN+Yt4mGp03iiX
c0pzQrEw+HxYcsA0KAUCQDKCo5OTBB0FLpH+ZgTqkeBabt3lNYFphAqEqLyC6q10
+WMlWY/jvLyQYldbvP3ENgahGKlv99SKytSb9MFQlnc=
-----END RSA PRIVATE KEY-----
-----BEGIN CERTIFICATE-----
MIIC+DCCAmGgAwIBAgIJAKhuyXeddEVVMA0GCSqGSIb3DQEBBQUAMFwxCzAJBgNV
BAYTAkdCMRIwEAYDVQQIEwlCZXJrc2hpcmUxEDAOBgNVBAcTB05ld2J1cnkxFzAV
5/ncthk9QyZwLrz1/WEC/9qbST/aYGEz4lOMc8tPu9vKh9CAYI42J6zu51Y=
-----END CERTIFICATE-----


Each client will need the ‘certificate’ part of the file, so we can use the grep command to extract this information.

grep -A 100 CERTIFICATE /etc/openldap/ssl/ldap-server.pem > /etc/openldap/ssl/ldap-client.pem

The ‘/etc/openldap/slapd.conf’ file now references our key to activate TLS encryption of all incoming connections to the server. The LDAP daemon won’t start properly unless the files in the ‘/etc/openldap/ssl directory’ are owned by the ldap user. We need to make sure of this with: -

chown -R ldap.ldap /etc/openldap/ssl

The next step is to configure the migration tools used to migrate your existing users into LDAP. To do this, you need to edit the migrate_common.ph file: -

vi /usr/share/migrationtools/migrate_common.ph

The only things you need to change are the lines below. The DEFAULT_BASE should be set to your host.


# Default DNS domain
$DEFAULT_MAIL_DOMAIN = "localhost";

# Default base
$DEFAULT_BASE = "dc=ldap,dc=test,dc=lan";


You now need to copy a default DB_CONFIG file which sets cache and tuning options for the Berkley database backend (this also needs to be writeable by the ldap user). This file may be under a different directory depending on your version of Fedora, but you can find it with: -

find /usr -name "DB_CONFIG.example" -print

Assuming this is found in ‘/usr/share/openldap-servers/DB_CONFIG.example’, you need to copy this example file to your /var/lib/ldap directory and rename it as DB_CONFIG.

cp /usr/share/openldap-servers/DB_CONFIG.example /var/lib/ldap/DB_CONFIG

Once you’ve done this, you need to make sure that the ldap user has access to it: -

chown -rf ldap.ldap /var/lib/ldap

Lastly, test your configuration by running the command, and check that it returns ‘config file testing succeeded’:-

slaptest -u

If you have no problems, you’ve successfully completed your basic LDAP server configuration. You should now be able to start your slapd service with the following: -

service slapd start

You can test that the service started correctly and is listening on the LDAPS port with: -

netstat -lt | grep ldap

You should get back something like: -


tcp 0 0 *:ldap *:* LISTEN
tcp 0 0 *:ldaps *:* LISTEN


If everything looks good, set slapd to start on boot with: -

chkconfig slapd on

To test the server configuration, run: -

ldapsearch -x -b '' -s base '(objectclass=*)' namingContexts

which should return success as below: -


# extended LDIF
#
# LDAPv3
# base <> with scope baseObject
# filter: (objectclass=*)
# requesting: namingContexts
#

#
dn:
namingContexts: dc=ldap,dc=test,dc=lan

# search result
search: 2
result: 0 Success

# numResponses: 2
# numEntries: 1


We should now have a base LDAP server running and configured for our LDAP domain. However we do not have any users (People) or groups (Group) configured from our /etc/passwd, /etc/shadow and /etc/group files. So now we need to set up our base, authentication and group files. This is done via a migration of your existing local UNIX accounts already configured on the server which are converted into an LDIF file for loading into LDAP. First however, we need to create a template ‘base.ldif’ file which creates the base structure of our directory (ldap.test.lan), which we will import into the LDAP database first.

Configuring the base LDAP entry

vi ~/base.ldif

Add the following, replacing dn and dc entries with the dc’s from your host.


dn: dc=ldap,dc=test,dc=lan
dc: test
objectClass: top
objectClass: domain

dn: ou=People,dc=ldap,dc=test,dc=lan
ou: People
objectClass: top
objectClass: organizationalUnit

dn: ou=Group,dc=ldap,dc=test,dc=lan
ou: Group
objectClass: top
objectClass: organizationalUnit


Now that we have the base information for our LDAP database structure, we can import that information into our LDAP database (using the password you created above:-

ldapadd -x -W -D "cn=admin,dc=ldap,dc=test,dc=lan" -f ~/base.ldif

You should get some output similar to the following: -


Enter LDAP Password:
adding new entry "dc=ldap,dc=test,dc=lan"
adding new entry "ou=People,dc=ldap,dc=test,dc=lan"
adding new entry "ou=Group,dc=ldap,dc=test,dc=lan"


Importing users as LDAP People entries

Now, we will use the migration script whose settings we modified earlier to create an .ldif file which we will use to populate LDAP with all our existing users pulled from ‘/etc/passwd’.

/usr/share/migrationtools/migrate_passwd.pl /etc/passwd ~/people.ldif

Once you’ve created this .ldif file, the entries can be imported into LDAP:-

ldapadd -x -W -D "cn=admin,dc=ldap,dc=test,dc=lan" -f ~/people.ldif

Importing groups as LDAP Group entries

We will now do the same with the user groups with the following: -

/usr/share/migrationtools/migrate_group.pl /etc/group ~/group.ldif

…and then import the group entries into LDAP: -

ldapadd -x -W -D "cn=admin,dc=test,dc=lan" -f ~/group.ldif

Now that we have our LDAP database populated with user information, it’s time to test our work. You can use the ldapsearch command to look for your username as below: -

ldapsearch -x "cn=[user]"

You should get something back like: -


# extended LDIF
#
...
# user, Group, ldap.test.lan
dn: cn=user,ou=Group,dc=ldap,dc=test,dc=lan
objectClass: posixGroup
objectClass: top
cn: [user]
userPassword:: R2NyfXE0fDPJ=
gidNumber: 500
# search result
search: 2
result: 0 Success
...


Add new LDAP People/Group entries

That will have imported the user details present on the server into LDAP. But it’s safe to assume that you’ll want to add other users. To add a new user, you’d create a .ldif for the user account and the group. Then we import these files into the LDAP server like we did with the base, people and groups details previously.

For example, if I wanted to add a user called ‘david’, I’d create a ‘david.ldif’ file such as below: -

vi ~/david-user.ldif

…and populate it with: -


dn: uid=david,ou=People,dc=ldap,dc=test,dc=lan
uid: david
cn: David Lightman
objectClass: account
objectClass: posixAccount
objectClass: top
objectClass: shadowAccount
userPassword: {crypt}$6$XemGNmYE5f3QAFo/vt7Uld/gUcP/2N7/R.uw5SK.
shadowLastChange: 14846
shadowMax: 99999
shadowWarning: 7
loginShell: /bin/bash
uidNumber: 501
gidNumber: 501
homeDirectory: /home/david
gecos: David Ligntman


Then the same process for the group: -

vi ~/david-group.ldif

Populated with: -


dn: cn=david,ou=Group,dc=ldap,dc=test,dc=lan
objectClass: posixGroup
objectClass: top
cn: david
userPassword: {crypt}x
gidNumber: 501


You’ll notice the password has an ‘x’ in it, much like the ‘/etc/passwd’ entry for the encrypted password, as all passwords are kept in ‘/etc//shadow’ these days. Similar format, though. Lastly, these files will be added to LDAP: -


# ldapadd -x -W -D "cn=admin,dc=ldap,dc=test,dc=lan" -f ~/david-user.ldif
Enter LDAP Password:
adding new entry "uid=david,ou=People,dc=ldap,dc=test,dc=lan"



# ldapadd -x -W -D "cn=admin,dc=ldap,dc=test,dc=lan" -f ~/david-group.ldif
Enter LDAP Password:
adding new entry "cn=david,ou=Group,dc=ldap,dc=test,dc=lan"


…and you should now have a new user called ‘david’. The absolute last thing you need to do is open a port in your firewall for LDAP to listen on. Since the default LDAP port is 389, we’ll open that using iptables.

iptables -A INPUT -m state --state NEW -m tcp -p tcp --dport 389 -j ACCEPT

That’s the LDAP server configuration and data migration done.

Client LDAP configuration

Right, the next next is to step up a client Fedora system to authenticate via the LDAP server. First, make sure you have installed the required packages: -

yum -y install openldap openldap-clients openldap-devel nss_ldap

LDAP clients are configured using the ‘/etc/openldap/ldap.conf’ file. You need to make sure that the file refers to the LDAP server’s IP address for the domain ‘ldap.test.lan’ which we set up. So, edit the config file with: -

vi /etc/openldap/ldap.conf

Make sure it looks something like this: -


HOST 192.168.1.2
BASE dc=ldap,dc=test,dc=lan


Next, we’ll need to tweak the ‘/etc/nsswitch.conf’ file. This configuration file defines the order in which the operating system searches login databases for login information when users attempt to login. You’ll want to configure it to first search its ‘/etc/passwd’ file. If it doesn’t find the user password information there, it goes to the LDAP server. The easiest way set this up is to use the ‘/usr/bin/authconfig-tui’ command, which provides a graphical tool.

1. Select LDAP.
2. Give the LDAP server’s IP address, which is 192.168.1.2 in this case.
3. Give the base DN as dc=ldap,dc=test,dc=lan
4. Select TLS.
5. Use MD5 and shadow passwords.

When you’re done, look at the ‘/etc/nsswitch.conf’ file and make sure it has references to LDAP. It should look something like: -


passwd: files ldap
shadow: files ldap
group: files ldap

hosts: files dns

bootparams: nisplus [NOTFOUND=return] files

ethers: files
netmasks: files
networks: files
protocols: files
rpc: files
services: files

netgroup: files ldap

publickey: nisplus

automount: files ldap
aliases: files nisplus


You’ll need to copy the encryption key from the server’s ‘/etc/openldap/ssl/ldap-client.pem’ location to ‘/etc/openldap/cacerts’. You can use sftp for this: -


# cd /etc/openldap/cacerts
# sftp root@192.168.1.2 -p 22
get /etc/openldap/ssl/ldap-client.pem
bye


You previously created a user named ‘ldap’ in the group on the LDAP server. You now need to make sure that this user has a home directory on the LDAP client. The example in this section creates the directory and makes ldap the owner. As you can see, the LDAP server correctly gets its user information about ‘ldap’ from 192.168.1.2; the chown command doesn’t complain about the ‘ldap’ user not existing in the client’s ‘/etc/passwd’ file.

Check that the ldap user doesn’t exist in the client’s ‘/etc/passwd’ file.

grep ldap /etc/passwd

There shouldn’t be anything returned. Next, you’ll create the home directory, copy a Bash login profile file into it, and modify the ownership of the directory and all the files to user ‘ldap’:-


# mkdir /home/ldap
# chmod 700 /home/ldap/
# chown -R ldap.ldap /home/ldap
# ls -la /home


Next you need to edit the /etc/ldap.conf file: -

vi /etc/ldap.conf

And make sure it looks something like: -


uri ldaps://ldap.test.lan/
#ssl start_tls
tls_cacertdir /etc/openldap/cacerts


Testing your LDAP client/server setup

Restart your LDAP server service to make sure all configuration is loaded and then to test, open another terminal window on the client and try to login as an LDAP-only user, such as the ‘david’ account we set up earlier: -

su - david

You should be able to login fine! :-) If not, check your /var/log/messages file for any error messages which might tell you what went wrong. But that is about it! Further documentation on the uses of LDAP can be found here.