HA Linux Mail Server – Part 0

I’ve been working on a project to learn more about RHCS in general and decided to build a HA Mail server.

Technologies Used:
– Centos 6.6
– Postfix
– Dovecot (IMAP)
– OpenLDAP
– GlusterFS
– Roundcube

Over the coming weeks I’ll be posting the steps required to set this up.

The general idea is to have two RHCS services


  • VIP for postfix/dovecot
  • glusterfs for delivered mail, postfix and dovecot configuration files


  • VIP for mysql
  • glusterfs for mysql config and data


Each node runs the following locally

  • httpd + roundcube
  • OpenLDAP replicating from another LDAP server

While the above two services could certainly be cluster services, this is not required.

We NAT our public IP to the VIP for mail-svc.

To avoid SSH key issues, copy your SSH keys from /etc/ssh to the other node (So SSH to the public IP will not result in errors after a failover)



Using Live CDs to help repair/diagnose a PC

When I work on a Windows machine, I usually end up coming across a missing/corrupt driver. On Windows it is next to impossible to figure out what kind of hardware is in your computer without having the driver installed. You could google for a device id, but you’re going to get a lot of wrong answers.

A better solution is using a live CD. Any recent one will work; the tools you’re using will work regardless.

lspci – List all PCI Devices

01:00.0 VGA compatible controller: nVidia Corporation GeForce 8400M GS (rev a1)
03:01.0 FireWire (IEEE 1394): Ricoh Co Ltd R5C832 IEEE 1394 Controller (rev 05)
03:01.1 Generic system peripheral [0805]: Ricoh Co Ltd R5C822 SD/SDIO/MMC/MS/MSPro Host Adapter (rev 22)
03:01.2 System peripheral: Ricoh Co Ltd R5C843 MMC Host Controller (rev 12)
03:01.3 System peripheral: Ricoh Co Ltd R5C592 Memory Stick Bus Host Adapter (rev 12)
03:01.4 System peripheral: Ricoh Co Ltd xD-Picture Card Controller (rev 12)
09:00.0 Ethernet controller: Broadcom Corporation NetLink BCM5906M Fast Ethernet PCI Express (rev 02)
0c:00.0 Network controller: Intel Corporation PRO/Wireless 3945ABG Network Connection (rev 02)

Want to know the model of your network card, even if Linux doesn’t support it? No quite sure what kind of nvidia card you have? Done.

You might say, “Well sure, that’s nifty. But what about USB devices?”

lsusb – List USB Devices

Bus 004 Device 002: ID 046d:c50e Logitech, Inc. MX-1000 Cordless Mouse Receiver

dmesg – Outputs nifty stuff about your computer

From hard drives, to being able to see if the machine detected a new USB device to what wireless cards were detected dmesg offers a ton of information. I suggest using grep and more; my dmesg is 541 lines long, and it wouldn’t be surprising for yours to be more.

It it also a great place to find error messages (failling hardware generally shows up here).

[    7.616000] sd 2:0:0:0: [sda] 156301488 512-byte hardware sectors (80026 MB)

[   16.076000] iwl3945: Tunable channels: 11 802.11bg, 13 802.11a channels

Want to see how much memory is actually being detected? (Windows and many BIOS’s round these numbers)

cat /proc/meminfo

Want to know a bunch of information about the CPU in the machine? Then cpuinfo is for you.

cat /proc/cpuinfo

Migration of a live linux system

1. Prep machine or VM for the migration. On the destination server, boot with a live CD (CentOS, Debian, etc.). Partition drives.

I am assuming you are using just one partition, adding other partitions is trivial.

2. Mount drive and transfer system


mkdir /media/chroot
mount /dev/sda1 /media/chroot
rsync -az –exclude ‘proc’ –exclude ‘sys’ –exclude ‘dev’ –exclude ‘media’ –exclude ‘mnt’ root@source_server:/ /media/chroot
mkdir /media/chroot/dev
mkdir /media/chroot/proc
mkdir /media/chroot/sys
mount –bind /dev /media/chroot/dev


3.  Chroot into system


chroot /media/chroot
mount /proc
mount /sys
mv /boot/grub/device.map /boot/grub/device.map.old
install-grub /dev/sda


Also make sure to edit /etc/fstab with your new device IDs/partitions. You can view a list of device IDs by doing


ls -l /dev/disk/by-uuid


When you are done, reboot.

3. System show now boot successfully


Linux copy GPT partition table with dd

I recently had to copy the partition table of a 3TB disk in a situation where tools such as sfdisk could not be installed.

Since GPT table length is dependant on the number of partitions, you need to do some investigation.

In this case, it was a ‘QNAP’ server that had fdisk (no GPT support) and parted.

On a working drive, run

parted -ms /dev/sda print

Note the number of partitions.

Formula = (128*N)+1024

Where N is the number of partitions you have. In this case I had 4, so I end up with a value of 1536

dd if=/dev/sda of=GPT_TABLE bs=1 count=1536

You now have a backup of a valid partition table you can apply to another drive

dd if=GPT_TABLE of=/dev/sdb bs=1 count=1536

Once this was done, you can manually re-add the drive.

mdadm –manage /dev/md0 –add /dev/sdb3

If you are wondering how we determined the sd[a-z], we accomplished this through hot-swapping the drive to generate logs indicating the drive.

Now why this supposedly automated RAID product required this…

Automate Restart of D-Link DCS-900 Cameras

I have some older D-Link DCS-900 cameras being used with ZoneMinder for a security system at home.

While these cameras are normally very stable (no lock-ups for weeks or months at a time) it still happens. To make things work more smoothly, I decided to find a way to restart the devices with cron.

I decided to have the cameras be reset twice a month (on the 1st and 15th of each month at 4am).

crontab -e

0 4 1,15 * * wget –http-user=admin –http-password=YOUR-PASSWORD -O /dev/null –post-data=”Reset= Yes ” http://camera1/Reply.html


You can remove the –http-user and –http-password sections if you do not have an admin user/password set.

Ubuntu 8.10 LDAP Server with TLS

This should also work with Ubuntu 9.04

apt-get install slapd ldap-utils libnss-ldap libpam-ldap migrationtools

Much like Debian, we must perform an extra step to make our changes permanent.

dpkg-reconfigure slapd

Omit OpenLDAP server configuration? No
DNS domain name: your-server
Organization name: whatever
Database backend to use: HDB
Do you want the database to be removed when slapd is purged? No
Move old database? Yes
Admin password: <password>
Confirm: <password>
Allow LDAPv2 protocol: No

Now we will populate LDAP. You should probably add a temporary LDAP user for testing. After we populate LDAP we’ll remove it.

adduser test
cd /usr/share/migrationtools

Edit migrate_common.ph and replace “padl.com” with your domain and dc=padl,dc=com with dc=YOUR,dc=DOMAIN

./migrate_base.pl > /tmp/base.ldif

./migrate_passwd.pl /etc/passwd /tmp/passwd.ldif

./migrate_group.pl /etc/group /tmp/group.ldif

Remove the top section of /tmp/base.ldif

dn: dc=dev
dc: dev
objectClass: top
objectClass: domain

Restart LDAP and add our information to LDAP

/etc/init.d/slapd restart

ldapadd -x -W -D 'cn=admin,dc=example,dc=net' < /tmp/base.ldif
ldapadd -x -W -D 'cn=admin,dc=example,dc=net' < /tmp/passwd.ldif
ldapadd -x -W -D 'cn=admin,dc=example,dc=net' < /tmp/group.ldif

To enable PAM I suggest

cd /tmp
wget http://st0rage.org/files/pam.d.tar
tar xf pam.d.tar -C /etc

Alternativly, you can try the following Ubuntu specific method:

auth-client-config -t nss -p lac_ldap
pam-auth-update ldap

Please note that I was unable to change user passwords with ‘passwd’ using this method.

Remove the test user

userdel test

Set up TLS Certificate

mkdir /etc/ldap/ssl
cd /etc/ldap/ssl

openssl req -new -nodes -out req.pem -keyout key.pem
#No challenge password, leave empty
openssl rsa -in key.pem -out new.key.pem
openssl x509 -in req.pem -out ca-cert -req -signkey new.key.pem -days 9999
mv new.key.pem server.pem
cat ca-cert >> server.pem

Enable TLS on server

ldapmodify -x -D cn=admin,cn=config -W

Paste in the following:

dn: cn=config
add: olcTLSCACertificateFile
olcTLSCACertificateFile: /etc/ldap/ssl/server.pem
add: olcTLSCertificateFile
olcTLSCertificateFile: /etc/ldap/ssl/server.pem
add: olcTLSCertificateKeyFile
olcTLSCertificateKeyFile: /etc/ldap/ssl/server.pem

Press ENTER a few times to finish modifying and then Control+C when you see:

modifying entry "cn=config"

Edit /etc/default/slapd and add

SLAPD_SERVICES="ldap:/// ldaps:/// ldapi:///"

Restart slapd

/etc/init.d/slapd restart

If it fails, try:

chown -R openldap /etc/ldap/ssl
/etc/init.d/slapd restart

Client configuration

vi /etc/ldap.conf

uri ldaps://server
port 636
ssl start_tls
ssl on
tls_checkpeer no
tls_cacertfile /etc/ldap/ssl/server.pem

vi /etc/nsswitch.conf

passwd: compat ldap
group: compat ldap
shadow: compat ldap
hosts: files dns
networks: files
protocols: db files
services: db files
ethers: db files
rpc: db files
netgroup: ldap

Now test! Make sure to stop nscd if you’re using it

/etc/init.d/nscd stop


$ id test
uid=1001(test) gid=1001(test) groups=1001(test)

Remember to keep an eye on /var/log/auth.log if you run into any problems

Recover from accidental mkfs using fsck

Recently I accidentally ran mkfs.ext3 on the wrong RAID1 array.

As soon as I realized what had happened, I shut power off from the PC.

Booting to a live CD, I ran

mdadm –assemble /dev/md0 /dev/sda1 /dev/sdb1

fsck -y /dev/md0

I was able to then mount the drive

mkdir /media/md0

mount /dev/md0 /media/md0

ls /media/md0/lost+found

And the majority of my files were saved.

Check to see if your filesystem is ext2 or ext3

mount | grep md0

If it’s ext2, convert it to ext3

umount /media/md0

tune2fs -j /dev/md0

WARNING: If you choose to run fsck manually (without -y), choosing “n” to Clear? and “y” to Fix? will cause you to lose that file.

Debian Corefiles

If you have any issues with programs segfaulting, it can be useful to allow coredumps in order to see what happened.

You first have to allow coredumps

vi /etc/security/limits.conf

*  –  core  100000

@users  –  core  0

core is measured in kbytes, and core files can be fairly large (the simple script below generates a 350kb file).  Do not set this to unlimited for regular users!

echo 1 > /proc/sys/kernel/core_uses_pid

mkdir /tmp/core

echo “/tmp/core/dump” > /proc/sys/kernel/core_pattern

ulimit -c 100000

To test it let’s create a script that will cause a coredump.

vi core_dump.sh

kill -s $$

chmod +x core_dump.sh

ls /tmp/cores

dump.14594 core_dump.sh

To view the corefile, you need gdb.

apt-get install gdb

gdb -c dump.13594

Ubuntu 8.04 Hardy LDAP Client

Ubuntu 7.10 was a nightmare when it came to setting up ldap, but 8.04 improves this process quite a bit.

We are going to set up a Hardy client on a desktop machine, which involves using NFS (for /home) and allowing all desktop users to do desktop tasks.

apt-get install libpam-ldap libnss-ldap nss-updatedb libnss-db nfs-common nscd

Answer the questions; unlike Debian they should actually be put in the configuration file.

Make sure to transfer over your certifiate if you use SSL. I like to use /etc/ldap/ssl

Edit /etc/ldap.conf (which both libnss and libpam use).

base dc=example,dc=com

#This is important! Don't use ldap:///
uri ldap://example.com/
ldap_version 3
rootbinddn cn=admin,dc=example,dc=com
port 389
bind_policy soft
pam_password crypt
ssl start_tls
tls_checkpeer no
tls_cacertfile /etc/ldap/ssl/cert.pem
nss_initgroups_ignoreusers avahi,avahi-autoipd,backup,bin,daemon,dhcp,games,gdm,gnats,haldaemon,hplip,irc,klog,libuuid,list,lp,mail,man,messagebus,news,polkituser,proxy,pulse,root,sync,sys,syslog,uucp,www-data

Now edit /etc/ldap/ldap.conf

BASE    dc=example,dc=com
URI    ldap://example.com
TLS_CACERT /etc/ldap/ssl/cert.pem


account    sufficient   pam_ldap.so
account    required     pam_unix.so


auth       sufficient   pam_ldap.so
auth       required     pam_unix.so nullok_secure use_first_pass


password   sufficient   pam_ldap.so
password   required     pam_unix.so nullok obscure min=4 max=8 md5


session    required     pam_unix.so
session    required     pam_mkhomedir.so skel=/etc/skel/
session    optional     pam_ldap.so


passwd: files ldap
group: files ldap
shadow: files ldap

hosts:          files mdns4_minimal [NOTFOUND=return] dns mdns4
networks:       files

protocols:      db files
services:       db files
ethers:         db files
rpc:            db files

Now we want to make sure users are assigned to the correct groups when they log in, so add the following to /etc/security/groups.conf

Hal does not recognize this, however, so delete the following entries from /etc/dbus-1/system.d/hal.conf

<deny send_interface="org.freedesktop.Hal.Device.Volume"/>
<deny send_interface="org.freedesktop.Hal.Device.Volume.Crypto"/>

We need to edit /etc/pam.d/gdm for the groups.conf file to take effect, so add the following

auth optional pam_group.so

As root, run

nss_updatedb ldap

To mount /home over NFS, add the following to /etc/fstab       /home   nfs defaults 0 0

Universal Plug and Play (uPnP) in Debian

Before I begin, uPnP does have some security implications. Users that can send uPnP requests can open any port to any other computer. Please read the comments below for more details.

If you are currently using a Linux router on a home network, you should consider uPnP. uPnP allows computers to request ports be opened for them, so applications that require open ports can work without work of the owner of the Linux router.

apt-get install libupnp0 linux-igd

Edit /etc/default/upnpd and specify your external and internal interfacecs; upnp does not support multiple interfaces but I suspect multiple instances of this program could accomplish this; just replace the /etc/init.d/upnpd file with a custom one like this


case $1 in


upnpd eth0 eth1

upnpd eth0 eth2



killall upnpd



Restart the daemon, and you should see clients requesting ports in /var/log/syslog (grep for upnp).

If you do not, make sure that you see the following line in “route”       *            U     0      0        0 eth3

Where eth3 is your internal interface. If not, add it with

route add -net netmask eth3

Also, if you are blocking any ports with your firewall configuration, you may have to do the following to allow your clients request to reach the server.

iptables -A INPUT -s -p tcp --dport 2869 -j ACCEPT
iptables -A INPUT -s -p udp --dport 1900 -j ACCEPT

Where is your internal network.

A good way to test to make sure this is working is using the Azureus bittorrent client’s NAT checking tool.

Below is a two network firewall script that I am currently using with uPnP; I can’t make any guarantees to how secure it is but it works quite well.

/sbin/ip addr add dev eth0 $IP2
/sbin/ip addr add dev eth0 $IP3
/sbin/depmod -a
/sbin/modprobe ip_tables
/sbin/modprobe ip_conntrack
/sbin/modprobe ip_conntrack_ftp
/sbin/modprobe ip_conntrack_irc
/sbin/modprobe iptable_nat
/sbin/modprobe ip_nat_ftp
/sbin/modprobe ip_nat_irc
echo "1" > /proc/sys/net/ipv4/ip_forward
echo 1 > /proc/sys/net/ipv4/tcp_syncookies
echo "65535" > /proc/sys/net/ipv4/ip_conntrack_max
echo 1 > /proc/sys/net/ipv4/icmp_echo_ignore_broadcasts
echo "1" > /proc/sys/net/ipv4/icmp_ignore_bogus_error_responses
echo "1" > /proc/sys/net/ipv4/conf/all/accept_redirects
echo "0" > /proc/sys/net/ipv4/conf/all/log_martians
echo 30 > /proc/sys/net/ipv4/tcp_fin_timeout
echo 9000 > /proc/sys/net/ipv4/tcp_keepalive_time
echo 1 > /proc/sys/net/ipv4/tcp_window_scaling
echo 0 > /proc/sys/net/ipv4/tcp_sack
echo 3000 > /proc/sys/net/ipv4/tcp_max_syn_backlog
$IPT -A DROPLOG -j ULOG -m limit --limit 2/hour --ulog-nlgroup 1 --ulog-qthreshold 20
$IPT -t nat -A POSTROUTING -s -o eth0 -j SNAT --to-source $IP2
$IPT -t nat -A POSTROUTING -s -o eth0 -j SNAT --to-source $IP3
$IPT -t nat -A POSTROUTING -s -o eth0 -j MASQUERADE
$IPT -t nat -A POSTROUTING -s -o eth0 -j MASQUERADE
$IPT -t nat -A POSTROUTING -s -d -j ACCEPT
$IPT -t nat -A POSTROUTING -s -d -j ACCEPT
$IPT -t nat -A POSTROUTING -s -d -j ACCEPT
$IPT -t nat -A POSTROUTING -s -d -j ACCEPT
$IPT -t nat -A POSTROUTING -s -d -j ACCEPT
$IPT -A INPUT -p tcp --dport 22 -j ACCEPT
$IPT -A INPUT -p tcp --dport 80 -j ACCEPT
#Allow uPnP from internal clients
$IPT -A INPUT -s -p tcp --dport 2869 -j ACCEPT
$IPT -A INPUT -s -p udp --dport 1900 -j ACCEPT
$IPT -A INPUT -s -p tcp --dport 2869 -j ACCEPT
$IPT -A INPUT -s -p udp --dport 1900 -j ACCEPT
$IPT -A INPUT -i eth2 -j ACCEPT
$IPT -A INPUT -i eth3 -j ACCEPT
$IPT -A INPUT -p tcp -s --dport 3128 -j ACCEPT
$IPT -A INPUT -p tcp -s --dport 3128 -j ACCEPT
$IPT -t nat -A PREROUTING -i eth1 -p tcp --dport 80 -j DNAT --to
$IPT -t nat -A PREROUTING -i eth2 -p tcp --dport 80 -j DNAT --to
$IPT -t nat -A PREROUTING -i eth3 -p tcp --dport 80 -j DNAT --to
$IPT -t nat -A PREROUTING -i eth4 -p tcp --dport 80 -j DNAT --to
$IPT -t nat -A PREROUTING -i eth5 -p tcp --dport 80 -j DNAT --to
$IPT -A INPUT -i eth0 -p tcp --syn --destination-port 3128 -j DROPLOG