Dbmail? A great Open Source email system, especially for LAMP/MySql administrators

A couple of weeks ago, I was thinking about how I may build an advanced search utility for my own email archive. One way to make complex queries on the archive seemed to be to put it all into a relational database. Since the Dbmail system stores email in that way, I asked its developers and Harald Reindl (an email administrator at The Lounge who already uses Dbmail: I found him in the PostFix Mailing list archives) if Dbmail could be used in that way.

The feedback I got made me change my mind about how to rebuild my own email search system, for the reasons explained below. At the same time, how and why Harald and his company use Dbmail seemed really interesting. Here’s the story.

About doing complex email searches with Dbmail

Harald explained to me that:

you should only do searches in the mail client, via IMAP. The Dbmail database is not nice to search for messages because they splitted in all their mime-parts and many db-records. Therefore, even if your search is successful, it would be hard to get a complete message without studying the Dbmail sources. So, since the search is running over IMAP and with the capabilities of Imap you can’t do more complex searches, even if the backend is a relational database. However, searches can be faster, just because the messages are splitted and indexed on the other side. Even if Dovecot in the latest versions builds an index too, so I would not expect any difference. It is surely possible to make a backend in PHP or what else language to search in the Dbmail database, but be careful about references to not display messages not owned by the user who starts the search

Why you may want to use Dbmail

Harald: I chose Dbmail because it has a 100% Mysql-backend configuration and the possibility to have a synchronized backup-slave in the network, which you can stop everytime to make consistent snapshots for offsite backups without interrupting the mail server. We are using Dovecot as proxy in front of Dbmail for several reasons:

  • it supports more auth mechanisms than dbmail
  • it supports TLS/SSL directly
  • it supports replaces (% to @) since historically many users are configured with %
  • postfix supports dovecot directly for SASL-Auth, so you have the same auth-mechs and encryption options for pop3, Imap and smtp
  • security: I think it would be hard to exploit Dbmail through Dovecot (whereas exploiting directly dovecot seems harder, since it has only the user-logins)

We decide to migrate to Dbmail because we were running Apple servers (*brrr*) with Eudora mail server and I needed a replacement running on Linux/Vmware-hosts.

Since my main job is PHP/Mysql-developer, a full db-driven server gives me options to write special interfaces for all needs, doing cron jobs for notifies, cleanups, implementing auto-reply-backends and many nice things without touching text configurations.

Postfix is also nearly 100% Mysql-driven in our environment. There are great options for forwarders/aliases on both sides. If there is anything to do you only have to figure out which of both components can do what you want best, with the smallest side-effects on the whole system.

It was a really hard job to write a PHP-backend with 20.000 lines of code in few weeks, while learning much about mail servers. However, this has been running perfectly since the summer of 2009 with only a few “WTFs” and optimizations, but they are because of little know-how at the beginning.

This means after two months working day and night there was a complete solution, and for the second mailserver the whole virtual machine was cloned in 2010 and needed only minimal configuration. The Mysql replication is a big improvement for backups, here is how we use it:

  • VMware-ESXi-Cluster
  • Mailserver on one host
  • Clone of the first machine on the second
  • Replication between both of them
  • Replication is a separate Mysql instance, read-only port 3307
  • the replication can be used for Postfix as fallback, since readonly is enough for that
  • on the backup-machine a normal instance with a copy of the db is running…
  • …so you can start dbmail-imapd with this instance and directly access it via Thunderbird
  • once a week both mysqld are stopped and rsync-ed from replication to backup
  • before that happens the last backup goes to “mysql-last-week”
  • once per day the replication is stopped and a offsite-backup per rsync done

So we have permanently access to the mailbox versions from last sunday and we can switch a week back with a simple script and restore a customer with imapsync between both machines and have a daily backup on the other end of the city. And through all the time it takes to do this, the mail services are not down for one second. I would not know how to do this with a file driven mail server, because there are permanently files changed and nobody knows if the backup is clean enough if it is ever needed.

How to reject spam from certain countries (if you must really, really do it)?

Every now and then, a question like this pops up on some email server management forum:

  I'd like to be able to reject connections from remote IP addresses if they're from certain countries.

The usual reason is either that Continue reading How to reject spam from certain countries (if you must really, really do it)?

Review, Linux E-Mail,set up, maintain, and secure a small office e-mail server

(this is a review that i originally posted somewhere on Slashdot, IIRC)

Linux E-mail, Second Edition is a book written for Packt Publishing by I. Haycox, A. McDonald, M. Back, R. Hildebrandt, P.B.Koetter, D. Rusenko and C. Taylor. Linux E-mail contains Continue reading Review, Linux E-Mail,set up, maintain, and secure a small office e-mail server

Virtual Personal Email Servers, legal, management and economics issues

After I published Wanted: Virtual Personal Email Servers I got lots of feedback. This is an edited summary of a particularly interesting one, from John of JDPFu.com, reproduced with his permission. My comments and answers are at the end of this page.

John’s comments

Marco, the Virtual Personal Email Server (VPES) that you recently asked for already exists, but not for end users. There are many pseudo-all-in-1 solutions. To solve your needs, I’d begin with ebox, a small business distro, and maybe look at Amahi for a home inclined distro. There’s always the Perfect Setup series to install a panel-based server setup like we all see on hosting ISPs.

Zimbra for a VPES?

The most user friendly free email management interface with all the bells and whistles that also supports virtual hosts is from Zimbra, but the setup is a bear and the system requirements are huge for what it provides. A simple, small Postfix, Dovecot, squirrelmail setup will easily run in 256MB of RAM and 1.5G of disk. It will require very little CPU. Zimbra won’t install on less than 1GB of RAM and it uses 30-50% of 2 CPUs even when it isn’t busy. I’ve been running 3 virtual email domains on a single Zimbra host (Xen virtualhost) for almost 2 years. You get much more than you’re asking for with the free Zimbra – enterprise calendaring is the main thing that I needed. I doubt an end user could install, configure and run Zimbra. It is too complex. There are pre-built Zimbra virtual appliances (Marco: like this?), but I’ve never tried any of them – search them on Freshmeat. There are other competitors to Zimbra, but if you want MS-Outlook calendaring, none are free, including Zimbra.

VPES DNS configuration and management

Setting up Postfix, Dovecot and Squirrelmail is relatively easy. The DNS configuration gets complicated since so many DNS providers have different interfaces, so scripting that would have to be limited. Further, each existing provider probably has contracts that prevent reselling services without joining their “network”.

The issue with a 1-minute solution for private email domains is that there are 3 parts to the problem and they should all be performed separately, by different service providers. Only GoDaddy or another registrar/host/DNS/SSL provider would be able to make it happen with just a few clicks almost immediately. I suspect, those suppliers have little desire to enter a low profit market like vanity email domains. There is too much higher profit work still to be done for them.

About selling VPES hosting

I (John) spent 20 min earlier today considering whether I could make any profit creating an easy setup script with a fairly low monthly price point. For me, it wasn’t possible to do and compete with Google business hosting at $50/user/year. Yahoo! does it for $35/yr for 1 user or $10/month for multiple users. You probably need to bring your own registered domain and DNS provider to them.

VPES and legal/data retention issues

BTW, the laws for email metadata retention are different in the EU than in the USA. In a previous job, I performed system design work to allow the company (a very large ISP/telecom) to meet EU mandated law enforcement access to email logs. The required data was date, time, to, from, subject and mail server interactions, but not the body of the email. The laws are less clear in the USA, but any data can be subpoenaed. If you do not have the data, then you cannot provide it.

Marco’s answers

First, a bit of context

In case this wasn’t clear yet, personally I have already solved my own VPES needs almost two years ago, by manually setting up my own email server for all my email and purchasing a domain name also for this purpose. With the exception of Zimbra, my current setup is very similar to what John suggests and I am quite happy about it. So at this point I don’t personally need any “system integrator”, and I am not interested in offering such a service myself, because I have already solved my own needs and I have nor skills nor interest to become a real sw developer or hosting provider.

The point of my first VPES articles here and at the Stop! was to stimulate developers and hosting providers to look at the potential of this market, because there are many people who would use a wrapped-up interface (and pay for it) but wouldn’t do everything manually as I did. Please also see my comment about Citadel and other packages falling short of being a complete VPES.

Speaking of VPEs price and DNS issues…

I do know (cfr the “Is your DNS clean” box of my “Build your own email server with Postfix” tutorial) that there are parts of the VPES puzzle that, as things are today, can’t be put inside a 1-click, do-it-yourself package. Probably DNS handling and domain name registration could be part of the services offered by a VPES hosting provider, for a one-time fee.

I also know that a VPES is not, today at least, something that could cost less than 35 or even 50 USD/year. On one hand I feel/hope that creating demand will bring prices down. On the other, I am not that concerned about price, and I don’t think that a potential VPES provider should try to compete on price with Yahoo, Gmail and friends. Not initially at least.

I am paying more than 100USD/year right now because I value and want the advantages that full email control gives me. My point is that there already are today millions of people that WOULD pay 100USD/year or more for the same freedom and control… if only somebody would spare them of ever looking at a command prompt.

When I set my email server up by myself, I didn’t do it to save a few tenths dollars a year. This is (until price structures won’t change, at least) about freedom, privacy and control, not having a vanity domain after the @ of one’s email address. I was forced to do it by myself, because the only offers on the market where:

  1. Gmail, Yahoo etc… <=50 USD/year
  2. some “email hosting providers” that wanted ~120/150 USD/year but would NOT provide all the features I’ve listed and would not be flexible
  3. Hire a consultant to do everything per my specs, which would have costed me some thousands dollars.

but if somebody had offered me a real VPES virtual box… I’d have bought it at ~150USD/year. Besides, who says that a virtual box must only be for one person? A whole family may very well do with one, until the kids are old enough to pay and manage their own by themselves. Ditto for small businesses. What do readers think?

Comments to, Wanted, Virtual Personal Email Servers

These are the comments I got when I wrote Wanted: Virtual Personal Email Servers:

(me:) Today this article was announced on LinuxToday and this is the first comment it got:

I didn’t feel like creating an account on his site just to make this comment (where’s OpenID please?) but in case he reads the comments here – the obvious solution is Citadel, which is a nice compact installation that does pretty much everything he’s looking for in a single package.

Here is my answer:

Art, thanks for your feedback. I agree that creating an account is bothering. It is definitely my intention to set up as soon as possible OpenID and/or some captcha system so that even unregistered users can contribute feedback directly on the website. I only have to find the time :-(

With respect to Citadel: personally, I don’t need it. I already built and use my own, fully customized VPES almost two years ago, it’s doing great and I am very happy with it. The point of this article is that:

  1. the majority of email users will simply refuse to switch to anything that isn’t complete and manageable from the first minute from one web control panel. They won’t install and configure stuff from the command line as I already did
  2. As good as it is, Citadel is NOT “pretty much everything” I am suggesting. Far from it. For example, the Citadel page about spam says: “Citadel has the ability to easily and seamlessly integrate with SpamAssassin. Here’s how to make it work for you. First, of course, you must install SpamAssassin. The steps for installing SpamAssassin are beyond the scope of this document. Go to the SpamAssassin web site…”
    In other words, to build what I call a complete VPES you have to do quite some work today, even if you start from Citadel. Not a problem for me, again, but not what I am suggesting.
  3. therefore, the point of my article was to suggest to developers and hosting providers (since I am no developer, have no interest or skills to be an hosting provider AND I already have my VPES anyway) to do the extra step to integrate all these things in ONE bundle to get more users and paying customers.


C. D. Rigby wrote:
Dynamic DNS

Excellent article, and for me, quite timely. I am in the process of setting up Postfix for myself on my home “server” (an old laptop running ArchLinux). So, the article provides good pointers for my roll-ones-own solution.

More advanced users may wish to run their own VPES at home. Many people have a dynamic IP address from their ISP. This configuration is almost always cheaper than an account with a fixed, routable IP address. So, I would add to your list, conditional on a self-managed, at-home VPES, the need for a Dynamic DNS service. Web searching for “dynamic DNS” will turn up lots of options.

Me: About timeliness and VPESs at home

about the “timely” part: as I wrote at the end, a VPES is to email what diaspora is to social networking with Facebook: an ideal/necessary complement.

I have not gone too much into the “VPES at home” part for several reasons. Speaking only of price, in many countries ISPs don’t allow running any server off cheap residential access contracts (even when they tolerate it but only until they have a bad day): in those cases (which is mine too) a VPES in a data center is actually cheaper than running a VPES at home. Even if it is more expensive than it should for the reasons explained in my piece, that is lack of properly sized hosting packages.

Wanted, Virtual Personal Email Servers

(update 2010/05/29: here’s another article about VPES legal, management and economics issues)

The way email is normally used today has several serious limits that I recently explained in another article. I also pointed out that one of the biggest obstacles to personal email management is lack of user demand for Virtual Personal Email Server (VPES) software and hosting packages. A VPES may run into any computer in your home or in some external datacenter, but that is another issue. Here I only want to look at the software side, that is to explain what are, in my opinion, the technical requirements and features of the perfect VPES. You are welcome to add your own in the comments and, if you’re a hosting provider already offering VPES, to add a link to your offer in the same way.

Web hosting providers only offer… web packages

Today, hosting providers (including those who “sell” virtual machines with full root access) offer packages that are optimized for easy set up and management of websites: bandwidth, virtual RAM, disk space and therefore price of those offers are only dimensioned for people who want to publish online some content. The only choice you have is between “cheap but slow and small website and “very expensive, but very big and fast website, plus some intermediate offer of the same kind. However, email hosting has different characteristics than web hosting: for example, it is not a real time service, so it has lower CPU, bandwidth (and probably disk space) requirements than most websites, but it must not relay spam and must block what the user (not its delegates or any third parties!) considers spam. So, even if turn-key VPES software packages already existed (see next paragraph) one obstacle to their large scale deployment is lack of corresponding hosting offers: sure, you could buy a (Virtual Server dimensioned for) web hosting account and use it for your VPES, but it wouldn’t be an efficient use of your money, just underutilization of something built and sold to do something else. Besides, even 10 Euro a month, that is the starting price for many reliable hosting packages, would be too expensive for many users.

Where is my email control panel?

Let’s now see what a VPES should be and look like. There’s very little or no software to develop to build it. All the free software necessary to run a VPES already exists today and is usable without being a programmer! I already run my own email server, and to do it I never had to write, modify or compile any software program: I only had to read a few tutorials in order to give the right value to some configuration options.

This said, in order to build my own VPES, I had to manually edit configuration files in a character terminal. I’m perfectly fine with that, but it is a fact of life that many people who’d like to have a VPES and pay a few bucks for it, will simply refuse to work in that way. Even if they have no problem whatever to set the same options through a web form.

Therefore, the other big obstacle (=business opportunity) to real personal email management is the lack of Linux-based distributions that are VPESs, that is distributions that contain all and only the software needed for a VPES plus an integrated web-based interface to configure all its parts. Something like this already exists today, as the screenshots in this tutorial on Postfix configuration with Webmin prove: what’s missing is bundling everything together so that you go from installation to a configuration panel like that without intermediate steps, and never needs to use anything else for configuration.

VPES software and feature list

Here is a list, by no means complete. When present, package names in parenthes indicate what I use on my own server, but you’re welcome to suggest alternatives. Just keep in mind, however, that the typical VPES user couldn’t care less of debating whether PostFix is better than Exim, or even to know which MTA server is actually running under the hood:

Software components:

  • MTA server: this is the program that actually receives your email from the Internet and forwards your messages to the Internet (Postfix)
  • Imap server: manages the mailboxes that contain received email, serving their content to the webmail or mail program chosen by the user (Dovecot)
  • Webmail interface, to read and write email from any web browser (Squirrelmail)
  • Backup utility to duplicate configuration info and mailboxes on other computers
  • Backup restore utility
  • Antispam system (SpamAssassin, bogofilter)
  • DNS configuration interface to set up the domain name part of your desired email address(es) and email identification functions like SPF or DKIM
  • Web interface to check if the server looks like a spam source
  • OpenSSL to encrypt web connections, with web interface to generate a certificate
  • database to store user and domain names
  • web based interface to manage user and domain names
  • password strength checking utility
  • GPG to digitally sign and/or encrypt email
  • Calendaring and address book functionality for each user of the VPES
  • Webmin or similar to manage through a web browser all of the above plus updates of software and antispam rules
  • Apache web server to run all the web interfaces mentioned above


  • secure default configuration: no open relay, default set of SpamAssassin rules enabled, enforcement of secure passwords, only crypted HTTP connections allowed, mandatory SMTP authorization, software updates possible with signed packages from trusted sources…
  • support for multiple virtual domains, so one VPES can be used both for you @yourfamily.com, your.son @yourfamily.com, you @yourbusiness.com and so on
  • (why not?) wrapping everything in a virtual machine (a-la TurnKey Linux) so the whole thing can be installed in a snap even on Windows or Mac system or in the cloud.

What else? Please tell!

I am sure that the lists above are incomplete and that they can be improved. Your contributions are welcome, just register to add them in the comments or send them directly to me. What matters is to start serious discussion on how to build a really integrated VPES entirely out of Free Software. I also welcome general feedback about making Personal Email Management really popular, especially because it could (should?) become the email part of the Freedom Box.