update 2012/07/13: I have realized only today that from March 2012 Scratch is also available with a GPL v2 license.

Recently there have been two separate discussions on two italian Free Software mailing lists, about the meaning and obligations of the license for the source code of the educational software Scratch. In both cases, I asked (and received) confirmation of my understanding of the license directly from the “Help@Scratch”. The discussions were about two topics that are, I believe, interesting and relevant for everybody considering usage or development of Free Software, especially (but not only) in educational and non-profit contexts. Therefore, I’m publishing the answers I got from the Scratch team here (with their approval), hoping they may be useful to everybody else with the same doubts in the future (but please see the disclaimer below!).

Does contributing your code to some FOSS project automatically give away your copyright on that code?

Point 4 of the Scratch source code license says:

copies or derivative works must retain the Scratch copyright notice and license

Reading this, some members of those mailing lists said that they would NOT use Scratch and would actively recommend others to NOT use because their understood the sentence above to mean something like: if you write extra code to modify or extend Scratch you HAVE to assign the copyright of their changes to them.

My understanding, instead, was that this means that if I modify the source code and (respecting all the other conditions) redistribute the resulting program:

1. I must include with all the source code the original notice that says that copyright of the original code belongs to the original authors
2. but copyright of the code I write remains to me. In other words, I do not need to assign the copyright of my modifications to the Scratch team to contribute.

In other words, I’ve always understood (even before the OpenOffice/LibreOffice saga confirmed this to me) that contributing your code to a FOSS project with whatever license doesn’t automatically means that the copyright of that code goes to the original licensor. The answer from the Scratch team confirmed this interpretation, in two separate messages:

the copyright belongs to whoever makes the modification. However, they are compelled by the terms of the license to make their code available to others under the Scratch license. So the modified code must be made available to others under the same terms that the original source was made available to the programmer that modified it. But that’s not the same as assigning the copyright to us.

A programmer of Scratch added his interpretation:

The author of any modifications to Scratch owns the copyright to any new code that they write. The copyright for their code is NOT automatically assigned to MIT. MIT could not, for example, include their changes in a Scratch release without their permission. However, under the “share alike” clause in the Scratch license, they must share the source code for their changes. “Share alike” clauses are quite common in open source licenses.

A derivative work is bound by all the terms of the Scratch license, including the non-commercial clause, so creators of modified or extended versions of Scratch cannot sell their derivative works without permission. However, anyone who modifies Scratch is allowed to use and their distribute derivative work for free or use it in other ways.

For example, the Open University created a modified version of Scratch that they hope will be used by over 10,000 students over the next five or six years. They are allowed to do that under the terms of the Scratch license.

Note that the Scratch source license is irrelevant to people who merely wish to use Scratch (which includes most students and educators). The license for the “binary” Scratch packages does not even include a non-commercial clause, so people can, for example, distribute it in an educational package or package it with a book

Can you be paid to develop software “for non commercial use only”?

The other topic of discussion was about another point of the Scratch source code license:

The Scratch source code license allows you to distribute derivative works based on the Scratch source code for non-commercial uses subject to the following restrictions…

The mailing list got stuck on this case:

• suppose some public school, Public Administration (PA), NGO… decides it needs a modified version of Scratch for non commercial use, for example to use it internally, or to give it away for free, only to give free classes to kids
• but the only programmer who’s skilled enough and available to modify the software wants to be paid by the school or NGO. Is this allowed by the license?

My (Marco) understanding is that if that school/NGO/PA pays the programmer to modify the Scratch code and release both the modified source code with the same license as Scratch and the executable program for free, they aren’t violating any license. Why? Because what I see as being used commercially here are only the skills and time of the programmer and any tool (e.g. compilers) he or she may use to modify the program, NOT the source code or binaries of Scratch (or any other software licensed with a non-commercial clause) in and by themselves.

Others, instead, said that the license means that nobody can ask or take money for building the derivative works, even if those derivative works will never be sold or used to make money.

When I asked again to “Help@Scratch” who was right, I got this answer:

You are correct. The non-commercial clause just means that you can’t make a modification and then sell the finished product. It’s totally fine to pay programmers to work on it… [that clause is there because] especially in the early days of Scratch before it was well known, we wanted to be sure some company with a big marketing budget didn’t take the code, change the About page, and then sell it to schools for $50 a license. We’re a non-profit, and our marketing budget is $0, so we have to be careful about such things.

DISCLAIMER: I published this exchange for two reasons. First, I hope that it will help to clarify the meaning of “copyright assignment” and “non-commercial use” when software is concerned, especially to non native English speakers. In the second place, I hope to gather more feedback and info on these topics. This said, I am not a lawyer, and laws are different from country to country! Therefore, do not assume that this page is 100% accurate and complete! Ask a lawyer in your country if you need a definitive, up to date answer (and in such case, please let me know it!) Thanks.

Background

Rss is a wonderful system to get headlines of online news from many independent sources and browse them as quickly as possible, without subscribing to any website, giving away personal information and/or depending on any third-party website to aggregate everything for you.

In order to save time and to not depend on any Rss reader, I have written two simple scripts. One downloads all the RSS feeds I want to read and saves them in a format suitable for further processing. The other reads that temporary file and generates one single HTML page with all the news titles and links. The format of the temporary file is very simple. It’s just plain text with three fields per line, separated by a “|” (pipe) character: feed name, article title and article URL. Here’s an example of that file:

  Repubblica|Pisani, difesa a oltranza "Non ho dato notizie a Iorio"|http://example.com/url_1.html
  Repubblica|Caffè, sigaretta, persino l'email così la pausa diventa un privilegio|http://example.com/url_2.html
  Repubblica|L'ultimo trucco "ad aziendam" di Berlusconi il 'padrone' del paese corrompe la democraz
  ia|http://example.com/url_3.html

The real problem is how to generate that file, that is how to download, parse and reformat RSS from the command line. Here’s how I do it. It works almost perfectly, with one exception explained below, for which I ask for your help.

Rss downloader script

The simplest way I’ve found to download and parse Rss feeds is the Python feedparser module. Once it is installed, it only takes 15 lines of code to generate the list shown above:

   1#! /usr/bin/python
   2
   3 import sys
   4 import feedparser
   5 import socket
   6
   7 timeout = 120
   8 socket.setdefaulttimeout(timeout)
   9
  10 feed_name = sys.argv[1]
  11 feed_url     = sys.argv[2]
  12 d               = feedparser.parse(feed_url)
  13
  14 for s in d.entries:
  15	print feed_name + "|" + unicode(s.title).encode("utf-8") + "|" + unicode(s.link).encode("utf-8") + "\n"

The scripts takes as argments the feed name and the RSS URL (lines 10 and 11). Line 12 is the one that actually downloads the feed and saves all its content in an object named “d”. The timeout in lines 7/8 is needed to not have the script freeze when some website is unreachable. The last two lines look at each element of the RSS object and print (together with the feed name) the title (s.title) and URL (s.link) of each entry. That’s it, really.

One little problem: encoding

As I said, the script works almost perfectly as is, and I hope you’ll find it useful. The only problem I haven’t solved yet is how to handle non-ASCII characters in URLs and, especially, news titles. As an example of what I mean, here’s what I get when I convert to HTML the three lines shown above.

(in case it matters, this happens on Fedora 14 x86_64). As you can see, the accented letters are messed up. Similar things happen with quotes and other non-ASCII stuff. How do I fix this? Before I added the encode("utf-8") command it looked even worst (**), but there’s something still missing here. I have tried to figure out what, but I must say the relevant Python documentation isn’t so simple and easy to find (or recognize at least), so your feedback is very welcome. Thanks!

(**) this is why I believe that the problem is, and should be fixed, in the Python script itself and not in the other script that creates the HTML page, but I may be wrong. Regardless of this, I want to understand better how encoding is handled in Python

Yesterday Sergio, a user of OpenOffice Impress, sent to the OpenOffice.org discussion list his list of the “Major Gaps of OpenOffice Impress 3.3 vs. Microsoft Office PowerPoint”.

Sergio compiled the list because, as much as he likes OpenOffice, “after struggling for over 1 year, sadly he had to stop using Open Office Impress and go back to Microsoft Power Point”.

Personally, I have experienced and can confirm most of what Sergio lists as “File Processing issues”. I haven’t encountered the other problems, but that may be because I use Impress very little these days, and I only need it for very simple slideshows. I don’t even know yet, for lack of personal experience, if and how the current LibreOffice version of Impress would be different. However, I am very curious to know if such differences exists. Above all, since I strongly support the OpenDocument format used by OpenOffice, LibreOffice and many other software programs, I want these issues to be solved.
Therefore, after speaking with Sergio, I’ve reformatted his report and put it here where it’s easier to find it than as a mailing list attachment, and easier to comment without subscribing to a mailing list. Your feedback is welcome!

Impress File Processing issues

• Slow speed of processing even with high efficiency PCs (major problem !): Many tasks are performed very very slow !
• Cutting slides: very, very slow
• Copying and pasting slides from one impress file to another: very, very slow
• Acquiring a slide change, even in the text: quite slow
• Saving files: very slow
• Opening files: very slow

Copy and past slides from one impress file to another

When a graphic is present in the slide layout, it gets deleted when the slide is pasted and copied in the destination file (major problem).
the color format of the slide in the source file gets changed when the slides gets pasted in the destination file (In PowerPoint, when you paste the slide in the destination file you are asked whether to retain the original format, including colours, layout graphic, etc.)

Changing page (slide) (to the following or to the previous one) in normal view

In “normal” view, it is not possible to shift easily to one page to the following or the previous one, using for instance the side scroll bar or the mouse scroll wheel. This is possible only when the zoom size of the page/slide very small, not with operative size. You have to necessarily click on the new slide into the left frame with the miniatures slides. This is very cumbersome.

Icons view

It is not possible to view all the icons of the formatting toolbar, unless you set a very large window size. Please allow to arrange the toolbar in 2 lines, even when it is integrated in the menù bar.
Please allow to change the order of icons within a toolbar.

Formatting in OpenOffice.org Impress

Bulleted list: I can’t set easily and automatically a space or a tab between each bullet and the first character of the paragraph (this option is present in “Open Office ” Word)

Increase or decrease indent of a paragraph or a bulletted list: I can’t let the icon left-to-right or right-to-left appear in the Formatting Toolbar, and therefore it is difficult to increase or decrease the indent (this option is present in OpenOffice Word)

Multiple selection of non-consecutive text: it is not possible, within the text in a same text cell, to select multiple, non-consecutive words or sentences or different non-consecutive sentences of a bulletted list (these options are possible in Open Office Word using “CTRL”),
Similarly, within a table, it is not possible to select multiple, non consecutive words, or sentences or cells (this is possible in Open Office Word using “CTRL”).

Formatting multiple text cells at the same time: after you select multiple text cells, the tool bar “Formatting” disappears. Therefore, you have to go to the Edit toolbar or right click and make one change at a time in the text format, which is very time-consuming.

Formatting tables: there is no way to select a column or a line just putting the cursor at the top of the column or before the line.

Changing the column width: putting the cursor onto one column border (starting from the second column from the left), clicking and dragging it in order to enlarge or reduce the column width: there is no way to retain the original width of the side columns (this is partly possible in Open Office Word by clicking at the same time the CTRL).

When the file is saved and re-opened, especially when an Impress file is saved as Microsoft PowerPoint and then re-opened as Impress file, tables gets often increased in line-spacing (very difficult to reduce back) and, consequently, in the overall height, so that they often get outside the slide (major problem!)

WordPress is quick and easy to install and update, but the quicker you can make these operations the better, right? If you have shell access to the server where your WordPress copy is installed, it is possible to perform all the operations in Step 1 of the Manual WordPress Update Procedure with the shell script below. It will save you a few minutes, which may seem too little, but is great if you maintain more than one copy of WordPress. That, however, is not the main reason to use a script like this. Its bigger advantage is reducing the possibility of human error by doing things by hand, at the prompt or with the mouse is the same.

The scripts takes two parameters: the first is the complete URL of the zipped WordPress files that you need in order to upgrade. The second is the directory, called WP_ROOT, where the WordPress installation you need to upgrade lives. You will note that in my script $WP_ROOT is only the name of that directory, not the complete path to it, which in the script is $HTML_ROOT/$WP_ROOT. I have do this because I have several, independent copies of WordPress running on the same server, all living in different subdirectories of my $HTML_ROOT, named with the name of each blog. Therefore, this choice of variables sounds more natural to me. When I want to update http://stop.zona-m.net, I will type `upgrade_wordpress.sh http://wordpress.org/wordpress-3.1.1.zip stop`. When I want to update http://strider.zona-m.net, instead, I will type “strider” instead of “stop”, and so on. Here is the script. Enjoy it, but remember to read the warnings at the end of this page first!

  #! /bin/bash
  # upgrade_wordpress.sh
  # This script performs automatically the operations described in the section
  # "Manual Update - Step 1" of http://codex.wordpress.org/Updating_WordPress
  # Copyright 2011 M. Fioretti http://stop.zona-m.net
  # Released under GPL V2 license
  #
  # $1 URL of latest WordPress version, zipped
  # $2 Root directory of the WordPress installation that must be upgraded

  WP_ROOT=$2
  HTML_ROOT=/var/www/html/wp
  TEMP_DIR=/tmp/temp_wp_update

  # Step 1.1 and 1.2: get latest WordPress, unpack it in temporary directory
  mkdir $TEMP_DIR
  cd    $TEMP_DIR
  wget  $1
  unzip -q wordpress*.zip

  # Step 1.3: Delete the old wp-includes and wp-admin directories
  rm -rf $HTML_ROOT/$WP_ROOT/wp-includes $HTML_ROOT/$WP_ROOT/wp-admin

  # Step 1.4: "upload" the new wp-includes and wp-admin directories
  cp -r -p $TEMP_DIR/wordpress/wp-includes $TEMP_DIR/wordpress/wp-admin $HTML_ROOT/$WP_ROOT

  # Step 1.5: "Upload" the individual files from the new wp-content folder
  # to your existing wp-content folder, overwriting existing files.

  cd $TEMP_DIR/wordpress/wp-content/
  find . -type f | sort  > $TEMP_DIR/new_wp_content_files_list
  tar cf $TEMP_DIR/wp-content-files-to-replace.tar -T $TEMP_DIR/new_wp_content_files_list
  cd  $HTML_ROOT/$WP_ROOT/wp-content
  tar xf $TEMP_DIR/wp-content-files-to-replace.tar

  # Step 6: "Upload" all new loose files from the root directory of the new
  # version to your existing wordpress root directory

  cd     $TEMP_DIR/wordpress
  tar cf $TEMP_DIR/loose_files_in_wp_rootdir.tar license.txt readme.html *php
  cd     $HTML_ROOT/$WP_ROOT
  tar xf $TEMP_DIR/loose_files_in_wp_rootdir.tar

  # Final step: remove temporary files, remember to check the configuration

  rm -rf $TEMP_DIR
  echo "you should take a look at the wp-config-sample.php file, to see if any new settings have been introduced that you might want to add to your own wp-config.php"

Warnings

If you reload the WordPress admin page just after running the script you’ll get the “let’s upgrade the database” button, and everything should be OK after that. I have tested this script myself to update both http://tips.zona-m.net/ and this website from WordPress 2.8.1 on a Centos VPS and as far as I can see it worked just fine.

As everything done with shell scripts, howeverm this thing is powerful enough to both save you time and to hurt your blog if there is some bug or if something changes in the official WordPress instructions before I notice it and update this page. Of course, the script is provided as is, without warranties, and you should (because you should do it anyway!!!) back everything up right before running it.

The script could have been even shorter. I deliberately wrote it in that way to have the same steps as the procedure described at WordPress.org, and make it easier to update if/when something changes in WordPress. If you find errors or ways to improve it, please write them as a comment and I will update the page as soon as possible. Thanks.

A couple of weeks ago, I was thinking about how I may build an advanced search utility for my own email archive. One way to make complex queries on the archive seemed to be to put it all into a relational database. Since the Dbmail system stores email in that way, I asked its developers and Harald Reindl (an email administrator at The Lounge who already uses Dbmail: I found him in the PostFix Mailing list archives) if Dbmail could be used in that way.

The feedback I got made me change my mind about how to rebuild my own email search system, for the reasons explained below. At the same time, how and why Harald and his company use Dbmail seemed really interesting. Here’s the story.

About doing complex email searches with Dbmail

Harald explained to me that:

you should only do searches in the mail client, via IMAP. The Dbmail database is not nice to search for messages because they splitted in all their mime-parts and many db-records. Therefore, even if your search is successful, it would be hard to get a complete message without studying the Dbmail sources. So, since the search is running over IMAP and with the capabilities of Imap you can’t do more complex searches, even if the backend is a relational database. However, searches can be faster, just because the messages are splitted and indexed on the other side. Even if Dovecot in the latest versions builds an index too, so I would not expect any difference. It is surely possible to make a backend in PHP or what else language to search in the Dbmail database, but be careful about references to not display messages not owned by the user who starts the search

Why you may want to use Dbmail

Harald: I chose Dbmail because it has a 100% Mysql-backend configuration and the possibility to have a synchronized backup-slave in the network, which you can stop everytime to make consistent snapshots for offsite backups without interrupting the mail server. We are using Dovecot as proxy in front of Dbmail for several reasons:

• it supports more auth mechanisms than dbmail
• it supports TLS/SSL directly
• it supports replaces (% to @) since historically many users are configured with %
• postfix supports dovecot directly for SASL-Auth, so you have the same auth-mechs and encryption options for pop3, Imap and smtp
• security: I think it would be hard to exploit Dbmail through Dovecot (whereas exploiting directly dovecot seems harder, since it has only the user-logins)

We decide to migrate to Dbmail because we were running Apple servers (*brrr*) with Eudora mail server and I needed a replacement running on Linux/Vmware-hosts.

Since my main job is PHP/Mysql-developer, a full db-driven server gives me options to write special interfaces for all needs, doing cron jobs for notifies, cleanups, implementing auto-reply-backends and many nice things without touching text configurations.

Postfix is also nearly 100% Mysql-driven in our environment. There are great options for forwarders/aliases on both sides. If there is anything to do you only have to figure out which of both components can do what you want best, with the smallest side-effects on the whole system.

It was a really hard job to write a PHP-backend with 20.000 lines of code in few weeks, while learning much about mail servers. However, this has been running perfectly since the summer of 2009 with only a few “WTFs” and optimizations, but they are because of little know-how at the beginning.

This means after two months working day and night there was a complete solution, and for the second mailserver the whole virtual machine was cloned in 2010 and needed only minimal configuration. The Mysql replication is a big improvement for backups, here is how we use it:

• VMware-ESXi-Cluster
• Mailserver on one host
• Clone of the first machine on the second
• Replication between both of them
• Replication is a separate Mysql instance, read-only port 3307
• the replication can be used for Postfix as fallback, since readonly is enough for that
• on the backup-machine a normal instance with a copy of the db is running…
• …so you can start dbmail-imapd with this instance and directly access it via Thunderbird
• once a week both mysqld are stopped and rsync-ed from replication to backup
• before that happens the last backup goes to “mysql-last-week”
• once per day the replication is stopped and a offsite-backup per rsync done

So we have permanently access to the mailbox versions from last sunday and we can switch a week back with a simple script and restore a customer with imapsync between both machines and have a daily backup on the other end of the city. And through all the time it takes to do this, the mail services are not down for one second. I would not know how to do this with a file driven mail server, because there are permanently files changed and nobody knows if the backup is clean enough if it is ever needed.

Around October 2010 I migrated from Drupal to WordPress my bilingual websites Stop and Strider. Eighteen months later, WordPress has confirmed to be better than Drupal for my own needs, as far as those two websites are concerned (I still stick to Drupal for other websites).

There is, however, one part of those WordPress websites that has just become a big problem for me, and is how to keep them (looking) multilingual. I think I have found a solution for it and have explained it here for two reasons. First if I am right, this trick will be probably useful to many other WordPress users. The other is caution. I said I think I have a solution: I have done a little testing on a dummy installation (see below) and everything seems to work. However, before implementing it on the actual blogs, I’d really like to have some feedback from the community, just to make sure I am not forgetting something important (which I may have very well done).

When I started those two websites, I wanted them to have one name/base URL as simple and short as possible, so that I could just tell people “go to stop.zona-m.net” and then they would immediately find the version in their language, versus “go to {it,en}.stop.zona-m.net”. English content would have URLs like stop.zona-m.net/2010/11/article-title/, while all Italian content would have the /it/ suffix, as in stop.zona-m.net/it/2010/11/article-title/ and there would be automatic links from each version to the other. Drupal 6 does this, no problem.

Do you REALLY need a multilanguage blog?

When I moved to WordPress the best way to keep this architecture with one WordPress install seemed, and very likely is, the WPML plugin with English as primary language. After a while, however, I realized that WPML doesn’t seem to do all I need and had in Drupal. This may be my own fault but, for example, I found no way in WPML to:

• use language-dependent widgets, ie how to tell WPML that a certain widget, e.g. an RSS feed, must be shown only in the English or Italian version of each page (update 2011/04/15: Amir, one of the WPML maintainers, points out in the comments to this post that this is now possible with WPML and it is explained here)
• link to each other, as Drupal lets you do, the different names in each language of each category, in order to have automatic links between, for example, http://stop.zona-m.net/category/digiworld/ and its italian equivalent http://stop.zona-m.net/it/category/digimondo/, or to have automatic preselection of the right categories in the new target language when you translate an existing article.

In addition to this, I also realized something more important. Very likely, no matter how cool it looks, I don’t really need the links with the nice flags between english and italian versions of each page. By this I mean that my readers don’t really need it and almost never use it, because almost all of them only use the website in only one language.

Finally, at the beginning of 2011, WPML announced it would become a commercial/proprietary product. Putting all this together made me decide to find another solution.

Making two copies of WordPress look like one bilingual website

Those two websites run on a Virtual Private Server, on which I can add databases, change apache settings and so on as much as I want. This and all I explained in the previous section made me consider splitting each one of those websites in two independent, monolanguage WordPress installs. The result would be something a little bit less convenient to manage for me than it was with WPML, but nothing substantial, and almost transparent for all my readers.

As long, of course, as no URL changes in the migration! I want all the English URLs to remain the same (no problem) but I also do NOT want to “lose” or change the italian URLs. Here’s how I have achieved this on a test install, please tell me if there are weaknesses!

I have set up a test bilingual blog at bits.zona-m.net, with the same versions of WordPress and all the plugins as the real websites, in the /var/www/html/wp/tips directory, associated to the wp_tips MySql database and with this Apache server configuration:

  <VirtualHost *:80>
      ServerAdmin marco@digifreedom.net
      DocumentRoot /var/www/html/wp/tips
      ServerName tips.zona-m.net
      AccessFileName .htaccess
      CustomLog logs/tips.zona-m.net.access.log combined
      ErrorLog  logs/tips.zona-m.net.error.log
  </VirtualHost>

Then I created a few posts in it, both in English and Italian, with links to each other, to have something meaningful on which to test the following method.

In order to split that WordPress bilingual blog in two independent ones without breaking any URL already created, first I cloned its MySql database:

  # mysqldump -u root -p wp_tips > ~/dumb_wptips_bilingual.sql
  # mysql -u root -p
  > create database wptips_it;
  > use wptips_it;
  > source /root/dumb_wptips_bilingual.sql;
  > GRANT ALL PRIVILEGES ON wptips_it.* TO "sameuser_ofwptips"@"localhost" IDENTIFIED BY "password";

then I duplicated the whole, already working WordPress installation of tips.zona-m.net to another directory:

  cp -r -p /var/www/html/wp/tips /var/www/html/wp/tips_it

and modified the wp-config.php file in tips_it to point at the wptips_it database. Next, I modified the Apache configuration to “fetch” all and only the URLs starting with http://tips.zona-m.net/it from that second directory. In order to do this it is necessary to create an alias in the httpd.conf file:

  <VirtualHost *:80>
      ServerAdmin marco@digifreedom.net
      DocumentRoot /var/www/html/wp/tips
      ServerName tips.zona-m.net
      Alias /it /var/www/html/wp/tips_it
      AccessFileName .htaccess
      CustomLog logs/tips.zona-m.net.access.log combined
      ErrorLog  logs/tips.zona-m.net.error.log
  </VirtualHost>

and to modify the .htaccess files in the two folders as follows:

  # more /var/www/html/wp/tips/.htaccess
  <IfModule mod_rewrite.c>
  RewriteEngine On
  RewriteBase /
  RewriteRule ^index\.php$ - [L]
  RewriteCond %{REQUEST_FILENAME} !-f
  RewriteCond %{REQUEST_FILENAME} !-d
  RewriteRule . /index.php [L]
  </IfModule>
  # more /var/www/html/wp/tips_it/.htaccess
  <IfModule mod_rewrite.c>
  RewriteEngine On
  RewriteBase /it/
  RewriteRule ^index\.php$ - [L]
  RewriteCond %{REQUEST_FILENAME} !-f
  RewriteCond %{REQUEST_FILENAME} !-d
  RewriteRule . /it/index.php [L]
  </IfModule>
  #

At that point, “all” was left was to convince the cloned WordPress installation in tips_it and its MySql database wptips_it that they now live at another base URL with the it/ suffix, that is tips.zona-m.net/it. It seemed to me that the simplest/safest method in this case among those recommended in the official Change the WordPress URL page is the one called “Edit functions.php”. So I added these two lines to the functions.php file of the active theme:

  # cat /var/www/html/wp/tips_it/wp-content/themes/twentyten/functions.php
  <?php
  /*
  update_option('siteurl','http://tips.zona-m.net/it');
  update_option('home','http://tips.zona-m.net/it');
  */
  /**
   * TwentyTen functions and definitions
   *

then, as the WordPress page above recommends, I loaded the admin page a couple of times and immediately after I removed those two lines. Last but not least, I deactivated English in WPML in the “Italian” blog and Italian in the “English” one. Done! Apparently, everything works, as you can check by yourself by going to http://tips.zona-m.net and http://tips.zona-m.net/it. All the URL created when there was only one bilingual WPML/WordPress install are still valid with the same website URL: you can still visit both http://tips.zona-m.net/2011/04/here-we-go-with-my-second-english-post/ and http://tips.zona-m.net/it/2011/04/versione-italiana-del-secondo-post/, even if they are now served by different copies of WordPress. The only thing that has been lost, but as I said I do wonder if I really needed it, are the links between the several versions of each page.

If this works, I will migrate the real websites to this structure, deactivate/remove WPML completely, then upgrade both of them to the last version of WordPress, customized their layout separately and go on. The only things I should lose are, I believe:

• possibility to handle posts in both languages from the same browser window. Worth the hassle for me, if it both maintains the old URLs and lets me customize each language at will
• clear signs for Italian visitors that there is also an English version of each website and vice-versa. I feel that just adding a link in the top menu bar to the home pages of each language should be enough, since I’m pretty confident that almost nobody so far actually used both two languages
• (totally wild guess, not sure if this really is an existing issue) search engines temporarily pissed off and de-ranking each post because it doesn’t have links from the version in the other language???

What do you think? Have I missed something? Is there any security hole or other weakness that will come and bite me if I migrate the real website? Should I do something else? Please let me know!

Every now and then, a question like this pops up on some email server management forum:

I’d like to be able to reject connections from remote IP addresses if they’re from certain countries.

The usual reason is either that the sender only receives spam from those countries and is convinced that this will always be the case, or that, since the sender doesn’t need or want to exchange email with anybody in those countries, why bother at all with filtering messages from there?

Personally, I feel that both reasons are very weak, to say the least, and could turn against (the users of ) the system administrator that implements them. I’ve explained why in another post. However, this is a relatively frequent question and if an email system administrator really wants to do it, at the very least, he or she ought to do it right. Besides, even if I don’t do it myself, I do want to understand how all this works, both for the fun of it and because the same tricks may be used to flag or delay, instead of plainly rejecting, email that “comes from certain countries”. For all these reasons, I’ve reformatted here some tips appeared in a recent thread on the postfix list. Please remember that (see above and bottom) as far as this page is concerned, I’m just an editor! I have just rearranged in a more readable form an email discussion! If there are errors or other ways to add this function to a web server, just let me know in the comments and I’ll update this page as soon as possible, because I do want it to be as accurate and complete as possible.

As far as I understand, there several ways to do geographically based email rejection in PostFix, but with other servers it should be possible to do similar things.

IP-based black lists

If you want this mere as an anti spam measure, I advise to use IP-based black lists. They are more powerful and are adapted for spam hosts around the globe as they appear. However, the original poster reported that “I’m already using three RBL’s (b.barracudacentral.org, zen.spamhaus.org and dnsbl.sorbs.net) yet I’m still seeing a fair amount of spam coming in from Russian and Romanian IP ranges that isn’t blocked.

Per-country access map, policy plugin, milter, or deep content filter

How to do this with PostFix (thanks Wietse!) is explained in these pages (if you know how to do the same with other servers, please let me know)

• http://www.ipdeny.com/
• http://www.postfix.org/access.5.html
• http://www.postfix.org/cidr_table.5.html
• http://www.postfix.org/SMTPD_POLICY_README.html
• http://www.postfix.org/MILTER_README.html
• http://www.postfix.org/SMTPD_PROXY_README.html
• http://www.postfix.org/FILTER_README.html

Check URI/domain based DNSBLs

You may also want to try this script: it scans mail headers for domains and checks them against 3 URI/domain
based DNSBLs. hese are user configurable, so you can add more URI/domain DNSBLs if you wish. Do NOT use IP based DNSBLs with this, only URI/domain DNSBLs.

“I’d use both of the above tools regardless of the Russian/Romanian spam problem. And if you absolutely will never receive mail from these countries, or very infrequently, using ipdeny.com as Wietse mentioned is a sure fire way to block the entire country. I use it myself against Russia, Romania, China, Korea, Malaysia, and a few others. If you do this it’s a good idea to implement the dnswl.org whitelist.”

Alternatively, you can also use DNSBL maps like http://countries.nerd.dk/, keeping in mind, as their author himself says, that “countries.nerd.dk is NOT a list of spammers, it is an IP-to-country DNS mapping service”.

Postfix PCRE bot spam killer

give this a go. It is reported as simple to implement. It targets dynamic IP hosts and some others that simply shouldn’t be sending SMTP mail directly. It stops quite a bit that Zen (CBL+PBL), SORBS DUL, and other DNSBLs miss. It’s a simple PCRE table consisting of 1600+ regular expressions that matche rDNS strings of undesirable hosts. Myself and a few others on this list use it with good results. “It’s not magic, but just another good tool to have in your AS arsenal”.

DNSWL

Postfix > 2.8 can query DNSWL directly. For earlier versions you’ll need to get permission to rsync the data file. You’ll need to run it through a script to add “permit_auth_destination” to each IP line transforming it into a CIDR table.

SQL access maps

you can convert the GeoIP map into an sql access map (see also the GeoIP how to). With that you can adjust your sql queries to implement whatever you want to do. You can convert the GeoIP map to a DNSBL and use rbldnsd

Other solutions

Use Milter-Greylist based geoip detection, this Python GeoIP policy server, these Header checks or the list in this (not so recent) article on Asian spam.

Again, any pointer / suggestion on how to improve this page is welcome.

(this is a review that i originally posted somewhere on Slashdot, IIRC)

Linux E-mail, Second Edition is a book written for Packt Publishing by I. Haycox, A. McDonald, M. Back, R. Hildebrandt, P.B.Koetter, D. Rusenko and C. Taylor. Linux E-mail contains all the information you need to handle all email services for a small organization on your own Linux server: send and receive messages, provide access to email accounts from any web browser, block as much viruses and spam as possible, possibly before they even enter your server, backup all email and configuration data in the most effective way and, finally, configuration of some email clients. Since I have written some basic tutorials on the same argument and had already configured my own email server, using almost the same set of tools described in this book, I got a free copy from Packt Publishing for review.

Target readership

“Linux E-mail” is aimed to (quoting from the Preface) “beginner or intermediate level System Administrators in small businesses, who want to set up a Linux-based e-mail server without spending a lot of time in becoming experts in individual applications”. I can only confirm this assertion, but I think there is also another, maybe larger category of Free Software users who may find this particular book useful. I refer to anybody who doesn’t run a full fledged mail server, but still wants as much protection as possible against spam and viruses on his or her Linux desktop, and maybe is also interested to know how email delivery works behind the curtains: this book can be an interesting, effective resource also for this category of readers. In any case, you do need at least a basic understanding of the structure of a Linux system and of the command line to use this book.

Structure of the book

The first chapter, “Linux and E-mail Basics”, begins by explaining why, in this Gmail era, a small organization should still bother to manage its own email by itself. After that, the chapter goes into how email travels through the Internet, which kind of servers and protocols are involved in each phase and the structure of an email message. Finally, there is one thing which I particularly appreciated, because you need to get it right if you’re managing your own email: the role of DNS and DNS records in email delivery.

After this introduction, Linux E-mail does not describe just the principles and general, theoretical criteria that you should follow when setting up a Linux email server. The authors have chosen one specific program (Postfix, Courier-IMAP, Squirrelmail, Procmail, SpamAssassin and ClamAV) for each part of the job. The book devotes at least one chapter to each of them, describing in full detail what each program does, how it works, how it fits in the big picture and how to install, configure and also test it in the most common scenarios. The last chapter, instead, explains how to backup the server, using a few tricks that you’ll find quite useful for every kind of backup, not just email ones.

What’s good

This book has two main advantages. First, it is very complete. Secondly, it has all you really need for this particular task in one place, organized in a coherent way and explained in a clear, easy to understand style. Sure, you could find almost everything that’s in the book scattered over a lot of man pages, online tutorials and mailing lists. After all, as I said, that’s just the way I myself put together my server a couple years ago. However, just because I did it myself through online docs, I can confirm that having a book like this on your desk makes you save a lot of time and (above all) makes it much easier to understand how all the pieces work together. Email handling and terminology can be quite time-consuming to figure out without assistance. I found particularly useful from this point of view the chapters about:

• stopping spam with Postfix (good explanation, with examples, of SMTP restrictions!)
• troubleshooting Postfix problems
• authenticate Postfix users with Cyrus-SASL
• dry testing and debugging procmail recipes
• ClamAV post installation testing

Another thing that is good, in case you were wondering, is that most of this information is not going to be outdated before you even buy the book. All the programs described in Linux E-Mail are stable backend software which is quite unlikely to radically change in the short/medium term.

What’s less good

There are a few repetitions here and there, but that’s a minor issue. One thing that, personally, I found useless are the parts on compiling and installing each application from source code and those describing how to configure Microsoft Outlook and other clients to interact with the server, handle authentication and so on. This is all information that it’s already explained somewhere else and is not strictly about configuring a server. However, it’s well written and detailed as the rest of the book, so it will probably be useful to hter readers.

All in all, there is only one thing which, in my opinion, should have been mentioned in a book like this but isn’t: how to add support, at least for sending email, for the DKIM (DomainKeys Identified Mail) and SPF (Sender Policy Framework) protocols. I know that many skilled email administrators ignore (if not hate…) DKIM and SPF and understand the reasons.

However, when I set up my server I found that adding such support was necessary in order to get my email accepted without question by Hotmail and Yahoo. Maybe this is not true anymore, but I’ll leave my server as is (if it ain’t broken…). Readers starting as email administrators with “Linux E-Mail” must just remember that they’ll have to study this one topic without its support. Everything else they’ll need is in the book.

Wordpress is a great publishing system, but managing it manually can be a very time consuming process. This is especially true when you want to upload lots of posts, or if you would like to write content in your preferred, full-blown text editor and then have it “magically” appear online.

Wordpress takes care of these needs allowing remote posting via email or the WordPress XML-RPC interface (if you enable the WordPress, Movable Type, MetaWeblog and Blogger XML-RPC checkbox in goinig to Settings > Writing > Remote Publishing). The first method is explained here and in other places, but requires setting up a dedicated email account. For several reasons I preferred not to do it that way, so I looked at the other system.

PHP scripts for this purpose are here and here. The first uses the cURL capabilities of PHP to send the data over SSL. The second uses IXR, the Incutio XML-RPC Library for PHP, to “incorporate both client and server classes, as it is designed to hide as much of the workings of XML-RPC from the user as possible”.

Both those scripts work, and the second can also be used to edit existing posts or get lists of the latest published articles. However I was looking for something that didn’t require PHP (personal preference, really). Eventually, I found the WordPress-CLI utility by Leo Charre and have already used it successfully to upload hundreds of posts to several of my WordPress websites (see bottom of this page for examples). Here’s how I did it.

Wordpress-CLI installs like any other Perl Module, see the instructions in the README file. In order for it to work, however, I also had to download and install from CPAN the Perl Modules Getopt-Std-Strict-1.01 and LEOCHARRE-Debug-1.03.

Once everything is installed, you’ll have in your path a script called wordpress-upload-post. Run it at the command prompt or in a script, giving as options the name of the HTML file containing the post, plus its required publication date, title and category, as well as your user name and password, and you’ll have your post online.

To make things faster, I use wordpress-upload-post inside this script:

  #! /bin/bash
  # usage: post2wp.sh postfile blogname
  # postfile: text file containing post content in txt2tags format
  # blogname: name of blog

  POST=$1
  BLOGNAME=$2

  HTML=/tmp/tmp_wordpress_post.html
  ACCOUNTS_DIR= "$HOME/.blog_accounts"

  #########################################################################
  #extract title, category and publication date of the post
  TITLE=`grep '%TITLE: ' $POST | cut -c9-`
  CATEGORY=`grep '%CATEGORY: ' $POST | cut -c12-`
  DATE=`grep '%DATE: ' $POST | cut -c8-`

  YEAR=`echo $DATE | cut -c1-4`
  MONTH=`echo $DATE | cut -c5-6`
  DAY=`echo $DATE | cut -c7-8`
  HOUR=`echo $DATE | cut -c9-10`
  MIN=`echo $DATE | cut -c11-12`
  DATE="$HOUR:$MIN $YEAR/$MONTH/$DAY"

  rm -f $HTML.tmp*
  txt2tags -t xhtml --no-headers -i $POST -o $HTML

  ###########################################################################
  # source blog parameters: user name, password, XML_RPC url

  if [ -e $ACCOUNTS_DIR/$BLOGNAME ]
  then
    source $ACCOUNTS_DIR/$BLOGNAME
  else
    echo "Error! $BLOGNAME account file doesn't exist!"
    exit
  fi

  ###########################################################################
  # upload post to blog
  WP_OPTS="-D '$DATE' -t '$TITLE'  -c '$CATEGORY'  -u $USER -p '$PW' -x $XMLRPC_URL"
  WP_CMD="wordpress-upload-post $WP_OPTS $HTML"
  eval $WP_CMD
  exit

I use txt2tags as source format for most things I publish online. It is a very simple ASCII markup format, that I customize adding to each article to be published on WordPress comments like these:

  %TITLE: Is it OK for a School or Charity to accept software donations?
  %CATEGORY: Digiworld
  %DATE: 200707010900

The script extracts these variables from the source text and reformats them in the way required by wordpress-upload-post.

Next, it converts the article from the ASCII markup to HTML calling the txt2tags Python Script. Password, user name and the URL of the wordpress blog to use to upload the HTML post are in a separate file ($ACCOUNTS_DIR/$BLOGNAME) that has this format:

  USER='your user name'
  PW='your password'
  XMLRPC_URL='http://your.blog.home.page/xmlrpc.php'

and is read right after generating the HTML version of the post. The last part of the script, “upload post to blog” concatenates all the parameters in one option string WP_OPTS, build the publishing command WP_CMD and evaluates, thus publishing your post online. Enjoy! Of course, if you only post once in a while you won’t save much time, but if you ever need, as I did, to post lots of stuff, try this!

What’s missing

wordpress-upload-post and this script work great (see bottom of page), but they aren’t perfect. The biggest limit right now of WordPress-CLI is that you can’t specify WordPress tags or, if you have a multilingual blog, the language of the current post. I’d also like to use it to add comments to existing posts, but that’s not essential really. I discussed these things with Leo. His answer is that “some of these things simply won’t work. For example- adding tags- because xmlrpc.php does not implement a call to add a tag. I’ve made some hacks to be able to do so- but this works on a local/server level. [also] I can’t recall right now if they have a comment call”. Leo also asked me to forward to all readers of this page this invitation:

I’d be open to actually share in maintenance of things like WordPress::XMLRPC and a WordPress::CLI revised version. If you want to do this level of changes/additions, we could set up a branch off some cvs server… and… implement bugzilla on some server- to keep track of changes and todos.

Another thing I’d like to figure out is a way to upload images to WordPress so that they are associated to that post and get thumbnails. If that were possible, it would be easy to figure out in advance the right URLs for both the images and their thumbnails, and add them to the txt2tags source. As it is today, in the rare cases where I need to upload with this script a post that does have images, I add them in a second moment by hand. Suggestions?

How I’ve used this script

If you are curious to what real posts written in txt2tags format, then converted to HTML and automatically uploaded to a WordPress blog look like, and don’t mind a bit of self-promotion, have a look at the following links. I have already used the script above to:

• upload (see example above) Is it OK for a School or Charity to accept software donations? and all the other chapters of the Family Guide to Digital Freedom, generating its Table of Contents right after the upload with a direct query to the WordPress MySql database!
• upload the whole text of my Open Data, Open Society report
• rebuild from scratch, semi-automatically, the whole website of the Run Up to date Linux Everywhere project
• publish most of the English content on my backpacking website

Gnuplot is a really great plotting utility, that can be used either interactively or automatically, from inside scripts of all sorts. However, sometimes it can be quite difficult to use simply because there are lots of documentation, but it is hard to figure out exactly what piece of documentation you should read and where it is.

This is a big problem, because the way you plot data, that is which Gnuplot options you set, can make a huge difference in the readability of the plot. For example, I had this ASCII file called initial_data.dat, that lists the number of daily visitors to four different websites I follow (the complete file used for this article is downloadable from the link at the end of this page):

  20110101 2481 89 896 98
  20110102 2341 83 1341 762
  20110103 560 208 1795 890
  20110104 936 409 1665 419
  20110105 534 562 937 341
  20110106 171 728 953 612
  20110107 200 199 1297 569
  20110108 521 557 990 295
  20110109 1592 227 535 466
  20110110 1363 211 21 299
  20110111 437 222 110 302
  ...

The file format is very simple: the first column is the date, then there are four space-separated columns, one for each website. However, if you tell gnuplot to plot those numbers as simple lines with these commands (the plot command here is wrapped around for readability, but it must stay all on one line!):

  set terminal png size 1400, 600
  set output "with_lines.png"
  set key center top
  set style fill solid
  set xdata time
  set timefmt "%Y%m%d"
  set format x "%b %d"
  set ytics 300
  set y2tics 300 border
  set xlabel "Website traffic from 20110101 to 20110322"

  set style line 1 linetype 1 pointtype 0 linewidth 1 linecolor 6
  set style line 2 linetype 2 pointtype 0 linewidth 1 linecolor 7
  set style line 3 linetype 3 pointtype 0 linewidth 1 linecolor 8
  set style line 4 linetype 4 pointtype 0 linewidth 1 linecolor 9
  plot ["20110101":"20110131"][:] 'initial_data.dat' using 1:5 t   \
                                  "Website 4" w lines linestyle 4, \
   'initial_data.dat' using 1:4 t "Website 3" w lines linestyle 3, \
   'initial_data.dat' using 1:3 t "Website 2" w lines linestyle 2, \
   'initial_data.dat' using 1:2 t "Website 1" w lines linestyle 1

you’ll get the diagram on the left, that isn’t really informative: the lines continuously overlap, so it’s hard to see quickly which website had the highest traffic on each day. It’s even harder to see the total amount of traffic on each day. The solution is to stack the graphs, that is to use each line as the foundation on which to draw the others. In other words, instead of drawing four lines, one for each website, I wanted to draw:

• traffic for website 1 using the numbers in the second columns, as they are in the data file
• traffic for website 2 //summing its numbers in column 3 with the corresponding numbers in column 2
• traffic for website 3 //summing its numbers in column 4 with the corresponding numbers in column 2 and 3
• traffic for website 4 //summing its numbers in column 5 with the corresponding numbers in column 2, 3 and 4

It is possible to do this kind of stuff directly in Gnuplot, by calling utilities like sed or awk directly from within the Gnuplot command file as explained in this page. However, in my opinion it is much better and cleaner to do stuff like this before calling gnuplot, with an auxiliary script in Perl, Python or whatever strikes your fancy. The reason is that in this way you can pre-process the numbers in much more flexible, and self-documenting ways that Gnuplot is capable of: another way to say this is that it is better to keep number calculation and number plotting completely separate. For example, I also run an expanded version of the script below that calculates and stacks on the fly the moving averages over one month of the traffic to each website.

In this case, using a separate script to “stack” the data is also quick. All I had to do to rearrange the numbers as described above was to feed the data file to another script and save the result as another data file:

cat initial_data.dat | perl stack_data.pl > stacked_data.dat

Stack_data.pl is the simple Perl script below:

  #! /usr/bin/perl

  use strict;
  my $I;
    while (<>) {                            # read the data file from Standard Input
      chomp;
      my @FIELDS = split ' ', $_;           # put each column in a different
      my $DATE = shift @FIELDS;             # field of the @FIELDS array
      print "$DATE ";
      my $LINE_TOTAL = 0;
      for ($I = 0; $I <= $#FIELDS; $I++) {  # re-print the data, but making of
         $LINE_TOTAL += $FIELDS[$I];        # each column the sum of all the ones
         print " $LINE_TOTAL";              # preceeding it
      }
      print "\n";                           # print the new numbers to Standard Output
   }

Here is what you get when plotting the stacked_data.dat file with the same style of the initial graph, that is with the same “plot” command. That is much better but, in my opinion, it wasn’t good enough yet. Personally, I think filled areas are more informative in cases like this. The way to obtain them is to use another style in the plot command. Replace the last line of the Gnuplot instruction file with this one (on ONE line!):

  plot ["20110101":"20110131"][:]
  'initial_data.dat' using 1:5 t "Website 4" w filledcurves x1 linestyle 4,
  'initial_data.dat' using 1:4 t "Website 3" w filledcurves x1 linestyle 3,
  'initial_data.dat' using 1:3 t "Website 2" w filledcurves x1 linestyle 2,
  'initial_data.dat' using 1:2 t "Website 1" w filledcurves x1 linestyle 1

and this is what the plot will look like. Nicer, uh? Please note that the order in which each column is read and plotted in the command above is important. When Gnuplot draws a new line or area, it does it over what it has already drawn or paint. With lines (second diagram in this page) it makes no practical difference. With areas, instead, it’s esential to paint them in the right order. This means that you must plot first the highest area (“Website 4″ in our example) that is the one that is the sum of all the original columns. Then comes the one that is the sum of only the first three columns etc…