Of course it does, it's get and post requests which your browser does to server. And if you need to modify in bulk records in your web application (Koha in our example) you might wonder about writing a script which does job for you.

It's not really hard. WWW::Mechanize provides us with web browser scriptable in perl and following scripts logs in into Koha's interanet and edit items specified in file:

#!/usr/bin/perl

use warnings;
use strict;

use WWW::Mechanize;
use Data::Dump qw(dump);

# we will use %d and %s to insert values from file
my $url_template = 'https://localhost:8443/cgi-bin/koha/cataloguing/addbiblio.pl?biblionumber=%d';

our ( $user, $passwd );
require 'config.pl'; # edit $user and $password in config.pl file

my $login_url = 'https://localhost:8443'; # Koha intranet

my $mech = WWW::Mechanize->new( autocheck => 1 );


warn "# login $login_url\n";
$mech->get( $login_url );

$mech->submit_form(
	fields => {
		userid => $user,
		password => $passwd,
	},
);

sub modify_field; # declare later

while( <> ) {
	chomp;
	my @v = split(/\s+/,$_);
	warn "<< ",dump(@v),$/;

	my $url = sprintf $url_template, @v;
	warn "# url $url\n";
	$mech->get( $url );

	my $form = $mech->form_number( 1 ); # XXX 1st form

	# XXX edit fields

	modify_field $form => 'tag_008_subfield_00' => sub { s/^(.{24}).(.+)/$1d$2/ };

	$mech->submit;
}

exit;

# magic to find field name by partial match from beginning
sub modify_field {
	my ( $form, $field, $coderef ) = @_;

	my @inputs = $form->inputs;
	my ( $name, $value ) = map { $_->name, $_->value } grep { defined $_->name && $_->name =~ /^\Q$field\E/ } $form->inputs;
	die "can't find $field in ", $mech->dump_forms unless $name && $value;

	$_ = $value; $coderef->($value);
	my $new = $_;

	if ( $value eq $new ) {
		warn "WARNING: $name not changed [$value]\n" if $value eq $new;
		return;
	}

	warn "$name\n\tOLD: $value\n\tNEW: $new\n";

	$mech->field( $name, $new );
}

This script proved to be very useful for us, and hopefully it might be useful for other users of Koha also.

As you know by now, I'm working pure free software implementation of RFID support for our library. This time, I decided to tackle problem of printing RFID cards using EVOLIS Dualys printer.

This is experimental support for EVOLIS Dualys 3 printer with black ribbon (K) to provide pixel-exact driver with support for two-side printing.

Existing cups driver is available at

http://www.evolis.com/eng/Drivers-Support/Product-support/Dualys-3

but I haven't been able to make it print on both sides of cards, partly because using dumplex option in cups seems to segfault GhostScript and/or rastertoevolis cups filter depending on combination of duplex options.

I also needed pixel perfect transfer to printer, and cups bitmap format is always in color, leaving final pixel modifications down to cups filter which always produced differences between file sent to printer and perfect black and white rendition of it.

SCRIPTS

Current toolset consists of following scripts:

• inkscape-render.pl card/template.svg 201008159999 login Name Surname

  Generate pdf files from Inkscape SVG template in card/ using print-front and print-back object IDs. Layers doesn't work since we can't toggle visilbity easily. To print more than one object gruop them and change ID of group.

  After pdf files are created, GhostScript is used to rasterize them into pbm (monochrome) bitmaps.

• evolis-driver.pl front.pbm back.pbm > evolis.commands

  Provides driver which generates printer command stream to print two-sided card from pbm files.

• evolis-simulator.pl evolis

  Simulator for EVOLIS printer commands which is useful for development. It creates one pbm file per page printed.

• scripts/evolis-command.pl

  Command-line interface to send commands to printer and receive responses. Supports readline for editing and history. Requires local parallel port connection, probably to USB parallel device.

EXAMPLE

Following is simple walk-through from svg image in Inkscape to evolis command stream which can be executed in top-level directory of this distribution:

  ./scripts/inkscape-render.pl card/ffzg-2010.svg 201008159999 Ime Prezime
  ./scripts/evolis-driver.pl out/201008159999.front.pbm out/201008159999.back.pbm > evolis
  ./scripts/evolis-simulator.pl evolis
  qiv evolis*.pbm

Allmost two years ago, I began my experiments with RFID, writing support for 3M 810 RFID Reader. Then I tried to make web interface for RFID it in Koha.

This woked, but having separate Comet server was too much complexity for me, so I decided to implement JSONP directly in RFID driver code. Hack, I allready had HTTP server, and local JavaScript interface, so why not?

Move to JSONP allowed me to use original Koha web interface, and just overlay RFID information and form submission code as simple jQuery snippet.

But, two years of development and trying out different approaches produced not-quite-production-quality code. So, I began rewrite called Biblio::RFID. It splits RFID reader support from HTTP and JSONP servers and couples this with documentation and tests. I have production use for it this summer, involving programming of RFID cards as they are printed out, so expect it to change during next few weeks. After that I will push it to CPAN, but I would love to get feedback and comments before that.

Update: It's also availale at CPAN.

As you might know by now, I started an informal podcast in Croatian about Free and Open Source software called Razgovori o slobodnom softveru. This blog post will try to summarize my experiences and serve as information for podcast guests on how to get the best possible audio quality.

Every podcast currently starts as google wave. While this isn't ideal, it does work for now. I would love some other service which would allow me to see information color-coded by users. Those notes are later used to write blog post with all links and information about episode.

First, let's get Skype out of way. Only reason why we aren't using free software for audio conversation is simple fact that everybody has Skype and there is great skype-call-recorder which allows us to get uncompressed audio recording with local audio on one channel and remove one on another which enables easy matching of audio levels for local and remote speakers.

While we are at audio levels, let me share a little secret: right after recording audio from Skype, I use Levelator to adjust audio levels. There is Linux version of it, but it doesn't seem to work for me on recent Debian. But, effect is so valuable to good quality podcast that I just use Windows version under wine.

As far as guest audio is concerned I have found following notes really important for my guests:

• use any microphone other than one on your laptop. Built in microphones catch typing sounds and all huss and buzz from machine itself. Even cheapest possible microphone will generate better results, just compare Ivan's audio from episode 0 (audio via built-in laptop microphone) versus episode 1 (which is cheap plastic external microphone)
• if you have headset with microphone, use it. Adjust your microphone level to be above your nose. That will prevent recording of air that you breath.
• When you need to take a pause while speaking, just do it! I edit whole podcast to remove bloopers and breading anyway, but it helps if you repeat last word of full sentence when you start over so it will fit in context (and intonation) of previous material.

Much more useful information can be found in Skype for interviews by Doug Kaye and Paul Figgiani (guys behind Conversations Network which also implemented Levelator mentioned above), so if you need to record on Windows or Mac take a look there.

For editing audio I use audacity which is great audio editor, but I still have to learn all keyboard shortcuts because using mouse to edit audio is too slow and cumbersome if you want real flow of montage (and I did video montage back in 1990 on local TV station, so I know unnecessary mouse clicks when I see them). It would be nice if labels moved when I do cut of audio, because editing of audio leaves labels at old positions which isn't useful since audio moved. To give you some idea of how much effort is involved in editing audio (mostly removing silences, bloopers and breading): for last 35 minute podcast, it took 420 edits to get it into final state. And most of single day. It's about the same as creating good presentation: I often need 6 times more than presentation length to create it. For audio editing, you can count on 10-15 times ratio for editing.

Finally, I hop over to some creative commons audio site (to be honest, it was first one in google search for CC music), pick one of top tracks and extract intro and outro music from it to put it at beginning of podcast and end. Finally, audio introduction and web address information for end are recorded and put on top of music.

I have two screens, and 9 virtual dekstops on my machines. Which sometimes makes it tricy to remember on which virtual desktop is xterm connected to klin. Fortunatly, I have that information in window title, so something as simple as:

xdotool search --name klin windowactivate

-- XXX dpavlin: search windows and active them
awful.key({ modkey }, "F12",
  function () 
    awful.prompt.run({ prompt = "window: " },
      mypromptbox[mouse.screen].widget,
      function (find)
        awful.util.spawn("xdotool search --name "..find.." windowactivate", false)
      end)
  end),

So you have a car. And you are a geek. You know kind of I don't really know or care how cars work. And then you got graph like this from your car:

If you take a look at OBD2, you might think that it's only useful to your car mechanic. But, as you can clearly see above, any technology which can draw graphs and uses computers is interesting to me.

You can take two different paths to getting sensor data from your car using open source software. Freediag is project which support dumb (basically level converters between your car OBD2 connector and RS232 or USB) cables. This approach requires very funky boud rates on serial port and lot of knowledge inside software, but offers future-prof route because you can hack software.
So I decided to buy one of those cables (available locally in Zagreb) and ended up with VW cable which did dump some data, but freediag didn't really helped in decoding them.

So, in next step, I opted for one of ELM327 based devices (which decode car protocol within microcontroller) and moved to OBD GPS Logger for data collection. This did work, and result is graph included in this post. In the process, I figured out why you can disconnect both cables (to car and to usb) from micro-controller in metal box: it's the only way to reset it, sigh!

However, now I have a few observations about state of free software for OBD2: I wasn't able to found any software which would introspect all my sensors and log them. This is quite strange since you can inquire car about supported sensors, so that will probably be next thing for me to do. If you have more information, please let me know.

We all speek utf-8 thease days, don't we? Well, not really... I got CSV file export and I couldn't guess encoding from simply looking into it any more. So I wrote gist to dump all Croatian 8-bit encodings in utf-8:

#!/bin/sh -x

file=$1

function encoding {
        echo "# $1"
        head $file | iconv -f $1 -t utf-8
}

encoding cp850
encoding cp852
encoding cp1250
encoding cp1252
encoding iso-8859-1
encoding iso-8859-2
encoding mac
encoding MAC-CENTRALEUROPE

Example usage:

./test-8bit-encodings.sh data/ESB_izvadak-tekuci.csv | vi -R -c 'set nowrap' -

I have spent few last weeks with my head down, adding persistence and changes tracking to Mojo Facets, turning it into much more interesting alternative to web-based data stores like DabbleDB. Idea was simple: I had all data in memory, I should be able to edit it, right?

Well, as it always turns out, if was about three weeks of development, but it moved Mojo Facets into much more interesting use case of making small tweaks to your input data.
Problem is how to keep those changes? Mojo Facets is never master provider for data so saving some kind of audit log which can be applied back on master data is of paramount importance. After all, if you edit that data, you might want to apply those changes back when you re-generate source file or pull new version from some other system.

First idea was to add simple audit log which records all requests in Mojo. I decided to call requests with parameters actions and store them on disk under /tmp. All I had to add was re-submit form in browser and a bit of interface around it. Same form with all parameters can turn Mojo Facets into peer-to-peer application: I just added checkbox which can change destination URL in action to another Mojo Facets installation and I got nice replication of actions to another instance.

But, all was not well. Editing data in browser generates update to specific entry in your dataset, so I decided also to record changes which include old and new field value, and all unique keys for this dataset.

This seems like such a small statement, but getting it up to point where you can load some data, edit it in browser and than apply that changes back on original data (after reboot) or on different dataset with same unique field.
Even better, it should be possible to apply changes log to master data. I prefer to think of it as a replication log to another system.

To integrate better with other systems, filters got export (and import) option which dumps them in simple, one line per entry text file which is accessible over http. It's perfect format it you want to quickly xargs that data into another script, for example to generate more source data with something as simple as:

 cat /srv/mojo_facets/public/export/isi-cro2.js/filter.autor.119 | \
 xargs -i ./bin/isi-download-results.pl 'CA={}'

Speaking of more dataset sources, I also added parsing on html tables, which should allow users at some point to just drop multiple tables from results page into single directory and load them as dataset. Your telebanking doesn't have export you need? No problem! Just save all pages to disk and you are ready to do.
Right now, table parsing needs a bit of heuristics to be really useful. It searches for table on page with correct number of columns, and has support for extracting of header or repeating first row (normal <td>) for column names.

All that would be unusable without profiling to turn it really snappy. This is first time I used Google Chrome for serious development, and while I still dislike it's inspector (firebug's dom inspector is much nicer to me), Speed Tracer extension was very useful for front-end part including network throuput and html/javascript overhead. On server side, I used Devel::NYTProf, and I will talk about it at Slobodni Fastival 4 in Čakovec, so see you there...

My server side faceted browser just got a bit better. In fact, it become 10 times better. But, let's try to explain this story step by step...

This week I will try to introduce faceted joins. Primary motivation is great Plants For A Future database which consists of more than one text file.

Use case is something like following:
I would like to know all plants which can have medical use, are edable and have perennial habitat (so I don't have to re-plant them every year).

And you can watch the video to see how easily this can be done:

But, this still doesn't make MojoFacets 10 times better than before. This is quite small dataset (still about 10 times bigger than Exhibit could handle), but I had new problem: 100Mb source file a bit less than 30000 items. To make it scale more I implemented pre-calculated filters and sorts. They serve same usage as indexes do in relational databases, but they are calculated on demand and stored in memory.

Let's see in action how does it work with ~30000 items:

In this video, we saw:

• starting memory usage of ~13Mb
• 100Mb dataset with 29869 items
• filter by autor with 45644 taking ~10s
• use regex filter ic,
• godina_izdavanja is numeric facet
• jezik filter using cro slo ser regexps and toggle it
• show popup title on filters
• turn off filters to show under 4s load time
• at the end, we consumed ~260Mb of memory

Like every year, we had our local Linux conference. It was very intense event (for first year I'm involved in real organization) and I can say it's all just a big blurb.

I had two tutorials, one about my Virtual LDAP and another one about creating Google like (horizontally scalable) cluster from library building. In this one, I covered a whole bunch of tools which I ended up using during last year:

• Webconverger is the easiest way to deploy Firefox on kiosks for public Internet access
• PXElator - full stack solution to network booting and spawning machines
• Sack - horizontally scalable (across cores or nodes) in-memory perl hash with remote code execution (close to data)
• mongoDB which I use for audit log in PXElator and feed it back to Sack after finding CouchDB too slow.
• Sysadmin Cookbook as a way to document HOWTO or SOP documents
• bak-git for tracking configuration changes
• Gearman and Narada didn't get all attention they deserved, partly because i wasn't able to make Narada work (I tried perl and php version in preparation for tutorial). But, I hope that I managed to transfer part of my fascination with distributed fork approach.

During the conference I wrote small project to index git log messages using Sphinx which might help you to get started with it.

I love btrfs, and he hates me back. Tonight he decided to take away my holiday morning by spinning single core and bringing server down for three and half hours until I migrated everything back to ext4.

I will miss snapshots, but with all troubles I had so far, it's not worth it. See you in few years...

Can MySQL be slow? Yes it can. It depends on disk speed (even if you have fancy SSD drives) and some tuning can turn down query time dramatically, especially if you are currently using default settings and/or your accumulated several gigabytes of data. Sure, there is documentation about optimizing MySQL, and bunch of information on Internet but where to start?

Tune MySQL parameters

For a start, download mysqltuner.pl or tuning-primer.sh and run them. While they won't give you exactly same recommendations it will give you nudge into right direction.

I would especially recommend setting innodb_buffer_pool_size to some reasonable number. This will provide immediate performance improvement, especially if you are running with default Debian settings and have couple of gigs of data.

Find slow queries

But linking to scripts wouldn't be worth a blog post. Next step is to find which queries are slow on your system. To do that, you can use Maatkit or more specifically mk-query-digest. You can feed it with /var/log/mysql/mysql-slow.log or run in on live system:

$ wget maatkit.org/get/mk-query-digest
$ perl mk-query-digest --processlist localhost --interval 0.01

If you are can modify application, dig into show queries. In my case, I found out that session handling in Koha takes more than 75% of all MySQL query time. I decided to leave this for future improvement. But, if you can't or don't want to modify application, simply move to next topic.

Tune Linux

Yoshinori Matsunobu had great presentation Linux Performance Tuning and Stabilization Tips at recent MySQL Conference & Expo in which he covered all various ways to optimize Linux for MySQL workloads. I will summarize here just few parts which I found useful for my workload which involves multiple LXCs, each with own logical volume using btrfs and sitting on battery backed RAID controller.

• turn off readhead

  hdparm -a 0 /dev/sdb
  

• use noop or deadline scheduler for block device on which is logical volume with MySQL

  echo deadline > /sys/block/sdb/queue/scheduler
  

• reduce swappiness

  echo 0 > /proc/sys/vm/swappiness
  

• disable write barriers (only with battery backed controller!)

  mount /mnt/koha -o remount,nobarrier
  

Results

So did all of this made sense? It's best to have clear goal when optimizing, and mine was less than 3 second web response. Without any MySQL tweaking and 3Gb database it was taking 5 seconds or more for each response. To my amazement, without any modification in application itself, I managed to produce visible improvement, even if only real comment I got was: It's not slow today :-)

I have been following btrfs for quite some time. First I got kernel oopses with full disk (which have been fixed since), and then I began testing snapshots for incremental backup. Few weeks ago, I have taken a plunge and migrated by production server over to btrfs. I will try to summary first few weeks of my experience with it.

For a start, forget about using btrfs-tools which come with your distribution (Debian in my case). It's probably too old to include delete snapshot option which is really needed if you don't want to fill up your disk eventually. So, hop over to btrfs-progs-unstable and compile your own utilities.

With latest utilities at hand, I decided to make logical volume for each of my virtual machines. Before installing (or migrating) machine, create sub-volume. This is important because snapshot work on file-system or sub-volume level, and if you want to create incremental snapshots, you need to have sub-volume to snapshot.

root@prod:~# lvcreate -L 50G -n koha raid5
  Logical volume "koha" created

root@prod:~# mkfs.btrfs /dev/raid5/koha

WARNING! - Btrfs v0.19-15-g8f55b76-dirty IS EXPERIMENTAL
WARNING! - see http://btrfs.wiki.kernel.org before using

fs created label (null) on /dev/raid5/koha
        nodesize 4096 leafsize 4096 sectorsize 4096 size 50.00GB
Btrfs v0.19-15-g8f55b76-dirty

root@prod:~# mkdir /mnt/koha

root@prod:~# mount /dev/raid5/koha /mnt/koha

root@prod:~# btrfsctl -S rootfs /mnt/koha
operation complete
Btrfs v0.19-15-g8f55b76-dirty

Now, you are ready to install your machine in /mnt/koha/rootfs. After you have done that, you can create backup snapshots using something like this:

root@prod:~# mkdir /mnt/koha/.snap

root@prod:~# btrfsctl -s /mnt/koha/.snap/2010-04-19 /mnt/koha/rootfs/
operation complete
Btrfs v0.19-15-g8f55b76-dirty

Even, better, you can hop over to my sysadmin cookbook and fetch mksnap shell script which will create hourly snapshots of your machine for nice incremental backups. This is all nice and well, but after a while you will see that your disk-space increases all the time (which is expected because you are creating one snapshot every hour, collecting all changes).

As you can see in graph above, after two weeks of such usage, I figured out what I will run out of disk space eventually, and even worse, disk fragmentation begin to take toll on performance of my server. So, I implemented small perl script to expire snapshots older than 3 days (but keep single midnight snapshot for each day). I also decided to create cron job to defragment file-system every morning.

dpavlin@prod:~$ cat /etc/cron.d/btrfs-defrag

PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin

# m h   dom mon dow     user    command
15 7    * * *           root    btrfsctl -d /mnt/koha/rootfs/

This did help a bit, but not good enough. I have three basic services on this box: Apache web server running perl cgi scripts, MySQL database and Zebra indexer. So, in a final step of despair (before going back to ext4) I decided to move Zebra to RAID1 volume which is on two separate disks. And it did make a huge change.

So, is btrfs ready for usage in production? That depends on your IO load. If you have to more-or-less random IO workloads (like RDBMS and full-text indexer in my example) spreading it over multiple disks will provide better performance than any choice of file-system. But, if snapshots are something useful for your use-case, give btrfs a try. Also have in mind that recovery tools for btrfs are... non-existent. So make backups (which is good idea anyway) and remember that btrfs snapshots on same disks don't count as backup.

I am huge fan of Exhibit faceted browsing of data. However, Exhibit is implemented in JavaScript within your browser and that makes it unusable for larger amounts of data (more than 300 or so). In my case, 3800 elements is unusably slow even in latest Chrome or Firefox.

Something had to be done. If JavaScript inside browser isn't up to the task, you might wonder what would happen if you moved processing back to server side, and use browser just for task which they are good at: displaying generated HTML pages.

So, let me introduce MojoFacets - server-side facet browser based on Mojolicious with a bit of jQuery UI.

Although it seems strange to promote server-side solutions in 2010, this approach still makes sense. For just 40Mb of memory usage server side (including Mojo and dataset) you can get fast and usable facet browsing.

I had to make 10 sequences in our Koha installation which uses MySQL. To my horror MySQL doesn't have CREATE SEQUENCE (insert MySQL vs PostgreSQL rant here). Even worse, MySQL manual suggest usage of LAST_INSERT_ID which would mean that I had to create 10 tables just to track my sequences.

There is better solution (based on StackOverflow question: Emulating a transaction-safe SEQUENCE in MySQL) outlined below:

create table ffzg_zs_seq (
        name varchar(2) unique not null,
        current integer unsigned not null
);

insert into ffzg_zs_seq values ('PA',100000);
insert into ffzg_zs_seq values ('PB',100000);
insert into ffzg_zs_seq values ('PC',100000);
insert into ffzg_zs_seq values ('PD',100000);
insert into ffzg_zs_seq values ('PE',100000);
insert into ffzg_zs_seq values ('DD',100000);
insert into ffzg_zs_seq values ('MR',100000);
insert into ffzg_zs_seq values ('DR',100000);
insert into ffzg_zs_seq values ('FO',100000);
insert into ffzg_zs_seq values ('SE',100000);

delimiter |
         
create function ffzg_zs_nextval( seq_name varchar(2) )
returns integer unsigned
begin        
        update ffzg_zs_seq set current = ( @next_val := current + 1 ) where name = seq_name ;
        return @next_val;
end|
    
delimiter ;

update ffzg_zs_seq
set current=(
        select  
                max(substring_index(itemcallnumber,' ',-1))
        from items
        where substring_index(itemcallnumber,' ',1) = ffzg_zs_seq.name
);

Usage is simple, allmost as good as built-in sequences:

mysql> select ffzg_zs_nextval('DD');
+-----------------------+
| ffzg_zs_nextval('DD') |
+-----------------------+
|                100178 | 
+-----------------------+
1 row in set (0.00 sec)