Pasky’s Log

One commercial project I was working on for Raspberry Pi involved playing various MP3 samples when a button is pushed. The original implementation used mplayer to play back the samples, however the issue is that there was up to 1500ms latency between mplayer was executed and start of playback.

I didn’t do detailed profiling, but I think two factors causing high latency of mplayer were that (i) just loading all the .so libraries mplayer depends on can take many hundreds of milliseconds (ii) the file is being scanned for whatever stuff, streams detected etc. and that can also take some extra time; perhaps I could force mplayer to realize this is a simple MP3 file, but (i) is still the much bigger factor.

I wanted to avoid recoding all the samples to wav. That would allow me to use aplay directly and the playback starts immediately, but it would also feel really silly; decoding of MP3 is not the bottleneck, just the latency of mammoth software loading and initializing itself is. I also didn’t try mpd as that might have been a bit painful to set up.

Another point worth noting is that I didn’t use the crappy on-board PWM audio but a $3 chinese USB soundcard (which is still much better than PWM audio). And using reasonably up-to-date Raspbian Wheezy. So I tried…

• mplayer -slave -idle, started in parallel with my program and receiving commands via FIFO. It hangs after the first file (even though it works fine when ran without -slave).
• cmus running in parallel with my program, controlled by cmus-remote. Convincing it to use ALSA device of my choice was really hard, but eventually I managed, only to hear my files sped up about 20x.
• madplay I couldn’t convince about using a non-default ALSA device at all.
• mpg123 started immediately and could play back the MP3 files on a non-default ALSA device. Somehow, the quality was very low though (telephone grade) and there was an intense high-pitched clip at the end of the playback.
• mpg321 I couldn’t convince to produce any sound and anyway it had about 800ms latency before playback started, probably due to its libao dependency.
• sox, or rather AUDIODEV=hw:1 play worked! (After installing a package with MP3 support for sox.) No latency, normal quality, no clips, no hangs. Whew.

Verdict: There still is a software on Linux that can properly and quickly play MP3 files on Raspberry Pi, though it was a challenge to find it. I didn’t think of sox at first and I was almost giving up hope. BTW, normally you would use sox and play for applying a variety of audio transformations and effects in a batch/pipeline fashion and it can do a lot of awesome magic.

I promised to post some minutes from the BOF in $SUBJ here for people who don’t remember all the tool names:

• [l]imit in mutt is very powerful functionality; my other blogpost describes notmuch integration to that
• new mutt-kz has good virtual folder support as notmuch integration; perhaps future of state-of-art text mail clients
• sup is interesting gmail-like text client, but way too slow!
• alot is worth a look as nice notmuch frontend; no screenshots on net though
• notmuch can filter by folder label, so single db is fine for all your folders
• dovecot sync (dsync)
• lookg at images when reading mail remotely – screenenv (set $DISPLAY based on last active screen client), new tool needed for seamless transfer of files back to local machine is needed!
• maildir sync using VCS (bazaar) instead of imap (read mails using “thick client”) (ccxcz)
• prioritization of downloaded mails by using uucp for transfer (lmw)
• sending mails by feeding them to procmail which decides how to send them (ccxcz)
• automatic addressbook building: lbdb (little brother db)
• Trojita is Qt MUA with very fast IMAP.
• another mutt tip: set edit_headers will make mutt not ask about recipient, subject etc. before starting editor, but let you put the headers in instead

For about 13 years now, I’m running the Muaddib IRC bot that serves a range of Czech channels. Its features varied historically, but the main one is providing conversational AI services (it learns from people talking to him and replies back based on the learnt stuff). It runs the Megahal Markov chain algorithm, using the Hailo implementation right now.

Sometimes, I need to reset its brain. Most commonly when the server happens to hit a disk full situation, something no Megahal implementation seems to be able to deal with gracefully. :-) (Hailo is SQLite-based.) Thankfully, it’s a simple sed job with all the IRC logs archived. However, Muaddib always had trouble with non-ASCII data, mixing a variety of encodings and liking to produce a gibberish result.

So, historically, people used to talk to Muaddib using ISO-8859-2 and UTF8 encodings and now I had mixed ISO-8859-2/UTF8 lines and I wanted to convert them all to UTF8. Curiously, I have not been able to quickly Google out a solution and had to hack together my own (and, well, dealing with Unicod ein Perl is never something that goes quickly). For the benefit of fellow Google wanderers, here is my take:

perl -MEncode -ple 'BEGIN { binmode STDOUT, ":utf8"; }
  $_ = decode("UTF-8", $_, sub { decode("iso-8859-2", chr(shift)) });'

It relies on the Encode::decode() ability to specify a custom conversion failure handler (and the fact that Latin2 character sequences that are also valid UTF-8 sequences are fairly rare). Note that Encode 2.35 (found in Debian squeeze) is broken and while it documents this feature, it doesn’t work. Encode 2.42_01 in Debian wheezy or latest CPAN version (use perl -MCPAN -e 'install Encode' to upgrade) works fine.

On Wednesday, I have bought myself an Android phone, as my good old S-E C510 suffered from worse and worse charging problems. I have found that I find it pretty much impossible to type on a touchscreen and did not see any improvement even after light practice (on a spare second hand Android phone I acquired just for its sensor – sometime in the future, maybe it will drive a quadcopter). So, I went for Sony-Ericsson Xperia Pro (codename iyukan) with its hardware keyboard. It’s a pretty neat phone, my only complaint is a difficult-to-press power button.

However, just after turning it on for the first time, the phone prompted me to upgrade it from Android 2.3 to Android 4. The fool I was, thinking that newer is better and wanting to summarily get rid of all the preloadware apps… And since a friend told me that CyanogenMod works all right on this phone, I would need a Windows PC to upgrade to Android 4 the Sony Ericsson way, I like to have full control over the systems I use and I like CM’s tray design ;-), I went for it.

First, some tips and tricks for fellow Googlers that come by this post in a need to get CyanogenMod working on their Xperia Pro:

• Do not expect CyanogenMod wiki to be a place to document even critical issues, learn about them and solving them. Your only shot is hitting the issue blindly and then followup wild googling and IRC. More on that below.
• Ignore stock CyanogenMod. What you want is using CyanogenMod fork FreeXperia (FXP) which contains CM tuned for Xperia phones, with both custom kernel and set of drivers and applications. Follow the regular CyanogenMod flashing howto, just use .zip files provided by FreeXperia. The latest CM9.1 Xperia version FXP136 worked quite well for me, aside of wifi troubles (more on that below), camera autofocus on touching the cmaera button and maybe some compass weirdness (I didn’t verify that yet, but there are workarounds in the tracker in case it proves to be a real issue).
• If you insist on stock CyanogenMod 9.0.0-rc2, replace the boot.img you will be flashing (kernel image) with the one from FXP136, or your phone will essentially refuse to start up with applications like the Setup Wizard crashing and if you manage to get past that, the phone being quite sluggish.
• New FXP WiFi drivers for Xperia Pro (wl12xx, specifically wl1271) have support for some extended powersaving features that depend on RX streaming. On some APs, that means the device will receive packets only up to 100ms after it transmits packets itself – any packets coming after that will be lost, which means that communication with sites that take a little to process your requests (e.g. the Market) or using any kind of streaming breaks.
  I have spent the whole last night fiddling with wifi and binary patching wl12xx.ko to tweak the parameters, but I just didn’t manage to get it working with my Wifi AP. However, over the night I have tilted to thinking that this is slightly more likely bug in powersaving support of my AP rather than in the wifi firmware, which is simply using more aggressive powersaving modes now than other Android phones visiting my home wifi network before and other devices like notebooks. I have pretty much given up on debugging this now and will just buy a new AP, since with all other APs I have came by so far the phone works fine (but there are scattered reports about this problem on the net).
• My phone refuses to properly authenticate with my AP (always stuck in the “Obtaining IP address stage”, but in fact it never comes to DHCP, instead it fails right after authentication), wpa-supplicant logs WPA: EAPOL-Key Replay Counter did not increase - dropping packet and that’s it. After I restart my AP, the authentication succeeds… once; if I disconnect, I won’t connect again anymore. Again, this happens just with my AP, so maybe there is some connection to the previous problem, perhaps some authentication packets being dropped… This happens with WEP, WPA-PSK TKIP or AES, … The only workaround I have found is to restart the AP.
• Before, my phone would get stuck in a different way, believing that its rightful IP address is 169.254.222.something and never asking using DHCP for an actual IP address. The solution to that problem is to open a terminal, su, and rm /data/misc/dhcp/*.leases. Also, don’t panic if you are to connect to eduroam; even though the WiFi authentication dialog will show phase 2 to be “None”, that does not mean wpa_supplicant on the phone is not internally using MSCHAP. :-)

So, in the end, the phone has eaten much of my last three days, and it was not spent installing and fiddling with neat apps but debugging some frustrating issues. I hope it will serve me better from now on… :-)

But this has been also an interesting lesson in dysfunctioning open source projects – yes, I mean CyanogenMod and FreeXperia. First of all.

The problem is that the projects are very unfriendly to their audience. Sure, CyanogenMod has a pretty front website and after some very non-straightforward you may even reach a straightforward HOWTO for your phone that you may follow to do the installation, but the project becomes unfriendly once you need to do some powerusery things with your phone or even start taking look at the source and doing some development. First, I should take a note that some of the issues are probably FreeXperia specific. Let’s take a look at some of the problems:

• Bad overview documentation. I found no way to actually learn on my own about FreeXperia and its relationship to CyanogenMod (which is still not completely clear to me). Even long after first hints to “use FXP136″ or whatever, I was clueless about what the “FXP” actually meant.
• Bad release documentation. On Xperia Pro, the latest CyanogenMod official is 9.0.0-RC2. There appears to be absolutely no way to learn about what kind of state is it in – what blocker bugs are there to keep this at RC2? Is it worth waiting for 9.0.0? It appears to be all just in the minds of the maintainers so the only way to decide which version of CM to pick is to waste time trying to install it. Also, FreeXperia homepage caries essentially no documentation either, not even linking a fairly essential companion forum thread.
• Bad detailed documentation. It appears that the only way to learn about issues and try to solve them is either asking on the forum and navigating its unwieldy paginated threads, or asking on IRC and hoping someone knowledgeable is by accident following the channel at that moment. There is a Wiki but most attempts to document issues and help out fellow users or simply correct factual errors appear to be reverted without explanation.
• Bad development documentation. Xperia Pro is actually a huge exception here since there is an actual HOWTO on compiling CyanogenMod for it using the arcane build system. However, trying to navigate the masses of github repositories of both CyanogenMod and FreeXperia and understanding how they relate, in which repository and in which branch can I actually find the kernel I’m running and where does my wl21xx module come from has taken me several hours anyway. While FreeXperia is supposed to be an “open source” project, there is actualy no word on its homepage about where to get the sources and how are they built; you are on your own in the GitHub maze (and no, there is no link to GitHub’s FreeXperia account on its homepage either). I’d say this is on the verge of violating GPL, though probably not quite behind the line yet…
• Less than ideal developer attitude. The people at IRC are mostly very helpful and I thank them again for all their help. But I have been rather discouraged by my wiki experience and why should I even bother reporting bugs?

I complained a bit about some of these issues in the past few days. A fellow IRC user asked “would you rather developers spend their time on documentation than fixing bugs?”. I think resounding “YES” is in order. Most basic documentation (what is what) does not take long to write and goes a long way. Also, putting effort to fixing bugs is usually no excuse for bad attitude to users.

It seems to me that to be a happy CyanogenMod user, you either do not actually put much effort in poking the system and you are lucky to have a most mainstream device with all the major issues ironed out, or you go all the way to become a core developer and learn about all the details. If you are stuck somewhere in-between, willing to get to the bones of the system to solve your problem but just wanting to solve your problem, CyanogenMod/FreeXperia gives you no choice but to spend days learning about all the ways things work and are getting done.

Given that, it is actually surprising to me that it still works as well as it does. It is an interesting case study in open source dynamics. I think it will be interesting to see whether FreeXperia can survive for long time as the original developers, who don’t work in a much open environment, wear out and enthusiasm of newcoming fresh developers will be required… Let’s watch and learn!

Few years ago, as a school project I have written hed. It’s yet another terminal hexadecimal editor, but with few unique features.

Thanks to its splay tree file representation, it is able to very efficiently handle editing and even inserting to huge files; the file is not loaded in memory as a whole, just the modified parts are saved, and therefore you are able to edit even files many gigabytes in size efficiently. You can also save just the swap file separately as a “working diff” and restore your changes later on top of unmodified original file.

It uses vi-like keybindings (including marks and yank/paste registers or :!). It also features an “expression” concept that lets you efficiently compose search, substitute or jump expressions composed from a variety of data representations, supporting arithmetic operators and register references. E.g. using special register “. (data under cursor), you can use command #”. to jump to file offset written under cursor.

I’m writing about it again now since I just pushed out Debian packaging for the editor, so you can easily make Debian or Ubuntu packages for yourself from the source (it also has existing OpenSUSE packaging). Try it out! I’m not maintaining the project anymore, but Petr Tesarik will gladly accept any patches or feedback (or I will too, forwarding it to him :-).

I love Perl and it’s my language of choice for much of the software I write (between shell at one extreme and C at the other). However, there is one thing Perl really sucks at – Unicode and UTF8 encoding support. It is not that the features aren’t there, but that getting it to work is so tricky. It is so much tricks to remember already that I started writing them down:

http://brmlab.cz/user/pasky/perl-utf8

It’s a wiki, anyone is welcome to contribute. :-)

So, thanks to very helpful Rickta59 on #43oh IRC channel, I got my Launchpad v1.5 serial communication working. The key piece of information I was missing:

This is even drawn on the board, but it just didn’t occur to me that I need to do this simple thing. Most examples seem to use hardware UART, and Energia Serial class also uses hardware UART.

It is still very flaky:

• For the first ten seconds, communication is impossible. Wait for timeout messages to appear in dmesg, then you can start communication.
• When the board is sending data, something must be reading them on the host side. If not, the driver collapses and you need to replug the device.
• The latter might be circumvented by direct USB communication without involving the tty driver.

So, it is rather fragile, but usable! Let’s enjoy our Launchpads for projects where this is not a big issue…

I found out that the situation with MSP430 is not as bad as it seemed. This post is mostly obsolete, but I’m leaving the text up for the benefit of Google index and other desperate people struggling with their Launchpad. :-)

This blogpost serves as a big fat warning to the future ones that might be about to follow in my footsteps:

It’s a sad reality but that’s just how it is, to the best of my knowledge, and after a lot of research and doing unbelievable things to kernel drivers etc. To clarify a bit, basic programming using mspdebug works, but you cannot communicate between host and board using USB serial. This seems to have worked with much older USB chips but not with the ones used by TI in current versions of the board (I got Launchpad with MSP-EXP430G2 ordered in May 2012).

Some fun technical details to help google index and guide others diagnosing this:

[186808.775510] usb 1-1.2: new full-speed USB device number 7 using ehci_hcd
[186808.891778] usb 1-1.2: New USB device found, idVendor=0451, idProduct=f432
[186808.891788] usb 1-1.2: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[186808.891794] usb 1-1.2: Product: Texas Instruments MSP-FET430UIF
[186808.891800] usb 1-1.2: Manufacturer: Texas Instruments
[186808.891804] usb 1-1.2: SerialNumber: CFFF4695F6C11445
[186808.924900] cdc_acm 1-1.2:1.0: This device cannot do calls on its own. It is not a modem.
[186808.924914] cdc_acm 1-1.2:1.0: No union descriptor, testing for castrated device
[186808.925029] cdc_acm 1-1.2:1.0: ttyACM0: USB ACM device
[186808.927595] usbcore: registered new interface driver cdc_acm
[186808.927603] cdc_acm: USB Abstract Control Model driver for USB modems and ISDN adapters
[186818.963279] generic-usb 0003:0451:F432.0001: usb_submit_urb(ctrl) failed
[186818.963332] generic-usb 0003:0451:F432.0001: timeout initializing reports
[186818.964177] generic-usb 0003:0451:F432.0001: hiddev0,hidraw0: USB HID v1.01 Device [Texas Instruments Texas Instruments MSP-FET430UIF] on usb-0000
:00:1a.0-1.2/input1
[186818.964262] usbcore: registered new interface driver usbhid
[186818.964269] usbhid: USB HID core driver

This is what my dmesg says the first time the board is plugged in. mspdebug works fine but any attempt of serial communication over /dev/ttyACM0 (talking to TI-provided sample UART code). OBTW if you are actually wondering how to compile and upload stuff on this baby:

msp430-gcc -mmcu=msp430g2553 -Wall -O3 -o uart_01_9600 msp430g2xx3_uscia0_uart_01_9600.c
mspdebug rf2500 prog\ uart_01_9600

For USB interface, TI includes its own crazy USB-enabled microcontroller on board that provides a HID-ish interface (for mspdebug) and an ACM-ish interface (for UART emulation) on a single port (which is nicely confusing). The serial part is supposed to be handled by ti_usb_3410_5052 kernel driver, which grabs a firmware and attempts to reflash the USB microcontroller so that it presents a more sensible serial USB interface (pretty crazy, eh?). However, the rf2500 variant of this chip appears to be too new and simply not supported either by the firmware or the firmware uploader.

Tweaking USB ids in the driver (f430 -> f432) does not help. Getting ti_3410.fw that Debian helpfully does not ship does not help. Manually binding the driver to USB does not help. The furthest I get is that the driver indeed tries to flash the ti_3410.fw firmware to device, but just times out doing that (I think maybe I bricked the serial part of the USB microcontroller by now):

[193053.430662] ti_usb_3410_5052 1-1.2:1.0: TI USB 3410 1 port adapter converter detected
[193054.443490] usb 1-1.2: ti_download_firmware - error downloading firmware, -110
[193054.443528] ti_usb_3410_5052: probe of 1-1.2:1.0 failed with error -5

Oh, and mspdebug rf2400 exit before any serial communication (I have found a tip somewhere) does not help either. An obviously-working UART code for MSP430G2553 would be welcome too, to triple-rule-out a uC-side firmware problem. (The launchpad board is awesome but rx/tx leds are sorely missing. I know, I could grab an oscilloscope… but how many hours have I already wasted by this?)

So, what seemed to be a great Arduino replacement turns to dust for me since the whole point of 80% of my Arduino projects is to talk to a computer… That said, if (after) you make it work, you will get one, or maybe even two Launchpads for free from me.

At our university department, many people still haven’t migrated to UTF8 and are still happily using ISO-8859-2 – mainly due to the amount of legacy text (TeX, …) documents.
Nowadays, support for non-UTF8 is slowly waning though, and CUPS is a prime example. Most of (shabby anyway) support for non-UTF8 encodings have been removed few years ago. It is still possible to force CUPS to print text files in non-UTF8 encoding if you extract the appropriate files from ancient version (1.2 or some-such) of CUPS to /usr/share/cups/charset/ and print using e.g. lpr -o document-format='text/plain;charset=iso-8859-2'. However, there is simply no support for lpr automatically setting the charset based on your locale.

We decided that the best way to go is to simply auto-detect the encoding using the awesome enca package and convert text files from this encoding to UTF8. This should be actually fairly fool-proof in practice, unless you are dealing with an extremely mixed set of languages. Making own CUPS filter is easy – just change texttops entries in /etc/cups/mime.conv to textautoencps and create a new /usr/lib/cups/filter/textautoencps file:

#!/bin/bash
 
if [ $# == 0 ]; then
  echo >&2 "ERROR: $0 job-id user title copies options [file]"
  exit 1
fi
 
{ if [ $# -ge 6 ]; then
    cat $6
  else
    cat
  fi; } |
    enconv -x utf-8 -L czech |
    /usr/lib/cups/filter/texttops "${@:0:6}"

At our university department, people sometimes need to run expensive or long-term computations. We have few servers reserved for computations, but frequently it is useful to run computations on machines in the offices since some of them are fairly powerful and mostly get only very light use CPU-wise.

However, such computations must never impair any interactive or more pressing use of the machine. Therefore, we want to limit scheduling priority of the computations, limit total memory used by the computations and allow *anyone* kill *any* running computation. It turns out that this is not as trivial to achieve as I hoped.

In comes Computations under control: compctl – cgroup-based control of publicly limitable and stopable tasks. It is a tool that allows anyone to execute a command (or start screen) such that it is marked as a computation. Then, it allows anyone else to limit the total amount of memory allocated for all computations and to stop a specific computation or all computations on a machine. It uses cgroups to keep track of computations and limit the total memory usage, and a simple client-server architecture to perform priviledged tasks.

I hope it will be useful for someone else too. :-) Feel free to send in patches, and extra pairs of eyeballs checking the security would be welcome too. Top on my TODO list is simple debian package and a more verbose compctl –list output.