Archive

Archive for the ‘linux’ Category

SMTP from Exim-equipped roaming notebook (SSH smarthost)

February 13th, 2014 No comments

I don’t send email from my notebook often, dealing with my correspondence on my server machine via ssh. When I need to do it, it’s usually when I’m sending Git patches or something like that. I didn’t meet much trouble with sending it directly, but SMTP servers of Debian-involved people are some of the most picky one can meet and I decided it’ll be best if I switch the exim4 on my notebook to smarthost mode where all mail is relayed via my main server.

So that should be trivial to do, right? Wrong, apparently. I figured I’d use SMTP auth, but it just seems mind-bogglingly complicated to configure if you don’t want to spend an evening on it. The client part is fairly easy (probably both on exim4 and postfix), but setting up postfix server to do SMTP auth (for just a single person) is really silly stuff. Maybe not so crazy if you use PAM / shadow for authentication, but that means that on my notebook, I’d have to store (in plaintext) my server password anyone could use to log in – no way. It seems I could switch to Dovecot and somehow pass it a simple password to use, but at that point my patience ran out and I just backed off a litle.

Why not just use ssh for smarthost SMTP transport? Authentication via ssh is something everyone understands nowadays, it does the best job there, no silly passwords involved and you can just pipe SMTP through it. You wouldn’t do that at in a company setting with Windows notebooks, but for a single geek, it seems ideal.

Someone already did set up ssh as exim transport, but that’s for exim3. So here follows a super-quick HOWTO to do this with exim4:

  • Set up ssh key on client:
    sudo -u Debian-exim /bin/bash
    ssh-keygen # go with the default, and empty password, this will be used in an automated way
    ssh me@server.example.org # to fill up known_hosts; it will fail yet
    cat ~/.ssh/id_rsa.pub # this is my public key
    exit # ..the sudo
    
  • Set up ssh key on server – paste the public key printed by the cat above to ~me/.ssh/authorized_keys and prepend command="nc -w1 localhost smtp",no-agent-forwarding,no-port-forwarding,no-X11-forwarding to the key line. This key can now be used only for mail relaying.
  • Do dpkg-reconfigure exim4-config and configure smarthost mode. Also use it to find out whether you are using split or big configuration. You will also probably want to enable “mailname hiding”, otherwise your return-path will contain an unroutable address.
  • Set up ssh transport in exim4 – add the following to the config file:
    ssh_pipe:
      debug_print = "T: ssh_pipe for smarthost delivery"
      driver = pipe
      path = "/bin:/usr/bin:/usr/local/bin"
      command = "ssh me@server.example.org nc -w1 localhost smtp"
      use_bsmtp
      message_prefix = "HELO mynotebook.example.org\r\n"
      delivery_date_add
      envelope_to_add
    

    (it would be nicer if we used the actual smarthost configuration option value and our notebook’s hostname instead of hardcoded strings, I guess).

  • In the smarthost: section of the configuration file, replace transport = remote_smtp_smarthost with transport = ssh_pipe.
  • /etc/init.d/exim4 reload and voilá, sending mail from anywhere should work now!

I *wish* setting up roaming SMTP nodes would be way easier nowadays and I wouldn’t have to eventually spend about 90 minutes on this stuff…

Categories: linux Tags: , , ,

systemd: journal listing on /dev/tty12

February 12th, 2014 3 comments

Inspired by the Debian CTTE deliberations on the new default init for Debian, I installed systemd on my notebook after tonight’s forced reboot and played with it a little.

(And I like it! I was very sceptical when hearing about systemd first, but after reading a lot of discussions and trying it myself, I find most of the problematic points either fixed already or a load of FUD. The immediate big selling point for me is actually journald, it and its integration with systemctl is really awesome. I’ll actually find systemd more useful on servers than desktops, I think.)

While it’s a nice exercise for anyone wanting to get familiar with systemd, I still decided to share a tidbit – service file that will make log entries show up on /dev/tty12. Many people run with rsyslogd set up for this, you’ll want to disable that (by default, all journal entries are forwarded to rsyslog). The advantage of showing journal entries instead is mainly color coding. :)

The file listing follows, or get it here.

# Simple systemd service that will show journal contents on /dev/tty12
# by running journalctl -af on it.
# Install by:
#  - Saving this as /etc/systemd/system/journal@tty12.service
#  - Running systemctl enable journal@tty12
#  - Running systemctl start journal@tty12
# journald can also log on console itself, but current Debian version won't
# show timestamps and color-coding.
# systemd is under LGPL2.1 etc, this is inspired by getty@.service.

[Unit]
Description=Journal tail on %I
Documentation=man:journalctl(1)
After=systemd-user-sessions.service plymouth-quit-wait.service systemd-journald.service
After=rc-local.service

# On systems without virtual consoles, don't start any getty. (Note
# that serial gettys are covered by serial-getty@.service, not this
# unit
ConditionPathExists=/dev/tty0

[Service]
# the VT is cleared by TTYVTDisallocate
ExecStart=/bin/sh -c "exec /bin/journalctl -af > /dev/%I"
Type=idle
Restart=always
RestartSec=1
UtmpIdentifier=%I
TTYPath=/dev/%I
TTYReset=yes
TTYVHangup=yes
#TTYVTDisallocate=yes
TTYVTDisallocate=no
KillMode=process
IgnoreSIGPIPE=no

# Unset locale for the console getty since the console has problems
# displaying some internationalized messages.
Environment=LANG= LANGUAGE= LC_CTYPE= LC_NUMERIC= LC_TIME= LC_COLLATE= LC_MONETARY= LC_MESSAGES= LC_PAPER= LC_NAME= LC_ADDRESS= LC_TELEPHONE= LC_MEASUREMENT= LC_IDENTIFICATION=

[Install]
Alias=getty.target.wants/journal@tty12.service

(P.S.: Creating this service file – my very first one – took me 10 minutes total, including studying documentation and debugging two stupid mistakes I made.)

Categories: linux Tags: , ,

GPS souřadnice českých měst a obcí

February 1st, 2014 1 comment

Pro zobrazování poloh dopadů meteosond na IRC jsem potřeboval v jednoduchém CSV formátu seznam souřadnic českých měst, ale ukázalo se, že je překvapivě obtížné něco takového získat. Sice existuje tabulka na jednom astronomickém webu, výběr tam zahrnutých obcí je ale docela divný, někde je místo obce jen její část, atd.

Nakonec jsem zvolil postup “udělej si sám”, a to kombinací seznamu na Wikipedii, Google Geocoding API a trochy XPath.

Seznam rozumné podmnožiny měst mohu získat třeba pomocí:

curl 'http://cs.wikipedia.org/w/index.php?title=Seznam_obc%C3%AD_s_roz%C5%A1%C3%AD%C5%99enou_p%C5%AFsobnost%C3%AD&action=edit' |
  sed -ne 's/^# \[\[\([^]|]*|\)*\([^]]*\)\]\].*/\2/p' | sort

Mám-li zase jméno obce, její souřadnice mohu získat tímto zaklínadlem:

m=Aš; curl -s 'http://maps.googleapis.com/maps/api/geocode/xml?address='"${m// /+},+CZ"'&sensor=false' |
  xmllint --xpath '//location[lat or lng]//text()' -

(Důležitý trik je to ,CZ, jinak bude Google znát spoustu Kolínů a Aš bude znamenat Americká Samoa. Alternativně si můžete z výsledků vyfiltrovat ty české pomocí XPath //result[address_component/short_name/text()="CZ"]/geometry/location[lat or lng]//text().)

Teď už to pro vygenerování jednoduchého CSV stačí spojit dohromady:

curl 'http://cs.wikipedia.org/w/index.php?title=Seznam_obc%C3%AD_s_roz%C5%A1%C3%AD%C5%99enou_p%C5%AFsobnost%C3%AD&action=edit' |
  sed -ne 's/^# \[\[\([^]|]*|\)*\([^]]*\)\]\].*/\2/p' | sort |
  while read m; do
    echo -n $m
    curl -s 'http://maps.googleapis.com/maps/api/geocode/xml?address='"${m// /+},+CZ"'&sensor=false' |
      xmllint --xpath '//location[lat or lng]//text()' - |
      tr -s '\n' ' ' | tr ' ' ','
    echo
    sleep 0.1
  done | sed 's/,$//'

Rádi byste hotové CSV?

Bonus: Podobně vygenerované CSV s pražskými částmi (katastrálními územími).

Categories: linux Tags: , , , , ,

Mice TV!

January 20th, 2014 No comments

Chido has two mouse-pets (Acomys Caihirinus, actually) and we finally did what we already planned to do long ago:

vlc http://pasky.or.cz:8090/mouses.flv

A live video stream of our mouse palace!


Some technical trivia – the IP cam used is Edimax IC-3110 (it’s pretty crappy, not recommended) and we are restreaming using vlc invocation:

cvlc -L --sout "#transcode{vcodec=mp4v,vb=1024,scale=1}:duplicate{dst=http{mux=ts,dst=:8090/mouses.flv},select=noaudio}" --no-sout-rtp-sap --no-sout-standard-sap --sout-ts-shaping=1000 --sout-ts-use-key-frames --ttl=40 rtsp://admin:PASSWORD@192.168.6.X:554/ipcam.sdp

(I did not get h264 FLV stream working reliably, unfortunately. I tried #transcode{vcodec=h264,venc=x264{keyint=20},vb=4096,scale=1} and duplicate{dst=http{mux=ffmpeg{mux=flv},dst=...,select=noaudio}.)

Categories: linux Tags: , , ,

Playing MP3 on Raspberry Pi with low latency

May 11th, 2013 1 comment

One commercial project I was working on for Raspberry Pi involved playing various MP3 samples when a button is pushed. The original implementation used mplayer to play back the samples, however the issue is that there was up to 1500ms latency between mplayer was executed and start of playback.

I didn’t do detailed profiling, but I think two factors causing high latency of mplayer were that (i) just loading all the .so libraries mplayer depends on can take many hundreds of milliseconds (ii) the file is being scanned for whatever stuff, streams detected etc. and that can also take some extra time; perhaps I could force mplayer to realize this is a simple MP3 file, but (i) is still the much bigger factor.

I wanted to avoid recoding all the samples to wav. That would allow me to use aplay directly and the playback starts immediately, but it would also feel really silly; decoding of MP3 is not the bottleneck, just the latency of mammoth software loading and initializing itself is. I also didn’t try mpd as that might have been a bit painful to set up.

Another point worth noting is that I didn’t use the crappy on-board PWM audio but a $3 chinese USB soundcard (which is still much better than PWM audio). And using reasonably up-to-date Raspbian Wheezy. So I tried…

  • mplayer -slave -idle, started in parallel with my program and receiving commands via FIFO. It hangs after the first file (even though it works fine when ran without -slave).
  • cmus running in parallel with my program, controlled by cmus-remote. Convincing it to use ALSA device of my choice was really hard, but eventually I managed, only to hear my files sped up about 20x.
  • madplay I couldn’t convince about using a non-default ALSA device at all.
  • mpg123 started immediately and could play back the MP3 files on a non-default ALSA device. Somehow, the quality was very low though (telephone grade) and there was an intense high-pitched clip at the end of the playback.
  • mpg321 I couldn’t convince to produce any sound and anyway it had about 800ms latency before playback started, probably due to its libao dependency.
  • sox, or rather AUDIODEV=hw:1 play worked! (After installing a package with MP3 support for sox.) No latency, normal quality, no clips, no hangs. Whew.

Verdict: There still is a software on Linux that can properly and quickly play MP3 files on Raspberry Pi, though it was a challenge to find it. I didn’t think of sox at first and I was almost giving up hope. BTW, normally you would use sox and play for applying a variety of audio transformations and effects in a batch/pipeline fashion and it can do a lot of awesome magic.

Categories: linux, software Tags: , , , , , ,

Short minutes from “Text Mail Clients” BOF @ LinuxDays Prague 2012

October 21st, 2012 No comments

I promised to post some minutes from the BOF in $SUBJ here for people who don’t remember all the tool names:

  • [l]imit in mutt is very powerful functionality; my other blogpost describes notmuch integration to that
  • new mutt-kz has good virtual folder support as notmuch integration; perhaps future of state-of-art text mail clients
  • sup is interesting gmail-like text client, but way too slow!
  • alot is worth a look as nice notmuch frontend; no screenshots on net though
  • notmuch can filter by folder label, so single db is fine for all your folders
  • dovecot sync (dsync)
  • lookg at images when reading mail remotely – screenenv (set $DISPLAY based on last active screen client), new tool needed for seamless transfer of files back to local machine is needed!
  • maildir sync using VCS (bazaar) instead of imap (read mails using “thick client”) (ccxcz)
  • prioritization of downloaded mails by using uucp for transfer (lmw)
  • sending mails by feeding them to procmail which decides how to send them (ccxcz)
  • automatic addressbook building: lbdb (little brother db)
  • Trojita is Qt MUA with very fast IMAP.
  • another mutt tip: set edit_headers will make mutt not ask about recipient, subject etc. before starting editor, but let you put the headers in instead
Categories: linux Tags: , , , ,

Conversion from mixed UTF8 / legacy encoding data to UTF8

September 23rd, 2012 No comments

For about 13 years now, I’m running the Muaddib IRC bot that serves a range of Czech channels. Its features varied historically, but the main one is providing conversational AI services (it learns from people talking to him and replies back based on the learnt stuff). It runs the Megahal Markov chain algorithm, using the Hailo implementation right now.

Sometimes, I need to reset its brain. Most commonly when the server happens to hit a disk full situation, something no Megahal implementation seems to be able to deal with gracefully. :-) (Hailo is SQLite-based.) Thankfully, it’s a simple sed job with all the IRC logs archived. However, Muaddib always had trouble with non-ASCII data, mixing a variety of encodings and liking to produce a gibberish result.

So, historically, people used to talk to Muaddib using ISO-8859-2 and UTF8 encodings and now I had mixed ISO-8859-2/UTF8 lines and I wanted to convert them all to UTF8. Curiously, I have not been able to quickly Google out a solution and had to hack together my own (and, well, dealing with Unicod ein Perl is never something that goes quickly). For the benefit of fellow Google wanderers, here is my take:

perl -MEncode -ple 'BEGIN { binmode STDOUT, ":utf8"; }
  $_ = decode("UTF-8", $_, sub { decode("iso-8859-2", chr(shift)) });'

It relies on the Encode::decode() ability to specify a custom conversion failure handler (and the fact that Latin2 character sequences that are also valid UTF-8 sequences are fairly rare). Note that Encode 2.35 (found in Debian squeeze) is broken and while it documents this feature, it doesn’t work. Encode 2.42_01 in Debian wheezy or latest CPAN version (use perl -MCPAN -e 'install Encode' to upgrade) works fine.

Perl and UTF8

June 24th, 2012 1 comment

I love Perl and it’s my language of choice for much of the software I write (between shell at one extreme and C at the other). However, there is one thing Perl really sucks at – Unicode and UTF8 encoding support. It is not that the features aren’t there, but that getting it to work is so tricky. It is so much tricks to remember already that I started writing them down:

http://brmlab.cz/user/pasky/perl-utf8

It’s a wiki, anyone is welcome to contribute. :-)

Categories: linux, software Tags: , , ,

Texas Instrument Launchpad MSP430 and Linux

June 11th, 2012 2 comments

I found out that the situation with MSP430 is not as bad as it seemed. This post is mostly obsolete, but I’m leaving the text up for the benefit of Google index and other desperate people struggling with their Launchpad. :-)

This blogpost serves as a big fat warning to the future ones that might be about to follow in my footsteps:

Currently sold TI Launchpad MSP430
is not properly supported by Linux
as of 2012-06-01

It’s a sad reality but that’s just how it is, to the best of my knowledge, and after a lot of research and doing unbelievable things to kernel drivers etc. To clarify a bit, basic programming using mspdebug works, but you cannot communicate between host and board using USB serial. This seems to have worked with much older USB chips but not with the ones used by TI in current versions of the board (I got Launchpad with MSP-EXP430G2 ordered in May 2012).


Some fun technical details to help google index and guide others diagnosing this:

[186808.775510] usb 1-1.2: new full-speed USB device number 7 using ehci_hcd
[186808.891778] usb 1-1.2: New USB device found, idVendor=0451, idProduct=f432
[186808.891788] usb 1-1.2: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[186808.891794] usb 1-1.2: Product: Texas Instruments MSP-FET430UIF
[186808.891800] usb 1-1.2: Manufacturer: Texas Instruments
[186808.891804] usb 1-1.2: SerialNumber: CFFF4695F6C11445
[186808.924900] cdc_acm 1-1.2:1.0: This device cannot do calls on its own. It is not a modem.
[186808.924914] cdc_acm 1-1.2:1.0: No union descriptor, testing for castrated device
[186808.925029] cdc_acm 1-1.2:1.0: ttyACM0: USB ACM device
[186808.927595] usbcore: registered new interface driver cdc_acm
[186808.927603] cdc_acm: USB Abstract Control Model driver for USB modems and ISDN adapters
[186818.963279] generic-usb 0003:0451:F432.0001: usb_submit_urb(ctrl) failed
[186818.963332] generic-usb 0003:0451:F432.0001: timeout initializing reports
[186818.964177] generic-usb 0003:0451:F432.0001: hiddev0,hidraw0: USB HID v1.01 Device [Texas Instruments Texas Instruments MSP-FET430UIF] on usb-0000
:00:1a.0-1.2/input1
[186818.964262] usbcore: registered new interface driver usbhid
[186818.964269] usbhid: USB HID core driver

This is what my dmesg says the first time the board is plugged in. mspdebug works fine but any attempt of serial communication over /dev/ttyACM0 (talking to TI-provided sample UART code). OBTW if you are actually wondering how to compile and upload stuff on this baby:

msp430-gcc -mmcu=msp430g2553 -Wall -O3 -o uart_01_9600 msp430g2xx3_uscia0_uart_01_9600.c
mspdebug rf2500 prog\ uart_01_9600

For USB interface, TI includes its own crazy USB-enabled microcontroller on board that provides a HID-ish interface (for mspdebug) and an ACM-ish interface (for UART emulation) on a single port (which is nicely confusing). The serial part is supposed to be handled by ti_usb_3410_5052 kernel driver, which grabs a firmware and attempts to reflash the USB microcontroller so that it presents a more sensible serial USB interface (pretty crazy, eh?). However, the rf2500 variant of this chip appears to be too new and simply not supported either by the firmware or the firmware uploader.

Tweaking USB ids in the driver (f430 -> f432) does not help. Getting ti_3410.fw that Debian helpfully does not ship does not help. Manually binding the driver to USB does not help. The furthest I get is that the driver indeed tries to flash the ti_3410.fw firmware to device, but just times out doing that (I think maybe I bricked the serial part of the USB microcontroller by now):

[193053.430662] ti_usb_3410_5052 1-1.2:1.0: TI USB 3410 1 port adapter converter detected
[193054.443490] usb 1-1.2: ti_download_firmware - error downloading firmware, -110
[193054.443528] ti_usb_3410_5052: probe of 1-1.2:1.0 failed with error -5

Oh, and mspdebug rf2400 exit before any serial communication (I have found a tip somewhere) does not help either. An obviously-working UART code for MSP430G2553 would be welcome too, to triple-rule-out a uC-side firmware problem. (The launchpad board is awesome but rx/tx leds are sorely missing. I know, I could grab an oscilloscope… but how many hours have I already wasted by this?)


So, what seemed to be a great Arduino replacement turns to dust for me since the whole point of 80% of my Arduino projects is to talk to a computer… That said, if (after) you make it work, you will get one, or maybe even two Launchpads for free from me.

Using CUPS to print text files in non-UTF8 charset encoding

May 17th, 2012 No comments

At our university department, many people still haven’t migrated to UTF8 and are still happily using ISO-8859-2 – mainly due to the amount of legacy text (TeX, …) documents.
Nowadays, support for non-UTF8 is slowly waning though, and CUPS is a prime example. Most of (shabby anyway) support for non-UTF8 encodings have been removed few years ago. It is still possible to force CUPS to print text files in non-UTF8 encoding if you extract the appropriate files from ancient version (1.2 or some-such) of CUPS to /usr/share/cups/charset/ and print using e.g. lpr -o document-format='text/plain;charset=iso-8859-2'. However, there is simply no support for lpr automatically setting the charset based on your locale.

We decided that the best way to go is to simply auto-detect the encoding using the awesome enca package and convert text files from this encoding to UTF8. This should be actually fairly fool-proof in practice, unless you are dealing with an extremely mixed set of languages. Making own CUPS filter is easy – just change texttops entries in /etc/cups/mime.conv to textautoencps and create a new /usr/lib/cups/filter/textautoencps file:

#!/bin/bash
 
if [ $# == 0 ]; then
  echo >&2 "ERROR: $0 job-id user title copies options [file]"
  exit 1
fi
 
{ if [ $# -ge 6 ]; then
    cat $6
  else
    cat
  fi; } |
    enconv -x utf-8 -L czech |
    /usr/lib/cups/filter/texttops "${@:0:6}"
Categories: linux, software Tags: , , ,