Dobrica Pavlinušić's random unstructured stuff
Dobrica Pavlinušić's random unstructured stuff: Revision 4

This is the home page for Dobrica Pavlinušić's random unstructured stuff.

  • Power cycle network switch using Arduino and relay

    Our top-of-switch rack decides to die randomly from time to time. It was somewhat inconvenient since it also killed most of our infrastructure including primary and secondary DNS so I needed a solution quickly. Since different rack is still on the network, I should be able to hack something and finally connect my Arduino knowledge and sysadmin realm, right? Think of it as power cycle watchdog based on network state.

    First thing was to figure out what was happening with the switch. It seemed like it was still working (LEDs did blink), but only thing that helped was power cycle. So as a first strep, I connected serial console (using RS-232 extension cable) to on-board serial port (since it doesn't seem to work using cheap CH340 based USB serial dongles) and I didn't expect this:

    0x37491a0 (bcmCNTR.0): memPartAlloc: block too big 6184 bytes (0x10 aligned) in partition 0x30f6d88
    0x3cd6bd0 (bcmCNTR.1): memPartAlloc: block too big 6184 bytes (0x10 aligned) in partition 0x30f6d88
    0x6024c00 (simPts_task): memPartAlloc: block too big 1576 bytes (0x10 aligned) in partition 0x30f6d88
    0x6024c00 (simPts_task): memPartAlloc: block too big 1576 bytes (0x10 aligned) in partition 0x30f6d88
    When I google messages like this I get two types of answers:
    1. beginner questions about VxWorks which summ up to: you have memory leak
    2. errors from switches with boardcomm chipset from various vendors
    There is basically no solution. We are running latest firmware, and internet doesn't have any idea what to do.
    Serial console did emit a lot of messages, but didn't respond to input at all. I would at last expect that watchdog timer in the switch will reset it once it manages to fragment it's own memory so much that it has stopped forwarding packets, oh well.... What else can I do?

    IEC power cable with relay

    I wanted something what I can plug in between the existing switch with IEC power connector with USB on the other end that can be plugged into any USB port for control.


    Since this is 220V project (and my first important one), I tried to do it as safe as possible.

    • I started with a power cable, that I cut in half and put ferrules on all wires to be sure that connectors will grip those wires well.
    • Then I replaced the power plug with IEC connector so it's can be inserted in any power cable. In this case, we soldered wires ends, since ferrules where too big to fit into connector housing. We did wrap a wire around a screw in a connector correctly, so tightening the screw will not displace the wire.
    • Finally I connected cheap 10A 250VAC relay which should be enough for fully loaded 48 port gigabit network switch that draws round 80W.
    • To make sure that rest of the system can't just power cycle device connected at any time, I connected live wire through normally closed pins on a relay. This means that this cable should work as-is (without powering it at all) and when powered, since board has pull-up resistor on the relay to VCC, the relay will be in the same sate, passing power to device.
    • Finally I checked all three power cable wires with multi-meter and got around 0.2 ohms which mans that whole thing works for now.
    At this point we should note that this relay board has only three pins (IN, GND and VCC) and has no optical isolation to 220V side. Since isolation would require us to provide additional power supply for 220V side, it was acceptable a risk.

    Putting it into a box

    I really wanted to somehow fix wires and protect the bottom of the relay board (which has 220V on it) from shorting to something, so I used an old box from a dairy product and created a housing for electronics.


    If you look carefully, you will notice that I had to cut the case all the way through to pass through the power cable (that has a zip-tie on inside to prevent it from pulling out). The Case will be fixed using hot glue and a lid, so this won't be a problem.
    Warning and label on the lid is also nice touch, and shouldn't be skipped when creating a thing which you won't be only user of.

    Arduino software

    You will also notice that relay is connected to A7, which didn't work out. Let me explain:
    The idea is to use Arduino default pin state (INPUT) as a state in which the pin will stay most of the time. This makes pin floating, and we can inspect pull-up on relay board and report if we see it. When we want to activate the relay, we'll flip pin to output, pull it down, and activate the relay.
    Code is available at and it can't be much simpler:

     * power cycle switch
     * relay is connected across 5V relay through normally closed pins so that failure of arduino doesn't kill power to switch.
     * to activate relay on this board, signal pin has to be pulled to ground.and coil draw is 76 mA when active
     * board has pull up on input pin to it's vcc
    #define RELAY_PIN 2
    void setup() {
      pinMode(LED_BUILTIN, OUTPUT);
      pinMode(RELAY_PIN, INPUT); // don't modify pin state
      Serial.print("Relay pin on reset: ");
    void loop() {
      if ( Serial.available() ) {
        char c =;
        if ( c == '0' ) {
          pinMode(RELAY_PIN, OUTPUT);
          digitalWrite(RELAY_PIN, LOW); // activate relay
          digitalWrite(LED_BUILTIN, HIGH); // led on
        } else if ( c == '1' ) {
          pinMode(RELAY_PIN, INPUT);
          digitalWrite(LED_BUILTIN, LOW); // led off
        } else {
    Simple is good: I toyed with idea of automatically releasing the relay from Arduino code, and when I started to implement timeout configuration on Arduino side, I remembered what this will be plugged into random server USB port, without avrdude and any handy way to update firmware on it, so I decided to just leave simplest possible commands:
    • 1 - ON (outputs H) - power on, default
    • 0 - OFF (outputs L) - power off, relay active

    Hot glue galore

    Then, I applied liberal amount of hot-glue to fix power cables and board in place. It worked out pretty well. You will also notice that the relay pin has moved to D2.




    And here it is, installed between existing switch power cable and switch, connected to only USB port still available in rack which is still on network.

    cron and serial port

    Idea is simple: we'll use cron to ping primary and secondary DNS IP addresses and if any of these fail, we'll send 0 to turn power off, wait 3 seconds, and send 1 to turn power back on.
    Implementation, however, is full of quirks, mostly because we don't want to depend on additional utilities installed, and we need to wait for Arduino to reset after connecting to serial port (and to give it time to display value of relay pin) before we start turning power off.

    #!/bin/sh -e
    ping -q -c 5 > /dev/shm/ping && ping -q -c 5 >> /dev/shm/ping || (
    test -e /dev/shm/reset && exit 0 # reset just once
    cp /dev/shm/ping /dev/shm/reset  # store failed ping
    date +%Y-%m-%dT%H:%M:%S
    cat /dev/shm/ping
    trap "exit" INT TERM
    trap "kill 0" EXIT
    stty -F $dev speed 115200 raw
    cat < $dev &
            sleep 3 # wait for reset and startup message
            echo 0  # off
            sleep 3
            echo 1  # on
            sleep 1
    ) | cat > $dev
    kill $!
    ) # ping subshell
    It's started from crontab with user which has dialout group membership so he can open /dev/ttyUSB0:
    dpavlin@ceph04:~$ ls -al /dev/ttyUSB0 
    crw-rw---- 1 root dialout 188, 0 Dec 28 01:44 /dev/ttyUSB0
    dpavlin@ceph04:~$ id
    uid=1001(dpavlin) gid=1001(dpavlin) groups=1001(dpavlin),20(dialout),27(sudo)
    dpavlin@ceph04:~$ crontab -l | tail -1
    */1 *  *   *   *     /home/dpavlin/
    This will execute script every minute. This allows us to detect error within minute. However, switch boot takes 50s, so we can't just run this script every minute, because it will result in constant switch power cycles. But since we are resetting switch just once this is not a problem.

    With this in place, your network switch will not force you to walk to it so you can power cycle it any more. :-)
    And it's interesting combination of sysadmin skills and electronics which might be helpful to someone.

    remote access

    If we want to access our servers while switch doesn't work, it's always useful to create few shell scripts on remote nodes which will capture IP addresses and commands which you will need to execute to recover your network.

    dpavlin@data:~/ffzg$ cat 
    #!/bin/sh -xe
    ssh microcom -p /dev/ttyUSB0 -s 115200
    dpavlin@data:~/ffzg$ cat 
    ssh microcom -p /dev/ttyS1 -s 9600

  • LSI MegaRAID slow compared to md

    This week I learned valuable lesson: if you are using MIPS from 2008 in your RAID controller, you can't really expect it to be faster than more modern Intel CPU when doing RAID 10 on disks.

    It all started with failure of SSD in our bcache setup which sits on top of MegaRAID RAID10 array. Since this required me take one of ganeti nodes down, it was also a good opportunity to add one more disk (we where running 6 disks and one SSD) and switch to software md RAID10 so we can use all 7 disks in RAID10. In this process, I did some benchmarking and was shocked with results.

    First, let's see original MegaRAID configuration:

    # lsblk --scsi -m
    sda  0:0:6:0    disk ATA      WDC WD1002FBYS-1 0C12      sda  931.5G root  disk  brw-rw----
    sdb  0:0:7:0    disk ATA      INTEL SSDSC2BW24 DC32      sdb  223.6G root  disk  brw-rw----
    sdc  0:2:0:0    disk DELL     PERC H310        2.12      sdc    2.7T root  disk  brw-rw----
    # hdparm -tT /dev/sdc
     Timing cached reads:   13920 MB in  1.99 seconds = 6981.76 MB/sec
     Timing buffered disk reads: 1356 MB in  3.00 seconds = 451.81 MB/sec
    and let's compare that with JBOD disks on controller and creating md array:
    # hdparm -Tt /dev/md0
     Timing cached reads:   13826 MB in  1.99 seconds = 6935.19 MB/sec
     Timing buffered disk reads: 1888 MB in  3.01 seconds = 628.05 MB/sec
    So, TL;DR is that you are throwing away disk performance if you are using hardware RAID. You didn't expect that? I didn't. In retrospect it's logical that newish Intel CPU can process data much faster than slowish MIPS on RAID controller, but on the other hand only difference is RAID overhead because same controller is still handling disks with software raid.

    I also wrote document with a lot of console output and commands to type if you want to do the same:

  • Edit any textarea in vi with syntax highlight

    Today I was explaining xclip utility and found this useful snippet which I wrote back in 2011 that allows you to edit browser textarea in terminal using vi with syntax highlighting.

    # workflow:
    # 1. focus textarea in browser you want to edit
    # 2. press ctrl+a then ctrl+c
    # 3. switch to terminal and start this script with optional extensioni for highlight: xclip-vi html
    # 4. edit file in vi, and save it
    # 5. switch back to browser, and press ctrl+v in already selected textarea
    test -z "$ext" && ext=txt
    xclip -out > /tmp/$$.$ext && vi /tmp/$$.$ext && xclip -in -selection clipboard < /tmp/$$.$ext
 github gist

  • Running legacy binary junk after upgrade in chroot

    These days, my work seems more like archeology than implementing newest and coolest web technologies, but someone has to keep an eye on old web servers which are useful to people.

    In this case, it's two major Debian releases upgrade which of course did result in some amount of breakage as you would expect. Most of breakage was easy to fix, but we had one site which had only pyc files, and python2.6 with correct modules wasn't supported on current distribution.

    First idea was to use python de-compiler to generate source python files, and this only got me to different python errors which I didn't know how to fix. so, this was dead end.

    I did have a backup of machine before upgrade on zfs pool, so next logical idea was to run minimal amount of old binaries to keep the site up. And I decided to do it in chroot so I can easily share mysql socket from current installation into pre-upgrade squeeze. Let's see what was involved in making this happen.

    1. make writable copy of filesystem

    Our backups are on zfs pool, so we cloned all three disks into same layout as they are mounted on filesystem:

    2018-06-08.10:07:53 zfs clone lib15/backup/ lib15/export/
    2018-06-08.10:08:19 zfs clone lib15/backup/ lib15/export/
    2018-06-08.10:08:38 zfs clone lib15/backup/ lib15/export/

    2. nfs export it to upgraded server

    Take a note of crossmnt parameter, this will allow us to mount all descending filesystems automatically.

    root@lib15:~# grep pauk /etc/exports 
    root@lib15:~# exportfs -va

    3. mount nfs export

    root@pauk:~# grep squeeze /etc/fstab  /mnt/squeeze    nfs     defaults,rw     0 0

    4. start chroot with old version of service

    To make this work, I first edited apache configuration in chroot to start it on different port, and configured front-end server to redirect site to new port.

    root@pauk:/home/dpavlin# cat 
    mount --bind /dev $to/dev
    mount --bind /sys $to/sys
    mount --bind /proc $to/proc
    mount --bind /var/run/mysqld/ /mnt/squeeze/var/run/mysqld/
    /usr/sbin/chroot $to /etc/init.d/apache2 start

    5. redirect apache traffic to server in chroot

    You will have to insert something like this in current apache configuration for vhost:

        Order allow,deny
        Allow from all
    <Location /german2/>
        RequestHeader set Host ""
        ProxyPreserveHost On

    You might scream at me that in days of containers, systemd and other better solutions chroot is obsolete. In my opinion, it's just a right tool for a job from time to time, if you have backup :-)

  • DORS/CLUC 2018: linux+sensor+device-tree+shell=IoT ?

    This year on DORS/CLUC 2018 I decided to talk about device tree in Linux kernel, so here are blurb, presentation and video of that lecture.

    You have one of those fruity *Pi arm boards and cheep sensor from China? Some buttons and LEDs? Do I really need to learn whole new scripting language and few web technologies to read my temperature, blink a led or toggle a relay?
    No, because your Linux kernel already has drivers for them and all you need is device tree and cat.

    Below is transcript of talk:
    Hello. How are you?
    This is the participation part, come on
    you're not the first I time here!
    OK, my name is Dobrica Pavlinušić and I will try to
    persuade you today that you can do with
    your Linux something which you might not
    have thought of by yourself I hope in a
    sense in last year and a half I noticed
    that I am using microcontrollers for
    less and less and that I'm using my
    Linux more and more for more or less the
    same tasks and in that progress process
    I actually learned something which I
    want to share with you today in a sense
    my idea is to tell you how to do
    something with your arm single board
    computer in this lecture we will talk
    mostly about Allwinner boards but if
    you want a hint if you want to buy
    some arm computer please buy the board
    which is supported by armbian. armbian  is the project which actually
    maintains the distribution for our
    boards and is currently the best
    distribution for arms aside from me
    raspbian for Raspberry Pi but raspbian
    supports only Raspberry Pi but if you have
    any other board please take a look if
    there is an armbian port. if there isn't
    try to contribute one and if you are
    just deciding which board to buy my
    suggestion is buy the one which is
    already supported on the other hand if
    you already did something similar you
    might have found some references on the
    internet about device three and it
    looked like magic
    so we'll try to dispel some of that
    magic today unfortunately when you start
    playing with it one of the first things
    you will want to do is recompile the
    kernel on your board so be prepared to
    compile additional drivers if they are
    not already included. armbian again
    wins because it comes with a lot of
    a lot of drivers already included and
    this year I will not say anything which
    requires soldering which might be good
    for you if you are afraid of the heat
    but it will be a little bit more than
    just connecting few wires not much more
    for a start let's start with the warning
    for example you have a arms in arms
    small arm board and you want to have a
    real clock in it you know the one which
    keeps the time when the board is powered
    off has the battery and so on if you buy
    the cheapest one from China which is
    basically for Arduino you will buy the
    device which is 5 volt device which your
    arm single board computer isn't. you can
    modify the board removing two resistors
    if you want to but don't tell anyone I2C, and will mostly talk about i2c
    sensors here, should be 5 volt tolerant
    so if you by mistake just connected and
    your data signals are really 5 volt you
    won't burn your board but if you are
    supplying your sensor with 5 volts
    please double-check that your that is
    sure to connect it to your board and
    nothing bad will happen this is the only
    warning I have for the whole lecture in
    this example I showed you a really
    simple way in which you can take the
    sensors run i2cdetect, detect its
    address in this case it's 68 - and then
    - load 1 kernel module and all of the
    sudden your Raspberry Pi will have battery backed clock, just like your laptop does
    but how did this journey all started for
    me? about two years ago I was very
    unsatisfied with the choice of pinouts
    which you can download from the internet
    I was thinking something along the lines
    wouldn't be it wouldn't it be nice if I
    print the pinout for any board I have
    with perfect 2.54 millimeters pin
    spacing which I can put beside my pins
    and never make a mistake of plugging the
    wire in the wrong pin and we all know
    that plugging the wire in the wrong pin
    is always the first problem you have on
    the other hand you say oh this is the
    great idea and you are looking at your
    pin out which is from the top of the
    board and you are plugging the wires
    from the bottom of the board and all of
    the sudden your pin out has to be
    flipped but once you write a script
    which actually displays the pin out it's
    trivially easy to get to add options to
    flip it horizontally or vertically and
    create either black and white pin out if
    you are printing it a laser or color pin
    out if you are printing it on some kind
    of inkjet so once you have that SVG
    which you can print and cut with the
    scissors and so on it's just a script on
    your machine so you could also have the
    common line output and then it went all
    south I started adding additional data
    which you can see on this slide in
    square brackets with the intention of
    having additional data for each pin for
    example if I started SPI I want to see
    that this pin is already used so I will
    not by mistake plug something into the
    SPI pins if I already have the SPI pin
    started if I have the serial port on
    different boards your serial might be
    UART4 on this particular CPU but it's
    the only serial in your Linux system so
    it will be /dev/ttyS0 for
    example so I wanted to see all the data
    and in the process I actually saw a lot
    of things which kernel know and I didn't
    so today talking to you about it of
    course in comment line because you might
    rotate your board well plug in the wires
    you can also do
    all the flips and things you you already
    saw in the in the graphic part so let's
    start with the sensor okay I said cheap
    sensor for eBay we'll get the cheap
    sensors from eBay but this sensor is from
    some old PowerPC Macintosh. It was
    attached to the disk drive and the
    Macintosh used it to measure the
    temperature of the disk drive. you know
    that was in the times before the smart
    had a temperature and I said hmm this
    is the old sensor, kernel surely
    doesn't have support for it, but, oh look,
    just grep through the kernel
    source and indeed there is a driver and
    this was a start I said hmm
    driver in kernel, I don't have to use
    Arduino for it - now that I know that
    driver is there and I have kernel module compiled, we said that prerequisite
    is that we can compile the kernel, what
    do we actually have to program or do to
    make this sensor alive? not more than
    this a echo in the middle of the slide
    you just echo the name of the module and
    the i2c address the i2c addresses we saw
    it before we can get it with i2cdetect by just connecting the sensor and
    the new device will magically appear it
    will be shown in the sensors if you have
    lm-sensors package installed but if
    you don't you can always find the same
    data in /sys/ file system which is full
    of wonders and as we'll see in non
    formatted way so the first two digits
    actually the last three digits digits
    are the decimal numbers and the all the
    other are the the integer celsius in
    this case. but you might say - I
    don't want to put that echo in my startup
    script! or I would like to have that as
    soon as possible I don't want to depend
    on the userland
    to actually start my sensor and believe
    it or not because your smart phones have
    various sensors in
    them, there is a solution in the Linux kernel
    for that and it's called the device tree
    so this is probably the simplest form of
    the device tree which still doesn't look
    scary, but it i will, stay with me, and it
    again defines our module the address
    which is 49 in this case and i2c 1
    interface just like we did in that echo
    but in this case this module will be
    activated as soon as kernel starts up
    as opposed to the end of your boot up
    process. one additional thing that kernel
    has and many people do not use is
    ability to load those device trees
    dynamically the reason why those most
    people don't use it is because they are
    on to old kernels I think you have to
    have something along the lines of 4.8
    4.8 or newer to actually have the
    ability to load the device trees live
    basically you are you are using /sys/kernel/config directory and this script
    just finds where you have it mounted and
    loads your device tree live word of
    warning currently although it seems like
    you can do that on Raspberry Pi the API
    is there the model is compiled,
    everything is nice and Diddley, kernel
    even says that device tree overlay is
    applied, it doesn't work on the raspberry
    pi because raspberry pi is different but
    if you gave a any other platform live
    loading is actually quite nice and
    diddley. so now we have some sensor and it
    works or it doesn't
    and we somewhat suspect that kernel
    developers didn't write a good driver
    which is never ever the case if you
    really want to implement some driver
    please look first at the kernel source
    tree there probably is the
    implementation better than the one you
    will write and you can use because the
    kernel is GPL you can use that
    implementation as a reference because in
    small experience with those drivers in
    kernel they are really really nice but
    what do you do well to debug it?
    I said no soldering and I didn't say but
    it would be nice if I could do that
    without additional hardware. Oh, look
    kernel has ability to debug my i2c
    devices and it's actually using tracing.
    the same thing I'm using on my servers
    to get performance counters. isn't that
    nice. I don't have to have a logic
    analyzer. I can just start tracing and
    have do all the dumps in kernel. nice,
    unexpected, but nice! so let's get to the
    first cheap board from China so you
    bought your arm single base computer
    single board computer and you want to
    add few analog digital converters to it
    because you are used to Arduino and you
    have some analog sensor or something and
    you found the cheapest one on eBay and
    bought few four five six because they
    are just the $ each, so what do you do
    the same thing we saw earlier you just
    compile the models say the address of
    the interface and it will appear and
    just like it did in the last example so
    everything is nice but but but you read
    the datasheet of that sensor and the
    sensor other than 4 analog inputs also
    has 1 analog output which is the top
    pin on the left denoted by AOUT, so you
    want to use it
    oh this kernel model doesn't is not very
    good it doesn't have ability to control
    that of course it does but how do you
    find it?
    my suggestion is actually to search
    through the /sys/ for either address of
    your i2c sensor, which is this
    case is 48, or for the word output or
    input and you will actually get all
    files, because in linux everything is a
    file, which are defined in driver of this
    module, and if you
    look at it, there is actually out0_output
    out0_output file in which you can
    turn output on or off so we are all
    golden. kernel developers didn't forget
    to implement part of the driver for this
    sensor all golden my original idea was
    to measure current consumptions of arm
    boards because I'm annoyed by the random
    problems you can have just because your
    power supply is not powerful enough so
    it's nice actually to monitor your power
    usage so you will see what changes, for
    example, you surely... we'll get that, remind
    me to tell you how much power does the
     the additional button take
    that's actually interesting thing which
    you wouldn't know if you don't measure
    current so you buy the cheapest possible
    eBay sensor for current, the right one
    is the ina219 which is
    bi-directional current sensing so you
    can put it between your battery and solar panel and you will see
    whether the battery is charging or
    discharging, or if you need more channels
    I like an ina3221 which has 3 channels,
    the same voltage
    but 3 different channels, so you can
    power 3 arm single computers from one
    sensor if you want to. and of course once
    you have that again the current is in
    some file and it will be someting...
    But, I promised you IOT? right? nothing
    I said so far is IOT! where is the
    where are the things? buzzwords are missing! OK, challenge
    accepted! Let's make a button! you know it's like a
    blink LED. So buttons, because I was
    not allowed to use soldering iron in
    this talk, I'm using old buttons from old
    scanner. nothing special 3 buttons, in
    this case with hardware debounce, but we
    don't care.
    4 wires 3 buttons.
    how hard can it be?
    well basically it can be really really
    simple  this is the smallest
    font so if you see how to read this I
    congratulate you! in this case I am
    specifying that I want software pull up, in this first fragment on the top.
    I could have put some
    resistors, but you said no soldering
    so here I am telling to my
    processor. please do pull up on
    those pins. and then I'm defining three
    keys. as you can see email. connect and
    print. which generate real Linux
    keyboard events. so if you are in X and
    press that key, it will generate that key,
    I thought it would it would be better to
    generate you know the magic multimedia
    key bindings as opposed to A, B and C
    because if I generated a ABC and was its
    console I would actually generate
    letters on a login prompt which I didn't
    want so actually did it and it's quite
    quite easy. In this case I'm using gpio-keys-polled which means that my CPU
    is actually pulling every 100
    milliseconds those keys to see whether
    they their status changed and since the
    board is actually connected through the
    current sensing - sensor I mentioned earlier I'm
    getting additional: how many mA?
    every 100 milliseconds, pulling 3 keys?
    60! I wouldn't expect my power
    consumption to rise by 60 milliamps
    because I am pulling 3 keys every 100
    milliseconds! but it did and because I
    could add sensors to the linux without
    programming drivers, I know that! why am I
    using polling because on allwinner
    all pins are not interrupt capable so in
    the sense all pins cannot generate
    interrupts on allwinner
    raspberry pi in this case is different
    on raspberry pi every pin can be
    interrupt pin on Allwinner that is not
    the case. so the next logical question is
    how do I know when I'm sitting in front
    of my board whether the pin can get the
    interrupt or not? my suggestion is ask
    the kernel. just grap through the debug
    interface of the kernel through pinctrl
    which is basically the the thing
    which configures the pins on your arm
    CPU and try to find the irq
    will surely get the list of the pins
    which are which are irq capable take in
    mind that this will be different on
    different arm architectures so
    unfortunately on allwinner
    it will always look the same because it
    is allwinner architecture. actually
    sunxi ! but on the Raspberry Pi for
    example this will be somewhat different
    but the kernel knows, and the grep is
    your friend. so you wrote the device
    three you load it either live or some
    something on some other way you connect
    your buttons and now let's try does it really
    work? of course it does! we will start
    evtest which will show us all the
    input devices we have the new one is the
    gpio-3-buttons, which is the same name
    as our device three overlay and we can see
     the same things we saw in
    device three we defined three keys with
    this event but we free-of-charge
    got for example keyboard repeat because
    this is actually meant to be used for
    keyboards our kernel is automatically
    implementing repeat key we can turn it
    off but this is example of one of the
    features which you probably wouldn't
    implement yourself if you are connecting
    those three keys to your Arduino but but
    but this is still not the
    if this is the Internet it should have
    some kind of
    Internet in it some buzzwords for
    example mqtt and it really can just
    install trigger-happy demon which is
    nice deamon which listens to
    input events and write a free file
    configuration which will send the each
    key pressed order MQTT and job done i
    did the internet button without a line
    of code the configuration one side note
    here if you are designing some some
    Internet of Things thingy which even if
    you are only one who will use it it's a
    good idea but if you are doing it for
    somebody else
    please don't depend on the cloud because
    I wouldn't like for my door to be locked
    permanently with me outside just because
    my internet connection isn't working
    think about it of course you can use any
    buttons in this case this was the first
    try actually like the three buttons
    better than this one that's why this
    these buttons are coming second and this
    board with the buttons has one
    additional nice thing and that is the
    LED we said that will cover buttons and
    LEDs right unfortunately this LED is 5
    volts so it won't light up on 3.3 volts
    but when you mentioned LEDs something
    came to mind how can I use those LEDs
    for something more useful than just
    blinking them on or off we'll see you
    later then turning them on or off it is
    also useful you probably didn't know
    that you can use Linux triggers to
    actually display status of your MMC card
    CPU load network traffic or something
    else on the LEDs itself either the LEDs
    which you already have on the board but
    if your manufacturer didn't provide
    enough of them you can always just add
    random LEDs, write device tree and
    define the trigger for them, and this is
    an example of that
    and this example is actually for this
    board which is from the ThinkPad
    ThinkPad dock to be exact, which
    unfortunately isn't at all visible
    on this picture, but you will believe me,
    has actually 2 LEDs and these 3
    keys and 2 LEDs actually made the arm
    board which doesn't have any buttons on
    it or status LEDs somewhat flashy with
    buttons. that's always useful on the
    other hand here I would just want to
    share a few hints with you for a start
    first numerate your pins because if you
    compared the the picture down there
    which has seven wires and are numerated
    which is the second try be the first
    nodes on the up you will see that in
    this case i thought that there was eight
    wires so the deducing what is connected
    where when you have the wrong number of
    wires is maybe not the good first step
    on the other hand we saw the keys what
    about rotary encoders? for years I was
    trying to somehow persuade Raspberry Pi
    1 as the lowest common denominator of
    all arm boards you know cheap slow and
    so on to actually work with this exact
    rotary encoder the cheapest one from the
    Aliexpress of course see the pattern
    I tried Python I try the attaching
    interrupt into Python I tried C code and
    nothing worked at least didn't work
    worked reliably and if you just write
    the small device tree say the correct
    number of steps you have at your rotary
    encoder because by default it's 24 but
    this particular one is 20 you will get
    perfect input device for your Linux with
    just a few wires
    amazing so we saw the buttons we saw the
    LEDs we have everything for IOT except
    the relay so you saw on one of the
    previous pictures this relay box it's
    basically four relays separated by
    optocouplers which is nice and my
    suggestion since you can't in device
    three you can't say this pin will be
    output but I want to initially drive it
    high or I want to initially drive it low
    it seems like you can say that it's
    documented in documentation it's just
    not implemented there on every arm
    architecture so you can write it in
    device three but your device tree will
    ignore it. you but you can and this might be
    somewhat important because for example
    this relay is actually powering all your
    other boards and you don't want to
    reboot them just because you reboot the
    machine which is actually driving the
    relay so you want to control that pin as
    soon as possible so my suggestion is
    actually to explain to Linux kernel that
    this relays is actually 4 LEDs which is
    somewhat true because the relay has the
    LEDs on it and then use LEDs which do
    have the default state which works to
    actually drive it as soon as possible as
    the kernel boot because kernel will boot
    it will change the state of those pins
    from input to output and set them
    immediately to correct value so you
    hopefully want want power cycle your
    other boards, and then you can use the
    LEDs as you would normally use them in
    any other way
    if LEDs are interesting to you have in
    mind that on your on each of your
    computers you have at least two LEDs but
    this caps lock and one is non lock on
    your keyboard and you can use those same
    triggers I mentioned earlier on your
    existing Linux machine using those
    triggers so for example your caps lock
    LED can blink as your network traffic
    does something on your network really
    it's fun on the other hand if you have a
    Raspberry Pi and you defined everything
    correctly you might hit into some kind
    of problems that particular chip has
    default pull ups which you can't turn on
    for some pins which are actually
    designed to be clocks of this kind or
    another so even if you are not using
    that pin as the SPI clock whatever you
    do in your device tree you won't be able to
    turn off, actually you will turn
    off the the setting in the chip to for
    the pull up but the pull up will be
    still there actually thought that there
    is a hardware resistor on board but
    there isn't it's inside the chip just
    word of warning so if anything I would
    like to push you towards using Linux
    kernel for the sensors which you might
    not think of as the first choice if you
    just want to add some kind of simple
    sensor to to your Linux instead of
    Arduino over serial port which I did and
    this is the solution which might with
    much less moving parts or wiring pie and
    in the end once you make your own shield
    for us but if I you will have to write
    that device tree into the EEPROM anyway so
    it's good to start learning now so I
    hope that this was at least useful very
    interesting and if you have any
    additional questions I will be glad to
    to answer them and if you want to see
    one of those all winner board which can
    be used with my software to show the pin
    out here is one board which kost
    actually borrowed me yesterday and in which
    yesterday evening I actually ported the
    my software which is basically just
    writing the definition of those pins
    over here and the pins on the header
    which I basically copy pasted from the
    excel sheet just to show that you can
    actually do that for any board you have
    with just really pin out in textual file
    it's really that simple
    here are some additional
    links and do you have any questions?

  • DORS/CLUC 2017: bro - what's in your network?

    This year on DORS/CLUC 2017, I talked about bro which is great network analysys tools everybody should know, but dispite it's long history is not as well known as you would suspect.

  • etckeeper, bind jnl files and git-pack memory problems

    For last few years, one of first tools which we install on each new server is etckeeper. It saved us couple of times, and provides nice documentation about changes on the system.

    However, git can take a lot of space if you have huge files which change frequently (at least daily since etckeeper has daily cron job to commit changes done that day). In our case, we have bind which stores jnl files in /etc/bind which results in about 500 Kb change each day for 11 zones we have defined.

    You might say that it doesn't seem to be so bad, but in four months, we managed to increase size of repository from 300 Mb to 11 Gb. Yes, this is not mistake, it's 11000 Mb which is increase of 36 times! Solution for this is to use git gc which will in turn call git-pack to compress files. But this is where problems start -- git needs a lot of RAM to do gc. Since this machine has only 1 Gb of RAM, this is not enough to run git gc without running out of memory.

    Last few times, I transferred git repository to other machine, run git gc there and than transfered it back (resulting in nice decrease from 11 Gb back to 300 Mb), however, this is not ideal solution. So, let's remove bind jnl files from etckeeper...

    Let's start with our 11 Gb git repo, copy it to another machine which has 64 Gb or RAM needed for this operation.

    root@dns01:/etc# du -ks .git
    11304708        .git
    root@dns01:~# rsync -ravP /etc/.git
    Now, we will re-create local files because we need to find out which jnl files are used so we can remove them from repo.
    root@build:/srv/dns01/etc# git reset --hard
    ls bind/*.jnl | xargs -i git filter-branch -f --index-filter 'git rm --cache --ignore-unmatch {}'
    echo 'bind/*.jnl' >> .gitignore
    git commit -m 'ignore ind jnl files' .gitignore
    Now, finally we can shrink our 11 Gb repo!
    root@build:/srv/dns01/etc# du -kcs .git
    11427196        .git
    root@build:/srv/dns01/etc# git gc
    Counting objects: 38117, done.
    Delta compression using up to 18 threads.
    Compressing objects: 100% (27385/27385), done.
    Writing objects: 100% (38117/38117), done.
    Total 38117 (delta 27643), reused 12846 (delta 10285)
    Removing duplicate objects: 100% (256/256), done.
    root@build:/srv/dns01/etc# du -ks .git
    414224  .git
    # and now we can copy it back...
    root@dns01:/etc# rsync -ravP .
    Just as side note, if you want to run git gc --aggressive on same repo, it won't finish with 60 Gb or RAM and 100 Gb of swap, which means that it needs more than 150 Gb of RAM.

    So, if you are storing modestly sized files which change a lot, have in mind that you might need more RAM to run git gc (and get disk usage under control) than you might have.

  • Let's hack cheap hardware - 2016 edition

    Last week I head pleasure to present at two conferences in two different cities: DORS/CLUC 2016 and Osijek Mini Maker Faire on topic of cheap hardware from China which can be improved with little bit of software or hardware hacking. It was well received, and I hope thet you will find tool or two in it which will fill your need.

    I hope to see more hacks of STM8 based devices since we have sdcc compiler with support for stm8, cheap SWIM programmer in form of ST-Link v2 (Chinese clones, which are also useful as ARM SWD programmers) and STM8 has comparable features to 8-bit AVR micro-controllers but cheaper.

  • Debian OpenLDAP with GnuTLS and OpenSSL certificates

    Every few years we have to renew SSL certificates. And there is always something which can go wrong. So I decided to reproduce exact steps here so that Google can find it for next unfortunate soul who has same problem.

    Let's examine old LDAP configuration:

    deenes:/etc/ldap/slapd.d# grep ssl cn\=config.ldif 
    olcTLSCACertificateFile: /etc/ssl/certs/
    olcTLSCertificateFile: /etc/ssl/certs/
    olcTLSCertificateKeyFile: /etc/ssl/private/
    We need to convert OpenSSL key into format which GnuTLS understands:
    deenes:/etc/ssl/private# certtool -k < star_ffzg_hr.key > /tmp/star_ffzg_hr.gnutls.key
    Than we need to create certificate which includes our certificate and required chain in same file:
    deenes:/etc/ldap/slapd.d# cat /etc/ssl/certs/star_ffzg_hr.crt /etc/ssl/certs/DigiCertCA.crt > /etc/ssl/certs/chain-star_ffzg_hr.crt
    All is not over yet. OpenLDAP doesn't run under root priviledges, so we have to make sure that it's user is in ssl-cert group and that our certificates have correct permissions:
    deenes:/etc/ldap/slapd.d# id openldap
    uid=109(openldap) gid=112(openldap) groups=112(openldap),104(ssl-cert)
    deenes:/etc/ldap/slapd.d# chgrp ssl-cert \
    /etc/ssl/certs/DigiCertCA.crt \
    /etc/ssl/certs/star_ffzg_hr.crt \
    /etc/ssl/certs/chain-star_ffzg_hr.crt \
    deenes:/etc/ldap/slapd.d# chmod 440 \
    /etc/ssl/certs/DigiCertCA.crt \
    /etc/ssl/certs/star_ffzg_hr.crt \
    /etc/ssl/certs/chain-star_ffzg_hr.crt \
    deenes:/etc/ldap/slapd.d# ls -al \
    /etc/ssl/certs/DigiCertCA.crt \
    /etc/ssl/certs/star_ffzg_hr.crt \
    /etc/ssl/certs/chain-star_ffzg_hr.crt \
    -r--r----- 1 root ssl-cert 3764 Jan 19 09:45 /etc/ssl/certs/chain-star_ffzg_hr.crt
    -r--r----- 1 root ssl-cert 1818 Jan 17 16:13 /etc/ssl/certs/DigiCertCA.crt
    -r--r----- 1 root ssl-cert 1946 Jan 17 16:13 /etc/ssl/certs/star_ffzg_hr.crt
    -r--r----- 1 root ssl-cert 5558 Jan 19 09:23 /etc/ssl/private/star_ffzg_hr.gnutls.key
    Finally, we can modify LDAP configuration to use new files:
    deenes:/etc/ldap/slapd.d# grep ssl cn\=config.ldif 
    olcTLSCACertificateFile: /etc/ssl/certs/DigiCertCA.crt
    olcTLSCertificateFile: /etc/ssl/certs/chain-star_ffzg_hr.crt
    olcTLSCertificateKeyFile: /etc/ssl/private/star_ffzg_hr.gnutls.key
    We are done, restart slapd and enjoy your new certificates!

  • FSec 2015 - Raspberry PI for all your GPIO needs

    When I started playing with Raspberry Pi, I was a novice in electronics (and I should probably note that I'm still one :-).

    But since then, I did learn a few things, and along that journey I also figured out that Raspberry Pi is great little device which can be used as 3.3V programmer for AVR, JTAG, SWD or CC111x devices (and probably more).

    I collected all my experiences in presentation embedded below which I had pleasure to present at FSec conference this year. I hope you will find this useful.

  • DORS/CLUC 2015: AVR component tester

    Few weeks ago, we had our annual conference DORS/CLUC 2015 on which I had interesting (hopefully) presentation about AVR component tester. Since then, we got video recording of conference, so below you can find embedded presentation and video recording (in Croatian).

  • Overview of ganeti cluster from command line: ps, kvm, proc and tap

    We have been running ganeti cluster in our institution for more than a year now. We did two cycles of machine upgrades during that time, and so far we where very pleased with ability of this cloud platform. However, last week we had a problem with our instances -- two of them got owned and started generating DoS service attack to external resources. From our side it seemed at first like our upstream link is over saturated, and we needed to way to figure out why it is.


    At first, it seemed like this would be easy to do. Using dstat,i I found that we are generating over 3 Gb/s traffic every few seconds to outside world. We have 1 Gb/s upstream link, but our bonded interfaces on ganeti nodes can handle 3 Gb/s of traffic, so for a start we where saturating our own link.

    But which instance did that? I had to run dstat on every node in our cluster until I found two nodes which had instances which where overloading our link. Using iftop I was able to get hostname and IP address of instances which I wanted to shut down. However, this is where problems started. We didn't have DNS entries for them, and although I had IP and mac address of instances I didn't had easy way to figure our which instance has that mac.

    Than I figured out that I can get mac from kvm itself, using ps. Once I found instances it was easy to stop then and examine what happened with them.

    But, this got me thinking. Every time I have a troubleshooting problem with ganeti, I basically use more or less same command-line tools to figure out what is going on. But I didn't have a tool which would display me some basic stats about instance, but including mac addresses and network traffic (which in our configuration are tap devices added to bridges). So I wrote, gnt-info which presents nice overview of your instances in ganeti cluster which you can grep to drill-down into particular instance or host.

  • Controlling 315 MHz light sockets using Arduino

    We all read hackaday, and when I read Five Dollar RF Controlled Light Sockets post I decided that I have to buy some. However, if you read comments on original Cheap Arduino Controlled Light Sockets - Reverse Engineering RF post and especially comments, you will soon figure out that ordering same looking product from China might bring you something similar but with different internals.

    In my case, all four light sockets turn on or off with any button press on remote which was a shame. When I opened remote and socket, I also had bad surprise. My version didn't have any SPI eeprom, but just two chips, ST F081 FB 445 in remote and ST ED08 AFB422 in light bulb (in picture hidden below receiver board).

    remote.jpg socket-top.jpg socket-bottom.jpg

    But, I already had acquired two sets so I wanted to see what I can do with them. Since I couldn't read eeprom to figure out code, I decided to use rtl-sdr to sniff radio signals and try to command them using cheap 315 MHz Arduino module.

    I used gqrx to sniff radio signals and I was not pleased. Remote drifted all over the place mostly around 316 MHz and it was some trial and error to capture signals which are generated when buttons are pressed. However, I have verified that it's sending same signal multiple times no matter which keys I press (which would explain why four pins on remote are soldered together).

    After a while I had two traces (since I have two sets of light sockets) and could decode binary data which is sent from following picture:


    How I knew that one set is transmitting 1000100110110000000000010 and another one 1011001001011111000000010. From looking into timing in audacity, it seemed that each bit is encoded in short-long or long-short sequence where short one is about third of long one, and one bit is about 1200 ms. I cheated here a little and stuck scope into scope into transmit trace on remote to verify length of pulses just to be sure.

    So as next step I wrote simple Arduino sketch to try it out:

    #define TX_PIN 7
    #define LED_PIN 13
    char *code = "1000100110110000000000010";
    //char *code = "1011001001011111000000010";
    void setup() {
      pinMode(LED_PIN, OUTPUT);
      pinMode(TX_PIN, OUTPUT);
    void loop() {
      digitalWrite(LED_PIN, HIGH);
      for(int i = 0; i < strlen(code); i++) {
        int i1 = 300;
        int i2 = 900;
        if (code[i] == '1' ) {
          i1 = 900;
          i2 = 300;
        digitalWrite(TX_PIN, HIGH);
        digitalWrite(TX_PIN, LOW);
      digitalWrite(LED_PIN, LOW);  
    So, I compiled it, uploaded to Arduino and... nothing happens. Back to the drawing board, I guess.

    When I was looking into gqrx I could see that signal is sent as long as I'm holding button up to 10 seconds. From experience before I know that this cheap receivers need some tome to tune into frequency so next logical step was to send same signal multiple times. And guess what, when I sent same singal twice with 2000 ms delay between them everything started to work.

    Again somewhat. Light socket in far corner of hall seemed to have problems receiving signal which would put two light socket in hall in opposite state: one would be on and another would be off. This was fun, and could be fixed with simple antenna on Arduino module (since currently I don't have any) but I will conclude that your IoT device should send different codes for on and off state so something like this won't happen to you.

    Then I got carried away and added commands to change all parameters to experiment how sensitive receiver is. You can find full code at;a=blob;f=light_sockets/light_sockets.ino With this experiments I found out that you don't have to be precise with timings (so my oscilloscope step was really not needed). Receiver works with 500 ms low and 1100 ms high (for total of 1600 ms per bit) on high end, down to 200 ms for low and 800 ms for high (for total of 1000 ms per bit).

    I suspect that chips are some kind of 26 bit remote encoders/decoders but I can't find any trace of datasheet on Internet. This is a shame, because I suspect that it's possible to program light sockets to respond to any code and in theory address each of them individually (which was my goal in beginning). However poor construction quality, and same code for on and off state (combined with poor reception) makes me wonder if this project is worth additional time.

  • Reusing servos from old printers with Arduino

    I must confess that I'm pack rat. When I see old printer, something inside my head tries to figure out what I can do with all parts inside it instead of passing it to land-fill. However, I'm sysadmin and software guy, so JTAGs and programming is more up my wally than hardware. However, I decided to figure out how to drive one of servos using Arduino and this is my journey through this experience.

    So I started with printer disassembly and got one stepper motor and some gears on it. It is Mitsumi M42SP-6TE. It has four wires and I couldn't find any data sheet about it. So what do I do now?


    First some educated guesses.I assumed that it's 12V servo. This was somewhat influenced by examining similar Mitsumi MP42SP-6NK motor which have rating of 12V or 24V. Using unimer and taking ohm measurement between wires I confirmed that it has 10 Ω between coils which means it's bipolar, having two different coils which had both to be driven at the same time.


    To connect it to Arduino, I acquired some time ago clone of Adafruit motor shield. When you buy cheap clones you expect some problems, and mine was fact that screw terminals on board weren't cut flash with board, so I had to use flat cutters and shorten them to prevent motor power from shorting with ICSP header on Arduino and USB connector on Uno. I also used red electrical tape and put it on USB connector just to be safe(r).


    I also needed to add power jumper (white on picture) to provide power from Arduino (which in turn is powered by 12V 1A adapter). However, in this configuration L293D H-bridge becomes very hot to touch, so for testing I modified StepperTest example to provide me with serial control and powered Arduino from USB port (from which it draws 0.42 A and stepper still works with 5V supply which makes my 12 V assumption somewhat questionable). This enabled me to deduce that this stepper is also 7.5° which takes 48 steps to do full turn (small red dot on stepper gear helped to verify this). I also verified that top gear has 13:1 ratio to stepper motor making gear mechanism useful for smaller movements and better tork.

    I hope this blog post will motive you to take old printers, scanners, faxes and similar devices apart and take useful parts out if it. Re-using boards for driving steppers is also very interesting, but this particular printer didn't come with power supply (and it has strange connector) and driver chip on it doesn't have any publicly available info, so this will have to wait some other printer which will decide to give up it's parts for my next project...

  • FSec 2014 - I can haz your board with JTAG


    Last week I had pleasure of attending FSec 2014, annual security conference. Just like last year, I had hardware presentation, this time about reverse engineering NComputing CPLD dongle. You can find it on or embedded below.

    I had great time at conference, but I'm somewhat wondering did audience got something from my lecture. It was very interesting for me to figure out JTAG pinout on this board, and connect it to various JTAG programmers (all with their's good and bad sides) and I noticed that there are not any introductory text on the web how to approach this problem for the first time. So, I decided to present this topic in hope that this will motivate other people to take a hack at some board which would otherwise end up on e-waste of even worse, land-fill. And, who can resist call of free hardware which you can re-purpose? :-)