Quick jump:  What's new

 
Go to:
 
Weblog: howto   
in Dobrica Pavlinušić's random unstructured stuff
RaspberryPi
mount-rpi-image.sh

Contents: [Dobrica Pavlinušić's random unstructured stuff]




Interestring projects

TODO: test them out and write full description on this wiki :-)

development

OV7670

DHT11/DHT22

BlinkM + i2c

CEC - control your TV

http://www.raspberrypi.org/forum/viewtopic.php?f=35&t=15749

root@raspberrypi:/home/pi# git clone https://github.com/Pulse-Eight/libcec.git
root@raspberrypi:/home/pi# cd libcec/
root@raspberrypi:/home/pi/libcec# apt-get install autoconf automake libtool liblockdev1-dev libudev-dev libraspberrypi-dev
root@raspberrypi:/home/pi/libcec# sh -x bootstrap
root@raspberrypi:/home/pi/libcec# ./configure --with-rpi-include-path=/opt/vc/include/ --with-rpi-lib-path=/opt/vc/lib/

##############################################################################

libCEC version 2:1:0 configured

Compilation flags:
  CXXFLAGS :       -g -O2 -fPIC -Wall -Wextra -Wno-missing-field-initializers  -Wno-psabi
  libCEC LDFLAGS : -lbcm_host -llockdev -lrt -ldl -lpthread  -L/opt/vc/lib/ -lvcos -lvchiq_arm -ludev  
  client LDFLAGS : -lrt -ldl -lpthread 

Configured features:
  Pulse-Eight CEC Adapter :             yes
  Pulse-Eight CEC Adapter detection :   yes
  Raspberry Pi support :                yes
  TDA995x support :                     no

You can now build libCEC by running:
  make

##############################################################################

make install

TV on

pi@raspberrypi ~ $ echo "on 0" | cec-client -s

Hints from forum

screen saver

http://www.raspberrypi.org/phpBB3/viewtopic.php?p=84831#p84831

# prevent screensaver from powering down display
setterm -blank 0 -powerdown 0 > /dev/tty0

# clear the display and turn off the flashing cursor
clear > /dev/tty0
setterm -cursor 0 > /dev/tty0

# turn the cursor back on when done with omxplayer
setterm -cursor 1 > /dev/tty0

YouTube transcoding in VLC

http://www.raspberrypi.org/phpBB3/viewtopic.php?p=163658#p163658

cvlc 'youtube.com/watch?v=XXXXXXXXXX' :sout='#transcode{vcodec=h264,vb=0,fps=0,scale=0,height=720,acodec=mp4a,ab=128,channels=2,samplerate=44100}:http{mux=ffmpeg{mux=flv},dst=:8080/youtube}' :sout-keep

Education materials

Raspberry Pi Accessories Starter Kit

http://www.conrad.com/ce/en/product/409208/Raspberry-Pi-Accessories-Starter-Kit?ref=searchDetail

  • 1x Rapberry sticker

GPU

Arduino

JTAG

Arduino ProMini 3.3V 8MHz

Disable serial port console

edit /boot/cmdline.txt and remove all ttyAMA0 leaving:

pi@raspberrypi ~ $ cat /boot/cmdline.txt
dwc_otg.lpm_enable=0 console=tty1 root=/dev/mmcblk0p2 rootfstype=ext4 elevator=deadline rootwait bcm2708.w1_gpio_pin=4

also, comment out ttyAMA0 in /etc/inittab

connect Arduino serial

RX->TX
TX->RX

https://github.com/dpavlin/avrdude-rpi/tree/fix_channel_in_use

pin definitions in GPIO.RPi are physical pins on board not BCM or wiring ip!

gpio1.png

flasing

dpavlin@blue:~/Arduino/RF433_Sockets$ scp /tmp/build*.tmp/*.hex pi@rpi.tv:/tmp/
RF433_Sockets.cpp.hex                                        100%   19KB  18.6KB/s   00:00

pi@raspberrypi ~ $ cat avrdude.sh 
#!/bin/sh
hex=`ls -t /tmp/*.hex | head -1`
echo "flash hex: $hex"
sudo /home/pi/avrdude-rpi/avrdude-autoreset -C /home/pi/avrdude.conf -v -v -patmega328p -carduino -P/dev/ttyAMA0 -b57600 -D -Uflash:w:$hex:i 

pi@raspberrypi ~ $ ./avrdude.sh
flash hex: /tmp/RF433_Sockets.cpp.hex

avrdude: Version 6.1, compiled on Sep 17 2014 at 05:41:35
         Copyright (c) 2000-2005 Brian Dean, http://www.bdmicro.com/
         Copyright (c) 2007-2014 Joerg Wunsch

         System wide configuration file is "/home/pi/avrdude.conf"
         User configuration file is "/root/.avrduderc"
         User configuration file does not exist or is not a regular file, skipping

         Using Port                    : /dev/ttyAMA0
         Using Programmer              : arduino
         Overriding Baud Rate          : 57600
done with autoreset
strace: |/home/pi/avrdude-rpi/autoreset: Broken pipe
strace: |/home/pi/avrdude-rpi/autoreset: Broken pipe
         AVR Part                      : ATmega328P
         Chip Erase delay              : 9000 us
         PAGEL                         : PD7
         BS2                           : PC2
         RESET disposition             : dedicated
         RETRY pulse                   : SCK
         serial program mode           : yes
         parallel program mode         : yes
         Timeout                       : 200
         StabDelay                     : 100
         CmdexeDelay                   : 25
         SyncLoops                     : 32
         ByteDelay                     : 0
         PollIndex                     : 3
         PollValue                     : 0x53
         Memory Detail                 :

                                  Block Poll               Page                       Polled
           Memory Type Mode Delay Size  Indx Paged  Size   Size #Pages MinW  MaxW   ReadBack
           ----------- ---- ----- ----- ---- ------ ------ ---- ------ ----- ----- ---------
           eeprom        65    20     4    0 no       1024    4      0  3600  3600 0xff 0xff
           flash         65     6   128    0 yes     32768  128    256  4500  4500 0xff 0xff
           lfuse          0     0     0    0 no          1    0      0  4500  4500 0x00 0x00
           hfuse          0     0     0    0 no          1    0      0  4500  4500 0x00 0x00
           efuse          0     0     0    0 no          1    0      0  4500  4500 0x00 0x00
           lock           0     0     0    0 no          1    0      0  4500  4500 0x00 0x00
           calibration    0     0     0    0 no          1    0      0     0     0 0x00 0x00
           signature      0     0     0    0 no          3    0      0     0     0 0x00 0x00

         Programmer Type : Arduino
         Description     : Arduino
         Hardware Version: 2
         Firmware Version: 1.16
         Vtarget         : 0.0 V
         Varef           : 0.0 V
         Oscillator      : Off
         SCK period      : 0.1 us

avrdude: AVR device initialized and ready to accept instructions

Reading | ################################################## | 100% 0.01s

avrdude: Device signature = 0x1e950f
avrdude: safemode: lfuse reads as 0
avrdude: safemode: hfuse reads as 0
avrdude: safemode: efuse reads as 0
avrdude: reading input file "/tmp/RF433_Sockets.cpp.hex"
avrdude: writing flash (6770 bytes):

Writing | ################################################## | 100% 1.98s

avrdude: 6770 bytes of flash written
avrdude: verifying flash memory against /tmp/RF433_Sockets.cpp.hex:
avrdude: load data flash data from input file /tmp/RF433_Sockets.cpp.hex:
avrdude: input file /tmp/RF433_Sockets.cpp.hex contains 6770 bytes
avrdude: reading on-chip flash data:

Reading | ################################################## | 100% 1.49s

avrdude: verifying ...
avrdude: 6770 bytes of flash verified

avrdude: safemode: lfuse reads as 0
avrdude: safemode: hfuse reads as 0
avrdude: safemode: efuse reads as 0
avrdude: safemode: Fuses OK (E:00, H:00, L:00)
strace: |/home/pi/avrdude-rpi/autoreset: Broken pipe
strace: |/home/pi/avrdude-rpi/autoreset: Broken pipe
strace: |/home/pi/avrdude-rpi/autoreset: Broken pipe
strace: |/home/pi/avrdude-rpi/autoreset: Broken pipe
strace: |/home/pi/avrdude-rpi/autoreset: Broken pipe

avrdude done.  Thank you.

strace: |/home/pi/avrdude-rpi/autoreset: Broken pipe

DS1307 Tiny RTC

Check if module has 5V pull-up I2C resistors! If it does (like Tiny RTC has R2 and R3) you will have to remove them.

tiny-rtc-3.3v-remote-r2-r3.jpg

root@raspberrypi:/home/pi# apt-get install i2c-tools

root@raspberrypi:/home/pi# modprobe i2c-dev

root@raspberrypi:/home/pi# i2cdetect -y 1
     0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f
00:          -- -- -- -- -- -- -- -- -- -- -- -- -- 
10: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- 
20: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- 
30: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- 
40: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- 
50: 50 -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- 
60: -- -- -- -- -- -- -- -- 68 -- -- -- -- -- -- -- 
70: -- -- -- -- -- -- -- --                         

root@raspberrypi:/home/pi# modprobe rtc-ds1307

root@raspberrypi:/home/pi# echo ds1307 0x68 > /sys/class/i2c-adapter/i2c-1/new_device

root@raspberrypi:/home/pi# hwclock --systohc -D --noadjfile --utc
hwclock from util-linux 2.20.1
Using /dev interface to clock.
Assuming hardware clock is kept in UTC time.
Time elapsed since reference time has been 0.722720 seconds.
Delaying further to reach the new time.
Setting Hardware Clock to 11:53:41 = 1429876421 seconds since 1969
ioctl(RTC_SET_TIME) was successful.

DS18B20

root@raspberrypi:/home/pi# grep w1 /boot/config.txt 
dtoverlay=w1-gpio

# reboot

root@raspberrypi:/home/pi# cat /sys/bus/w1/devices/28-0000043c5507/w1_slave 
50 05 4b 46 7f ff 0c 10 1c : crc=1c YES
50 05 4b 46 7f ff 0c 10 1c t=85000

Hmm, you will also want to remove R1 pullup to make it work?

avrdude

Using Raspberry Pi as Arduino ISP programmer

root@raspberrypi:/home/pi# avrdude -c linuxgpio -p atmega328p

avrdude: AVR device initialized and ready to accept instructions

Reading | ################################################## | 100% 0.00s

avrdude: Device signature = 0x1e950f

avrdude: safemode: Fuses OK (E:05, H:DA, L:FF)

avrdude done.  Thank you.

logic analyzer

rotary encoder

http://www.bobrathbone.com/raspberrypi_rotary.htm

SWD

http://www.disk91.com/2015/technology/programming/swd-programming-using-a-raspberrypi/
https://github.com/disk91/PySWD

pi@raspberrypi ~ $ git clone https://github.com/disk91/PySWD
pi@raspberrypi ~ $ cd PySWD/


PIC

esp8089

PS/2 GPIO

GPIO header annotations

Signal generator

pi@rpi2 ~/WsprryPi $ sudo ./wspr --test-tone 1000000
A test tone will be generated at frequency 1.000000 MHz

Using local mbox device file with major 249.
Transmitting test tone on frequency 1.000000 MHz
Press CTRL-C to exit!

LCD panels

http://blog.reasonablycorrect.com/raw-dpi-raspberry-pi/

Pi Zero USB OTG

libcomposite

PATA IDE emulation

DPI LCD screen

i2c clock stretching

permalink
Parallella


power

Update boot flash

This is required to get CR10 led working

enviroment before u-boot update:

zynq-uboot> env print
baudrate=115200
bitstream_image=parallella.bit.bin
bootcmd=run modeboot
bootdelay=0
devicetree_image=devicetree.dtb
devicetree_size=0x20000
ethact=zynq_gem
ethaddr=04:4f:8b:00:10:13
fdt_high=0x20000000
initrd_high=0x20000000
ipaddr=192.168.0.99
kernel_image=uImage
kernel_size=0x500000
modeboot=run qspiboot
qspiboot=echo Configuring PL and Booting Linux...;mmcinfo;fatload mmc 0 0x4000000 ${bitstream_image};fpga load 0 0x4000000 0x3dbafc;fatload mmc 0 0x3000000 ${kernel_image};fatload mmc 0 0x2A00000 ${devicetree_image};bootm 0x3000000 - 0x2A00000
serverip=192.168.0.101
stderr=serial
stdin=serial
stdout=serial

Environment size: 655/131068 bytes

flasing output:

zynq-uboot> mmcinfo
Device: SDHCI
Manufacturer ID: 3
OEM: 5344
Name: SU08G 
Tran Speed: 50000000
Rd Block Len: 512
SD version 2.0
High Capacity: Yes
Capacity: 7.4 GiB
Bus Width: 4-bit
zynq-uboot> fatload mmc 0 0x4000000 parallella.7020.flash.bin
reading parallella.7020.flash.bin

4351800 bytes read
zynq-uboot> sf probe 0 0 0
SF: Detected N25Q128 with page size 64 KiB, total 16 MiB
zynq-uboot> sf erase 0 0x1000000
SF: Successfully erased 16777216 bytes @ 0x0
zynq-uboot> sf write 0x4000000 0 0x$filesize
SF: program success 4351800 bytes @ 0x0

# I skipped set ethaddr because it was already set correctly

zynq-uboot> set AdaptevaSKU SKUA101040
zynq-uboot> saveenv
Saving Environment to SPI Flash...
SF: Detected N25Q128 with page size 64 KiB, total 16 MiB
Erasing SPI flash...SF: Successfully erased 131072 bytes @ 0x4e0000
Writing to SPI flash...SF: program success 131072 bytes @ 0x4e0000
done



GPIO

dpavlin@parallella:~/parallella-utils$ make gpiotest
gcc -o gpiotest gpiotest.c para_morse.c para_gpio.c -Wall -lrt

dpavlin@parallella:~/parallella-utils$ sudo ./gpiotest 
[sudo] password for dpavlin: 
GPIOTEST - Basic test of para_gpio

Initializing...
Success, pausing 5 seconds
Setting direction (forces output to 0!)...
Pausing 5 seconds
Reading multiple times
> 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 
Toggling pin 100000 times...
Took 1.432 seconds, 69818 updates/sec
Write / Read test, 100000 times...
Took 2.618 seconds, 76384 ops/sec
Setting output to 0
Verifying level... OK
Disabling output and monitoring input
Gave up waiting for the input to transition
Setting output to 1
Verifying level... OK
Disabling output and monitoring input
Input flipped after 0 cycles
Closing




FPGA

sudo mknod /dev/xdevcfg c 250 0
sudo -s
cat parallella.bit.bin > /dev/xdevcfg
exit

# programming time
cat /sys/devices/amba.1/f8007000.devcfg/prog_done

/sys/bus/iio/devices/iio:device0/in_temp0_raw
/sys/bus/iio/devices/iio:device0/in_temp0_offset
/sys/bus/iio/devices/iio:device0/in_temp0_scale

T = ((in_temp0_raw + in_temp0_offset) * in_temp0_scale) / 1000)

script:

#!/bin/bash
raw=`cat /sys/bus/iio/devices/iio:device0/in_temp0_raw`
offset=`cat /sys/bus/iio/devices/iio:device0/in_temp0_offset`
scale=`cat /sys/bus/iio/devices/iio:device0/in_temp0_scale`

c_temp=`echo "scale=1;(($raw + $offset) * $scale) / 1000" | bc`
f_temp=`echo "scale=1;(($c_temp * 9) / 5) + 32" | bc`

echo
echo "Zynq Temp: $c_temp C / $f_temp F"
echo

Modified to display temperature in loop: temp.sh

headless stream

http://forums.parallella.org/viewtopic.php?f=48&t=1213

removed html traces from device tree

serial

hardware

software

device tree

sudo apt-get install device-tree-compiler

# Compiling a device tree blob from source (.dts-->.dtb):
dtc -I dts -O dtb -o devicetree.dtb zynq-parallella1-headless.dts

# To get source from compiled device tree (.dtb-->.dts):
dtc -I dtb -O dts -o devicetree.dts devicetree.dtb


/dev

see how to modify devicetree at http://elinux.org/Parallella_Speed_Up_Ubuntu

Debian

http://elinux.org/Parallella_Debian

Epiphany

permalink
Huawei E220


Working configuration

(for me at least :-)

/etc/ppp/peers/huawei-e220

dpavlin@llin:~$ cat /etc/ppp/peers/huawei-e220
# /etc/ppp/peers/huawei-e220 pppd script
# pppd call huawei-e220

debug
kdebug 3
nodetach
#persist

/dev/ttyUSB0
460800
idle 7200
noipdefault
defaultroute
usepeerdns

connect "/usr/sbin/chat -vf /etc/chatscripts/huawei-e220.chat"  

# avoid compression:
noccp
#nobsdcomp
novj
# t-mobile specific?
#ipcp-restart 8
#ipcp-max-configure 50
#ipcp-accept-local
#ipcp-accept-remote
#ipcp-max-failure 20

lcp-echo-failure 0
lcp-echo-interval 0

modem
crtscts

require-pap
refuse-chap

show-password
user YOUR_USERNAME@DOMAIN
password YOUR_PASSWORD

passive

/etc/chatscripts/huawei-e220.chat

dpavlin@llin:~$ cat /etc/chatscripts/huawei-e220.chat
TIMEOUT 3
ABORT BUSY 
ABORT 'NO CARRIER' 
ABORT VOICE 
ABORT 'NO DIALTONE' 
ABORT 'NO DIAL TONE' 
ABORT 'NO ANSWER' 
ABORT DELAYED
"" ATZ
OK ATQ0V1E1S0=0&C1&D2
OK AT+COPS?
OK AT+CGDCONT=1,"ip","carnet.vip.hr"
OK ATDT*99#
CONNECT ""

Links

Turn off pin

$ cu -l /dev/ttyUSB0
at+clck="sc",0,"PIN!"
OK

libusb driver program

Compilation

Get source from http://www.kanoistika.sk/bobovsky/archiv/umts/huaweiAktBbo.c

cc     huaweiAktBbo.c   -o huaweiAktBbo -lusb

hal problem

If you modem is restarting like mad, it's problem with hal.

CD image update

Since device can emulate USB CDROM, I will try to update image on it to provide bios update image instead of useless (to me) Windows drivers.

dmesg

dpavlin@t61p:~$ uname -a
Linux t61p 2.6.30-1-686-bigmem #1 SMP Mon Aug 3 17:32:39 UTC 2009 i686 GNU/Linux
dpavlin@t61p:~$ dmesg
[ 3405.568120] usb 4-1: new full speed USB device using uhci_hcd and address 2
[ 3405.726188] usb 4-1: New USB device found, idVendor=12d1, idProduct=1003
[ 3405.726196] usb 4-1: New USB device strings: Mfr=1, Product=2, SerialNumber=0
[ 3405.726203] usb 4-1: Product: HUAWEI Mobile
[ 3405.726208] usb 4-1: Manufacturer: HUAWEI Technologies
[ 3405.726384] usb 4-1: configuration #1 chosen from 1 choice
[ 3405.792930] Initializing USB Mass Storage driver...
[ 3405.796244] usb-storage: probe of 4-1:1.0 failed with error -1
[ 3405.796289] usbcore: registered new interface driver usb-storage[ 3405.796296] USB Mass Storage support registered.
[ 3405.952077] usb 4-1: USB disconnect, address 2
[ 3407.432107] usb 4-1: new full speed USB device using uhci_hcd and address 3
[ 3407.590171] usb 4-1: New USB device found, idVendor=12d1, idProduct=1003
[ 3407.590180] usb 4-1: New USB device strings: Mfr=1, Product=2, SerialNumber=0
[ 3407.590187] usb 4-1: Product: HUAWEI Mobile
[ 3407.590192] usb 4-1: Manufacturer: HUAWEI Technologies
[ 3407.590373] usb 4-1: configuration #1 chosen from 1 choice
[ 3407.603280] usb-storage: probe of 4-1:1.2 failed with error -1[ 3407.650695] usbcore: registered new interface driver usbserial
[ 3407.650725] USB Serial support registered for generic
[ 3407.650819] usbcore: registered new interface driver usbserial_generic
[ 3407.650824] usbserial: USB Serial Driver core
[ 3407.663176] USB Serial support registered for GSM modem (1-port)
[ 3407.663293] option 4-1:1.0: GSM modem (1-port) converter detected
[ 3407.663458] usb 4-1: GSM modem (1-port) converter now attached to ttyUSB0
[ 3407.663478] option 4-1:1.1: GSM modem (1-port) converter detected
[ 3407.663576] usb 4-1: GSM modem (1-port) converter now attached to ttyUSB1
[ 3407.663609] usbcore: registered new interface driver option
[ 3407.663614] option: v0.7.2:USB Driver for GSM modems

power

root@t61p:~/t61p/usb_modeswitch/usb_modeswitch-1.0.2# lsusb
Bus 004 Device 003: ID 12d1:1003 Huawei Technologies Co., Ltd. E220 HSDPA Modem / E270 HSDPA/HSUPA Modem
Bus 004 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub

root@t61p:~/t61p/usb_modeswitch/usb_modeswitch-1.0.2# echo suspend >  /sys/bus/usb/devices/4-1/power/level
root@t61p:~/t61p/usb_modeswitch/usb_modeswitch-1.0.2# dmesg

fetchrss: http://feeds.delicious.com/v2/rss/dpavlin/e220?count=15
  • There was an error: 500 Can't connect to feeds.delicious.com:80 (Bad hostname 'feeds.delicious.com')


E171

Found modem         : E171
Model               : Huawei E171
IMEI                : 354807044078459
Serial NR.          : 4BA5TA10B2706637
Firmware            : 11.126.85.01.143
Compile date / time : Jun 21 2010 20:27:27
Dashboard version   : UTPS11.300.05.15.143_MAC11.301.09.01.143
Chipset             : Qualcomm MSM6290
Voice feature       : disabled
SIM Lock status     : unlocked
Wrong codes entered : 0 (unlock attempts left : 10)

E1752

Found modem         : E1752
Model               : Huawei E1752
IMEI                : 353145034241600
Serial NR.          : O47NAA19A1600110
Firmware            : 11.126.13.00.00
Compile date / time : Jul 05 2010 10:16:42
Dashboard version   : UTPS11.300.05.22.56_MAC11.300.08.19.56
Chipset             : Qualcomm MSM6290
Voice feature       : disabled
SIM Lock status     : unlocked
Wrong codes entered : 0 (unlock attempts left : 10)

AT^CVOICE=0 Enable Voice
AT^CVOICE=1 Disable Voice
AT^CVOICE=? Check status
=========
AT^U2DIAG=0 – switch the device in modem mode only
AT^U2DIAG=1 – device in modem mode + CD-ROM
AT^U2DIAG=255 – modem mode + CD-ROM + Card Reader
AT^U2DIAG=256 – modem mode + Card Reader
permalink
PostgreSQL

SQL snippets for PostgreSQL



Size of transaction log

select sum((pg_stat_file('pg_xlog/' || file)).size)
from pg_ls_dir('pg_xlog') as file
where file ~ '^[0-9A-F]';

Size of tables in current database

select relname,pg_relation_size(oid) as size
from pg_class
where relkind = 'r' and relname not like 'pg_%' order by size desc;

Number of rows in all tables

http://stackoverflow.com/questions/2596670/how-do-you-find-the-row-count-for-all-your-tables-in-postgres

SELECT 
  nspname AS schemaname,relname,reltuples::int
FROM pg_class C
LEFT JOIN pg_namespace N ON (N.oid = C.relnamespace)
WHERE 
  nspname NOT IN ('pg_catalog', 'information_schema') AND
  relkind='r' 
ORDER BY reltuples DESC;

Script to display database size

Script: pg_size

all databases

$ pg_size

all databases, grouped by user

$ pg_size -u

Active locks

select * from pg_locks where pid not in (select procpid from pg_stat_activity);

Graphics

Taken from http://lca2007.linux.org.au/att_data/Miniconfs(2f)PostgreSQL/attachments/advanced.pdf

# CREATE AGGREGATE array_accum (anyelement)
(
    sfunc = array_append,
    stype = anyarray,
    initcond = '{}'
);

# select d, count(*),
       array_to_string(array_accum('+'::text),'') as graph
       from hits group by 1 order by 1 asc;
     d       | count | graph
------------+-------+----------
 2007-01-13 |      8 | ++++++++
 2007-01-14 |      7 | +++++++
 2007-01-15 |      4 | ++++
(3 rows)

Current query activity

select datid,datname,procpid,current_query,query_start from pg_stat_activity where current_query not like '<IDLE>' ;

To actually see queries you also have to edit postgresql.conf to include:

stats_command_string = yes

Quickly convert database table to SQLite

pg_dump --inserts --no-owner --no-privileges --table hr_edu_orgs a3c | grep -v ^SET | sqlite3 var/a3c.sqlite
permalink
Hitchhikers Guide to RFID
Contents: [Dobrica Pavlinušić's random unstructured stuff]


There are many on-line resources about RFID. However, most of them are not well suited for beginners. So, if you just got RFID reader (3M in this case) and want high-level overview of what it is and what it can do, you are out of luck.

Until now, that is. This guide should help you decide if RFID is right thing for your library and when you make choice to implement it, how to do it.

This document will describe my experience with 3M 810 reader using RFID 501: RFID Standards for Libraries RFID_501.pdf

RFID tag

Best way to think about RFID tags is like contact-less readable barcode.

Our particular tags come in two form: RFID stickers (to be placed on books) and plastic credit-card like cards (we use them for patrons).

Have in mind that established practice is to have different RFID systems for books and patrons (we are use same one). When we asked about using same system for books and patrons, we got reply: "we don't have experience with it".
In practice, we have problem with 3M selfcheck software in some special conditions where book reader have patron card in range it gets network connection error with SIP server.
Since normal configuration of selfcheck stations is to have two different systems for patrons and books this problem doesn't show up for other users.

Each tag has unique serial number (SID) assigned by manufacturer and used in RFID collision detection protocol. It looks like hexadecimal number starting with letter E0 like this: E00401003123AA26.

It's best to think of SID as unique identifier of physical tag.
Your information system will have it's own ID (barcode?) for some item.
There are cases in which you might want to change physical tag sticker on book because it's damaged. In that case, you will change SID of that item, but not barcode (which is data programmed on tag itself).

Our initial idea was to use data programmed on chip for everything, and just ignore SIDs, but we found out that there is class of RFID devices which can read ONLY SID from chip (in our case it's photocopying system).
3M software does record SIDs to log file when programming chips, but that's all. It essential ignores it for all practical intends and purposes.

Chips have 7 blocks of user data on it, each block with 4 bytes which enables us to store 28 bytes of user specified data on each tag.

blank tag

3M Manufacturing Blank

Easiest case is blank tag, in which all data on chip is 0x55

0	55 55 55 55		blank tag
1	55 55 55 55
2	55 55 55 55
3	55 55 55 55
4	55 55 55 55
5	55 55 55 55
6	00 00 00 00 

Generic blank

Generic blank seems to erase only first three blocks with zeros:

00	00 00 00 00
01	00 00 00 00
02	00 00 00 00

while rest of tag will be unchanged including rest of data on tag

programmed tag

Tags programmed with 3M software have following data layout on them:

0	04 is 00 tt		i [4 bit] = number of item in set	[1 .. i .. s]
					s [4 bit] = total items in set
					tt [8 bit] = item type

1	dd dd dd dd		dd [16 bytes] = barcode data
2	dd dd dd dd
3	dd dd dd dd
4	dd dd dd dd

5	bb bl ll ll		b [12 bit] = branch [unsigned]
				l [20 bit] = library [unsigned]
6	cc cc cc cc		c [32 bit] = custom signed integer

This basically means that your barcode or identifier of item or patron can have up to 16 characters (by default numeric, but you can extend that to handle alphanumeric and special character if you need that) and three integer values: branch 0 .. 4095, library 0 .. 1048575 and custom data -2147483648 .. 2147483647.

You might want to use those values to uniquely identify your library and branch so that your RFID tags in books won't collide with other libraries.
If you leave decision just to providers of equipment, you might end up with 300000 tags which have plain and simple 0 in those fields. Guess which value will have tags of next library which that provider will have? My guess would be 0 also.

Writing correct numbers in that fields is not enough. If you want to use 3M software, you will also have to setup it to ignore all other tags which doesn't match your library and branch.

security

There is also single byte called AFI or security which can be changed without accessing content of chip. This byte is also readable by more primitive RFID devices like doors to check if book have been checked out from library.

3M is using 0xD7 (215) value for secured items (door will beep) and 0xDA (218) as unsecured. It seems that all other values are ignored.
(I would guess that other manufacturers are using different values)

As I mentioned before, since we don't have any special values in branch, library or custom field, we have situations in which patron cards get secured when patron walks by checkout counter and 3M software is left in checkout mode.
This triggers door to ring when patron passes which is not ideal.

disable tag

3M software have option to disable tags. Initial examination showed that it's simply programming of tag with following content:

00	ff 00 00 00
01	00 00 00 00
02	00 00 00 00
03	00 00 00 00
04	00 00 00 00
05	00 00 00 00
06	00 00 00 00

and security set to d7 (this might be value from tag before disabling it, I'll have to re-check this)

While 3M software will ignore tags programmed with this content, there is not permanent disabling of tag since it can be programmed using other software.

RFID reader device

Reader consists of several part:

  • black pad - reader antenna
  • reader - small box with micro controller and usb port
  • software

Reader is recognized as USB serial device with it's own protocol on serial port. We are mostly interested in it's protocol and our ability to use reader and tags with our custom software.

At first, I assumed that protocol with RFID readers is some kind of standard.
After extensive search on Internet I wasn't able to find any documentation about this particular protocol (I even tried to compare it with existing open source implementations just to be sure).

So, only solution was to do clean-room reverse engineering, and using that technique I developed perl module which can talk with RFID reader which is available at http://svn.rot13.org/index.cgi/RFID

After initial reverse engineering of protocol I rewrote support for 3M and CPR reader which is available at https://github.com/dpavlin/Biblio-RFID

Related blog posts

fetchrss: http://mjesec.ffzg.hr/~dpavlin/blog/mt/mt-search.cgi?tag=RFID&Template=feed&IncludeBlogs=1
  • There was an error: 500 Internal Server Error


More information

If this was too geeky for you here is some additional materials:

ISO standard

  • ISO 15962.2004 - object identifier structure
  • ISO 15693 - RFID (layer 2)
  • ISO 18000 Part 3 Mode 1 - 13.56MHz
  • ISO/IEC JTC1/SC17/WG8

ISO/IEC 14443, Proximity cards

The Standard series ISO/IEC 14443 consists of 4 parts, which are:

ISO/IEC 14443-1 Physical characteristics

17n1363t.doc 17n1363b.doc

ISO/IEC 14443-2 Radio frequency power and signal interface

17n1522t.pdf 17n1522c.doc

ISO/IEC 14443-2/AMD2 Amendment 2: Bit rates of fc/64, fc/32 and fc/16

17n2343T.pdf 17n2343F.doc

ISO/IEC 14443-3 Initialization and anticollision

17n1531t.pdf 17n1531c.doc

ISO/IEC 14443-3 Amendment 1: Bit rates of fc/64, fc/32 and fc/16

17n2342T.pdf 17n2342F.doc

ISO/IEC 14443-4 Transmission protocol

17N1689T.pdf 17n1689c.doc

ISO/IEC 15693, Vicinity cards

ISO/IEC 15693-1 Physical characteristics

17n1355t.doc 17n1355b .doc

ISO/IEC 15693-2 Air interface and initialisation

17n1486.pdf 17n1486c.doc

ISO/IEC 15693-3 Anticollision and transmission protocol

17n1692t.pdf 17n1692c.doc

ISO/IEC 10373-6, 10373-7, Test methods for the contactless integrated circuit(s) cards

ISO/IEC 10373-6 Proximity cards

17n1695t.pdf 17n1695c.doc

ISO/IEC 10373-6/AMD1 Amendment 1: Additional PICC test methods

17n2258t.pdf 17n2258t.doc 17n2258C.doc

ISO/IEC 10373-6/AMD2 Amendment 2: Improved RF test methods

17n2225t.pdf 17n2225F.doc

ISO/IEC 10373-7 Vicinity cards

17n1697t.pdf 17n1697c.doc

ISO/IEC 10536, Close-coupled cards

ISO/IEC 10536-1 Physical characteristics

17n1480t.PDF 17n1480c.doc
permalink
Shell

Welcome to scratch pad of usefull shell snippets



remove all php4 packages with apt-get

dpkg -l | grep php4 | awk '{ print $2 }' | xargs apt-get remove -y

use socat as simple serial program

socat - /dev/ttyS0,b9600,echo=0,raw

list first ten files from srchives

ls *.tar.gz *.tgz | xargs -i sh -c "ls -hs '{}' ; echo ; tar tvfz '{}' | head -10 ; echo"

ls *.tar.bz2 *.tbz | xargs -i sh -c "ls -hs '{}' ; echo ; tar tvfj '{}' | head -10 ; echo"

Yes, it sucks that tar doesn't know how to handle compression formats automatically...

tail recent logs and put output in file

Useful to take a quick view into last lines, reload page (and capture newly created errors/infos) and store that in single file for easy review (in vim :-)

tail -f /var/log/apache2/*.log | tee ~/log

port forwarder using socat

socat -d -d TCP4-LISTEN:1280,fork TCP4:192.168.212.5:80

remove all filesystem cache

echo 3 > /proc/sys/vm/drop_caches
permalink
DNS-323

D-Link DNS-323 network attached storage



Overview

dmesg.txt

dlink-DFDADE:~# uname -a
Linux dlink-DFDADE 2.6.12.6-arm1 #30 Mon Aug 18 14:19:14 CST 2008 armv5tejl GNU/Linux

dlink-DFDADE:~# free
            total      used      free    shared   buffers    cached
Mem:        61904     54808      7096         0     11824     30592
-/+ buffers/cache:     12392     49512
Swap:     1060208         0   1060208

dlink-DFDADE:~# cat /proc/cpuinfo 
Processor      : ARM926EJ-Sid(wb) rev 0 (v5l)
BogoMIPS       : 331.77
Features       : swp half thumb fastmult edsp java 
CPU implementer : 0x41
CPU architecture: 5TEJ
CPU variant    : 0x0
CPU part       : 0x926
CPU revision   : 0
Cache type     : write-back
Cache clean    : cp15 c7 ops
Cache lockdown : format C
Cache format   : Harvard
I size         : 32768
I assoc        : 1
I line length  : 32
I sets         : 1024
D size         : 32768
D assoc        : 1
D line length  : 32
D sets         : 1024

Hardware       : MV-88fxx81
Revision       : 0000
Serial         : 0000000000000000

RAID

Personalities : [linear] [raid0] [raid1] 
md0 : active raid0 sda2[0] sdb2[1]
      2925532672 blocks 64k chunks
      
unused devices: <none>
dlink-DFDADE:~# hdparm -tT /dev/sda /dev/sdb /dev/md0

/dev/sda:
 Timing cached reads:   210 MB in  2.01 seconds = 104.48 MB/sec
 Timing buffered disk reads:   72 MB in  3.00 seconds =  24.00 MB/sec

/dev/sdb:
 Timing cached reads:   212 MB in  2.01 seconds = 105.47 MB/sec
 Timing buffered disk reads:  104 MB in  3.03 seconds =  34.32 MB/sec

/dev/md0:
 Timing cached reads:   208 MB in  2.01 seconds = 103.48 MB/sec
 BLKGETSIZE failed: File too large

dlink-DFDADE:~# dd_rescue /dev/sda /dev/null
dd_rescue: (info): ipos:   1044480.0k, opos:   1044480.0k, xferd:   1044480.0k
                   errs:      0, errxfer:         0.0k, succxfer:   1044480.0k
             +curr.rate:    37169kB/s, avg.rate:    32123kB/s, avg.load: 60.7%

dlink-DFDADE:~# dd_rescue /dev/md0 /dev/null
dd_rescue: (info): ipos:    729536.0k, opos:    729536.0k, xferd:    729536.0k
                   errs:      0, errxfer:         0.0k, succxfer:    729536.0k
             +curr.rate:    37176kB/s, avg.rate:    37502kB/s, avg.load: 76.5%

Debian

2.6.26-2-orion5x

raid0

Linux dlink-DFDADE 2.6.26-2-orion5x #1 Thu Aug 20 05:04:03 UTC 2009 armv5tel GNU/Linux
Personalities : [raid1] [raid6] [raid5] [raid4] [raid0] 
md1 : active raid0 dm-1[1] dm-0[0]
      2097024 blocks 64k chunks
      
md0 : active raid1 sda4[0] sdb4[1]
      513984 blocks [2/2] [UU]
      
unused devices: <none>

/dev/sda:
 Timing cached reads:   212 MB in  2.01 seconds = 105.39 MB/sec
 Timing buffered disk reads:   72 MB in  3.05 seconds =  23.58 MB/sec

/dev/sdb:
 Timing cached reads:   218 MB in  2.01 seconds = 108.26 MB/sec
 Timing buffered disk reads:   88 MB in  3.00 seconds =  29.33 MB/sec

/dev/md1:
 Timing cached reads:   208 MB in  2.01 seconds = 103.46 MB/sec
 Timing buffered disk reads:   60 MB in  3.07 seconds =  19.57 MB/sec

Linux dlink-DFDADE 2.6.26-2-orion5x #1 Thu Aug 20 05:04:03 UTC 2009 armv5tel GNU/Linux
Personalities : [raid1] [raid6] [raid5] [raid4] [raid0]
md1 : active (auto-read-only) raid1 dm-1[1] dm-0[0]
      1048512 blocks [2/2] [UU]
        resync=PENDING

md0 : active raid1 sda4[0] sdb4[1]
      513984 blocks [2/2] [UU]

unused devices: <none>

/dev/sda:
 Timing cached reads:   220 MB in  2.00 seconds = 109.73 MB/sec
 Timing buffered disk reads:   82 MB in  3.01 seconds =  27.22 MB/sec

/dev/sdb:
 Timing cached reads:   216 MB in  2.02 seconds = 107.12 MB/sec
 Timing buffered disk reads:   90 MB in  3.09 seconds =  29.14 MB/sec

/dev/md1:
 Timing cached reads:   208 MB in  2.02 seconds = 103.19 MB/sec
 Timing buffered disk reads:   52 MB in  3.05 seconds =  17.06 MB/sec

2.6.30

http://www.cyrius.com/journal/debian/orion/d-link/dns-323/dns-323-fan-control

dlink-DFDADE:~/mdadm# ./test.sh 
+ ./remove.md1.sh
+ mdadm --manage --stop /dev/md1
mdadm: error opening /dev/md1: No such file or directory
+ mdadm --manage --remove /dev/md1
mdadm: error opening /dev/md1: No such file or directory
+ yes
+ mdadm --create --verbose /dev/md1 --level=0 --raid-devices=2 --force /dev/vga/raid.a /dev/vgb/raid.b
mdadm: chunk size defaults to 64K
mdadm: /dev/vga/raid.a appears to be part of a raid array:
    level=raid1 devices=2 ctime=Sat Sep  5 14:54:52 2009
mdadm: /dev/vgb/raid.b appears to be part of a raid array:
    level=raid1 devices=2 ctime=Sat Sep  5 14:54:52 2009
Continue creating array? mdadm: array /dev/md1 started.
+ ./hdparm-test.sh
+ test -d out
+ uname -a
+ cat /proc/mdstat
+ hdparm -tT /dev/sda /dev/sdb /dev/md1
++ date +%Y%m%d_%H%M%S
+ tee out/20090905_151032
Linux dlink-DFDADE 2.6.30-1-orion5x #1 Tue Aug 18 04:19:30 UTC 2009 armv5tel GNU/Linux
Personalities : [raid1] [raid0] 
md1 : active raid0 dm-0[1] dm-1[0]
      2097024 blocks 64k chunks
      
md0 : active raid1 sda4[0] sdb4[1]
      513984 blocks [2/2] [UU]
      
unused devices: <none>

/dev/sda:
 Timing cached reads:   246 MB in  2.01 seconds = 122.14 MB/sec
 Timing buffered disk reads:  132 MB in  3.00 seconds =  43.93 MB/sec

/dev/sdb:
 Timing cached reads:   242 MB in  2.01 seconds = 120.27 MB/sec
 Timing buffered disk reads:  138 MB in  3.01 seconds =  45.87 MB/sec

/dev/md1:
 Timing cached reads:   234 MB in  2.01 seconds = 116.15 MB/sec
 Timing buffered disk reads:  130 MB in  3.03 seconds =  42.85 MB/sec

+ ./remove.md1.sh
+ mdadm --manage --stop /dev/md1
mdadm: stopped /dev/md1
+ mdadm --manage --remove /dev/md1
+ yes
+ mdadm --create --verbose /dev/md1 --level=1 --raid-devices=2 --spare-devices=0 --force /dev/vga/raid.a /dev/vgb/raid.b
mdadm: /dev/vga/raid.a appears to be part of a raid array:
    level=raid0 devices=2 ctime=Sat Sep  5 15:10:31 2009
mdadm: /dev/vgb/raid.b appears to be part of a raid array:
    level=raid0 devices=2 ctime=Sat Sep  5 15:10:31 2009
mdadm: size set to 1048512K
Continue creating array? mdadm: array /dev/md1 started.
+ ./hdparm-test.sh
+ test -d out
+ uname -a
++ date +%Y%m%d_%H%M%S
+ cat /proc/mdstat
+ hdparm -tT /dev/sda /dev/sdb /dev/md1
+ tee out/20090905_151114
Linux dlink-DFDADE 2.6.30-1-orion5x #1 Tue Aug 18 04:19:30 UTC 2009 armv5tel GNU/Linux
Personalities : [raid1] [raid0] 
md1 : active (auto-read-only) raid1 dm-0[1] dm-1[0]
      1048512 blocks [2/2] [UU]
        resync=PENDING
      
md0 : active raid1 sda4[0] sdb4[1]
      513984 blocks [2/2] [UU]
      
unused devices: <none>

/dev/sda:
 Timing cached reads:   116 MB in  2.01 seconds =  57.84 MB/sec
 Timing buffered disk reads:  118 MB in  3.00 seconds =  39.32 MB/sec

/dev/sdb:
 Timing cached reads:   114 MB in  2.01 seconds =  56.79 MB/sec
 Timing buffered disk reads:  140 MB in  3.03 seconds =  46.27 MB/sec

/dev/md1:
 Timing cached reads:   234 MB in  2.02 seconds = 116.07 MB/sec
 Timing buffered disk reads:  110 MB in  3.01 seconds =  36.49 MB/sec

updates

  • Research on FOSS foundations

    I worked on research on FOSS foundations and published two reports:

    Growing Open Source Projects with a Stable Foundation

    This primer covers non-technical aspects that the majority of projects will have to consider at some point. It also explains how FOSS foundations can help projects grow and succeed.

    This primer explains:

    • What issues and areas to consider
    • How other projects and foundations have approached these topics
    • What FOSS foundations bring to the table
    • How to choose a FOSS foundation

    You can download Growing Open Source Projects with a Stable Foundation.

    Research report

    The research report describes the findings of the research and aims to help understand the operations and challenges FOSS foundations face.

    This report covers topics such as:

    • Role and activities of foundations
    • Challenges faced and gaps in the service offerings
    • Operational aspects, including reasons for starting an org and choice of jurisdiction
    • Trends, such as the "foundation in a foundation" model
    • Recommendations for different stakeholders

    You can download the research report.

    Acknowledgments

    This research was sponsored by Ford Foundation and Alfred P. Sloan Foundation. The research was part of their Critical Digital Infrastructure Research initiative, which investigates the role of open source in digital infrastructure.

  • ledger2beancount 2.6 released

    I released version 2.6 of ledger2beancount, a ledger to beancount converter.

    Here are the changes in 2.6:

    • Round calculated total if needed for price==cost comparison
    • Add narration_tag config variable to set narration from metadata
    • Retain unconsummated payee/payer metadata
    • Ensure UTF-8 output and assume UTF-8 input
    • Document UTF-8 issue on Windows systems
    • Add option to move posting-level tags to the transaction itself
    • Add support for the alias sub-directive of account declarations
    • Add support for the payee sub-directive of account declarations
    • Support configuration file called .ledger2beancount.yaml
    • Fix uninitialised value warning in hledger mode
    • Print warning if account in assertion has sub-accounts
    • Set commodity for commodity-less balance assertion
    • Expand path name of beancount_header config variable
    • Document handling of buckets
    • Document pre- and post-processing examples
    • Add Dockerfile to create Docker image

    Thanks to Alexander Baier, Daniele Nicolodi, and GitHub users bratekarate, faaafo and mefromthepast for various bug reports and other input.

    Thanks to Dennis Lee for adding a Dockerfile and to Vinod Kurup for fixing a bug.

    Thanks to Stefano Zacchiroli for testing.

    You can get ledger2beancount from GitHub.

  • beancount2ledger 1.3 released

    I released version 1.3 of beancount2ledger, the beancount to ledger converter that was moved from bean-report ledger into a standalone tool.

    You can get beancount2ledger from GitHub or via pip install.

    Here are the changes in 1.3:

    • Add rounding postings only when required (issue #9)
    • Avoid printing too much precision for a currency (issue #21)
    • Avoid creating two or more postings with null amount (issue #23)
    • Add price to cost when needed by ledger (issue #22)
    • Preserve posting order (issue #18)
    • Add config option indent
    • Show metadata with hledger output
    • Support setting auxiliary dates and posting dates from metadata (issue #14)
    • Support setting the code of transactions from metadata
    • Support mapping of account and currency names (issue #24)
    • Improve documentation:
      • Add user guide
      • Document limitations (issue #12)

  • ledger2beancount 2.5 released

    I released version 2.5 of ledger2beancount, a ledger to beancount converter.

    Here are the changes in 2.5:

    • Don't create negative cost for lot without cost
    • Support complex implicit conversions
    • Handle typed metadata with value 0 correctly
    • Set per-unit instead of total cost when cost is missing from lot
    • Support commodity-less amounts
    • Convert transactions with no amounts or only 0 amounts to notes
    • Fix parsing of transaction notes
    • Keep tags in transaction notes on same line as transaction header
    • Add beancount config options for non-standard root names automatically
    • Fix conversion of fixated prices to costs
    • Fix removal of price when price==cost but when they use different number formats
    • Fix removal of price when price==cost but per-unit and total notation mixed
    • Fix detection of tags and metadata after posting/aux date
    • Use D directive to set default commodity for hledger
    • Improve support for postings with commodity-less amounts
    • Allow empty comments
    • Preserve leading whitespace in comments in postings and transaction headers
    • Preserve indentation for tags and metadata
    • Preserve whitespace between amount and comment
    • Refactor code to use more data structures
    • Remove dependency on Config::Onion module

    Thanks to input from Remco Rijnders, Yuri Khan, and Thierry. Thanks to Stefano Zacchiroli and Kirill Goncharov for testing my changes.

    You can get ledger2beancount from GitHub

  • ledger2beancount 2.4 released

    I released version 2.4 of ledger2beancount, a ledger to beancount converter.

    There are two notable changes in this release:

    1. I fixed two regressions introduced in the last release. Sorry about the breakage!
    2. I improved support for hledger. I believe all syntax differences in hledger are supported now.

    Here are the changes in 2.4:

    • Fix regressions introduced in version 2.3
      • Handle price directives with comments
      • Don't assume implicit conversion when price is on second posting
    • Improve support for hledger
      • Fix parsing of hledger tags
      • Support commas as decimal markers
      • Support digit group marks through commodity and D directives
      • Support end aliases directive
      • Support regex aliases
      • Recognise total balance assertions
      • Recognise sub-account balance assertions
    • Add support for define directive
    • Convert all uppercase metadata tags to all lowercase
    • Improve handling of ledger lots without cost
    • Allow transactions without postings
    • Fix parsing issue in commodity declarations
    • Support commodities that contain quotation marks
    • Add --version option to show version
    • Document problem of mixing apply and include

    Thanks to Kirill Goncharov for pointing out one regressions, to Taylor R Campbell for for a patch, to Stefano Zacchiroli for some input, and finally to Simon Michael for input on hledger!

    You can get ledger2beancount from GitHub

  • beancount2ledger 1.1 released

    Martin Blais recently announced that he'd like to re-organize the beancount code and split out some functionality into separate projects, including the beancount to ledger/hledger conversion code previously provided by bean-report.

    I agreed to take on the maintenance of this code and I've now released beancount2ledger, a beancount to ledger/hledger converter.

    You can install beancount2ledger with pip:

    pip3 install beancount2ledger
    

    Please report issues to the GitHub tracker.

    There are a number of outstanding issues I'll fix soon, but please report any other issues you encounter.

    Note that I'm not very familiar with hledger. I intend to sync up with hledger author Simon Michael soon, but please file an issue if you notice any problems with the hledger conversion.

    Version 1.1 contains a number of fixes compared to the latest code in bean-report:

    1.1 (2020-07-24)

    • Preserve metadata information (issue #3)
    • Preserve cost information (lot dates and lot labels/notes) (issue #5)
    • Avoid adding two prices in hledger (issue #2)
    • Avoid trailing whitespace in account open declarations (issue #6)
    • Fix indentation issue in postings (issue #8)
    • Fix indentation issue in price entries
    • Drop time information from price (P) entries
    • Add documentation
    • Relicense under GPL-2.0-or-later (issue #1)

    1.0 (2020-07-22)

    • Split ledger and hledger conversion from bean-report into a standalone tool
    • Add man page for beancount2ledger(1)

  • ledger2beancount 2.3 released

    I released version 2.3 of ledger2beancount, a ledger to beancount converter.

    There are three notable changes with this release:

    1. Performance has significantly improved. One large, real-world test case has gone from around 160 seconds to 33 seconds. A smaller test case has gone from 11 seconds to ~3.5 seconds.
    2. The documentation is available online now (via Read the Docs).
    3. The repository has moved to the beancount GitHub organization.

    Here are the changes in 2.3:

    • Improve speed of ledger2beancount significantly
    • Improve parsing of postings for accuracy and speed
    • Improve support for inline math
    • Handle lots without cost
    • Fix parsing of lot notes followed by a virtual price
    • Add support for lot value expressions
    • Make parsing of numbers more strict
    • Fix behaviour of dates without year
    • Accept default ledger date formats without configuration
    • Fix implicit conversions with negative prices
    • Convert implicit conversions in a more idiomatic way
    • Avoid introducing trailing whitespace with hledger input
    • Fix loading of config file
    • Skip ledger directive import
    • Convert documentation to mkdocs

    Thanks to Colin Dean for some feedback. Thanks to Stefano Zacchiroli for prompting me into investigating performance issues (and thanks to the developers of the Devel::NYTProf profiler).

    You can get ledger2beancount from GitHub

  • ledger2beancount 2.2 released

    I released version 2.2 of ledger2beancount, a ledger to beancount converter.

    Here are the changes in 2.2:

    • Show warning for unknown apply directive
    • Recognize apply rate directive (an alias of apply fixed)
    • Don't convert meta-data on ignored virtual postings but keep as comments
    • Update location of beancount repository

    You can get ledger2beancount from GitHub.

    Thanks to GitHub user MarinBernard for reporting a bug with virtual postings!

  • ledger2beancount 2.1 released

    I released version 2.1 of ledger2beancount, a ledger to beancount converter.

    Here are the changes in 2.1:

    • Handle postings with posting dates and comments but no amount
    • Show transactions with only one posting (without bucket)
    • Adding spacing between automatic declarations
    • Preserve preliminary info at the top

    You can get ledger2beancount from GitHub.

    Thanks to Thierry (thdox) for reporting a bug and for fixing some typos in the documentation. Thanks to Stefano Zacchiroli for some good feedback.

  • ledger2beancount 2.0 released

    I released version 2.0 of ledger2beancount, a ledger to beancount converter.

    Here are the changes in 2.0:

    • Handle comments in account and commodity declarations
    • Handle transactions with a single posting (without bucket)
    • Handle empty metadata values
    • Rewrite Emacs modeline

    You can get ledger2beancount from GitHub.


permalink
SocialText Open Wiki Install

Here are some quick notes about installation on clean Debian etch machine

This page is aimed towards system administrators, if you want information how to customize design of page take a look at SocialText CSS design (which is included on bottom of this page).

Contents: [Dobrica Pavlinušić's random unstructured stuff]


Helper packages

sudo apt-get install wv xpdf-utils unrtf unzip links
sudo cpan MP3::Tag

Install source

svn co https://repo.socialtext.net:8999/svn/socialtext/trunk socialtext
cd socialtext/nlw

# checkout lastest Socialtext::Resting client
svn export --force https://repo.socialtext.net:8999/svn/socialtext-clients/trunk/rest/perl/Socialtext-Resting/lib/Socialtext/ lib/Socialtext/

./configure --socialtext-open 1 --apache-proxy 0 --force-ssl-login 1
make
sudo bash
make install
update-rc.d -f apache-perl remove
update-rc.d st-apache defaults
update-rc.d ceqlotron defaults
/etc/init.d/st-apache start
/etc/init.d/ceqlotron start

Update existing checkout

cd socaialtext
svn update
cd nlw
./configure --socialtext-open 1 --apache-proxy 0 --force-ssl-login 1
make
make test
sudo make install
sudo /etc/init.d/st-apache stop
sudo /etc/init.d/ceqlotron restart
sudo /etc/init.d/st-apache start

Add new admin user

sudo -u www-data st-admin create-user --email dpavlin@rot13.org --password wikidream \
  --first-name Dobrica --last-name Pavlinusic
sudo -u www-data st-admin add-workspace-admin --email dpavlin@rot13.org

Configuring mailer

postfix

Install and configure postfix for mail delivery

apt-get install postfix
vi /etc/postfix/main

Important lines are following:

alias_maps = hash:/etc/aliases, hash:/etc/aliases.deliver
alias_database = hash:/etc/aliases, hash:/etc/aliases.deliver

They add /etc/aliases.deliver (which SocialText generates with names of wiki workspaces) to configuration.

Next, we need to create /etc/aliases.deliver.db which is binary version of database needed for postfix. It's extremly important that aliases.deliver file is owned by www-data because postfix picks user which will do e-mail delivery based on owner of aliases file. If your owner of /var/www/socialtext is some other user than www-data you will have to modify following command appropriately:

# make www-data owner of file
chown www-data:root /etc/aliases.deliver
# allow root group to write in /etc/
chmod g+w /etc/
# create /etc/aliases.deliver.db
sudo postalias /etc/aliases.deliver

Now restart postfix as root user

/etc/init.d/postfix restart

crontab

Add following to /etc/crontab to refresh aliases database for postfix regularly:

*/10 *  * * *   root    postalias /etc/aliases.deliver

Handling e-mail errors

sudo tail /var/log/mail.log | grep st-admin

dsn=5.3.0, status=bounced (Command died with status 13: "/usr/local/bin/st-admin deliver-email --workspace bio". Command output: open nlw-stats.log: Permission denied at /usr/local/share/perl/5.8.8/Socialtext/Statistics.pm line 103. END failed--call queue aborted. )

This problem is reported on socialtext-devel list, but no response so far.

# sudo vi +24 /usr/local/share/perl/5.8.8/Socialtext/Statistics.pm
Readonly my $LOG_NAME => 'nlw-stats.log';
Readonly my $LOG_PATH => $ENV{APACHE_LOG_DIR}
    ? "$ENV{APACHE_LOG_DIR}/$LOG_NAME"       
    : "/tmp/$LOG_NAME";

Customize template

Remove "Did you know?" box from dashboard

sudo vi /usr/share/nlw/template/view/homepage

Or from your socialtext/nlw checkout (so that make install will install correct one):

vi share/template/view/homepage

Fix RSS feed errors

If you are using {fetchrss http://blog.rot13.org/index.xml} and having problems with RSS feeds, try to re-installing two perl modules:

sudo cpan XML::RSS XML::Atom

Change e-mail address of workspace

This configuration change will feed responses to e-mails generated by wiki back into wiki. You might not want to do that! :-)

st-admin set-workspace-config --workspace rot13 \
  email_notification_from_address '"random unstructured stuff" <rot13@wiki.rot13.org>'

Access simple version of page

Every SocialText wiki has a lite version available at http://wiki.example.com/lite/page/:workspace/

For this wiki it's http://saturn.ffzg.hr/lite/page/rot13

Configure sane resource usage

Since SocialText wiki is designed to run on dedicated box, it's resource requiements are not for faint-hearted users. However, if you are installing it on machine with constraint hardware (or virtual machine) you might want to decrease number of apache-perl server which are created on startup and kept running. Each apache child takes about 80Mb of RAM, so lowering number of children will have beneficial effect on your memory usage.

If you open /etc/apache-perl/st-httpd.conf you will see comment that describes command used for generating this configuration. So, let's edit command limits, re-create config and restart apache:

vi +485 dev-bin/gen-config
dev-bin/gen-config --sitewide --apache-proxy=0 --hostname=saturn.ffzg.hr
/etc/init.d/st-apache reload

Convert RSS feed URLs to absolute

If you need to fetch RSS feeds from remote URLs and those providers create only relative URLs in feeds (like svnweb does) that this patch might help you: FetchRSSPlugin.pm.diff

It does dirty magic to prepend hostname before URL in feeds making them work from Socialtext wiki which is on different host.

Problems with 8-bit UTF-8 encoded characters in URLs

It seems that trunk of socialtext

Repository Root: https://repo.socialtext.net:8999/svn/socialtext
Repository UUID: 593ce9bc-2028-0410-bb98-c566b643a948
Revision: 6127
Node Kind: directory
Schedule: normal
Last Changed Author: jjp
Last Changed Rev: 6028
Last Changed Date: 2007-12-10 19:38:21 +0100 (Mon, 10 Dec 2007)

has problems with CGI.pm 3.31. As a temporary work-around use backpan.sh to download CGI.pm 3.29 which is known to work.

Enable html wafl in workspace

This enables .html markup which enables embedded html into wiki page

st-admin set-workspace-config --workspace razmjenavjestina allows_html_wafl 1

Upgrade to 2.2

There are various problems with this upgrade.

Socialtext::Resting

For a start, you will have to install Socialtext::Resting clinet in the middle of make install because import of help workspace will fail.

You can get it from https://repo.socialtext.net:8999/svn/socialtext-clients/trunk/rest/perl/Socialtext-Resting

Missing directory

mkdir /etc/apache-perl/socialtext-conf.d

Missing skins

http://code.google.com/p/socialtext-skins/wiki/SkinningIssues



Problem not yet solved.


This page collect various parts which used to be in Socialtext Open Wiki Install, but since they are more oriented towards wiki admins and designers, it's here as separate page.



New skin (CSS design) creation

cd /usr/share/nlw/css/
cp -r st rot13
st-admin set-workspace-config --workspace rot13 skin_name rot13
vi rot13/screen.css

See also CSS kit at SocialText Open Wiki

Keep local changes in SVN

It's important to save local modifications, because next make install will overwrite your changes in /usr/share/nlw. One possibility is to do overlay svn checout over /usr/share/nlw and than svn add and svn commit every change:

cd /usr/share
svn co file:///data/st-custom/share nlw.svn
cd nlw.svn
find . | cpio -pdu ../nlw/
cd - && rm -Rf nlw.svn
cd nlw
svn info

permalink
https using stunnel

Quick guide based on stunnel for mysql - server and client to add https to existing http server on demand.



Installation

apt-get install stunnel4

Create certificate

cd /etc/stunnel
openssl req -x509 -nodes -days 365 -newkey rsa:1024 -keyout https.pem -out https.pem

Configuration file

/etc/stunnel/httpd.conf

cert = /etc/stunnel/https.pem
sslVersion = SSLv3
debug = 7

[https]
accept  = 443
connect = 80
TIMEOUTclose = 0

Start tunnel

stunnel4 /etc/stunnel/https.conf
permalink
Cisco VPN

Use vpnc instead of Cisco VPN client



Source

svn co http://svn.unix-ag.uni-kl.de/vpnc/trunk vpnc
cd vpnc
make
sudo make install

Support template

pcf2vpnc Support_template.pcf > /etc/vpnc/foobar.conf

Hash password security, eh?

Start connection

sudo /usr/local/sbin/vpnc --no-detach --debug 2 foobar

Fix routes

I didn't like that all my traffic was going over VPN

sudo route del default
route add default gw 192.168.1.1
route add -net 192.168.4.0 netmask 255.255.255.0 dev tun0
permalink
Weblog Navigation
Loading...
Weblog Archives
  • Loading...