2012年1月30日 星期一

Semantic Patch (spatch)/generic patch (spdiff)

Backporting the Linux kernel, for good - was: Re: semantic patch inference [LWN.net]
http://lwn.net/Articles/458918/

Backporting the Linux kernel, for good / Linux Plumbers Conference: Developing the Kernel, Libraries and Utilities / September 7-9, 2011 / Santa Rosa, CA
http://www.linuxplumbersconf.org/2011/ocw/proposals/771
http://www.linuxplumbersconf.org/2011/ocw//system/presentations/771/original/kernel-backport-for-good.odp



spatch
man - spatch (1) - apply a semantic patch file to a set of C files
http://www.digipedia.pl/man/doc/view/spatch.1/

Coccinelle: A Program Matching and Transformation Tool for Systems Code
http://coccinelle.lip6.fr/
apt-get install ocaml-findlib python2.6-dev #install the dev package of your python version
make all http://www.blogger.com/img/blank.gif#or, make world


Semantic Patches for specifying and automating Collateral Evolutions
http://coccinelle.lip6.fr/papers/semantic-patches-talk-ols07.odp
http://www.diku.dk/hjemmesider/ansatte/julia/eurosys-padioleauy.pdf

Semantic Patches
http://coccinelle.lip6.fr/papers/ols07-padioleau.pdf

Pet Peaves about Platform code, and arch_reset
http://lists.infradead.org/pipermail/linux-arm-kernel/2011-November/072104.html
Example of using spatch by Tony Lindgren


patchparse (neither source nor binary found)

Understanding Collateral Evolution in Linux Device Drivers
http://coccinelle.lip6.fr/papers/RR-5769.pdf
http://www.diku.dk/hjemmesider/ansatte/julia/eurosys-padioleauy.pdf

Programs used to produce the figures in the paper: Understanding Collateral Evolution in Linux Device Drivers
patch analyzer (patchparse)/interface analyzer
http://coccinelle.lip6.fr/patch_analyzer_and_co.html


spdiff

Generic Patch Inference
http://www.diku.dk/hjemmesider/ansatte/julia/andersen_ase08.pdf

spdiff -- A tool for Semantic Patch Inference
http://www.diku.dk/hjemmesider/ansatte/jespera/doc.html
apt-get install ocaml ocaml-native-compilers menhir
make depend
make opt

(Ubuntu 10.04)
Error: make: *** No rule to make target `genericparser.cmo', needed by `spdiff'. Stop.
#345792 - xml-light_2.2-1: FTBFS: No rule to make target `xml_parser.cmi' - Debian Bug report logs
http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=345792

Error: ocamlyacc genericparser.mly
File "genericparser.mly", line 31: syntax error

spdiff pre-built binary
http://www.diku.dk/hjemmesider/ansatte/jespera/downloads/spdiff.opt

Stripping kernel module

Re: Can modules be stripped?
http://www.linux-mips.org/archives/linux-mips/2002-04/msg00031.html

strip_module.

#!/bin/sh
#
# Given a list of objects, strip all static symbols except those
# required by insmod.
#
# Copyright Keith Owens . GPL.
# Sat Feb 1 12:52:17 EST 1997
#
# Mainly intended for reducing the size of modules to save space
# on emergency and install disks. Be aware that removing the
# static symbols reduces the amount of diagnostic information
# available for oops. Not recommended for normal module usage.
#
# This code requires the modules use MODULE_PARM and EXPORT_.
# Do not strip modules that have not been converted to use
# MODULE_PARM or are using the old method of exporting symbols.
# In particular do not use on modules prior to 2.1.20 (approx).
#
# The objects are stripped in /tmp, only if the strip works is
# the original overwritten. If the command line to strip the
# symbols becomes too long, the strip is done in multiple passes.
# Running strip_module twice on the same object is safe (and a
# waste of time).
#

cat > /tmp/$$.awk <<\EOF
BEGIN {
strip = "/usr/bin/objcopy";
nm = "/usr/bin/nm";
cp = "/bin/cp";
mv = "/bin/mv";
rm = "/bin/rm";
tmp = "/tmp";
command_size = 400; # arbitrary but safe

getline < "/proc/self/stat";
pid = $1;
tmpcopy = tmp "/" pid ".object";
nmout = tmp "/" pid ".nmout";

for (i = 1; i < ARGC; ++i)
strip_module(ARGV[i]);

do_command(rm " -f " tmpcopy " " nmout);

exit(0);
}

function strip_module(object,
keep_symbol, to_strip, symbol, command, changed) {
do_command(cp " -a " object " " tmpcopy);
do_command(nm " " tmpcopy " > " nmout);
# delete array_name sometimes breaks, internal error, play safe
for (symbol in keep_symbol)
delete keep_symbol[symbol];
for (symbol in to_strip)
delete to_strip[symbol];
new_module_format = 0;
while ((getline < nmout) > 0) {
$0 = substr($0, 10);
# b static variable, uninitialised
# d static variable, initialised
# r static array, initialised
# t static label/procedures
if ($1 ~ /[bdrt]/)
to_strip[$2] = "";
else if ($1 != "?")
keep_symbol[$2] = "";
else if ($0 ~ /\? __ksymtab_/)
keep_symbol[substr($2, 11)] = "";
else if ($0 ~ /\? __module_parm_/)
keep_symbol[substr($2, 15)] = "";
if ($0 ~ /\? __module/)
new_module_format = 1;
}
close(nmout);
command = "";
changed = 0;
if (new_module_format) {
for (symbol in to_strip) {
if (!(symbol in keep_symbol)) {
changed = 1;
if (length(command) > command_size) {
do_command(strip command " " tmpcopy);
command = "";
}
command = command " --strip-symbol=" symbol;
}
}
}
if (command != "") {
changed = 1;
do_command(strip command " " tmpcopy);
}
if (changed)
do_command(mv " " tmpcopy " " object);
}

function do_command(command) {
if ((ret = system(command)) != 0)
giveup("command \"" command "\" failed " ret, ret);
}

function giveup(message, ret) {
print "strip_module: " message > "/dev/stderr";
exit(ret);
}
EOF

awk -f /tmp/$$.awk "$@"
ret=$?
rm -f /tmp/$$.awk
exit $ret



How much strip(1)-ing is okay for kernel modules?
http://unix.stackexchange.com/questions/25421/how-much-strip1-ing-is-okay-for-kernel-modules
strip --strip-unneeded
make INSTALL_MOD_STRIP=1 modules_install.
strip --strip-debug *.ko (= strip -g *.ko)

2012年1月19日 星期四

Cavium Multicore Programming

Cavium University Program
http://university.cavium.com/

Cavium University Program - Download Center
http://university.cavium.com/download_center.html

OpenFlow - Enabling Innovation in Your Network
http://www.openflow.org/

cnusers.org - Welcome to cnusers.org
http://www.cnusers.org/
Octeon SDK v2.2 from cnusers.org
http://www.cnusers.org/index.php?option=com_remository&Itemid=32&func=select&id=32

2012年1月9日 星期一

Open Source CPU -- OpenRISC

Linux 3.1 - Linux Kernel Newbies
http://kernelnewbies.org/Linux_3.1#head-37c60fa1253db74ce7d224718a71f5836bd5be09

OpenRISC is an Open Source CPU from the OpenCores project that brings to the world of hardware all the same advantages that Open Source software has known for so long. The aim of the project is to create free and open source computing platforms available under the GNU (L)GPL license, and a set of free, open source implementations of the architecture and open source software development tools, libraries, operating systems and applications. The implementation merged in this release is the 32-bit OpenRISC 1000 family (or1k).

OR1200 OpenRISC Processor - OR1K :: OpenCores
http://opencores.org/or1k/OR1200_OpenRISC_Processor

2012年1月8日 星期日

VirtualBox can't operate in VMX root mode

modprobe -r kvm_intel

virtualbox.org • View topic - VirtualBox can't operate in VMX root mode.
https://forums.virtualbox.org/viewtopic.php?t=7796

svnversion: Getting the Revision Number of your Subversion (SVN) Working Copy

svnversion .

Getting the Revision Number of your Subversion Working Copy
http://www.zorched.net/2006/07/20/getting-the-revision-number-of-your-subversion-working-copy/

2012年1月5日 星期四

git-filter-branch: Checklist For Shrinking A Repository

git-filter-branch(1): Rewrite branches - Linux man page
http://linux.die.net/man/1/git-filter-branch

Checklist For Shrinking A Repository

git-filter-branch is often used to get rid of a subset of files, usually with some combination of --index-filter and --subdirectory-filter. People expect the resulting repository to be smaller than the original, but you need a few more steps to actually make it smaller, because git tries hard not to lose your objects until you tell it to. First make sure that:
  • You really removed all variants of a filename, if a blob was moved over its lifetime. git log --name-only --follow --all -- filename can help you find renames.
  • You really filtered all refs: use --tag-name-filter cat -- --all when calling git-filter-branch.
Then there are two ways to get a smaller repository. A safer way is to clone, that keeps your original intact.
  • Clone it with git clone file:///path/to/repo. The clone will not have the removed objects. See git-clone(1). (Note that cloning with a plain path just hardlinks everything!)
If you really don't want to clone it, for whatever reasons, check the following points instead (in this order). This is a very destructive approach, so make a backup or go back to cloning it. You have been warned.
  • Remove the original refs backed up by git-filter-branch: say git for-each-ref --format="%(refname)" refs/original/ | xargs -n 1 git update-ref -d.
  • Expire all reflogs with git reflog expire --expire=now --all.
  • Garbage collect all unreferenced objects with git gc --prune=now (or if your git-gc is not new enough to support arguments to --prune, use git repack -ad; git prune instead).

2012年1月3日 星期二

[QEMU] Running Linux 2.6.33 ARM versatilepb

Compiling Linux kernel for QEMU ARM emulator
http://balau82.wordpress.com/2010/03/22/compiling-linux-kernel-for-qemu-arm-emulator/
versatilepb has 4 PrimeCell UART (PL011)(ttyAMA0-3), LCD tty0.

2.6.33 (2.6.34, failed at 2.6.35 and after), 2010q1
make ARCH=arm versatile_defconfig
make ARCH=arm menuconfig
If RAMdisk size might > 4MB, change CONFIG_BLK_DEV_RAM_SIZE to higher value.
Enable CONFIG_AEABI, disable CONFIG_MODULES. Build zImage
make ARCH=arm CROSS_COMPILE=arm-none-linux-gnueabi- all
Booting should be fine, but lack of rootfs.
qemu-system-arm -M versatilepb -m 128M -kernel zImage
Create a CPIO rootfs
The CPIO rootfs doesn't need /dev/console.
cat > test.c << eof
#include <stdio.h>
void main() {
printf("Hello World!\n");
while(1);
}
eof
arm-none-linux-gnueabi-gcc -static test.c -o test
echo test | cpio -o --format=newc > rootfs

CPIO: Both kernel and user space message goes to tty0 by default.
qemu-system-arm -M versatilepb -m 128M -kernel zImage -initrd rootfs -append "root=/dev/ram rdinit=/test"

CPIO: Both kernel and user space message goes to ttyAMA0
qemu-system-arm -M versatilepb -m 128M -kernel zImage -initrd rootf -append "root=/dev/ram rdinit=/test console=ttyAMA0" -nographic



ERROR: I had no issue booting zImage, but ran into following error while running userspace programs, no matter the "hello world" test or busybox, with my old CodeSoucery toolchain:
test (1): undefined instruction: pc=00017104
Code: e3a0e000 e1a0c00e e5d03000 e2433030 (e6ef2073)
After a lot of try-and-error, I finally found that the old CodeSoucery toolchain that I downloaded long times ago doesn't work, while the one downloaded recently does, WTF....
arm-2009q1-203-arm-none-linux-gnueabi-i686-pc-linux-gnu.tar.bz2
old downloaded md5sum: a88fb553950d1b47ecefa9929dea04a8
recently downloaded md5sum: 1a6e88782f08b09a0e6cef545a1712ec
It might be caused by the default ARM arch used by the compiler.

I use buildroot to build rootfs and lib, BR2_generic_arm must be set. Enabling BR2_arm1136jf_s would cause similar issue. It might be caused by unsupported instruction of new arch.


In RAM disk, if /dev/console is missing, both tty0 or ttyAMA0 will not see "Hello World". /dev/console can be created by
# mknod dev/console c 5 1


(TODO: start from busybox download?)

1. basic rootfs: /etc, /dev, ...
2. library: /lib (or build all as static)
3. busybox, enable CONFIG_STATIC if no library


By default, both kernel and user space message goes to tty0
qemu-system-arm -M versatilepb -m 128M -kernel zImage -append "root=/dev/ram0 rw init=/test user_debug=31" -initrd ramdisk.img.gz
qemu-system-arm -M versatilepb -m 128M -kernel zImage -append "root=/dev/ram0 rw init=/test user_debug=31 console=tty0" -initrd ramdisk.img.gz

Both kernel and user space message goes to ttyAMA0
qemu-system-arm -M versatilepb -m 128M -kernel zImage -append "root=/dev/ram0 rw init=/test user_debug=31 console=ttyAMA0 " -initrd ramdisk.img.gz -nographic

kernel message will goes to both tty0 and ttyAMA0, but user space message only goes to the ttyAMA0(the last one)
qemu-system-arm -M versatilepb -m 128M -kernel zImage -append "root=/dev/ram0 rw init=/test user_debug=31 console=tty0 console=ttyAMA0 " -initrd ramdisk.img.gz -nographic


Enable ethernet:
qemu-system-arm -M versatilepb -m 256M -kernel output/zImage -append "root=/dev/ram0 rw init=/linuxrc user_debug=31 console=ttyAMA0" --nographic -initrd output/ramdisk.img.gz -net nic,macaddr=00:00:00:00:02:00 -net tap,ifname=tap2,script=no


Eanble 2 ethernet result in Segmentation fault of QEMU:
qemu-system-arm -M versatilepb -m 256M -kernel output/zImage -append "root=/dev/ram0 rw init=/linuxrc user_debug=31 console=ttyAMA0" --nographic -initrd output/ramdisk.img.gz -net nic,macaddr=00:00:00:00:02:00 -net tap,ifname=tap0,script=no -net nic,macaddr=00:00:00:00:01:00 -net tap,ifname=tap1,script=no


Eanble PCI and E1000 result in emulated Linux panic:
qemu-system-arm -M versatilepb -m 256M -kernel output/zImage -append "root=/dev/ram0 rw init=/linuxrc user_debug=31 console=ttyAMA0" --nographic -initrd output/ramdisk.img.gz -net nic,macaddr=00:00:00:00:02:00 -net tap,ifname=tap0,script=no -net nic,model=e1000,macaddr=00:00:00:00:01:00 -net tap,ifname=tap1,script=no

Kernel panic:
Intel(R) PRO/1000 Network Driver - version 7.3.21-k5-NAPI
Copyright (c) 1999-2006 Intel Corporation.
PCI: enabling device 0000:00:0c.0 (0000 -> 0003)
Unable to handle kernel paging request at virtual address 44000400
(...............)
[<c01ab10c>] (e1000_io_write+0x0/0x8) from [<c01b4a6c>] (e1000_reset_hw+0xc0/0x1f4)
[<c01b4a6c>] (e1000_reset_hw+0xc0/0x1f4) from [<c028b9bc>] (e1000_probe+0x3d0/0xa98)
[<c028b9bc>] (e1000_probe+0x3d0/0xa98) from [<c0163b58>] (local_pci_probe+0x18/0x1c)
[<c0163b58>] (local_pci_probe+0x18/0x1c) from [<c01642fc>] (pci_device_probe+0x58/0x8c)
[<c01642fc>] (pci_device_probe+0x58/0x8c) from [<c0198f38>] (driver_probe_device+0xa8/0x15c)
[<c0198f38>] (driver_probe_device+0xa8/0x15c) from [<c019904c>] (__driver_attach+0x60/0x84)
[<c019904c>] (__driver_attach+0x60/0x84) from [<c0198800>] (bus_for_each_dev+0x44/0x74)
[<c0198800>] (bus_for_each_dev+0x44/0x74) from [<c0198168>] (bus_add_driver+0x9c/0x218)
[<c0198168>] (bus_add_driver+0x9c/0x218) from [<c0199320>] (driver_register+0xa8/0x134)
[<c0199320>] (driver_register+0xa8/0x134) from [<c0164528>] (__pci_register_driver+0x38/0xac)
[<c0164528>] (__pci_register_driver+0x38/0xac) from [<c001a040>] (e1000_init_module+0x34/0x88)
[<c001a040>] (e1000_init_module+0x34/0x88) from [<c002438c>] (do_one_initcall+0x5c/0x1bc)
[<c002438c>] (do_one_initcall+0x5c/0x1bc) from [<c0008574>] (kernel_init+0x90/0x144)
[<c0008574>] (kernel_init+0x90/0x144) from [<c0025e38>] (kernel_thread_exit+0x0/0x8)

Debian -- Debian Releases

Debian -- Debian Releases
http://www.debian.org/releases/


The "unstable" distribution is called sid.
(..........)
* The next release of Debian is codenamed "wheezy" — no release date has been set
* Debian 6.0 ("squeeze") — current stable release
* Debian GNU/Linux 5.0 ("lenny") — obsolete stable release
* Debian GNU/Linux 4.0 ("etch") — obsolete stable release
* Debian GNU/Linux 3.1 ("sarge") — obsolete stable release
* Debian GNU/Linux 3.0 ("woody") — obsolete stable release
* Debian GNU/Linux 2.2 ("potato") — obsolete stable release
* Debian GNU/Linux 2.1 ("slink") — obsolete stable release
* Debian GNU/Linux 2.0 ("hamm") — obsolete stable release