2016年9月10日 星期六

Make Ubuntu kernel

Edit debian.master/changelog, add some string to distinguish from the original one:

linux (4.5.7-040507.201606100436-test15+23e5a3b) unstable; urgency=low
Add execution permission to build scripts:
chmod a+x debian/rules; chmod a+x debian/scripts/*; chmod a+x debian/scripts/misc/*;
"make oldconfig"
debian/rules updateconfigs
debian/scripts/misc/oldconfig ARCH
"make menuconfig":
fakeroot debian/rules clean
fakeroot debian/rules editconfigs #
Build:
fakeroot debian/rules clean
fakeroot debian/rules binary-headers binary-generic
Build image with dbgsym:
fakeroot time debian/rules binary-headers binary-generic skipdbg=false
https://wiki.ubuntu.com/Kernel/BuildYourOwnKernel
https://wiki.ubuntu.com/KernelTeam/GitKernelBuild
https://wiki.ubuntu.com/Kernel/MainlineBuilds


Ubuntu prebuilt kernel:
https://launchpad.net/~kernel-ppa/+archive/ubuntu/ppa
https://launchpad.net/~canonical-kernel-team/+archive/ubuntu/ppa
http://kernel.ubuntu.com/~kernel-ppa/mainline/
http://kernel.ubuntu.com/~kernel-ppa/mainline/daily/current/




2016年9月8日 星期四

宜一荒野限定冷笑話/腦筋急轉彎

太陽爸和太陽媽,生了個太陽兒子,我們該說什麼祝賀詞恭喜他們? 生日快樂
那這樣總共有幾顆太陽呢?一顆,因為太陽他爸是漂流木,太陽他媽是母老虎,所以只有一顆陳太陽

https://youtu.be/MOI3etHEYRM


誰最會講笑話?酵母,因為會發酵
不可以對誰講笑話? 海浪,因為海浪笑了 會海嘯
(原始:在哪邊不可以講笑話?海邊,因為海笑(嘯)了很恐怖)

綿羊被剃毛後為什麼會睡不著呢?因為 失綿 了

黑人和白人結婚會生出什麼? 灰狼
小黃 小白 小黑 和 小紅 坐車誰會吐?  小白兔 和 小黃兔

2016年9月2日 星期五

Kernel Crash Dump

sudo apt install linux-crashdump

Verification
cat /proc/cmdline
dmesg | grep -i crash
kdump-config show

cat /proc/sys/kernel/sysrq

Active kdump
kdump-config load

Enabling various types of panics
To make Linux kernel to panic on different situations please use:

echo 1 > /proc/sys/kernel/hung_task_panic          # panic when hung task is detected
echo 1 > /proc/sys/kernel/panic_on_io_nmi          # panic on NMIs from I/O
echo 1 > /proc/sys/kernel/panic_on_oops            # panic on oops or kernel bug detection
echo 1 > /proc/sys/kernel/panic_on_unrecovered_nmi # panic on NMIs from memory or unknown
echo 1 > /proc/sys/kernel/softlockup_panic         # panic when soft lockups are detected
echo 1 > /proc/sys/vm/panic_on_oom                 # panic when out-of-memory happens

Test Crash kernel
sudo -s
echo c > /proc/sysrq-trigger

Get vmlinux with debug symbol
sudo tee /etc/apt/sources.list.d/ddebs.list << EOF
deb http://ddebs.ubuntu.com/ $(lsb_release -cs)          main restricted universe multiverse
deb http://ddebs.ubuntu.com/ $(lsb_release -cs)-security main restricted universe multiverse
deb http://ddebs.ubuntu.com/ $(lsb_release -cs)-updates  main restricted universe multiverse
deb http://ddebs.ubuntu.com/ $(lsb_release -cs)-proposed main restricted universe multiverse
EOF

sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys ECDCAD72428D7C01
sudo apt-get update
sudo apt-get install linux-image-$(uname -r)-dbgsym

Inspecting the crash dump using crash

Debug kernel: /usr/lib/debug/boot/
Crash dump: /var/crash/

crash <debug kernel> <crash dump>


Kernel/CrashdumpRecipe - Ubuntu Wikihttps://wiki.ubuntu.com/Kernel/CrashdumpRecipe

Kernel Crash Dump
https://help.ubuntu.com/lts/serverguide/kernel-crash-dump.html

2016年9月1日 星期四

Random abnormal high CPU sys usage related to timer

Test files and logs are available in GitHub
https://github.com/mkl0301/abnormal-cpu-load


20160910 update:
Up to now I still don't know clearly whether this is simply a display issue, or something is actually being executed.
  1. Even with the issue happening, the application seems works normally.
  2. But with previous perf report, the data sampling rate when the issue happen is higher than the rate when issue not happen.
I happened to found that on v3.13, enabling CONFIG_CONTEXT_TRACKING_FORCE can fix this issue. But unfortunately the kernel I'm using is 3.4, which doesn't have the context tracking feature added, so the hunting keeps on....

Forcing context tracking only works until 4.5, doing so on 4.6 and after still see the issue.



Random abnormal high CPU sys usage related to timer
https://lkml.org/lkml/2016/8/26/383

We were having issue with our userspace application which
__sometimes__ result in high CPU sys usage at each execution. The high
sys CPU usage persist until the application is killed.

We simplified the application to just creating a timer and its handler
then does nothing, but looping and sleeping for the timer to be
triggered. With top, the CPU that running the application usually
takes almost 0% for sys usage. But sometimes it will occupy certain
amount of sys usage, up to 100% at most of the time on my embedded
device.

On my laptop, Intel Core i5-4200U, running Ubuntu 14.02.2, Linux
3.13.0-45-generic and 4.4.0-34-generic, the issue can be reproduced
with lower sys usage (7~50%). The same can be reproduced with
buildroot+vanilla kernel 4.7 and 3.13.

Restart the application could temporarily fix the the issue, but there
are chances to happen again.

/proc/timer_stats, /proc/interrupts, and perf didn't show any abnormal value or useful clue.
Comparing the good and fail log got the following
  • The /proc/timer_stats is almost the same, but perf events shows extra softirq/timer events.
  • The perf sample of the failed case is much more than the good case, but the ratio of the sampled functions are basically the same.


One issue is found that seems related, but no further action.

Keystone II Linux: Random High CPU usage - userspace application using 1 full core - Linux forum - Linux - TI E2E Community
http://e2e.ti.com/support/embedded/linux/f/354/p/433791/1553204



The Documentation/cpu-load.txt discussed a situation that the cpu load reported by top might underestimated. The C code is very alike to ours. But it's discussing a different topic.

Documentation/cpu-load.txt
https://www.kernel.org/doc/Documentation/cpu-load.txt
https://lkml.org/lkml/2007/2/12/6




Why you should avoid using SIGALRM for timer – Linux, Embedded, Android and Security blog
https://nativeguru.wordpress.com/2015/02/19/why-you-should-avoid-using-sigalrm-for-timer/

Linux timer | 菜鳥的三年成長史
https://wirelessr.gitbooks.io/working-life/content/linux_timer.html
The best timer | 菜鳥的三年成長史
https://wirelessr.gitbooks.io/working-life/content/the_best_timer.html


Linux Timers | Blog | Upvoid
https://upvoid.com/devblog/2014/05/linux-timers/

kernel debug


Linux kernel tracing
http://www.slideshare.net/vh21/linux-kernel-tracing

Ktap A New Scripting Dynamic Tracing Tool For Linux
http://events.linuxfoundation.org/sites/events/files/lcjpcojp13_zhangwei.pdf

Using GDB to Help You Read Linux Kernel Code without Running It - Google Docs
https://docs.google.com/document/d/1w1nPmCLpeRN3kEYglzHwozhWXY0ddT9oe6JCZocsPNE/edit#heading=h.8bskh6ejh6u7