Hellish go debugging story

> Monkey see program crash
> blow heat gun at memory
> find and blacklist corrupted RAM bits
> still see crash
> compile 100s of kernel (coz poor man have no ECC memory)
> compare kernel configs
> sees CONFIG_OPTIMIZE_INLINING=n and flipping it fixes crash
> still don’t know which inlined function triggers the crash
> build 100 more kernel, test each of 'em in qmeu
> compiling part of the kernel with CONFIG_OPTIMIZE_INLINING on.
> in hopes of narrowing down the problematic object.
> finds .../vdso/vclock_gettime.o.
> vDSO, ofcourse.
> remembers it is added to all programs running in userspace.
> flips the CONFIG_OPTIMIZE_INLINING again and compares vDSO’s ASM.
> it does orq on memory after 4KiB.
> recall why allocate 4KiB of stack in the first place? ^[1]
> coz, -fstack-check was turned on in his GCC version.
> realizes go reimplement how memory allocations are made
> realizes go reimplement how and system calls are invoked
> check golang’s source code
> see a preallocated 104 Bytes reserved for vDSO shared object
> dam, that’s not enough! stackprobe orq require 4KiB.
> now stackprobe crosses 104B program’s current thread’s memory
> it marks a random bit on memory it doesn’t even own.
> a poor thread reads that random corrupted bit
> the poor thread simply cries and crashes.
> Fascinating.

Debugging an evil Go runtime bug, from heat guns to kernel compiler flags (2017): https://marcan.st/2017/12/debugging-an-evil-go-runtime-bug/

Footnotes

1. it’s there to mitigate stack clash vulnerability (where stack expansion at one point crosses another program’s memory region - allowing attackers to write arbitrary code there and exploiting target program to execute it. win.) ↩︎