Debugging notes

From Noah.org
Revision as of 09:51, 31 August 2010 by Root (talk | contribs)
Jump to navigationJump to search


Summary of memory used by a program

This will give the max memory used by a program while it was running. It is unclear if this includes child processes and shared libraries. I use `command` here to start `time` because `time` is a Bash built-in as well as an external command. In this case I want the command, not the built-in.

command time --format "%MK" COMMAND

Debugging with LD_PRELOAD hooks

See http://michael.toren.net/slides/lkm-alternatives/slide006.html

LD_PRELOAD_notes

Debugging with `strace`

For these examples, I usually open two xterm windows for experimentation. In one shell I use `strace` to attach to the PID of the other shell. You can get the PID of a shell by running `echo $$` in the shell you want to watch.

This will show all file related activity by the process, $PID. Note that -e trace=file is synonymous with -e file. The man page for `strace` explains the expression syntax for the -e option.

strace -p $PID -f -e trace=file

You can use the -c option to get a count of each system call made by the process. The count is made after filtering by the -e expression. The count is printed when `strace` exits. The following will show a count of each file related system call.

strace -p $PID -c -f -e trace=file

This will show a count of all system calls.

strace -p $PID -c -f

This will show what data is written by the process, $PID.

strace -p $PID -f -e write

The '-f' option tells `strace` to follow children that are forked and execed. Note that this is a best effort and that strace can miss a few system calls of the child while it starts up. This can be significant in the real world.

The following will trace data that is read and written by the process, $PID:

strace -f -e write,read -p $PID

The following will trace data that is written by the process, $PID. This is useful for tracing stdout of shells and the like. I don't know why I need both -e options. I think I should need only -e write=2, but if I do that then I see every other system call. Nor can I do -e write,write=2. This syntax gives me a headache... At any rate, this mostly does what I want, but not quite -- it does not actually print re response back from the shell in the hex dump:

strace -f -s 1024 -e write -e write=2 -p $PID

This will show files created by a process. Note that files can be created and opened for writing using 'creat' as well as 'open'. Here I filter a lot of the open calls. The "-o /proc/self/fd/1" forces output to stdout.

strace -o /proc/self/fd/1 -p $PID -f -e creat,open | grep -v O_RDONLY

In theory, you could also force output to stdout with '-o \|cat', but piping through cat seems to take more time, so `strace` misses more child calls when it tried to follow them. It is also slower to use `strace's` own built-in filter option '-e'. It is faster to pipe through grep for later filtering. For example, this will often miss 'open' calls to open files for writing:

strace -o \|cat -p $PID -f -e creat -e open | grep -v O_RDONLY

But will work a little better:

strace -o \|cat -p $PID -f | grep -v O_RDONLY | grep open

You might miss a file open or creat when using redirects from the parent shell. If you use this in a command-line pipe stream that the process you trace has the file opened for it already as file descriptor 1. This should be no surprise...

strace -f -e write echo foo > foo.txt