Debugging notes

Summary of memory used by a program

This will give the max memory used by a program while it was running. It is unclear if this includes child processes and shared libraries. I use `command` here to start `time` because `time` is a Bash built-in as well as an external command. In this case I want the command, not the built-in. UPDATE WARNING: The `command` tool does not seem to work as I had expected in Bash. There is an alternative that seems to work and that is to put a \ back-slash in from the command you want to be loaded as a program instead of a shell built-in.

This shows the resident set size.

command time --format "%MK" [COMMAND]
# OR
\time --format "%MK" [COMMAND]

The following is a more concrete example. This compares the effect of using read modifiers in ImageMagic conversion.

# With read modifier:
command time --format "%MK" find ~/Pictures -type f -exec convert {}'[480x272]' -format jpg -quality 80 -resize '480x272>' -gravity center -extent 480x272 -background white -flatten {}.preview.jpg \;
# Without read modifier:
command time --format "%MK" find ~/Pictures -type f -exec convert {} -format jpg -quality 80 -resize '480x272>' -gravity center -extent 480x272 -background white -flatten {}.preview.jpg \;

Debugging with LD_PRELOAD hooks

See http://michael.toren.net/slides/lkm-alternatives/slide006.html

See also LD_PRELOAD_notes.

Debugging with ptrace(1)

The ptrace(1) system call is used to attach a trace to a process. See the following article, ptrace notes, for information on using ptrace for debugging down to the CPU instruction level.

Debugging with `perf trace`

perf trace is similar to strace but it is newer and uses a different subsystem in the kernel to gather its trace information.

perf trace echo "hello world"

Debugging with `strace`

See also perf trace.

For these examples, I like to open two terminal windows. In one shell I use `strace` to attach to the PID of the other shell. You can get the PID of a shell by running `echo $$` in the shell you want to watch.

This will show all file related activity by the process, $PID. Note that -e trace=file is synonymous with -e file. The man page for `strace` explains the expression syntax for the -e option.

strace -p $PID -f -e trace=file

You can use the -c option to get a count of each system call made by the process. The count is made after filtering by the -e expression. The count is printed when `strace` exits. The following will show a count of each file related system call.

strace -p $PID -c -f -e trace=file

This will show a count of all system calls.

strace -p $PID -c -f

This will show what data is written by the process, $PID.

strace -p $PID -f -e write

The '-f' option tells `strace` to follow children that are forked and execed. Note that this is a best effort and that strace can miss a few system calls of the child while it starts up. This can be significant in the real world.

The following will trace data that is read and written by the process, $PID:

strace -f -e write,read -p $PID

The following will trace data that is written by the process, $PID. This is useful for tracing stdout of shells and the like. I don't know why I need both -e options. I think I should need only -e write=2, but if I do that then I see every other system call. Nor can I do -e write,write=2. This syntax gives me a headache... At any rate, this mostly does what I want, but not quite -- it does not actually print re response back from the shell in the hex dump:

strace -f -s 1024 -e write -e write=2 -p $PID

This will show files created by a process. Note that files can be created and opened for writing using 'creat' as well as 'open'. Here I filter a lot of the open calls. The "-o /proc/self/fd/1" forces output to stdout.

strace -o /proc/self/fd/1 -p $PID -f -e creat,open | grep -v O_RDONLY

In theory, you could also force output to stdout with '-o \|cat', but piping through cat seems to take more time, so `strace` misses more child calls when it tried to follow them. It is also slower to use `strace's` own built-in filter option '-e'. It is faster to pipe through grep for later filtering. For example, this will often miss 'open' calls to open files for writing:

strace -o \|cat -p $PID -f -e creat -e open | grep -v O_RDONLY

But will work a little better:

strace -o \|cat -p $PID -f | grep -v O_RDONLY | grep open

You might miss a file open or create when using redirects from the parent shell. If you use this in a command-line pipe stream that the process you trace has the file opened for it already as file descriptor 1. This should be no surprise...

strace -f -e write echo foo > foo.txt