Debugging notes
Contents
Summary of memory used by a program
This will give the max memory used by a program while it was running. It is unclear if this includes child processes and shared libraries. I use `command` here to start `time` because `time` is a Bash built-in as well as an external command. In this case I want the command, not the built-in. UPDATE WARNING: The `command` tool does not seem to work as I had expected in Bash. There is an alternative that seems to work and that is to put a \ back-slash in from the command you want to be loaded as a program instead of a shell built-in.
This shows the resident set size.
command time --format "%MK" [COMMAND] # OR \time --format "%MK" [COMMAND]
The following is a more concrete example. This compares the effect of using read modifiers in ImageMagic conversion.
# With read modifier: command time --format "%MK" find ~/Pictures -type f -exec convert {}'[480x272]' -format jpg -quality 80 -resize '480x272>' -gravity center -extent 480x272 -background white -flatten {}.preview.jpg \; # Without read modifier: command time --format "%MK" find ~/Pictures -type f -exec convert {} -format jpg -quality 80 -resize '480x272>' -gravity center -extent 480x272 -background white -flatten {}.preview.jpg \;
Debugging with LD_PRELOAD hooks
See http://michael.toren.net/slides/lkm-alternatives/slide006.html
See also LD_PRELOAD_notes.
Debugging with ptrace(1)
The ptrace(1) system call is used to attach a trace to a process. See the following article, ptrace notes, for information on using ptrace for debugging down to the CPU instruction level.
Debugging with `perf trace`
perf trace is similar to strace but it is newer and uses a different subsystem in the kernel to gather its trace information.
perf trace echo "hello world"
Debugging with `strace`
See also perf trace.
For these examples, I like to open two terminal windows. In one shell I use `strace` to attach to the PID of the other shell. You can get the PID of a shell by running `echo $$` in the shell you want to watch.
The '-f' option tells `strace` to follow children that are forked and exec'ed. Note that this is a best effort and that strace can miss a few system calls by the child while it starts up. This can be significant in the real world.
This will show all file related activity by the process, $PID. Note that -e trace=file is synonymous with -e file. The man page for `strace` explains the expression syntax for the -e option.
strace -p $PID -f -e trace=file
The following will show network activity by the process, $PID:
strace -p $PID -f -e trace=network
You can use the -c option to get a count of each system call made by the process. The count is made after filtering by the -e expression. The count is printed when `strace` exits. The following will show a count of each file related system call.
strace -p $PID -c -f -e trace=file
This will show a count of all system calls.
strace -p $PID -c -f
This will show what data is written by the process, $PID.
strace -p $PID -f -e write
The following will trace data that is read and written by the process, $PID:
strace -f -e write,read -p $PID
The following will trace data that is written by the process, $PID. This is useful for tracing stdout of shells and the like. I don't know why I need both -e options. I think I should need only -e write=2, but if I do that then I see every other system call. Nor can I do -e write,write=2. This syntax gives me a headache... At any rate, this mostly does what I want, but not quite -- it does not actually print re response back from the shell in the hex dump:
strace -f -s 1024 -e write -e write=2 -p $PID
This will show files created by a process. Note that files can be created and opened for writing using 'creat' as well as 'open'. Here I filter a lot of the open calls. The "-o /proc/self/fd/1" forces output to stdout.
strace -o /proc/self/fd/1 -p $PID -f -e creat,open | grep -v O_RDONLY
In theory, you could also force output to stdout with '-o \|cat', but piping through cat seems to take more time, so `strace` misses more child calls when it tried to follow them. It is also slower to use `strace's` own built-in filter option '-e'. It is faster to pipe through grep for later filtering. For example, this will often miss 'open' calls to open files for writing:
strace -o \|cat -p $PID -f -e creat -e open | grep -v O_RDONLY
But will work a little better:
strace -o \|cat -p $PID -f | grep -v O_RDONLY | grep open
You might miss a file open or create when using redirects from the parent shell. If you use this in a command-line pipe stream that the process you trace has the file opened for it already as file descriptor 1. This should be no surprise...
strace -f -e write echo foo > foo.txt