Debugging

From Helpful
Jump to: navigation, search

Shell, admin, and both:

Shell - command line and bash notes · shell login - profiles and scripts ·· find and xargs and parallel · screen and tmux
Linux admin - disk and filesystem · users and permissions · Debugging · security enhanced linux · health and statistics · kernel modules · YP notes · unsorted and muck
Logging and graphing - Logging · RRDtool and munin notes
Network admin - Firewalling and other packet stuff ·


Remote desktops
VNC notes
XDMCP notes



This article/section is a stub — probably a pile of half-sorted notes, is not well-checked so may have incorrect bits. (Feel free to ignore, fix, or tell me)

What-does-it-spit-out debugging

print statements

For simple bugs this is simple, effective, though you should later either take them out or turn them into filterable logging instead.


Logging

Live debugging

From the IDE

Terms

  • Step into: Go into the called function's workings. (does the same as 'over' if not a call)
  • Step over: Perform evaluation/call, but don't go into it, and don't display ("I trust this, give me the next thing at the current level")
  • Step out: "I've seen enough of this function's guts, go (back) to its caller."

Note: while 'over' may suggest skipping code execution, none of the options skip execution - it just means skipping its debugger display.


Debug and performance inspection tools

strace

This article/section is a stub — probably a pile of half-sorted notes, is not well-checked so may have incorrect bits. (Feel free to ignore, fix, or tell me)

(On solaris, see truss)


finds syscalls a program makes, prints to stderr.

Great for debugging when programs don't say enough, and for workload characterization for things that seem slow.

strace can run a command for you, or attach to a running process via its PID (-p)

For things that fork off, you'll want -f

To compare time spent in each syscall, use
-c
.


You can filter what syscalls are reported using their names, and with a few existing categorizations:

  • file: any file-related calls (access, state, read, write, etc.)
  • desc: file descriptor related
  • process: process management, e.g. fork, wait,l exec
  • network
  • signal
  • ipc
  • memory


For example:

# only mention open() calls
strace -eopen ls     
# see whether there are a bulk of stat()s done (for how many, use -c)
strace -estat,fstat,lstat,newfstatat  find /tmp
# see what progrms are invoked
strace -f -eprocess  service apache2 status 
# summarize time spent in each syscall
strace -c  ls -l /proc


# comparing e.g. ls on small and large dirs
strace -c ls -l /data/largedir
For example, say you notice that
ls -l
seems to do a lot of lgetxattr()s.
-c
would show that for large directories getdents dwarfs everything, and in general it's nearly free assuming that you've just lstat()ted.

ltrace

An execution shim that shows calls into shared libraries.

https://en.wikipedia.org/wiki/Ltrace

pcstat

Page Cache statistics

For individual files, find how much is in the page cache

See also :



echo l > /proc/sysrq-trigger

...gives the backtrace for what's on each CPU.

Writes it to dmesg, which often also goes to something like kern.log, syslog / messages

kernel profiler sources

kprobes

(kernel functions)

kernel tracepoints

dtrace

userspace profiler sources

uprobes

(userspace functions)

USDT (Userland Statically Defined Tracing)

A way of embedding dtrace probes into an app


LTTng userspace

profiler tools / frontends

(Note that some basic things can be gotten from /proc)


dtrace (solaris, also freebsd, linux, osx, smartos)

Very cool tools originating on Solaris.

The rest are ports, some close to the Solaris version, some further away.


perf (linux)

a.k.a. perf_events


https://perf.wiki.kernel.org/index.php/Main_Page

http://www.brendangregg.com/perf.html


systemtap (linux)

systemtap seems to imitate dtrace decently


https://en.wikipedia.org/wiki/SystemTap

https://sourceware.org/systemtap/

https://sourceware.org/systemtap/tutorial.pdf

ftrace

BPF (linux, bsd, more?)

extended Berkeley Packet Filter (eBPF, also just BPF) originated in network packet filtering, but grew so flexible that it is also very useful for system tracing (so now needs a new name).

The thing that allows it to be safe, flexible, and fast is that it compiles user requests into sandboxed bytecode in the kernel.



https://en.wikipedia.org/wiki/Berkeley_Packet_Filter

LTTng

OProfile

https://en.wikipedia.org/wiki/OProfile

Unsorted

Instruments (osx)

https://en.wikipedia.org/wiki/Instruments_(software)

Xperf (windows)

https://blogs.msdn.microsoft.com/ntdebugging/2008/04/03/windows-performance-toolkit-xperf/



Unsorted

http://dtrace.org/blogs/brendan/

NVidia profiler tools

Post-mortem debugging

Debug symbols

GNU debugger