Debugging and troubleshooting

From CSP

Introduction

CSP is a bit challenging to debug, since the main application runs in Python with the core components loaded as extension modules. This page provides some tips for debugging; it is a work in progress, so feel free to correct or expand on it as needed. Note that except for logging, most of the details here are specific to the GNU/Linux platform.

Logging

CSP actually runs under Python. sim.py is the main entry point, which starts the Python interpreter, parses command line flags, and loads the core extension modules that drive the simulation. The two main extension modules are _csplib and _cspsim (.so under GNU/Linux, .dll under Windows). Both of these modules write to a common log file (sim.log) that can contain useful debugging information. The logs are the first place you should look if you are having trouble running the simulation. Note that logs are only produced if the extension modules are built in debug mode. (This is the default build mode under GNU/Linux). If built in "release mode", no logs will be written and all the symbolic debug information that is very helpful for diagnosing problems will be omitted.

By default the log is written to sim.log in the current directory. The content of each of this log can be restricted in two ways. The first restriction is a priority threshold. Each log entry has an associated priority, ranging from 0 (low priority) to 4 (high priority). These numeric priorities also have associated symbolic names: DEBUG, INFO, WARNING, ERROR, FATAL. The default priority threshold is set in the sim.ini file (typically located in the same directory as sim.py). The LoggingLevel entry under the Debug section sets the numeric logging threshold for sim.log messages. Setting this to 2 or higher will prevent many low priority messages from being recorded, and generally improves the performance of the simulation. If you encounter problems, especially during startup, you should decrease this value to 0 to log all messages. The priority threshold can also be overridden by specifying --logpri=XXX when running sim.py. The value of this flag can either be numeric (same as the .ini file), or symbolic (e.g. "INFO").

The second restriction is by message category. Each message is assigned a category, from a limited set defined in csplib/util/LogConstants.h. By default all categories are logged, but this can create a lot of noise when you are trying to focus on a specific type of problem. To limit the set of message categories, specify the --logcat=XXX flag when running sim.py. Here the flag value is a set of categories, separated by colons (":"). For example, --logcat=NETWORK:PHYSICS:TERRAIN will cause only messages in those three categories to be logged. You can also exclude individual message categories; for example --logcat=ALL:-NETWORK:-PHYSICS logs everything except network and physics messages.

Under GNU/Linux, the simulation can record stack traces when internal faults are detected. These stack traces are written to the log file and can be very helpful for locating bugs. The raw stack traces are usually somewhat cryptic, but there is a tool in csp/tools/ called trace which can decode the stack traces into a more readable form. Simply copy the stack trace from the log, run trace from the same directory in which you ran sim.py, paste the stack trace, and hit <ctrl-d>. It may take a little time to resolve the symbols, so be patient. The output gives the full call stack at the time of the error, as well as function names, source files, and line numbers. This often provides a good starting point for debugging, either by inspection of the source code or by setting breakpoints in a debugger.

Attaching GDB

If log messages aren't sufficient to diagnose the problem, you'll probably need to use a debugger. Under GNU/Linux, that means gdb. The following assumes you are somewhat familiar with gdb; if not you should look at the gdb man page and online documentation.

First, run sim.py with the --pause flag. This will start the simulation to the point of loading the _csplib and _cspsim extension modules, print some instructions for starting gdb, and then pause until you hit <enter>. Basically, just follow the instructions that sim.py prints to start gdb in a separate terminal window and attach to the running Python process. gdb will freeze the process so that you can set breakpoints as needed. When you are ready, enter 'c' at the gdb prompt to unfreeze (continue) the process, and then hit <enter> in sim.py window to unpause the simulation. Note that you probably don't want to be in fullscreen mode while doing this (see the FullScreen option in sim.ini).

Windows users: Please refer to Debugging_the_Windows_build for information about how to use the Visual Studio debugger.