
Debugging the CGI binaries:

The typical sign of trouble is an Error 500 display in your
WEB browser when accessing the CGI binaries, and the following
message in your Apache error log:

[Fri Mar 25 11:02:40 2005] [error] Premature end of script headers: hgTracks

This is usually a simple configuration problem.  Items to verify:

1.  the hg.conf file in the cgi-bin directory specifies the correct
	user names and passwords for MySQL database access.
	See also: README.mysql.setup
2.  The cgi-bin directory is set to permissions 755 and not 775 or 777
	When permissions are too permissive for this directory, Apache
	errors out with suexec permission violations.
3.  Verify change history of the database hgcentral:
	http://genome.ucsc.edu/admin/mirror.html#step7
    Rarely, changes in this database require corresponding changes
	in the source code.  Make sure your code and version of
	hgcentral are synchronized.  Newer versions of hgcentral
	database with old source code are OK.  The problem is when
	you have new source code that expects new features in hgcentral.

If these items are OK, then you can check the actual operation of
a cgi binary.  Go to the source tree directory of the cgi binary,
for example hgTracks:
	kent/src/hg/hgTracks

In this directory, run a 'make compile' to produce a binary that
is left in this directory.  This binary can be run from the command
line:

$ ./hgTracks

By itself with no arguments, it should produce the default tracks
display HTML page for the Human genome.  This assumes you have set
up your $HOME/.hg.conf file to allow access to the MySQL databases.
(See also: README.mysql.setup).  A binary execution failure should
be obvious at this stage of the game.  If it exits because of SIGSEGV
we can run it under a debugger for specifics.  More on this below.

If the problem is specific to a particular set of tracks being
displayed, or particular genomes or options, command line arguments
can be given to these CGI binaries to provide the URL inputs
that a CGI binary would normally see.

To prepare the binaries for operation under a debugger, go to
the src/inc directory and edit the common.mk configuration file.
Change "COPT=-O" to read: "COPT=-g"
GNU gcc will allow "-O" with "-g", and some bugs will only exhibit
themselves with -O on.  However the optimizations with -O can
sometimes confuse the debugger's sense of location due to
optimization rearrangement of code.
Also eliminate the -Wuninitialized option from the HG_WARN definition
to avoid constant warnings about that being incompatible with -g.

Rebuild the source tree:

$ cd kent/src
$ make clean
$ make libs
$ cd hg/hgTracks
$ make compile

The hgTracks binary will now have all symbol information in it and
it can be operated under a debugger such as ddd (or gdb, etc...).

For the case of specific options or tracks causing problems,
to find the full set of options in effect for the failure case,
when your WEB browser is at the Error 500 display page, edit
the displayed URL in your WEB browser to call the cgi binary cartDump:
http://<your server>/cgi-bin/cartDump 

This will display all environment variables in effect at the time
of the crash.  Most of the track display options that are marked
as "hide" can be ignored.  That is their default setting already.
The important ones are the db, position, and specific options for
the track under consideration.  The command line can be formatted
just as if it was a URL string.  For example:

$ ./hgTracks "db=hg17&trackControlsOnMain=0&position=chr4:56214201-56291736"

Or with spaces between the arguments:

$ ./hgTracks db=hg17 trackControlsOnMain=0 position=chr4:56214201-56291736

Remember to protect special characters on the command line from
shell interpretation by appropriate quoting.

At this point, running under a debugger, with a command line for
specific options, a crash of the binary should give you some clue
about the problem by checking the stack backtrace to see what function
is failing.  It is highly doubtful you will be finding problems
in the source code for the crashes.  The almost universal cause
for failure are the data inputs to the binaries.  For example,
violations of the SQL structures expected from the database tables.
Missing data files in the /gbdb/ hierarchy, and so forth.

If you are developing code for special track displays, the most
common form of problem is a memory violation while using some
of the specialized structures, hash lists, etc.  Your stack backtrace
will usually highlight these situations.

====================================================================
This file last updated: $Date: 2005/03/28 19:49:59 $
