Skip to content

Server trouble‐shooting

Jump to bottom
David Anderson edited this page Sep 13, 2024 · 2 revisions

Log files

Each server component (scheduler, feeder, transitioner, etc.) has its own log file. These files are in the log_HOSTNAME subdirectory of the project directory. The logs have entries for errors, indicated by CRITICAL; e.g.

make_work_mt.out:1601921:2022-08-16 20:36:53.1759 [CRITICAL] can't find wu wu_multi_thread_nodelete

This generally means that something that needs to happen isn't happening; you need to figure it out and fix it.

In addition, log entries can describe normal events. To control the verbosity of the log files:

  • Scheduler: set the desired logging options
  • File upload handler: set fuh_debug_level.
  • daemons: pass the cmdline arg "-d N" (1=least verbose, 4=most verbose) If you run server components with -d 4, their database queries will be logged. This is useful for tracking down database-level problems.

If you're interested in the history of a particular job, grep for WU#12345 or RESULT#12345 (where 12345 represents the ID) in the log files. The html/ops pages also provide an interface for this.

Examining the database

The admin web interface provides a web-based interface for browsing your project's database.

You can also use MySQL tools such as

Examining shared memory

The command

bin/show_shmem

will print a textual summary of the contents of the shared-memory structure that caches jobs and information about applications.

Trouble-shooting the job pipeline

  • Are workunits (jobs) getting created correctly? Examine the database to see. If you're using a work generator, check its log file.
  • Are results (job instances) getting created? Examine the database to see. If you don't see results, check the transitioner log file.
  • Are jobs getting into shared memory? Use show_shmem (see above). You should see jobs. If not, check the feeder log file.
  • Is the scheduler sending jobs? If not, check its log file, preferably with the following log flags:
    • <debug_version/>: show details of app version selection
    • <debug_send/>: show details of job assignment
    • <debug_quota/>: show details of quota enforcement
  • Are clients processing jobs correctly? Check the status and stderr output of completed jobs.
  • Are output files getting uploaded? Check the file upload handler log file.
  • Are jobs getting validated? Check the validator log file.
  • Are jobs getting assimilated? Check the assimilator log file.

Debugging the scheduler

If the scheduler is acting incorrectly or crashing, and you like mucking around in C++ source code, you can run it under a debugger like gdb. The scheduler is a CGI program; it reads a request from stdin and writes a reply to stdout. So you can debug it as follows:

  • Copy the "scheduler_request_X.xml" file from a client to the machine running the scheduler. (X = your project URL)
  • Run the scheduler under the debugger, giving it this file as stdin, i.e.:
gdb cgi
(set a breakpoint if desired)
r < scheduler_request_X.xml
  • You may have to doctor the database as follows to keep the scheduler from rejecting the request:
update host set rpc_seqno=0, rpc_time=0 where hostid=N

As an alternative to this, edit sched/handle_request.cpp, and put a call to debug_sched("debug_sched"); just before sreply.write(fout, sreq);. Then, after recompiling, touch a file called 'debug_sched' in the project root directory. This will cause transcripts of all subsequent scheduler requests and replies to be written to the cgi-bin/ directory with separate small files for each request. The file names are sched_request_H_R and sched_reply_H_R where H=hostid and R=rpc sequence number. This can be turned off by deleting the 'debug_sched' file.

To get core files for scheduler crashes, uncomment the following line in sched/sched_main.cpp, and recompile:

#define DUMP_CORE_ON_SEGV 1

Home

Clone this wiki locally