Mod Perl Icon Mod Perl Icon Controlling and Monitoring the Server


[ Prev | Main Page | Next ]

Table of Contents:


The Writing Apache Modules with Perl and C book can be purchased online from O'Reilly and Amazon.com.
Your corrections of either technical or grammatical errors are very welcome. You are encouraged to help me to improve this guide. If you have something to contribute please send it directly to me.

[TOC]


Restarting techniques

All of these techniques require that you know the server PID (Process ID). The easiest way to find the PID is to look it up in the httpd.pid file. With my configuration it exists as /usr/local/var/httpd_perl/run/httpd.pid. It's easy to discover where to look at, by checking out the httpd.conf file. Open the file and locate the entry PidFile:

  PidFile /usr/local/var/httpd_perl/run/httpd.pid

Another way is to use the ps and grep utilities:

  % ps auxc | grep httpd_perl

or maybe:

  % ps -ef | grep httpd_perl

This will produce a list of all httpd_perl (the parent and the children) processes. You are looking for the parent process. If you run your server as root - you will easily locate it, since it belongs to root. If you run the server as user (when you don't have a root access, most likely all the processes will belong to that user (unless defined differently in the httpd.conf), but it's still easy to know 'who is the parent' -- the one of the smallest size...

You will notice many httpd_perl executables running on your system, but you should not send signals to any of them except the parent, whose pid is in the PidFile. That is to say you shouldn't ever need to send signals to any process except the parent. There are three signals that you can send the parent: TERM, HUP, and USR1.

[TOC]


Implications of sending TERM, HUP, and USR1 to the server

We will concentrate here on the implications of sending these signals to a mod_perl enabled server. For documentation on the implications of sending these signals to a plain Apache server see http://www.apache.org/docs/stopping.html .

TERM Signal: stop now

Sending the TERM signal to the parent causes it to immediately attempt to kill off all of its children. This process may take several seconds to complete, following which the parent itself exits. Any requests in progress are terminated, and no further requests are served.

That's the moment that the accumulated END blocks will be executed! Note that if you use Apache::Registry or Apache::PerlRun, then END blocks are being executed upon each request (at the end).

HUP Signal: restart now

Sending the HUP signal to the parent causes it to kill off its children like in TERM (Any requests in progress are terminated) but the parent doesn't exit. It re-reads its configuration files, and re-opens any log files. Then it spawns a new set of children and continues serving hits.

The server will reread its configuration files, flush all the compiled and preloaded modules, and rerun any startup files. It's equivalent to stopping, then restarting a server.

Note: If your configuration file has errors in it when you issue a restart then your parent will not restart but exit with an error. See below for a method of avoiding this.

USR1 Signal: graceful restart

The USR1 signal causes the parent process to advise the children to exit after their current request (or to exit immediately if they're not serving anything). The parent re-reads its configuration files and re-opens its log files. As each child dies off the parent replaces it with a child from the new generation of the configuration, which begins serving new requests immediately.

The only difference between USR1 and HUP is that USR1 allows children to complete any in-progress request prior to killing them off.

By default, if a server is restarted (ala kill -USR1 `cat logs/httpd.pid` or with HUP signal), Perl scripts and modules are not reloaded. To reload PerlRequire's, PerlModule's, other use()'d modules and flush the Apache::Registry cache, enable with this command:

 PerlFreshRestart On              (in httpd.conf) 

Make sure you read Evil things might happen when using PerlFreshRestart.

It's worth mentioning that restart or termination can sometimes take quite a lot of time. Check out the PERL_DESTRUCT_LEVEL=-1 option during the mod_perl perl Makefile.PL stage, which speeds this up and leads to more robust operation in the face of problems, like running out of memory. It is only usable if no significant cleanup has to be done by perl END blocks and DESTROY methods when the child terminates, of course. What constitutes significant cleanup? Any change of state outside of the current process that would not be handled by the operating system itself. So committing database transactions is significant but closing an ordinary file isn't.

Some folks prefer to specify signals using numerical values, rather than symbolics. If you are looking for these, check out your kill(3) man page. My page points to /usr/include/sys/signal.h, the relevant entries are:

  #define SIGHUP     1    /* hangup, generated when terminal disconnects */ 
  #define SIGTERM   15    /* software termination signal */
  #define SIGUSR1   30    /* user defined signal 1 */

[TOC]


Using apachectl to control the server

Apache's distribution provides a nice script to control the server. It's called apachectl and it's installed into the same location with httpd. In our scenario - it's /usr/local/sbin/httpd_perl/apachectl.

Start httpd:

  % /usr/local/sbin/httpd_perl/apachectl start 

Stop httpd:

  % /usr/local/sbin/httpd_perl/apachectl stop

Restart httpd if running by sending a SIGHUP or start if not running:

  % /usr/local/sbin/httpd_perl/apachectl restart

Do a graceful restart by sending a SIGUSR1 or start if not running:

  % /usr/local/sbin/httpd_perl/apachectl graceful    

Do a configuration syntax test:

  % /usr/local/sbin/httpd_perl/apachectl configtest 

Replace httpd_perl with httpd_docs in the above calls to control the httpd_docs server.

There are other options for apachectl, use help option to see them all.

It's important to understand that this script is based on the PID file which is PIDFILE=/usr/local/var/httpd_perl/run/httpd.pid. If you delete the file by hand - apachectl will fail to run.

Also, notice that apachectl is suitable to use from within your Unix system's startup files so that your web server is automatically restarted upon system reboot. Either copy the apachectl file to the appropriate location (/etc/rc.d/rc3.d/S99apache works on my RedHat Linux system) or create a symlink with that name pointing to the the canonical location. (If you do this, make certain that the script is writable only by root -- the startup scripts have root privileges during init processing, and you don't want to be opening any security holes.)

[TOC]


SUID start-up scripts

For those who wants to use SUID startup script, here is an example for you. This script is SUID to root, and should be executable only by members of some special group at your site. Note the 10th line, which ``fixes an obscure error when starting apache/mod_perl'' by setting the real to the effective UID. As others have pointed out, it is the mismatch between the real and the effective UIDs that causes Perl to croak on the -e switch.

Note that you must be using a version of Perl that recognizes and emulates the suid bits in order for this to work. The script will do different things depending on whether it is named start_http, stop_http or restart_http. You can use symbolic links for this purpose.

 #!/usr/bin/perl
 
 # These constants will need to be adjusted.
 $PID_FILE = '/home/www/logs/httpd.pid';
 $HTTPD = '/home/www/httpd -d /home/www';
 
 # These prevent taint warnings while running suid
 $ENV{PATH}='/bin:/usr/bin';
 $ENV{IFS}='';
 
 # This sets the real to the effective ID, and prevents
 # an obscure error when starting apache/mod_perl
 $< = $>;
 $( = $) = 0; # set the group to root too
 
 # Do different things depending on our name
 ($name) = $0 =~ m|([^/]+)$|;
 
 if ($name eq 'start_http') {
     system $HTTPD and die "Unable to start HTTP";
     print "HTTP started.\n";
     exit 0;
 }
 
 # extract the process id and confirm that it is numeric
 $pid = `cat $PID_FILE`;
 $pid =~ /(\d+)/ or die "PID $pid not numeric";
 $pid = $1;
 
 if ($name eq 'stop_http') {
     kill 'TERM',$pid or die "Unable to signal HTTP";
     print "HTTP stopped.\n";
     exit 0;
 }
 
 if ($name eq 'restart_http') {
     kill 'HUP',$pid or die "Unable to signal HTTP";
     print "HTTP restarted.\n";
     exit 0;
 }
 
 die "Script must be named start_http, stop_http, or restart_http.\n";

[TOC]


Monitoring the Server. A watchdog.

With mod_perl many things can happen to your server. The worst one is the possibility that the server will die when you will be not around. As with any other critical service you need to run some kind of watchdog.

One simple solution is to use a slightly modified apachectl script which I called apache.watchdog and to put it into the crontab to be called every 30 minutes or even every minute - if it's so critical to make sure the server will be up all the time.

The crontab entry:

  0,30 * * * * /path/to/the/apache.watchdog >/dev/null 2>&1

The script:

  #!/bin/sh
    
  # this script is a watchdog to see whether the server is online
  # It tries to restart the server if it's
  # down and sends an email alert to admin 
  
  # admin's email
  EMAIL=webmaster@somewhere.far
  #EMAIL=root@localhost
    
  # the path to your PID file
  PIDFILE=/usr/local/var/httpd_perl/run/httpd.pid
    
  # the path to your httpd binary, including options if necessary
  HTTPD=/usr/local/sbin/httpd_perl/httpd_perl
        
  # check for pidfile
  if [ -f $PIDFILE ] ; then
    PID=`cat $PIDFILE`
    
    if kill -0 $PID; then
      STATUS="httpd (pid $PID) running"
      RUNNING=1
    else
      STATUS="httpd (pid $PID?) not running"
      RUNNING=0
    fi
  else
    STATUS="httpd (no pid file) not running"
    RUNNING=0
  fi
      
  if [ $RUNNING -eq 0 ]; then
    echo "$0 $ARG: httpd not running, trying to start"
    if $HTTPD ; then
      echo "$0 $ARG: httpd started"
      mail $EMAIL -s "$0 $ARG: httpd started" </dev/null >& /dev/null
    else
      echo "$0 $ARG: httpd could not be started"
      mail $EMAIL -s "$0 $ARG: httpd could not be started" </dev/null >& /dev/null
    fi
  fi

Another approach, probably even more practical, is to use the cool LWP perl package , to test the server by trying to fetch some document (script) served by the server. Why is it more practical? Because, while server can be up as a process, it can be stuck and not working, So failing to get the document will trigger restart, and ``probably'' the problem will go away. (Just replace start with restart in the $restart_command below.

Again we put this script into a crontab to call it every 30 minutes. Personally I call it every minute, to fetch some very light script. Why so often? If your server starts to spin and trash your disk's space with multiply error messages, in a 5 minutes you might run out of free space, which might bring your system to its knees. And most chances that no other child will be able to serve requests, since the system will be too busy, writing to an error_log file. Think big -- if you are running a heavy service, which is very fast, since you are running under mod_perl, adding one more request every minute, will be not felt by the server at all.

So we end up with crontab entry:

  * * * * * /path/to/the/watchdog.pl >/dev/null 2>&1

And the watchdog itself:

  #!/usr/local/bin/perl -w
  
  use strict;
  use diagnostics;
  use URI::URL;
  use LWP::MediaTypes qw(media_suffix);
  
  my $VERSION = '0.01';
  use vars qw($ua $proxy);
  $proxy = '';    

  require LWP::UserAgent;
  use HTTP::Status;
  
  ###### Config ########
  my $test_script_url = 'http://www.stas.com:81/perl/test.pl';
  my $monitor_email   = 'root@localhost';
  my $restart_command = '/usr/local/sbin/httpd_perl/apachectl restart';
  my $mail_program    = '/usr/lib/sendmail -t -n';
  ######################
  
  $ua  = new LWP::UserAgent;
  $ua->agent("$0/Stas " . $ua->agent);
  # Uncomment the proxy if you don't use it!
  #  $proxy="http://www-proxy.com";;
  $ua->proxy('http', $proxy) if $proxy;
  
  # If returns '1' it's we are alive
  exit 1 if checkurl($test_script_url);
  
  # We have got the problem - the server seems to be down. Try to
  # restart it. 
  my $status = system $restart_command;
  #  print "Status $status\n";
  
  my $message = ($status == 0) 
              ? "Server was down and successfully restarted!" 
              : "Server is down. Can't restart.";
    
  my $subject = ($status == 0) 
              ? "Attention! Webserver restarted"
              : "Attention! Webserver is down. can't restart";
  
  # email the monitoring person
  my $to = $monitor_email;
  my $from = $monitor_email;
  send_mail($from,$to,$subject,$message);
  
  # input:  URL to check 
  # output: 1 if success, o for fail  
  #######################  
  sub checkurl{
    my ($url) = @_;
  
    # Fetch document 
    my $res = $ua->request(HTTP::Request->new(GET => $url));
  
    # Check the result status
    return 1 if is_success($res->code);
  
    # failed
    return 0;
  } #  end of sub checkurl
  
  # sends email about the problem 
  #######################  
  sub send_mail{
    my($from,$to,$subject,$messagebody) = @_;
  
    open MAIL, "|$mail_program"
        or die "Can't open a pipe to a $mail_program :$!\n";
   
    print MAIL <<__END_OF_MAIL__;
  To: $to
  From: $from
  Subject: $subject
  
  $messagebody
  
  __END_OF_MAIL__
  
    close MAIL;
  } 

[TOC]


Running server in a single mode

Often while developing new code, you will want to run the server in single process mode. See Sometimes it works Sometimes it does Not and Names collisions with Modules and libs Running in single process mode inhibits the server from ``daemonizing'', allowing you to run it more easily under debugger control.

  % /usr/local/sbin/httpd_perl/httpd_perl -X

When you execute the above the server will run in the fg (foreground) of the shell you have called it from. So to kill you just kill it with Ctrl-C.

Note that in -X mode the server will run very slowly while fetching images. If you use Netscape while your server is running in single-process mode, HTTP's KeepAlive feature gets in the way. Netscape tries to open multiple connections and keep them open. Because there is only one server process listening, each connection has to time-out before the next succeeds. Turn off KeepAlive in httpd.conf to avoid this effect while developing or you can press STOP after a few seconds (assuming you use the image size params, so the Netscape will be able to render the rest of the page).

In addition you should know that when running with -X you will not see any control messages that the parent server normally writes to the error_log. (Like ``server started, server stopped and etc''.) Since httpd -X causes the server to handle all requests itself, without forking any children, there is no controlling parent to write status messages.

[TOC]


Starting a personal server for each developer

If you are the only developer working on the specific server:port - you have no problems, since you have a complete control over the server. However, many times you have a group of developers who need to concurrently develop their own mod_perl scripts. This means that each one will want to have control over the server - to kill it, to run it in single server mode, to restart it again, etc., as well to have control over the location of the log files and other configuration settings like MaxClients, etc. You can work around this problem by preparing a few httpd.conf file and forcing each developer to use:

  httpd_perl -f /path/to/httpd.conf  

I have approached it in other way. I have used the -Dparameter startup option of the server. I call my version of the server

  % http_perl -Dsbekman

In httpd.conf I wrote:

  # Personal development Server for sbekman
  # sbekman use the server running on port 8000
  <IfDefine sbekman>
  Port 8000
  PidFile /usr/local/var/httpd_perl/run/httpd.pid.sbekman
  ErrorLog /usr/local/var/httpd_perl/logs/error_log.sbekman
  Timeout 300
  KeepAlive On
  MinSpareServers 2
  MaxSpareServers 2
  StartServers 1
  MaxClients 3
  MaxRequestsPerChild 15
  </IfDefine>
  
  # Personal development Server for userfoo
  # userfoo use the server running on port 8001
  <IfDefine userfoo>
  Port 8001
  PidFile /usr/local/var/httpd_perl/run/httpd.pid.userfoo
  ErrorLog /usr/local/var/httpd_perl/logs/error_log.userfoo
  Timeout 300
  KeepAlive Off
  MinSpareServers 1
  MaxSpareServers 2
  StartServers 1
  MaxClients 5
  MaxRequestsPerChild 0
  </IfDefine>

What we have achieved with this technique: Full control over start/stop, number of children, separate error log file, and port selection. This saves me from getting called every few minutes - ``Stas, I'm going to restart the server''.

To make things even easier. (In the above technique, you have to discover the PID of your parent httpd_perl process - written in /usr/local/var/httpd_perl/run/httpd.pid.userfoo) . We change the apachectl script to do the work for us. We make a copy for each developer called apachectl.username and we change 2 lines in script:

  PIDFILE=/usr/local/var/httpd_perl/run/httpd.pid.sbekman
  HTTPD='/usr/local/sbin/httpd_perl/httpd_perl -Dsbekman'

Of course you think you can use only one control file and know who is calling by using uid, but since you have to be root to start the server - it is not so simple.

The last thing was to let developers an option to run in single process mode by:

  /usr/local/sbin/httpd_perl/httpd_perl -Dsbekman -X

In addition to making life easier, we decided to use relative links everywhere in the static docs (including the calls to CGIs). You may ask how using the relative link you will get to the right server? Very simple - we have utilized the mod_rewrite to solve our problems:

In access.conf of the httpd_docs server we have the following code: (you have to configure your httpd_docs server with --enable-module=rewrite )

  # sbekman' server
  # port = 8000
  RewriteCond  %{REQUEST_URI} ^/(perl|cgi-perl)  
  RewriteCond  %{REMOTE_ADDR} 123.34.45.56
  RewriteRule ^(.*)           http://nowhere.com:8000/$1 [R,L]
  
  # userfoo's server
  # port = 8001
  RewriteCond  %{REQUEST_URI} ^/(perl|cgi-perl)  
  RewriteCond  %{REMOTE_ADDR} 123.34.45.57
  RewriteRule ^(.*)           http://nowhere.com:8001/$1 [R,L]
  
  # all the rest
  RewriteCond  %{REQUEST_URI} ^/(perl|cgi-perl)  
  RewriteRule ^(.*)           http://nowhere.com:81/$1 [R]
  

where IP numbers are the IPs of the developer client machines (where they are running their web browser.) (I have tried to use REMOTE_USER since we have all the users authenticated but it did not work for me)

So if I have a relative URL like /perl/test.pl written in some html or even http://www.nowhere.com/perl/test.pl in my case (user at machine of sbekman) it will be redirected by httpd_docs to http://www.nowhere.com:8000/perl/test.pl.

Of course you have another problem: The CGI generates some html, which should be called again. If it generates a URL with hard coded PORT the above scheme will not work. There 2 solutions:

First, generate relative URL so it will reuse the technique above, with redirect (which is transparent for user) but it will not work if you have something to POST (redirect looses all the data!).

Second, use a general configuration module which generates a correct full URL according to REMOTE_USER, so if $ENV{REMOTE_USER} eq 'sbekman', I return http://www.nowhere.com:8000/perl/ as cgi_base_url. Again this will work if the user is authenticated.

All this is good for development. It is better to use the full URLs in production, since if you have a static form and the Action is relative but the static document located on another server, pressing the form's submit will cause a redirect to mod_perl server, but all the form's data will be lost during the redirect.

[TOC]


Wrapper to emulate the server environment

Many times you start off debugging your script by running it from your favorite shell. Sometimes you encounter a very weird situation when script runs from the shell but dies when called as a CGI. The real problem lies in the difference between the environment that is being used by your server and your shell. An example can be a different perl path or having PERL5LIB env variable which includes paths that are not in the @INC of the perl compiled with mod_perl server and configured during the startup.

The best debugging approach is to write a wrapper that emulates the exact environment of the server, by first deleting the environment variables like PERL5LIB and calling the same perl binary that it is being used by the server. Next, set the environment identical to the server's by copying the perl run directives from server startup and configuration files. It will also allow you to remove completely the first line of the script - since mod_perl skips it and the wrapper knows how to call the script.

Below is the example of such a script. Note that we force the -Tw when we call the real script. (I have also added the ability to pass params, which will not happen when you call the cgi from the web)

  #!/usr/local/bin/perl -w    
   
  # This is a wrapper example 
   
  # It simulates the web server environment by setting the @INC and other
  # stuff, so what will run under this wrapper will run under web and
  # vice versa. 
  
  #
  # Usage: wrap.pl some_cgi.pl
  #
  
  BEGIN{
    use vars qw($basedir);
    $basedir = "/usr/local";
  
    # we want to make a complete emulation, 
    # so we must remove the user's environment
    @INC = ();
  
    # local perl libs
    push @INC,
      qw($basedir/lib/perl5/5.00502/aix
         $basedir/lib/perl5/5.00502
         $basedir/lib/perl5/site_perl/5.005/aix
         $basedir/lib/perl5/site_perl/5.005
        );
  }
  
  use strict;
  use File::Basename;
  
    # process the passed params
  my $cgi = shift || '';
  my $params = (@ARGV) ? join(" ", @ARGV) : '';
  
  die "Usage:\n\t$0 some_cgi.pl\n" unless $cgi;
  
    # Set the environment
  my $PERL5LIB = join ":", @INC;
  
    # if the path includes the directory 
    # we extract it and chdir there
  if ($cgi =~ m|/|) {
    my $dirname = dirname($cgi);
    chdir $dirname or die "Can't chdir to $dirname: $! \n";
    $cgi =~ m|$dirname/(.*)|;
    $cgi = $1;
  }
  
    # run the cgi from the script's directory
    # Note that we invoke warnings and Taint mode ON!!!
  system qq{$basedir/bin/perl -I$PERL5LIB -Tw $cgi $params};

[TOC]


Log Rotation

A little bit off topic but good to know and use with mod_perl where your error_log can grow at a 10-100Mb per day rate if your scripts spit out lots of warnings...

To rotate the logs do:

  mv access_log access_log.renamed
  kill -HUP `cat httpd.pid`
  sleep 10; # allow some children to complete requests and logging
  # now it's safe to use access_log.renamed
  .....

The effect of SIGUSR1 and SIGHUP is detailed in: http://www.apache.org/docs/stopping.html .

I use this script:

  #!/usr/local/bin/perl -Tw
  
  # this script does a log rotation. Called from crontab.
  
  use strict;
  $ENV{PATH}='/bin:/usr/bin';
  
  ### configuration
  my @logfiles = qw(access_log error_log);
  umask 0;
  my $server = "httpd_perl";
  my $logs_dir = "/usr/local/var/$server/logs";
  my $restart_command = "/usr/local/sbin/$server/apachectl restart";
  my $gzip_exec = "/usr/bin/gzip";
  
  my ($sec,$min,$hour,$mday,$mon,$year) = localtime(time);
  my $time = sprintf "%0.2d.%0.2d.%0.2d-%0.2d.%0.2d.%0.2d", $year,++$mon,$mday,$hour,$min,$sec;
  $^I = ".".$time;
  
  # rename log files
  chdir $logs_dir;
  @ARGV = @logfiles;
  while (<>) {
    close ARGV;
  }
  
  # now restart the server so the logs will be restarted
  system $restart_command;
  
  # compress log files
  foreach (@logfiles) {
      system "$gzip_exec $_.$time";
  }

Randal L. Schwartz contributed this:

Cron fires off setuid script called log-roller that looks like this:

    #!/usr/bin/perl -Tw
    use strict;
    use File::Basename;
    
    $ENV{PATH} = "/usr/ucb:/bin:/usr/bin";
    
    my $ROOT = "/WWW/apache"; # names are relative to this
    my $CONF = "$ROOT/conf/httpd.conf"; # master conf
    my $MIDNIGHT = "MIDNIGHT";  # name of program in each logdir
    
    my ($user_id, $group_id, $pidfile); # will be set during parse of conf
    die "not running as root" if $>;
    
    chdir $ROOT or die "Cannot chdir $ROOT: $!";
    
    my %midnights;
    open CONF, "<$CONF" or die "Cannot open $CONF: $!";
    while (<CONF>) {
      if (/^User (\w+)/i) {
        $user_id = getpwnam($1);
        next;
      }
      if (/^Group (\w+)/i) {
        $group_id = getgrnam($1);
        next;
      }
      if (/^PidFile (.*)/i) {
        $pidfile = $1;
        next;
      }
     next unless /^ErrorLog (.*)/i;
      my $midnight = (dirname $1)."/$MIDNIGHT";
      next unless -x $midnight;
      $midnights{$midnight}++;
    }
    close CONF;
    
    die "missing User definition" unless defined $user_id;
    die "missing Group definition" unless defined $group_id;
    die "missing PidFile definition" unless defined $pidfile;
    
    open PID, $pidfile or die "Cannot open $pidfile: $!";
    <PID> =~ /(\d+)/;
    my $httpd_pid = $1;
    close PID;
    die "missing pid definition" unless defined $httpd_pid and $httpd_pid;
    kill 0, $httpd_pid or die "cannot find pid $httpd_pid: $!";
    
    
    for (sort keys %midnights) {
      defined(my $pid = fork) or die "cannot fork: $!";
      if ($pid) {
        ## parent:
        waitpid $pid, 0;
      } else {
        my $dir = dirname $_;
        ($(,$)) = ($group_id,$group_id);
        ($<,$>) = ($user_id,$user_id);
        chdir $dir or die "cannot chdir $dir: $!";
        exec "./$MIDNIGHT";
        die "cannot exec $MIDNIGHT: $!";
      }
    }
    
    kill 1, $httpd_pid or die "Cannot sighup $httpd_pid: $!";

And then individual MIDNIGHT scripts can look like this:

    #!/usr/bin/perl -Tw
    use strict;
    
    die "bad guy" unless getpwuid($<) =~ /^(root|nobody)$/;
    my @LOGFILES = qw(access_log error_log);
    umask 0;
    $^I = ".".time;
    @ARGV = @LOGFILES;
    while (<>) {
      close ARGV;
    }

Can you spot the security holes? Our trusted user base can't or won't. :) But these shouldn't be used in hostile situations.

[TOC]


Preventing from modperl process from going wild

Sometimes calling an undefined subroutine in a module can cause a tight loop that consumes all memory. Here is a way to catch such errors. Define an autoload subroutine:

  sub UNIVERSAL::AUTOLOAD {
    my $class = shift;
    warn "$class can't \$UNIVERSAL::AUTOLOAD!\n";
  }

It will produce a nice error in error_log, giving the line number of the call and the name of the undefined subroutine.

Sometimes an error happens and causes the server to write millions of lines into your error_log file and in a few minutes to put your server down on its knees. For example I get an error Callback called exit show up in my error_log file many times. The error_log file grows to 300 Mbytes in size in a few minutes. You should run a cron job to make sure this does not happen and if it does to take care of it. Andreas J. Koenig is running this shell script every minute:

  S=`ls -s /usr/local/apache/logs/error_log | awk '{print $1}'`
  if [ "$S" -gt 100000 ] ; then
    mv  /usr/local/apache/logs/error_log /usr/local/apache/logs/error_log.old
    /etc/rc.d/init.d/httpd restart
    date | /bin/mail -s "error_log $S kB on inx" myemail@domain.com
  fi

It seems that his script will trigger restart every minute, since once the logfile grows to be of 100000 lines, it will stay of this size, unless you remove or rename it, before you do restart. On my server I run a watchdog every five minutes which restarts the server if it is getting stuck (it always works since when some modperl child process goes wild, the I/O it causes is so heavy that other brother processes cannot normally to serve the requests.) See Monitoring the Server for more hints.

Also check out the daemontools from ftp://koobera.math.uic.edu/www/daemontools.html :

  ,-----
  | cyclog writes a log to disk. It automatically synchronizes the log
  | every 100KB (by default) to guarantee data integrity after a crash. It
  | automatically rotates the log to keep it below 1MB (by default). If
  | the disk fills up, cyclog pauses and then tries again, without losing
  | any data.
  `-----

[TOC]


The Writing Apache Modules with Perl and C book can be purchased online from O'Reilly and Amazon.com.
Your corrections of either technical or grammatical errors are very welcome. You are encouraged to help me to improve this guide. If you have something to contribute please send it directly to me.
[ Prev | Main Page | Next ]

Written by Stas Bekman.
Last Modified at 08/05/1999
Mod Perl Icon Use of the Camel for Perl is
a trademark of O'Reilly & Associates,
and is used by permission.