0
Feb. 18, 2017  
In Articles by Iunary

For some reasons you will find yourself writing some magical lines of code or having a small/large application that needs to run all the time and shouldn't fail.

To accomplish this, we need something to keep an eye on that magic or application and ensure to keep it up and running all the time, here comes the usefulness of supervisor.

This story assumes that you are familiar with the command line and Sysadmin basics.

Intro

Supervisor is a simple and popular python application for process controlling and monitoring. on the other hand it's a program that runs and watch other programs to ensure they are executed all the time.

On the next lines, we are going to demonstrate how to use it effectively to maintain your applications for a better deployments.

Installation and configuration

Installation

To install Supervisor on Ubuntu or Debian, we can simply run the following as a super user:

apt-get install supervisor -y

Once it has completed, the supervisor daemon should be already running. To make sure it is, run the following command.

service supervisor status

The command will return 'is running', in case it's not. You can start it by running.

service supervisor start

Now we have done with the installation of the supervisor and ensured of supervisord running. We can write our first configuration file for our application.

Configuration

I chose celery for this demonstration. Supervisor is a configuration file driven. We are going to write a configuration file for celery to be monitored and controlled by supervisor. Before that we are going to configure our supervisord web interface.

Configuration files for supervisor are located in /etc/supervisor folder, by adding the following lines to the supervisor configuration file supervisord.conf we are going to be able to access it through our browser.

[inet_http_server]
port=*:9001

By checking the the url http://127.0.0.1:9001 we can access the web interface of our supervisor doemon and it's empty now.

image

Supervisor comes also with supervisorctl, a client command line that allow us to control the processes. It provides a shell like interface to the features provided by supervisord. From supervisorctl, we can connect to different supervisord processes, get status on the subprocesses controlled by, stop and start subprocesses of, and get lists of running processes of a supervisord.

As we see on the previous screenshot we don't have any monitored process yet, so let's create a configuration file for celery application.

We need to create a new configuration file at /etc/supervisor/conf.d/ folder and we are going to call it celery.conf, the extension .conf is so important as we see on the supervisor configuration file, it includes any file with .conf extension.

[include]
files = /etc/supervisor/conf.d/*.conf

Now we need to tell Supervisord how to run and monitor celery. We are going to put the following lines in our celery.conf

[program:celeryd]
command=/usr/bin/celery worker -B --loglevel=INFO
user=yusuf
numprocs=1
stdout_logfile=/var/log/celery/worker.log
stderr_logfile=/var/log/celery/worker.err
autostart=true
autorestart=true
startsecs=10
stopwaitsecs = 600
killasgroup=true
priority=998

Now let's check the web interface again, we see that our monitored application appeared on the list.

We can use the client command line by running

supervisorctrl status

In case there are no errors on our config file it will return something like that, telling us that our application is running and monitored by supervisor.

celeryd RUNNING pid 27, uptime 0:05:07

##Supervisor and Virtualenv

On the previous celery configuration, we installed celery globally on our system. For many reasons python developers prefer to use virtualenv for both development and production mode, so we are going to modify our previous configuration, in case celery is installed on a python virtual environment.

Our celery.conf will looks like that.

[program:celeryd]
environment=PYTHONPATH='/path/to/your/env'
command=/path/to/your/env/bin/celery worker --loglevel=INFO
command=/usr/bin/celery worker -B --loglevel=INFO
user=yusuf
numprocs=1
stdout_logfile=/var/log/celery/worker.log
stderr_logfile=/var/log/celery/worker.err
autostart=true
autorestart=true
startsecs=10
stopwaitsecs = 600
killasgroup=true
priority=998

The PYTHONPATH section is setting up the python path to the virtualenv. So when supervisor goes to execute the command it will be executing it within the virtualenv.

Supervisor and Superlance

Superlance is a python package of plugin utilities for monitoring and controlling processes that run under supervisor by listening to supervisord events.

Installation

It can be installed easily by typing the following command:

pip install superlance

Superlance comes with five even listeners:

  • httpok

    This plugin is meant to be used as a supervisor event listener, subscribed to TICK_* events. It tests that a given child process which must in the RUNNING state, is viable via an HTTP GET request to a configured URL. If the request fails or times out, httpok will restart the “hung” child process.

  • crashmail

    This plugin is meant to be used as a supervisor event listener, subscribed to PROCESS_STATE_EXITED events. It email a user when a process enters the EXITED state unexpectedly.

  • memmon

    This plugin is meant to be used as a supervisor event listener, subscribed to TICK_* events. It monitors memory usage for configured child processes, and restarts them when they exceed a configured maximum size.

  • crashmailbatch

    Similar to crashmail, crashmailbatch sends email alerts when processes die unexpectedly. The difference is that all alerts generated within the configured time interval are batched together to avoid sending too many emails.

  • fatalmailbatch

    This plugin sends email alerts when processes fail to start too many times such that supervisord gives up retrying. All of the fatal start events generated within the configured time interval are batched together to avoid sending too many emails.

  • crashsms

    Similar to crashmailbatch except it sends SMS alerts through an email gateway. Messages are formatted to fit in SMS.

In this story we are going to talk only about crashmail and configure it for our needs.

We are going to edit supervisord.conf file by adding the following lines to the configuration file:

[eventlistener:crashmail]
command=crashmail -a -m [email protected]
events=PROCESS_STATE_EXITED

If you want to get notified only for a specific application, for example celeryd in our case:

[eventlistener:crashmail]
command=crashmail -p celeryd -m [email protected]
events=PROCESS_STATE_EXITED

For more information about superlancer i recommend you to check their documentation at http://superlance.readthedocs.org/en/latest/


0 Comments

There are currently no comments. You can be first!

Sign in