For some reasons you will find yourself writing some magical lines of code or having a small/large application that needs to run all the time and shouldn't fail.
To accomplish this, we need something to keep an eye on that magic or application and ensure to keep it up and running all the time, here comes the usefulness of supervisor.
This story assumes that you are familiar with the command line and Sysadmin basics.
Supervisor is a simple and popular python application for process controlling and monitoring. on the other hand it's a program that runs and watch other programs to ensure they are executed all the time.
On the next lines, we are going to demonstrate how to use it effectively to maintain your applications for a better deployments.
To install Supervisor on Ubuntu or Debian, we can simply run the following as a super user:
apt-get install supervisor -y
Once it has completed, the supervisor daemon should be already running. To make sure it is, run the following command.
service supervisor status
The command will return 'is running', in case it's not. You can start it by running.
service supervisor start
Now we have done with the installation of the supervisor and ensured of supervisord running. We can write our first configuration file for our application.
I chose celery for this demonstration. Supervisor is a configuration file driven. We are going to write a configuration file for celery to be monitored and controlled by supervisor. Before that we are going to configure our supervisord web interface.
Configuration files for supervisor are located in /etc/supervisor folder, by adding the following lines to the supervisor configuration file supervisord.conf we are going to be able to access it through our browser.
By checking the the url http://127.0.0.1:9001 we can access the web interface of our supervisor doemon and it's empty now.
Supervisor comes also with supervisorctl, a client command line that allow us to control the processes. It provides a shell like interface to the features provided by supervisord. From supervisorctl, we can connect to different supervisord processes, get status on the subprocesses controlled by, stop and start subprocesses of, and get lists of running processes of a supervisord.
As we see on the previous screenshot we don't have any monitored process yet, so let's create a configuration file for celery application.
We need to create a new configuration file at /etc/supervisor/conf.d/ folder and we are going to call it celery.conf, the extension .conf is so important as we see on the supervisor configuration file, it includes any file with .conf extension.
[include] files = /etc/supervisor/conf.d/*.conf
Now we need to tell Supervisord how to run and monitor celery. We are going to put the following lines in our celery.conf
[program:celeryd] command=/usr/bin/celery worker -B --loglevel=INFO user=yusuf numprocs=1 stdout_logfile=/var/log/celery/worker.log stderr_logfile=/var/log/celery/worker.err autostart=true autorestart=true startsecs=10 stopwaitsecs = 600 killasgroup=true priority=998
Now let's check the web interface again, we see that our monitored application appeared on the list.
We can use the client command line by running
In case there are no errors on our config file it will return something like that, telling us that our application is running and monitored by supervisor.
celeryd RUNNING pid 27, uptime 0:05:07
##Supervisor and Virtualenv
On the previous celery configuration, we installed celery globally on our system. For many reasons python developers prefer to use virtualenv for both development and production mode, so we are going to modify our previous configuration, in case celery is installed on a python virtual environment.
Our celery.conf will looks like that.
[program:celeryd] environment=PYTHONPATH='/path/to/your/env' command=/path/to/your/env/bin/celery worker --loglevel=INFO command=/usr/bin/celery worker -B --loglevel=INFO user=yusuf numprocs=1 stdout_logfile=/var/log/celery/worker.log stderr_logfile=/var/log/celery/worker.err autostart=true autorestart=true startsecs=10 stopwaitsecs = 600 killasgroup=true priority=998
The PYTHONPATH section is setting up the python path to the virtualenv. So when supervisor goes to execute the command it will be executing it within the virtualenv.
Superlance is a python package of plugin utilities for monitoring and controlling processes that run under supervisor by listening to supervisord events.
It can be installed easily by typing the following command:
pip install superlance
Superlance comes with five even listeners:
This plugin is meant to be used as a supervisor event listener, subscribed to TICK_* events. It tests that a given child process which must in the RUNNING state, is viable via an HTTP GET request to a configured URL. If the request fails or times out, httpok will restart the “hung” child process.
This plugin is meant to be used as a supervisor event listener, subscribed to PROCESS_STATE_EXITED events. It email a user when a process enters the EXITED state unexpectedly.
This plugin is meant to be used as a supervisor event listener, subscribed to TICK_* events. It monitors memory usage for configured child processes, and restarts them when they exceed a configured maximum size.
Similar to crashmail, crashmailbatch sends email alerts when processes die unexpectedly. The difference is that all alerts generated within the configured time interval are batched together to avoid sending too many emails.
This plugin sends email alerts when processes fail to start too many times such that supervisord gives up retrying. All of the fatal start events generated within the configured time interval are batched together to avoid sending too many emails.
Similar to crashmailbatch except it sends SMS alerts through an email gateway. Messages are formatted to fit in SMS.
In this story we are going to talk only about crashmail and configure it for our needs.
We are going to edit supervisord.conf file by adding the following lines to the configuration file:
[eventlistener:crashmail] command=crashmail -a -m [email protected] events=PROCESS_STATE_EXITED
If you want to get notified only for a specific application, for example celeryd in our case:
[eventlistener:crashmail] command=crashmail -p celeryd -m [email protected] events=PROCESS_STATE_EXITED
For more information about superlancer i recommend you to check their documentation at http://superlance.readthedocs.org/en/latest/
There are currently no comments. You can be first!Sign in