18

How do I ensure sub-processes are stopped when I stop Supervisord?

I'm using Supervisord to run two Celery workers. The command for each worker is:

command=/usr/local/myapp/src/manage.py celery worker --concurrency=1 --loglevel=INFO

When I start supervisord, they run perfectly, but when I run sudo service supervisord stop, it reports my workers stop:

celery_1: stopped
celery_2: stopped

However, when I run ps aux | grep celery I see they're still running:

www-data 27667  0.0  1.5 318420 64360 ?        S    10:45   0:00 manage.py celery worker --concurrency=1 --loglevel=INFO
www-data 27668  0.0  1.5 318420 318416 ?        S    10:45   0:00 manage.py celery worker --concurrency=1 --loglevel=INFO
Improve this question
3
  • Hey @Certin, did you come to a solution for this? Commented Sep 26, 2013 at 11:57
  • would be interested in a solution as well .. Commented Nov 13, 2013 at 5:00
  • Cerin: Did you find a solution for that ? Commented Mar 12, 2014 at 16:07

5 Answers 5

Reset to default
12

i believe stopping the supervisor service stops the supervisor daemon, not the supervisor process that is managing your celeryworkers

supervisorctl stop all should allow you stop the workers, and also allow you to start/restart them

http://supervisord.org/running.html#running-supervisorctl

Improve this answer
Sign up to request clarification or add additional context in comments.

1 Comment

The signal that supervisor sends to the celery workers is not going to kill them immediately, if they are still processing tasks at that moment they will keep running until those tasks are done. So it might take a long time for your workers to shut down even after calling supervisorctl stop all
9

The reason I ended up here is that when I was using supervisor to start and stop my celery processes, the celery workers were not stopping, which lead to an accumulation of workers.

I tried various settings in supervisor, such as stopsignal, and killasgroup. None of them stopped the workers.

At this point, it appears that the celery process does not send signals to the workers when it is stopped.

What I ended up doing was to add:

ps auxww | grep 'celery worker' | awk '{print $2}' | xargs kill -9

to my scripts. This is from the celery docs

At the end of this command, I tried kill -TERM instead of kill -9, hoping that would kill the processes more gracefully. But that always left one process.

Improve this answer

2 Comments

You should rather use pgrep. Using ps | grep may have some issues such as the grep process being itself in the list. pgrep -f 'celery worker' | xargs kill -9 or simply pkill -f 'celery worker'.
I added the stopasgroup=true to the [program:x] section of supervisor (v4.2.0) and it worked as a charm. Might be more favorable then manual intervention in some cases.
1

Not sure, but it seams like in this case, supervisor is managing sh scripts. Not python threads. Can you display the ps auxf | grep celery , while supervisord are running? May be editing of the command, like this: command=python /path/to/manage.py ..., will be helpful.

Improve this answer

Comments

1

Take a look in your supervisord.log. Maybe supervisord is sending a SIGKILL to celery? If it is, try to increase the timeout (e.g. stopwaitsecs=600).

https://github.com/celery/celery/issues/102

Improve this answer

Comments

1

I've had such issues with django channels, settings stopasgroup=true in suppervisor config (as suggested here) fixed the issue. I don't understand why, however.

Improve this answer

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.