Celery with Django in production

0. Background
1. Create the dedicated user and group
2. Create the celery configuration file
3. Create the systemd file
4. Restart the server
5. References

0. Background

Three months ago, I deployed my react, Django project on a ubuntu server. I used celery for a task queue. Today I tried to look celery environment and I got confused about the files and my celery configuration. Now In this post, I will try to explain all the things that you should know about the celery configuration on the server.

If you never heard about celery, check out the celery official documentation, First steps with celery and First steps with Django before reading this post.

In the celery official documentation, you will find the three possible ways to daemoniz your celery with python, i.e. init-script, Init-script: celerybeat and using systemd. In this post, I will only explain about the systemd demonization.

1. Create the dedicated user and group

First of all, you need to create the dedicated user and group as celery,

<span>sudo </span>groupadd celery
<span>sudo </span>useradd <span>-g</span> celery celery
<span>sudo </span>groupadd celery
<span>sudo </span>useradd <span>-g</span> celery celery
sudo groupadd celery
sudo useradd -g celery celery

Enter fullscreen mode Exit fullscreen mode

2. Create the celery configuration file

Now let’s create the celery configuration file inside the /etc/default/celeryd directory,

Note: Please replace the user with your ubuntu user, CELERY_APP_NAME, CELERYD_CHDIR, and CELERY_BIN.

<span># most people will only start one node:</span>
<span>CELERYD_NODES</span><span>=</span><span>"worker1"</span>
<span># but you can also start multiple and configure settings</span>
<span># for each in CELERYD_OPTS</span>
<span>#CELERYD_NODES="worker1 worker2 worker3"</span>
<span># alternatively, you can specify the number of nodes to start:</span>
<span>#CELERYD_NODES=10</span>
<span># Absolute or relative path to the 'celery' command:</span>
<span>CELERY_BIN</span><span>=</span><span>"/home/user/.virtualenvs/venv/bin/celery"</span>
<span>#CELERY_BIN="/virtualenvs/def/bin/celery"</span>
<span># App instance to use</span>
<span># comment out this line if you don't use an app</span>
<span>CELERY_APP</span><span>=</span><span>"celery_app_name"</span>
<span># or fully qualified:</span>
<span>#CELERY_APP="proj.tasks:app"</span>
<span># Where to chdir at start.</span>
<span>CELERYD_CHDIR</span><span>=</span><span>"/home/user/django-project/"</span>
<span># Extra command-line arguments to the worker</span>
<span>CELERYD_OPTS</span><span>=</span><span>"--time-limit=300 --concurrency=8"</span>
<span># Configure node-specific settings by appending node name to arguments:</span>
<span>#CELERYD_OPTS="--time-limit=300 -c 8 -c:worker2 4 -c:worker3 2 -Ofair:worker1"</span>
<span># Set logging level to DEBUG</span>
<span>#CELERYD_LOG_LEVEL="DEBUG"</span>
<span># %n will be replaced with the first part of the node name.</span>
<span>CELERYD_LOG_FILE</span><span>=</span><span>"/var/log/celery/%n%I.log"</span>
<span>CELERYD_PID_FILE</span><span>=</span><span>"/var/run/celery/%n.pid"</span>
<span># Workers should run as an unprivileged user.</span>
<span># You need to create this user manually (or you can choose</span>
<span># a user/group combination that already exists (e.g., nobody).</span>
<span>CELERYD_USER</span><span>=</span><span>"celery"</span>
<span>CELERYD_GROUP</span><span>=</span><span>"celery"</span>
<span>CELERYD_LOG_LEVEL</span><span>=</span><span>"INFO"</span>
<span># If enabled PID and log directories will be created if missing,</span>
<span># and owned by the userid/group configured.</span>
<span>CELERY_CREATE_DIRS</span><span>=</span>1
<span># most people will only start one node:</span>
<span>CELERYD_NODES</span><span>=</span><span>"worker1"</span>
<span># but you can also start multiple and configure settings</span>
<span># for each in CELERYD_OPTS</span>
<span>#CELERYD_NODES="worker1 worker2 worker3"</span>
<span># alternatively, you can specify the number of nodes to start:</span>
<span>#CELERYD_NODES=10</span>

<span># Absolute or relative path to the 'celery' command:</span>
<span>CELERY_BIN</span><span>=</span><span>"/home/user/.virtualenvs/venv/bin/celery"</span>
<span>#CELERY_BIN="/virtualenvs/def/bin/celery"</span>

<span># App instance to use</span>
<span># comment out this line if you don't use an app</span>
<span>CELERY_APP</span><span>=</span><span>"celery_app_name"</span>
<span># or fully qualified:</span>
<span>#CELERY_APP="proj.tasks:app"</span>

<span># Where to chdir at start.</span>
<span>CELERYD_CHDIR</span><span>=</span><span>"/home/user/django-project/"</span>

<span># Extra command-line arguments to the worker</span>
<span>CELERYD_OPTS</span><span>=</span><span>"--time-limit=300 --concurrency=8"</span>
<span># Configure node-specific settings by appending node name to arguments:</span>
<span>#CELERYD_OPTS="--time-limit=300 -c 8 -c:worker2 4 -c:worker3 2 -Ofair:worker1"</span>

<span># Set logging level to DEBUG</span>
<span>#CELERYD_LOG_LEVEL="DEBUG"</span>

<span># %n will be replaced with the first part of the node name.</span>
<span>CELERYD_LOG_FILE</span><span>=</span><span>"/var/log/celery/%n%I.log"</span>
<span>CELERYD_PID_FILE</span><span>=</span><span>"/var/run/celery/%n.pid"</span>

<span># Workers should run as an unprivileged user.</span>
<span># You need to create this user manually (or you can choose</span>
<span># a user/group combination that already exists (e.g., nobody).</span>
<span>CELERYD_USER</span><span>=</span><span>"celery"</span>
<span>CELERYD_GROUP</span><span>=</span><span>"celery"</span>
<span>CELERYD_LOG_LEVEL</span><span>=</span><span>"INFO"</span>
<span># If enabled PID and log directories will be created if missing,</span>
<span># and owned by the userid/group configured.</span>
<span>CELERY_CREATE_DIRS</span><span>=</span>1
# most people will only start one node:
CELERYD_NODES="worker1"
# but you can also start multiple and configure settings
# for each in CELERYD_OPTS
#CELERYD_NODES="worker1 worker2 worker3"
# alternatively, you can specify the number of nodes to start:
#CELERYD_NODES=10

# Absolute or relative path to the 'celery' command:
CELERY_BIN="/home/user/.virtualenvs/venv/bin/celery"
#CELERY_BIN="/virtualenvs/def/bin/celery"

# App instance to use
# comment out this line if you don't use an app
CELERY_APP="celery_app_name"
# or fully qualified:
#CELERY_APP="proj.tasks:app"

# Where to chdir at start.
CELERYD_CHDIR="/home/user/django-project/"

# Extra command-line arguments to the worker
CELERYD_OPTS="--time-limit=300 --concurrency=8"
# Configure node-specific settings by appending node name to arguments:
#CELERYD_OPTS="--time-limit=300 -c 8 -c:worker2 4 -c:worker3 2 -Ofair:worker1"

# Set logging level to DEBUG
#CELERYD_LOG_LEVEL="DEBUG"

# %n will be replaced with the first part of the node name.
CELERYD_LOG_FILE="/var/log/celery/%n%I.log"
CELERYD_PID_FILE="/var/run/celery/%n.pid"

# Workers should run as an unprivileged user.
# You need to create this user manually (or you can choose
# a user/group combination that already exists (e.g., nobody).
CELERYD_USER="celery"
CELERYD_GROUP="celery"
CELERYD_LOG_LEVEL="INFO"
# If enabled PID and log directories will be created if missing,
# and owned by the userid/group configured.
CELERY_CREATE_DIRS=1

Enter fullscreen mode Exit fullscreen mode

Now lets change the owner of the celery log and PID file,

<span>chown</span> <span>-R</span> celery:celery /var/log/celery/
<span>chown</span> <span>-R</span> celery:celery /var/run/celery/
<span>chown</span> <span>-R</span> celery:celery /var/log/celery/
<span>chown</span> <span>-R</span> celery:celery /var/run/celery/
chown -R celery:celery /var/log/celery/
chown -R celery:celery /var/run/celery/

Enter fullscreen mode Exit fullscreen mode

3. Create the systemd file

Now lets create the systemd file inside /etc/systemd/system/celery.service directory and write following content,

Note: Please replace the user with your ubuntu user and make sure to check the path to bin celery file (this file will available probably inside your virtual environment)

<span>[</span>Unit]
<span>Description</span><span>=</span>Celery Service
<span>After</span><span>=</span>network.target
<span>[</span>Service]
<span>Type</span><span>=</span>forking
<span>User</span><span>=</span>celery
<span>Group</span><span>=</span>celery
<span>EnvironmentFile</span><span>=</span>/etc/default/celeryd
<span>WorkingDirectory</span><span>=</span>/home/user/django-project
<span>ExecStart</span><span>=</span>/home/user/.virtualenvs/venv/bin/celery multi start <span>${</span><span>CELERYD_NODES</span><span>}</span> <span>\</span>
  <span>-A</span> <span>${</span><span>CELERY_APP</span><span>}</span> <span>--pidfile</span><span>=</span><span>${</span><span>CELERYD_PID_FILE</span><span>}</span> <span>\</span>
  <span>--logfile</span><span>=</span><span>${</span><span>CELERYD_LOG_FILE</span><span>}</span> <span>--loglevel</span><span>=</span><span>${</span><span>CELERYD_LOG_LEVEL</span><span>}</span> <span>${</span><span>CELERYD_OPTS</span><span>}</span>
<span>ExecStop</span><span>=</span>/home/user/.virtualenvs/venv/bin/celery <span>${</span><span>CELERY_BIN</span><span>}</span> multi stopwait <span>${</span><span>CELERYD_NODES</span><span>}</span> <span>\</span>
  <span>--pidfile</span><span>=</span><span>${</span><span>CELERYD_PID_FILE</span><span>}</span>
<span>ExecReload</span><span>=</span>/home/user/.virtualenvs/venv/bin/celery <span>${</span><span>CELERY_BIN</span><span>}</span> multi restart <span>${</span><span>CELERYD_NODES</span><span>}</span> <span>\</span>
  <span>-A</span> <span>${</span><span>CELERY_APP</span><span>}</span> <span>--pidfile</span><span>=</span><span>${</span><span>CELERYD_PID_FILE</span><span>}</span> <span>\</span>
  <span>--logfile</span><span>=</span><span>${</span><span>CELERYD_LOG_FILE</span><span>}</span> <span>--loglevel</span><span>=</span><span>${</span><span>CELERYD_LOG_LEVEL</span><span>}</span> <span>${</span><span>CELERYD_OPTS</span><span>}</span>
<span>[</span>Install]
<span>WantedBy</span><span>=</span>multi-user.target
<span>[</span>Unit]
<span>Description</span><span>=</span>Celery Service
<span>After</span><span>=</span>network.target

<span>[</span>Service]
<span>Type</span><span>=</span>forking
<span>User</span><span>=</span>celery
<span>Group</span><span>=</span>celery

<span>EnvironmentFile</span><span>=</span>/etc/default/celeryd
<span>WorkingDirectory</span><span>=</span>/home/user/django-project
<span>ExecStart</span><span>=</span>/home/user/.virtualenvs/venv/bin/celery multi start <span>${</span><span>CELERYD_NODES</span><span>}</span> <span>\</span>
  <span>-A</span> <span>${</span><span>CELERY_APP</span><span>}</span> <span>--pidfile</span><span>=</span><span>${</span><span>CELERYD_PID_FILE</span><span>}</span> <span>\</span>
  <span>--logfile</span><span>=</span><span>${</span><span>CELERYD_LOG_FILE</span><span>}</span> <span>--loglevel</span><span>=</span><span>${</span><span>CELERYD_LOG_LEVEL</span><span>}</span> <span>${</span><span>CELERYD_OPTS</span><span>}</span>
<span>ExecStop</span><span>=</span>/home/user/.virtualenvs/venv/bin/celery <span>${</span><span>CELERY_BIN</span><span>}</span> multi stopwait <span>${</span><span>CELERYD_NODES</span><span>}</span> <span>\</span>
  <span>--pidfile</span><span>=</span><span>${</span><span>CELERYD_PID_FILE</span><span>}</span>
<span>ExecReload</span><span>=</span>/home/user/.virtualenvs/venv/bin/celery <span>${</span><span>CELERY_BIN</span><span>}</span> multi restart <span>${</span><span>CELERYD_NODES</span><span>}</span> <span>\</span>
  <span>-A</span> <span>${</span><span>CELERY_APP</span><span>}</span> <span>--pidfile</span><span>=</span><span>${</span><span>CELERYD_PID_FILE</span><span>}</span> <span>\</span>
  <span>--logfile</span><span>=</span><span>${</span><span>CELERYD_LOG_FILE</span><span>}</span> <span>--loglevel</span><span>=</span><span>${</span><span>CELERYD_LOG_LEVEL</span><span>}</span> <span>${</span><span>CELERYD_OPTS</span><span>}</span>

<span>[</span>Install]
<span>WantedBy</span><span>=</span>multi-user.target
[Unit]
Description=Celery Service
After=network.target

[Service]
Type=forking
User=celery
Group=celery

EnvironmentFile=/etc/default/celeryd
WorkingDirectory=/home/user/django-project
ExecStart=/home/user/.virtualenvs/venv/bin/celery multi start ${CELERYD_NODES} \
  -A ${CELERY_APP} --pidfile=${CELERYD_PID_FILE} \
  --logfile=${CELERYD_LOG_FILE} --loglevel=${CELERYD_LOG_LEVEL} ${CELERYD_OPTS}
ExecStop=/home/user/.virtualenvs/venv/bin/celery ${CELERY_BIN} multi stopwait ${CELERYD_NODES} \
  --pidfile=${CELERYD_PID_FILE}
ExecReload=/home/user/.virtualenvs/venv/bin/celery ${CELERY_BIN} multi restart ${CELERYD_NODES} \
  -A ${CELERY_APP} --pidfile=${CELERYD_PID_FILE} \
  --logfile=${CELERYD_LOG_FILE} --loglevel=${CELERYD_LOG_LEVEL} ${CELERYD_OPTS}

[Install]
WantedBy=multi-user.target

Enter fullscreen mode Exit fullscreen mode

4. Restart the server

Now everything is setup. Now you just need to restart the service

<span>sudo </span>systemctl daemon-reload
<span>sudo </span>systemctl <span>enable </span>celery
<span>sudo </span>systemctl restart celery
<span>sudo </span>systemctl daemon-reload
<span>sudo </span>systemctl <span>enable </span>celery
<span>sudo </span>systemctl restart celery
sudo systemctl daemon-reload
sudo systemctl enable celery
sudo systemctl restart celery

Enter fullscreen mode Exit fullscreen mode

Now your celery is ready to use in production. You can check the status of the celery manually by typing following code,

celery <span>-A</span> CELERY_APP_NAME worker <span>-l</span> INFO
celery <span>-A</span> CELERY_APP_NAME worker <span>-l</span> INFO
celery -A CELERY_APP_NAME worker -l INFO

Enter fullscreen mode Exit fullscreen mode

5. References

1. Celery documentation
2. Daemonization celery in production: stackoverflow
3. Celery not running (Permission Denied): stackoverflow

原文链接：Celery with Django in production