Flag of Ukraine
SymfonyCasts stands united with the people of Ukraine

Deployment & Supervisor

Keep on Learning!

If you liked what you've learned so far, dive in!
Subscribe to get access to this tutorial plus
video, code and script downloads.

Start your All-Access Pass
Buy just this tutorial for $12.00

With a Subscription, click any sentence in the script to jump to that part of the video!

Login Subscribe

So... how does all of this work on production? It's a simple problem really: on production, we somehow need to make sure that this command - messenger:consume - is always running. Like, always.

Some hosting platforms - like SymfonyCloud - allow you to do this with some simple configuration. You basically say:

Yo Cloud provider thingy! Please make sure that bin/console messenger:consume is always running. If it quits for some reason, start a new one.

If you're not using a hosting platform like that, it's ok - but you will need to do a little bit of work to get that same result. And actually, it's not just that we need a way to make sure that someone starts this command and then it runs forever. We actually don't want the command to run forever. No matter how well you write your PHP code, PHP just isn't meant to be ran forever - eventually your memory footprint will increase too much and the process will die. And... that's perfect! We don't want our process to run forever. Nope: what we really want is for messenger:consume to run, handle... a few messages... then close itself. Then, we'll use a different tool to make sure that each time the process disappears, it gets restarted.

Hello Supervisor

The tool that does that is called supervisor. After you install it, you give it a command that you always want running and it stays up all night constantly eating pizza and watching to make sure that command is running. The moment it stops running, for any reason, it puts down the pizza and it restarts the command.

So let's see how Supervisor works and how we can use it to make sure our worker is always running. Because I'm using a Mac, I already installed Supervisor via Brew. If you're using Ubuntu, you can install it via apt. By the way, you don't actually need to install & configure Supervisor on your local machine: you only need it on production. We're installing it so we can test and make sure everything works.

Supervisor Configuration

To get it going, we need a supervisor configuration file. Google for "Messenger Symfony" and open the main documentation. In the middle... there's a spot that talks about supervisor. Copy the configuration file. We could put this anywhere: it doesn't need to live in our project. But, I like to keep it in my repo so I can store it in git. In... how about config/, create a new file called messenger-worker.ini and paste the code inside.

[program:messenger-consume]
command=php /path/to/your/app/bin/console messenger:consume async --time-limit=3600
user=ubuntu
numprocs=2
autostart=true
autorestart=true
process_name=%(program_name)s_%(process_num)02d

The file tells Supervisor which command to run and other important info like which user it should run the process as and the number of processes to run. This will create two worker processes. The more workers you run, the more messages can be handled at once. But also, the more memory & CPU you'll need.

Now, locally, I don't need to run supervisor... because we can just manually run messenger:consume. But to make sure this all works, we're going to pretend like my computer is production and change the path to point to use my local path: /Users/weaverryan/messenger... which if I double-check in my terminal... oop - I forgot the Sites/ part. Then, down here, I'll change the user to be weaverryan. Again, you would normally set this to your production values.

Oh, and if you look closely at the command, it's running messenger:consume async. Make sure to also consume async_priority_high. The command also has a --time-limit=3600 option. We'll talk more about this and some other options in a bit, but this is great: it tells the worker to run for 60 minutes and then exit, to make sure it doesn't get too old and take up too much memory. As soon as it exits, Supervisor will restart it.

Running Supervisor

Now that we have our config file, we need to make sure Supervisor can see it. Each Supervisor install has a main configuration file. On a Mac where it's installed via Brew, that file is located at /usr/local/etc/supervisord.ini. On Ubuntu, it should be /etc/supervisor/supervisord.conf.

Then, somewhere in your config file, you'll find an include section with a files line. This means that Supervisor is looking in this directory to find configuration files - like ours - that will tell it what to do.

To get our configuration file into that directory, we can create a symlink: ln -s ~/Sites/messenger/config/messenger-worker.ini then paste the directory.

ln -s ~/Sites/messenger/config/messenger-worker.ini /usr/local/etc/supervisor.d/

Ok! Supervisor should now be able to see our config file. To run supervisor, we'll use something called supervisorctl. Because I'm on a Mac, I also need to pass a -c option and point to the configuration file we were just looking at. If you're on Ubuntu, you shouldn't need to do this - it'll know where to look already. Then say reread: that tells Supervisor to reread the config files:

supervisorctl -c /usr/local/etc/supervisord.ini reread

By the way, you may need to run this command with sudo. If you do, no big deal: it will execute the processes themselves as the user in your config file.

Cool! It sees the new messager-consume group. That names comes from the key at the top of our file. Next, run the update command... which would restart any processes with the new config... if they were already running... but our's aren't yet:

supervisorctl -c /usr/local/etc/supervisord.ini update

To start them, run start messenger-consume:*:

supervisorctl -c /usr/local/etc/supervisord.ini start messenger-consume:*

That last argument - messenger-consume:* isn't very obvious. When you create a "program" called messenger-consume, this creates what's called a "homogeneous process group". Because we have processes=2, this group will run two processes. By saying messenger-consume:* it tells Supervisor to start all processes inside that group.

When we run it... it doesn't say anything... but... our worker commands should now be running! Let's go stop our manual worker so that only the ones from Supervisor are running. Now,

tail -f var/log/messenger.log

This will make it really obvious whether or not our messages are being handled by those workers. Now, upload a few photos, delete a couple of items, move over and... yea! It's working! It's actually working almost twice as fast as normal because we have twice the workers.

And, now we can have some fun. First, we can see the process id's created by Supervisor by running:

ps -A | grep messenger:consume

Tip

You can also use ps aux, which will work on more operating systems.

There they are: 19915 and 19916. Let's kill one of those:

kill 19915

And run that again:

ps -A | grep messenger:consume

Yes! 19916 is still there but because we killed the other one, supervisor started a new process for it: 19995. Supervisor rocks.

Next, let's talk more about the options we can use to purposely make workers exit before they take up too much memory. We'll also talk about how to restart workers on deploy so that they see the new code and a little detail about how things can break if you update your message class.

Leave a comment!

24
Login or Register to join the conversation
Tim K. Avatar

In case anybody has the same problem, that the installation of SUPERVISOR is not allowed on production-server, I solved this with a workaround by using a cronjob (starting every 5 minutes) with this bash-script:


#!/bin/sh

# check with '$ which php' and add to variable
PHP_FOLDER="/usr/bin/php"

# give your root folder of your symfony-project
SYMFONY_FOLDER="$HOME/your/symfony/project"

# define the transporters that shall be treated
TRANSPORTERS="async_1 async_2"

# define timezone to avoid mismatch between php for console and php for web (or server). Different
# timezones, would delay the treatment of the messages in the queues. E.g. check timezone with
# SERVER time: $ echo $TZ
# PHP-Console time: $ php -r 'echo date_default_timezone_get();'
TIMEZONE="Europe/Berlin"

# Show folder (main intention is to exit if folder not exists)
echo "Change folder to:"
cd "$SYMFONY_FOLDER" || exit
pwd

echo ""
echo "Stop any running Workers"
env -i "$PHP_FOLDER" -q "$SYMFONY_FOLDER"/bin/console messenger:stop-workers

echo ""
echo "Starting Worker (messenger:consume)"
env -i "$PHP_FOLDER" -d date.timezone=$TIMEZONE -q "$SYMFONY_FOLDER"/bin/console messenger:consume -vv $TRANSPORTERS --memory-limit=100M

Happy coding!

1 Reply

Hey Tim,

Thank you for sharing your workaround with others! Good idea to start the command via CRON ;)

Cheers!

Reply
Default user avatar
Default user avatar Rémi Dck | posted 2 years ago

Hello again !
A tiny precision, wich I faced : on ubuntu server I had to precise the directory where the supervisor should find the app so my worker looks like :
[program:messenger-consume]
command=php bin/console messenger:consume async --time-limit=3600
directory=/var/www/html/my-app

One other thing about supervisor, it seems he prefer now working with .conf files and not .ini (easy to fix, but you have to think about this --> http://supervisord.org/runn...

Are aware of a trouble for the FlySystem to delete files in a handler? I can't do it on production while I'm ok on local dev (still searching btw)

1 Reply

Just in case, the correct link is: http://supervisord.org/runn...

Reply

Hey Remi,

Yeah, it makes sense. In our example we use the absolute path so it works. If you use a relative path like only "bin/console" - you have to specify the directory of your project. Thank you for some tips! Might be useful for other users.

About problem with deleting files in a handler. Hm, most probably you have different permissions, so it sounds like permission problem. Is it for all the folders? Or some of them are removed successfully but some are not?

Cheers!

Reply
Rémi W. Avatar
Hm, most probably you have different permissions, so it sounds like permission problem. Is it for all the folders? Or some of them are removed successfully but some are not?


Mmm... Not sure because my filesystem works perfectly in the controller in production... I'll explore further and keep you update (i didn't try it syncronously yet to make a differential diagnosis)

Reply

Hey @Rémi Dck

I also believe that the file deletion problem is due to permissions. I think you just have to tell supervisor to run your worker as your web user (usually named as "www"). Here is an example of how to do it but I didn't give it a try: https://stackoverflow.com/a...

Cheers!

Reply
Default user avatar

OMG you're the best Diego. Thanks a lot !!!
I dug in and the tail is a little bit tricky (to me, i'm not a unix user...)
The first step is finding the web user, like you said and the best command for this is less /etc/passwd. For Ubuntu 18.04 seems to be www-data
Next in messenger-worker.conf change user name and add environnement targeting the home directory of www-data
Aaaand done ! Love it.

1 Reply

Awesome! Excellent job man because I wrote the name wrong, it usually is www-data not just www :p

Reply
Marko Avatar

Hi. What is/are the possible strategy/strategies to use if the Consumer throws an exception for any reason ? Assuming that the environment is Production. Thanks.

Reply

Hey Marko,

The best strategy is to do a few more retries after some time, and if still no success - move that message into a failed messages queue where you can retry the message manually when the problem will be fixed. That's the strategy we use on SymfonyCasts, and IIRC we covered it in this course :)

Cheers!

Reply
Marko Avatar

Hello Victor! Thanks a lot for your quick answer. Does Supervisor have any role in this issue (in production) ?

Reply

Hey Marko,

Supervisor is about a different problem, it helps to restart the worker on the server, so it's a bit different. If for some reasons you mean that your worker crashes, i.e. stops on the production that you have to restart it manually again - then yes, you need that supervisor that will restart the worker for you. But the problems you described relates to different things if I understand you correctly.

Cheers!

Reply
Rainer S. Avatar
Rainer S. Avatar Rainer S. | posted 2 years ago

I really hate asking this question.
Is there an easy windows solution?

Reply

Sven

Yeah that's a good question, I'm hot sure but probably this link https://github.com/alexsilv... will help you, but I think you already saw this link. Unfortunately I can't advise you some real production example for windows, also there is not so many windows servers, that's why there is not so many solutions for it!

Cheers! Hope it will help!

Reply
Chalice Avatar
Chalice Avatar Chalice | posted 2 years ago

Has anyone by any chance run supervisor on an AWS EB setup? (Elastic Beanstalk, using .ebextension files)

Reply

Hey Chad,

Unfortunately, I have not run it personally. Though it looks like there's some solutions around the internet, e.g.: https://gist.github.com/vra...

I hope this helps!

Cheers!

Reply
Chalice Avatar

Thanks, this is one of the articles I used to get things going.

Reply

Hey Chad,

Ah, great! Sorry, can't say something more about it, have never done it before. Probably someone who use it could suggest you a good tip about it.

Cheers!

Reply
Chalice Avatar

the following `.ebextension` config file is working to install supervisor and start messenger:consume, but the `.env` file is required to be populated with all the database credentials and such

files:
"/tmp/messenger-consume.conf":
mode: "000644"
owner: root
group: root
content: |
[program:messenger-consume]
command=php /var/www/html/bin/console messenger:consume async --time-limit=1800 --memory-limit=340M --sleep=30
user=webapp
numprocs=1
autostart=true
autorestart=true
process_name=%(program_name)s_%(process_num)02d

"/tmp/supervisord.conf":
mode: "000644"
owner: root
group: root
content: |
; supervisor config file

[unix_http_server]
file=/var/run/supervisor.sock ; (the path to the socket file)
chmod=0700 ; sockef file mode (default 0700)

[supervisord]
user=root
logfile=/var/log/supervisor/supervisord.log ; (main log file;default $CWD/supervisord.log)
pidfile=/var/run/supervisord.pid ; (supervisord pidfile;default supervisord.pid)
childlogdir=/var/log/supervisor ; ('AUTO' child log dir, default $TEMP)

; the below section must remain in the config file for RPC
; (supervisorctl/web interface) to work, additional interfaces may be
; added by defining them in separate rpcinterface: sections
[rpcinterface:supervisor]
supervisor.rpcinterface_factory = supervisor.rpcinterface:make_main_rpcinterface

[supervisorctl]
serverurl=unix:///var/run/supervisor.sock ; use a unix:// URL for a unix socket

; The [include] section can just contain the "files" setting. This
; setting can list multiple files (separated by whitespace or
; newlines). It can also contain wildcards. The filenames are
; interpreted as relative to this file. Included files *cannot*
; include files themselves.

[include]
files = /etc/supervisor/conf.d/*.conf
; Change according to your configurations

commands:
01_01_install_supervisord:
command: |
sudo easy_install supervisor
sudo mkdir -p /etc/supervisor/conf.d/
sudo mkdir -p /var/log/supervisor
sudo touch /var/log/supervisor/supervisord.log
ignoreErrors: true

container_commands:
01_01_copy_supervisor_configuration_files:
command: |
sudo mv /tmp/supervisord.conf /etc/supervisord.conf
sudo mv /tmp/messenger-consume.conf /etc/supervisor/conf.d/messenger-consume.conf
ignoreErrors: true
02_01_start_supervisord:
command: |
sudo /usr/local/bin/supervisord -c /etc/supervisord.conf
sudo /usr/local/bin/supervisorctl reread
sudo /usr/local/bin/supervisorctl update
sudo /usr/local/bin/supervisorctl restart all
sudo /usr/local/bin/supervisorctl start messenger-consume:*
ignoreErrors: true
Reply

Hey Chad,

Thank you for sharing this solution with others! I'm glad you were able to get it working.

Cheers!

Reply
Ajie62 Avatar

Hello,

When running supervisorctl -c /usr/local/etc/supervisord.ini start messenger-consume:*, I get zsh: no matches found: messenger-consume:*. But it seems the workers are actually working, because I tried to upload an image and it worked. In the video, there's no message after you start the supervisor with messenger-consume:*. So I'm just wondering if I'm doing something wrong...

Thanks!

Reply

Hey Jérôme !

Hmm. Try this command to see what's going on:


supervisorctl -c /usr/local/etc/supervisord.ini start

I definitely don't like that "no matches found". Also try using the stop command to see if it stops anything. Here's a screenshot of what starting and stopping looks on my machine - you can see how "start" sometimes doesn't have any output (if it's already started) but sometimes does.

https://imgur.com/Pz601gU

Also, the zsh: no matches found: messenger-consume:* is odd... the error should come from supervisorctl... not from "zsh". Try surrounding messenger-consume:* with quotes. I don't know zsh well... but I'm wondering if it's somehow interpreting something differently...

Cheers!

Reply
Ajie62 Avatar

Hey, thanks for you answer! The problem was oh-my-zsh. I removed it an now everything works fine. I assume I could have solved the problem and keep oh-my-zsh, but I wanted to save time and keep learning :) Thank you again, weaverryan

Reply
Cat in space

"Houston: no signs of life"
Start the conversation!

This tutorial is built with Symfony 4.3, but will work well on Symfony 4.4 or 5.

What PHP libraries does this tutorial use?

// composer.json
{
    "require": {
        "php": "^7.1.3",
        "ext-ctype": "*",
        "ext-iconv": "*",
        "composer/package-versions-deprecated": "^1.11", // 1.11.99
        "doctrine/annotations": "^1.0", // v1.8.0
        "doctrine/doctrine-bundle": "^1.6.10", // 1.11.2
        "doctrine/doctrine-migrations-bundle": "^1.3|^2.0", // v2.0.0
        "doctrine/orm": "^2.5.11", // v2.6.3
        "intervention/image": "^2.4", // 2.4.2
        "league/flysystem-bundle": "^1.0", // 1.1.0
        "phpdocumentor/reflection-docblock": "^3.0|^4.0", // 4.3.1
        "sensio/framework-extra-bundle": "^5.3", // v5.3.1
        "symfony/console": "4.3.*", // v4.3.2
        "symfony/dotenv": "4.3.*", // v4.3.2
        "symfony/flex": "^1.9", // v1.18.7
        "symfony/framework-bundle": "4.3.*", // v4.3.2
        "symfony/messenger": "4.3.*", // v4.3.4
        "symfony/property-access": "4.3.*", // v4.3.2
        "symfony/property-info": "4.3.*", // v4.3.2
        "symfony/serializer": "4.3.*", // v4.3.2
        "symfony/validator": "4.3.*", // v4.3.2
        "symfony/webpack-encore-bundle": "^1.5", // v1.6.2
        "symfony/yaml": "4.3.*" // v4.3.2
    },
    "require-dev": {
        "easycorp/easy-log-handler": "^1.0.7", // v1.0.7
        "symfony/debug-bundle": "4.3.*", // v4.3.2
        "symfony/maker-bundle": "^1.0", // v1.12.0
        "symfony/monolog-bundle": "^3.0", // v3.4.0
        "symfony/stopwatch": "4.3.*", // v4.3.2
        "symfony/twig-bundle": "4.3.*", // v4.3.2
        "symfony/var-dumper": "4.3.*", // v4.3.2
        "symfony/web-profiler-bundle": "4.3.*" // v4.3.2
    }
}