Matar a los trabajadores antes de tiempo y en el despliegue

Hey Roman A.!

Ok, good question - this can be a bit of a confusing part - and it depends on your setup and deploy process.

If I run messenger:stop-workers command, supervisor anyway will restart all workers. Did I understand correctly?

Yes, you do understand correctly!

Here's the super important thing:

A) On deploy, you somehow need to stop/restart your workers so that they see the new code. The messenger:stop-workers is one easy way to do this.

B) BUT, however you stop/restart your workers, it needs to be done gracefully. What I mean is, you can't stop the workers in the middle of handling a message (as you correctly already knew). The messenger:stop-workers command does this.

So basically, your issue is a bit specific to Docker / your deployment strategy. It looks like (tell me if I'm wrong) that you're deploying with Docker. And so, as part of your deploy, you stop the old containers. Is that right? If so, you probably don't need to call messenger:stop-workers because stopping the container will stop those workers anyways. But that doesn't solve your problem. The real important part is that you need to "kill" your old "worker" containers gracefully. If you send a SIGTERM, then Messenger will finish its current message and THEN exit - https://github.com/symfony/symfony/blob/f4ff77cc0867d46c944594451c46689aa55c5ffb/src/Symfony/Component/Messenger/EventListener/StopWorkerOnSigtermSignalListener.php#L24 - so you should keep the old containers alive until that happens. Also if you are using Docker, you might not need supervisor. A Docker container itself is centered around a process. It depends on your setup, but if you start a container that runs a worker, often people will have another Docker mechanism set up so that if that process (and thus the container) ever quits, a new one is restarted. That's sort of... built-in supervisor :).

Let me know if this makes sense. A lot of this depends on how you're deploying... and I'm making a lot of assumptions about that... so I could be totally wrong :).

Cheers!

Hi, @weaverryan ! Thank you for your good answer!

You understood correctly that I'm deploying with Docker. My deploying strategy looks like:
1. Pull new docker image from the registry
2. Stop service by docker-compose down
3. Delete old containers
4. Start a new container with service

Will my workers die gracefully If I just use "docker-compose down"? Must I call "messenger:stop-workers" before I kill my container?

It is very interesting about workers without the supervisor. Could you explain a little bit more about how I can avoid using the supervisor and use only Docker? I understand that it is kinda out of Symfony scope, but maybe you can share some links about that theme?

Hey Roman A.!

I'm not a Docker expert, so take what I say with a "grain of salt", but I'll do my best to give you some advice :).

> Will my workers die gracefully If I just use "docker-compose down"? Must I call "messenger:stop-workers" before I kill my container?

It will shutdown gracefully. Well, more specifically, here's what happens:

A) docker-compose down sends a TERM signal to the process
B) If messenger:consume is currently handling a message, it will *ignore* TERM (which is basically a "request" to terminate) until it finishes the message. Once it finishes the message fully, it will exit.

The one catch is that, by default, docker-compose down will only "wait" 10 seconds for the process to quit before it takes the container down anyways. That's the "timeout" argument on that command and it's configurable. So you should configure it to be higher than the longest a message should take to handle, to avoid the container quitting too early.

> It is very interesting about workers without the supervisor. Could you explain a little bit more about how I can avoid using the supervisor and use only Docker?

I'm not sure how you're configuring Docker now, but here are two different ways of doing things:
A) You start a container that runs supervisor that runs messenger:consume. If you do this, I think it will all work correctly - when you "down" the Docker container, that should kill both supervisor and messenger:consume "gracefully", but I've not tested it.

B) You start a container whose "process" is actually "php bin/console messenger:consume" itself - along with some argument like --memory-limit=128M (you should ALWAYS pass some flags like this to messenger:consume so that it doesn't run forever - regardless of how you're running all of this). This will mean that the process will exit occasionally... which means that the container will shut down. Certain docker infrastructure setups are built to handle this: you will have other pieces that constantly restart containers (if I understand things correctly) as soon as they exit. So basically, whenever the process exits (and so, the container stops), something else re-creates the container.

I hope that helps. I'm VERY much not an expert on the infrastructure & deployment stuff with Docker - so I'm talking at a "high level" here :).

Cheers!

Just a tip: processing TERM signal in messenger requires pcntl php extension.
Another tip: supervisor has option stopwaitsecs (The number of seconds to wait for the OS to return a SIGCHLD to supervisord after the program has been sent a stopsignal. If this number of seconds elapses before supervisord receives a SIGCHLD from the process, supervisord will attempt to kill it with a final SIGKILL. Default: 10). You should probably increase it to prevent undesired sigkills.

Hey Thomas,

Sounds correct, yes. Well, if you're using a Symfony Cloud / Platform.sh for example - it should be already done behind the scene for you. But with your custom deploy system the strategy should be like this I think. Well, probably we can even simplify it to this:

1. Deploy the new code
2. Run messenger:stop-workers command

Then the supervisor should take care of re-running the workers again for you. I think it should be enough too.

Cheers!

Hi,

IIRC Stopping supervisor will stop all workers it created, so you don't need to run messenger:stop-workers if you are stopping supervisor. This stop-workers command is helpful when you need to restart messenger without stopping supervisor.

Cheers

Hey apphancer!

First a wee note: on Ubuntu, ps -A is not returning information on the messenger processes, however ps -S seems to be doing the trick

Ha! This is why we can't have nice things :p. I just checked man ps on my Mac and then on an (older) Ubuntu version and both of these flags basically had completely different meanings. 🙃 Thanks for the note.

My problem is that when running messenger:stop-workers, the worker does not stop probably because of APCu

That is possible - it's possible that your CLI php is not using APCu. Unfortunately (but by design), if the caching system fails (like because the APCu extension isn't installed for your CLI php), the cache component doesn't fail - it just doesn't cache (it's done this way so that a caching failure doesn't take down your site). I would run php -m at your terminal and check to see if apcu is there. Ubuntu is funny because they usually use a different php.ini file for your CLI vs the web. And so, you might have it configured only for one but not the other. Run php --ini to see what the CLI php.init path is.

It looks like when I comment out the app: '%cache_adapter%' line, messenger:stop-workers starts working

This definitely tells me that the APCU cache is failing in the CLI... or in both places. Btw, if you wanted, you could temporarily put debug code in this class - https://github.com/symfony/symfony/blob/5.x/src/Symfony/Component/Cache/Adapter/ApcuAdapter.php - to see what's going on, like in the constructor and also in doSave().

Let me know what you find out!

Cheers!

I also was mislead by this `ps -A`, maybe adding a note for linux users: go for the classic `ps aux`

Hey Jean-Christophe,

Thank you for sharing your solution with others and confirming it works for you on Linux systems!

Cheers!

php -m returns apcu as present. But my bad: the result of bin/console messenger:stop-workers is a beautiful green [OK] Signal successfully sent to stop any running workers. which made me miss out the line above: WARNING [cache] Failed to save key "workers.restart_requested_timstamp" of type double...
So I went into the ini file for PHP cli and added apc.enable_cli = On. I am no longer getting that warning now, but the worker is still not receiving the stop signal.

I've added a bit of debug code into the ApcuAdapter class. Inside the doSave() method when running bin/console messenger:stop-workers, it looks like it is successfully persisting the data, e.g.

array(1) {
    [0]=>
    string(46) "XxIXZBaWIQ:workers.restart_requested_timestamp"
}

This same array is passed in doFetch() every few seconds while the worker is running. However, apcu_fetch($ids, $ok) returns an empty array therefore return $values; is also empty.
Not sure if this is something due to a misconfiguration with APCu. Everything works fine with the PDO adapter, so probably will leave it to PDO considering soon I will probably switch to Redis anyway.

Hey apphancer!

Good debugging! Unfortunately, I don't have an answer about why it's ultimately not seeing that cache key - it's definitely a misconfiguration with APCu, but I can't imagine *what* the misconfiguration is. It might not be worth the time digging in if you have other caching options - I would just use them :).

Cheers!

It would be awesome if you could pass in a "max age" for the "deliveredAt" property so the worker will re-consume the message if "deviveredAt" is > X seconds old.

I didn't see anything obvious to do that in the vendor code....

acually I DO SEE!
vendor/symfony/messenger/Transport/Doctrine/Connection.php:line 281

    private function createAvailableMessagesQueryBuilder(): QueryBuilder
    {
        $now = new \DateTime();
        $redeliverLimit = (clone $now)->modify(sprintf('-%d seconds', $this->configuration['redeliver_timeout']));

        return $this->createQueryBuilder()
            ->where('m.delivered_at is null OR m.delivered_at < ?')
            ->andWhere('m.available_at <= ?')
            ->andWhere('m.queue_name = ?')
            ->setParameters([
                $redeliverLimit,
                $now,
                $this->configuration['queue_name'],
                . . .

Now I just need to figure out if I can set "$this->configuration['redeliver_timeout']", it's default is 3600, 1 hour! 15 minutes would be more then enough for me. :)

I am a dope! RTFM, https://symfony.com/doc/cur..., under options "redeliver_timeout"

Can't see the forest for the trees. ugh, quitting time.

lol :)
I'm glad you could fix your problem. Thanks for sharing your solution

Cheers!

Hey JBMan!

Hmm, an excellent question! What command are you using to clear the cache? The cache "pool" that's created for the "restart" signal uses cache.app as its parent. And cache.app is meant to be a store that is persistent across deploys. That's a fancy way of saying that Symfony has two main cache systems: cache.system (which is cleared between deploys) and cache.app (which persists between deploys). So... it shouldn't be a problem. But, what are you seeing?

Cheers!

Using symfony for many years and first time read about difference of meaning for two cache systems!

Hey Yanosh,

Yeah, probably still a little-known fact. I'm glad you found something interesting for you :)

Cheers!

PHP Deployer creates a completely new directory when deploying a new version of my app. So var/cache will be empty after the deployment.

I didn't know cache.app is meant to be persistent across deploys. Based on your answer I started using Redis for the cache.app pool. After that the worker restarts work fine.

Thank you very much for your help!

Hey JBMan!

Woo! Awesome :).

I didn't know cache.app is meant to be persistent across deploys. Based on your answer I started using Redis for the cache.app pool.

Well-done. Yes, I don't know if the purpose of cache.app is as obvious as it should be - I might tweak the recipe to add some more comments around it in cache.yaml. Anyways, I'm glad we got it sorted!

Cheers!

Oh my gosh... duh! Thanks for the ping on that