Async Emails with Messenger

Hey Christophe R.!

Hmm, yes this sounds tricky. Ok, a few questions:

A) You mentioned that your email is sent by a queue. When you have this problem, are you actually routing the emails to some sort of "async" queue? And then when you run bin/console messenger:consume, THAT is when the email is sent (and the styles are missing)? Or are you sending to Messenger, but then the message is actually handled "sync". If it is this situation, you might have this problem: https://github.com/symfony/symfony/issues/34972

B) In general, I think that your message is either being rendered multiple times for some reason (and so, the 2nd render has no styles) or you are handling your messages in a worker... and after it sends the first email, encore needs to be reset so that it works properly for the 2nd message... though I think that Encore is automatically reset between handling messages via messenger:consume... but I'm not 100% positive.

To help debug, I would put some debugging code inside this method - https://github.com/symfony/symfony/blob/5.4/src/Symfony/Component/Mailer/Messenger/MessageHandler.php#L29-L32 - that's the Messenger handler class that is called to actually SEND the email message. I would be interested to see if it's being called multiple times.

Let me know what you find out :).

Cheers!

Thanks for your answer!

This morning, in local dev I did another test, I Put a dump($message) in MessageHandler.php and I ask 2 requests passwords (email). They are in the queue (so no email sent) and in the profiler, the email section looks to be ok.

When I consumed, the first message has the css and the second not (but it is 2 differents email because 2 tokens differents for the reset)

Here the dump

I follow all your courses about API Platform, async email and symfony stuff so my code inspired by you and I added the reset :
$this->entrypointLookup->reset();
in the constructor of my Mailer service. (May be it need to be another place?)

If I send message with MESSENGER as doctrine default, I have the same issue.

If I remove messenger from composer, It looks to be ok each time, So I am thinking the issue is with Messenger and Mailer...

I am not sure to understand the A), but I think it is the case where the email looks to be good when sending in the queue and when I consume, css is missing with the second one (with my morning example)

Tricky fact, If when I sent another email in the queue while consume is running, the css is missing.
IF I stop consume, resent email, reconsume, the email has his css...

Hey Christophe R.!

This is excellent debugging :). So the issue is fairly simple. When you run Messenger (without restarting), the first email will look correct. But then, when it sends a second email, nothing has called $entrypointLookup->reset() yet... so the CSS is missing. Adding that line to your Mailer service won't help, because that code isn't being executed in the worker: it's calling Symfony's mailer service directly.

So, hmm. I know how to fix this... but I'm surprised that I haven't seen this before as this should be affecting anyone who is using Encore in their emails with Messenger. I thought (but now I think I was wrong) that Messenger was "resetting" the Symfony services between messages (which would call reset() on the entrypoint lookup. We use emails and Messenger on our site, but we don't use Encore yet (we had a system already that built the email HTML & CSS outside of Symfony).

Anyways, here's the fix: register a listener/subscriber on the WorkerMessageReceivedEvent event. Inside, if the message is an instance of SendEmailMessage - https://github.com/symfony/symfony/blob/5.4/src/Symfony/Component/Mailer/Messenger/SendEmailMessage.php - then call $this->entrypointLookup->reset().

That'll reset the Encore stuff before each email is actually delivered. On my end, I need to add a note about this to the tutorial. Let me know how it goes and if you need any more guidance!

Cheers!

Actually, here is the class you'll need: https://gist.github.com/wea... - try that out and let me know if you have any issues :)

Wowwww Ryan, you are too strong! It's WORK ! Thank you!
I will test in prod next week but no more issue in local dev!

Awesome! Thank you for confirming the fix! We'll add a note to the video for it soon - I appreciate you brining up the issue :)

Hey Nick F.!

Oh no! Do you remember (other than the xsl extension) what Composer issues you had? Did you download the "start" code and then upgrade to 4.4? Or just start a new 4.4 project and "adapt" the code into it? If there is an issue, I'd like to know so that we can smooth it out.

[messenger] Error thrown while handling message Symfony\Component\Mailer\Messenger\SendEmailMessage. Sending for retry #1 using 1000 ms delay. Error: "Connection to "smtp.mailtrap.io:2525" timed out."
["message" => Symfony\Component\Mailer\Messenger\SendEmailMessage { …},"class" => "Symfony\Component\Mailer\Messenger\SendEmailMessage","retryCount" => 1,"delay" => 1000,"error" => "Connection to "smtp.mailtrap.io:2525 " timed out.","exception" => Symfony\Component\Messenger\Exception\HandlerFailedException { …}]
And it tries this 3 times before removing the message from the transport.
BUT the emails ARE being sent. So why is there an error?

Hmm. Is the email sent just ONE time, or is it sent 3 times? I suspect just once - but I want to be sure. Also, if you stop running messenger:consume and THEN trigger the email to be sent (i.e. register), is the email sent? I can't think of why this would be happening: you are clearly "routing" the email to an async transport, which means that Messenger will no longer be handling it sync. However, my first guess is that is IS somehow being handed sync and then also sent to the transport. But... as I said, that's just a guess.

[messenger] Error thrown while handling message Symfony\Component\Mailer\Messenger\SendEmailMessage. Sending for retry #2 using 2000 ms delay. Error: "Expected response code "354" but got code "250",

Hmm. So, in the SMTP internals, when Symfony sends a message it does this:

1) it Sends a "DATA" line, which tells the SMTP server that you're about to starting sending data. the SMTP server should reply with a 354 code, which means "Start mail input"

2) After Sending the data, Symfony sends a special ending syntax and expects the SMTP server to reply with 250, meaning "Requested action was completed".

For some reason, the SMTP server that you're using seems to be replying to the first with a 250. I have no idea why it would do that. Here's the core part of Symfony that I'm referencing: https://github.com/symfony/symfony/blob/083a32dee34bc0cb67f4d81262a39c132f8097e0/src/Symfony/Component/Mailer/Transport/Smtp/SmtpTransport.php#L202-L207

Is this coming from Mailtrap? Or something else? Let me know what you find out!

Cheers!

Hey Matthias,

Thank you for kind words about SymfonyCasts tutorial! I'm really happy to hear it!

Hm, EmailMiddleware is something that you created, right? You created this for getting access to SentMessage? Are you sure your middleware is called in the correct order? And are you sure your middleware is called at all? Probably you need to call you middleware a bit later?

Did you try to debug the stacktrace with Blackfire? It might help to track the chain of calls that you can follow and if there's any chances to get that SentMessage object.

P.S. I looked into code a bit, and it looks like when we're talking about Messenger, the message is sent in MessageHandler::__invoke(), i.e. there's where the send() method is called which in turn return SentMessage object, see: https://github.com/symfony/... - but I don't see any return statements in this MessageHandler::__invoke() that would help to access SentMessage with Messenger :/

I hope this helps somehow. Please, let us know if it's not.

Cheers!

OMG. It seems that it's the missing "return" in the default MessageHandler. When i update the Class with the return statement, i can access the SentMessage Object from the getResult Method delivered by HandledStamp. Cool.

So that i can go forward with my project code without change symfony codebase, i created my own MessageHandler to handle the SendEmailMessage. Registered it over the services tag "mailer.messenger.message_handler". A kind of expected, the handlers are both registered.

services.yaml
`
services :
[...]

App\MessageHandler\SendEmailMessageHandler:
    tags:
        - name: mailer.messenger.message_handler
          handles: Symfony\Component\Mailer\Messenger\SendEmailMessage

`

result of debug:messenger
`
The following messages can be dispatched:

Symfony\Component\Mailer\Messenger\SendEmailMessage

  handled by App\MessageHandler\SendEmailMessageHandler  
  handled by mailer.messenger.message_handler            

`

Is there an other way to really overwrite or replace the default MessageHandler? Because at the moment both handler sending an Email ... again: as expected :)

Looking really forward to the "final fix".

Hey Matthias,

Glad this workaround works for you! I wonder if that "return" statement should be in the Symfony code base by default... probably it's a good idea to create a PR about it and see the feedback of community.

Hm, I suppose it should be sent only once... how does your messenger.yaml file config look like? Especially your routing section there. Also, I though the correct tag should be "messenger.message_handler" according to the docs, no? See https://symfony.com/doc/cur...

Cheers!

Upon further investigation it looks like Symfony is trying to serialize all the Doctrine Objects passed to the mail via 'context' and though I haven't found the closure yet, it is probably somewhere inside those entities. But IMHO it is definetly not neccessary to serialize all of that just for an async mail to be sent. I'd still like to know how to handle closures with Messenger but for now it should suffice to render the mail template and then serialize it. Is there a way to do so?

EDIT: Okay so I've found at least a closure. As mentioned above all the Doctrine Objects are passed to the serializer. And with them Entity Manger Object(s ?). They have LazyObjectState and there is an initializer that is a Closure.

... Leading me back to my question: Is it really neccessary to serialize ALL THAT? And how come this is not a common problem? I am probably not the only one using Doctrine Object when creating an email.

EDIT2: And a similar problem was just described over at Stackoverflow. Looks like those Lazy Ghost Entity Managers introduced in 6.2 are causing some trouble...

Hey Eric,

Yes, the problem is that when you make your messages async - you force them to be saved somewhere, e.g. in the DB, before sending via Messenger, and so to be properly saved they should serialize objects you pass to the emails. But thankfully, there's a simple solution for this! Just stop passing objects into your email template :) If you need a user first name and email in that - you don't need to pass the whole object like you do with page templates - just pass simple scalar values of user's first name and email.

Otherwise, if you really don't want to get rid of passing objects to the email template - then you probably need to worry about configuring your entities to be serialized properly. But I think the best practice is to pass just simple scalar values to the template instead.

Cheers!

Hey Victor,
thanks for the reply. I actually didn't take into consideration just passing scalar values. But as I also happen to be a lazy dev, I made it work with passing objects. It is probably a bit context dependend but my usecase is as follows: After handling a form a confirmation mail is send. This mail makes use of objects from the form data.
All this Lazy Ghost EntityManager Shenanigan gets inserted into the objects after they are flushed. So just before I do a clone
` $entityDTO = clone $entityObject;

        $entityManager->persist($entityObject);
        $entityManager->flush();
        ...
        $email = (new TemplatedEmail())
                    -> [...]
                    -> context([
                        'entity' => $entityDTO
                    ]);`

Looks like it all works just fine that way.

Hey Eric,

Interesting solution, thanks for sharing it with others! Glad to hear it works for you, but just in case add some tests to be calm on upgrades :)

Cheers!

Hey Dang

An option would be to listen to the event Symfony\Component\Messenger\Event\WorkerMessageHandledEvent. Then, you can check if the message instance inside the envelop is equals to your email message. Does it make sense?

Cheers!

Hey MolloKhan ,
Thanks a lot for that hint. I've created a Event Subsciber to listen to that Symfony\Component\Messenger\Event\WorkerMessageHandledEvent.
It works. But actually, my email is sending inside a MessageHandler of "other" Message (I call the mailer->send(email) inside MessageHandler). So when I listen to WorkerMessageHandledEvent, I get back the Message when it done, but not the event of the mailer it self. I can see the line "Message Symfony\Component\Mailer\Messenger\SendEmailMessage handled by Symfony\Component\Mailer\Messenger\MessageHandler::__invoke" while the worker running. And after that is my MeesageHandler is handled.
So can I catch the event while the workers is running and for exact the mailer event?

Hey Trinh, sorry for my slow reply.

So, the email is sent via another message handler, in that case, your subscriber should be triggered twice, isn't it? one per each message been handled, unless, Symfony Messenger does not trigger the event when a message is dispatched inside a handler... I'm not sure about that but if that's the case, an option would be that you dispatch a custom event inside your MessageHandler class so you can listen to it

Hey Steven,

Yes, that's a known WebpackEncoreBundle problem, you need to reset its state every time you send an email. Take a look at the docs: https://github.com/symfony/... . And actually, we covered this issue in this course: https://symfonycasts.com/sc... - you probably missed this :)

So, bring back the Messenger and try this solution ;)

I hope this helps!

Cheers!

Thank you Victor! I had skipped the PDF video as I wasn't working with PDFs.

Hey Steven,

Ah, now I see why you missed this :) Yeah, a known problem for Webpack Encore, you just should be aware of it to be able to solve when needed. It's not a bug actually, it's just a compromise that makes Encore shine in other cases - it prevents rendering script and link tags twice when you use template inheritance in Twig :)

Cheers!

Hey Metin,

Yeah, I think I know your problem. It's similar to request context when you generate URLs in async emails - because there's no request, you need to set the request context to generate links properly, i.e. see https://symfony.com/doc/4.1... . What about translation - it's the same, no request, so no locale. This probably mean that you have to set the locale manually. You can do it via translator globally e.g. "$translator->setLocale('fr');" for example, or pass the correct locale to the "trans()" method - it should be possible via the last argument, see method signature. I'd recommend to try the first way and set the correct local manually on the translator before trying to translate.

I suppose you can even do it globally via events, listen to WorkerMessageReceivedEvent::class and set the correct locale - it should give you correct context when your emails will be sent async. I'm not sure, but probably you will also need to set the locale on ConsoleEvents::class event as well, but it might be redundant if your problem is only in async emails.

I hope this helps!

Cheers!

Hey Shaun,

I don't know for sure, have never done this before, but I'd say no. Thanks to the new Symfony Mailer, messages are just data objects now, and so you can easily serialize/deserialize them. It was a problem with SwiftMailer because its messages were "heavy". I'd recommend you to refactor to Symfony Mailer to be able to use it with Messenger component that should work out of the box without extra coding. And most probably it shouldn't be a hard refactor, but depends on your project though.

I hope this helps!

Cheers!

Hey Thibault V.!

I believe that this is, unfortunately, just the "nature of worker processes". It's not just your inky templates that are affected - if you change any PHP code while developing locally, you need to restart your workers so that your workers see the new code. This is why we gracefully restart workers on deploy. But, I can give you 2 options:

1) In dev, use a "sync://" transport. Then only use your real transport in the prod environment (I can point you how to do this if you need a little pointer).

2) If you use the symfony binary, it has a little-known feature to "watch" for changes and restart your workers for you. It's this:

symfony run -d --watch=config,src,templates,vendor symfony console messenger:consume async

That runs your messenger:consume command in the background as a daemon and restarts it automatically whenever it sees changes to the listed directories.

Let me know if any of this helps ;).

Cheers!

Hey weaverryan,

Thanks for this command, it worked like a charm :)

Cheers !

Hi Christina V.!

Excellent question :). So if you're sending a reminder for an event, my guess is that you would do this with some sort of CRON job? For example, you would run a CRON job every 1 minute... or 5 minutes, query the database to find all emails that need to be sent and then send them? If you were thinking of some totally different way, then let me know.

Assuming I'm right, then, as you said, you might find that you need to send 20 emails all at once. If you did not use Messenger and use Mailer directly, then, if something fails, you'll get a normal exception. You could certainly wrap the sending in a "try-catch" and NOT mark it as sent, so that it would be tried again on the next loop. Here is some fake code to illustrate:

$eventRegistrants = [...];
foreach ($eventRegistrants as $eventRegistrant) {
    // ...

    try { 
        $this->mailer->send(...);
        $eventRegistration->setReminderEmailSent(true); // used to know that this was sent!
    } catch (\Exception $e) {
        // do nothing, don't mark the "registrant email" as sent so that
        // we will try to send it again on the next loop
        // maybe log an error :) 
    }
}

This is a fine setup. Like anything, you need to code carefully to avoid problems if suddenly 10,000 emails need to be sent at once, but there are no real problems with this. Also, Mailer itself as a way to specify a "fallback" transport. This allows Mailer to automatically try sending through a 2nd transport if the first transport fails for some reason.

If you decide to use Messenger, then the error handling is pushed into a different spot. First, you would not need the try-catch: you would just do $this->mailer->send() and that's it. In reality, this would "queue" the email to be sent. Then, if there IS a problem delivering the email from your "worker", Messenger's automatic retry system will takeover. By default, Messenger will automatically try to "process" a message 3 times (with some delays in between) before sending it to a "failure queue" so you can inspect what happened and retry.

So in both situations, you can handle your failures gracefully. Messenger would really be more about performance or scalability in this setup: in theory your one CRON job could very quickly queue 1000 new emails to be sent, and then multiple workers could share the load of sending those emails quickly. But, you may not have that problem :).

Let me know if this helps!

Cheers!

Hey @weaverryan.

Sorry for the late reply. With this corona crisis, I'm out of space !

Well, I don't have a CRON (yet) but yes, at the end, this is something that we plan to have.
After you reply, I get deeper in the new Messenger Component, I missed some stuff and now it's all crystal clear!

Thanks for your reply and coments.
Always a pleasure to learn from here.

Cheers

huh tried again today and it works correctly. Nothing has changed in my config. Could it be a "cache" issue?

Hey Lydie,

Hm, weird... yeah, sounds like a cache issue... Especially if we're talking about prod. In dev mode cache should recompile... but in case you use some virtualization tools and sharing files between host and target machines - it might not always work correctly, though works most of the time. Anyway, in any weird situation it would be always a good idea to try to clear the cache first and see if it helps. If not - dig further :)

I'm happy it works for you! And thanks for mentioning it works now!

Cheers!

Hello!

In fact I was talking about local development (but with db on a remote server). I do not really understand what happened but even clearing manually the cache did not work. I think it worked again after closing phpstorm and stopping local server. If it appears again, I will just try that and see if it helps :)

Hey Lydie,

Hm, interesting. I wonder what platform do you use? Are you on Windows? Probably start with stop/start the webserver... if it does not help, only then try to close/open PhpStorm. Though, it's weird that you have to do these steps.

Cheers!