Async Emails with Messenger

Keep on Learning!

If you liked what you've learned so far, dive in!
Subscribe to get access to this tutorial plus
video, code and script downloads.

Start your All-Access Pass
Buy just this tutorial for $12.00

With a Subscription, click any sentence in the script to jump to that part of the video!

Login Subscribe

Sending an email - like after we complete registration - takes a little bit of time because it involves making a network request to SendGrid. Yep, sending emails is always going to be a "heavy" operation. And whenever you're doing something heavy... it means your user is waiting for the response. That's... not the end of the world... but it's not ideal.

So... when a user registers, instead of sending the email immediately, could we send it... later and return the response faster? Of course! Thanks to Symfony's Messenger component, which has first-class integration with Mailer.

Installing & Configuring Messenger

First: in our editor, open .env.local and, for simplicity. let's change the MAILER_DSN back to use Mailtrap. To install Messenger... you can kinda guess the command. In your terminal, run:

composer require messenger

Messenger is super cool and we have an entire tutorial about it. But, it's also simple to get set up and running. Let's see how.

The recipe for Messenger just did a few things: it created a new messenger.yaml configuration file and also added a section in .env. Let's go find that.

62 lines .env
... lines 1 - 55
###> symfony/messenger ###
# Choose one of the transports below
# MESSENGER_TRANSPORT_DSN=amqp://guest:guest@localhost:5672/%2f/messages
# MESSENGER_TRANSPORT_DSN=doctrine://default
# MESSENGER_TRANSPORT_DSN=redis://localhost:6379/messages
###

Here's the 30 second description of how to get Messenger set up. In order to do some work "later" - like sending an email - you need to configure a "queueing" system where details about that work - called "messages" - will be sent. Messenger calls these transports. Because we're already using Doctrine, the easiest "queueing" system is a database table. Uncomment that MESSENGER_TRANSPORT_DSN to use it.

Next, open config/packages/messenger.yaml - that's the new config file:

framework:
messenger:
# Uncomment this (and the failed transport below) to send failed messages to this transport for later handling.
# failure_transport: failed
transports:
# https://symfony.com/doc/current/messenger.html#transport-configuration
# async: '%env(MESSENGER_TRANSPORT_DSN)%'
# failed: 'doctrine://default?queue_name=failed'
# sync: 'sync://'
routing:
# Route your messages to the transports
# 'App\Message\YourMessage': async

and uncomment the transport called async.

framework:
messenger:
... lines 3 - 5
transports:
... line 7
async: '%env(MESSENGER_TRANSPORT_DSN)%'
... lines 9 - 16

Making Emails Async

Great. As soon as you install Messenger, when Mailer sends an email, internally, it will automatically start doing that by dispatching a message through Messenger. Hit Shift + Shift to open a class called SendEmailMessage.

Specifically, Mailer will create this object, put our Email message inside, and dispatch it through Messenger.

Now, if we only installed messenger, the fact that this is being dispatched through the message bus would make... absolutely no difference. The emails would still be handled immediately - or synchronously.

But now we can tell Messenger to "send" instances of SendEmailMessage to our async transport instead of "handling" them - meaning delivering the email - right now. We do that via the routing section. Go copy the namespace of the SendEmailMessage class and, under routing, I'll clear out the comments and say Symfony\Component\Mailer\Messenger\, copy the class name, and paste: SendEmailMessage. Set this to async.

framework:
messenger:
... lines 3 - 11
routing:
# Route your messages to the transports
# 'App\Message\YourMessage': async
'Symfony\Component\Mailer\Messenger\SendEmailMessage': async

Hey! We just made all emails async! Woo! Let's try it: find the registration page.... register as "Fox", email thetruthisoutthere15@example.com, any password, agree to the terms and register!

You may not have noticed, but if you compared the response times of submitting the form before and after that change... this was way, way faster.

Checking out the Queue

Over in Mailtrap... there are no new messages. I can refresh and... nothing. The email was not delivered. Yay! Where is it? Sitting & waiting inside our queue... which is a database table. You can see it by running:

php bin/console doctrine:query:sql 'SELECT * FROM messenger_messages'

That table was automatically created when we sent our first message. It has one row with our one Email inside. If you look closely... you can see the details: the subject, and the email template that will be rendered when it's delivered.

Running the Worker

How do we actually send the email? In Messenger, you process any waiting messages in the queue by running:

php bin/console messenger:consume -vv

The -vv adds extra debugging info... it's more fun. This process is called a "worker" - and you'll have at least one of these commands running at all times on production. Check out our Messenger tutorial for details about that.

Cool! The message was "received" from the queue and "handled"... which is a fancy way in this case to say that the email was actually delivered! Go check out Mailtrap! Ah! There it is! The full correct email... in all its glory.

By the way, in order for your emails to be rendered correctly when being sent via Messenger, you need to make sure that you have the route context parameters set up correctly. That's a topic we covered earlier in this tutorial.

So... congrats on your new shiny async emails! Next, let's make sure that the "author weekly report" email still works... because... honestly... there's going to be a gotcha. Also, how does sending to a transport affect our functional tests?

Leave a comment!

  • 2020-05-12 Thibault Jtc

    Hey weaverryan,

    Thanks for this command, it worked like a charm :)

    Cheers !

  • 2020-05-08 weaverryan

    Hey Thibault Jtc!

    I believe that this is, unfortunately, just the "nature of worker processes". It's not just your inky templates that are affected - if you change any PHP code while developing locally, you need to restart your workers so that your workers see the new code. This is why we gracefully restart workers on deploy. But, I can give you 2 options:

    1) In dev, use a "sync://" transport. Then only use your real transport in the prod environment (I can point you how to do this if you need a little pointer).

    2) If you use the symfony binary, it has a little-known feature to "watch" for changes and restart your workers for you. It's this:


    symfony run -d --watch=config,src,templates,vendor symfony console messenger:consume async

    That runs your messenger:consume command in the background as a daemon and restarts it automatically whenever it sees changes to the listed directories.

    Let me know if any of this helps ;).

    Cheers!

  • 2020-05-07 Thibault Jtc

    Hello guys,

    I'm having an issue with what seems to be "cache" when developing my email templates using Inky.

    --> When modifying any template, I need to restart the messenger worker for my changes to be taken into account. Is it linked to messenger or is it linked to the "inky_to_html" method not parsing the template twice ? (like the styles where we need to manually reset)

    I haven't found any related issues anywhere in the comments nor google and even though I could continue restarting my worker on local, it is getting somewhat annoying and I would like to find a more effective solution :)

    Thanks in advance !

  • 2020-04-29 Caeema

    Hey @weaverryan.

    Sorry for the late reply. With this corona crisis, I'm out of space !

    Well, I don't have a CRON (yet) but yes, at the end, this is something that we plan to have.
    After you reply, I get deeper in the new Messenger Component, I missed some stuff and now it's all crystal clear!

    Thanks for your reply and coments.
    Always a pleasure to learn from here.

    Cheers

  • 2020-04-24 weaverryan

    Hi Caeema!

    Excellent question :). So if you're sending a reminder for an event, my guess is that you would do this with some sort of CRON job? For example, you would run a CRON job every 1 minute... or 5 minutes, query the database to find all emails that need to be sent and then send them? If you were thinking of some *totally* different way, then let me know.

    Assuming I'm right, then, as you said, you might find that you need to send 20 emails all at once. If you did *not* use Messenger and use Mailer directly, then, if something fails, you'll get a normal exception. You could certainly wrap the sending in a "try-catch" and NOT mark it as sent, so that it would be tried again on the next loop. Here is some fake code to illustrate:


    $eventRegistrants = [...];
    foreach ($eventRegistrants as $eventRegistrant) {
    // ...

    try {
    $this->mailer->send(...);
    $eventRegistration->setReminderEmailSent(true); // used to know that this was sent!
    } catch (\Exception $e) {
    // do nothing, don't mark the "registrant email" as sent so that
    // we will try to send it again on the next loop
    // maybe log an error :)
    }
    }

    This is a fine setup. Like anything, you need to code carefully to avoid problems if suddenly 10,000 emails need to be sent at once, but there are no real problems with this. Also, Mailer itself as a way to specify a "fallback" transport. This allows Mailer to automatically try sending through a 2nd transport if the first transport fails for some reason.

    If you decide to use Messenger, then the error handling is pushed into a different spot. First, you would not need the try-catch: you would just do $this->mailer->send() and that's it. In reality, this would "queue" the email to be sent. Then, if there IS a problem delivering the email from your "worker", Messenger's automatic retry system will takeover. By default, Messenger will automatically try to "process" a message 3 times (with some delays in between) before sending it to a "failure queue" so you can inspect what happened and retry.

    So in both situations, you can handle your failures gracefully. Messenger would really be more about performance or scalability in this setup: in theory your one CRON job could very quickly queue 1000 new emails to be sent, and then multiple workers could share the load of sending those emails quickly. But, you may not have that problem :).

    Let me know if this helps!

    Cheers!

  • 2020-04-23 Caeema

    Hi everybody,

    I used this tutorial, and adapt it for a CMS I develop with Symfony 4.4

    For this project, I need to be able to send emails to multiple users.
    For example, send a reminder for an event to all users.

    I'm working with Messenger too, but I was wondering... How can we manage a potential error ?
    Let's say that on the 20 emails that has to be send, 2 emails are not send (whatever the reason)...
    Is it better to manage this with Mailer or with Messenger ? And how ?

    Thanks for your ideas/suggestions because the documentation for this 2 "new" functionality is quite silent about it.

  • 2020-03-09 weaverryan

    Hey nfleming65!

    Oh no! Do you remember (other than the xsl extension) what Composer issues you had? Did you download the "start" code and then upgrade to 4.4? Or just start a new 4.4 project and "adapt" the code into it? If there is an issue, I'd like to know so that we can smooth it out.

    > [messenger] Error thrown while handling message Symfony\Component\Mailer\Messenger\SendEmailMessage. Sending for retry #1 using 1000 ms delay. Error: "Connection to "smtp.mailtrap.io:2525" timed out."
    ["message" => Symfony\Component\Mailer\Messenger\SendEmailMessage { …},"class" => "Symfony\Component\Mailer\Messenger\SendEmailMessage","retryCount" => 1,"delay" => 1000,"error" => "Connection to "smtp.mailtrap.io:2525
    " timed out.","exception" => Symfony\Component\Messenger\Exception\HandlerFailedException { …}]
    > And it tries this 3 times before removing the message from the transport.
    > BUT the emails ARE being sent. So why is there an error?

    Hmm. Is the email sent just ONE time, or is it sent 3 times? I suspect just once - but I want to be sure. Also, if you *stop* running messenger:consume and THEN trigger the email to be sent (i.e. register), is the email sent? I can't think of why this would be happening: you are clearly "routing" the email to an async transport, which means that Messenger will no longer be handling it sync. However, my first guess is that is IS somehow being handed sync and then *also* sent to the transport. But... as I said, that's just a guess.

    > [messenger] Error thrown while handling message Symfony\Component\Mailer\Messenger\SendEmailMessage. Sending for retry #2 using 2000 ms delay. Error: "Expected response code "354" but got code "250",

    Hmm. So, in the SMTP internals, when Symfony sends a message it does this:

    1) it Sends a "DATA" line, which tells the SMTP server that you're about to starting sending data. the SMTP server should reply with a 354 code, which means "Start mail input"

    2) After Sending the data, Symfony sends a special ending syntax and expects the SMTP server to reply with 250, meaning "Requested action was completed".

    For some reason, the SMTP server that you're using seems to be replying to the first with a 250. I have no idea why it would do that. Here's the core part of Symfony that I'm referencing: https://github.com/symfony/...

    Is this coming from Mailtrap? Or something else? Let me know what you find out!

    Cheers!

  • 2020-03-07 nfleming65

    Trying to use the starter code when I'm running symfony 4.4 has caused me so many composer headaches
    On another note, I'm getting this error in my console from the messenger:

    [messenger] Error thrown while handling message Symfony\Component\Mailer\Messenger\SendEmailMessage. Sending for retry #1 using 1000 ms delay. Error: "Connection to "smtp.mailtrap.io:2525" timed out."
    ["message" => Symfony\Component\Mailer\Messenger\SendEmailMessage { …},"class" => "Symfony\Component\Mailer\Messenger\SendEmailMessage","retryCount" => 1,"delay" => 1000,"error" => "Connection to "smtp.mailtrap.io:2525
    " timed out.","exception" => Symfony\Component\Messenger\Exception\HandlerFailedException { …}]

    And it tries this 3 times before removing the message from the transport.

    BUT the emails ARE being sent. So why is there an error?

    There was also an error message mentioning an unexpected response code:

    [messenger] Error thrown while handling message Symfony\Component\Mailer\Messenger\SendEmailMessage. Sending for retry #2 using 2000 ms delay. Error: "Expected response code "354" but got code "250",
    with message "250 2.1.0 Ok"." ["message" => Symfony\Component\Mailer\Messenger\SendEmailMessage { …},"class" => "Symfony\Component\Mailer\Messenger\SendEmailMessage","retryCount" => 2,"delay" => 2000,"error" => "Expecte
    d response code "354" but got code "250", with message "250 2.1.0 Ok".","exception" => Symfony\Component\Messenger\Exception\HandlerFailedException { …}]

  • 2020-02-03 Victor Bocharsky

    Hey Matthias,

    Glad this workaround works for you! I wonder if that "return" statement should be in the Symfony code base by default... probably it's a good idea to create a PR about it and see the feedback of community.

    Hm, I suppose it should be sent only once... how does your messenger.yaml file config look like? Especially your routing section there. Also, I though the correct tag should be "messenger.message_handler" according to the docs, no? See https://symfony.com/doc/cur...

    Cheers!

  • 2020-01-24 Matthias

    OMG. It seems that it's the missing "return" in the default MessageHandler. When i update the Class with the return statement, i can access the SentMessage Object from the getResult Method delivered by HandledStamp. Cool.

    So that i can go forward with my project code without change symfony codebase, i created my own MessageHandler to handle the SendEmailMessage. Registered it over the services tag "mailer.messenger.message_handler". A kind of expected, the handlers are both registered.

    services.yaml

    services :
    [...]
    App\MessageHandler\SendEmailMessageHandler:
    tags:
    - name: mailer.messenger.message_handler
    handles: Symfony\Component\Mailer\Messenger\SendEmailMessage

    result of debug:messenger

    The following messages can be dispatched:

    -----------------------------------------------------------
    Symfony\Component\Mailer\Messenger\SendEmailMessage
    handled by App\MessageHandler\SendEmailMessageHandler
    handled by mailer.messenger.message_handler
    -----------------------------------------------------------

    Is there an other way to really overwrite or replace the default MessageHandler? Because at the moment both handler sending an Email ... again: as expected :)

    Looking really forward to the "final fix".

  • 2020-01-24 Victor Bocharsky

    Hey Matthias,

    Thank you for kind words about SymfonyCasts tutorial! I'm really happy to hear it!

    Hm, EmailMiddleware is something that you created, right? You created this for getting access to SentMessage? Are you sure your middleware is called in the correct order? And are you sure your middleware is called at all? Probably you need to call you middleware a bit later?

    Did you try to debug the stacktrace with Blackfire? It might help to track the chain of calls that you can follow and if there's any chances to get that SentMessage object.

    P.S. I looked into code a bit, and it looks like when we're talking about Messenger, the message is sent in MessageHandler::__invoke(), i.e. there's where the send() method is called which in turn return SentMessage object, see: https://github.com/symfony/... - but I don't see any return statements in this MessageHandler::__invoke() that would help to access SentMessage with Messenger :/

    I hope this helps somehow. Please, let us know if it's not.

    Cheers!

  • 2020-01-21 Matthias

    Hey Victor,

    first of all thanks a lot for your tutorials. They helped me a lot to move my mindset from symfony 3.4 to 4.4. Big step. Your video helped especially for the main concepts of sf4.

    As it seems, i run into a problem which is not very common, measured by search results i found on google. Today i found this tutorial and you use the same Mail setup as i do. So, hopefully you can help me out.
    Over all i have a working setup after switching from SwiftMailer with a self written spooling system to the brand new mailer component with async dsn (sendgrid). Thanks to that, i could delete a lot of code ;)

    After sending an email via mailer, i store the base email in the database to use the activity callback provided from sendgrid (like: deliverd, opened, clicked). Sendrid uses their own message id to communicate the status from the specific message. This is stored in the response header 'x-message-id'. In the code debug i can see that this header is correctly set by the sendgrid transport (SendgridApiTransport:59). It's stored in the SentMassge Object.

    Here is the thing: I use your code from the "messenger" tutorial with the EmailMiddleware. With this approach i cant access the SentMessage Object anymore. I only got the TemplatedEmail Object. The TemplatedEmail Object does not store the Header, given by Sendgrid response.
    The return value of getResult() from HandledStamp is null.

    I hope you have a hint what i do wrong at the moment. When i read the changes from Fabien, related to this topic, it should work (https://github.com/symfony/... but i have no clue what i'm doing wrong.

    Hopefully you have an idea.

  • 2020-01-18 Lydie

    Hello!

    I am on a mac.

    Have a nice we!

  • 2020-01-17 Victor Bocharsky

    Hey Lydie,

    Hm, interesting. I wonder what platform do you use? Are you on Windows? Probably start with stop/start the webserver... if it does not help, only then try to close/open PhpStorm. Though, it's weird that you have to do these steps.

    Cheers!

  • 2020-01-16 Lydie

    Hello!

    In fact I was talking about local development (but with db on a remote server). I do not really understand what happened but even clearing manually the cache did not work. I think it worked again after closing phpstorm and stopping local server. If it appears again, I will just try that and see if it helps :)

  • 2020-01-16 Victor Bocharsky

    Hey Lydie,

    Hm, weird... yeah, sounds like a cache issue... Especially if we're talking about prod. In dev mode cache should recompile... but in case you use some virtualization tools and sharing files between host and target machines - it might not always work correctly, though works most of the time. Anyway, in any weird situation it would be always a good idea to try to clear the cache first and see if it helps. If not - dig further :)

    I'm happy it works for you! And thanks for mentioning it works now!

    Cheers!

  • 2020-01-15 Lydie

    huh tried again today and it works correctly. Nothing has changed in my config. Could it be a "cache" issue?

  • 2020-01-13 Lydie

    Hello!

    I got a strange behavior when enabling the messenger to send email async:

    1st case:
    -> I sent the mail to the queue
    -> Checking email in database, the email looks correct (links / translation)
    -> Start the messenger:consume (-vv)

    => I got the error "Unable to generate a URL for the named route "admin_homepage" as such route does not exist." but the route exists

    2nd case:
    -> I start the messenger:consume (-vv)
    -> I sent email to the queue (here I can not check if the email looks correct in database as the consumer is handling the email directly)

    => I reveiced in mailtrap the email but no link are generated and translation is not done. No error in messenger:consume output.

    Any idea of what could miss?

    Thx!