Secrets Management Setup

Keep on Learning!

If you liked what you've learned so far, dive in!
Subscribe to get access to this tutorial plus
video, code and script downloads.

Start your All-Access Pass
Buy just this tutorial for $10.00

With a Subscription, click any sentence in the script to jump to that part of the video!

Login Subscribe

My favorite new feature in Symfony 4.4 and 5 - other than the fact that Messenger and Mailer are now stable - is probably the new secrets management system, which is as cool as it sounds.

Secrets?

Here's the deal: every app has a set of config values that need to be different from machine to machine, like different on my local machine versus production. In Symfony, we store these as environment variables.

One example is MAILER_DSN:

62 lines .env
... lines 1 - 38
###> symfony/mailer ###
MAILER_DSN=null://null
# in Symfony 4.4 and higher, the syntax is
# MAILER_DSN=null://default
###
... lines 44 - 62

While developing, I want to use the null transport to avoid sending real emails. But on production, this value will be different, maybe pointing to my SendGrid account.

We reference environment variables with a special syntax - this one is in config/packages/mailer.yaml: %env()% with the variable name inside: MAILER_DSN:

framework:
mailer:
dsn: '%env(MAILER_DSN)%'

If you look at the full list of environment variables, you'll notice that there are two types: sensitive and non-sensitive variables.

For example, MAILER_DSN is a "sensitive" variable because the production value probably contains a username & password or API key: something that, if someone got access to it, would allow them to use our account. So, it's not something that we want to commit to our project.

But other values are not sensitive, like WKHTMLTOPDF_PATH:

62 lines .env
... lines 1 - 44
###> knplabs/knp-snappy-bundle ###
WKHTMLTOPDF_PATH=/usr/local/bin/wkhtmltopdf
WKHTMLTOIMAGE_PATH=/usr/local/bin/wkhtmltoimage
###
... lines 49 - 62

This might need to be different on production, but the value is not sensitive: we don't need to keep it a secret. We could actually commit its production value somewhere in our app to make deployment easier if we wanted to.

So... why are we talking about this? Because, these sensitive, or "secret" environment variables make life tricky. When we deploy, we need to somehow set the MAILER_DSN variable to its secret production value, either as a real environment variable or probably by creating a .env.local file. Doing that safely can be tricky: do you store the secret production value in a config file in this repository or in some deploy script? You can, but then it's not very secure: the less people that can see your secrets - even people on your team - the better.

The Vault Concept

One general solution to this problem is something called a vault. The basic idea is simple: you encrypt your secrets - like the production value for MAILER_DSN - and then store the encrypted value. The "place" where the encrypted secrets are stored is called the "vault". The secrets inside can only be read if you have the decryption password or "private key".

This makes life easier because now your secrets can safely be stored in this "vault", which can just be a set of files on your filesystem or even a cloud vault service. Then, when you deploy, the only "secret" that you need to have available is the password or private key. Some vaults also allow other ways to authenticate.

Introducing Symfony's Secrets "Vault"

None of this "vault" stuff has anything to do with Symfony: it's just a cool concept and there are various services & projects that support the idea - the most famous being HashiCorp's Vault.

But, in Symfony 4.4, a new secrets system was added to let us do all this cool stuff out-of-the-box.

Here's the goal: instead of having MAILER_DSN as an environment variable, we're going to move this to be an "encrypted secret".

Dumping an Env Var for Debugging

To see how this all works clearly, let's add some debugging code to dump the MAILER_DSN value. Open config/services.yaml and add a new bind - $mailerDsn set to %env(MAILER_DSN)% - so we can use this as an argument somewhere:

... lines 1 - 11
services:
# default configuration for services in *this* file
_defaults:
... lines 15 - 17
# setup special, global autowiring rules
bind:
... lines 20 - 24
$mailerDsn: '%env(MAILER_DSN)%'
... lines 26 - 51

I forgot my closing quote... which Symfony will "gently" remind me in a minute.

Next, open src/Controller/ArticleController.php. In the homepage action, thanks to the bind, we can add a $mailerDsn argument. Dump that and die:

... lines 1 - 13
class ArticleController extends AbstractController
{
... lines 16 - 28
public function homepage(ArticleRepository $repository, $mailerDsn)
{
dump($mailerDsn);die;
... lines 32 - 36
}
... lines 38 - 64
}

Now, refresh the homepage. Booo. Let's go fix my missing quote in the YAML file. Refresh again and... perfect: the current value is null://null.

That's no surprise: that's the value in .env and we are not overriding it in .env.local:

62 lines .env
... lines 1 - 38
###> symfony/mailer ###
MAILER_DSN=null://null
... lines 41 - 62

Converting an Env Var to a Secret

Ok, as soon as you have an environment variable that you want to convert to a secret, you need to fully remove it as an environment variable: do not set it as an environment variable anywhere anymore. I'll remove MAILER_DSN from .env and if we were overriding it in .env.local, I would also remove it from there:

62 lines .env
... lines 1 - 38
###> symfony/mailer ###
# MAILER_DSN=null://null
... lines 41 - 62

Not surprisingly, when you refresh, we're greeted with a great big ugly error:

The environment variable is not found.

Bootstrapping the Secrets Vault

So how do we make MAILER_DSN an encrypted secret? With a fancy new console command:

php bin/console secrets:set MAILER_DSN

That will ask us for the value: I'll go copy null://null - you'll learn why I'm choosing that value in a minute - and paste it here. You don't see the pasted value because the command hides the input to be safe.

The Public/Encryption & Private/Decryption Keys

Hit enter and... awesome! Because this was the first time we added something to the secrets vault, Symfony needed to create the vault - and it did that automatically. What does that actually mean? It means that it created several new files in a config/secrets/dev directory.

Let's go check them out: config/secrets/dev. Ooooo.

To "create" the secrets vault, Symfony created two new files, which represent "keys": a private decrypt key and a public encrypt key. If you look inside, they're just fancy text files: they return a long key value.

The public encrypt file is something that is safe to commit to your repository. It's used to add, or "encrypt" a secret, but it can't read encrypted secrets. By committing it, other developers can add new secrets.

The private decrypt key - as its name suggests - is needed to decrypt and read secrets.

One set of Secrets per Environment

Now normally, the "decrypt" key is private and you would not commit it to your repository. However, as you may have noticed, Symfony maintains a different set of secrets per environment. The vault we created is for the dev environment only. In the next chapter, we'll create the vault for the prod environment.

Anyways, because secrets in the dev environment usually represent safe "defaults" that aren't terribly sensitive, it's ok to commit the private key for the dev environment. Plus, if you didn't commit it, other developers on your team wouldn't be able to run the app locally... because Symfony wouldn't be able to read the dev secrets.

Committing the dev Keys

Let's add these to git:

git status

Then git add config/secrets and also add .env:

git add config/secrets .env

This added all 4 files. The other two files store info about the secrets themselves: each secret will be stored in its own file and the "list" file just helps us get the full list of secrets that exist. Commit this:

git commit -m "setting up dev environment vault"

%env()% Automatically Looks for Secrets

And now I have a pleasant surprise: go over and refresh the homepage. It works! That's by design: the %env()% syntax is smart:

... lines 1 - 11
services:
# default configuration for services in *this* file
_defaults:
... lines 15 - 17
# setup special, global autowiring rules
bind:
... lines 20 - 24
$mailerDsn: '%env(MAILER_DSN)%'
... lines 26 - 51

It first looks for a MAILER_DSN environment variable. If it finds one, it uses it. If it does not, it then looks for a MAILER_DSN secret. That's why... it just works.

bin/console secrets:list

To get a list of all the encrypted secrets, you can run:

php bin/console secrets:list

Yep - just one right now. Add --reveal to see the values. By the way, this "reveal" only works because the decrypt file exists in our app.

Next: our app will not currently work in the prod environment because there is no prod vault and so no MAILER_DSN prod secret. Let's fix that and talk a bit about deployment.

Leave a comment!

  • 2020-07-06 weaverryan

    Good work around! The Pr has been merged, but not tagged yet. I would expect a tag fairly soon, as the last one was 3 weeks ago.

    Cheers!

  • 2020-07-06 Vinz Stoned Orgies

    Just saw that yes Kiuega A ;) Thanks! In the meantime I could make it work by hardcoding SYMFONY_DECRYPTION_SECRET in my .env prod file with its encoded value...

  • 2020-07-06 Kiuega A

    Hey ! (ami Français), On SF 4 and 5, there seems to be nothing. No update available for recipes, it's strange.
    On the other hand, did you end up succeeding in generating the environment variable SYMFONY_DECRYPTION_SECRET ?

    EDIT : Nicolas Grekas said that it was OK, and that we will just have to wait for the 5.1.3 release

  • 2020-07-06 Vinz Stoned Orgies

    Hi Kiuega A this has been fixed and merged into SF3.4 apparently => https://github.com/symfony/...

  • 2020-07-06 Kiuega A

    Hello, any news on this subject? I am facing the same problem (php 7.3.5).

    I was able to add a first variable in PROD, but when I wanted to edit it, I got the same error.

    I also do not have the environment variable "SYMFONY_DECRYPTION_SECRET", I did not manage to generate it. When I tried the command : php -r 'echo base64_encode(require "config/secrets/prod/prod.decrypt.private.php");' :

    I got this error
    Parse error: syntax error, unexpected 'private' (T_PRIVATE) in Command line code on line 1

  • 2020-06-18 Vinz Stoned Orgies

    weaverryan haha yes, cool! You should consider participating to the "web2day" event, it is quite big now, and there are lots of conferences dedicated to tech ;) could be awesome.
    PS: I registered on Disqu's using FB login, that's why!

  • 2020-06-18 weaverryan

    Come on Vinz Stoned Orgies! Where is your brand consistency? :p Ok, when I hover over picks44, I can see VinZ :D

    Btw, I see you're from Nantes! We visited there years ago - after the last SymfonyCon in Paris - lovely place - saw the machines de l'île of course ;)

  • 2020-06-18 Vinz Stoned Orgies

    weaverryan I am the one saying he his using PHP 7.2 ;)

  • 2020-06-18 weaverryan

    Hey Vinz Stoned Orgies!

    Great! I mean, not great having a bug... of course :). But at least we know others have the issue and there will be an effort to fix it. I just "pushed" on that PR to try to keep it moving. It's interesting that, on that issue, some people are able to repeat and others cannot. What version of PHP are you on? If we could identify the reason why it happens on some machines but not others, that could help a lot. I tried on PHP 7.4 (and did not get the bug).

    Cheers!

  • 2020-06-17 Vinz Stoned Orgies

    weaverryan the issue has been raised at SF, it seems like it is indeed a bug ==> https://github.com/symfony/...

  • 2020-06-16 weaverryan

    Awesome! Let me know what you find out! It definitely sounds "fishy", but I can't repeat it :/

  • 2020-06-11 Vinz Stoned Orgies

    weaverryan yes that's correct, that's only occuring for the 2nd secret... I'll try to set a blanck copy like you did to see if has something to do with my local dev environment... and let you know! Thanks as always.

  • 2020-06-11 weaverryan

    Hey Vinz Stoned Orgies!

    Yea, this sounds SUPER weird, so I'm checking into it further. So far, I can't repeat the error - I must be missing something that you have. Here's what I've tried so far:

    1) I started a new project, and tried to create 2 secrets:


    symfony new prod_decrypt_bug
    cd prod_decrypt_bug
    ./bin/console secrets:set --env=prod SECRET1
    ./bin/console secrets:set --env=prod SECRET2

    Both secrets were created without any problems.

    2) I downloaded the course code from this page, unzipped it, and moved into the "finish" directory. Then I:


    composer install
    # I deleted the prod vault so that I could start from scratch
    rm -rf config/secrets/prod
    ./bin/console secrets:set --env=prod MAILER_DSN
    ./bin/console secrets:set --env=prod MAILER_DSN2

    Again, this had no problems.

    Am I doing something different than you are? I assumed that you're getting the error right when you call secrets:set for the 2nd secret, is that correct? Or do you get the error at some other point?

    Let me know!

    Cheers!

  • 2020-06-11 Vinz Stoned Orgies

    I have the same problem with a second project now! I downloaded a fresh copy of SF5.1 and installed the latest packages I needed, adapted and copied the "old" files(aseets, src, templates, config...), and it did the exact same : first prod secret is generated, second one raises the issue. Unless I add the SYMFONY_DECRYPTION_SECRET env var in my .env.prod.local file... This is insane.

  • 2020-06-10 Vinz Stoned Orgies

    Diego Aguiar I was using a solid 4.4 version at the start of the project, then I upgraded it to 5.0 and 5.1... Maybe it is missing a package or something?

  • 2020-06-09 Diego Aguiar

    That's weird :)
    What Symfony version do you have?

  • 2020-06-09 Vinz Stoned Orgies

    So even weirder, if I manually set a SYMFONY_DECRYPTION_SECRET env var within my .env.prod.local file, it now works!!! But I still have my config/secrets/prod/prod.decrypt.private.php...
    I'm lost.

  • 2020-06-09 Vinz Stoned Orgies

    Hi Diego Aguiar,
    If I do a project search for "SYMFONY_DECRYPTION_SECRET" within PhpStorm, it only find references in the cache...
    This env is not set anywhere in my project, and I have generated a config/secrets/prod/prod.decrypt.private.php (that's how I managed to generate the first secret, but not the second...). This is weird, really.

  • 2020-06-08 Diego Aguiar

    Hey Vinz Stoned Orgies

    Yes, you whether set that env variable up or you ensure that the file config/secrets/prod/prod.decrypt.private.php exists in your project. If that's not the case, double check your env vars and parameters to see if something is trying to use the SYMFONY_DECRYPTION_SECRET env var.

    Cheers!

  • 2020-06-08 Vinz Stoned Orgies

    I have no idea, I don't remember setting this value anywhere in my project!
    I this something that needs to be done?

  • 2020-06-08 weaverryan

    Hey Vinz Stoned Orgies!

    Hmm. I don't see SYMFONY_DECRYPTION_SECRET anywhere in these files? Where/how are you setting this?

    To give you a bit more context, the default value of the "decryption environment value" is base64:default::SYMFONY_DECRYPTION_SECRET. What I mean is, when Symfony tries to decrypt your fault, it looks for this SYMFONY_DECRYPTION_SECRET environment variable , then "base64 decodes" it. It almost looks like the SYMFONY_DECRYPTION_SECRET value is set to itself SYMFONY_DECRYPTION_SECRET... or something similar.

    Cheers!

  • 2020-06-08 Vinz Stoned Orgies

    Sure!
    My .env file has:


    APP_ENV=dev
    APP_SECRET=03cbf876e68a6826b4009ef99b30acd9
    GOOGLE_ANALYTICS_TRACKING_ID=null
    HTTP_PROTOCOL=https
    SITE_BASE_SCHEME=$HTTP_PROTOCOL
    SITE_BASE_HOST=desmdc.local
    SITE_BASE_URL=$SITE_BASE_SCHEME://$SITE_BASE_HOST
    CORS_ALLOW_ORIGIN=^https?://(localhost|127\.0\.0\.1)(:[0-9]+)?$

    I don't use a .env.local for now as I don't need to override anything

    And in my services.yml I have many declarations of env parameters (which some are supposed to be stored in the secret vault) like :


    parameters:
    asset.request_context.base_path: '%env(SITE_BASE_URL)%'
    asset.request_context.secure: true
    router.request_context.host: '%env(SITE_BASE_HOST)%'
    router.request_context.scheme: '%env(SITE_BASE_SCHEME)%'
    App\Service\Geocoder:
    arguments:
    $mapQuestApiKey: '%env(GEOCODING_API_KEY)%'
    $env: '%env(APP_ENV)%'
    App\Service\MyDesoutterCloud:
    arguments:
    $iwidApiBaseUri: '%app.iwid_api_base_uri%'
    $iwidApiAdminEmail: '%env(IWID_API_EMAIL)%'
    $iwidApiAdminPassword: '%env(IWID_API_PWD)%'
    $env: '%env(string:APP_ENV)%'
    $cache: '@mdc.cache'
  • 2020-06-08 weaverryan

    Hey Vinz Stoned Orgies!

    Ah, circular reference!

    > ... runs away

    Let's... see if we can figure this out ;). It looks like, somehow, you have a parameter that is referencing itself. If you follow the error message, it looks like eventually Symfony sees that the SYMFONY_DECRYPTION_SECRET environment variable is set to env(base64:default::SYMFONY_DECRYPTION_SECRET)"), which then causes a loop.

    Can you post the contents of all of the relevant files? Like .env, .env.local, services.yaml (if you have anything relevant there)?

    Cheers!

  • 2020-06-05 Vinz Stoned Orgies

    Hi there, I managed to create a first secret for the prod env, but when I try to set a second, it just raise an error:
    Fatal error: Symfony\Component\DependencyInjection\Exception\ParameterCircularRe
    ferenceException: Circular reference detected for parameter "env(base64:default:
    :SYMFONY_DECRYPTION_SECRET)" ("env(base64:default::SYMFONY_DECRYPTION_SECRET)" >
    "env(default::SYMFONY_DECRYPTION_SECRET)" > "env(SYMFONY_DECRYPTION_SECRET)" >
    "env(base64:default::SYMFONY_DECRYPTION_SECRET)"). in C:\laragon\www\DESmdc\vend
    or\symfony\dependency-injection\Container.php:389

    Any clue ?

  • 2020-02-24 Vladimir Sadicov

    Hey Tomasz Gąsior

    Yep you are totally right, you can use dd() for it, since Symfony 4.1.

    Cheers!

  • 2020-02-22 Tomasz Gąsior

    4:30 — there is dd() function for this. ;)