Flag of Ukraine
SymfonyCasts stands united with the people of Ukraine

Secrets Management Setup

Keep on Learning!

If you liked what you've learned so far, dive in!
Subscribe to get access to this tutorial plus
video, code and script downloads.

Start your All-Access Pass
Buy just this tutorial for $10.00

With a Subscription, click any sentence in the script to jump to that part of the video!

Login Subscribe

My favorite new feature in Symfony 4.4 and 5 - other than the fact that Messenger and Mailer are now stable - is probably the new secrets management system, which is as cool as it sounds.

Secrets?

Here's the deal: every app has a set of config values that need to be different from machine to machine, like different on my local machine versus production. In Symfony, we store these as environment variables.

One example is MAILER_DSN:

62 lines .env
... lines 1 - 38
###> symfony/mailer ###
MAILER_DSN=null://null
# in Symfony 4.4 and higher, the syntax is
# MAILER_DSN=null://default
###
... lines 44 - 62

While developing, I want to use the null transport to avoid sending real emails. But on production, this value will be different, maybe pointing to my SendGrid account.

We reference environment variables with a special syntax - this one is in config/packages/mailer.yaml: %env()% with the variable name inside: MAILER_DSN:

framework:
mailer:
dsn: '%env(MAILER_DSN)%'

If you look at the full list of environment variables, you'll notice that there are two types: sensitive and non-sensitive variables.

For example, MAILER_DSN is a "sensitive" variable because the production value probably contains a username & password or API key: something that, if someone got access to it, would allow them to use our account. So, it's not something that we want to commit to our project.

But other values are not sensitive, like WKHTMLTOPDF_PATH:

62 lines .env
... lines 1 - 44
###> knplabs/knp-snappy-bundle ###
WKHTMLTOPDF_PATH=/usr/local/bin/wkhtmltopdf
WKHTMLTOIMAGE_PATH=/usr/local/bin/wkhtmltoimage
###
... lines 49 - 62

This might need to be different on production, but the value is not sensitive: we don't need to keep it a secret. We could actually commit its production value somewhere in our app to make deployment easier if we wanted to.

So... why are we talking about this? Because, these sensitive, or "secret" environment variables make life tricky. When we deploy, we need to somehow set the MAILER_DSN variable to its secret production value, either as a real environment variable or probably by creating a .env.local file. Doing that safely can be tricky: do you store the secret production value in a config file in this repository or in some deploy script? You can, but then it's not very secure: the less people that can see your secrets - even people on your team - the better.

The Vault Concept

One general solution to this problem is something called a vault. The basic idea is simple: you encrypt your secrets - like the production value for MAILER_DSN - and then store the encrypted value. The "place" where the encrypted secrets are stored is called the "vault". The secrets inside can only be read if you have the decryption password or "private key".

This makes life easier because now your secrets can safely be stored in this "vault", which can just be a set of files on your filesystem or even a cloud vault service. Then, when you deploy, the only "secret" that you need to have available is the password or private key. Some vaults also allow other ways to authenticate.

Introducing Symfony's Secrets "Vault"

None of this "vault" stuff has anything to do with Symfony: it's just a cool concept and there are various services & projects that support the idea - the most famous being HashiCorp's Vault.

But, in Symfony 4.4, a new secrets system was added to let us do all this cool stuff out-of-the-box.

Here's the goal: instead of having MAILER_DSN as an environment variable, we're going to move this to be an "encrypted secret".

Dumping an Env Var for Debugging

To see how this all works clearly, let's add some debugging code to dump the MAILER_DSN value. Open config/services.yaml and add a new bind - $mailerDsn set to %env(MAILER_DSN)% - so we can use this as an argument somewhere:

... lines 1 - 11
services:
# default configuration for services in *this* file
_defaults:
... lines 15 - 17
# setup special, global autowiring rules
bind:
... lines 20 - 24
$mailerDsn: '%env(MAILER_DSN)%'
... lines 26 - 51

I forgot my closing quote... which Symfony will "gently" remind me in a minute.

Next, open src/Controller/ArticleController.php. In the homepage action, thanks to the bind, we can add a $mailerDsn argument. Dump that and die:

... lines 1 - 13
class ArticleController extends AbstractController
{
... lines 16 - 28
public function homepage(ArticleRepository $repository, $mailerDsn)
{
dump($mailerDsn);die;
... lines 32 - 36
}
... lines 38 - 64
}

Now, refresh the homepage. Booo. Let's go fix my missing quote in the YAML file. Refresh again and... perfect: the current value is null://null.

That's no surprise: that's the value in .env and we are not overriding it in .env.local:

62 lines .env
... lines 1 - 38
###> symfony/mailer ###
MAILER_DSN=null://null
... lines 41 - 62

Converting an Env Var to a Secret

Ok, as soon as you have an environment variable that you want to convert to a secret, you need to fully remove it as an environment variable: do not set it as an environment variable anywhere anymore. I'll remove MAILER_DSN from .env and if we were overriding it in .env.local, I would also remove it from there:

62 lines .env
... lines 1 - 38
###> symfony/mailer ###
# MAILER_DSN=null://null
... lines 41 - 62

Not surprisingly, when you refresh, we're greeted with a great big ugly error:

The environment variable is not found.

Bootstrapping the Secrets Vault

So how do we make MAILER_DSN an encrypted secret? With a fancy new console command:

php bin/console secrets:set MAILER_DSN

That will ask us for the value: I'll go copy null://null - you'll learn why I'm choosing that value in a minute - and paste it here. You don't see the pasted value because the command hides the input to be safe.

The Public/Encryption & Private/Decryption Keys

Hit enter and... awesome! Because this was the first time we added something to the secrets vault, Symfony needed to create the vault - and it did that automatically. What does that actually mean? It means that it created several new files in a config/secrets/dev directory.

Let's go check them out: config/secrets/dev. Ooooo.

To "create" the secrets vault, Symfony created two new files, which represent "keys": a private decrypt key and a public encrypt key. If you look inside, they're just fancy text files: they return a long key value.

The public encrypt file is something that is safe to commit to your repository. It's used to add, or "encrypt" a secret, but it can't read encrypted secrets. By committing it, other developers can add new secrets.

The private decrypt key - as its name suggests - is needed to decrypt and read secrets.

One set of Secrets per Environment

Now normally, the "decrypt" key is private and you would not commit it to your repository. However, as you may have noticed, Symfony maintains a different set of secrets per environment. The vault we created is for the dev environment only. In the next chapter, we'll create the vault for the prod environment.

Anyways, because secrets in the dev environment usually represent safe "defaults" that aren't terribly sensitive, it's ok to commit the private key for the dev environment. Plus, if you didn't commit it, other developers on your team wouldn't be able to run the app locally... because Symfony wouldn't be able to read the dev secrets.

Committing the dev Keys

Let's add these to git:

git status

Then git add config/secrets and also add .env:

git add config/secrets .env

This added all 4 files. The other two files store info about the secrets themselves: each secret will be stored in its own file and the "list" file just helps us get the full list of secrets that exist. Commit this:

git commit -m "setting up dev environment vault"

%env()% Automatically Looks for Secrets

And now I have a pleasant surprise: go over and refresh the homepage. It works! That's by design: the %env()% syntax is smart:

... lines 1 - 11
services:
# default configuration for services in *this* file
_defaults:
... lines 15 - 17
# setup special, global autowiring rules
bind:
... lines 20 - 24
$mailerDsn: '%env(MAILER_DSN)%'
... lines 26 - 51

It first looks for a MAILER_DSN environment variable. If it finds one, it uses it. If it does not, it then looks for a MAILER_DSN secret. That's why... it just works.

bin/console secrets:list

To get a list of all the encrypted secrets, you can run:

php bin/console secrets:list

Yep - just one right now. Add --reveal to see the values. By the way, this "reveal" only works because the decrypt file exists in our app.

Next: our app will not currently work in the prod environment because there is no prod vault and so no MAILER_DSN prod secret. Let's fix that and talk a bit about deployment.

Leave a comment!

30
Login or Register to join the conversation
MattWelander Avatar
MattWelander Avatar MattWelander | posted 3 months ago

Hi, what is the difference between putting application-wide parameters in the top section parameters: of the services.yaml file, and putting them as you did there under

services:
_defaults:
bind:

This is how I currently do it

1) In the .env file define for example a variable called basepath

2) In services.yaml I do this:

parameters:
basepath: '%env(basepath)%'

3) In every controller where I want to access these values I have done this:

class EntryController extends AbstractController
{
//Replacement for the old way of accessing app-wide parameters i.e. $this->container->getParameter('xxx')
private $globalParameters;

public function __construct(ParameterBagInterface $params)
{
$this->globalParameters = $params;
}

4) To actually get one of the variables inside a function, I then do this:
$basepath = $this->globalParameters->get('basepath');

IS there just an easier way to do this now, where I can put all app-wide variables under that bind: section and just pass them as an argument to my controller functions?

Reply

Hey Mattias,

Using the bind key is just for convenience when a parameter will be used in many places.
I wouldn't inject the ParameterBag, I'd only inject the parameter that I need:


parameters:
basepath: '%env(basepath)%'

services:
App\SomeService:
arguments:
$basePath: '%env(basepath)%'

Cheers!

Reply
MattWelander Avatar

Oh, but I use my parameters in normal controllers, not in services.

Is my way the correct way of doing it then? Since all of my controllers aren't defined as services at all.

Reply

Are you sure your controllers are not services? That's the default since Symfony 4.4, if you can inject services into your controller's arguments, then they are defined as services. I don't usually inject parameters into my controllers, what I usually do is inject the parameter directly into the service that's going to use it. But, if for some reason you're going to use it in a controller, then you can do precisely what you did or just add a new bind key

Cheers!

Reply
picks Avatar

Hi there, I managed to create a first secret for the prod env, but when I try to set a second, it just raise an error:
Fatal error: Symfony\Component\DependencyInjection\Exception\ParameterCircularRe
ferenceException: Circular reference detected for parameter "env(base64:default:
:SYMFONY_DECRYPTION_SECRET)" ("env(base64:default::SYMFONY_DECRYPTION_SECRET)" >
"env(default::SYMFONY_DECRYPTION_SECRET)" > "env(SYMFONY_DECRYPTION_SECRET)" >
"env(base64:default::SYMFONY_DECRYPTION_SECRET)"). in C:\laragon\www\DESmdc\vend
or\symfony\dependency-injection\Container.php:389

Any clue ?

Reply

Hey picks!

Ah, circular reference!

> ... runs away

Let's... see if we can figure this out ;). It looks like, somehow, you have a parameter that is referencing itself. If you follow the error message, it looks like eventually Symfony sees that the SYMFONY_DECRYPTION_SECRET environment variable is set to env(base64:default::SYMFONY_DECRYPTION_SECRET)"), which then causes a loop.

Can you post the contents of all of the relevant files? Like .env, .env.local, services.yaml (if you have anything relevant there)?

Cheers!

Reply
picks Avatar

Sure!
My .env file has:


APP_ENV=dev
APP_SECRET=03cbf876e68a6826b4009ef99b30acd9
GOOGLE_ANALYTICS_TRACKING_ID=null
HTTP_PROTOCOL=https
SITE_BASE_SCHEME=$HTTP_PROTOCOL
SITE_BASE_HOST=desmdc.local
SITE_BASE_URL=$SITE_BASE_SCHEME://$SITE_BASE_HOST
CORS_ALLOW_ORIGIN=^https?://(localhost|127\.0\.0\.1)(:[0-9]+)?$

I don't use a .env.local for now as I don't need to override anything

And in my services.yml I have many declarations of env parameters (which some are supposed to be stored in the secret vault) like :


parameters:
asset.request_context.base_path: '%env(SITE_BASE_URL)%'
asset.request_context.secure: true
router.request_context.host: '%env(SITE_BASE_HOST)%'
router.request_context.scheme: '%env(SITE_BASE_SCHEME)%'
App\Service\Geocoder:
arguments:
$mapQuestApiKey: '%env(GEOCODING_API_KEY)%'
$env: '%env(APP_ENV)%'
App\Service\MyDesoutterCloud:
arguments:
$iwidApiBaseUri: '%app.iwid_api_base_uri%'
$iwidApiAdminEmail: '%env(IWID_API_EMAIL)%'
$iwidApiAdminPassword: '%env(IWID_API_PWD)%'
$env: '%env(string:APP_ENV)%'
$cache: '@mdc.cache'
Reply
Kiuega Avatar

Hello, any news on this subject? I am facing the same problem (php 7.3.5).

I was able to add a first variable in PROD, but when I wanted to edit it, I got the same error.

I also do not have the environment variable "SYMFONY_DECRYPTION_SECRET", I did not manage to generate it. When I tried the command : php -r 'echo base64_encode(require "config/secrets/prod/prod.decrypt.private.php");' :

I got this error
Parse error: syntax error, unexpected 'private' (T_PRIVATE) in Command line code on line 1

Reply
picks Avatar
picks Avatar picks | Kiuega | posted 2 years ago | edited

Hi Kiuega this has been fixed and merged into SF3.4 apparently => https://github.com/symfony/...

Reply
Kiuega Avatar

Hey ! (ami Français), On SF 4 and 5, there seems to be nothing. No update available for recipes, it's strange.
On the other hand, did you end up succeeding in generating the environment variable SYMFONY_DECRYPTION_SECRET ?

EDIT : Nicolas Grekas said that it was OK, and that we will just have to wait for the 5.1.3 release

Reply
picks Avatar
picks Avatar picks | Kiuega | posted 2 years ago | edited

Just saw that yes Kiuega ;) Thanks! In the meantime I could make it work by hardcoding SYMFONY_DECRYPTION_SECRET in my .env prod file with its encoded value...

Reply

Good work around! The Pr has been merged, but not tagged yet. I would expect a tag fairly soon, as the last one was 3 weeks ago.

Cheers!

Reply

Hey picks!

Hmm. I don't see SYMFONY_DECRYPTION_SECRET anywhere in these files? Where/how are you setting this?

To give you a bit more context, the default value of the "decryption environment value" is base64:default::SYMFONY_DECRYPTION_SECRET. What I mean is, when Symfony tries to decrypt your fault, it looks for this SYMFONY_DECRYPTION_SECRET environment variable , then "base64 decodes" it. It almost looks like the SYMFONY_DECRYPTION_SECRET value is set to itself SYMFONY_DECRYPTION_SECRET... or something similar.

Cheers!

Reply
picks Avatar

I have no idea, I don't remember setting this value anywhere in my project!
I this something that needs to be done?

Reply

Hey picks

Yes, you whether set that env variable up or you ensure that the file config/secrets/prod/prod.decrypt.private.php exists in your project. If that's not the case, double check your env vars and parameters to see if something is trying to use the SYMFONY_DECRYPTION_SECRET env var.

Cheers!

Reply
picks Avatar
picks Avatar picks | MolloKhan | posted 2 years ago | edited

Hi MolloKhan,
If I do a project search for "SYMFONY_DECRYPTION_SECRET" within PhpStorm, it only find references in the cache...
This env is not set anywhere in my project, and I have generated a config/secrets/prod/prod.decrypt.private.php (that's how I managed to generate the first secret, but not the second...). This is weird, really.

Reply
picks Avatar

So even weirder, if I manually set a SYMFONY_DECRYPTION_SECRET env var within my .env.prod.local file, it now works!!! But I still have my config/secrets/prod/prod.decrypt.private.php...
I'm lost.

Reply

That's weird :)
What Symfony version do you have?

Reply
picks Avatar
picks Avatar picks | MolloKhan | posted 2 years ago | edited

MolloKhan I was using a solid 4.4 version at the start of the project, then I upgraded it to 5.0 and 5.1... Maybe it is missing a package or something?

Reply
picks Avatar

I have the same problem with a second project now! I downloaded a fresh copy of SF5.1 and installed the latest packages I needed, adapted and copied the "old" files(aseets, src, templates, config...), and it did the exact same : first prod secret is generated, second one raises the issue. Unless I add the SYMFONY_DECRYPTION_SECRET env var in my .env.prod.local file... This is insane.

Reply

Hey picks!

Yea, this sounds SUPER weird, so I'm checking into it further. So far, I can't repeat the error - I must be missing something that you have. Here's what I've tried so far:

1) I started a new project, and tried to create 2 secrets:


symfony new prod_decrypt_bug
cd prod_decrypt_bug
./bin/console secrets:set --env=prod SECRET1
./bin/console secrets:set --env=prod SECRET2

Both secrets were created without any problems.

2) I downloaded the course code from this page, unzipped it, and moved into the "finish" directory. Then I:


composer install
# I deleted the prod vault so that I could start from scratch
rm -rf config/secrets/prod
./bin/console secrets:set --env=prod MAILER_DSN
./bin/console secrets:set --env=prod MAILER_DSN2

Again, this had no problems.

Am I doing something different than you are? I assumed that you're getting the error right when you call secrets:set for the 2nd secret, is that correct? Or do you get the error at some other point?

Let me know!

Cheers!

Reply
picks Avatar
picks Avatar picks | weaverryan | posted 2 years ago | edited

weaverryan yes that's correct, that's only occuring for the 2nd secret... I'll try to set a blanck copy like you did to see if has something to do with my local dev environment... and let you know! Thanks as always.

1 Reply

Awesome! Let me know what you find out! It definitely sounds "fishy", but I can't repeat it :/

Reply
picks Avatar
picks Avatar picks | weaverryan | posted 2 years ago | edited

weaverryan the issue has been raised at SF, it seems like it is indeed a bug ==> https://github.com/symfony/...

Reply

Hey picks!

Great! I mean, not great having a bug... of course :). But at least we know others have the issue and there will be an effort to fix it. I just "pushed" on that PR to try to keep it moving. It's interesting that, on that issue, some people are able to repeat and others cannot. What version of PHP are you on? If we could identify the reason why it happens on some machines but not others, that could help a lot. I tried on PHP 7.4 (and did not get the bug).

Cheers!

Reply
picks Avatar
picks Avatar picks | weaverryan | posted 2 years ago | edited

weaverryan I am the one saying he his using PHP 7.2 ;)

Reply

Come on picks! Where is your brand consistency? :p Ok, when I hover over picks44, I can see VinZ :D

Btw, I see you're from Nantes! We visited there years ago - after the last SymfonyCon in Paris - lovely place - saw the machines de l'île of course ;)

Reply
picks Avatar
picks Avatar picks | weaverryan | posted 2 years ago | edited

weaverryan haha yes, cool! You should consider participating to the "web2day" event, it is quite big now, and there are lots of conferences dedicated to tech ;) could be awesome.
PS: I registered on Disqu's using FB login, that's why!

Reply

4:30 — there is dd() function for this. ;)

Reply

Hey TomaszGasior

Yep you are totally right, you can use dd() for it, since Symfony 4.1.

Cheers!

Reply
Cat in space

"Houston: no signs of life"
Start the conversation!

What PHP libraries does this tutorial use?

// composer.json
{
    "require": {
        "php": "^7.3.0",
        "ext-iconv": "*",
        "antishov/doctrine-extensions-bundle": "^1.4", // v1.4.2
        "aws/aws-sdk-php": "^3.87", // 3.110.11
        "composer/package-versions-deprecated": "^1.11", // 1.11.99
        "doctrine/doctrine-bundle": "^2.0", // 2.0.6
        "doctrine/doctrine-migrations-bundle": "^1.3|^2.0", // 2.1.2
        "doctrine/orm": "^2.5.11", // v2.7.2
        "doctrine/persistence": "^1.3.7", // 1.3.8
        "easycorp/easy-log-handler": "^1.0", // v1.0.9
        "http-interop/http-factory-guzzle": "^1.0", // 1.0.0
        "knplabs/knp-markdown-bundle": "^1.7", // 1.8.1
        "knplabs/knp-paginator-bundle": "^5.0", // v5.0.0
        "knplabs/knp-snappy-bundle": "^1.6", // v1.7.0
        "knplabs/knp-time-bundle": "^1.8", // v1.11.0
        "league/flysystem-aws-s3-v3": "^1.0", // 1.0.23
        "league/flysystem-cached-adapter": "^1.0", // 1.0.9
        "league/html-to-markdown": "^4.8", // 4.8.2
        "liip/imagine-bundle": "^2.1", // 2.3.0
        "nexylan/slack-bundle": "^2.1", // v2.2.1
        "oneup/flysystem-bundle": "^3.0", // 3.3.0
        "php-http/guzzle6-adapter": "^2.0", // v2.0.1
        "sensio/framework-extra-bundle": "^5.1", // v5.5.3
        "symfony/asset": "5.0.*", // v5.0.2
        "symfony/console": "5.0.*", // v5.0.2
        "symfony/dotenv": "5.0.*", // v5.0.2
        "symfony/flex": "^1.0", // v1.17.6
        "symfony/form": "5.0.*", // v5.0.2
        "symfony/framework-bundle": "5.0.*", // v5.0.2
        "symfony/mailer": "5.0.*", // v5.0.2
        "symfony/messenger": "5.0.*", // v5.0.2
        "symfony/monolog-bundle": "^3.5", // v3.5.0
        "symfony/security-bundle": "5.0.*", // v5.0.2
        "symfony/sendgrid-mailer": "5.0.*", // v5.0.2
        "symfony/serializer-pack": "^1.0", // v1.0.2
        "symfony/twig-bundle": "5.0.*", // v5.0.2
        "symfony/twig-pack": "^1.0", // v1.0.0
        "symfony/validator": "5.0.*", // v5.0.2
        "symfony/webpack-encore-bundle": "^1.4", // v1.7.2
        "symfony/yaml": "5.0.*", // v5.0.2
        "twig/cssinliner-extra": "^2.12", // v2.12.0
        "twig/extensions": "^1.5", // v1.5.4
        "twig/inky-extra": "^2.12" // v2.12.0
    },
    "require-dev": {
        "doctrine/doctrine-fixtures-bundle": "^3.0", // 3.3.0
        "fzaninotto/faker": "^1.7", // v1.8.0
        "symfony/browser-kit": "5.0.*", // v5.0.2
        "symfony/debug-bundle": "5.0.*", // v5.0.2
        "symfony/maker-bundle": "^1.0", // v1.14.3
        "symfony/phpunit-bridge": "5.0.*", // v5.0.2
        "symfony/profiler-pack": "^1.0", // v1.0.4
        "symfony/var-dumper": "5.0.*" // v5.0.2
    }
}