Migrations

Hey @Ali!

Hmmm - that is super weird. How is your DATABASE_URL configured? Are you using the Symfony binary + Docker setup like we're using in this tutorial? Or are you specifying it in .env (or .env.local)? If you're specifying in .env or .env.local, what does your DATABASE_URL look like? And what does your config/packages/database.yaml look like?

I'm not personally familiar with this error, but it looks like it's usually due to a missing "serverVersion" configuration. This is normally (but not always) configured as a ?serverVersion= on the end of your DATABASE_URL - for example https://github.com/symfony/...

Cheers!

A port number after the localhost IP (127.0.0.1) was giving me trouble. Removing it, left it as the default and it worked.

Hey Sherri

I'm sorry you had to go thorugh all those problems there's where Docker shrines. Anyways, I'm glad to know you could fix the problems and move forward. Thanks for sharing your solution

Cheers!

Hey Dirk!

So each time we update the app and need to migrate, we have to edit the migration file to avoid losing all of the"manual created tables". Is there a way to avoid this?

Yes, there is! Go into your config/packages/doctrine.yaml file and add a doctrine.dbal.schema_filter option:

doctrine:
    dbal:
         schema_filter: ~^(?!t_)~

This would ignore any tables prefixed with t_ - reference https://symfony.com/doc/current/bundles/DoctrineMigrationsBundle/index.html#manual-tables

So the only weird part is getting this regular expression correct: you are basically giving it a regular expression of all the tables that it should worry about (so that it filters "out" the manual ones). So I think something like ~^(?!(custom_table1|custom_table2))~ where you list your custom tables.

Let me know if that helps!

Cheers!

Hey weaverryan !
Yes it worked and I am very thankful for the help and the great tutorials. But this also creates a problem at a different point in the app where I use the schemaManager to list all table names unfortunately are the tables I need, the tables that are ignored by the schema_filter and don't get shown. This is bit problematic and I don't have a plan how I could fix it.

Hey Dirk!

Hmm. Yea, unfortunately, you can't have it "both ways": all parts of Doctrine use this schema_filter, including the schema manager. Here is what I would do, if you are ok getting a bit fancier ;).

A) Set the array of table names you want to ignore to a parameter:

# config/services.yaml
parameters:
    extra_tables: [custom_table1, custom_table2]

B) Remove the schema_filter option. There is another way to do this that can use the parameter above. You would do this by leveraging a class from DoctrineBundle - you need to register it as a service:

# config/services.yaml
services:
    # ...

    doctrine.ignore_table_filter:
        class: Doctrine\Bundle\DoctrineBundle\Dbal\BlacklistSchemaAssetFilter
        arguments: ['%extra_tables%']
        tags: ['doctrine.dbal.schema_filter']

This accomplishes the same as the schema_filter, just using a lower-level mechanism. I'm using it so that we can read the array extra_tables parameter. This is important because NOW, wherever you're using the SchemaManager in your code to list all the table names, I would inject this parameter and then return the full list of tables as all of the tables from SchemaManager + the tables in the extra_tables parameter.

Let me know if that makes any sense ;). Some code may not be exactly right - I'm just typing it here.

Cheers!

This solution works perfectly but there is one thing that bothers me. I have to hard code all of the tables which are a lot and they could change or new ones get added. So I was thinking about using a prefix like in the schema_filter but I don't know if this is possible. Nevertheless I got a lot further with my project thanks to your help.

Hey Boris Z. !

This isn't something I've needed to do before but... after some quick googling, this seems like a surprisingly big mess :/. It seems that version 3 of DoctrineMigrationsBundle seems to have dropped the support for using different connections or entity managers - https://github.com/doctrine... - which is a very unfortunate oversight.

And, fixing it in your app, is non-trivial. Fortunately, I found this post - https://jmsche.fr/en/blog/u... - which was exactly the type of thing I was thinking was necessary (but he already did the work!). Let me know if it helps.

Cheers!

weaverryan, hello again!
You asked me to write if it helps. And yes, it is)
I have made a proxy migrate command as in example and also a diff command in the same way. It was even easier than I expected

Woohoo! Thanks for following up!

> But I still don't understand why there is no native solution in migrations bundle

I think its removal was basically an accident. And now... it's hard to ask anyone to contribute it that doesn't need it (the maintainers of the bundle clearly don't have this use-case). So, someone just needs to do the work - that proxy command is probably a good guide ;).

Cheers!

weaverryan thank you very much! It looks like exactly the solution I was looking for. I'll try it and write here.
But I still don't understand why there is no native solution in migrations bundle. The only needed thing is to make possible to add multiple configurations to the migrations config

Hey Sean,

Migrations are pretty straightforward... so probably the only rule - write good migrations ;) But serious, the most important part of executing migrations on production - to do not lose any existent data. That's super important... and that's something you should care about. After each generated migration - check it and make sure it's really safe for production data.

Also, if you have automated deploys - you will need to pass "--no-interaction" option to the command to avoid any interactive questions, otherwise it's step will just hang your deployment process.

And usually devs only care about queries in up() method and ignore ones in down(). If something will be incorrect in the migration - you would probably need to rollback manually, i.e. manually execute some queries to sync the DB.

That's probably all I can think about now. Hope it will help :)

Cheers!

Thanks for the input. I really appreciate it.

Hey Sean,

No problem, that's a good question to wonder about as nobody want to lose any real data on production ;) Btw, one more tip, just in case, create a backup of your production DB before deploying if you're not confident in your migrations.

Cheers!

Hey be_tnt!

Yes, I know this situation - the upgrade from migrations 2 to 3 is kind of annoying. So, first, you are absolutely sure that your migrations table has been cleaned out? I'm asking because, if i remember correctly (and I'm not sure if I am), the default name of the migrations table also changed during the upgrade. So it may be that your migrations table IS still there, but now Doctrine migrations is looking for a table with a different name. Double check that first.

Otherwise, you *can* manually fix it by copying the table from your existing instance or by manually inserting rows in that table for the old migrations. You shouldn't need to do this, because nothing should have cleaned out your migrations table, but in case something funny happened, it's definitely ok to do that :).

Cheers!

Thx for your quick answer.
I got 3 environments: local, dev and prod and for all of these, the "old" table for migrations has been deleted when upgrading :( Inserting rows manually could take a little but of time as I would need to add 64 records :o) I will check in my backup to see if I can find it.
Any other solutions? :o)

Hey be_tnt!

> the "old" table for migrations has been deleted when upgrading

That is super weird and unfortunate :/.

> will check in my backup to see if I can find it.

This is the easiest solution :). Fortunately, no data was lost... it's just an annoying thing to need to fix. I can think of 2 work arounds:

1) Locally, clean out your database, then run all of your migrations so that you can get a migrations_versions table (or whatever it's called on your version) that you can then copy up to production. A quick, hacky way to avoid 64 manual inserts.

2) You could delete all of the old migration files and start "from scratch". It would look something like this:

A) Checkout to an older branch that matches production.
B) Delete all the old migration files
C) Empty your database (so, zero tables)
D) Run make:migration

This will create a single migration file that contains all the SQL from the 64. You can then move that into your "main" branch and delete the other 64 (I would probably rename the new migration to a date that is older than any new ones).

So... let me know what you end up doing ;).

Cheers!

So I found out the old table in my backup, adapted the records with the new schema. Then I copied all my old migrations files (63 in fact) to the new migrations folder and finally execute make:migration to check

I got 2 warnings:
[WARNING] You have 63 available migrations to execute.
[WARNING] You have 63 previously executed migrations in the database that are not registered migrations.

Any idea?

ok I found it. The '\' between DoctrineMigrations and Version (Version column) was removed during the INSERT. I have put double '\' instead of a single one and now it works ! :)

Ex: INSERT INTO `doctrine_migration_versions` VALUES ('DoctrineMigrations\\Version20190704123309','2020-01-29 15:22:52',0);

Thx for your help!

Ah, good job be_tnt! Happy you've got things back and working :)

Cheers!