WEBVTT NOTE Created by CaptionSync from Automatic Sync Technologies www.automaticsync.com 00:00:00.286 --> 00:00:06.316 align:middle Welcome! Good afternoon! 00:00:06.686 --> 00:00:12.226 align:middle I'm here to talk about building apps for immutable servers. 00:00:12.886 --> 00:00:19.346 align:middle But that's not slightly true. 00:00:19.466 --> 00:00:22.626 align:middle While I was building the talk, I slightly changed the title 00:00:22.766 --> 00:00:27.066 align:middle so it's more scaling apps with immutable servers. 00:00:28.586 --> 00:00:31.986 align:middle And as just a disclaimer, there will be no code. 00:00:32.666 --> 00:00:34.106 align:middle We are not going to build any app. 00:00:34.676 --> 00:00:40.286 align:middle So, you should expect to know more about servers, and how you can use immutable servers 00:00:40.286 --> 00:00:43.636 align:middle for your app and how you should use them, and if you should use them. 00:00:44.966 --> 00:00:47.146 align:middle So, a bit about me. 00:00:47.496 --> 00:00:48.596 align:middle My name is Daniel Gomes. 00:00:48.596 --> 00:00:49.276 align:middle I am Portuguese. 00:00:49.276 --> 00:00:49.906 align:middle I'm from Lisbon. 00:00:49.906 --> 00:00:52.636 align:middle I'm a software engineer at Teamleader. 00:00:53.106 --> 00:00:58.046 align:middle I am also the organizer of @phplx, the user group, the PHP user group in Lisbon. 00:00:58.556 --> 00:01:01.296 align:middle You can find me on twitter and sometimes I blog. 00:01:01.636 --> 00:01:05.396 align:middle So, before I start I just want to know, how many Portuguese people I have here. 00:01:05.586 --> 00:01:06.656 align:middle Can you just raise your hand? 00:01:08.976 --> 00:01:11.596 align:middle A few? Probably 10. 00:01:11.936 --> 00:01:12.436 align:middle That's nice. 00:01:13.496 --> 00:01:15.366 align:middle So let's get started. 00:01:15.366 --> 00:01:17.016 align:middle Some topics that I'm going to cover. 00:01:17.016 --> 00:01:19.156 align:middle We are going to cover different types of scaling. 00:01:19.556 --> 00:01:24.056 align:middle We are going to cover configuration drift and management, different types of servers, 00:01:24.426 --> 00:01:29.006 align:middle and also a build process for your web or application server. 00:01:29.006 --> 00:01:35.026 align:middle So, at the beginning, usually when everyone at the office starts to work in a new product 00:01:35.026 --> 00:01:41.336 align:middle or a new application, you start some wire framing, some designs and into the site. 00:01:41.336 --> 00:01:44.946 align:middle And then you start doing the code, there. 00:01:45.066 --> 00:01:47.646 align:middle and then you need to decide our first infrastructure. 00:01:48.296 --> 00:01:53.736 align:middle But usually, everybody already should do something like this, did do something like this. 00:01:54.016 --> 00:01:57.596 align:middle Is that you put everything in one server because you are just starting, 00:01:57.596 --> 00:02:01.796 align:middle you want to just have something really quickly to the customer to get some feedback. 00:02:02.096 --> 00:02:04.286 align:middle Right? So you end up with something like this. 00:02:04.396 --> 00:02:08.606 align:middle How many have you ever ended up with the application, the database, cron jobs, 00:02:09.086 --> 00:02:11.796 align:middle whatever in the same machine that your application is running. 00:02:12.426 --> 00:02:13.216 align:middle Can you raise your hand? 00:02:14.716 --> 00:02:17.086 align:middle Yeah, 80% of the room. 00:02:17.086 --> 00:02:18.996 align:middle So, I have been there as well. 00:02:18.996 --> 00:02:20.636 align:middle So, everybody does this and this is okay. 00:02:20.636 --> 00:02:21.576 align:middle No problem. 00:02:22.556 --> 00:02:25.146 align:middle Then you launch your app, everything went well. 00:02:25.596 --> 00:02:28.006 align:middle Then, users start using your application, 00:02:28.536 --> 00:02:33.646 align:middle you start having some rough problems and your server is unavailable. 00:02:34.176 --> 00:02:37.316 align:middle This is a good thing because if your server is unavailable 00:02:37.826 --> 00:02:41.086 align:middle because of the users - it's, it's nice, it's working. 00:02:41.656 --> 00:02:42.736 align:middle You are getting results, right? 00:02:43.666 --> 00:02:49.916 align:middle So, the CPU, memory, whatever it is - is not handling the usage, the load, 00:02:50.256 --> 00:02:52.006 align:middle and then it's the time that you say: well. 00:02:53.036 --> 00:02:53.876 align:middle we need to scale, right? 00:02:55.806 --> 00:02:57.096 align:middle It's easy, right? 00:02:57.366 --> 00:02:58.116 align:middle Scale is easy. 00:02:58.696 --> 00:03:00.556 align:middle Is that easy? 00:03:00.636 --> 00:03:02.346 align:middle Scaling? Do you think? 00:03:03.616 --> 00:03:09.396 align:middle Do you think that you can scale this thing? 00:03:09.526 --> 00:03:12.986 align:middle You are correct in one point, but in another one - you are not correct. 00:03:12.986 --> 00:03:13.566 align:middle Yes and no. 00:03:15.206 --> 00:03:16.926 align:middle You can vertical scale it, right? 00:03:17.906 --> 00:03:18.766 align:middle That's the first step. 00:03:18.796 --> 00:03:23.106 align:middle So, you add more memory if you need, you add more CPU, 00:03:23.886 --> 00:03:31.796 align:middle more this or you're just fine tuning your php.ini, your PHP-FPM, Nginx, whatever you have. 00:03:32.506 --> 00:03:34.946 align:middle These are the options that you have with the vertical scaling. 00:03:35.416 --> 00:03:39.196 align:middle Otherwise, then you would have more hardware to add more CPU or memory. 00:03:39.816 --> 00:03:43.856 align:middle So, and for horizontal scaling. 00:03:45.196 --> 00:03:47.066 align:middle So, if you want to add... 00:03:47.196 --> 00:03:49.406 align:middle so, the goal with the horizontal scaling is, basically, 00:03:49.406 --> 00:03:52.146 align:middle going from one machine to more than one machine. 00:03:52.226 --> 00:03:58.116 align:middle In this example, is 4 instances of your server, but it can be 2 instances, 3, 10, 20. 00:03:58.416 --> 00:04:02.356 align:middle It's depending on your, on your use case of course. 00:04:02.526 --> 00:04:10.156 align:middle So, let's say that with the current server that we had, we create a cluster 00:04:10.156 --> 00:04:11.876 align:middle and we have this kind of architecture. 00:04:12.626 --> 00:04:14.326 align:middle So the question is: Will this work? 00:04:15.536 --> 00:04:16.146 align:middle Of course, not. 00:04:16.456 --> 00:04:17.836 align:middle Yeah? Nope! 00:04:18.926 --> 00:04:24.076 align:middle Why? Because you have persistence and state inside our server. 00:04:24.076 --> 00:04:27.836 align:middle So, if I do a change in this machine, in these instances, 00:04:28.126 --> 00:04:31.616 align:middle what will happen to the other ones, they will not be in sync, right? 00:04:31.706 --> 00:04:33.286 align:middle So, what is the point of this? 00:04:33.286 --> 00:04:34.576 align:middle Does not make any sense. 00:04:35.146 --> 00:04:39.696 align:middle So, what you need - you need new architecture, this is not the only way to go. 00:04:39.696 --> 00:04:40.996 align:middle This is an example, of course. 00:04:41.336 --> 00:04:44.776 align:middle So, you put the load balancer on front of your application tier 00:04:44.776 --> 00:04:50.446 align:middle or web point tier you have a cluster of databases, a cluster for AMQP, 00:04:50.916 --> 00:04:53.566 align:middle a cluster for any kind of infrastructure... 00:04:53.936 --> 00:04:56.026 align:middle the persistence that you need. 00:04:56.376 --> 00:05:01.246 align:middle Okay? You can also have your assets in a CDN, whatever. 00:05:02.886 --> 00:05:05.306 align:middle So, and you end up with something like this, right? 00:05:05.766 --> 00:05:06.396 align:middle That's okay. 00:05:06.816 --> 00:05:07.426 align:middle This should work. 00:05:09.246 --> 00:05:15.856 align:middle But yeah, you have it, everything is working fine, it kind of handled the load, 00:05:16.236 --> 00:05:17.566 align:middle but then you have a bug in production. 00:05:18.846 --> 00:05:19.496 align:middle This happens. 00:05:19.836 --> 00:05:23.316 align:middle Who did add a bug in production? 00:05:23.316 --> 00:05:25.626 align:middle Everyone, I think everyone, the ones are 00:05:25.626 --> 00:05:27.856 align:middle that don't raise their hands are a bit shy I would say. 00:05:28.806 --> 00:05:33.946 align:middle So, and yeah, sometimes you are in a rush or it's in the middle of the night 00:05:33.946 --> 00:05:38.896 align:middle and you just need, I just want to fix this thing and I went to go sleep and you do SSH... 00:05:39.176 --> 00:05:41.126 align:middle Oh, I figured out what it is, I change it. 00:05:41.126 --> 00:05:41.916 align:middle It's okay. 00:05:42.786 --> 00:05:44.306 align:middle But in a cluster, this is not a good idea. 00:05:44.306 --> 00:05:48.936 align:middle Right? So, the problem was solved and you go to sleep. 00:05:49.776 --> 00:05:50.306 align:middle You think. 00:05:51.156 --> 00:05:52.306 align:middle But was it really solved? 00:05:52.306 --> 00:05:54.676 align:middle If you have horizontal scaling. 00:05:55.016 --> 00:05:59.486 align:middle If you only change one instance, do you think it's solved? 00:05:59.626 --> 00:06:01.096 align:middle No. I see some... 00:06:01.966 --> 00:06:02.886 align:middle some heads saying no. 00:06:02.886 --> 00:06:04.816 align:middle So yeah, you are right, it's not solved. 00:06:04.816 --> 00:06:05.916 align:middle This is what happens. 00:06:06.376 --> 00:06:08.346 align:middle So you changed one instance on the top right. 00:06:08.856 --> 00:06:09.906 align:middle That's the fixed one. 00:06:10.316 --> 00:06:12.146 align:middle But the other ones - you didn't change them. 00:06:13.236 --> 00:06:15.406 align:middle So, you have an inconsistent state in your... 00:06:15.516 --> 00:06:16.836 align:middle in your instance, in your cluster? 00:06:16.906 --> 00:06:17.946 align:middle This is not good. 00:06:19.606 --> 00:06:23.086 align:middle And, at this point, you enter into the configuration drift. 00:06:23.806 --> 00:06:27.306 align:middle And configuration drift is basically manual adopt changes 00:06:27.666 --> 00:06:29.526 align:middle to your servers that are not recorded. 00:06:30.016 --> 00:06:33.496 align:middle If you don't record them - you cannot reapply them to the other instances. 00:06:34.306 --> 00:06:38.706 align:middle So you have difference, different states between your servers in your cluster. 00:06:39.126 --> 00:06:44.476 align:middle So, this is configuration drift, and this is your cluster 00:06:44.476 --> 00:06:46.736 align:middle after a few months if you continue to do that. 00:06:46.736 --> 00:06:53.506 align:middle If you change the instance with SSH without recording and reapplying to the other instances, 00:06:53.736 --> 00:06:54.946 align:middle you end up with something like this. 00:06:55.876 --> 00:07:00.716 align:middle And what happens is that your servers become a snowflake server. 00:07:01.026 --> 00:07:04.296 align:middle I don't know how many of you know what is a snowflake server? 00:07:06.056 --> 00:07:06.946 align:middle Some hands.. 00:07:06.946 --> 00:07:11.686 align:middle so, yeah. So probably, most of you, or a huge part, 00:07:11.686 --> 00:07:17.606 align:middle a big part of you will have snowflake servers today that are long running servers. 00:07:18.756 --> 00:07:20.936 align:middle There are no consistency between them in your cluster. 00:07:22.026 --> 00:07:23.206 align:middle They hard to reproduce. 00:07:23.206 --> 00:07:28.026 align:middle So, you cannot just build a new, a new server from the image that you have 00:07:28.486 --> 00:07:30.226 align:middle because if you don't record the changes, 00:07:30.226 --> 00:07:33.176 align:middle you cannot reapply them to a new server that you launch. 00:07:33.906 --> 00:07:35.056 align:middle So, you cannot reproduce. 00:07:35.056 --> 00:07:37.776 align:middle And this is the most important for me - lack of confidence. 00:07:38.346 --> 00:07:43.286 align:middle Do trust your, your server, your cluster, if you have different states in your instances? 00:07:44.946 --> 00:07:45.316 align:middle No, right? 00:07:47.456 --> 00:07:55.606 align:middle So, how can we solve this? 00:07:58.466 --> 00:08:03.536 align:middle So, you can use configuration management with automation tools. 00:08:04.336 --> 00:08:08.646 align:middle And the configuration management is just the process of systematically, 00:08:08.986 --> 00:08:14.526 align:middle systematically handling changes to a system in a way that it maintains integrity over time. 00:08:16.376 --> 00:08:17.316 align:middle Let's see an example. 00:08:17.546 --> 00:08:23.946 align:middle So, let's say that I want to spin up a new server or I have a running server and I went 00:08:23.946 --> 00:08:26.666 align:middle to do some changes to my, to my server. 00:08:27.216 --> 00:08:30.556 align:middle What you need to do is you first change your configuration files 00:08:31.496 --> 00:08:36.936 align:middle and then you run the configuration management to apply that changes into all your servers. 00:08:36.936 --> 00:08:41.896 align:middle So, that way you end up with all your servers in the desire state. 00:08:43.566 --> 00:08:45.476 align:middle Let's see how this works. 00:08:45.636 --> 00:08:48.076 align:middle So I have my 4 instances. 00:08:48.206 --> 00:08:48.706 align:middle I have my... 00:08:48.706 --> 00:08:56.856 align:middle I want to apply the state D. I run my, my configuration management and after it runs, 00:08:57.606 --> 00:09:02.256 align:middle hopefully if nothing goes wrong, everything will be on state D. Yeah, 00:09:02.256 --> 00:09:05.936 align:middle because even if you run configuration management, something can go wrong 00:09:06.486 --> 00:09:08.996 align:middle if you don't have internet and stuff like that. 00:09:10.136 --> 00:09:14.476 align:middle So what tools can we use to... 00:09:14.476 --> 00:09:16.696 align:middle for configuration management? 00:09:16.696 --> 00:09:17.746 align:middle There are a lot, there are... 00:09:17.746 --> 00:09:19.026 align:middle These are some that I know. 00:09:20.586 --> 00:09:24.786 align:middle You have Chef, Puppet, SaltStack, Ansible. 00:09:25.126 --> 00:09:27.516 align:middle How many of you have already worked with some of those tools? 00:09:28.006 --> 00:09:32.626 align:middle So, how many of you work more with Ansible or Puppet? 00:09:32.626 --> 00:09:38.146 align:middle Ansible? Yeah, I think, yeah, most of the people I think works with Ansible. 00:09:38.786 --> 00:09:39.746 align:middle I prefer Ansible too. 00:09:39.796 --> 00:09:41.996 align:middle It's more easy, for me. 00:09:43.186 --> 00:09:45.816 align:middle So, let's do just a quick recap. 00:09:46.136 --> 00:09:49.986 align:middle So you have seen different types of scalings, vertical and horizontal. 00:09:50.176 --> 00:09:52.316 align:middle You have seen what configuration drift is. 00:09:53.606 --> 00:09:57.456 align:middle How you turn your servers, all your services can become a snowflake server 00:09:57.686 --> 00:09:59.026 align:middle and the problems that they have. 00:09:59.836 --> 00:10:03.066 align:middle And how you can use configuration management to solve those problems. 00:10:03.066 --> 00:10:07.406 align:middle Right? So we are reaching half of the presentation. 00:10:07.406 --> 00:10:13.176 align:middle So this will be more topics so, I think it's the perfect time for you, if you have any questions 00:10:13.176 --> 00:10:16.676 align:middle so far to make it now because probably at the end you will not remember. 00:10:17.086 --> 00:10:17.976 align:middle Any questions so far? 00:10:19.176 --> 00:10:21.096 align:middle Nope. So, let's continue. 00:10:22.606 --> 00:10:24.856 align:middle So, let me present you Phoenix servers. 00:10:25.206 --> 00:10:26.876 align:middle Anyone of you know, Phoenix servers? 00:10:29.866 --> 00:10:31.406 align:middle Only one I think. 00:10:31.936 --> 00:10:36.986 align:middle Oh, nice! So, Phoenix Server! 00:10:37.056 --> 00:10:43.766 align:middle Quoting Martin Fowler: a server should be like a phoenix, like regularly rising from the ashes. 00:10:44.786 --> 00:10:45.426 align:middle Yeah, that's true. 00:10:45.616 --> 00:10:49.246 align:middle So, if you do this, you will avoid configuration drifts. 00:10:49.846 --> 00:10:56.546 align:middle You have disposal, disposable servers and servers that can be built from scratch, right? 00:10:57.346 --> 00:11:01.596 align:middle Because you can apply everything with the configuration management. 00:11:01.906 --> 00:11:06.146 align:middle So, you're sure always the state when the server is built. 00:11:06.426 --> 00:11:12.226 align:middle So you can just terminate an instance, and create a new one, and you are done. 00:11:12.666 --> 00:11:13.826 align:middle That's nice. 00:11:15.356 --> 00:11:17.086 align:middle So, let's see an example. 00:11:17.956 --> 00:11:20.056 align:middle So I have my 4 instances. 00:11:20.056 --> 00:11:21.216 align:middle I terminate an instance. 00:11:21.456 --> 00:11:22.406 align:middle Instance terminated. 00:11:22.976 --> 00:11:23.966 align:middle A new one is spinning up. 00:11:24.866 --> 00:11:28.776 align:middle I apply state C, run the configuration management, and voila. 00:11:30.036 --> 00:11:30.366 align:middle It's done. 00:11:31.716 --> 00:11:32.916 align:middle Everything's working fine. 00:11:33.566 --> 00:11:38.896 align:middle Cool. So, spinning up new servers is not a problem anymore. 00:11:39.616 --> 00:11:44.576 align:middle You are not afraid of doing that, that everything will not be in the same state 00:11:45.136 --> 00:11:47.476 align:middle because you have the tools in place to help you. 00:11:49.286 --> 00:11:53.136 align:middle But, is idempotence guaranteed here? 00:11:53.296 --> 00:11:54.446 align:middle Do you think? 00:11:56.016 --> 00:11:57.496 align:middle Nope. Yeah, you're right. 00:11:58.706 --> 00:11:59.146 align:middle Do you know why? 00:12:02.156 --> 00:12:05.896 align:middle What if the package repository are down when you are doing this? 00:12:07.446 --> 00:12:12.126 align:middle And imagine that you are having a major outage or you have a problem in your servers 00:12:12.516 --> 00:12:15.806 align:middle and you really need to deploy a new version of your server. 00:12:16.866 --> 00:12:23.186 align:middle And, S3 was down for 48 hours, something like that, and it was a mess. 00:12:23.186 --> 00:12:28.646 align:middle Nobody could download anything because other repositories were there in S3 a couple 00:12:28.646 --> 00:12:31.996 align:middle of few years ago, I remember that, was really crazy: what the... 00:12:32.046 --> 00:12:33.846 align:middle is only S3? 00:12:33.846 --> 00:12:35.176 align:middle And I cannot download anything. 00:12:36.066 --> 00:12:40.566 align:middle I'm not even talking about Composer, installing Apache or Nginx. 00:12:40.566 --> 00:12:41.076 align:middle It can happen. 00:12:42.616 --> 00:12:47.366 align:middle So, let's say that you have this process that you spin up a new server, 00:12:47.646 --> 00:12:52.976 align:middle you run the configuration management, the configuration management will install packages, 00:12:53.326 --> 00:12:58.586 align:middle create folders, create users, upload the app, and then when it's finished, 00:12:58.586 --> 00:13:00.976 align:middle I have my server ready and in the desired state. 00:13:01.316 --> 00:13:07.086 align:middle So let's say that at certain point the internet or the repositories are unavailable. 00:13:07.936 --> 00:13:11.586 align:middle This means, that I can create the folder, I can create the users, 00:13:12.006 --> 00:13:13.146 align:middle but I cannot install the package. 00:13:13.516 --> 00:13:15.886 align:middle So I will not have my server in my desired state. 00:13:16.136 --> 00:13:24.726 align:middle So, yeah, configuration management is nice, but if I don't have internet that does not serve me. 00:13:25.556 --> 00:13:29.436 align:middle Right? If I need to deploy right now without the internet, it will not work. 00:13:30.606 --> 00:13:35.526 align:middle And that's not good for our business and for you, if you are running critical stuff. 00:13:36.826 --> 00:13:38.836 align:middle So how can we fix this? 00:13:39.156 --> 00:13:40.136 align:middle Anyone has an idea? 00:13:40.696 --> 00:13:50.386 align:middle What? That would work, but if that instance has problems, you're not fixing anything, 00:13:50.386 --> 00:13:55.146 align:middle you are just ensuring that we have a new, but yeah, that would be, could be a solution in it. 00:13:56.056 --> 00:14:00.466 align:middle But yeah, you can have immutable servers - that will help you. 00:14:00.466 --> 00:14:01.016 align:middle Let's see why. 00:14:02.726 --> 00:14:08.966 align:middle So, according to Kief Morris: An immutable server is a server, that once deployed, 00:14:08.966 --> 00:14:14.266 align:middle is never modified, merely replaced with a new updated instance. 00:14:15.106 --> 00:14:16.046 align:middle This is really powerful. 00:14:19.416 --> 00:14:21.076 align:middle So what does this mean? 00:14:21.146 --> 00:14:25.526 align:middle This means that when I do a build, I have the final image with everything baked in. 00:14:25.946 --> 00:14:28.766 align:middle This means that my application is there, 00:14:29.356 --> 00:14:33.146 align:middle all packages are installed, everything is configured. 00:14:33.146 --> 00:14:38.146 align:middle It's just spinning up a new, a new server and application will start running. 00:14:38.556 --> 00:14:39.846 align:middle You don't need to do anything. 00:14:40.116 --> 00:14:41.526 align:middle You don't need, well, you need the internet 00:14:41.596 --> 00:14:46.036 align:middle because if you are using aws - you need, yeah, but that's... 00:14:46.336 --> 00:14:48.326 align:middle than you cannot do anything right. 00:14:48.326 --> 00:14:52.296 align:middle If you don't have the internet in your local machine and if you need that to spin 00:14:52.296 --> 00:14:55.586 align:middle up new servers - that immutable servers won't help you 00:14:55.586 --> 00:15:01.096 align:middle because that's not the problem of the immutable servers. 00:15:01.096 --> 00:15:05.276 align:middle You cannot perform any change after the image is built. 00:15:05.806 --> 00:15:11.686 align:middle So, if you want to, if you did a mistake and you want to fix it, you have to build a new image. 00:15:12.116 --> 00:15:18.136 align:middle Okay? And you need to include all scripts, everything you needed for the application 00:15:18.136 --> 00:15:26.666 align:middle to start on boot because you cannot SSH into the machine, you cannot do anything like that. 00:15:26.986 --> 00:15:30.996 align:middle Okay? That's the point and the goal of immutable servers. 00:15:31.396 --> 00:15:36.696 align:middle So, this is really easy to scale out, to deploy and rollback, right? 00:15:37.426 --> 00:15:43.626 align:middle Because let's say that you deploy something, you have a bug, and if you don't save the image 00:15:43.626 --> 00:15:47.626 align:middle of your last version, you have to build everything again to roll back. 00:15:48.046 --> 00:15:49.006 align:middle That is not a good thing. 00:15:49.706 --> 00:15:54.396 align:middle But usually people's save the last image so that will not be an issue, but still. 00:15:55.776 --> 00:16:00.506 align:middle So, therefore you have much more confidence in what you have 00:16:00.716 --> 00:16:02.366 align:middle in your servers, in your infrastructure. 00:16:03.946 --> 00:16:05.056 align:middle You rely on them. 00:16:05.376 --> 00:16:05.816 align:middle You can... 00:16:06.046 --> 00:16:12.546 align:middle you have reliability on your infrastructure and what's the most important - trust. 00:16:13.016 --> 00:16:14.626 align:middle Your business will trust what you have. 00:16:14.926 --> 00:16:18.426 align:middle You are going to trust, everyone will trust in what you have. 00:16:18.426 --> 00:16:23.796 align:middle Right? And I think this is also very important for me. 00:16:23.796 --> 00:16:29.416 align:middle You can sleep very quietly and you know that it's not because your server is having issues 00:16:29.416 --> 00:16:31.336 align:middle that you are not, you are going to wake up. 00:16:31.336 --> 00:16:34.996 align:middle If you have auto-scaling then it will also help you. 00:16:35.536 --> 00:16:37.546 align:middle But let's talk about auto scaling after. 00:16:38.776 --> 00:16:45.686 align:middle And also, I don't know if you know the 12 factor app, but the 12 factor app is a methodology 00:16:45.686 --> 00:16:50.206 align:middle for software as a service that basically says these 12 points. 00:16:50.206 --> 00:16:54.596 align:middle I'm not listing all the points here, but it says that you should have this possibility, 00:16:54.976 --> 00:17:00.386 align:middle your configuration should be in the environment for the application. 00:17:01.276 --> 00:17:03.636 align:middle You should have dev and production parity, 00:17:04.366 --> 00:17:08.166 align:middle and the backing services should be attached to your application server. 00:17:08.636 --> 00:17:12.846 align:middle Backing services is your database, AMQP, whatever, 00:17:13.486 --> 00:17:18.726 align:middle anything that is not your application running, not your application code running. 00:17:19.076 --> 00:17:24.806 align:middle So, any service that it uses to run, etc. And also, 00:17:25.106 --> 00:17:28.346 align:middle this one I discovered while I was doing some research for it. 00:17:28.756 --> 00:17:33.726 align:middle There is the Reproducible-Builds.org that is, 00:17:33.726 --> 00:17:39.436 align:middle they define that a reproducible build is something that you can verify: 00:17:39.776 --> 00:17:42.226 align:middle the path from source code to binary. 00:17:42.756 --> 00:17:49.106 align:middle So, this means that if you, if you give the source code, the build, the environment, 00:17:49.106 --> 00:17:51.786 align:middle and the build instructions - every time that you do that build, 00:17:52.056 --> 00:17:54.486 align:middle you will have the same outcome always. 00:17:54.916 --> 00:17:55.706 align:middle Exactly the same. 00:17:55.896 --> 00:17:58.276 align:middle It's like the checksum. 00:17:59.206 --> 00:18:05.166 align:middle Also, Symfony is part of this organization and there is a blog post on the resources. 00:18:06.456 --> 00:18:09.766 align:middle So, how do we do, how do we build it? 00:18:17.296 --> 00:18:20.836 align:middle You need continuous Integration and continuous deployment pipelines. 00:18:21.536 --> 00:18:26.086 align:middle Continuous deployment is not necessary but it would be nice because it makes sense: 00:18:26.136 --> 00:18:30.086 align:middle if you do a change, it can be very quickly to go to production 00:18:30.366 --> 00:18:34.156 align:middle if you're trusting what you are doing, depends on your process, of course. 00:18:35.396 --> 00:18:38.886 align:middle And one very important distinction that you need to do is: 00:18:39.416 --> 00:18:41.996 align:middle a build is different from running your server. 00:18:42.696 --> 00:18:50.416 align:middle If you build - you create an image that can then be launched, can be used to create a new server. 00:18:50.636 --> 00:18:53.286 align:middle Okay? A build just creates an image, period. 00:18:53.736 --> 00:18:56.096 align:middle It's just that. 00:18:56.306 --> 00:18:59.456 align:middle So let's say that you have this simple build process. 00:18:59.456 --> 00:19:00.496 align:middle This is just an example. 00:19:01.176 --> 00:19:06.016 align:middle So, you spin up a new server, you run your configuration management, 00:19:06.616 --> 00:19:11.726 align:middle you configure the application environment, you have the server in your desired state. 00:19:12.216 --> 00:19:14.656 align:middle Then, you bake in everything that you need for your app 00:19:15.286 --> 00:19:17.316 align:middle and you have your application final image. 00:19:18.066 --> 00:19:21.096 align:middle Okay? That's a simple one. 00:19:22.186 --> 00:19:26.106 align:middle It's more or less easier, but it can have some issues. 00:19:26.666 --> 00:19:33.256 align:middle Let's say that you want to apply some security updates. 00:19:33.676 --> 00:19:34.866 align:middle It can be more tricky. 00:19:35.596 --> 00:19:38.136 align:middle I usually prefer a multistep build process. 00:19:38.136 --> 00:19:39.226 align:middle This is my preference. 00:19:39.226 --> 00:19:45.206 align:middle Okay? So, basically, I like to have a base image where I have my OS hardening 00:19:45.966 --> 00:19:50.716 align:middle and all common tools that I might use because you might have a server for PHP, 00:19:50.716 --> 00:19:55.786 align:middle you might have server for Go, you can have different servers, but you want your base server 00:19:55.786 --> 00:19:58.606 align:middle to have all the tools that you need for your infrastructure. 00:19:58.606 --> 00:20:02.696 align:middle So, we don't have to configure different things for different types of servers. 00:20:03.236 --> 00:20:08.726 align:middle Okay? Then I have my application base image where I install all the software needed 00:20:08.826 --> 00:20:13.236 align:middle for the application to run, I create the folders, the users, permissions, 00:20:13.616 --> 00:20:17.826 align:middle everything that you need before you just upload the application. 00:20:18.606 --> 00:20:22.296 align:middle And you put everything that it needs to. 00:20:22.446 --> 00:20:27.836 align:middle Okay? It's just the basic stuff to run: Nginx, Apache, that sort of things. 00:20:27.836 --> 00:20:35.196 align:middle Okay? And then, the last build would be creating the application image, 00:20:35.956 --> 00:20:38.766 align:middle the final image that you use to deploy to create new servers. 00:20:38.886 --> 00:20:46.066 align:middle So, you upload your app, boots, you, you make any script to boot the application on start up - 00:20:46.476 --> 00:20:49.526 align:middle composer install, to clear cache, all that stuff. 00:20:49.676 --> 00:20:53.496 align:middle That way you know that when you deploy the application, 00:20:53.496 --> 00:20:55.616 align:middle the server is ready to receive traffic. 00:20:56.316 --> 00:21:01.546 align:middle Okay? So system upgrades, security updates. 00:21:01.546 --> 00:21:06.666 align:middle Usually, I like to do them on the base image, but this means that if I do them 00:21:06.666 --> 00:21:10.836 align:middle in the base image - I will have to rebuild everything. 00:21:11.076 --> 00:21:12.006 align:middle It's a chain, right? 00:21:12.836 --> 00:21:16.986 align:middle If I do a change in the first - I have to build again everything. 00:21:17.876 --> 00:21:23.876 align:middle It's a bit overload, it can be costly in terms of time, but yeah, it's, 00:21:24.056 --> 00:21:26.256 align:middle I think it's a more structured process. 00:21:27.326 --> 00:21:31.016 align:middle It has advantage and disadvantage, but yeah. 00:21:31.816 --> 00:21:37.546 align:middle So, any application security by application is not your application, 00:21:37.546 --> 00:21:41.656 align:middle not a bug you have in your security... 00:21:41.966 --> 00:21:47.646 align:middle issue you have in your application, it can be a security in Nginx, so a security problem 00:21:47.646 --> 00:21:52.686 align:middle in Nginx or PHP-FPM, or whatever, that should be done in the base image 00:21:52.966 --> 00:21:56.016 align:middle or any configuration should also be done there. 00:21:56.016 --> 00:22:00.886 align:middle So this is basically, this is a multistep process, 00:22:00.966 --> 00:22:02.876 align:middle I like this process, it has a bit overhead. 00:22:03.786 --> 00:22:07.736 align:middle It's not for everyone, but yeah, it's an example. 00:22:08.976 --> 00:22:16.526 align:middle So building tips for a Symfony app, you should run the composer install, cache clear, 00:22:16.526 --> 00:22:19.346 align:middle clear any cache that you might have in your application 00:22:19.746 --> 00:22:22.596 align:middle and you build your assets or push them to a CDN. 00:22:23.206 --> 00:22:26.066 align:middle So, there are more but these are the ones that I remembered. 00:22:28.876 --> 00:22:33.336 align:middle So what tools can you use to help us building our servers? 00:22:35.566 --> 00:22:38.756 align:middle We have some, this is just a subset of them 00:22:38.756 --> 00:22:41.656 align:middle because otherwise it will be a slide full of logos. 00:22:42.656 --> 00:22:43.916 align:middle But I will just mentioned some. 00:22:43.916 --> 00:22:47.386 align:middle So we can use Vault, I don't know, if how many of you know Vault? 00:22:47.576 --> 00:22:52.466 align:middle So, probably one third of the audience. 00:22:52.466 --> 00:22:59.566 align:middle Vault is, basically to store your secrets and that way you don't need to have ENV vars 00:22:59.566 --> 00:23:02.756 align:middle in your, in your application and all that stuff. 00:23:02.756 --> 00:23:04.726 align:middle Vault will store your secrets for you. 00:23:04.926 --> 00:23:07.116 align:middle And when, in your build process you go to Vault 00:23:07.466 --> 00:23:09.966 align:middle to fetch those secrets, depending on your environment. 00:23:10.786 --> 00:23:12.306 align:middle Packer, how many of you know Packer? 00:23:13.726 --> 00:23:15.436 align:middle A bit less, probably 10 percent. 00:23:15.896 --> 00:23:18.256 align:middle So, Packer is a really nice tool from HashiCorp. 00:23:18.256 --> 00:23:18.966 align:middle I really love it. 00:23:19.606 --> 00:23:22.126 align:middle With Packer you can build your server 00:23:22.126 --> 00:23:26.046 align:middle for different platforms at the same time in parallel. 00:23:26.266 --> 00:23:26.776 align:middle It's amazing. 00:23:26.996 --> 00:23:29.296 align:middle So you can create your production container. 00:23:29.296 --> 00:23:33.006 align:middle Let's say you want to use Docker, you can create your Docker production container. 00:23:33.006 --> 00:23:40.376 align:middle If you use Vagrant for development, you can also create your Vagrant image, you can deploy, 00:23:40.376 --> 00:23:46.226 align:middle you can create stuff for Amazon, for Google in the same, in same configuration file. 00:23:46.646 --> 00:23:47.476 align:middle It's really powerful. 00:23:48.136 --> 00:23:53.996 align:middle You have Docker, of course, Azure, AWS, Google Platform, Chef, Puppet. 00:23:54.336 --> 00:23:58.986 align:middle So, there are a lot of tools to help us to create these immutable servers. 00:24:03.326 --> 00:24:07.766 align:middle So, what are the next steps? 00:24:08.416 --> 00:24:13.326 align:middle If you can have immutable servers, you are safe, you can trust in our infrastructure, 00:24:13.776 --> 00:24:16.806 align:middle but then you can start doing more crazy things. 00:24:16.806 --> 00:24:21.756 align:middle So, we can start thinking about high availability, auto scaling. 00:24:22.616 --> 00:24:24.656 align:middle How many of you already do auto scaling here? 00:24:25.716 --> 00:24:28.586 align:middle Hmm, a few. 00:24:30.026 --> 00:24:32.756 align:middle Probably, five percent of the, of the audience. 00:24:33.226 --> 00:24:39.126 align:middle Nice. So, if you have immutable servers - it's really easy for you to just say: Okay, 00:24:39.126 --> 00:24:43.686 align:middle if my CPU is at 80 percent, launch me a new instance. 00:24:44.346 --> 00:24:44.746 align:middle It's easy. 00:24:45.346 --> 00:24:46.316 align:middle It's everything baked in. 00:24:46.316 --> 00:24:51.656 align:middle It's just a matter of a minute, probably, depending on where you have your infrastructure. 00:24:51.986 --> 00:24:52.736 align:middle And now it's built. 00:24:52.736 --> 00:25:00.506 align:middle It should be really easy to auto scale and also to scale down your cluster. 00:25:01.236 --> 00:25:05.486 align:middle Or, if I'm not having that much traffic - kill me some instances and you save money. 00:25:06.566 --> 00:25:09.836 align:middle Also you can auto scale your web tier, application tier. 00:25:09.836 --> 00:25:13.566 align:middle It will always depend on your architecture that you have in your infrastructure. 00:25:13.566 --> 00:25:18.966 align:middle Okay? How many of you have heard about Chaos Monkey from Netflix? 00:25:20.366 --> 00:25:22.736 align:middle Okay. 2% more or less? 00:25:22.796 --> 00:25:24.126 align:middle Three, five people? 00:25:24.126 --> 00:25:29.656 align:middle Yeah. So Chaos Monkey was a tool that Netflix had the need to create for them 00:25:29.716 --> 00:25:31.586 align:middle to test their own infrastructure. 00:25:32.046 --> 00:25:36.256 align:middle So, basically, Chaos Monkey goes into your cluster and says: 00:25:36.366 --> 00:25:38.906 align:middle hmm, I will kill this instance now. 00:25:39.676 --> 00:25:40.966 align:middle And it shuts down the instance. 00:25:42.056 --> 00:25:47.436 align:middle To stress test their application, how it reacts to a failure of an instance suddenly. 00:25:48.936 --> 00:25:49.736 align:middle That's a nice tool. 00:25:50.186 --> 00:25:55.026 align:middle How many of you would have the courage to just go now to your laptop and say: 00:25:55.026 --> 00:25:56.306 align:middle I will terminate this instance now. 00:26:00.046 --> 00:26:01.226 align:middle And it will run? 00:26:01.826 --> 00:26:02.686 align:middle I saw to it. 00:26:02.856 --> 00:26:04.856 align:middle Yeah, so that's nice. 00:26:04.856 --> 00:26:07.376 align:middle That's a good thing, but many of you will not have 00:26:07.376 --> 00:26:10.816 align:middle that confidence to do things like that, right? 00:26:11.366 --> 00:26:16.396 align:middle But then Netflix said: Well, one instance in one cluster is cool, is fine. 00:26:16.886 --> 00:26:19.246 align:middle But what if I shut down my entire region? 00:26:20.036 --> 00:26:21.336 align:middle That's Chaos Kong. 00:26:22.316 --> 00:26:26.986 align:middle So they basically say: Okay, you are good in one region 00:26:27.066 --> 00:26:29.236 align:middle if one instance or two failed - you're okay. 00:26:29.896 --> 00:26:32.496 align:middle Let's try shutdown the entire region. 00:26:35.316 --> 00:26:36.456 align:middle Now the fun starts. 00:26:37.106 --> 00:26:38.036 align:middle So the guys are crazy. 00:26:38.036 --> 00:26:41.696 align:middle Of course, many of us will not need anything like this, but yeah, 00:26:41.896 --> 00:26:45.726 align:middle when you have immutable servers and servers that you can trust, 00:26:45.726 --> 00:26:47.846 align:middle you can start thinking about these kinds of things. 00:26:48.626 --> 00:26:53.286 align:middle Of course you are not going to need them always or in every single job 00:26:53.286 --> 00:26:54.986 align:middle that you have or probably in your life. 00:26:55.616 --> 00:26:59.926 align:middle It all depends who you are, because there is not many Netflix out there I would say. 00:27:01.726 --> 00:27:03.446 align:middle So, some final notes. 00:27:05.916 --> 00:27:09.666 align:middle The goal of this talk is not for you to come out of here and tomorrow, 00:27:09.666 --> 00:27:13.886 align:middle start doing some immutable servers because as you can see it's a lot of work. 00:27:14.756 --> 00:27:17.156 align:middle It's some overhead, and you might not need them. 00:27:17.216 --> 00:27:22.366 align:middle So, my goal with this talk is more for you to think what you have in your application 00:27:22.636 --> 00:27:25.186 align:middle and start thinking, if I want to go to immutable servers, 00:27:25.466 --> 00:27:27.816 align:middle what I should do now to prepare for that. 00:27:28.436 --> 00:27:30.106 align:middle And that's a good exercise that you should do 00:27:30.486 --> 00:27:33.686 align:middle because you always say: Oh, we don't need to scale. 00:27:33.686 --> 00:27:36.536 align:middle But if you don't need this now, let's not do it. 00:27:37.066 --> 00:27:40.316 align:middle But when you know that you are going to need to scale, right? 00:27:40.896 --> 00:27:44.806 align:middle And if tomorrow something happens and your application starts receiving a lot of traffic, 00:27:45.326 --> 00:27:48.556 align:middle you will not have the time to change from vertical to horizontal. 00:27:49.616 --> 00:27:52.286 align:middle Trust me, it's not something like this and it's done. 00:27:52.556 --> 00:27:56.766 align:middle It's not. So, you should prepare really prepare your application, your infrastructure 00:27:57.356 --> 00:28:01.626 align:middle to start thinking about immutable servers to be able to achieve that one day. 00:28:01.626 --> 00:28:09.366 align:middle This does not mean that you are going to need them, so only use them if you really need them. 00:28:09.706 --> 00:28:16.306 align:middle Okay? That's my advice for you because they are really tricky. 00:28:16.956 --> 00:28:21.816 align:middle They are really useful, really nice, but they can be really tricky to get them. 00:28:22.086 --> 00:28:25.396 align:middle Okay? I have some resources here. 00:28:25.396 --> 00:28:27.596 align:middle I will probably update this slide. 00:28:27.596 --> 00:28:31.616 align:middle I will then share the slides and update some resources, but there are some resources 00:28:31.616 --> 00:28:36.026 align:middle that you can find out more information about this. 00:28:36.026 --> 00:28:37.006 align:middle Uh, thank you. 00:28:37.956 --> 00:28:41.256 align:middle Uh, I work at Teamleader, we are hiring, so if you like what we are doing 00:28:41.256 --> 00:28:42.706 align:middle and you want to find out, talk with me. 00:28:43.276 --> 00:28:44.056 align:middle Any questions? 00:28:47.186 --> 00:28:48.066 align:middle Questions, anyone? 00:28:48.366 --> 00:28:53.986 align:middle There is one, or I never throw a thing like this - it's my first time. 00:28:53.986 --> 00:28:56.476 align:middle You? Let me see if I'm. 00:28:56.766 --> 00:28:57.686 align:middle Ah, sorry! 00:29:09.236 --> 00:29:13.006 align:middle Yeah, a state in Symfony. 00:29:15.086 --> 00:29:16.246 align:middle Um, session. 00:29:18.286 --> 00:29:22.896 align:middle For instance, you should not store your sessions in your application server. 00:29:26.826 --> 00:29:31.016 align:middle Otherwise, if the user, the next request goes to another server, 00:29:31.466 --> 00:29:33.606 align:middle the server will not know if it's authenticated or not. 00:29:34.816 --> 00:29:39.196 align:middle Right? If you use the sessions for storing the state of the user 00:29:39.196 --> 00:29:44.586 align:middle for the application for instance. 00:29:46.826 --> 00:29:55.236 align:middle (Question inaudible) It depends on your use case. 00:29:55.236 --> 00:29:58.046 align:middle If the authentication user or anything that you have 00:29:58.046 --> 00:30:01.326 align:middle in your session is critical for you, then it's a problem. 00:30:02.126 --> 00:30:05.976 align:middle There is no rule for all the use cases - 00:30:06.016 --> 00:30:09.796 align:middle it depends on your application, what is critical for you or not. 00:30:10.416 --> 00:30:18.836 align:middle So yeah, for, for some applications, having state in, uh, in different states on the session 00:30:18.836 --> 00:30:21.886 align:middle in different places might not be a problem, but for others not. 00:30:22.736 --> 00:30:29.236 align:middle If I'm dealing like invoicing or Ecommerce that you want to check out stuff - that's an issue. 00:30:29.336 --> 00:30:33.586 align:middle Right? Can you throw it? 00:30:35.206 --> 00:30:37.066 align:middle Oh, there is another one. 00:30:37.066 --> 00:30:38.336 align:middle Oh, that will be art. 00:30:38.956 --> 00:30:39.426 align:middle Can you pass? 00:30:39.926 --> 00:30:42.146 align:middle I will throw for you. 00:30:42.956 --> 00:30:47.916 align:middle Well, I'm not good at this. 00:30:48.136 --> 00:30:51.726 align:middle Hi. I have one question about logging. 00:30:52.856 --> 00:30:55.716 align:middle For the immutable servers, we have to keep the log, 00:30:55.716 --> 00:30:58.236 align:middle of the logs of the log files of the applications. 00:30:58.886 --> 00:31:02.326 align:middle No, you should have event streams, um, log streams. 00:31:02.446 --> 00:31:07.306 align:middle So you should log, you should not store anything, logs, anything in your machine. 00:31:08.246 --> 00:31:16.416 align:middle So before the instance go down, like when it is de-scaling the system, so it should send the log 00:31:16.416 --> 00:31:20.846 align:middle to the streams, ports, or it's like it's a cron job type of thing 00:31:20.846 --> 00:31:25.506 align:middle like that will just upload the log files in a timely fashion manner. 00:31:25.896 --> 00:31:27.586 align:middle Yeah, yeah, true. 00:31:27.806 --> 00:31:31.866 align:middle When I say terminate is not like brute force terminate an instance. 00:31:31.866 --> 00:31:34.136 align:middle When you terminate the instance, you should have graceful, 00:31:34.806 --> 00:31:37.706 align:middle your application should graceful terminate everything. 00:31:37.706 --> 00:31:44.166 align:middle So, and also in a PHP-FPM and Nginx will not allow you, you can, you can configure it, 00:31:44.366 --> 00:31:48.556 align:middle to only terminate after it's not receiving more traffic and stuff like that. 00:31:49.036 --> 00:31:53.486 align:middle Okay? So yeah, you should be taking in consideration also um, 00:31:53.686 --> 00:31:56.396 align:middle finishing, finishing up the requests. 00:31:56.396 --> 00:32:00.616 align:middle If it's a cron job - you should send a signal for your application 00:32:01.016 --> 00:32:04.316 align:middle to terminate the, the workers and stuff like that. 00:32:04.736 --> 00:32:08.416 align:middle Yeah, there is some preparation for having these kinds of things. 00:32:09.086 --> 00:32:10.116 align:middle That's why it's not that easy. 00:32:11.706 --> 00:32:11.986 align:middle Thank you. 00:32:13.446 --> 00:32:17.686 align:middle Anymore questions? 00:32:18.066 --> 00:32:20.536 align:middle One more here. 00:32:21.966 --> 00:32:23.716 align:middle Go ahead. Sorry! 00:32:24.666 --> 00:32:24.926 align:middle Thank you. 00:32:25.576 --> 00:32:26.536 align:middle Are you... 00:32:26.646 --> 00:32:31.436 align:middle in my company we are using something similar and I was wondering about the overhead. 00:32:32.296 --> 00:32:39.076 align:middle So you have a certain build in production of that image and then 00:32:39.076 --> 00:32:43.356 align:middle of course you push some changes to the master replication and then you get another image, 00:32:44.376 --> 00:32:46.546 align:middle but during the night when you sleep quietly, 00:32:47.336 --> 00:32:48.996 align:middle you'll probably have to deploy the previous image. 00:32:48.996 --> 00:32:50.826 align:middle Right? The one that is in sync with the others. 00:32:51.746 --> 00:32:57.926 align:middle You don't start the full deployment of the new version of the server that you build. 00:32:57.926 --> 00:32:58.926 align:middle This get can get a lot of... 00:32:58.926 --> 00:33:03.096 align:middle You should create probably a new cluster and when everything is okay - you switch. 00:33:03.276 --> 00:33:07.186 align:middle Yeah. So there's a lot of overhead involved here. 00:33:07.876 --> 00:33:11.286 align:middle We have a problems this way, we always have to expand infrastructure. 00:33:11.656 --> 00:33:14.686 align:middle Yeah, but that, but that's the beauty, but you can create a new cluster 00:33:14.686 --> 00:33:18.666 align:middle with all the new instance and then just say: Okay, from now on I want the traffic to go 00:33:18.666 --> 00:33:22.876 align:middle to this site and if everything is not okay, let's move the traffic back. 00:33:22.876 --> 00:33:27.606 align:middle Or just say: Okay, let's try the new version, let's put just 10 percent of our traffic 00:33:27.606 --> 00:33:31.646 align:middle to the new version and if it's okay, let's move everything. 00:33:31.846 --> 00:33:36.406 align:middle So, it gives you a lot of flexibility, but yeah, also overhead, Yeah, there's like a one... 00:33:36.406 --> 00:33:41.836 align:middle if you don't need the immutable server and maybe you have to consider this 00:33:41.836 --> 00:33:45.116 align:middle when you start moving to this kind of deployment. 00:33:45.416 --> 00:33:48.466 align:middle That's why I'm saying that you should start preparing for them, 00:33:48.466 --> 00:33:50.876 align:middle but you should not go for them just because you want. 00:33:51.666 --> 00:33:54.326 align:middle You shouldn't go for them when you really need, but you should prepare 00:33:54.326 --> 00:33:57.036 align:middle for them yesterday because you never know. 00:33:58.536 --> 00:33:59.296 align:middle Any more questions? 00:34:00.056 --> 00:34:01.146 align:middle One more. I don't know. 00:34:01.146 --> 00:34:01.956 align:middle How many time do I have? 00:34:02.156 --> 00:34:05.426 align:middle I still have time. 00:34:06.026 --> 00:34:06.346 align:middle Thank you. 00:34:06.876 --> 00:34:12.286 align:middle Um, so this works for your application, of course, the immutable servers. 00:34:12.366 --> 00:34:13.946 align:middle But what do you do with databases? 00:34:14.086 --> 00:34:15.666 align:middle Do you have a replication cluster? 00:34:16.176 --> 00:34:17.016 align:middle The databases? 00:34:17.166 --> 00:34:17.906 align:middle It is a different thing. 00:34:18.676 --> 00:34:24.056 align:middle So, you can have a master/slave, you can have clusters, you can have them in multi-regions, 00:34:24.056 --> 00:34:26.676 align:middle then some sort of sync - that's up to you. 00:34:27.056 --> 00:34:29.806 align:middle But they need to be separated from your application. 00:34:29.806 --> 00:34:30.416 align:middle Yeah, of course. 00:34:30.446 --> 00:34:33.336 align:middle So that is something to consider separately from... 00:34:33.456 --> 00:34:34.316 align:middle alright, thank you. 00:34:38.606 --> 00:34:42.516 align:middle Anymore? I think no. 00:34:42.856 --> 00:34:52.096 align:middle So, thank you very much for listening to me.