SymfonyCasts stands united with the people of Ukraine
If you liked what you've learned so far, dive in!
Subscribe to get access to this tutorial plus
video, code and script downloads.
Run Gulp! Spoiler alert: Gulp is lying to you. It looks like everything runs in order: clean starts, clean finishes, then styles starts. But that's wrong. The truth is that everything is happening all at once, asynchronously. And to be fair, Gulp isn't really lying - it actually has no idea when each task actually finishes. Well, at least not yet.
Let's find out what's really going on.
Each line in a Gulp stream is asynchronous - like an AJAX call. This means
that before gulp.src() finishes, the next pipe() is already being called.
In fact, the whole function might finish before gulp.src() is done.
But we really need each line to run in order. So when you call pipe(),
it doesn't run what's inside immediately: it schedules it to be called once
the previous line finishes. The effect is like making an AJAX call, adding
a success listener, then making another AJAX call from inside it.
I wonder then, does the main.css file finish compiling before dinosaur.css
starts? Does the scripts wait for the styles task to finish? Let's find out.
Like with AJAX, each line returns something that acts like a Promise.
That means, for any line, we can write on to add a listener for when this
specific line actually finishes. When that happens, let's
console.log('start '+filename).
Copy this and add another listener to the last line. Change the text to "end":
| ... lines 1 - 14 | |
| app.addStyle = function(paths, outputFilename) { | |
| gulp.src(paths).on('end', function() { console.log('start '+outputFilename)}) | |
| ... lines 17 - 28 | |
| .pipe(gulp.dest('.')).on('end', function() { console.log('end '+outputFilename)}) | |
| }; | |
| ... lines 31 - 93 |
Ok, run gulp!
gulp
Woh! When it said it finished "styles", it really means it was done executing
the styles task. But things really finish way later. In fact, they don't
even start the process until later. And what's really crazy is that dinosaur.css
starts before main.css, even though main is the first style we add.
So, you can't depend on anything happening in order. But, what if you need to?
There's a bug with our manifest file - a race condition! Ah gross! Because
of the merge option, it opens up the manifest, reads the existing keys,
updates one of them, then re-dumps the whole file.
For styles, the manifest file is opened twice for main.css and dinosaur.css.
If one opens the file before the other finishes writing, when it writes,
it'll run over the changes from the first.
How can we make the first addStyle finish before starting the second?
It turns out the answer is easy. We can attach an end listener to any part
of the Gulp stream. Return the stream from addStyle. Then in styles,
attach an on('end') and only process dinosaur.css once the previous
call is finished:
| ... lines 1 - 14 | |
| app.addStyle = function(paths, outputFilename) { | |
| return gulp.src(paths).on('end', function() { console.log('start '+outputFilename)}) | |
| ... lines 17 - 29 | |
| }; | |
| ... lines 31 - 52 | |
| gulp.task('styles', function() { | |
| app.addStyle([ | |
| config.bowerDir+'/bootstrap/dist/css/bootstrap.css', | |
| config.bowerDir+'/font-awesome/css/font-awesome.css', | |
| config.assetsDir+'/sass/layout.scss', | |
| config.assetsDir+'/sass/styles.scss' | |
| ], 'main.css').on('end', function() { | |
| app.addStyle([ | |
| config.assetsDir+'/sass/dinosaur.scss' | |
| ], 'dinosaur.css'); | |
| }); | |
| }); | |
| ... lines 65 - 93 |
I know, it's ugly - we'll fix it, I promise! But let's see if it works:
gulp
Perfect! main.css starts and ends. Then dinosaur.css starts.
This is the key idea. But the syntax here is terrible. If we have 10 CSS files, we'll need 10 levels of nested listeners. That's not good enough.
To help fix this, I'll paste in some code I wrote:
| ... lines 1 - 53 | |
| var Pipeline = function() { | |
| this.entries = []; | |
| }; | |
| Pipeline.prototype.add = function() { | |
| this.entries.push(arguments); | |
| }; | |
| Pipeline.prototype.run = function(callable) { | |
| var deferred = Q.defer(); | |
| var i = 0; | |
| var entries = this.entries; | |
| var runNextEntry = function() { | |
| // see if we're all done looping | |
| if (typeof entries[i] === 'undefined') { | |
| deferred.resolve(); | |
| return; | |
| } | |
| // pass app as this, though we should avoid using "this" | |
| // in those functions anyways | |
| callable.apply(app, entries[i]).on('end', function() { | |
| i++; | |
| runNextEntry(); | |
| }); | |
| }; | |
| runNextEntry(); | |
| return deferred.promise; | |
| }; | |
| ... lines 84 - 129 |
It's an object called Pipeline - and it'll help us execute Gulp streams one
at a time. It has a dependency on an object called q, so let's go install
that:
npm install q --save-dev
On top, add var Q = require('q')
| ... lines 1 - 3 | |
| var Q = require('q'); | |
| ... lines 5 - 129 |
To use it, create a pipeline variable and set it to new Pipeline(). Now,
instead of calling app.addStyle() directly, call pipeline.add() with
the same arguments. Now we can move dinosaur.css out of the nested callback
and use pipeline.add() again. Woops, typo on pipeline!
| ... lines 1 - 84 | |
| gulp.task('styles', function() { | |
| var pipeline = new Pipeline(); | |
| pipeline.add([ | |
| config.bowerDir+'/bootstrap/dist/css/bootstrap.css', | |
| config.bowerDir+'/font-awesome/css/font-awesome.css', | |
| config.assetsDir+'/sass/layout.scss', | |
| config.assetsDir+'/sass/styles.scss' | |
| ], 'main.css'); | |
| pipeline.add([ | |
| config.assetsDir+'/sass/dinosaur.scss' | |
| ], 'dinosaur.css'); | |
| ... lines 98 - 99 | |
| }); | |
| ... lines 101 - 129 |
pipeline.add is basically queuing those to be run. So at the end, call
pipeline.run() and pass it the actual function it should call:
| ... lines 1 - 84 | |
| gulp.task('styles', function() { | |
| ... lines 86 - 94 | |
| pipeline.add([ | |
| config.assetsDir+'/sass/dinosaur.scss' | |
| ], 'dinosaur.css'); | |
| pipeline.run(app.addStyle); | |
| }); | |
| ... lines 101 - 129 |
Behind the scenes, the Pipeline is doing what we did before: calling addStyle,
waiting until it finishes, then calling addStyle again.
Try it!
gulp
Cool - we've got the same ordering.
Ok! Let's add this pipeline stuff to scripts. First, clean up my ugly debug
code. Make sure you actually return from addScript - we need that stream
so the Pipeline can add an end listener.
| ... lines 1 - 32 | |
| app.addScript = function(paths, outputFilename) { | |
| return gulp.src(paths) | |
| ... lines 35 - 133 |
Down in scripts work your magic! Create the pipeline variable, then
pipeline.add(). And, pipeline.run() to finish:
| ... lines 1 - 101 | |
| gulp.task('scripts', function() { | |
| var pipeline = new Pipeline(); | |
| pipeline.add([ | |
| config.bowerDir+'/jquery/dist/jquery.js', | |
| config.assetsDir+'/js/main.js' | |
| ], 'site.js'); | |
| pipeline.run(app.addScript); | |
| }); | |
| ... lines 112 - 133 |
Ok, try it!
gulp
Good, no errors! Use the Pipeline if you like it. But either way, remember that Gulp runs everything all at once. You can make one entire task wait for another to finish, but we'll talk about that later.
Hi there,
Thanks for the tutorial, it's really good !
I've got an issue here. Your Pipeline method is working great for me but :
The problem is that when I make a syntax error in the SCSS file, the plumber catch the error and gulp is still running, but nothing happen when I fix the syntax, no more CSS files are generated, it's like the watch styles is down.
Did I make something wrong ? Did you test the plumber with the Pipeline ?
Hi Kevin!
Ah, I think you're right! There is a bug here :). Fortunately, it looks like a well-known bug with a well-known workaround. Try updating both of your plugin.plumber calls (for CSS and JS) to look like this
.pipe(plugins.plumber(function(error) {
console.log(error.toString());
this.emit('end');
}))
Specifically, instead of just calling plumber(), you need to provide a callback. This callback should print the error (which plumber did automatically before adding this function) and then call this.emit('end'). That is the key: this tells the watch task that plumber is finished... which for some reason is the key to the whole thing.
Let me know if it works! I'm going to make an update to the tutorial.
Thanks for the question!
Hey again there !
I found a bug with the manifest file. It seems to work with 2 tasks (styles & scripts).
Then, I wanted to do 4 tasks (styles, styles_libraries, scripts and scripts libraries), in order to avoid generate the libraries files when I watch the custom styles, because it takes so long to generate the file when using some libraries. (~3s, couldn't wait so long to reload the browser)
After I had the 4 tasks, the manifest has gone crazy, it was rewriting when another task was still running even with the Pipeline.
I wonder if the problem comes from the fact that we initialize multiple Pipeline object and they can't communicate between them...? I don't have enough knowledge about Promise to be sure. Any ideas ?
Some threads say the manifest need a base path to have a good merge :
var config = {
baseRevManifestPath: 'app/Resources/assets',
revManifestPath: 'app/Resources/assets/rev-manifest.json'
};
.pipe(plugins.rev.manifest(config.revManifestPath, {
base: config.baseRevManifestPath,
merge: true
}))
.pipe(gulp.dest(config.baseRevManifestPath));
However, this doesn't fix the issue, I don't know if this is necessary...
To fix the issue, I had to finally use the deps array of gulp tasks and make styles_libraries dependant from styles, scripts dependant from styles_libraries and scripts_libraries dependant of scripts.
I don't know if it's a good solution, but it seems to work so far.
What do you think about this ?
Hey again there !
I found a bug with the manifest file. It seems to work with 2 tasks (styles & scripts) in the tutorial.
Then, I wanted to do 4 tasks (styles, styles_libraries, scripts and scripts libraries), in order to avoid generate the libraries files when I watch the custom styles, because it takes so long to generate the file when using some libraries. (~3s, too long for some CSS modifications)
After I had the 4 tasks, the manifest has gone crazy, it was rewriting when another task was still running even with the Pipeline.
I wonder if the problem comes from the fact that we initialize multiple Pipeline object and they can't communicate between them...? I don't have enough knowledge about Promise to be sure. Any ideas ?
Some threads say the manifest need a base path to have a good merge :
var config = {
baseRevManifestPath: 'app/Resources/assets',
revManifestPath: 'app/Resources/assets/rev-manifest.json'
};
.pipe(plugins.rev.manifest(config.revManifestPath, {
base: config.baseRevManifestPath,
merge: true
}))
.pipe(gulp.dest(config.baseRevManifestPath));
However, this doesn't fix the issue, I don't know if this is necessary...
To fix the issue, I had to finally use the deps array of gulp tasks and make styles_libraries dependant from styles, scripts dependant from styles_libraries and scripts_libraries dependant of scripts.
I don't know if it's a good solution, but it seems to work so far.
What do you think about this ?
Hey Ryan,
Really helpful tutorial. We've already started to take what we've learned here and started to apply it to a project. After watching the last video, it would seem like the Pipeline class would already exist somewhere as a library in the NPM database. Do you know of a library? Do you have plans to release it as a library? Or should someone like me create it?
Hey Startchurch!
Ah, really glad this was useful! So, I don't know of any such library, but I got the *exact* same impression as you: there was a logical problem... so I was expecting a library to already exist. To this day, the fact that there wasn't/isn't a library, leaves me thinking that there *might* be some other direction to take in solving this. But so far, I haven't seen anything. I don't have plans to release it as a library since (A) it's pretty small and (B) I still think that *eventually* a different method or library will be uncovered. But until then, it's working great for me and others.
Cheers!
I'm with you, it seems like an obvious problem, so it would seem that something would exist already.
What about a collection in memory of resources? When a resource get's updated in memory, the whole collection gets written? Maybe that idea wouldn't play well with gulp-rev.
// composer.json
{
"require": {
"php": ">=5.3.3",
"symfony/symfony": "2.6.*", // v2.6.4
"doctrine/orm": "~2.2,>=2.2.3", // v2.4.6
"doctrine/doctrine-bundle": "~1.2", // v1.2.0
"twig/extensions": "~1.0", // v1.2.0
"symfony/assetic-bundle": "~2.3", // v2.5.0
"symfony/swiftmailer-bundle": "~2.3", // v2.3.7
"symfony/monolog-bundle": "~2.4", // v2.6.1
"sensio/distribution-bundle": "~3.0", // v3.0.9
"sensio/framework-extra-bundle": "~3.0", // v3.0.3
"incenteev/composer-parameter-handler": "~2.0", // v2.1.0
"hautelook/alice-bundle": "~0.2" // 0.2
},
"require-dev": {
"sensio/generator-bundle": "~2.3" // v2.4.0
}
}
Hi there,
Thanks for the tutorial, it's really good !
I've got an issue here. Your Pipeline method is working great for me but :
The problem is that when I make a syntax error in the SCSS file, the plumber catch the error and gulp is still running, but nothing happen when I fix the syntax, no more CSS files are generated, it's like the watch styles is down.
Did I make something wrong ? Did you test the plumber with the Pipeline ?