The current state of affairs

Over the past month or so I’ve been trying to make sense of the current state of package and module management for JavaScript applications. There are so many different tools, many of which provide overlapping functionality. I’ve organised my thoughts on where these tools/frameworks intersect and differ, and the direction I think we will be moving going forward. With the rate of change in this landcape I wouldn’t be surprised if the specifics change over the coming 6 months, but some of the higher level trends should be relatevely stable.

Package Management

There are currently two major package repositories for JavaScript: Bower, and NPM. Bower is predominantly used for front end packages such as Backbone.js or Bootstrap, with NPM originally being used more heavily for server side packages. Over time, front end modules that traditionally resided in Bower have started to make their way into NPM. This is particularly prevalent with JavaScript libraries.  A major reason for this is that NPM requires packages to be authored using the CommonJS module specification. Technically this involves annotating your JavaScript file with two things:

• A module identifier to uniquely identify the module.
• Export statements for each of the functions that are to be exported and made available to external modules.

The powerful thing about the CommonJS module specification is that it provides a way to easily manage dependencies between different groupings of JavaScript functionality, which leads into the next point.

Module Loading

Though the idea of module loading has gained a lot of traction over the last few years, the majority of websites today still use classic method of immediately executed functions. For example to define a calculator module in vanilla Javascript I could do the following:

var calculator = (function () {

  return {
    add: function(x, y){
      return x + y;
    },
    subtract: function(x,y){
      return x - y;
    }
  };

}());

This pattern still allows us to build quite rich and complex applications, but it has a downside. The difficulty with this module pattern is that it requires developers to manually manage dependencies via script includes. For example, if the calculator module depended on a math module to provide the implementation of more advanced calculation functions then this file would need to be included before the script file containing the calculator function. This can quickly become hard to manage as the number of different include scripts in a project increases.

Over the past few years a better solution for this problem has started to emerge in the shape of a number of specifications and JavaScript frameworks:

• CommonJS: The CommonJS module format was developed by the Common JS List Participants. NodeJS core modules are implemented using the CommonJS module format, and NodeJS framework developers are encouraged to also follow this format. NodeJS provides a module loader system that works with modules in this format. The RequireJS site has a nice overview of how this format originated.
• RequireJS: RequireJS is a JavaScript framework uses the AMD (Asynchronous Module Definition) specification to provide an easy way to define and consume JavaScript modules. It is primarily targeted at the Web Browser environment, and as such has some advantages over CommonJS modules in this space. It also has a standard method to include multiple modules in the same file, and setting a function as a value (useful for constructor functions) which are the CommonJS format does not provide for. The latter point is actually provided by NodeJS as a kind of extension to the CommonJS format.
• Browserify: Browserify is a framework that adds support for Node (CommonJS) modules to the browser. Not all NPM modules are supported by Browserify. For example, if the NPM module is performing server side IO. Browserify is generally plugged into the JavaScript build pipeline to generate bundles for groupings of related JavaScript code.
• SystemJS, and ES6 Module Loader Polyfill: SystemJS is a dynamic module loader built on top of the ES6 Module Loader Polyfill. It handles any module format including CommonJS and AMD in addition to ES6 Modules. With transpilation tools such as Traceur and Babel it is starting to become feasible to write JavaScript applications in ES6 or ES7. This means that using SystemJS we can write more future proof JavaScript modules in ES6 while still taking advantage of the  existing ecosystem of CommonJS and AMD modules. In my opinion, while Browserify is a great project, it was more of a stepping stone towards where we need to head, with SystemJS being the best option going forward.

The example from above in the ES6 Module format would look something like this:

//------ calculator.js ------
export function add(x) {
    return x + y;
}
export function subtract(x, y) {
    return x - y;
}

This is cleaner than the classical JavaScript module pattern, and much simpler to fit into a large web application project JavaScript dependency structure.

Enter JSPM

With SystemJS we can dynamically load modules of any format into our JavaScript applications. Wouldn’t it be nice if we had a way of pulling these modules (including any related dependencies) into our applications, regardless of which repository they are stored in?

jspm is a package manager for the SystemJS universal module loader, built on top of the dynamic ES6 module loader — http://jspm.io/

JSPM allows you to load modules from NPM or Github. Some minor changes need to be made to existing packages to get them to work through JSPM. However, if you are worried that the package that you want to use is not supported, don’t be. There is a method to wire up the necessary plumbing for JSPM outside of the framework package, so you don’t need to be the package author to set this up. For front end development, JSPM can be used in place of both Bower and NPM (though may just wrap NPM in a lot of instances).

So, today the best way forward in my opinion is to use JSPM in combination with SystemJS to split your JavaScript application into ES6 compatible modules, which can depend on modules imported from JSPM. You can find out more about getting started with JSPM on the Github page.

How does bundling fit into this?

In an ideal world we would only load the required JavaScript modules as they are needed by pages. However, with the current version of HTTP, most browsers only support between 8 to 17 concurrent connections. This means that if we are on Chrome which supports 10 concurrent connections, and we are loading 20 scripts, the second set of 10 scripts will be blocked until the connections are freed up, and that doesn’t even factor in the other includes such as images and CSS which are also loaded as a part of this same pipeline. In the future we should see the implementation of HTTP2, which gets around the browser connection limits using Multiplexing

Multiplexing addresses these problems by allowing multiple request and response messages to be in flight at the same time; it’s even possible to intermingle parts of one message with another on the wire.

This, in turn, allows a client to use just one connection per origin to load a page.

 

Once HTTP2 has a broad enough implementation we’ll be able to asynchronously load scripts as needed in production, but we are not there yet. The stop gap for now is to bundle all of these separate modules together into a single JavaScript file (or a small group of JavaScript files containing related modules). JSPM has a command which would generally be run as a part of a Gulp/Grunt build process to bundle the dependencies ready to be served in production.

 

Summary

SystemJS, and JSPM allow us to break down our application into a set of modules. These modules can depend on JavaScript modules authored in any common JavaScript module format. We can write modules in a future proof way, bundling the modules and dependencies until it becomes feasible to remove this build in the future.