Using Webpack transparently with Django + hot reloading React components as a bonus

If you don’t already know webpack, you’ve some catching up to do.

Webpack is a module bundler that bundles javascript and other assets for the browser. It works really well for applications and javascript libraries and is very simple frontend build tool.

- Kent C. Dodds - https://egghead.io/lessons/javascript-intro-to-webpack

Objectives and reasoning

We’ll be setting up webpack and keeping it decoupled from django’s staticfiles system. Read my earlier post explaining why we we’ll be handling things this way and not integrating with staticfiles. We’ll be using webpack-bundle-tracker to extract information from webpack and django-webpack-loader to use the extracted information for django integration.

Setting up webpack

We’ll use npm to manage our frontend dependencies instead of managing them manually in one of the static files directories. You can optionally use bower as well in addition to npm.

First let’s setup npm in the root of your django project. This will generate a file called packages.json in your project root. It serves 2 purposes. Imagine requirements.txt and setup.py merged into one. That is packages.json for npm packages. If you use the --save or --save-dev flag when installing a package, it’ll save the packages as dependencies in the packages.json file. To reinstall the packages, all you need to is run npm install. Awesome, right? It gets better. The packages will be installed locally specific to your project under a directory called node_modules like virtualenv. To install a package globally, all you need to do is to use -g with npm install.

So, let’s generate a packages.json file in our project root using npm init

1
npm init

Npm dependencies

In addition to webpack, we’ll at least need the webpack-bundle-tracker plugin to extract useful information from webpack and store it in as json in a file. This file will act as the link between webpack and django.

Since we’ll be setting up webpack with an example reactjs app, we’ll also need babel. Babel is a great Javascript compiler that compiles ES6 into ES5 among other things. This lets use write next generation javascript today and still have run work in current browsers. Babel also supports react’s JSX language so we don’t need an additional compiler for that.

We’ll also need babel-loader to integrate babel with webpack. Webpack supports pluggable libraries called loaders that add support for different types of files and languages. Loaders can also be chained. For example, you can make a less file go through a less loader to compile it to css and then pass the output through a css loader. More on loaders here.

save vs save-dev

--save saves the packages you install as dependencies of your package. The packages that must be installed in order to run your package. --save-dev saves the packages as build dependencies, the packages that must be installed to hack on your package. Since we are not going to be publishing a real npm package, either one works. I like to use —save-dev as I only need the packages to build my bundles. Whatever the bundle depends on is included in the bundle it self.

Let’s install our first npm packages

1
npm install --save-dev react webpack webpack-bundle-tracker babel babel-loader

Create webpack config

1
2
3
mkdir -p assets/js
touch assets/webpack.config.js
touch assets/js/index.js

Let’s create a simple webpack config to load .jsx files using babel and use the webpack-bundle-tracker plugin to extract information to assets/webpack-stats.json. More on webpack configuration here.

webpack.config.js

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
var path = require("path");
var webpack = require('webpack');
var BundleTracker = require('webpack-bundle-tracker');

module.exports = {
context: __dirname,

entry: './js/index', // entry point of our app. assets/js/index.js should require other js modules and dependencies it needs

output: {
path: path.resolve('./assets/bundles/'),
filename: "[name]-[hash].js",
},

plugins: [
new BundleTracker({filename: './assets/webpack-stats.json'}),
],

module: {
loaders: [
{ test: /\.jsx?$/, exclude: /node_modules/, loader: 'babel-loader'}, // to transform JSX into JS
],
},

resolve: {
modulesDirectories: ['node_modules', 'bower_components'],
extensions: ['', '.js', '.jsx']
},
}

At this point, our directory structure will look something like this.

1
2
3
4
5
6
root/
├── manage.py
├── package.json
├── node_modules/
├── assets/ #added to STATICFILES_DIRS
│ └── webpack.config.js

Compiling our first bundle

Binaries shipped with node packages are installed to node_modules/.bin/ and it not added to $PATH automatically so we need to use fullpaths to the binaries. Installing binaries globablly like npm install -g webpack will add them to one of the binary location on in $PATH.

Let’s go ahead and compile our first bundle

1
./node_modules/.bin/webpack --config assets/webpack.config.js

This should create bundle at assets/bundles/main-[hash].js. This is good but we don’t want to create bundles manually every time we make changes to our code.

Webpack in watch mode

1
./node_modules/.bin/webpack --config assets/webpack.config.js --watch

This will leave the compiler running and compile bundles automatically when you change any of your source files. You’ll need to restart it if you make any changes to the webpack configuration though.


Example react app

Skip this part if you already have a react app running.

Let’s write a simple “hello, world” react app and use webpack to compile it. We refer to ./js/index as the entry point of our app in webpack.config.js which will look for index, index.js or index.jsx because we’ve added these three extensions to our webpack config under the key resolve.

assets/js/index.jsx

1
2
3
4
var React = require('react');
var App = require('./app');

React.render(<App/>, document.getElementById('react-app'));



assets/js/app.jsx

1
2
3
4
5
6
7
var React = require('react');

module.exports = React.createClass({
render: function(){
return <h1>Hello, world.</h1>
}
});

If you left webpack running in watch mode, it should automatically pick up the changes and compile a new bundle.


Django integration

Now that we’ve handed off the build process webpack, only thing we need on the django side is to know which bundle to include in our html pages. This is where django-webpack-loader comes in. It’ll also raise exceptions when webpack fails to build a bundle and django will show some useful information when in debug mode. To make sure django loads latest bundles during development, webpack loader will block any requests whenever it find webpack compiling code if DEBUG=True.

Requirements

1
pip install django-webpack-loader

Configuration

settings.py

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
import sys
import os

BASE_DIR = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))

STATICFILES_DIRS = (
os.path.join(BASE_DIR, 'assets'), # We do this so that django's collectstatic copies or our bundles to the STATIC_ROOT or syncs them to whatever storage we use.
)

WEBPACK_LOADER = {
'BUNDLE_DIR_NAME': 'bundles/',
'STATS_FILE': os.path.join(BASE_DIR, 'assets/webpack-stats.json'),
}

INSTALLED_APPS = (
...
'webpack_loader',
)

Usage

In templates

1
2
3
4
5
6
7
8
9
10
11
12
13
{% load render_bundle from webpack_loader %}
<!DOCTYPE html>
<html>
<head>
<meta charset="UTF-8">
<title>Example</title>
</head>

<body>
<div id="react-app"></div>
{% render_bundle 'main' %}
</body>
</html>

render_bundle will render the required <script> and <link> tags in the template.

Everything we need is in place now. Bundles will be automatically generated (provided you start webpack with —watch) and django will automatically pick up latest bundles from assets/bundles directory. In development, django will also block any requests while the bundles are being compiled. It’ll also raise raise if webpack fails to build at which point we can turn to our webpack process for more details.

Now that we’ve webpack working with django, let’s make things a little more fun by setting up hot reloading for our react components.


Bonus: Live editing react components

Since we are using pure webpack without any abstraction, we are free to use it however we want without the need to integrate any special with django. Whenever something new comes up for webpack, we can immediately use it without worrying if staticfiles, pipeline or compressor will support it or not. Decoupling FTW!

We’ll use a library called react-hot-loader by Dan Abramov. We’ll also need webpack-dev-server to build and serve our bundles if we want to hot reload any modules.

Requirements

1
npm install --save-dev webpack-dev-server react-hot-loader

Let’s modify assets/webpack.config.js to use webpack-dev-server and react-hot-loader

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
var path = require("path");
var webpack = require('webpack');
var BundleTracker = require('webpack-bundle-tracker');


module.exports = {
context: __dirname,
entry: [
'webpack-dev-server/client?http://localhost:3000',
'webpack/hot/only-dev-server',
'./js/index'
],

output: {
path: path.resolve('./assets/bundles/'),
filename: "[name]-[hash].js",
publicPath: 'http://localhost:3000/assets/bundles/', // Tell django to use this URL to load packages and not use STATIC_URL + bundle_name
},

plugins: [
new webpack.HotModuleReplacementPlugin(),
new webpack.NoErrorsPlugin(), // don't reload if there is an error
new BundleTracker({filename: './assets/webpack-stats.json'}),
],

module: {
loaders: [
// we pass the output from babel loader to react-hot loader
{ test: /\.jsx?$/, exclude: /node_modules/, loaders: ['react-hot', 'babel'], },
],
},

resolve: {
modulesDirectories: ['node_modules', 'bower_components'],
extensions: ['', '.js', '.jsx']
},
}

Instead of running webpack --watch, we’ll run webpack-dev-server to both compile and serve our bundles. The server will run on port 3000. publicPath in our webpack config refers to this server. Note that the server will by default keep the bunles in memory and not write to disk, so don’t be surprised if you don’t see anything new in assets/bundles/. Let’s use webpack-dev-server’s API to create a new instance of the server and pass webpack initialized with out config file to it. We’ll store this as server.js in our project root and use node run the server.

server.js

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
var webpack = require('webpack');
var WebpackDevServer = require('webpack-dev-server');
var config = require('./assets/webpack.config');

new WebpackDevServer(webpack(config), {
publicPath: config.output.publicPath,
hot: true,
inline: true,
historyApiFallback: true
}).listen(3000, '0.0.0.0', function (err, result) {
if (err) {
console.log(err);
}

console.log('Listening at 0.0.0.0:3000');
});

Taken from https://github.com/gaearon/react-hot-boilerplate/

Now instead of running webpack in watch mode, we run webpack-dev-server like this

1
node server.js

Done! Any changes made to the react components will reflect in the browser. No reload needed. Magic! right?

If you are interested in hot reloading react components, you should read this https://medium.com/@dan_abramov/the-death-of-react-hot-loader-765fa791d7c4


Production environments

Production bundles are different than local ones for various reason. I like to have slightly different webpack config for production, generate them locally and commit the bundle(s) and stats file to the codebase. As we store our bundles in assets and django is configured to look for static files in the assets directory, we don’t need to do anything special to with the bundles once they are generated. If you run collectstatic on your production servers after fetching new code, new bundles will be
there. If you configure static storage to use S3, then the bundles will be synched automatically to S3 ready for use in production.

Imporant: Make sure production config doesn’t use react-hot-loader or webpack-dev-server. Also make sure you use something like Uglify to compress your code and strip off any code only meant to be used in development.
Note: You should add our local webpack stats file and local bundles to .gitignore as they serve no purpose outside your local environment.

assets/webpack.prod.config.js

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
var config = require('./webpack.config.js');

config.output.path = require('path').resolve('./assets/dist');
config.output.pathName = '/production/path/to/bundle/directory'; // This will override the url generated by django's staticfiles

config.plugins = [
new BundleTracker({filename: './assets/webpack-stats-prod.json'}),

// removes a lot of debugging code in React
new webpack.DefinePlugin({
'process.env': {
'NODE_ENV': JSON.stringify('production')
}}),

// keeps hashes consistent between compilations
new webpack.optimize.OccurenceOrderPlugin(),

// minifies your code
new webpack.optimize.UglifyJsPlugin({
compressor: {
warnings: false
}
})
];
}



settings.py

1
2
3
4
5
if not DEBUG:
WEBPACK_LOADER.update({
'BUNDLE_DIR_NAME': 'dist/',
'STATS_FILE': os.path.join(BASE_DIR, 'assets/webpack-stats-prod.json'
})

Generate production bundles by invoking webpack one time with prod config

1
./node_modules/.bin/webpack --config assets/webpack.prod.config.js

This will create production bundles in ./assets/dist/ and ./assets/webpack-stats-prod.json

Let's modernize the way we handle frontend code with Django

The problem

Django is great but it’s frontend toolchain is stuck in the past. Imagine if someone told you to copy all your python module dependencies in your source tree and import them from there. Unthinkable, right? We’ve pip and virtualenv for that. We also have npm and bower for frontend packages but we still choose to manage frontend packages manually or write very complex wrappers over javascript tools so that we only have to deal with Python. I think this needs to change. The javascript community has come up with some awesome pieces of software. Npm is one of the best, probably the best package manager I’ve come across. Tools like grunt, gulp,
browserify, webpack are too good to ignore.

Problems with the currect approach

  • Manually maintaining dependencies is a pain.
  • No integration with managers like npm and bower.
  • Horrible for frontend engineers and designers to work with.
  • Backend and frontend systems are tightly coupled and sometimes limit each other.

What about django-npm, pipeline and compressor?

Apps like django-pipeline and django-compressor have done a great job with static assets. I’ve been a great fan of django-pipeline actually but I hate how they take away all the transparency in order to make things a little bit magical. They are limited in what they can do by the staticfiles system. They are opaque and make it harder to debug problems. They also need companion apps for to integrate any new javascript tool like gulp or browserify. Even after having all these wrappers, the experience can be quite rough at times. Documentation and resources available for javascript tools are not directly applicable sometimes when using these wrappers. You’ve an additional layer of software and configuration to worry about or wonder how your python configuration translates to the javascript. Things can be much much better.

Problems with wrappers

  • The are opaque, hard to debug and slow.
  • Limited by django’s staticfiles system.
  • Docs, stackoverflow, blog posts, written for the javascript tools don’t directly apply to the django wrapper apps.
  • Still limited by staticfiles.
  • Still very tightly coupled with django.

So should we abandon staticfiles?

No, but we should limit it to just collecting pre-bundled static assets to the target directory or static file servers. We should not hook post-processors into it. The build process for frontend assets should be completely decoupled from staticfiles.

How do we integrate the frontend tools with django then?

For simpler cases, we don’t even need to integrate. We can use things like gulp or grunt to compile assets and then use collectstatic to sync the builds, but we need some sort of integration to make things a bit smoother. During development, it makes sense to return 500 error code when a build fails so one knows immediately where to look for the problem. It also makes sense to block a request while a build is being compiled to make sure you don’t test older code in the browser. For production use, we can configure our frontend tools to use hashed names in the builds; It would be nice to have an easy way to get reference to hashed bundles in django. In my opinion, integration should stop here. We should not spawn processes from python, translate config in settings.py to native JS config.

I suggest we use bridges, not wrappers. Instead of writing wrappers around something like webpack and spawning webpack processes form within django, we should run webpack independently and pipe the output to django. If we can come up with a standard for this, we would only have to write a single bridge application for django. Then instead of writing django apps that wrap the javascript tools, we write plugins for the tools that emit useful data to be consumed by django or any other framework.

webpack-bundle-tracker + django-webpack-loader

django-webpack-loader and webpack-bundle-tracker implement a system like this. webpack-bundle-tracker plugin emits neccessary imformation about webpack’s compilation process and results so django-webpack-loader can consume it. django-webpack-loader does not care how you use webpack. You could control it from gulp, use the dev server, use the —watch mode or manually run it after every change. Head over to https://github.com/owais/django-webpack-loader/ to see how it all works or read this guide for setting it all up with optional live reload for react components.

What this solves

  • Use proper package managers like npm and bower instead of manually managing source files.
  • Use webpack transparently without any limitations and leverage all the documentation and resources.
  • Handle your frontend assets however you want. Run webpack in watch mode, use grunt, gulp, webpacks’ dev server or anything you desire. No limitations.
  • Make your frontend engineers and designers happy! :)
  • Completely decouple frontend build process from django. You could build and serve your static assets from a completely different system as long you give django access to the stats file generated by webpack-bundle-tracker.

Limitations of this approach (for now)

  • Harder to store static files in app directories (totally worth it).
  • Doesn’t integrate with static files provided by 3rd party apps yet.
  • Need to setup your frontend toolchain manually but that is very easy most of the time.

Remember The Rhythm

I recently switched to Rhythmbox from Banshee and I’m loving it. It’s fast, loads up almost instantly, doesn’t consume as much resources as Bansee and doesn’t crash after every 3 songs. I do miss some features that Banshee had though like remembering the last playing song. So, I created a plugin for rhythmbox just to do that. It remembers the last playing entry (song, radio station, podcast, etc), playback time, browser filters (genre, artist and album) and playlists. It remembers these things (except playback time) even if Rhythmbox crashes for some reason.

Install on Ubuntu

1
2
3
sudo add-apt-repository ppa:loneowais/ppa
sudo apt-get update
sudo apt-get install remember-the-rhythm

For other distros, install from the tarball

1
sudo make install

Banshee 2 in Ubuntu

Just noticed a small but nice change in Banshee 2. Usually, our music players minimize to system tray when closing the main window so that the music keeps playing in the background. We have to manually look for a quit function in order to entirely close the app or pause the music and let the app live in memory and consume cpu cycles. Banshee 2.0 has no quit function per se. Closing banshee while the music is playing hides the window, otherwise quits entirely.

We need to promote this bahavior across the freedesktop scene. Abstract underlying concepts as much as we can and make apps inteligent enough to decide for themselves.

Related post from the Canonical Design Team

An Update on GmailWatcher

In case you don’t already know, GmailWatcher is a Gmail notifier specifically designed for the Ubuntu Operating System. It relies heavily on Ubuntu specific packages like MeMenu, Notifications, DesktopCouch but I’m planning a stock Gnome version also so that people on Fedora, Suse and other rocking distribution can use it too.

Features

  • Multiple Accounts
  • Google Apps support
  • Secure password storage using Gnome-Keyring
  • Themable unread emails page
  • Preferences sync using U1

Right now, my target is to make it stable/usable enough in time for Maverick. I’m planning to release the first stable version and getting it into Universe for Maverick. In fact, I’ve already submitted it to REVU (MOTU Advocate, Anyone?).

Request

I intent to spend the next few days polishing the app. This post is essentially a call for user feedback. I would like to know what you don’t like about the way GmailWatcher works. Bugs, usability issues, appearance, anything that doesn’t count as a new feature.

Themes

You can also contribute some themes for the email summaries view. A GmailWatcher theme is a simple HTML page styled with CSS. It’s pretty basic right now. Once Maverick is released, I’ll add JavaScript and local media support. Take a look at lp:gmailwatcher-themes-extra in case you would like to add a theme or two.

Install

1
2
3
sudo add-apt-repository ppa:loneowais/ppa
sudo apt-get update
sudo apt-get install gmailwatcher

From TC 90 to Python

This Saturday I’ll be delivering a workshop on the Python programming language in my college. Our university has an age old course for Computer Engineering. C is the introductory language and is taught using the Turbo C compiler from the 90s, VB 6 is considered the best way to make GIUs. It’s time to introduce my friends to 2010 in Computer Science.

If this goes well, Django or PyGTK will be next.

Update

The workshop was successful. A lot of students showed up and really enjoyed python. \m/

Python Workshop

Announcing GmailWatcher

In case you don’t already know, GmailWatcher is a Gmail notifier specifically designed for the Ubuntu Operating System. It relies heavily on Ubuntu specific packages like MeMenu, Notifications, DesktopCouch but I’m planning a stock Gnome version also so that people on Fedora, Suse and other rocking distribution can use it too.

Features

  • Multiple Accounts
  • Google Apps support
  • Secure password storage using Gnome-Keyring
  • Themable unread emails page
  • Preferences sync using U1

Right now, my target is to make it stable/usable enough in time for Maverick. I’m planning to release the first stable version and getting it into Universe for Maverick. In fact, I’ve already submitted it to REVU (MOTU Advocate, Anyone?).

Request

I intent to spend the next few days polishing the app. This post is essentially a call for user feedback. I would like to know what you don’t like about the way GmailWatcher works. Bugs, usability issues, appearance, anything that doesn’t count as a new feature.

Themes

You can also contribute some themes for the email summaries view. A GmailWatcher theme is a simple HTML page styled with CSS. It’s pretty basic right now. Once Maverick is released, I’ll add JavaScript and local media support. Take a look at lp:gmailwatcher-themes-extra in case you would like to add a theme or two.

Install

1
2
3
sudo add-apt-repository ppa:loneowais/ppa
sudo apt-get update
sudo apt-get install gmailwatcher

Kargil


Kargil

Rest of the Set

FOSS Kashmir

Finally Mir Nazim has kicked off the Kashmir FOSS mailing list. Things should only soar from here on. Really excited. Check http://fosskashmir.appspot.com.