Using Org Babel for Literate Programming

Most people that have heard about Org mode are aware that it is a tool meant for creating TODO lists, taking notes, and even doing documentation to some extent.

What is not immediately clear about Org mode is that it is a very powerful tool for doing literate programming and a comfortable environment for working with active documents.

Rather than explain the ideas one by one, I will give a more practical example of what it is possible to do with it.

Example: Creating a monitoring HTTP endpoint

Let’s create a simple HTTP endpoint with Sinatra which will respond with some information about the host where it is running, like its uptime and the number of established connections.

Requirements

What we want is basically the output of the following commands:

uptime

…which will show something like this:

20:37  up 4 days,  9:48, 5 users, load averages: 0.67 0.86 1.11

And:

netstat -an | grep ESTABLISHED | wc -l

which shows a number:

5

The result from both commands would be aggregated into the response sent by the the server when we request /vars:

curl 127.0.0.1:9292/vars

Result:

Uptime:

20:37  up 4 days,  9:48, 5 users, load averages: 0.67 0.86 1.11

ESTABLISHED connections: 14

Preparation

We will start by creating a couple of folders, one for the content we write in Org, and another one for the actual implementation.

mkdir -p org
mkdir -p src
touch readme.org

(The readme.org file is of course optional, but still a good practice to follow.)

In the end, our directory structure will look something like this:

.
├── org
│   ├── app.org
│   └── run.org
├── readme.org
├── src
│   ├── Gemfile
│   ├── app.rb
│   └── config.ru

Implementation

Most of the contents of our application, we will develop within an org/app.org file.

touch org/app.org

Prototyping phase

Usually, while developing a program we go into a REPL to test out what would eventually make it into the program.

With Org Babel though, it is possible to call the code blocks within an Emacs buffer, so we can actually start prototyping using Org mode. Each time we press C-c C-c on top of the code block, the results would be refreshed:

*** Notes on how to get the results

We could use something simple like:

#+begin_src ruby :results output code
s = <<VARS
Uptime:
#{`uptime`}

ESTABLISHED connections:
#{`netstat -an | grep ESTABLISHED | wc -l`}

VARS

puts s
#+end_src

#+RESULTS:
#+BEGIN_SRC ruby
Uptime:
22:23  up 4 days, 11:34, 5 users, load averages: 0.58 0.46 0.30


ESTABLISHED connections:
       8


#+END_SRC

Using :tangle to create the files of the program

We can have a code block be tangled into a file by using the :tangle switch argument. For example, we can define the Gemfile and config.ru files within an Org mode buffer like this:

Dependencies from the app, just sinatra and webrick for the server is ok for now.

#+BEGIN_SRC ruby :tangle src/Gemfile
gem 'sinatra'
#+END_SRC

Needed to start the Sinatra application:

#+BEGIN_SRC ruby :tangle src/config.ru
require './app.rb'
run Sinatra::Application
#+END_SRC

Full example

The Sinatra application would then look something like this. From Emacs, we can either press C-c C-v t or C-c C-v C-t, (M-x org-babel-tangle also does it) to tangle the web of code blocks to their respective files.

#+TITLE: Monitoring HTTP endpoint

  This is a simple Sinatra application that provides
  the following endpoints:

  ‐ =/=        :: which responds =OK= in case all is good with the server.
  ‐ =/vars=    :: to get info about the system

  ** Bootstrapping the app

  *** Gemfile

  Dependencies from the app, just sinatra and webrick for now is ok.

  #+BEGIN_SRC ruby :tangle src/Gemfile
  gem 'sinatra'
  #+END_SRC

  *** Config.ru

  Needed to start the sinatra application

  #+BEGIN_SRC ruby :tangle src/config.ru
  require './app.rb'
  run Sinatra::Application
  #+END_SRC

  ** The App

  *** Import dependencies

  #+BEGIN_SRC ruby :tangle src/app.rb

  require 'sinatra'

  #+END_SRC

  *** =/= endpoint

  Just respond with =OK=.

  #+BEGIN_SRC ruby :tangle src/app.rb
  get '/' do
    'OK'
  end
  #+END_SRC

  *** =/vars= endpoint

  Respond with info about the system:

  #+BEGIN_SRC ruby :tangle src/app.rb
  get '/vars' do

  r = <<VARS
  Uptime:
  #{`uptime`}

  ESTABLISHED connections:
  #{`netstat -an | grep ESTABLISHED | wc -l`}

  VARS

  r
  end
  #+END_SRC

Running it

Now that, we have defined the program, it would be also helpful to define how to actually run it. We will write this in a org/run.org file. Note how we are including the org/app.org application as a dependency so that in case of changes those code blocks become tangled as well.

#+TITLE:   Running the Application
#+include: "org/app.org"

** Run it

To run it, we will need to get the dependencies first,
and then start it with bundler:

#+name: server
#+BEGIN_SRC sh :dir src
bundle install
bundle exec rackup
#+END_SRC

Now let's send some requests to it:

#+name: curl
#+BEGIN_SRC sh :wait 1
while true; do 
  curl 127.0.0.1:9292/vars 2> /dev/null
  sleep 1
done
#+END_SRC

Here we are giving the code block a #+name:, doing this make it possible to reuse the code block later on. I made a small gem that makes it possible to run these kind of blocks when they have a name.

gem install org-converge
org-run org/run.org

The output would look like this:

org-run org/run.org
...
Running code blocks now! (2 runnable blocks found in total)
server    -- started with pid 71840
curl      -- started with pid 71841
server    -- Using rack 1.5.2
server    -- Using rack-protection 1.5.3
server    -- Using tilt 1.4.1
server    -- Using sinatra 1.4.5
server    -- Using bundler 1.7.1
server    -- Your bundle is complete!
server    -- Use `bundle show [gemname]` to see where a bundled gem is installed.
server    -- [2014-12-11 22:56:42] INFO  WEBrick 1.3.1
server    -- [2014-12-11 22:56:42] INFO  ruby 2.1.2 (2014-05-08) [x86_64-darwin11.0]
server    -- [2014-12-11 22:56:42] INFO  WEBrick::HTTPServer#start: pid=71848 port=9292
server    -- 127.0.0.1 - - [11/Dec/2014 22:56:42] "GET /vars HTTP/1.1" 200 110 0.0527
curl      -- Uptime:
curl      -- 22:56  up 4 days, 12:07, 5 users, load averages: 0.78 0.84 0.76
curl      -- 
curl      -- 
curl      -- ESTABLISHED connections:
curl      --       12
curl      -- 
curl      -- 
server    -- 127.0.0.1 - - [11/Dec/2014 22:56:43] "GET /vars HTTP/1.1" 200 110 0.0210
curl      -- Uptime:
curl      -- 22:56  up 4 days, 12:07, 5 users, load averages: 0.78 0.84 0.76
curl      -- 
curl      -- 
curl      -- ESTABLISHED connections:
curl      --       12
curl      -- 
curl      -- 

Final remarks

I think there is a lot of potential in the approach from Org mode for Literate Programming, so it is worth a try. Take a look at the concepts exposed by Knuth in his paper on the matter, and you will find out that the core ideas about LP have not shown its age.

These days, most of my development starts from an Org mode buffer and it just continues there. With Org mode, you can basically start with a Readme (c.f. Readme Driven Development), and then just contine doing the full implementation of your source there in the same place, along with your notes (but which are not exported).

At the same time, it makes up for very useful documentation for others to pick up a project later on. And also Org mode file rendering is supported by Github! In case of bugs, please send me a ticket here.