Using Org Babel for Literate Programming
12 Dec 2014Most people that have heard about Org mode are aware that
it is a tool meant for creating TODO
lists, taking notes,
and even doing documentation to some extent.
What is not immediately clear about Org mode is that it is a very powerful tool for doing literate programming and a comfortable environment for working with active documents.
Rather than explain the ideas one by one, I will give a more practical example of what it is possible to do with it.
Example: Creating a monitoring HTTP endpoint
Let’s create a simple HTTP endpoint with Sinatra which
will respond with some information about the host where it is running,
like its uptime
and the number of established connections.
Requirements
What we want is basically the output of the following commands:
uptime
…which will show something like this:
20:37 up 4 days, 9:48, 5 users, load averages: 0.67 0.86 1.11
And:
netstat -an | grep ESTABLISHED | wc -l
which shows a number:
5
The result from both commands would be aggregated into
the response sent by the the server when we request /vars
:
curl 127.0.0.1:9292/vars
Result:
Uptime: 20:37 up 4 days, 9:48, 5 users, load averages: 0.67 0.86 1.11 ESTABLISHED connections: 14
Preparation
We will start by creating a couple of folders, one for the content we write in Org, and another one for the actual implementation.
mkdir -p org
mkdir -p src
touch readme.org
(The readme.org
file is of course optional, but still a good practice to follow.)
In the end, our directory structure will look something like this:
.
├── org
│ ├── app.org
│ └── run.org
├── readme.org
├── src
│ ├── Gemfile
│ ├── app.rb
│ └── config.ru
Implementation
Most of the contents of our application, we will develop within an
org/app.org
file.
touch org/app.org
Prototyping phase
Usually, while developing a program we go into a REPL
to test out what
would eventually make it into the program.
With Org Babel though, it is possible to call the code blocks within
an Emacs buffer, so we can actually start prototyping using Org mode.
Each time we press C-c C-c
on top of the code block, the results
would be refreshed:
*** Notes on how to get the results We could use something simple like: #+begin_src ruby :results output code s = <<VARS Uptime: #{`uptime`} ESTABLISHED connections: #{`netstat -an | grep ESTABLISHED | wc -l`} VARS puts s #+end_src #+RESULTS: #+BEGIN_SRC ruby Uptime: 22:23 up 4 days, 11:34, 5 users, load averages: 0.58 0.46 0.30 ESTABLISHED connections: 8 #+END_SRC
Using :tangle
to create the files of the program
We can have a code block be tangled into a file by using the
:tangle
switch argument. For example, we can define the Gemfile
and config.ru
files within an Org mode buffer like this:
Dependencies from the app, just sinatra and webrick for the server is ok for now.
#+BEGIN_SRC ruby :tangle src/Gemfile
gem 'sinatra'
#+END_SRC
Needed to start the Sinatra application:
#+BEGIN_SRC ruby :tangle src/config.ru
require './app.rb'
run Sinatra::Application
#+END_SRC
Full example
The Sinatra application would then look something like this.
From Emacs, we can either press C-c C-v t
or C-c C-v C-t
,
(M-x org-babel-tangle
also does it) to tangle the web of
code blocks to their respective files.
#+TITLE: Monitoring HTTP endpoint This is a simple Sinatra application that provides the following endpoints: ‐ =/= :: which responds =OK= in case all is good with the server. ‐ =/vars= :: to get info about the system ** Bootstrapping the app *** Gemfile Dependencies from the app, just sinatra and webrick for now is ok. #+BEGIN_SRC ruby :tangle src/Gemfile gem 'sinatra' #+END_SRC *** Config.ru Needed to start the sinatra application #+BEGIN_SRC ruby :tangle src/config.ru require './app.rb' run Sinatra::Application #+END_SRC ** The App *** Import dependencies #+BEGIN_SRC ruby :tangle src/app.rb require 'sinatra' #+END_SRC *** =/= endpoint Just respond with =OK=. #+BEGIN_SRC ruby :tangle src/app.rb get '/' do 'OK' end #+END_SRC *** =/vars= endpoint Respond with info about the system: #+BEGIN_SRC ruby :tangle src/app.rb get '/vars' do r = <<VARS Uptime: #{`uptime`} ESTABLISHED connections: #{`netstat -an | grep ESTABLISHED | wc -l`} VARS r end #+END_SRC
Running it
Now that, we have defined the program, it would be also helpful to
define how to actually run it. We will write this in a org/run.org
file.
Note how we are including the org/app.org
application as a
dependency so that in case of changes those code blocks become tangled
as well.
#+TITLE: Running the Application #+include: "org/app.org" ** Run it To run it, we will need to get the dependencies first, and then start it with bundler: #+name: server #+BEGIN_SRC sh :dir src bundle install bundle exec rackup #+END_SRC Now let's send some requests to it: #+name: curl #+BEGIN_SRC sh :wait 1 while true; do curl 127.0.0.1:9292/vars 2> /dev/null sleep 1 done #+END_SRC
Here we are giving the code block a #+name:
, doing this make it
possible to reuse the code block later on. I made a small gem that
makes it possible to run these kind of blocks when they have a name.
gem install org-converge org-run org/run.org
The output would look like this:
org-run org/run.org
...
Running code blocks now! (2 runnable blocks found in total)
server -- started with pid 71840
curl -- started with pid 71841
server -- Using rack 1.5.2
server -- Using rack-protection 1.5.3
server -- Using tilt 1.4.1
server -- Using sinatra 1.4.5
server -- Using bundler 1.7.1
server -- Your bundle is complete!
server -- Use `bundle show [gemname]` to see where a bundled gem is installed.
server -- [2014-12-11 22:56:42] INFO WEBrick 1.3.1
server -- [2014-12-11 22:56:42] INFO ruby 2.1.2 (2014-05-08) [x86_64-darwin11.0]
server -- [2014-12-11 22:56:42] INFO WEBrick::HTTPServer#start: pid=71848 port=9292
server -- 127.0.0.1 - - [11/Dec/2014 22:56:42] "GET /vars HTTP/1.1" 200 110 0.0527
curl -- Uptime:
curl -- 22:56 up 4 days, 12:07, 5 users, load averages: 0.78 0.84 0.76
curl --
curl --
curl -- ESTABLISHED connections:
curl -- 12
curl --
curl --
server -- 127.0.0.1 - - [11/Dec/2014 22:56:43] "GET /vars HTTP/1.1" 200 110 0.0210
curl -- Uptime:
curl -- 22:56 up 4 days, 12:07, 5 users, load averages: 0.78 0.84 0.76
curl --
curl --
curl -- ESTABLISHED connections:
curl -- 12
curl --
curl --
Final remarks
I think there is a lot of potential in the approach from Org mode for Literate Programming, so it is worth a try. Take a look at the concepts exposed by Knuth in his paper on the matter, and you will find out that the core ideas about LP have not shown its age.
These days, most of my development starts from an Org mode buffer and it just continues there. With Org mode, you can basically start with a Readme (c.f. Readme Driven Development), and then just contine doing the full implementation of your source there in the same place, along with your notes (but which are not exported).
At the same time, it makes up for very useful documentation for others to pick up a project later on. And also Org mode file rendering is supported by Github! In case of bugs, please send me a ticket here.