ops●tokyo

PaaS & Web Operations

@wallyqs

© 2015. All rights reserved.

At Hacker School in NY

06 Oct 2014

I managed to get into the Fall batch from Hacker School. Lots of things to focus on!

Some links:

About Org Converge

04 Jun 2014

In a past release from Org Ruby, I added the functionality to be able to turn off exporting the results attached to a code block. For example, here only the actual code would make it to the export:

#+begin_src sh :results code
echo 'hello'
#+end_src

#+RESULTS:
#+BEGIN_SRC sh
hello
#+END_SRC

Doing this involved adding support to parse the header arguments from a code block, so I decided to start playing with the idea of implementing tangling code blocks:

#+begin_src sh :tangle /etc/component.yml
port: 4000
#+end_src

This is just a subset of the functionality provided by the powerful Org Babel mode, and still very basic since it does not support #+MACROS for example, which would make feature more convenient.

After doing this, and since this is all Ruby, I imagined that since there is good interop with other reliable Ruby tools, it would be interesting to reopen foreman for actually running the blocks.

All this functionality is wrapped into a gem named org-converge, named that way in the sense that all reproducible runs should converge into a state.

In the future, I hope to have more Chef/Puppet/Ansible like features, like idempotency:

Idempotency example via :if and :unless switches

The way this work is by linking 2 different code blocks, so that one can define whether the second one should be executed.

In order to achieve this, for now the blocks need to be defined sequentially and use the exit status to check whether it should be run or not.

#+name: install-package
#+begin_src sh
echo "brew install something..."
echo "package is installed" > installed
#+end_src

#+name: package-already-installed
#+begin_src sh
if [[ -e installed ]]; then
  echo "Already installed"
  exit 0
else
  echo "Needs install"
  exit 1
fi
#+end_src

#+name: install-package-once-again
#+begin_src sh :unless package-already-installed
echo "Installing the package once again..."
echo "not run" >> installed
#+end_src

#+name: do-everything-even-if-package-installed
#+begin_src sh :if package-already-installed
echo "Doing this even if installed already..." 
echo "eof" >> installed
#+end_src

Now when we run it the first time, we would get the following output:

gem install org-converge

org-converge idempotent-run.org

[2014-11-28T13:18:13 -0500] package-already-installed -- Needs install
[2014-11-28T13:18:13 -0500] package-already-installed -- started with pid 47963
[2014-11-28T13:18:13 -0500] package-already-installed -- exited with code 1
[2014-11-28T13:18:13 -0500] install-package -- started with pid 47967
[2014-11-28T13:18:13 -0500] install-package -- Installing the package...
[2014-11-28T13:18:13 -0500] install-package -- exited with code 0

…but if we run it again, we would get the following:

org-converge idempotent-run.org

[2014-11-28T13:19:07 -0500] package-already-installed -- started with pid 48474
[2014-11-28T13:19:07 -0500] package-already-installed -- Already installed
[2014-11-28T13:19:07 -0500] package-already-installed -- exited with code 0
[2014-11-28T13:19:07 -0500] install-package -- Skipped since :unless clause matches check from 'package-already-installed'
[2014-11-28T13:19:07 -0500] Run has completed successfully.

Multiprocess run usage

I also have been using lately for doing multi process running, and I have found it pretty comfortable for when practicing literate programming with Org Babel.

An example, the following 3 scripts should be run in parallel with all output being flushed to the screen.

- Count some numbers with bash

#+name: bash_counter
#+begin_src sh :shebang #!/bin/bash

for i in `seq 1 5`; do 
  echo "Writing! $i"
  echo "hello $i from bash"
  sleep $(($RANDOM % 5))
done

#+end_src

- Count some numbers with ruby

#+name: ruby_counter
#+begin_src ruby :shebang #!/usr/bin/ruby
$stdout.sync = true

10.times do |n|
  puts "And now writing! #{n}"
  puts "Hello from Ruby"
  sleep 0.5
end
#+end_src

- Print some numbers with python

#+name: python_counter
#+begin_src python :shebang #!/usr/bin/python
for i in range(0, 20):
  print "From Python"

Replacing Jekyll YAML headers with Org mode buffer settings

06 Dec 2013

I took a quick look at what would be required to avoid having to use YAML headers for Jekyll and using only Org mode buffer settings to set the layout, title from a Jekyll post.

module Jekyll
 class Post
    def read_yaml(base, name, opts = {})
      self.content = File.read_with_options(File.join(base, name),
                                            merged_file_read_opts(opts))
      self.data ||= {}

      org_text = Orgmode::Parser.new(self.content)
      org_text.in_buffer_settings.each_pair do |key, value|
        self.data[key.downcase] = value
      end

      self.extracted_excerpt = self.extract_excerpt
    rescue => e
      puts "Error converting file #{File.join(base, name)}: #{e.message}"
    end
  end
end

This way, instead of having to write

---
title: something
---

I can just continue to use Org mode syntax:

#+title: Replacing Jekyll YAML headers with Org mode buffer settings

My fork of jekyll-org which has this feature enabled is here.

Update [2013-12-07 Sat]

I did some modifications again to plugin for converting Org mode texts:

  • Disable using Liquid tags and rely mostly on Org mode syntax for exporting
  • Remove the #+TITLE from in buffer settings to exporting it 2 times with different styles

The resulting plugin is here:

require 'org-ruby'

module Jekyll
  # This overrides having to use YAML in the posts
  # and instead use in buffer settings from Org mode
  class Post
    def read_yaml(base, name, opts = {})
      content = File.read_with_options(File.join(base, name),
                                       merged_file_read_opts(opts))
      self.data ||= {}

      org_text = Orgmode::Parser.new(content)
      org_text.in_buffer_settings.each_pair do |key, value|
        # Remove #+TITLE from the buffer settings to avoid double exporting
        org_text.in_buffer_settings.delete(key) if key =~ /title/i
        buffer_setting = key.downcase
        self.data[buffer_setting] = value
      end

      # Disable Liquid tags from the output
      self.content = <<ORG
{％ raw ％}
#{org_text.to_html}
{％ endraw ％}
ORG
      self.extracted_excerpt = self.extract_excerpt
    rescue => e
      puts "Error converting file #{File.join(base, name)}: #{e.message} #{e.backtrace}"
    end
  end

  # Based from https://github.com/eggcaker/jekyll-org
  class OrgConverter < Converter
    safe true
    priority :low

    def matches(ext)
      ext =~ /org/i
    end

    def output_ext(ext)
      ".html"
    end

    def convert(content)
      content
    end
  end

  module Filters
    def restify(input)
      site = @context.registers[:site]
      converter = site.getConverterImpl(Jekyll::OrgConverter)
      converter.convert(input)
    end
  end
end

Started to blog with Jekyll

24 Nov 2013

Starting a blog using the Left theme for Jekyll.

Compared to other solutions, this uses the org-ruby parser to export .org files into HTML. You can check the source for this site here: https://github.com/wallyqs/wallyqs.github.io

puts "Syntax highlighting works!"

There are still some pending things that I would like to improve:

  • Use the re-org tool to create and organize the posts that would finally go into the _posts directory for publishing. I think in the end using a YAML file would have been better than having to use environment variables.
  • Try to extend or patch Jekyll so that I can avoid having to write: 
    ---
layout: post
title:  Hello world
category: posts
---

    And instead cover this with Org syntax like:

    #+layout:   post
#+title:    Hello world
#+category: posts

Other than that, the setup promises to be very comfortable for writing, I’ll see how it goes.