Testing Vimscript with Vader.vim
2013. 10. 27.

TDD, or test-driven-development is already a well-established concept today, widely adopted by many software developers. So it's really not necessary for me to stress the importance of writing tests here again. But anyway, here's my story. In a nutshell, I learned that testing is important even for Vim plugins, and I created Vader.vim to test my own plugins.

On writing Vim plugins

Over time I've written a few Vim plugins. My typical Vim plugin starts as a collection of functions and commands extracted from my .vimrc. The feature set is usually quite small at this stage that I barely feel the need for an automated testing.

However once the code is formed into an independent plugin that can be used by anyone, not just me, I need to care much more about the details of it. Some random code in my .vimrc is "good enough" if it just works for me. Small errors are neglected since I know exactly what it's capable of and what its limits are.

But for a public Vim plugin, it's a whole different story. Now it has to run without errors on any popular platform, and it should not break even when the user is extremely imaginative and unorthodox.

Development-Driven-Test

And of course there will be feature requests. Naturally the code will keep growing over time, and soon it'll become much more complex than it was in its conception. Only then, I regret that I didn't pay enough attention to writing tests earlier.

Now if I want to add a feature or need to refactor some parts of the code, I have to be sure that it doesn't break any of its documented functionality. Now if it suddenly breaks, people will mock me for spending my time to write a free software they find it useful and making a small mistake in maintaining it. Obviously I don't want that to happen. But the plugin already has too many features and oddities, some of them even I lost track of, it's hard not to make a mistake if I keep testing it manually.

Semi-automated tests with Vim macros

This little story is what happened when I was passionately improving vim-easy-align, a text alignment plugin for Vim. There were already two popular plugins with the same goal, namely Tabular and Align.vim. Unfortunately, I was not completely satisfied with those two, so I made up my mind to start my own with a very small feature set that is custom-tailored to meet my own needs, and vim-easy-align was the result of the attempt. However, I'd gotten a little too ambitious over time and ended up implementing quite a lot of features. (Now I believe it has even more features than its predecessors.) Manual testing had become so time-consuming and error-prone, and finally I felt the great need for automated testing.

To keep it simple, I decided to use native Vim macros (or recordings) to record and replay keystrokes that transform the given file into the expected result. So if I want to test the plugin, I just open up an input file, replay the macro, and see if the modified buffer matches the expected result. Nothing fancy, but it worked pretty well for a while. However, as the number of test cases increased, the macro file had grown into a giant mess that I couldn't stand anymore. The biggest problem was its horrible readability. Whenever the result of the macro did not match the expected output, first thing I had to do was to figure out which part of the script was responsible for the breakage, but even that was never easy when all I was given was a sequence of several thousand keystrokes.

I wanted to have a better way. Yet I didn't care for writing English-like DSL for testing which I think rather complicates the problem than simplifying it. Simple JUnit-style assertions and macros that are readable should be good enough for most of the cases. Let's not over-engineer the problem. Let's keep it simple.

Bearing these in mind, I've finally written my own Vimscript testing tool, Vader.vim.

Vader.vim

Vader.vim is a simple tool designed to help you write Vimscript tests in a readable way. I wanted to make its interface simple and intuitive that it does not require a lengthy document. I wanted it to be instantly obvious to anyone who has the basic idea of testing.

Vader.vim defines its own .vader file format, which is a flat series of blocks of the following types.

If you have experience with any testing framework, I suppose you can easily conjecture what each type is for. Let's just see an example.

# Test case
Execute (test assertion):
  %d
  Assert 1 == line('$')

  setf python
  AssertEqual 'python', &filetype

Given ruby (some ruby code):
  def a
    a = 1
    end

Do (indent the block):
  vip=

Expect ruby (indented block):
  def a
    a = 1
  end

Do (indent and shift):
  vip=
  gv>

Expect ruby (indented and shifted):
    def a
      a = 1
    end

The code is self-explanatory. You can Execute some Vimscript and test it with Assertions. You're Given some ruby code, Do some work on it, and you Expect the result of it matches your expectation. Also, you can have arbitrary Vimscript executed Before and After each test case.

Well, that's all I have to say about .vader format.

Once you've finished writing a vader file, you can execute it with Vader command.

:Vader test.vader

And the result is given as follows.

Starting Vader: 1 suite(s), 3 case(s)
  Starting Vader: /Users/jg/.vim/plugged/vader.vim/example/test.vader
    (1/3) [EXECUTE] test assertion
    (2/3) [  GIVEN] some ruby code
    (2/3) [     DO] indent the block
    (2/3) [ EXPECT] indented block
    (3/3) [  GIVEN] some ruby code
    (3/3) [     DO] indent and shift
    (3/3) [ EXPECT] indented and shifted
  Success/Total: 3/3
Success/Total: 3/3 (assertions: 2/2)
Elapsed time: 0.160679 sec.

Congratulations. You just accidentally learned almost everything about Vader.vim.

Tests in Vader.vim

Vader.vim is a very new and immature project, but I've written/rewritten tests for most of my plugins with it and it has worked very well so far. If you're interested in writing your own tests with Vader, you can refer to the vader files I've written in the following links.

» capture | close