eddieantonio/blog

Python is a Compiled Language

2023-10-25T00:00:00+00:00

This blog post hopes to convince you that Python is a compiled language. And by “Python”, I don’t mean alternate versions of Python like PyPy, Mypyc, Numba, Cinder, or even Python-like programming languages like Cython, Codon, Mojo¹—I mean the regular Python: CPython! The Python that is probably installed on your computer right now. The Python that you got when you searched “python” on Google and downloaded the first thing that came up. The Python that you can pull up just by typing python3 into a fresh command line. That Python. Python is a compiled language.

Some context

Currently, I’m working on material for teaching students how to read and understand programming error messages. We are creating lessons for three programming languages: C, Python, and Java. One of the key points in teaching the nature of error messages is that they are generated at different stages: some error messages are reported at compile-time and others are reported while the program is actively running.

The first lesson was written for C, specifically using the GCC compiler. What these lessons demonstrated was that GCC splits the task of turning your code into a running program into various different stages:

preprocessing
lexical analysis
syntactic analysis
semantic analysis
linking

Furthermore, this lesson discusses the errors that may occur at each stage, and how that will affect the error messages that are presented. Importantly, an error in an earlier stage will prevent errors from being detected at a later stage.

While I was adapting this lesson for Java and Python, I realized that some things would have to change—for example, neither Python or Java have a preprocessor, and “linking” is not really the same thing in either Python or Java, so I just skipped these topics. However, I stumbled across an interesting insight.

Error messages can help you discover compilation stages

The fact that error messages are generated by different stages of the compiler, and compilers generally issue errors from earlier stages before continuing also means that you can discover the stages of your compiler by deliberately creating errors in a program.

So let’s discover the stages of the Python interpreter by playing a little game I like to call…

Which! Is! The! First! Error!

We are going to create a Python program with several errors, each of which attempts to induce a different kind of error message. We know that regular ol’ Python reports only one error message at a time—so the game is which error message will be reported first?

Here is the buggy program:

1 / 0
print() = None
if False
    ñ = "hello

Each line of code generates a different error message:

1 / 0 will generate ZeroDivisionError: division by zero.
print() = None will generate SyntaxError: cannot assign to function call.
if False will generate SyntaxError: expected ':'.
ñ = "hello will generate SyntaxError: EOL while scanning string literal.

The question is… which will be reported first? If you’re playing along at home, note that, unless otherwise stated, I will be reporting error messages from Python 3.12. Spoilers: the specific version of Python matters (more than I thought it would) so keep that in mind if you see different results.

Round 1

Now that we know the rules and the error messages that we are expecting, let’s start! Without running the code, which of the above error messages will be reported first?

Before I reveal the answer, think about what “interpreted” vs “compiled” language means to you. Here is a terrible Socratic dialogue that will hopefully make you personally reflect on what the difference is:

Socrates: A compiled language is one where the code is first put through a compiler before it is able to be run. An example is the C programming language. To run C code, first you have to run a compiler like gcc or clang, and then finally you can run your code. A compiled language gets converted to machine code—the ones and zeroes that your CPU understands.

Plato: But wait, isn’t Java a compiled language?

Socrates: Yes, Java is a compiled language.

Plato: But isn’t output of the regular Java compiler a .class file. That’s bytecode, isn’t it?

Socrates: That’s correct. Bytecode isn’t machine code, but Java is still a compiled language. This is because there are many problems that the compiler can catch, so you will need to correct errors before your program starts running.

Plato: What about interpreted languages?

Socrates: An interpreted language is one that relies on a separate program, aptly called an interpreter, to actually run your code. An interpreted language does not require the programmer to run a compiler first. Because of this, any errors that you make will be caught while your program is running. Python is an interpreted language—there is no separate compiler, and all errors that you make are caught at runtime.

Plato: If Python is not a compiled language, then why does the standard library include modules called py_compile and compileall?²

Socrates: Well, those modules just convert Python to bytecode. They don’t convert Python to machine code, so Python is still an interpreted language.

Plato: So, both Python and Java are converted to bytecode?

Socrates: Correct.

Plato: Then how is Python an interpreted language and yet Java is a compiled language instead?

Socrates: Because all errors in Python are caught at runtime.

Okay, enough of that. Let’s get to the actual answer. If you run code above in Python 3.12…

🥁🥁🥁🥁

You will get this error message:

  File "/private/tmp/round_1.py", line 4
    ñ = "hello  # SyntaxError: EOL while scanning string literal
        ^
SyntaxError: unterminated string literal (detected at line 4)

The first error message detected is on the last line of source code. What this tells us is that Python must read the entire source code file before running the first line of code. If you have a definition in your head of an “interpreted language” that includes “interpreted languages run the code one line at a time”, then I want you to cross that out!

What happened here?

I haven’t done a deep dive into the source code of the CPython interpreter to verify this, but I think the reason that this is the first error detected is because one of the first steps that Python 3.12 does is scanning (also known as lexical analysis). The scanner converts the ENTIRE file into a series of tokens before continuing to the next stage. A missing quotation mark at the end of a string literal is an error that is detected by the scanner—the scanner wants to turn the ENTIRE string into one big token, but it can’t do that until it finds the closing quotation mark. The scanner runs first, before anything else in Python 3.12, hence why this is the first error message.

Let’s close the quotation mark on line 4 to get rid of that error message and try again.

Round 2

With our first error fixed, this is what our code looks like:

1 / 0
print() = None
if False
    ñ = "hello"

There are still errors on lines 1, 2, and 3.

Which! Is! The! First! Error!

Make an educated guess on which will be the first error reported. I’ll spare you the strawman-y Socratic dialogue this time, and get straight to the point.

🥁🥁🥁🥁

File "/private/tmp/round_2.py", line 2
    print() = None
    ^^^^^^^
SyntaxError: cannot assign to function call here. Maybe you meant '==' instead of '='?"

It is the SyntaxError on line 2!

What happened here?

Once again, I did not check within the source code of CPython, but I am reasonably certain that the next stage is parsing (also known as syntactic analysis) and the parser reports the first error in the source code. Parsing the whole file happens before running the first line of code which means that Python does not even see the error on line 1 and reports the syntax error on line 2.

At this point, I will point out now that the program I wrote for this exercise is completely nonsensical and there is no right answer for how to fix the error. I have no idea what might be intended by print() = None, so I will fix this by replacing it with print(None) which also doesn’t make sense, but at least it’s syntactically correct.

Round 3

We fixed one syntax error, but there’s still another two errors in our file—one of which is also a syntax error. Here’s what our file looks like:

1 / 0
print(None)
if False
    ñ = "hello"

Which! Is! The! First! Error!

Recall that it seems that the syntax error took priority in round 2. Will it be the same in round 3?

🥁🥁🥁🥁

  File "/private/tmp/round_3.py", line 3
    if False
            ^
SyntaxError: expected ':'

Yes! The syntax error on line 3 took priority over the error on line 1.

What happened here

Just as in Round 2, the parser sees the entire file first before proceeding to the next stage. The fix is, as the error message (implicitly) suggests, to insert a colon before the end of the line.

Aside: Differences between Python versions

You might be wondering why I inserted two SyntaxErrors in the same file. Wouldn’t one be enough to show my point?

And here’s where the specific version of Python becomes relevant. If you try the same exercise in Python 3.8 or earlier, the line numbers for Rounds 2 and 3 are swapped!

In Python 3.8, the first error message reported for Round 2 is on line 3:

File "round_2.py", line 3
    if False
            ^
SyntaxError: invalid syntax

And after inserting the missing colon, Python 3.8 reports the following error message on line 2:

  File "round_3.py", line 2
    print() = None
    ^
SyntaxError: cannot assign to function call

Why are Python 3.8 and 3.12 reporting different errors? The reason is that Python 3.9 introduced a new parser. This parser is more capable than the previous, relatively naïve parser. The old parser (an LL(1) for you parsing nerds) was incapable of looking more than one token in advance which means that the parser in-and-of-itself technically accepted Python programs that are syntactically-invalid. Particularly, this limitation prevented the parser to identify whether the left-hand side of an assignment is a valid assignment target. The PEP linked earlier notes that the old parser accepts code that looks like this:

[x for x in y] = [1,2,3]

But this code makes no sense! In fact, the full Python grammar disallows this code. To fix this, a separate, hacky stage used to exist in Python that would check all the assignments and make sure that the left-side is actually something that can be assigned to. This occurred after parsing, hence why the error messages are swapped in the older versions of Python.

Round 4

Okay, with the last syntax error fixed in our program, this is what our program looks like:

1 / 0
print(None)
if False:
    ñ = "hello"

Okay, surely, now the next error message to be reported is the error on the first line. Right? …Right?

🥁🥁🥁🥁

Traceback (most recent call last):
  File "/private/tmp/round_4.py", line 1, in <module>
    1 / 0
    ~~^~~
ZeroDivisionError: division by zero

Yes, finally! At last, the error on the first line is reported.

Note also that this error message makes the first appearance of Traceback (most recent call last)—normally a staple of any Python error message, here appearing for the first time in the final round.

What happened here

Python was finally able to start running the code. Once all of the syntax errors are resolved, Python finally is able to interpret the first line, where it is forced to divide a number by zero, raising a runtime error called ZeroDivisionError! We know we’re in “runtime” because Python has printed Traceback (most recent call last) indicating that we have a stack trace. A stack trace is only something that can exist during runtime, which means that this error must have been caught during runtime.

This implies that the errors encountered in rounds 1–3 were… not… runtime errors? Then what were they?

Python is both a compiled and interpreted language

That’s right! The CPython interpreter really is an interpreter. But it also is a compiler. I hope the above exercise has demonstrated that Python must go through a few stages before ever running the first line of code:

scanning
parsing

Older versions of Python added an additional stage:

scanning
parsing
checking for valid assignment targets

Let’s compare this to the stages of compiling a C program from earlier:

~~preprocessing~~
lexical analysis (another term for “scanning”)
syntactic analysis (another term for “parsing”)
~~semantic analysis~~
~~linking~~

Python still performs some compilation stages before running any code. Python really does compile your code first, and just like Java, it compiles to bytecode. The implication for error reporting is that Python has compiler error messages and not every error message in Python occurs during runtime—in fact, only one of the four error messages above was raised at runtime, namely, ZeroDivisionError: division by zero.

You can actually compile all of your Python code beforehand using the compileall module on the command line:

$ python3 -m compileall .

This will place the compiled bytecode of all Python files in the current directory in __pycache__/ and show you any compiler errors. If you want to know what actually is IN that __pycache__/ folder, I did a talk for EdmontonPy which you should check out!

It is only after Python has been compiled to bytecode that the interpreter actually starts.³ I hope that the previous exercise has demonstrated that Python can indeed issue errors before runtime!

“Compiled vs. Interpreted language” is a false dichotomy

It’s a pet peeve of mine whenever a programming language is categorized as either a “compiled” or “interpreted” language. A language is not inherently compiled or interpreted; whether a language is compiled or interpreted (or both!) is an implementation detail.

And I’m not the only one that thinks like this. Laurie Tratt has a wonderful article arguing this exact same point by writing an interpreter that gradually becomes an optimizing compiler. It’s really neat!

Another fantastic resource is Bob Nystrom’s Crafting Interpreters. Here are some quotes from chapter 2:

What’s the difference between a compiler and an interpreter?

It turns out this is like asking the difference between a fruit and a vegetable. That seems like a binary either-or choice, but actually “fruit” is a botanical term and “vegetable” is culinary. One does not strictly imply the negation of the other. There are fruits that aren’t vegetables (apples) and vegetables that aren’t fruits (carrots), but also edible plants that are both fruits and vegetables, like tomatoes.

Fun fact: this was when first I learned what the deal is with tomatoes. In cooking, they’re a vegetable. And they’re, botanically, a fruit. They’re both. They can be both, and that’s okay. Wait, weren’t we talking about Python? Bob, bring us back:

When you run your Python program using [CPython], the code is parsed and converted to an internal bytecode format, which is then executed inside the VM. From the user’s perspective, this is clearly an interpreter—they run their program from source. But if you look under CPython’s scaly skin, you’ll see that there is definitely some compiling going on.

The answer is that it is both. CPython is an interpreter, and it has a compiler.

So why does it matter? Why is it counterproductive to make a hard distinction between “compiled” and “interpreted” languages?

“Compiled vs. Interpreted” limits what we think is possible with programming languages

A programming language doesn’t have to be defined by whether it’s compiled or interpreted! And thinking in such a rigid way limits what we believe a given programming language can do.

For instance, JavaScript is commonly lumped into the “interpreted language” category. But for a while, JavaScript running in Google Chrome would never be interpreted—instead, JavaScript was compiled directly to machine code!⁴ As a result, JavaScript can keep pace with C++. For this reason, I am really tired of arguments that say that interpreted languages are necessarily slow—performance is multifaceted and depends on much more than the “default” programming language implementation. JavaScript is fast now. Ruby is fast now. Lua has been fast for a while.

What about programming languages that are typically labelled as compiled?

Do you want a C interpreter? Here you go! Now you can write shell scripts that feature undefined behaviour!

REPLs for everybody!

One of the ways that I fell in love with programming was sitting in front of the REPL of languages like Python and Basic. That immediacy is what made programming thrilling for me, and is why I often reach for REPLs in my day-to-day life. But it seems like you can only have an interactive prompt for interpreted languages—so you can’t have a REPL for compiled languages, right?

Consider the Glasgow Haskell Compiler (GHC). One of the ways that I learned Haskell (and promptly forgot it 😉) is by messing around in the interactive console. But, as the name suggests, GHC really does compile Haskell code.

This makes being a “compiled” language no excuse for lacking an interactive interpreter! Do you want an interactive Java experience, similar to Python? It turns out that jshell has been bundled with Java since 2017.

But all of this is a distraction from the real distinction that we should be teaching students:

The true distinction: static vs. dynamic

The true distinction that we should be teaching students is the difference between properties of languages that can be determined statically—that is, by just staring at the code without running it—and properties that can only be known dynamically, during runtime.

Notice that I said “properties” and not “languages”. Every programming language chooses its own set of properties that can be determined either statically or dynamically, and taken together, this makes a language more “dynamic” or more “static”. Static versus dynamic is a spectrum, and yes, Python falls on the more dynamic end of the spectrum. A language like Java has far more static features than Python, but even Java includes things like reflection, which is inarguably a dynamic feature.

I find it understandable that dynamic vs. static is often conflated with compiled vs. interpreted—usually, languages that use interpreters have more dynamic features, like Python, Ruby, and JavaScript. Languages with more static features tend to be implemented without interpreters, like C++ and Rust. Then there’s Java that is somewhere in the middle.

Static type annotations in Python have been gradually (hehe) gaining adoption in codebases, and one of the expectations is that this can unlock performance benefits in Python code due to more things being known statically. Unfortunately, the truth is not that simple. It turns out that both types in Python (yes, just types in general—consider, metaclasses) and annotations themselves are dynamic features of Python, which makes static typing not the boon for performance that you would expect.

Conclusion

It doesn’t matter whether Python is compiled or interpreted. What matters is the set of properties that can be determined statically (that is, before runtime) in Python is relatively small, which means more errors tend to make themselves apparent at runtime than in programming languages that have more static features. This is the actual distinction that matters, and it is a far more granular and nuanced distinction than “compiled” vs “interpreted”. For this reason, I think it is important to emphasize the particular static and dynamic features when rather than teaching the tired distinction between “interpreted” and “compiled” languages.

EDIT: For source code for all the rounds (including a bonus round!) check out my GitHub repository. You can also see the results for errors across different implementations of Python, check out the different jobs in GitHub Actions.

I did not realize just how many Python compilers/Python-like languages exist. And this is very much a non-comprehensive list. ↩
Nobody would ever say this. ↩
Note that importing a module in Python happens at runtime, which means that any syntax errors in an imported module will give you error messages during runtime. You can still check for syntax errors in imported modules beforehand by using the compileall module. ↩
From what I understand, an interpreter called Ignition was added in 2016 to the V8 JavaScript engine, because the old “full-codegen” ate up a giant amount of memory. ↩

{n} times faster than C …with Python

2023-07-12T00:00:00+00:00

I was reading Owen Shepherd’s post “{n} times faster than C”, which explores how to hand-tune x86-64 assembly to make a certain problem faster (see below). Originally, this inspired me to write a short introduction to using Rust’s portable SIMD to manually speed up problems like this. I rewrote the problem in Rust (of course), used explicit SIMD, and observed a substantial speed-up.

However, that is not why I’m writing this blog post. Before getting started on writing that SIMD tutorial, I had an idea:

What if I compare the performance of my hand-rolled SIMD solution to a Python version that uses NumPy?

This innocent thought, dear reader, destroyed my productivity for the following two days. I learned, that you can indeed write plain ol’ Python code that can compute certain problems faster than a straightforward implementation in C and Rust. The secret, of course, is to write your Python program so that it’s doing all its work outside of Python, but I’m getting ahead of myself.

So instead of a tutorial on portable SIMD, we get this blog post — I’ve wasted so much time on this problem, I might as well write up some of my findings to wrap things up. This is how Python can be more performant than C or Rust.

But first, what is the problem we’re trying to solve?

The problem

The problem from Owen’s blog post is as follows:

Given a null-terminated buffer, maintain a count: increment that count for every s encountered. Decrement the count for every p encountered. Finally, return the count.

Here is Owen’s original C solution:

int run_switches(char *input) {
  int res = 0;
  while (true) {
    char c = *input++;
    switch (c) {
      case '\0':
        return res;
      case 's':
        res += 1;
        break;
      case 'p':
        res -= 1;
        break;
      default:
        break;
    }
  }
}

Owen’s post goes into detail about profiling this on his x86-64 machine, including hand-tuning assembly. I’m running this on my 2020 M1 MacBook Pro (see the appendix for more information on my testing setup), which uses an entirely different architecture (ARM AArch64 instead of x86-64), so Owen’s hand-tuned assembly won’t do me any good. So how does this code run on my computer?

I ran the above code with a test case of 12 MiB of random ASCII printable characters and got the following results:

Runtime: 7,794,642 nanoseconds per iteration 🦙
Throughput: 1.478 GiB/s

With a test case of 12 MiB of random s or p bytes, I get the following results:

Runtime: 41,617,334 nanoseconds per iteration 🐢
Throughput: 0.278 GiB/s or 284.9 MiB/s
Speedup: 0.19x

I didn’t do any investigation into why the same code is so much slower when given just s and p bytes, but if you peek at the compiled machine code (see it here on Compiler Explorer), it has 4 conditional branches. In general, modern CPUs work well if the branches are few, and if the outcome of the branches that do exist are predictable. Preferably, the only conditional branch you have is the one controlling the loop itself (e.g., while (condition) {...}). I’m guessing that the latter test case seems to make the branch predictor mispredict a lot, which is why it’s about 20% of the speed.

None of the remaining of the code examples seem to demonstrate an appreciable difference on performance when given either input, so I will only present the speed and any comparisons on the first test case, namely the random ASCII printable test case.

Rewrite it in Rust™

Naturally, being a tediously predictable Rustacean, I had to rewrite this in Rust. Here’s how that looks like:

use std::ffi::CStr;  // for C interoperability

pub fn rust_iter(s: &CStr) -> isize {
    s.to_bytes()
        .iter()
        .map(|c| match c {
            b's' => 1,
            b'p' => -1,
            _ => 0,
        })
        .sum()
}

On the random ASCII test case, it had this performance:

Runtime: 2,864,421 nanoseconds per iteration 🐆
Throughput: 3.979 GiB/s
Speedup: 2.72x

A few notes on this solution:

Unlike the C version, the Rust version knows the length of the buffer, and does not need to check for the null-terminator for each and every byte of the input.

Using iterators (that is, s.to_bytes().iter()) tends to enable a lot of optimizations in Rust, including auto-vectorization (that is, automatically using SIMD; see below).

Taking a quick peak at the assembly (see it here on Compiler Explorer) shows the use of instructions such as:

        movi    v1.16b, #112

Wow! movi is creating 16 copies of the number #112 (the ASCII value of the letter p) and putting them into a vector register v1!

And look!

        ldr     q19, [x11], #16

ldr loading 16 bytes from the input into a 128-bit wide-register q19, while incrementing the pointer by 16!

And then you have this absolute banger:

        cmeq    v20.16b, v19.16b, v1.16b

Which is using cmeq to simultaneously compare the 16 bytes from v1 against the 16 bytes just loaded into v19 (a.k.a., q19, the data from the input) and dumps the 16 individual comparisons into v20. Nice!

This is not the SIMD tutorial, so I won’t go into much depth about what’s happening here, but the Rust compiler definitely figured out how to automatically parallelize some parts of this algorithm, which is cool, and partially explains the speedup. Loading 16 bytes at a time is a whole lot faster than loading one byte at a time.

That said, the vectorized code that the compiler generated was not ideal. The majority of the instructions in the hot loop are the compiler attempting to maintain 16 simultaneous 64-bit sums, but in order to do that with the M1’s 128-bit wide SIMD registers, it must tediously shift bits around. This uses a lot of instructions and a lot of registers, which is work that should probably be avoided.

I thought that I could do better.

Rust portable SIMD: The fastest solution

Here is the Rust version that uses portable SIMD:

use std::ffi::CStr;
use std::simd::{u8x16, SimdInt, SimdPartialEq};

pub fn rust_portable_simd(s: &CStr) -> isize {
    let bytes = s.to_bytes();
    let (prefix, middle, suffix) = bytes.as_simd();

    let s = u8x16::splat(b's');
    let p = u8x16::splat(b'p');

    let mut result = 0;
    for &window in middle {
        let neg_ss = window.simd_eq(s).to_int();
        let neg_ps = window.simd_eq(p).to_int();
        let pairwise = neg_ps - neg_ss;

        result += pairwise.reduce_sum() as isize;
    }

    _count_scalar(prefix) + result + _count_scalar(suffix)
}

// Note: _count_scalar omitted for brevity

You can check out the assembly in Compiler Explorer.

So, how fast is this version?

Runtime: 584,040 nanoseconds per iteration 🚀
Throughput: 19.870 GiB/s
Speedup: 13.3x (compared to original C version)

584,040 nanoseconds is a bit difficult for me to conceptualize. I think milliseconds are bit easier to grok. 584,040 nanoseconds is 0.58 milliseconds. It took my three year old laptop just over half a millisecond to analyze over 12 and a half million letters. That’s sick. Computers are awesome. 🤘🏼

Predictably slow: the straightforward Python solution

The way that I generated test data for my benchmarks was by using NumPy, which is why I was using Python in the first place. Just for fun, let’s see what happens when we use the most route one Python code that you can imagine to iterate over the NumPy ndarray of random ASCII bytes and get a solution:

def python_for_loop(array):
    result = 0
    for element in array:
        if element == b"s":
            result += 1
        elif element == b"p":
            result -= 1
    return result

And how fast was this on my machine?

Runtime: 18,894,861,583 nanoseconds per iteration — that’s 18.89 seconds 🗿
Throughput: 0.617 MiB/s (measuring in gigabytes per second is a bit optimistic 💀)
Speedup: 0.000407x (compared to original C version)

So it turns out that calculating things one byte at a time in Python is really, really slow. Who knew!

However, there is another option.

Literally the first thing I tried in NumPy

This is where we get into the day-ruiner. I can’t remember my thought process clearly, but I have the vague notion in my head that operations that operate on entire vectors are really fast in NumPy, so try to think in vectors. And here is what I came up with:

def python_numpy(array):
    num_s = np.count_nonzero(array == b"s")
    num_p = np.count_nonzero(array == b"p")
    return num_s - num_p

This counts how many ss there are and then subtracts how many ps there are, and there’s your answer. How fast is it?

Runtime: 1,992,393 nanoseconds per iteration 🏎
Throughput: 5.800 GiB/s 🤯
Speedup: 3.91x (compared to original C version)

Yes. NumPy is nearly four times as fast as C. Of course, the original C code is hampered by having to iterate byte-by-byte to look for that null-terminator, not to mention that NumPy itself is a library written in C and Fortran, and it uses explicit SIMD parallelism to do its operations quickly.

However, I spent hours trying to replicate this algorithm with all its irreducible steps in Rust. That is, I could not find a way to write Rust code that could do the following:

Allocate a boolean vector whose elements are true if the corresponding byte in the input is equal to s.
Allocate a boolean vector whose elements are true if the corresponding byte in the input is equal to p.
Count how many non-zero values appear in the first vector.
Count how many non-zero values appears in the second vector.
Subtract the counts.

I’ll spare you the details, but in my attempts to write Rust code that faithfully does all these steps – steps that NumPy must do, given it’s working in Python – the best I could do is a solution that takes 3,769,195 nanoseconds per iteration or 3.031 GiB/s. It took me an entire working day of doing this, employing nasty tricks like using std::mem::transmute::<&[bool], &[u8]>(input) and unsafe { vec.set_len(input.len()) }. It got ugly! And I never managed to write any Rust code that did all that much faster than NumPy. 😖

Final results

	Implementation	Time per iteration	Throughput	Speedup
🥇	Rust - portable SIMD	584,040 ns	19.870 GiB/s	13.3x
🥈	Python - NumPy	1,992,393 ns	5.800 GiB/s	3.91x
🥉	Rust - iterators	2,864,421 ns	3.979 GiB/s	2.72x
	C - original	7,794,642 ns	1.478 GiB/s	1.00x
	Python - byte-by-byte	18,894,861,583 ns	<0.001 GiB/s	<0.01x

So Rust, using a manually-crafted SIMD algorithm is still the winner, but I am frankly alarmed to know that code can be nearly 4 times as fast as C with just NumPy, having to write only 4 lines of Python code.

There’s a lot more I could do here to investigate performance differences—I did not really employ any fancy profilers. I started to investigate NumPy’s source code and confirmed that the developers indeed wrote explicit SIMD, but that’s as far as I got. I’m curious how fast some straightforward Polars implementation would be.

Conclusion

You might not need to rewrite it in Rust, if you can somehow get NumPy to do all the heavy-lifting for you.

Appendix: Benchmarking set up

You can find all my code, including my janky benchmarking setup in this GitHub repository.

For all results, I used the same 12 MiB test case. For Rust and C, I used cargo bench to run benchmarks, however the docs are sparse, so I’m not entirely sure how many iterations were taken. For Python, I used timeit, taking 5 samples with 200 iterations per sample and reported the fastest mean time.

I compiled all C code with Clang with -O2 and -mcpu=apple-m1 to try to coax LLVM to generate code best suited for my machine. I compiled all Rust code with --release.

Here are the exact versions of the software I used.

Clang:

$ clang --version
Apple clang version 14.0.0 (clang-1400.0.29.202)
Target: arm64-apple-darwin21.6.0
Thread model: posix
InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin

Rust:

$ rustc --version
rustc 1.73.0-nightly (8ca44ef9c 2023-07-10)

Python and NumPy:

$ python3 --version
Python 3.11.0
$ python3 -c "import numpy; print('NumPy', numpy.__version__)"
NumPy 1.25.1

The machine I was testing on is just my personal laptop, a 2020 M1 MacBook Pro. Details of its specs are a bit complicated (4 performance cores?! 4 energy cores?! What? It has how large of a shared L2 cache?), but here is what macOS tells me:

$ uname -srm
Darwin 21.6.0 arm64
$ sysctl machdep.cpu.brand_string machdep.cpu.core_count hw.l1icachesize hw.l1dcachesize hw.l2cachesize
machdep.cpu.brand_string: Apple M1
machdep.cpu.core_count: 8
hw.l1icachesize: 131072
hw.l1dcachesize: 65536
hw.l2cachesize: 4194304

Installing Python on macOS (without going insane)

2020-01-26T00:00:00+00:00

If you’re like me (or Randall Munroe), the results of typing which -a python on your macOS machine is… devastating:

After many years of frustration, here are my recommendations for installing Python and Python-written utilities on macOS.

So what do I use Python for?

I use Python for library development, web development with Django, and scripting. I use Python primarily from the command line. I don’t really use Python for data science—at least not with Jupyter notebooks. I have absolutely no idea how to use conda. If you use Python primarily for data science, this guide is not written for you—there may be better solutions that I simply do not use. If your use case sounds similar to mine, please read on!

Do NOT install Python from python.org

It seems pretty obvious that you should install Python using the installer from Python’s official website.

The problem with this is that Python installs itself in a place that is difficult to manage without using administrator (i.e., sudo) privileges. This is okay for beginners, or people who only touch Python every so often. However, for my use cases, where I’m testing my code in multiple versions of Python, and have multiple virtual environments, this becomes bad news.

Homebrew

If you haven’t used Homebrew to install things on your Mac, go get it right now! I’ll wait.

We’re going to use Homebrew to install pyenv.

Do not install Python with Homebrew, though. Instead…

Manage multiple Python versions with `pyenv`

Install pyenv so that you can install multiple versions of Python and switch between them with ease:

brew install pyenv

Putting `pyenv` in charge by adding it to the `PATH`

Now you need to make sure your system can see the shims that pyenv creates. Shims are fake versions of all programs associated with a Python install. This allows pyenv to intercept a command and redirect it to the current or desired version of Python.

To do this, prepend the shim directory to your path. Add this line to the bottom of your shell startup file (either ~/.zshrc or ~/.bash_profile depending on either how new your Mac is or if you’ve customized your shell ¯\_(ツ)_/¯):

export PATH="$HOME/.pyenv/shims:$PATH"

The shim directory must be the first thing in your path, or at least, it has to come before /usr/bin/ and /usr/local/bin.

Now that you have added the line to your shell startup file, restart your shell for it to take effect. You can do this by closing your current terminal tab/window, and opening a new one, or in the same terminal/tab, you can do the following:

exec $SHELL

Let’s make sure our PATH looks right. To look at your path, use the following line:

printenv PATH

This is what my PATH environment variable looks like on my laptop:

/Users/eddieantonio/.pyenv/shims:/usr/local/bin:/usr/local/sbin:/bin:/usr/sbin:/sbin:/Library/TeX/texbin:/opt/X11/bin:/Users/eddieantonio/.local/bin

Kind of. I removed a few entries from my PATH to save myself from embarrassment.

Installing a Python version!

Now let’s install a modern version of Python. As of this writing, the current stable version of CPython is Python 3.8.1. Let’s install it with Pyenv!

pyenv install 3.8.1

If you want to see all the versions of Python that pyenv can install for you, you can list them:

pyenv install --list

This should be a pretty long list, with all kinds of alternate Python interpreters, including PyPy and Stackless. Install as many as you need!

Choose your default Python

Now tell pyenv the default version you want to use. Do this with pyenv global:

pyenv global 3.8.1

Now try it! Check what happens when you ask for the current Python version:

$ python --version
Python 3.8.1

Since I’m paranoid that something on my system will break because it’s expecting python to be Python 2.7 (yikes!) that’s pre-installed by Apple on my system, I do the following:

pyenv global system 3.8.1

This instructs to pyenv that it should check if the executable is installed on the system path first, and then try the selected Python version—in my case, CPython 3.8.1. You might be able to get away with pyenv global 3.8.1 and not use the pre-installed system Python at all.

If you don’t see the version you want to install, perhaps your installed version of pyenv is too old. Upgrade pyenv using Homebrew:

brew upgrade pyenv

pipx

I often want to install utilities that are written in Python, and available to install via pip. For example, I like installing black to format my code for me.

Do not do the following 🙅:

sudo pip install black

Instead, I’d like these globally-installed executables to be installed, without installing themselves on my Python path.

This is exactly the problem that pipx tries to solve.

Install it using Homebrew:

brew install pipx

Then install your favourite Python executables 👍!

pipx install black

Sadly, pipx will only install one package at a time, but we can do some arcane shell-fu to install multiple packages in one command line. Use xargs -n1 to call an command with exactly one of its input arguments:

# Install isort, mypy, snakeviz, pygments, and tqdm all on one line!
echo isort mypy snakeviz pygments tqdm | xargs -n1 pipx install

Managing project environments with poetry

For each of my projects, I want an isolated environment to install packages. Python has historically done this… poorly. The hacky solution for years has been virtual environments, but they require a lot of manual work, and you need to remember to create a virtual environment, activate it, place all the packages you installed in requirements.txt, and so on. What a mess!

Then came pipenv. I’ve used pipenv for about a year, and noticed a notable improvement in my workflow. However, some of the decisions that pipenv’s original creator made were questionable to say the least. pipenv is under new management, but some of the decisions and attitudes of the original creator have left scars on the current state of the project.

So then I switched to Poetry.

Poetry is my current choice for project dependency management. You can install it with Homebrew:

brew install poetry

Basic usage can be found on Poetry’s website, but here’s a quick tutorial:

Make a brand new project using poetry new:

poetry new my-awesome-app

cd into the project, and install some dependencies:

cd my-awesome-app
poetry add django

You can also specify dependencies you only use in development—that is, these packages should not be installed when somebody downloads your library, or deploys your app somewhere.

poetry add --dev black isort pylint

To actually run my application, I tend to use poetry shell to activate the current environment.

poetry shell

For example, in my Django app:

# this will not work -- not in virtualenv:
$ django-admin version
zsh: command not found: django-admin
$ poetry shell
Spawning shell within /Users/eddieantonio/Library/Caches/pypoetry/virtualenvs/my-awesome-app-VMVa3PrX-py3.7
# this works now 😄
(my-awesome-app-VMVa3PrX-py3.7) $ django-admin version
3.0.2

This is just scratching the surface of using Poetry. Read the docs to learn more.

Putting it all together: installing Python on macOS

Here is a dangerously copy-pastable snippet to install all the things:

# Install Homebrew:
/usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"

# Install pyenv, pipx, and poetry with Homebrew:
brew install pyenv pipx poetry

# Make sure pyenv is on the path:
echo 'export PATH="$HOME/.pyenv/shims:$PATH"' > ~/.zshrc

# Restart your shell (so we have the updated path):
exec $SHELL

# Install a modern version of Python using pyenv:
# (Python 3.8.1 is the latest stable version as of this writing)
pyenv install 3.8.1

# Tell pyenv to try using the system Python 2.x before using our pyenv
# installed Python 3.8.1
pyenv global system 3.8.1

# Install Python-based utilities globally using pipx:
# May I suggest black, isort, and mypy?
echo black isort mypy | xargs -n1 pipx install

Happy coding! 🍎🐍

Deploying a Parcel app with Now

2018-12-09T00:00:00+00:00

Last week, I deployed a single-page web application on Now.sh built with the Parcel bundler. Here’s how!

Setup

The files of my web application started off looking like this:

.
├── img/
├── index.html
├── main.js
├── package.json
└── styles.css

A single index.html entry point, with logic in main.js and styles provided by styles.css. There are some PNGs in img/ used for favicons. They’re all authored in a human-readable way—that is, with whitespace, and meaningful variable names.

Let’s bundle this with Parcel!

First, install parcel-bundler:

npm install parcel-bundler --save-dev

Then, build index.html with Parcel:

npx parcel build index.html

Next, in a new folder called dist/ you’ll see index.html and all the files included by it in dist/. But, they’ll be minified and content hashed!

With minification, index.html gets a modest 9.2% size decrease from 3.8 KiB to 3.45 KiB. However, styles.css goes from 2.7 KiB to 995 bytes, a 63.1% decrease!

Setting up `package.json`

I only want to commit my source code to GitHub; I don’t want to remember to build my package and push the resulting build products every time I change a single line of code (I know git pre-commit hooks exist, but… eww, no). Luckily, Now will run a build step for you before deploy!

I used Now’s official static builder. @now/static-builder will run a script specified in package.json. It expects that the results of the build will be created in a folder called dist/. It just so happens that Parcel places build output in dist/ by default!

To setup, @now/static-builder, I added the following line to the scripts key in package.json:

{
   "scripts": {
      "now-build": "parcel build index.html"
   }
}

Notice how this is the same line as before, just minus the npx part.

The nice thing about this, is that we can test if our build works by running the following command locally:

npm run now-build

Setting up `now.json`

To configure the @now/static-build builder, add a line to the builds key in now.json. Luckily, now.json doesn’t tend to have a lot of configuration, so the following is a complete now.json that you can copy-paste, and it will work in your app:

{
  "version": 2,
  "builds": [
    { "src": "package.json", "use": "@now/static-build" }
  ]
}

Deploying

From here, it’s up to your Now setup. I tend to push to a new branch in GitHub, and create a pull request. With Now’s GitHub Integration, this will automatically start the build and deploy it when it’s successful. I’ve added an alias to my now.json, so the latest pull request will be deployed as soon as it’s merged.

Other things

There’s an custom Now builder on npm, but I decided against using this, I’m wary of adding low star-count dependencies, ever since that event-stream incident. It turns out that using @now/static-build was fairly straightforward, so the custom package didn’t seem worth adding an extra requirement.

That’s all I got. I really like how relatively simple and painless it was to create a fast, minified, cache-friendly single-page web application with Now and Parcel. Getting a neat domain was pretty simple too! It’s bring us back to the simpler times when we’d deploy a new version of our application by uploading our PHP file via FTP to some random server. Ah, memories!

Why I made yet another Cree syllabics converter

2018-07-30T00:00:00+00:00

Image source: Ruben Quinn demonstrating the Cree syllabics star chart

Lately, I’ve been working on language technology for Plains Cree. Plains Cree is primarily written in two systems: standard Roman orthography (SRO) and syllabics. Since SRO uses the Latin alphabet—just like English—it is rather straightforward to type on a standard Canadian English keyboard. Syllabics keyboards are an ongoing struggle, however (I may expand on this some other time on this blog). Therefore, if one wants to write Cree in syllabics, it is sometimes easier to type it in SRO first, then use a transliterator to convert from SRO to syllabics.

In summary, I created a new bidirectional transliterator called cree-sro-syllabics (Edit: Try it online at syllabics.app!). It’s a new transliterator, because several transliterators already exist. So what do the others look like?

What are the freely available transliterators?

A quick Google search will net you at least the following SRO to syllabics transliterators.

However, none of these transliterators are perfect.

The issues

Word final “hk”

In syllabics, a word that ends with an “hk”—or “ᐦᐠ” in syllabics—are supposed end with “ᕽ” instead. However, this replacement can never occur in the middle of a word.

For example, the word “ê-wêpâpîhkêwêpinamâhk” (“we (and not you) are setting it swinging”), contains both a final “hk” and a “hk” cluster in the middle of the word. Its syllabic transcription is ᐁᐍᐹᐲᐦᑫᐍᐱᐊᒫᕽ.

Although the Algonquian Linguistic Atlas’s converter and syllabics.net’s converter both handle this, the Maskwacîs Converter does not, instead producing ᐁ ᐁᐧᐸᐱᐦᑫᐁᐧᐱᓇᒪᐦᐠ, with an erroneous “ᐦᐠ” cluster at the end.

Transliterating non-Cree words

Some transliterators attempt to convert every Latin character, even if it doesn’t make sense. Take the case of “Maskêkosihk Trail”—a road that goes from Edmonton to Enoch Cree Nation. The City of Edmonton unveiled the street sign, and, in the process, they unveiled an embarrassment:

Image source: CBC

Not only does the syllabics transliteration of the sign contain the “hk” error as mentioned above, but it half-transliterates the English word “trail” into syllabics. The result is that “trail” is rendered as “ᐟrᐊᐃl”, which contains Latin characters in the transliteration!

In my opinion, an SRO to syllabics transliterator should refuse to transliterate words that do not have the structure of a Cree word. However, all three of the mentioned transliterators do attempt to transliterate “trail” with differing results:²

Maskwacîs Cree Dictionary	ᐟrᐊᐃl
Algonquian Linguistic Atlas	ᐟᕒᐊᐃᐪ
Syllabics.net	ᐟᕒᐊᐃᓬ

Long vowels

Long vowels (âêîô) are distinct from short vowels (aio) in Cree. Long vowels are written with a dot above in syllabics. The exception is for “ê” because it is always long; as a result, some writers also drop the diacritic when writing “e” in SRO as well. It’s important to differentiate between long and short vowels, because it makes distinctions between words. For example, nipiy/ᓂᐱᕀ means “water” while nîpiy/ᓃᐱᕀ means “leaf”. However, there is such a thing as “plain” script, where the vowel dots are omitted, and pointed script where the vowels have all dots.

Another complication is that the “standard” Roman orthography in practice has multiple conventions for writing long vowels: using a macron (◌̄) and using a circumflex (◌̂).³

How do the various converters handle long vowel diacritics? The Maskwacîs converter does not produce dots for long vowels at all, however it accepts both macrons and circumflexes as input. The Algonquian Lingustic Atlas’s converter not only produces dots, but supports input in either macrons or circumflexes. The syllabics.net converter does worst of all, handling only macrons for long vowels. It simply spits out characters written with circumflexes. Additionally, it does not handle “ê” without an diacritics, which all other converters do.

Other odds and ends

Other issues for syllabics converters include how they deal with dashes, how they deal with combining diacritics, rather than pre-composed characters, and whether they produce the correct Unicode characters for the syllabics rather than very convincing look-alikes. There’s also the sandhi orthographic rule, but honestly, I’m not sure I fully comprehend how to apply this rule myself.

Summary

Here’s a breakdown of the previous issues, and whether each transliterator can handle it correctly.

	Word-final “hk”	Non-Cree words	Long vowels
Maskwacîs Cree Dictionary	❌	❌	❌
Algonquian Linguistic Atlas	✅	❌	✅
Syllabics.net	✅	❌	❌

Where’s the source code?

The most pressing issue to me personally is that I cannot find source code for any of these converters! This means that if other people want to incorporate a converter into their own app without an active internet connection, they can’t. They have to either reverse-engineer the converters online, or write their own code to do the conversion.

`cree-sro-syllabics`: an open-source Python and JavaScript library for syllabics conversion

My solution was to create a Python library that is free and open source.

EDIT: (2018-12-06) Now also available for JavaScript!

It handles all the issues previously mentioned. Try it with the following test cases:

The source code for cree-sro-syllabics can be found on its GitHub page, but it can also be seamlessly incorporated into a Python project that uses pip by installing it with:

pip install cree-sro-syllabics

EDIT: (2018-12-06) You can use npm to install cree-sro-syllabics in your JavaScript project:

npm install cree-sro-syllabics --save

Or you can copy-paste the .js file to your project (as long as you keep the AGPL license comment at the top!)

The future

I hope in future versions to add support for Woods Cree and other West Cree dialects. There are also a few interesting things that can be done to make sure SRO and syllabics conversions can be completely reversed without losing information about morpheme boundaries.

EDIT: (2018-12-06) In the original post, the library was called crk_orthography. It has been renamed to cree-sro-syllabics.

EDIT 2: (2018-12-06) cree-sro-syllabics now features beta support for Woods Cree (Th-dialect) and Swampy Cree (N-dialect).

I don’t have an iPhone to confirm this, but I believe this is the same converter bundled in the Cree Dictionary app. ↩
I wonder if the sign designer used the Maskwacîs transliterator to get this result. ↩
Anecdotally, I find that most writers near Edmonton and Maskwacîs prefer circumflexes to macrons; however noted Algonquian linguist Arok Wolvengrey prefers macrons. Heck, Jean Okimāsis writes her surname with a macron! ↩

Syntax and Sensibility: Using language models to detect and correct syntax errors

2018-01-15T00:00:00+00:00

Has this ever happened to you?

You’re writing source code, say this:

package ca.ualberta.cs.example;

public class Hello {
    public static void main(string args[]) {
        if (args.length != 2)
            int exitstatus = 2;
            system.out.println("not enough args");
            system.exit(exitstatus);
        }
        system.out.println("hello, world!");
    }
}

And when you go to compile it, you’re bombarded with error messages:

Hello.java:6: error: variable declaration not allowed here
            int exitstatus = 2;
                ^
Hello.java:10: error: <identifier> expected
        system.out.println("hello, world!");
                          ^
Hello.java:10: error: illegal start of type
        system.out.println("hello, world!");
                           ^
Hello.java:12: error: class, interface, or enum expected
}
^
4 errors

Four error messages! So the error must be on at least line 6, right? Right…?

After scratching your head for a while, you figure out that line 5, which looks like this:

if (args.length != 2)

…should actually have an open brace at the end of the line, like this:

if (args.length != 2) {

How the crumb was I supposed to figure that out from pouring over the error messages?

There’s got to be a better way

Introducing Sensibility, a tool designed to not only find where these syntax errors are, but also to tell you how you might be able to fix it!

The secret: use natural language models—specifically n-gram and LSTM language models—to figure out where the “unnatural” code lurks.

But Eddie… using NLP on code? That’s craziness!

Nope! It’s naturalness.

Sensibility can produce a valid fix for about 50% of all single-token syntax errors, often producing the true fix that a novice programmer would have made. For example, given the broken code above, Sensibility produces this output:

Hello.java:5:29: try inserting '{'
        if (args.length != 2)
                               ^
                               {
Hello.java:6:13: try replacing int with {
            int exitStatus = 2;
            ^
            {

Either of which will fix the file, but the former is probably what you intended :).

For more details, read our paper Syntax and Sensibility: Using language models to detect and correct syntax errors to appear in SANER 2018. Here’s a link to a preprint. And here’s the BibTeX.

@inproceedings{eddie2018SANER2018sasulmtdacse,
 accepted = {2017-12-18},
 author = {Eddie Antonio Santos and Joshua Charles Campbell and Dhvani Patel and Abram Hindle and José Nelson Amaral},
 authors = {Eddie Antonio Santos, Joshua Charles Campbell, Dhvani Patel, Abram Hindle, José Nelson Amaral},
 booktitle = {25th IEEE International Conference on Software Analysis, Evolution, and Reengineering (SANER 2018)},
 date = {2018-04-21},
 funding = {NSERC Discovery, MITACS Accelerate},
 location = {Campobasso, Italy},
 pagerange = {1--11},
 pages = {1--11},
 role = { Author},
 title = {Syntax and Sensibility: Using language models to detect and correct syntax errors},
 type = {inproceedings},
 url = {http://softwareprocess.ca/pubs/santos2018SANER-syntax.pdf},
 venue = {25th IEEE International Conference on Software Analysis, Evolution, and Reengineering (SANER 2018)},
 year = {2018}
}

The code is available on GitHub.

Special thanks to my coauthors, Joshua Charles Campbell, Dhvani Patel, Abram Hindle and José Nelson Amaral.

Anatomy of a crash repository

2016-11-18T00:00:00+00:00

What mysteries sleep deep within Ubuntu’s crash report repository? Find out in our latest preprint, Anatomy of a crash repository!

Ever wonder what functions crash most often? What signals cause the most crashes (Spoilers: it’s segfaults)? Then you should read our preprint!

This is a follow-up my previous paper with Joshua Campbell and our supervisor Abram Hindle.

Judging a Commit by Its Cover

2016-02-22T00:00:00+00:00

For the MSR 2016 Challenge, I applied data science to test a gut feeling I had: commits with weird, unusual log messages are often committing lower quality than code than boring, ordinary commit messages.

So: Are commits that look fishy…

…actually hiding dubious code?

Is my hunch true? (Spoilers: Marginally).

To test, we correlated build status from Travis-CI with n-gram language models on commit messages. Here’s the abstract:

Developers summarize their changes to code in commit messages. When a message seems “unusual,” however, this puts doubt into the quality of the code contained in the commit. We trained n-gram language models and used cross-entropy as an indicator of commit message “unusualness” of over 120 000 commits from open source projects. Build statuses collected from Travis-CI were used as a proxy for code quality. We then compared the distributions of failed and successful commits with regards to the “unusualness” of their commit message. Our analysis yielded significant results when correlating cross-entropy with build status.

Intrigued? Read the preprint! (And fork the replication code and data!)

EDIT: Added my presentation.

Acknowledgements to my supervisor, Abram Hindle, and my colleagues S. Kalen Romansky, and Shaiful Chowdhury for their reviews and comments!

The Unreasonable Effectiveness of Traditional Information Retrieval in Crash Report Deduplication

2016-02-06T00:00:00+00:00

Last week, Joshua Charles Campbell, Abram Hindle, and I wrote a paper on crash report bucketing.

You know when a program crashes, and it asks you to “Send a report,” so you just click “Okay” to make the message go away? This paper addresses the other side of that story, where several thousands of people dismissing that message with an innocent “Okay” every single day leads to a veritable deluge of crash reports that leave the developers with more reports than they know how to deal with.

We developed and evaluated a technique called PartyCrasher that’s like Google for crash reports. Basically, we get the thousands of crashes, do some very basic text search, and find a “bucket” of crashes that look most like it.

Here’s the official abstract:

Organizations like Mozilla, Microsoft, and Apple are flooded with thousands of automated crash reports per day. Although crash reports contain valuable information for debugging, there are often too many for developers to examine individually. Therefore, in industry, crash reports are often automatically grouped together in buckets. Ubuntu’s repository contains crashes from hundreds of software systems available with Ubuntu. A variety of crash report bucketing methods are evaluated using data collected by Ubuntu’s Apport automated crash reporting system. The trade-off between precision and recall of numerous scalable crash 7 deduplication techniques is explored. A set of criteria that a crash deduplication method must meet is presented and several methods that meet these criteria are evaluated on a new dataset. The evaluations presented in this paper show that using off-the-shelf information retrieval techniques, that were not designed to be used with crash reports, outperform other techniques which are specifically designed for the task of crash bucketing at realistic industrial scales. This research indicates that automated crash bucketing still has a lot of room for improvement, especially in terms of identifier tokenization.

Still interested? Read the preprint!

Authoring man pages in Markdown with Pandoc

2015-12-18T00:00:00+00:00

man pages are one of the most tenacious forms of programmer-oriented documentation. These manual pages have changed relatively little since their first publication in 1971. Regardless of whether or not they’re still relevant in the age of the internet, they remain an invaluable reference for any serious command line application or C library function.

Yet, in the whimsical technological wonderland that is 2015, one often wonders: there has got to be a better way of authoring man pages than in their native form: crusty old troff macros–document publishing’s equivalent of assembly language. Alas, dear reader, there is a way! It’s called Pandoc: a self-touted “universal document converter”. For our purposes, it can translate a modern expressive document formatting language such as Markdown into a troff-formated man page.

So let’s get started!

First, install Pandoc somehow. I recommend using cabal if you plan on writing your own Pandoc filters in Haskell. Sane people can probably get by just fine installing Pandoc using their favourite package manager, or following the instructions on Pandoc’s website.

The Markdown template

Although you may write your man page in any input format that Pandoc understands, I use Markdown since Pandoc provides a few Markdown extensions that are quite useful for man page authoring. Here’s the general template I use (view its raw Markdown source text):

There are a few things to note:

Headers

man pages generally have at least the following sections:

NAME — The name of the command or function and a minimal description.
SYNOPSIS — A terse listing of command line arguments or the function prototype.
DESCRIPTION — The bulk of the man page; describes in-detail how to use the command, or function.

man pages also tend to include some of the following sections:

OPTIONS — Command line options; this may also be a subsection of DESCRIPTION.
EXAMPLES — Informative usage examples. Personally, I prefer man pages that list the most common use cases in the DESCRIPTION, but it’s also handy to have a section purely dedicated to example usage.
FILES — Descriptions names of important files, such as configuration files.
ENVIRONMENT — Description of relevant environment variables that affect functionality.
BUGS — well-known bugs; however, I generally just throw a link to the corresponding issue tracker.
AUTHORS — Who made it?
SEE ALSO — References to other man pages.

Another useful resource on these headings is TLDP’s man page HOW-TO.

Section numbers

All manual pages reside in a numbered section. A man page is often referenced by its name and section number as in hello(1). A list of section numbers can be found in man’s man page, but some more common sections are section 1, general user commands; and section 3, C library functions.

The filename

For this file, I’ve chosen hello.1.md. The filename ends with .md to indicate it’s a Markdown document. Since its ultimate goal is to become a man page called hello.1, .md is prefixed with .1. Do not mistake .1 for a file extension! It’s just convention that a man page’s “file extension” is the manual section number it belongs to.

In order for man pages to be generally accessible, they must be stored in a path matching this pattern:

$PREFIX/share/man/man$SECTION/$NAME.$SECTION

Where $NAME is the name of the man page (i.e., you will type man $NAME to view it); $SECTION is the section number of the man page; and $PREFIX is a prefix to install tools in Unix such as /usr/local or /opt.

For example, my hello binary will live in the following path:

/usr/local/bin/hello

Its corresponding man page will live beside it here:

/usr/local/share/man/man1/hello.1

The metadata block

Pandoc’s Markdown parses the first three lines as metadata if they are prefixed with a % sign. The first line is special: it’s the document title, and for man pages in particular, it has the following format:

% NAME(#) Version | Header

In this case, we have a command named hello. It’s a general command, so its manual section is 1. It’s the very first major version of this command, so it gets Version 1.0. Finally, the header–separated from the rest of the title by a pipe character (|)–usually indicates what set of documentation this manual page belongs to.

% HELLO(1) Version 1.0 | Frivolous "Hello World" Documentation

If we provided author names using the second metadata line, Pandoc would automatically append an AUTHORS section to the very end of the man page. I have opted to write this section by hand, however, for greater control over the formatting of this section.

Useful Pandoc Markdown Extensions

I have found two Pandoc Markdown extensions particularly useful for man page authoring: definition lists and line blocks.

Definition lists are useful for describing particular command line arguments, files, and environment variables in depth.

-o, --output *file*

:  The --out option changes the default output to the specified file...

Line blocks, which preserve leading spaces and line breaks, are useful for writing multi-line synopses. Often, multiple lines are needed if there are several mutually-exclusive invocations (such as using the --help option in most commands).

| **hello** \[**-o**|**--out** _file_] \[_dedication_]
| **hello** \[**-v**|**--version**]
| **hello** \[**-h**|**--help**]

Note the escaped square brackets (\[ ... ]). Since line blocks still contain formatted text–only initial spacing and line-breaking are preserved–any Markdown syntax such as link brackets are still interpreted. This is useful for marking individual options with bold text and filenames with italics. However, this gets in the way when using idiomatic square brackets to denote optional components. The opening square bracket must be escaped to avoid text being interpreted as link text.

Building the man page

Now, to turn our Markdown into an actual man page, we just run this command:

pandoc --standalone --to man hello.1.md -o hello.1

The --standalone option is necessary; without it, Pandoc will naïvely translate the Markdown to troff without providing the header, footer, or the general man page structure to the output.

Building with a Makefile

I often create man pages in conjunction with a simple make build system. The following is a file you can copy paste or include into your Makefile to make it convert any Markdown-formatted man page into appropriate troff output.

To use, simply include pandoc-man.mk in your Makefile, set MANSECTION as appropriate for your man page (unnecessary if it’s in section 1), and declare the name of your desired man page as a build dependency. For my hello(1) example, my Makefile is as follows:

Useful Examples

I have authored a few man pages this way:

imgcat(1)
license(1)
unormalize(1), and its friends nfc(1), nfd(1), nfkc(1), nfkd(1) (which all point to the same man page by way of symbolic links).

As well, it’s useful to look at other well-written man pages for inspiration. For example, curl(1) is particularly good.

eddieantonio/blog

Python is a Compiled Language

Some context

Error messages can help you discover compilation stages

Which! Is! The! First! Error!

Round 1

What happened here?

Round 2

What happened here?

Round 3

What happened here

Aside: Differences between Python versions

Round 4

What happened here

Python is both a compiled and interpreted language

“Compiled vs. Interpreted language” is a false dichotomy

“Compiled vs. Interpreted” limits what we think is possible with programming languages

REPLs for everybody!

The true distinction: static vs. dynamic

Conclusion

{n} times faster than C …with Python

The problem

Rewrite it in Rust™

Rust portable SIMD: The fastest solution

Predictably slow: the straightforward Python solution

Literally the first thing I tried in NumPy

Final results

Conclusion

Appendix: Benchmarking set up

Installing Python on macOS (without going insane)

So what do I use Python for?

Do NOT install Python from python.org

Homebrew

Manage multiple Python versions with pyenv

Putting pyenv in charge by adding it to the PATH

Installing a Python version!

Choose your default Python

pipx

Managing project environments with poetry

Putting it all together: installing Python on macOS

Deploying a Parcel app with Now

Setup

Setting up package.json

Setting up now.json

Deploying

Other things

Why I made yet another Cree syllabics converter

What are the freely available transliterators?

The issues

Word final “hk”

Transliterating non-Cree words

Long vowels

Other odds and ends

Summary

Where’s the source code?

cree-sro-syllabics: an open-source Python and JavaScript library for syllabics conversion

The future

Syntax and Sensibility: Using language models to detect and correct syntax errors

There’s got to be a better way

Anatomy of a crash repository

Judging a Commit by Its Cover

The Unreasonable Effectiveness of Traditional Information Retrieval in Crash Report Deduplication

Authoring man pages in Markdown with Pandoc

The Markdown template

Headers

Section numbers

The filename

The metadata block

Useful Pandoc Markdown Extensions

Building the man page

Building with a Makefile

Useful Examples

Manage multiple Python versions with `pyenv`

Putting `pyenv` in charge by adding it to the `PATH`

Setting up `package.json`

Setting up `now.json`

`cree-sro-syllabics`: an open-source Python and JavaScript library for syllabics conversion