Hacking Atom

My experience writing an NLP-powered code suggestion engine

By Eddie Antonio Santos / @_eddieantonio

Let me tell you about my research!

Naturalness of Software

Natural Language Processing

$n$-gram language model

Can natural language processing be successfully applied to software?

Yes, probably.

A veritable buttload of applications!

Well, a lot of things!

Just a few of the applications are: pinpointing syntax errors in amateur code---code that looks a bit odd to our language models likely is the root cause of a syntax error, even if the compiler detects it somewhere else entirely.

We can find find common idioms in code---little snippets or templates of code, that are used a lot and infer what they mean, and what they mean about the code they're found in. And we can make it easier to input these snippets as well.

And, there is code completion and code suggestion. You can predict the next word or punctuation in a programming language by simply asking the language model what it thinks. Abram and pals developed one such application, that predicts exactly one word or punctuation (henceforth: token) in the future.

Enter: Gamboge

Multitoken code suggestion in Atom

Mean Surprise

$$S_{i} = \frac{I_{i}}{|t_{i}|}$$

(Steal this!)

In an n-gram language model, we care about the surprisal of a sequence of tokens. I could bore you about negative log probability, but it's actually pretty intuitive: given some tokens, how surprising is it to find it in the language model?

This give us an intuitive way at looking at the next token's probability. The most probable token following some sequence is the least surprising token after the sequence. Indeed, we call this measure the surprise of the token sequence. An event that is certain to happen, would have a probability of 100%. It is “not surprising”, so its surprisal value is 0. Similarly, an impossible event (according to our model) would leave us “infinitely surprised”.

Adding another token to a suggestion will almost always increases the overall surprisal of the suggestion (unless that next token follows the last 100% of the time). So, how do we prioritize longer, but more useful suggestions? Instead of sorting suggestions based on the surprisal of the entire suggestion, we'll sort it according to the average, or mean surprise of the entire suggestion. This means for single token suggestions, its surprisal doesn't change. But for multitoken suggestions, where some tokens contribute very little to the surprisal of the suggestion, the longer suggestion is given.

$S_{i}$ is the rank of suggestion $i$. $I_{i}$ here is the surprisal of all of the tokens in this suggestion. Therefore, just divide the total of tokens in the suggestion and we get an arithmetic mean. (A thought occurs: do we want mean surprise or mean probability? Because, mean probability would require log transforming...).

It's effectiveness is left as an exercise for the reader...

And that's all I have about the theoretical component.

Lessons Learned

It's just JavaScript (& npm!)

Developing for Atom, on one hand was really nice, because it's just javascript. It's built on top of Atom Electron.

Electron (formerly known as Atom Shell ) is basically the Chromium smushed together with Node.js. It's pretty rad! And you can use it independently of Atom to make desktop apps in HTML and Javascript. No bundler (Browserify, WebPack) required.

Does anybosdy use Slack? Slack is made using Electron!

You can just use npm and `require` like normal, which grants you one of the best parts of javascript. Plus it has most front-end libraries, incluing React and jQuery. It's beautiful.

That said, it's React. So don't touch the DOM!

The Atom Community is... uh...

Interesting priorities

CoffeeScript

Anyone down for a flamewar?

The Atom Community is... uh...

💩

Atom's Syntax Highligting Tokenizer

Insufficient for prediction!

require('esprima').tokenize('const hello = `world`');

Earlier, I mentioned that all you need for an n-gram are those precious, precious tokens; basically, a syntax highlighter should probably have a good idea of what tokens are in a language if it's going to highlight it. That said, using the tokens from the syntax highlighter is insufficient; its regular expressions often groups tokens together, or add whitespace to the defintion of a token that would otherwise lack it. Additionally, it knows nothing of the INDENT and DEDENT tokens in whitespace sensitive langauges (Python, Haskell, YAML, CoffeeScript), so it's completely useless for these langauges.

Luckily, one can use an external tool Esprima to do the tokenization for you.

Relying on Pre 1.0 APIs

BAD IDEA

I'd often get the rug pulled out underneath me as APIs were constantly changing. Not to mention the API docs were... lacking at best. I actually started a pull request to improve their docs for writing tests, and documenting the monkey-patching they did to make tests happen; but I ultimately did not complete it.

[Probably skip this] Also, the Vim-Mode was shit, and I actually learned plugin develpoing from having to patch the Vim-mode and make a PR.

At this point, I didn't know what React was, so I went ahead and modifed the DOM by hand. It's just JavaScript! But then they updated their React version which added aggressive checking of DOM invariants and inevitablly broke Gamboge. Gamboge has stayed in this state FOREVER.

Hacking Atom

My experience writing an NLP-powered code suggestion engine

Let me tell you about my research!

Naturalness of Software

Natural Language Processing

$n$-gram language model

Can natural language processing be successfully applied to software?

Yes, probably.

A veritable buttload of applications!

Enter: Gamboge

Multitoken code suggestion in Atom

Mean Surprise

(Steal this!)

Lessons Learned

It's just JavaScript (& npm!)

The Atom Community is... uh...

Interesting priorities

CoffeeScript

The Atom Community is... uh...

💩

Atom's Syntax Highligting Tokenizer

Insufficient for prediction!

Relying on Pre 1.0 APIs

BAD IDEA

Asynchronous testing in Jasmine 1.x

BAD IDEA

Conclusion

Resources

Natural Language Processing

Resources

Language Tools

Special Thanks