Kamal Marhubi

What happened to my hg advent series?

Sun, 09 Dec 2018 00:00:00 -0500

hg advent -m '06: debugging hg-git'

Sun, 09 Dec 2018 00:00:00 -0500

hg advent -m '05: heads and `hg commit --close`'

Thu, 06 Dec 2018 00:00:00 -0500

hg advent -m '04: there is no index'

Sat, 3 Dec 2018 00:00:00 -0500

hg advent -m '03: revsets!'

Sat, 3 Dec 2018 00:00:00 -0500

hg advent -m '02: extensions'

Sat, 2 Dec 2018 00:00:00 -0500

hg advent init

Sat, 1 Dec 2018 00:00:00 -0500

This is the start of a daily series in the run-up to Christmas where I learn mercurial. This first post will be about what's making me want to do this in the first place.

Rust should be an excellent open source citizen #rust2018

Wed, 31 Jan 2018 00:00:00 -0500

Here’s my late #rust2018 post. I contribute code to a lot of projects in a lot of different languages. I think it’s really important not to think about Rust in isolation, but in the context of the greater open source world. Here area few specific ways I think about this:

ease of contributing to projects in rust
interoperability with other languages (both directions)
ripgrep should be in debian
don’t fracture the ecosystem

The rest of this post is going through these in turn.

Rust projects should be easy to contribute to

I make a fair number of open source contributions, whether to fix small bugs, or scratch an itch. Sometimes the projects are in languages I don’t know well, or even at all. As a casual contributor, I want to be able to get in and make my change with as little incidental difficulty and frustration as possible.

But it doesn’t always work like that. The language a project is written in is a huge factor in how easy or hard it is.

My biggest wish for Rust in 2018 is that I would like to feel free to write programs in Rust without worrying that I’m excluding people from contributing.

I see a few ways that choice of language can affect who can contribute code changes, and how easily:

readability
writability
community coding style
tooling

Readability

To make a code change, you need to understand what the code is doing. The language itself plays a big part. For example, Python code is very easy to read, and Go is almost comically easy to read. It’s great!

Rust is doing ok here, but there’s room for improvement. The ergonomics work from 2017 will make a huge difference. It just needs to land! In particular, impl Trait and argument lifetimes will cut down on a huge amount of angle bracket noise, and match default bindings will get rid of most uses of the ref keyword, which doesn’t come up anywhere else in the language.

A core focus of 2017 was ergonomics and I don’t want that to go away. Rust should keep pushing on this throughout 2018, and beyond.

Writability

If you’re making a small code contribution to a program you use, often you get started by copy-paste-modify. Ideally the language doesn’t make that too hard. This is another place where ergonomics are important. Ownership and borrowing make this inherently harder than in other languages: I don’t know any other languages where

foo(x);
foo(x);

can be a compiler error!

The compiler messages keep getting better, and this really does make a difference.

Going beyond copy-paste-modify coding, Rust’s syntax has a fair number of gotchas. Luckily there’s a bunch of ergonomics improvements that should land soon. Even as a fairly experienced Rust programmer, I find the “how much &*[..] random noise do I need to add to make a thing work?” thing annoying. I’m really thankful for the effort that’s being poured into this.

I can’t wait to be able to stop saying to people, ‘I know it’s bad, but it’s being worked on’, and start saying ‘have you tried 1.28? All that nonsense is gone now!’

Community coding style

The kind of code that a language community prefers to write also makes a huge difference. There was a project I wanted to make a change to, and it happened to be written in Scala. The community coding style there is heavily slanted towards highly generic code, even when solving a very specific and concrete problem. I found it to be enough of an impediment that I never contributed.

Rust’s expressive type system is responsible for so much of for so much of how Rust can be low level and high level at the same time. Luckily it’s not quite expressive enough yet to allow monads to enter normal discourse. :-)

Most of the Rust projects whose code I’ve looked at have shied away from complicating things via incidental cleverness. I want the community to keep writing readable-if-verbose code as the type system learns more tricks.

Tooling

It should be dead simple to clone a project, build it, run its tests, and modify its dependencies.

Rust is doing a fantastic job of this: cargo is pretty damned amazing. I’m struggling to think of a crate I’ve worked that didn’t use the default cargo workflow for building, testing, and dependency management.

If you compare the experience to Go, it’s really striking. I actually feel dumb every time I try to contribute to a Go program. I’m sure that if you’re writing Go regularly, GOPATH and the seemingly random mishmash of plain go build, Makefiles, and other stuff are all easy to deal with. I’m usually trying to get in and make a small change, and move on. All that non-uniformity is a borderline showstopper. And that’s not even thinking about needing to modify dependencies! (Incidentally, I’m really excited to see Go making improvements here this year!)

If I had to pick one place I’d like to see get better, it’d be in finding the right crate for the job among all the ones that are out there. The crates.io search is not really all that great. I know that this is being worked on, and I’m excited to see what this looks like in a few months.

Interoperate well with other languages

Rust’s lack of a runtime is a huge strength. It can call into C without worrying about a GIL, or pinning threads or memory, or anything like that. And other languages can call into Rust just as easily as into C.

I want to see Rust really take advantage of this. Let’s avoid rewriting the world in pure Rust, and instead make using existing C and C++ libraries way easier. I’d like to see something like the Rust API guidelines, but focused on FFI wrapping libraries and the question of how to best turn a C API into an ergonomic and idiomatic Rust one. I want there to be a well-supported way of handling C dependencies, including being easily able to depend on system versions.

In the other direction, there are some really interesting projects out there focused on making it easy to write native extensions in Rust. Here I’m thinking of things like Helix for Ruby, Neon for node. It’s just magical that this is possible:

#[macro_use]                                | $ irb                              
extern crate helix;                         | >> require "console"               
                                            | >> Console.log("I'm in your Rust") 
ruby! {                                     | LOG: "I'm in your Rust"            
    class Console {                         |
        def log(string: &str) {             |
            println!("LOG: {:?}", string);  |
        }                                   |
    }                                       |
}                                           |

Rust has a lot to offer as a safe way to enter the world native extensions, whether for performance, low level capabilities, or just fun. I’d love to see these projects get more attention!

ripgrep should be in Debian

Or your favourite non-bleeding-edge distro. This sounds simple but it’s not. There’s a lot of work in working out to fit Rust into the Debian packaging rules. I think a lot of the policy work is done, but the tooling isn’t quite at the point where ripgrep and all its dependencies can be packaged. The six week release cycle that allows Rust to get better so quickly also works against this. As an example, the Firefox nightly PPA I use has had a failing build for the last week because Firefox now needs Rust 1.23.0 to build. I have no idea how to reconcile the Rust release cycle with the annual or semiannual release cycle many distros have, but it’s something the community must figure out.

Don’t fracture the ecosystem

The Rust async story is coming along nicely. But the community needs to take care not to split the crate ecosystem up unnecessarily. For example, historically Python had Twisted and Tornado that were two quite separate worlds with a lot of duplicated functionality. And in Scala, there’s the world of Twitter futures, the world of standard library Futures, and the world of akka. I think Rust is not at too much risk, but it’s something I worry about a bit.

Closing

I think a great way to judge how Rust is doing is to see there be more programs in Rust that aren’t about Rust. ripgrep is an inspiring example of taking the power of Rust, and making a difference to a much wider audience. It’s even included by default in at least a couple of text editors!

I’d love to see more of this in 2018.

Poking around /usr/lib/git-core

Fri, 07 Oct 2016 15:20:01 -0400

A somewhat unknown bit of git trivia is that git rebase is implemented as a ~600 line shell script. A few months ago, I did a full-distro upgrade on my machine. I kept working while it updated, and was surprised when git add --patch failed, because I was using other git commands just fine. Even weirder, plain git add worked! The message I got mentioned Perl and .pm files.

This made me a bit curious to find out what other bits of git were outside the git binary and not in C. Knowing that rebase was a shell script, and that the convention for external subcommands was git-$SUBCOMMAND, I could strace git to find out where that script lived:

$ strace -fe execve git rebase 2>&1 >/dev/null | grep git-rebase
[pid 21289] execve("/usr/lib/git-core/git-rebase", ["git-rebase"], [/* 51 vars */]) = 0

Ok, so /usr/lib/git-core is the place to look! How many executables in there?

$ find /usr/lib/git-core -type f -executable | wc -l
37

How many are shell scripts?

$ file /usr/lib/git-core/* | grep 'shell script' | wc -l
18

Let’s see which ones!

$ file /usr/lib/git-core/* \
> | grep 'shell script' \
> | cut -f1 -d: \
> | xargs basename -a \
> | column                                       
git-gui--askpass        git-filter-branch       git-submodule
git-citool              git-merge-one-file      git-stash
git-gui                 git-merge-resolve       git-rebase
git-difftool--helper    git-request-pull        git-bisect
git-mergetool           git-web--browse         git-quiltimport
git-merge-octopus       git-instaweb            git-subtree

Some of the surprising ones for me: git submodule, and git subtree because they seem ‘important’, as well as git-stash which seems like a ‘built-in’ facility. Of course, they are all important and built-in in that they come with git-core! I think this is mostly a reflection of my programming background: I’d never implement something complex in shell because I just can’t do it quickly enough for it to make any sense.

How about stats on the types of files?

$ find /usr/lib/git-core/ -type f -executable \
> | xargs file \
> | cut -d' ' -f 2- \
> | sed 's/^ *//;s/, interp.*//' \
> | sort \
> | uniq -c
     16 ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked
      3 Perl script text executable
     17 POSIX shell script, ASCII text executable
      1 POSIX shell script, ASCII text executable, with very long lines

We came here curious about git add --patch and Perl, and here we’ve got 3 Perl scripts. What are they?

$ file /usr/lib/git-core/* \
> | grep 'shell script' \
> | cut -f1 -d: \
> | xargs basename -a
git-difftool
git-relink
git-add--interactive

Aha! git-add--interactive sounds very likely. Of course we could have checked this much earlier on:

$ strace -fe execve git add -p 2>&1 >/dev/null | grep add
execve("/usr/bin/git", ["git", "add", "-p"], [/* 49 vars */]) = 0
[pid 22871] execve("/usr/lib/git-core/git", ["git", "add--interactive", "--patch", "--"], [/* 51 vars */]) = 0
[pid 22872] execve("/usr/lib/git-core/git-add--interactive", ["git-add--interactive", "--patch", "--"], [/* 51 vars */]) = 0

Proof. :-)

Spying on Android events without modifying source code

Mon, 03 Oct 2016 10:10:46 -0400

Let’s say you want to intercept events in an Android app, but don’t want to modify your source code. As a simple example, you want to add logging whenever the user clicks on any button in your app. I got a bit curious about whether it was possible so I spent some time last week figuring out how this might work. This was extra fun because I’ve never written and Android app, and it’s been years since I did anything in the Java ecosystem.

What’s hard about intercepting events on Android?

There are a few things going on that make this harder to do on Android than on the web. JavaScript and the DOM let you do all kinds of things at runtime—which is the only time! Adding <script src="blah.js"> can pretty much catch anything you want without modifying any other code.

In Java, once a class is loaded it’s fixed. You can do tricksy things to it via reflection, but that’s all you can do. You can’t get it to call custom code in response to a method call. On the JVM, you can provide a custom class loader and do even trickier things before a class is loaded, including modifying the bytecode. This would allow you to insert the custom code at the start of the method.

But Android doesn’t run the JVM, and it doesn’t run JVM bytecode. Instead the code is translated to DEX, another bytecode format. In current Android versions, even that isn’t what is actually run. Instead the DEX gets compiled further at install time, this time to native code for the device. This makes the modify-at-load-time approach seem somewhere between daunting and impossible.

So, runtime modification and instrumentation isn’t going to work. We can drop back to compile-time instrumentation instead. We could modify either the JVM bytecode before DEX translation, or the DEX bytecode. There’s a lot of support for JVM bytecode modification, and JVM bytecode is easier to modify¹ so we’ll go with that.

As one last thing that makes things more difficult, the base SDK classes are completely off limits. For example, instrumenting the base View class might be a useful thing to do, but we just can’t. They are preloaded in the zygote, which is sort of the primordial goop of a process that all apps launch from. This is to speed up app launch, and to save on some memory by allowing all apps to share those pages.

Let’s modify the bytecode

So now we know we want to modify JVM bytecode, let’s actually do it. There are a bunch of libraries that help with Java bytecode instrumentation. I went with ASM for this experiment because I’d heard of it before.

We only need to intercept one type of event to prove that this approach works. In this post we’ll just look at spying on clicks. There’s a View.OnClickListener interface that any listener implements. We need to check if a class implements that interface, and if so instrument its onClick(View) method².

ASM presents a streaming view of the contents of a class, and calls methods on classes you define as it encounters the bits of a class file. For example, it calls visit with the class name and some other stuff when it starts a new class, and it calls visitMethod on each method.

Because we only have this one-way streaming view of a class, we have to we’ll need to check the list of interfaces at the start and track whether we’re meant to do any work in an instance variable.

@Override
public void visit(
    int version,
    int access,
    String name,
    String signature,
    String superName,
    String[] interfaces) {

  // Call down the class visitor chain.
  cv.visit(version, access, name, signature, superName, interfaces);

  shouldInstrumentOnClick =
      Arrays.asList(interfaces).contains("android/view/View$OnClickListener");
}

Later, when we’re inspecting a method, we check to see if its name is onClick. If so, we make sure to instrument it. The way this works is by returning a modified MethodVisitor from visitMethod.

@Override
public MethodVisitor visitMethod(
    int access, String name, String desc, String signature, String[] exceptions) {
  // Get a method visitor from further down the class visitor chain.
  MethodVisitor mv = cv.visitMethod(access, name, desc, signature, exceptions);

  if (shouldInstrumentOnClick && name.equals("onClick")) {
    // Add our method visitor to the chain.
    mv = new LogClickAdapter(mv);
  }
  return mv;
}

Now the part that actually adds the code! We just want to insert some code at the start of the method, which we can do in visitCode in our MethodVisitor. ASM calls this library just before it goes through any bytecode instructions, so adding code here puts it at the start of the method. This isn’t the nicest thing to look at, as we’re mirroring the bytecode we’re adding:

@Override
public void visitCode() {
  mv.visitCode();

  mv.visitLdcInsn("SPY");
  mv.visitTypeInsn(NEW, "java/lang/StringBuilder");
  mv.visitInsn(DUP);
  mv.visitMethodInsn(INVOKESPECIAL, "java/lang/StringBuilder", "<init>", "()V", false);
  mv.visitLdcInsn("saw click on ");
  mv.visitMethodInsn(
      INVOKEVIRTUAL,
      "java/lang/StringBuilder",
      "append",
      "(Ljava/lang/String;)Ljava/lang/StringBuilder;",
      false);
  mv.visitVarInsn(ALOAD, 1);
  mv.visitMethodInsn(
      INVOKEVIRTUAL,
      "java/lang/StringBuilder",
      "append",
      "(Ljava/lang/Object;)Ljava/lang/StringBuilder;",
      false);
  mv.visitMethodInsn(
      INVOKEVIRTUAL, "java/lang/StringBuilder", "toString", "()Ljava/lang/String;", false);
  mv.visitMethodInsn(
      INVOKESTATIC, "android/util/Log", "d", "(Ljava/lang/String;Ljava/lang/String;)I", false);
  mv.visitInsn(POP);
}

That’s the ASM way to write out the bytecode corresponding to this Java code:

Log.d("SPY", "saw a click on " + view);

Luckily, there’s a nifty utility called ASMifier which outputs the ASM code necessary to generate a class file. Even if you are familiar with JVM bytecode, the ASMified version isn’t going to be fun to write, so that’s really handy.

Making it super easy: adding a plugin to the build

Great! Now we can modify classes to add our custom spying code. But the aim here is to make this require as little modification to the application as possible. Android uses Gradle as its standard app build tool, so we can wrap the whole thing up in a Gradle plugin. This would reduce it down to just adding a couple of lines to the build.gradle file for the app.

This actually turned out to be the most time consuming part! I already knew enough about JVM bytecode and compilation to make the ASM part fairly straightforward. All I had to do was learn enough about the ASM API to do what I needed to do. For the plugin, I had to get my head around some Gradle architecture, and the overall Android build system so that I could decide where to slot the instrumentation in.

In the end, it turned out that the Android build system folks did a great job of providing an API to transform classes before they get compiled to DEX. This is the well-named Transform API. After I defined the transform, the plugin’s body was pretty much just a call to registerTransform on the Android plugin. That handled setting up the Gradle task, its inputs and outputs and whatever else is involved.

What it looks like

This isn’t exactly exciting, perhaps unless you’re me and you just got it to work. But here’s one of the Android samples with and without instrumentation.

Without instrumentation

With instrumentation

As I click on the buttons, it logs lines with D/SPY and details on the thing I clicked on.

The diff

The only differences between the two are in the build.gradle file:

diff --git a/Application/build.gradle b/Application/build.gradle
index 990c615..8d05e53 100644
--- a/Application/build.gradle
+++ b/Application/build.gradle
@@ -1,15 +1,18 @@
 
 buildscript {
     repositories {
+        flatDir dirs: "/home/kamal/projects/asm/build/libs"
         jcenter()
     }
 
     dependencies {
+        classpath 'me:plugin:1'
         classpath 'com.android.tools.build:gradle:2.2.0'
     }
 }
 
 apply plugin: 'com.android.application'
+apply plugin: 'track-plugin'
 
 repositories {
     jcenter()

Exploring unfamiliar ecosystems can be fun

I had a bunch of fun doing this. For one thing, spying on programs is generally a fun activity. Especially if they don’t know you’re doing it. I rarely write programs that respond to user input. While I only tested this on a couple of Android samples, I still saw direct feedback from me clicking on things—even if that feedback was just a log line in the Android Studio console.

It was also rewarding to go from here’s-an-idea-I-have-no-idea-how-to-implement to totally-working-proof-of-concept in just a few days. Even more so because this was an entire code universe I hadn’t really set foot in before. I can write passable Java, but I’ve never touched a Java build system. My experience with Android is limited to having owned two or three Nexus devices over the years.

This makes me curious about when it’s possible to dive into an unfamiliar area and do something non-trivial in a short period of time. I think there has to be a degree of familiarity with at least some of what you’re trying to do. In this instance, I wasn’t stumbling with Java syntax, and I’d done some stuff with JVM bytecode before. The unfamiliar areas were Android, and the build tooling.

At the other end, when I first tried writing Rust, I was trying to do something I didn’t know how to do in a language I didn’t know. That turned out to be quite frustrating, and it was hard to make progress. It ended up delaying me learning Rust by a few months, because I gave up on the project and Rust along with it.

This makes me think there’s an analog of ‘innovation tokens’ at play here. Except instead of optimizing for reliability, we’re optimizing for a balance between challenge and fun. I’m really curious to hear other people’s experiences, so please get in touch!

In the JVM’s stack-based bytecode, inserted code just needs to ensure it doesn’t pop anything off the stack, or leave anything on the stack. Otherwise you can more or less insert any code with net-zero stack change. In the register-based DEX bytecode, some instructions can only work with a specific subset of the available registers, so you’ll almost certainly have to move data around and restore it afterwards. ↩
Tracking clicks on Android turns out to be fairly complicated. There are a bunch of ways a click handler can be registered. I think the most obvious ones are setOnClickListener, and setting the name of a method on the Activity in the the android:onclick XML attribute. The method I’ll outline only catches the setOnClickListener ones. The XML attirbutes are a bit trickier as they result in using an OnClickListener defined in the base SDK’s View class. That’s not something we can instrument. Instead, the way to handle those is to get them from the XML layout files, and instrument the named methods. ↩

Some things I learned about libdwarf

Mon, 25 Jul 2016 18:04:02 -0400

In the last few days, I got drafted / nerd-sniped into helping Julia Evans on her Ruby stacktrace spying program. Specifically, one of the next big things is to read type information out of the Ruby binary so that it’s not hardcoded to work for just one Ruby version.

Going in, I didn’t know much about DWARF, its data model, or how to access it programmatically. This post is a minidump of things I learned about libdwarf, including reasons I’m probably not going to use it.

What DWARF is, the very very short and incomplete version

From the DWARF standard’s homepage:

DWARF is a debugging file format used by many compilers and debuggers to support source level debugging.

Here are some of the things stored in DWARF data:

symbol names, like functions, parameters, variables
a way to go from source file lines to instruction addresses and vice versa, for setting breakpoints and stepping through functions
a way to find out where data for a variable is at a given point in the program
information about the types in a program, including their size and offsets of their fields

I’m sure there’s lots of other stuff in there too. For the task at hand, I’m mostly interested in the last of these. The Ruby stacktrace program currently works by using some struct definitions from header files from Ruby itself. This ties it to a specific Ruby version. We want to make it work on any Ruby version by looking up the struct definitions in the DWARF data.

libdwarf

Linked off the dwarfstd.org site is libdwarf, a C library for reading DWARF. This is great, because the DWARF format looks pretty complicated, and I definitely don’t want to write a reader from scratch! There’s an example program included with libdwarf to show you how to read debug info.

Things I found weird about libdwarf

I started putting together a Rust wrapper for libdwarf, but ended up finding it a little too weird.

NB: I don’t do much C programming beyond working with Linux system libraries, so these might not actually be weird practices for C libraries. But they were definitely weird for me!

Memory management tied to a specific libdwarf session

Some libdwarf functions allocate memory for the data they return. The library provides a special dwarf_dealloc function to free them. So far so good. But the deallocation also takes a handle to the Dwarf_Debug session that the memory was allocated for. This isn’t too bad: a Rust wrapped object could just hold a handle to the Dwarf_Debug session so that dwarf_dealloc can be called in the destructor. There is a small niggle in that the library will deallocate all these objects for you if you call close the Dwarf_Debug with dwarf_finish. This can all be handled in a reasonable way from the Rust side by tying the lifetimes of things to the Dwarf_Debug instance they came from.

Function names increment with library changes

Here are four function names from libdwarf:

dwarf_next_cu_header()
dwarf_next_cu_header_b()
dwarf_next_cu_header_c()
dwarf_next_cu_header_d()

Each one except for dwarf_next_cu_header_d() includes a note like:

It operates exactly like dwarf_next_cu_header_d() but is missing the header_type field. This is kept for compatibility. All code using this should be changed to use dwarf_next_cu_header_d().

I don’t know how to handle compatibility and API evolution in C, but this is definitely the first time I’ve seen something like this! It does mean that code compiled against older versions will continue to work, which is good. But it strikes me as odd overall.

Error handling

Most libdwarf functions take a Dwarf_Error* as their last argument. If you pass a non-null pointer, then error data will stored there. If a null pointer is passed, then the library calls abort() on your behalf instead. Except if you passed an error handler callback when creating the Dwarf_Debug instance: errors will get passed to your callback. It’s unclear to me if you are meant the dwarf_dealloc the error your callback gets passed in that case. If the callback returns (rather than exiting the program or throwing the error as an exception), then I think the whole call looks like nothing went wrong.

Oh and if you do pass a non-null Dwarf_Error*, you are in charge of freeing the error with dwarf_dealloc, passing the Dwarf_Debug instance, as mentioned above. Except if that error happened in the call to dwarf_init, in which case there is no valid Dwarf_Debug instance; in that case, you just free it with free().

Sometimes stateful iteration

There are a bunch of functions for iterating through bits and pieces of DWARF data. For some, there is a next function that takes the current item and gives the next one. Iteration can be started from the beginning by passing in NULL as the current item. But in one case, the iteration state is stored in the Dwarf_Debug instance. You must repeatedly call the same function until it returns NULL: here’s no other way to restart the iteration.

libdw

None of those things were too weird in themselves, but taken together I was wondering about alternatives. The alternative I’m looking at right now is libdw from elfutils. At a short glance, it has a more straightforward API. There’s also a potentially useful higher level “frontend library” called libdwfl. I’ll be taking a closer look this week and putting together a rough binding to see how that goes!

Things you could do with the Rust AST

Thu, 02 Jun 2016 06:47:58 -0400

Lately I’ve been contributing to rustfmt, which is a tool for formatting Rust code. Right now, it formats entire files, which is great if you keep your entire project formatted by rustfmt. If you’re adding code to an existing unrustfmted project, using rustfmt leads to big diffs where most of the changes are just formatting. These can be really annoying to review, so the feature I’m working on is to allow it to format just parts of files. Then you could just reformat the parts you actually changed, making things much nicer for whoever is reviewing the change.

Working on rustfmt has meant getting a bit of familiarity with Rust’s abstract syntax tree (AST), which is how the rustc compiler represents source code. The AST has nodes to represent each expression, statement, function, module, and all the other bits of syntax that make up a Rust program. It’s been interesting and kind of cool to work with, and I wanted to share a bit!

libsyntax and syntex_syntax

The Rust compiler’s own parsing code is in a library called libsyntax. Sadly for programs that aren’t rustc, this is a private library inside the compiler source repository. If you’re willing to use the nightly compiler, there are some incantations that will tell rustc to let you use these internal libraries, but it’s not really encouraged. The Rust folks want to be able to make changes as they keep working on making Rust better, and not have to worry about breaking outside projects.

But someone maintains a copy in Crates.io called syntex_syntax, with modifications to build with stable Rust. It’s part of a suite of tools for generating Rust code, initially created for a serialization framework.

The upshot of this is that we can just add the line

syntex_syntax = "0.33"

to our Cargo.toml file, and then we can parse Rust code in our programs!

A small example of working with the AST

To get an idea of what working with the AST looks like, let’s look at a small example: printing the number of arguments for each function in a file.

syntex_syntax has a Visitor trait you can implement to walk an AST. The default implementation just walks the whole tree and doesn’t do anything else. To write a custom visitor, you can implement the visit function for the node type you’re interested in, and leave the rest out. If you’re more used to Java-esque type systems, you can imagine inheriting from a base DefaultVisitor class, and overriding the functions relating to the syntactic elements you are interested in.

For our example, we override the implementation for visit_fn to store the funciton name and number of arguments in a hashmap. The full code is up on GitHub, but here are the important parts:

struct CountFnArgs<'a> {
    arg_counts: HashMap<String, usize>,
    // The codemap is necessary to go from a `Span` to actual line & column
    // numbers for closures.
    codemap: &'a CodeMap,
}

impl<'v, 'a> Visitor<'v> for CountFnArgs<'a> {
    fn visit_fn(&mut self,
                fn_kind: FnKind<'v>,
                fn_decl: &'v ast::FnDecl,
                block: &'v ast::Block,
                span: Span,
                _id: ast::NodeId) {
        let fn_name = match fn_kind {
            FnKind::ItemFn(id, _, _, _, _, _) |
            FnKind::Method(id, _, _) => id.name.as_str().to_string(),
            FnKind::Closure => format!("<closure at {}>", self.format_span(span)),
        };

        self.arg_counts.insert(fn_name, fn_decl.inputs.len());

        // Continue walking the rest of the funciton so we pick up any functions
        // or closures defined in its body.
        visit::walk_fn(self, fn_kind, fn_decl, block, span);
    }
}

fn count_fn_args(krate: &ast::Crate, codemap: &CodeMap) -> HashMap<String, usize> {
    let mut visitor = CountFnArgs {
        arg_counts: HashMap::new(),
        codemap: codemap,
    };
    visitor.visit_mod(&krate.module, krate.span, 0);

    visitor.arg_counts
}

Rust’s pattern matching really shines when working with ASTs. You can get a small glimpse of it here: we match on the FnKind to see if the function is a method, a free function (ItemFn), or a closure. In the first two cases, we just pull out the function’s name; in the third we get the source location to identify it better.

Here’s the output for our little program running on its own source:

FUNCTION                         ARGS
<closure at 40:19-40:39>         2
count_fn_args                    2
format_loc                       1
format_span                      2
main                             0
parse                            2
visit_fn                         6
visit_mac                        2

Not bad!

Things you can do with an AST

Being able to programmatically parse and work with source code opens up all kinds of cool possibilities. Here are a few.

Format Rust code!

As I kind of mentioned at the start of this post, rustfmt uses the AST to format code. First it parses a file into an AST. Then it walks the tree, recursively printing each node according to a bunch of formatting fules. The design document gives bit more background on the approach, and why rustfmt works the way it does.

Throughly test C APIs

The libc crate is full of function declarations, struct definitions, and constants for working with system libraries. It has a really cool and thorough test suite that makes sure all the items match what’s in the system C headers. It works by parsing the Rust source, and generating C code where it tests for equality of constants and matching type signatures and so on. It then generates some Rust code that calls the generated C code to check everything. So much code generation!

Here’s a snippet of the generated C code to give you an idea of what it’s like:

static int __test_const_O_RDONLY_val = O_RDONLY;
int* __test_const_O_RDONLY(void) {
    return &__test_const_O_RDONLY_val;
}


static int __test_const_O_WRONLY_val = O_WRONLY;
int* __test_const_O_WRONLY(void) {
    return &__test_const_O_WRONLY_val;
}


static int __test_const_O_RDWR_val = O_RDWR;
int* __test_const_O_RDWR(void) {
    return &__test_const_O_RDWR_val;
}

and it goes on and on and on for over 10,000 lines! The result is that the definitions in libc can be pretty safely relied on to match the definitions on the platform you’re working with, which is really great for doing systems programming (one of my favourite things).

Other things you could do!

There are all kinds of other things we can do! For example, it should be possible to build a syntax-aware find-and-replace tool to make some refactorings easier. I’m imagining something like FaceBook’s jscodeshift tool for JavaScript, or their spatch for PHP and some other languages. This would be great for situations like where you just changed the type of the third argument of that function from String to &str (string slice) and now you have to change all the call sites everywhere to add an & to take a reference.

If we’re willing to go a bit deeper into compiler internals—and require nighty Rust to build our project—we can also make tools that use type information. There’s no reason something like Hoogle couldn’t exist for Rust. Hoogle was amazing back when I used to write Haskell. You search by type signature. It’s kind of magical to be able to say “I have a a list and I want an int” and it would have the length function at the top of the results page.

DXR is an example of an exisiting project using type information in Rust. It’s an indexing engine and web interface with fancy cross referencing of source code. As an example, you can browse things around the visit_fn function in rustfmt. It’s really cool! This kind of interwingly source browsing can be really really powerful for understanding a new codebase.

I really love tools like these that make developers’ lives easier. If you have any ideas for Rust, or examples from other languages to “borrow”, please send them my way!

Thanks to Julia Evans for feedback on this post, and for suggesting I write it in the first place!

Segfaults are our friends and teachers

Mon, 25 Apr 2016 16:47:14 -0400

This afternoon, I got a segmentation fault in a Rust program, and was confused. This is Rust, I shouldn’t get segfaults! I quickly checked the couple of places I used unsafe, and they were either calling C functions (and checking the errors), or telling the compiler not to initialize some memory because I was going to write into it unconditionally before reading from it.

So, everything should be fine, right?

Evidently not:

$ cargo build --quiet
$ target/debug/kern-value 1
Segmentation fault

ok, but what is a segfault?

If you’ve written some C, you’ve almost certainly seen a segmentation fault at some point. I spent a long time thinking of it as a ‘thing that happens when you use pointers wrong in C’. That’s mostly true, but a segmentation fault actually has a very specific meaning. You get one when a process tries to access memory in a way that it’s not allowed to, or accessing invalid memory.

Here’s an example of a really simple program that will segfault:

$ cat segfault.c
#include <stdio.h>

int main(void) {
    char *uninitialized;
    printf("%c", *uninitialized);

    return 0;
}
$ gcc segfault.c
$ ./a.out
Segmentation fault

This program segfaults because the entire stack is set to 0 at program start. This includes the uninitialized pointer, which will be a null pointer, pointing to 0x0. When the program tries to print the value it points to, it attempts to read from the null address. That address is invalid, and that’s why we get a segmentation fault.

Segmentation faults are raised by the memory management unit (MMU), which is a piece of hardware! Sections of memory can have different access permissions on them: read, write, and execute. Operating systems use this to isolate processes, and to protect kernel memory from being written to by user code.

If a process tries to write to read-only memory, say, or execute non-executable memory, then the MMU hardware tells the kernel there was a segmentation fault. The kernel delivers the news to the process as a signal: SIGSEGV. By default, that terminates the process.

how we got the segfault

Some context: I’m playing around with reading and writing to pipes with large buffers. I’d previously been setting the pipe size to 1 MB, which is the maximum an unprivileged process can set it to on my machine:

$ cat /proc/sys/fs/pipe-max-size
1048576

My code had a constant PIPE_SIZE for the size of the pipe. I was also using it as the size of a stack-allocated buffer. For my experiments, I’d just changed that constant from 1 MB to 16 MB to test bigger pipes when running as root.

Realising this this set off something in the back of my mind: I vaguely remembered that stacks on Linux default to 8MB. Could the segfault be from going past the stack limit?

I switched to C to verify it:

$ cat -n test-stack.c
     1  #include <stdio.h>
     2
     3  #define BUF_SIZE (16*1024*1024)
     4
     5  int main(void) {
     6      char buf[BUF_SIZE];
     7      printf("%lu\n", sizeof(buf));
     8
     9      return 0;
    10  }
$ gcc -Wall -Wextra -Werror test-stack.c
$ ./a.out
Segmentation fault

Aha! A pretty minimal C program that has the same result. This program allocates a 16 MB array on the stack, and then prints its size. Except it segfaults instead. With the BUF_SIZE set to 8 KB less than 8 MB, everything works. With it set to 8 MB, it segfaults.¹ Hypothesis confirmed!²

seeing which instruction failed

I wanted to get an idea of exactly when the segfault happened, so I ran it under gdb:

$ gcc -g -Wall -Wextra -Werror test-stack.c
$ gdb -silent a.out
Reading symbols from a.out...done.
(gdb) run
Starting program: /tmp/a.out

Program received signal SIGSEGV, Segmentation fault.
0x0000000000400520 in main () at test-stack.c:7
7               printf("%lu\n", sizeof(buf));
(gdb) backtrace
#0  0x0000000000400520 in main () at test-stack.c:7
(gdb) disassemble
Dump of assembler code for function main:
   0x0000000000400506 <+0>:     push   %rbp
   0x0000000000400507 <+1>:     mov    %rsp,%rbp
   0x000000000040050a <+4>:     sub    $0x1000000,%rsp
   0x0000000000400511 <+11>:    mov    $0x1000000,%esi
   0x0000000000400516 <+16>:    mov    $0x4005b4,%edi
   0x000000000040051b <+21>:    mov    $0x0,%eax
=> 0x0000000000400520 <+26>:    callq  0x4003e0 <printf@plt>
   0x0000000000400525 <+31>:    mov    $0x0,%eax
   0x000000000040052a <+36>:    leaveq
   0x000000000040052b <+37>:    retq
End of assembler dump.

So the failure was on the call to printf. 0x1000000 is 16 MB, the size of the buffer. The sub $0x1000000,%rsp instruction is modifying the stack pointer register %rsp to make space for the buffer. When we get to the callq instruction, things exploded!

The callq instruction on x86 first pushes the return address onto the stack, then jumps to the called function. The return address is the current value instruction pointer register %ip, and is where the ret instruction in printf will jump back to. The segfault happens at this point: the CPU attempts to store the return address somewhere outside of the region of memory that’s set aside for the stack, and boom!

luck and guard pages

I think we’re actually kind of lucky to get a segfault. It’s quite possible the bit of memory that’s a few megabytes past our stack was writeable by our process. We’d then go happily corrupting whatever memory was there and probably all kinds of bad things would happen. Segmentation faults are actually our friends!

Modern compilers have stack protection features that help detect overflows. I’m very hazy on the details, but I think they include setting some pages of memory just after the stack to be read only. These are called guard pages. Attempts to write there would result in a segmentation fault. This helps catch stack overflow attempts that involve marching off the end of the stack. However, writing 8 MB after the end of the stack is beyond the guard pages, so we could have ended up in writeable memory.

setting stack size limits in the shell: ulimit

I took this as an opportunity to Learn. I wanted to make this program run without changing its source. At first I thought I could set the stack size with compiler options, but some searching revealed this wasn’t true on Linux. It turns out you can set it with the ulimit shell built-in³:

$ ulimit --soft --stack-size 32768
$ ./a.out
16777216

Success, and without changing the source or the binary! The --stack-size flag says we’re setting the stack size to 32768 KB or 32 MB. The --soft flag says to set the soft limit. I’ll just paste from my man ulimit on my system to explain the difference between hard and soft limits—the -S flag is short for --soft:

A hard limit can only be decreased. Once it is set it cannot be increased; a soft limit may be increased up to the value of the hard limit. If neither -H nor -S is specified, both the soft and hard limits are updated when assigning a new limit value, and the soft limit is used when reporting the current value.

setting stack size limits from inside your program: setrlimit(2)

If you use a lot of stack, it’s not great to force the user to run ulimit before running your program. Instead, you can set it yourself using the setrlimit(2) system call. Here’s proof it works:

$ cat -n test-stack.c
     1  #include <errno.h>
     2  #include <stdio.h>
     3  #include <sys/resource.h>
     4
     5  #define BUF_SIZE (16*1024*1024)
     6
     7  void run(void) {
     8      char buf[BUF_SIZE];
     9      printf("%lu\n", sizeof(buf));
    10  }
    11
    12  int main(void) {
    13      struct rlimit stack_limit = {
    14          .rlim_cur = 2 * BUF_SIZE,
    15          .rlim_max = RLIM_INFINITY,
    16      };
    17
    18      if (setrlimit(RLIMIT_STACK, &stack_limit)) {
    19          perror("setrlimit failed");
    20          return 1;
    21      }
    22
    23      run();
    24
    25      return 0;
    26  }
$ gcc -Wall -Wextra -Werror test-stack.c
$ ulimit -Ss 8192
$ ./a.out
16777216

A fun note: we had to put the buffer in a separate function. Otherwise the stack pointer would be adjusted before the call to setrlimit, and then that call would result in a segfault! This way the stack adjustment happens after calling setrlimit, and everything works out.

Allocating lots of stack space in main isn’t a great idea anyway, since you limit what’s available for the rest of your program. A static buffer would probably be better for this program; for others a heap buffer would work better.

diving into the depths for fun and learning

I had a fun hour or two investigating this. It’s not always possible to take the time in the moment, but it’s really rewarding when you can. I only vaguely knew about the ulimit command, and didn’t know anything at all about setrlimit(2). Since I’ve been programming in languages that output native code a bunch lately, I’ve also wanted to learn more about object file formats, linkers, in-process memory layout, and more. Learning a bit more about stack guard pages is a great step in that direction!

Thanks to Julia Evans for feedback on this post.

Curiously, I found that if I had a buffer size of even 1 byte over (8 MB - 8 KB), I still got the segfault. I’m not yet sure what’s going on there! ↩
Rust really likes to tout its memory safety. This seems like it should mean no segmentation faults, which is why I was confused at getting one. There’s a very specific set of behaviors that aren’t allowed in Rust programs. The Rust program got a segmentation fault because it attempted to write to inaccessible memory, but only through the stack pointer. None of the undefined behaviors disallow this, which I think is why it’s ok for this Rust program to segfault. ↩
ulimit must be a shell built-in and not an executable file because it sets a property of the shell’s process. Other fun examples include setting the working directory (cd), and setting environment variables (set -x in bash). These were some fun things I found out when building a shell, which is a totally worthwhile and fun exercise! ↩

100:10:1 update 7

Mon, 25 Apr 2016 00:00:00 -0400

It’s been a month since the last update, which is definitely too long. Here we go!

projects I worked since last update

cargo fmt-diff

After weeks of being “almost there”, I finally tidied up my initial work into a pull request! The pull request and commit messages are good if you want to see details of what I did. The tl;dr is that there is a --experimental-file-lines flag that allows you to specify FILE:RANGE,RANGE,... to limit formatting to specific lines. It only formats statements for now though!

I had to do a bit of redesigning of my initial approach. I had been assuming each run of the formatting function would affect only specific files. However, by default it formats reachable files in the same crate. This made me change the field on the configuration object from a set of lines for a single file to a map from file names to line sets.

There’s still some more refactoring work to put the line checks in the right place, as well as making it handle more than just statements. I wanted to get feedback on it before going further, since it’s already a +704 −131 change. I was hoping it’d fall in at closer to +400 net. Some of the surplus comes from including a few refactorings that could have been split out, but that I think make most sense in context of this PR.

silly key-value store

I did some experiments which resulted in the OOM killer going on a rampage on my laptop. This was fun!

oh that was cool! I just got the OOM killer to kill stuff because I tried to fill 4GB of pipe buffers on system with 4GB RAM. still running!
— Kamal Marhubi (@kamalmarhubi) April 13, 2016

This is the project I’m expecting to work on most this week. There will probably be a blog post or two as a result, and hopefully an exciting update next time!

projects I didn’t work on

tufcrates
containy-thing

Rust + nix = easier unix systems programming <3

Wed, 13 Apr 2016 00:00:00 -0400

Lately I’m writing lots of Rust, and I’m particularly interested in systems programming on unix. I’ve been using and contributing to a library called nix¹, whose mission is to provide ‘Rust friendly bindings to *nix APIs’.

In this blog post, I hope to convince you that you might want to reach for Rust and nix the next time you need to do some unix systems programming, especially if you aren’t fluent in C. It’s no harder to write, you won’t have to write more code, and it makes it much easier to avoid a few classes of mistakes.

First off, what is systems programming?

The term systems programming can mean all sorts of things, and depends a lot on context and who you’re talking to. For this blog post, here’s my incredibly precise definition:

Systems programming is programming where you spend more time reading man pages than reading the internet.

(As an aside, this means that systems programming is programming you can happily do on the subway if you happen to be in one of those places that doesn’t have connectivity in the tunnels, like New York. I definitely did this when taking the Q train back from the Recurse Center late at night!)

Here are a few examples of things that fall under this definition:

managing processes (eg, fork(2), execve(2), waitpid(2), signal(2), kill(2))
working with files (eg, open(2), ftruncate(2), unlink(2), read(2), write(2))
network programming (eg, socket(2), setsockopt(2), listen(2), sendfile(2))
Linux containers (eg, clone(2), unshare(2), pivot_root(2), mount(2))
interacting with hardware (eg, ioctl(2), mmap(2))

fork(2) and kill(2): an example of how badly things can go

Here’s a fairly innocuous looking C program that uses fork(2) and kill(2) to spawn and kill a process:

#include <signal.h>
#include <unistd.h>

int main(void) {
        pid_t child = fork();
        if (child) {  // in parent
                sleep(5);
                kill(child, SIGKILL);
        } else {  // in child
                for (;;);  // loop until killed
        }

        return 0;
}

This program compiles with no errors or warnings, not even with -Wall -Wextra -Werror. I recommend you don’t run it though, and here’s why. From the POSIX specification for fork(2):

Upon successful completion, fork() shall return 0 to the child process and shall return the process ID of the child process to the parent process. Both processes shall continue to execute from the fork() function. Otherwise, -1 shall be returned to the parent process, no child process shall be created, and errno shall be set to indicate the error.

And from the specification for kill(2):

If pid is -1, sig shall be sent to all processes (excluding an unspecified set of system processes) for which the process has permission to send that signal.

Putting the two together, that program could really ruin our day. If the fork() call fails for some reason², we store -1 in child. Later, we call kill(-1, SIGKILL), which tries to kill all our processes, and most likely hose our login. Not even screen or tmux will save us!³

It’s a pretty scary failure mode, and neither the library nor the language do anything at all to prevent us from having a terrible day.

Why fork and kill go so terribly together

I believe the main issue here is that the C library forces us to try and stick several meanings into one value. For fork(2), the return value is conveying three different things all at once:

whether or not the call succeeded (return value -1)
if it succeeded, whether or not we are in the child (return value 0)
if we are the parent, what the child’s PID is (strictly positive return value)

That’s a lot of information for one poor little pid_t—usually a 32-bit integer—to convey!

In the case of kill(2), the pid parameter conflates several different behaviors. From the POSIX specification again:

If pid is greater than 0, sig shall be sent to the process whose process ID is equal to pid.

If pid is 0, sig shall be sent to all processes (excluding an unspecified set of system processes) whose process group ID is equal to the process group ID of the sender, and for which the process has permission to send a signal.

If pid is -1, sig shall be sent to all processes (excluding an unspecified set of system processes) for which the process has permission to send that signal.

If pid is negative, but not -1, sig shall be sent to all processes (excluding an unspecified set of system processes) whose process group ID is equal to the absolute value of pid, and for which the process has permission to send a signal.

Some extra badness comes from C’s way of treating all non-zero integral values as truthy in conditions, so our if (child) check takes the true branch even when fork() failed and returned -1!

The combination of -1 as failure value from fork(2) and as a special value to kill(2) is unfortunate and makes this example especially bad. But other functions treat a pid value of -1 in a special way too, so even if we didn’t call kill(2) this could still turn out badly. For example, if we called waitpid(2) instead, we’d end up either blocking execution waiting for termination of a child that doesn’t exist, or reaping a child that some other thread is waiting for. While they won’t ruin our system in quite the same way as kill(-1, sig), neither are failure modes that should be so easy to end up in!

How nix and Rust help with the fork / kill problem

Here’s what this example would look like in Rust + nix:

extern crate nix;

use nix::sys::signal::*;
use nix::unistd::*;

fn main() {
    match fork().expect("fork failed") {
        ForkResult::Parent{ child } => {
            sleep(5);
            kill(child, SIGKILL).expect("kill failed");
        }
        ForkResult::Child => {
            loop {}  // until killed
        }
    }
}

We’ll go over it in detail below, but for now we’ll just notice that the structure and length are really similar to the C version. But it’s much safer, and won’t go on a process killing rampage!

The nix wrapper for fork(2) does two things to make it much easier to avoid accidentally killing all our processes. Both use Rust’s enums, which are effectively tagged unions.

Separating the parent and child returns with an enum

For the success case, nix’s fork() makes a really great use of a custom enum. Instead of returning just a plain pid_t, it returns a ForkResult type. The ForkResult enum looks like this:

pub enum ForkResult {
    Parent {
        child: pid_t
    },
    Child
}

We can read this definition as saying that a ForkResult is either Parent or Child. If it’s Parent then it contains a pid_t value named child, while if it’s Child then it contains no value. Rust has a pattern matching syntax for easily checking which variant an enum value is. If we have a variable fork_result, we can pattern match on it like this:

    match fork_result {
        ForkResult::Parent { child } => {
            // stuff to do if we're in the parent
        }
        ForkResult::Child => {
            // stuff do do if we're in the child
        }
    }

Separating the success and failure cases with Result

The other big thing Rust does to help is having a Result type that’s used to represent the return from functions that can fail. Similar to how ForkResult separated the parent and child cases, the built-in Result type separates successes from failures. It looks like this:

#[must_use]
pub enum Result<T, E> {
    Ok(T),
    Err(E),
}

We can read this definition as saying that a Result<T, E> is either Ok to indicate success, or Err to indicate failure. If it’s Ok, it contains a T value, while if it’s Err it contains an E value. For a specific case, you’d set T to be the successful return type, and E to be the type of error that can happen. And the #[must_use] attribute tells the compiler to warn us if we ignore a Result return value.⁴

For nix’s fork() function, the return type is Result<ForkResult, Errno>: our happy case is the ForkResult type we talked about earlier. Our sad case is an Errno value, which is simply an integer the OS uses to tell us why our call failed.

We could match on the return value of fork() directly like this:

    match fork() {
        Ok(ForkResult::Parent { child }) => {
            // stuff to do if we're in the parent
        }
        Ok(ForkResult::Child) => {
            // stuff do do if we're in the child
        }
        Err(errno) => {
            // stuff to do if there was an error
        }
    }

However, Rust has some idioms useful for dealing with Result values that make code a little bit tidier. The one we’ll rely on in this post is expect(). It unwraps the success value from an Ok result, or panics with a given error message if called on an Err result. It’s a handy way to just crash the program with a semi-useful error message when an error happens. That’s pretty much perfect for prototyping, or for quick and dirty programs.

With expect(), our match only has to consider the success cases:

    match fork().expect("fork failed") {
        ForkResult::Parent { child } => {
            // stuff to do if we're in the parent
        }
        ForkResult::Child => {
            // stuff do do if we're in the child
        }
    }

If fork() failed, our program will exit with an error message that looks something like this:

thread '<main>' panicked at 'fork failed: ENOMEM', ../src/libcore/result.rs:709

This tells us what failed, and why. For our tiny program, that’s enough to see what went wrong. For a more complicated program, we could ask for a backtrace by setting RUST_BACKTRACE=1 in the environment.

If you want to find out more about error handling in Rust, The Rust book has a chapter with a fantastic and detailed look at different approaches. I highly recommend reading it!

Join us!

I’ve really been enjoying doing this kind of programming in Rust. So much that I became a maintainer for nix! We’ve been exploring a few ways of using Rust’s features to help make systems programming safer and easier to not mess up.

If this kind of thing interests you too, come help out! We have good first bug label on our issue tracker, as well as a mentored bug label. We’d love to have your input and your help!

Thanks to Ant6n Dubrau, Bryan Newbold, Dan Luu, Julia Evans, and Mathieu Guay-Paquet for feedback on drafts of this post.

Confusingly for some, the library has nothing to do with the Nix package manager, NixOS, or any of the related projects. ↩
There are two main ways fork(2) can fail:
- the system is out of memory
- we’re at our process limit
Either of these can happen, and code should be ready if they do! ↩
I first came across this issue in a post on Rachel by the Bay—which incidentally is a great blog! ↩

Fun fact: when I first wrote the example for this post, I forgot to check the return value from kill(). Woops! But the compiler helpfully warned me that I was ignoring a Result return value:

$ cargo build
   Compiling fork-rs v0.1.0 (file:///home/kamal/projects/talks/2016-03-24-rc/fork-rs)
src/main.rs:13:13: 13:34 warning: unused result which must be used, #[warn(unused_must_use)] on by default
src/main.rs:13             kill(child, SIGKILL);
                           ^~~~~~~~~~~~~~~~~~~~~

↩

100:10:1 update 6

Tue, 22 Mar 2016 14:06:52 -0400

I said last time that I should keep a log; I didn’t. This is from memory + what I could convince git and GitHub to tell me. Writing this also makes me realize that now we’re two months into 100:10:1, I should promote some more projects to round out my list of 10. I’ll try and make some progress there this week!

projects I worked on this week

containy-thing

I made some really minor changes to containy-thing, which I haven’t pushed up to GitHub yet. I switched to using the released version of nix to pick up a couple of changes I’d made to enable mount(2), and make safe wrappers for getuid(2) and getgid(2).

This is kind of cool because it’s popping some yaks off the yak stack: I initially got involved with nix to make those changes. Somehow I wound up making lots more contributions, and becoming a maintainer. It’s a fun circle, because I’m coming back to containy-thing in order to get the CI for nix to run faster.

silly key-value store

I did some initial work on this still-mysterious project. I’m pretty excited for finally starting on it! Watch this space. :-)

projects I didn’t work on

cargo fmt-diff
tufcrates

100:10:1 update 5

Mon, 14 Mar 2016 11:59:21 -0400

I need to find a better way to track my time and know what I worked on. Here’s my recollection:

cargo fmt-diff

I reached a great state on Saturday: I can pass a set of line ranges in on the command line, and statements—but only statements—in that range will be reformatted! I probably spent half an hour trying different sets of arguments just to see it go. This is really exciting!

I still need to tidy up the code and send in the PR, but I’m at point where the infrastructure for this project is pretty much done. Next up is lots of heuristics for figuring out what to do when a syntactic element is partially in a modified line range. To get an idea: if you modify one line in a function body, you don’t want to reformat the whole function. But if you modify one line in a function declaration, you may want to reformat the whole declaration.

tufcrates

I finally printed off the latest paper from The Update Framework folks. I’m about half way through it. The paper makes some excellent extensions for community repositories, which is exactly what Crates.io is. I’ll continue reading the paper this week, and aim to write something about TUF and the problems it solves some time soon.

rust-bisect

Someone on the Rust IRC channel gave me a bit of feedback on rust-bisect. This was a) really exciting because someone was using my project, and b) really useful as I now have some idea of what would be good to add next.

Specifically, the immediate next step is to allow an interactive mode where you can run commands with the nightly version instead of needing to write a script. This would be analogous to the git-bisect without the run subcommand. I may end up changing the CLI to more closely match git-bisect in that case.

There’s are open questions of how to track progress, and how to override he user’s environment in the least intrusive and most friendly way. Current ideas I have:

have rust-bisect fork a shell with the environment set up so that rustc, cargo, et al will run the correct nightly version
- pros:
  - easy to keep track of state in the parent process
  - immediate access in the shell, so they can run rustc and cargo without prefixing
- cons:
  - it’s not obvious how to communicate the outcome up to the parent process from the shell: the obvious things involve terminating the shell and forking a new one, but
  - I would most likely be overriding the user’s shell preference
store state in the filesystem, and require that they prefix commands with rust-bisect run or a similar subcommand
- pros:
  - easier to communicate the outcome, as there’s no environment overriding going on
- cons:
  - it’s unclear where I should store the state. Current directory won’t work, because they might go up and down the tree while testing. Home directory might work, but would force at most one bisect at a time.
something akin to what virtualenv does, providing a script to source that overrides the current shell’s environment.
- pros:
  - it lets the user use their shell, and not prefix any commands
- cons:
  - I don’t like it
  - I’d have to provide scripts for a couple of different shells

I need to give this a bit more thought before implementing.

Projects I didn’t touch

containy-thing
silly key-value store