I’ve been thinking about what I wrote in my last post regarding closures and I am beginning to change my opinion about the correct solution. fn~ just seems so unfortunate. So, besides writing fn~, what are the other options? I just thought I’d write down a few of the other ideas I’ve come up with for later reference. Not saying any of the ideas in this post are good yet.

Just write &mut fn()

Maybe it’s not so bad. It is advertising the possibility that the closure may mutate its environment. This would mean that while &fn() is a valid type, it is a type that does not permit the function to be called, much as &&mut (pointer to a mutable borrowed pointer) does not permit the mutable borrowed pointer to be used.

At first I was thinking that there is also a valid interpretation for &fn, meaning a function that does not mutate the variable in its environment, but then I realize that per the DST proposal any &mut fn could be borrowed to &fn, and so that would not be sound.

Remove everything but borrowed closures

We could just only have borrowed closures. The type would be written fn[:bounds]() or once fn[:bounds](). There’d be no need to notate the kind of environment pointer: it’s always a borrowed pointer. All other uses of closures would be expressed using traits and impls.

Mainly this means that code which spawns traits would get somewhat verbose, because you would need to create a struct or some other type to capture all of the upvars. For larger tasks, this is not a big deal, but for some code it could be rather annoying. I imagine futures in particular would become much more verbose; enough so as to be nearly unusable.

On the upside, there’d be no more confusion about whether a closure copies its environment or not (no, it never does). Closure types would be simpler (no need to worry about sigils). You’d write fn() or once fn() in all but the most esoteric cases. The code to manage closures would become much simpler.

Add a new keyword for what is now called an “owned closure”

This is basically the fn~ solution with another name. Rather than writing fn~ to indicate a closure value that owns its environment, we could write proc (for procedure) or something like that. This avoids the annoying “sigil after the name”, at the cost of a new keyword.

Procedures could probably always be single-shot (that is, once). Almost all use cases for them (futures, tasks, etc) are single-shot, and the others could probably be accommodated with traits instead. But we could also distinguish between a proc and a once proc if we wanted.

Procedures would probably be less interoperable with functions, since the name does not particularly suggest interoperability. For example, I imagine you could not use a proc where a fn is expected. I don’t know of any time that this is actually important.

Using a different name also helps to draw a clear line between between “closures” (which reference the variables in the stack frame that created them) and “procedures” (which copy out from that stack frame). I personally would prefer to designate procedures with a different syntax, e.g., proc(x, y) { ... } in place of |x, y| ..., but this is not necessary (as an aside, I had hoped to write some today about why I think our current use of || to designate any kind of closure is troublesome and should be changed, before I realized that we’d have to address this problem I’m thinking over instead).

More ideas?

Ok, that’s most of the more radical ideas I’ve had so far. I’ll have to keep thinking on it.

I'd like to make an attempt at delivering regular status updates on the Gecko/Firefox build system and related topics. Here we go with the first instance. I'm sure I missed awesomeness. Ping me and I'll add it to the next update.

MozillaBuild Windows build environment updated

Kyle Huey released version 1.7 of our Windows build environment. It contains a newer version of Python and a modern version of Mercurial among other features.

I highly recommend every Windows developer update ASAP. Please note that you will likely encounter Python errors unless you clobber your build.

New submodule and peers

I used my power as module owner to create a submodule of the build config module whose scope is the (largely mechanical) transition of content from Makefile.in to moz.build files. I granted Joey Armstrong and Mike Shal peer status for this module. I would like to eventually see both elevated to build peers of the main build module.

moz.build transition

The following progress has been made:

• Mike Shal has converted variables related to defining XPIDL files in bug 818246.
• Mike Shal converted MODULE in bug 844654.
• Mike Shal converted EXPORTS in bug 846634.
• Joey Armstrong converted xpcshell test manifests in bug 844655.
• Brian O'Keefe converted PROGRAM in bug 862986.
• Mike Shal is about to land conversion of CPPSRCS in bug 864774.

Non-recursive XPIDL generation

In bug 850380 I'm trying to land non-recursive building of XPIDL files. As part of this I'm trying to combine the generation of .xpt and .h for each input .idl file into a single process call because profiling revealed that parsing the IDL consumes most of the CPU time. This shaves a few dozen seconds off of build times.

I have encounterd multiple pymake bugs when developing this patch, which is the primary reason it hasn't landed yet.

WebIDL refactoring

I was looking at my build logs and noticed WebIDL generation was taking longer than I thought it should. I filed bug 861587 to investigate making it faster. While my initial profiling turned out to be wrong, Boris Zbarsky looked into things and discovered that the serialization and deserialization of the parser output was extremely slow. He is currently trying to land a refactor of how WebIDL bindings are handled. The early results look very promising.

I think the bug is a good example of the challenges we face improving the build system, as Boris can surely attest.

Test directory reorganization

Joel Maher is injecting sanity into the naming scheme of test directories in bug 852065.

Manifests for mochitests

Jeff Hammel, Joel Maher, Ted Mielczarek, and I are working out using manifests for mochitests (like xpcshell tests) in bug 852416.

Mach core is now a standalone package

I extracted the mach core to a standalone repository and added it to PyPI.

Mach now categorizes commands in its help output.

Requiring Python 2.7.3

Now that the Windows build environment ships with Python 2.7.4, I've filed bug 870420 to require Python 2.7.3+ to build the tree. We already require Python 2.7.0+. I want to bump the point release because there are many small bug fixes in 2.7.3, especially around Python 3 compatibility.

This is currently blocked on RelEng rolling out 2.7.3 to all the builders.

Eliminating master xpcshell manifest

Now that xpcshell test manifests are defined in moz.build files, we theoretically don't need the master manifest. Joshua Cranmer is working on removing them in bug 869635.

Enabling GTests and dual linking libxul

Benoit Gerard and Mike Hommey are working in bug 844288 to dual link libxul so GTests can eventually be enabled and executed as part of our automation.

This will regress build times since we need to link libxul twice. But, giving C++ developers the ability to write unit tests with a real testing framework is worth it, in my opinion.

ICU landing

ICU was briefly enabled in bug 853301 but then backed out because it broke cross-compiling. It should be on track for enabling in Firefox 24.

Resource monitoring in mozbase

I gave mozbase a class to record system resource usage. I plan to eventually hook this up to the build system so the build system records how long it took to perform key events. This will give us better insight into slow and inefficient parts of the build and will help us track build system speed improvements over time.

Sorted lists in moz.build files

I'm working on requiring lists in moz.build be sorted. Work is happening in bug 863069.

This idea started as a suggestion on the dev-platform list. If anyone has more great ideas, don't hold them back!

Smartmake added to mach

Nicholas Alexander taught mach how to build intelligently by importing some of Josh Matthews' smartmake tool's functionality into the tree.

Source server fixed

Kyle Huey and Ted Mielczarek collaborated to fix the source server.

Auto clobber functionality

Auto clobber functionality was added to the tree. After flirting briefly with on-by-default, we changed it to opt-in. When you encounter it, it will tell you how to enable it.

Faster clobbers on automation

I was looking at build logs and identified we were inefficiently performing clobber.

Massimo Gervasini and Chris AtLee deployed changes to automation to make it more efficient. My measurements showed a Windows try build that took 15 fewer minutes to start - a huge improvement.

Upgrading to Mercurial 2.5.4

RelEng is tracking the global deployment of Mercurial 2.5.4. hg.mozilla.org is currently running 2.0.2 and automation is all over the map. The upgrade should make Mercurial operations faster and more robust across the board.

I'm considering adding code to mach or the build system that prompts the user when her Mercurial is out of date (since an out of date Mercurial can result in a sub-par user experience).

Parallelize reftests

Nathan Froyd is leading an effort to parallelize reftest execution. If he pulls this off, it could shave hours off of the total automation load per checkin. Go Nathan!

Overhaul of MozillaBuild in the works

I am mentoring a pair of interns this summer. I'm still working out the final set of goals, but I'm keen to have one of them overhaul the MozillaBuild Windows development environment. Cross your fingers.

About a month ago, I noticed a tweet from Coop:

Pouring out a little liquor for tinderbox today. Drinking the rest, because, you know, tinderbox.

It linked to a mozilla.dev.planning post describing the plan to end-of-life Tinderbox1.

As one of a handful of people who was required in an employment-capacity to support Tinderbox in production2,3, I can certainly understand the elation at getting rid of the aged continuous integration system. It hasn’t changed much (or seen much maintenance for that matter) since its original open source release fifteen years ago and certainly had plenty of warts4.

Having said that, part of me is sad at the… glee, for lack of a better word, at its demise.

Tinderbox is certainly antiquated by any modern standard, but it should not be forgotten that, having been released in 1998, it is very much the grandfather of continuous integration systems.

It may have “sucked,” but it facilitated a workflow and nurtured an ethos that is not only extremely important, but taken for granted today: namely the notion that individual developers should be “on the hook” when checking in, and have a responsibility to their peers to monitor the build and make sure the tree “stays green.”

It was Tinderbox that was largely responsible for introducing a generation of software engineers to this now-commonplace concept, and helping to get a previous generation of engineers to care about such things. Mozilla was the poster-child user for Tinderbox, but I know of at least VMware and Yahoo who used it years before Hudson/Jenkins and Buildbot existed.

Beyond that, it sports features that those systems TO THIS DAY do not:

1. Tinderbox put its build logic in the clients, and had them report to the server via email; this may seem odd now, but the asynchronous nature of that data flow meant that Tinderbox was surprisingly tolerant to network failures, something Jenkins and (especially) Buildbot both continue to handle horribly.
2. Tinderbox supported JSON output5 that allowed the development of its successor, tbpl, and other tools; it was one of the first CI systems to make its collection of data consumable in such a transparent format.
3. As mentioned above, Tinderbox was modular, separating out the logic of the client from the server, and using a simple API to communicate between them. This meant it was trivial to write a Tinderbox client in whatever language you preferred to write one in, as long as it output something the Tinderbox server expected; this is huge, and a major distinction between today’s systems, which follow a command-and-control model. I personally know Jenkins users who tell horror stories about 40+ minute startup times when tens-of-thousands of clients were involved. Tinderbox, by design, never had this problem.
4. Tinderbox facilitated continuous integration for open source projects in a way that still has yet to be replicated; say what you will about using email as a communication mechanism, but it allowed outside parties to set up and maintain “weird” platforms corporate sponsors of an open source project didn’t feel like investing in6, and yet still provide those CI results to the community in a highly-visible, inclusive way7. Good luck getting those Jenkins and Buildbot ports open through the corporate firewall today. And for all the hype about Git’s decentralization, there isn’t a continuous integration tool today built around a decentralized model like Tinderbox was.

It’s easy to hate on tools like Tinderbox when you purposefully ignore the context in which they were created8: Richard Stallman’s venerable GCC compiler was “a pile of crap”… if you forget it was the most widely used compiler before the EGCS fork, over a decade before “Clang” was more than a Klingon General9. “CVS sucks!” … yeah, compared to Git, Subversion, and Perforce. But not compared to RCS or Microsoft Source Safe. “Shell scripts are old-school garbage!”… except if you were trying to automate something in the 80s and early 90s, and didn’t want to have to learn about C pointers to do it.

It’s unfortunate that it seems unlikely Tinderbox will ever get the respect it truly deserves as one of the founding fathers of this idea of continuous integration that most of us take for granted today.

But perhaps that’s merely another example of Alan Kay’s indictment of our industry for its “disdain of history.”

In any event, to paraphrase Apollo 13 Mission control: “Farewell Tinderbox… and we thank you.”

_______________
1 And the requisite bug, which you can conveniently find find by searching for keyword “tinderbox-death”?
2 Coop also holds this distinction?
3 And as one who also has a number of (now embarrassing) checkins into that code base?
4 Including some pretty bad ones?
5 And possibly others?
6 Firefox on AIX, anyone??
7 It’s telling that the last few projects still using Mozilla’s Tinderbox instance fall pretty squarely into this category: Camino and Bugzilla?
8 Especially when they’re (merely) broadly used for fifteen years, but not actively maintained?
9 Not really?

A key step in the documentation process is the review. The Mozilla Developer Network (MDN) developer documentation team aspires to perform two types of review on each article on the wiki:

A technical review involves having someone with technical expertise in the article’s subject area read the article and ensure that it’s factually and technically accurate. This means making sure our explanations of the subject matter are right and make sense to the developers, as well as ensuring that any code samples are implemented correctly.

An editorial review is where someone reads the article and corrects any typos, grammatical mistakes, or stylistic or layout problems they can find.

When writing an article (or editing an existing article), there are checkboxes that let you request each of these types of review on your article; indeed, both of these flags are enabled by default for any newly-created page. Yellow banners are displayed at the top of the page to let readers know that these flags are set, as appropriate.

In the past, the only way to remove the review requests (for example, if you’ve read the article and didn’t see any factual errors) was to click the Edit button, scroll to the checkbox for the editorial review, toggle it off, and save the article.

Now there’s a better way!

Now, on pages with one or both of these review flags set, you’ll see a new “Quick review” box at the bottom of the article. This box looks like this:

The MDN Quick Review box.

All you have to do if you’ve completed a review is to turn on the appropriate checkbox(es) and click “Confirm Reviews.” The review is recorded and you have our gratitude for your help!

If you’d like to actually hunt down and take care of articles in need of review, there’s an easy way to find them:

• Articles that need editorial review
• Articles that need technical review

I’m hopeful that this streamlined approach to reviews will help encourage subject-matter experts to review content, and will help casual users of MDN get involved by reviewing content as well. We have some improvements planned for future updates to MDN that will make this even better, but this is a great start, and I’m grateful to our development team for putting this together!

I hold a lot of context in my head when it comes to the future of Mozilla's build system and the interaction with it. I wanted to perform a brain dump of sorts so people have an idea of where I'm coming from when I inevitably propose radical changes.

The sad state of build system interaction and the history of mach

I believe that Mozilla's build system has had a poor developer experience for as long as there has been a Mozilla build system. Getting started with Firefox development was a rite of passage. It required following (often out-of-date) directions on MDN. It required finding pages through MDN search or asking other people for info over IRC. It was the kind of process that turned away potential contributors because it was just too damn hard.

mach - while born out of my initial efforts to radically change the build system proper - morphed into a generic command dispatching framework by the time it landed in mozilla-central. It has one overarching purpose: provide a single gateway point for performing common developer tasks (such as building the tree and running tests). The concept was nothing new - individual developers had long coded up scripts and tools to streamline workflows. Some even published these for others to use. What set mach apart was a unified interface for these commands (the mach script in the top directory of a checkout) and that these productivity gains were in the tree and thus easily discoverable and usable by everybody without significant effort (just run mach help).

While mach doesn't yet satisfy everyone's needs, it's slowly growing new features and making developers' lives easier with every one. All of this is happening despite that there is not a single person tasked with working on mach full time. Until a few months ago, mach was largely my work. Recently, Matt Brubeck has been contributing a flurry of enhancements - thanks Matt! Ehsan Akhgari and Nicholas Alexander have contributed a few commands as well! There are also a few people with a single command to their name. This is fulfilling my original vision of facilitating developers to scratch their own itches by contributing mach commands.

I've noticed more people referencing mach in IRC channels. And, more people get angry when a mach command breaks or changes behavior. So, I consider the mach experiment a success. Is it perfect, no. If it's not good enough for you, please file a bug and/or code up a patch. If nothing else, please tell me: I love to know about everyone's subtle requirements so I can keep them in mind when refactoring the build system and hacking on mach.

The object directory is a black box

One of the ideas I'm trying to advance is that the object directory should be considered a black box for the majority of developers. In my ideal world, developers don't need to look inside the object directory. Instead, they interact with it through condoned and supported tools (like mach).

I say this for a few reasons. First, as the build config module owner I would like the ability to massively refactor the internals of the object directory without disrupting workflows. If people are interacting directly with the object directory, I get significant push back if things change. This inevitably holds back much-needed improvements and triggers resentment towards me, build peers, and the build system. Not a good situation. Whereas if people are indirectly interacting with the object directory, we simply need to maintain a consistent interface (like mach) and nobody should care if things change.

Second, I believe that the methods used when directly interacting with the object directory are often sub-par compared with going through a more intelligent tool and that productivity suffers as a result. For example, when you type make in inside the object directory you need to know to pass -j8, use make vs pymake, and that you also need to build toolkit/library, etc. Also, by invoking make directly, you bypass other handy features, such as automatic compiler warning aggregation (which only happens if you invoke the build system through mach). If you go through a tool like mach, you should automatically get the most ideal experience possible.

In order for this vision to be realized, we need massive improvements to tools like mach to cover the missing workflows that still require direct object directory interaction. We also need people to start using mach. I think increased mach usage comes after mach has established itself as obviously superior to the alternatives (I already believe it offers this for tasks like running tests).

I don't want to force mach upon people but...

Nobody likes when they are forced to change a process that has been familiar for years. Developers especially. I get it. That's why I've always attempted to position mach as an alternative to existing workflows. If you don't like mach, you can always fall back to the previous workflow. Or, you can improve mach (patches more than welcome!). Having gone down the please-use-this-tool-it's-better road before at other organizations, I strongly believe that the best method to incur adoption of a new tool is to gradually sway people through obvious superiority and praise (as opposed to a mandate to switch). I've been trying this approach with mach.

Lately, more and more people have been saying things like we should have the build infrastructure build through mach instead of client.mk and why do we need testsuite-targets.mk when we have mach commands. While I personally feel that client.mk and testsuite-targets.mk are antiquated as a developer-facing interface compared to mach, I'm reluctant to eliminate them because I don't like forcing change on others. That being said, there are compelling reasons to eliminate or at least refactor how they work.

Let's take testsuite-targets.mk as an example. This is the make file that provides the targets to run tests (like make xpcshell-test and make mochitest-browser-chrome). What's interesting about this file is that it's only used in local builds: our automation infrastructure does not use testsuite-targets.mk! Instead, mozharness and the old buildbot configs manually build up the command used to invoke the test harnesses. Initially, the mach commands for running tests simply invoked make targets defined in testsuite-targets.mk. Lately, we've been converting the mach commands to invoke the Python test runners directly. I'd argue that the logic for invoke the test runner only needs to live in one place in the tree. Furthermore as a build module peer, I have little desire to support multiple implementations. Especially considering how fragile they can be.

I think we're trending towards an outcome where mach (or the code behind mach commands) transitions into the authoratitive invocation method and legacy interfaces like client.mk and testsuite-targets.mk are reimplemented to either call mach commands or the same routine that powers them. Hopefully this will be completely transparent to developers.

The future of mozconfigs and environment configuration

mozconfig files are shell scripts used to define variables consumed by the build system. They are the only officially supported mechanism for configuring how the build system works.

I'd argue mozconfig files are a mediocre solution at best. First, there's the issue of mozconfig statements that don't actually do anything. I've seen no-op mozconfig content cargo culted into the in-tree mozconfigs (used for the builder configurations)! Oops. Second, doing things in mozconfig files is just awkward. Defining the object directory requires mk_add_options MOZ_OBJDIR=some-path. What's mk_add_options? If some-path is relative, what is it relative to? While certainly addressable, the documentation on how mozconfig files work is not terrific and fails to explain many pitfalls. Even with proper documentation, there's still the issue of the file format allowing no-op variable assignments to persist.

I'm very tempted to reinvent build configuration as something not mozconfigs. What exactly, I don't know. mach has support for ini-like configuration files. We could certainly have mach and the build system pull configs from the same file.

I'm not sure what's going to happen here. But deprecating mozconfig files as they are today is part of many of the options.

Handling multiple mozconfig files

A lot of developers only have a single mozconfig file (per source tree at least). For these developers, life is easy. You simply install your mozconfig in one of the default locations and it's automagically used when you use mach or client.mk. Easy peasy.

I'm not sure what the relative numbers are, but many developers maintain multiple mozconfig files per source tree. e.g. they'll have one mozconfig to build desktop Firefox and another one for Android. They may have debug variations of each.

Some developers even have a single mozconfig file but leverage the fact that mozconfig files are shell scripts and have their mozconfig dynamically do things depending on the current working directory, value of an environment variable, etc.

I've also seen wrapper scripts that glorify setting environment variables, changing directory, etc and invoke a command.

I've been thinking a lot about providing a common and well-supported solution for switching between active build configurations. Installing mach on $PATH goes a long way to facilitate this. If you are in an object directory, the mozconfig used when that object directory was created is automatically applied. Simple enough. However, I want people to start treating object directories as black boxes. So, I'd rather not see people have their shell inside the object directory.

Whenever I think about solutions, I keep arriving at a virtualenv-like solution. Developers would potentially need to activate a Mozilla build environment (similar to how Windows developers need to launch MozillaBuild). Inside this environment, the shell prompt would contain the name of the current build configuration. Users could switch between configurations using mach switch or some other magic command on the $PATH.

Truth be told, I'm skeptical if people would find this useful. I'm not sure it's that much better than exporting the MOZCONFIG environment variable to define the active config. This one requires more thought.

The integration between the build environment and Python

We use Python extensively in the build system and for common developer tasks. mach is written in Python. moz.build processing is implemented in Python. Most of the test harnesses are written in Python.

Doing practically anything in the tree requires a Python interpreter that knows about all the Python code in the tree and how to load it.

Currently, we have two very similar Python environments. One is a virtualenv created while running configure at the beginning of a build. The other is essentially a cheap knock-off that mach creates when it is launched.

At some point I'd like to consolidate these Python environments. From any Python process we should have a way to automatically bootstrap/activate into a well-defined Python environment. This certainly sounds like establishing a unified Python virtualenv used by both the build system and mach.

Unfortunately, things aren't straightforward. The virtualenv today is constructed in the object directory. How do we determine the current object directory? By loading the mozconfig file. How do we do that? Well, if you are mach, we use Python. And, how does mach know where to find the code to load the mozconfig file? You can see the dilemma here.

A related issue is that of portable build environments. Currently, a lot of our automation recreates the build system's virtualenv from its own configuration (not that from the source tree). This has and will continue to bite us. We'd really like to package up the virtualenv (or at least its config) with tests so there is no potential for discrepancy.

The inner workings of how we integrate with Python should be invisible to most developers. But, I figured I'd capture it here because it's an annoying problem. And, it's also related to an activated build environment. What if we required all developers to activate their shell with a Mozilla build environment (like we do on Windows)? Not only would this solve Python issues, but it would also facilitate simpler config switching (outlined above). Hmmm...

Direct interaction with the build system considered harmful

Ever since there was a build system developers have been typing make (or make.py) to build the tree. One of the goals of the transition to moz.build files is to facilitate building the tree with Tup. make will do nothing when you're not using Makefiles! Another goal of the moz.build transition is to start derecursifying the make build system such that we build things in parallel. It's likely we'll produce monolithic make files and then process all targets for a related class IDLs, C++ compilation, etc in one invocation of make. So, uh, what happens during a partial tree build? If a .cpp file from /dom/src/storage is being handled by a monolithic make file invoked by the Makefile at the top of the tree, how does a partial tree build pick that up? Does it build just that target or every target in the monolithic/non-recursive make file?

Unless the build peers go out of our way to install redundant targets in leaf Makefiles, directly invoking make from a subdirectory of the tree won't do what it's done for years.

As I said above, I'm sympathetic to forced changes in procedure, so it's likely we'll provide backwards-compatibile behavior. But, I'd prefer to not do it. I'd first prefer partial-tree builds are not necessary and a full tree build finishes quickly. But, we're not going to get there for a bit. As an alternative, I'll take people building through mach build. That way, we have an easily extensible interface on which to build partial tree logic. We saw this recently when dumbmake/smartmake landed. And, going through mach also reinforces my ideal that the object directory is a black box.

Semi-persistent state

Currently, most state as it pertains to a checkout or build is in the object directory. This is fine for artifacts from the build system. However, there is a whole class of state that arguably shouldn't be in the object directory. Specifically, it shouldn't be clobbered when you rebuild. This includes logs from previous builds, the warnings database, previously failing tests, etc. The list is only going to grow over time.

I'd like to establish a location for semi-persistant state related to the tree and builds. Perhaps we change the clobber logic to ignore a specific directory. Perhaps we start storing things in the user's home directory. Perhaps we could establish a second object directory named the state directory? How would this interact with build environments?

This will probably sit on the backburner until there is a compelling use case for it.

The battle against C++

Compiling C++ consumes the bulk of our build time. Anything we can do to speed up C++ compilation will work wonders for our build times.

I'm optimistic things like precompiled headers and compiling multiple .cpp files with a single process invocation will drastically decrease build times. However, no matter how much work we put in to make C++ compilation faster, we still have a giant issue: dependency hell.

As shown in my build system presentation a few months back, we have dozens of header files included by hundreds if not thousands of C++ files. If you change one file: you invalidate build dependencies and trigger a rebuild. This is why whenever files like mozilla-config.h change you are essentially confronted with a full rebuild. ccache may help if you are lucky. But, I fear that as long as headers proliferate the way they do, there is little the build system by itself can do.

My attitude towards this is to wait and see what we can get out of precompiled headers and the like. Maybe that makes it good enough. If not, I'll likely be making a lot of noise at Platform meetings requesting that C++ gurus brainstorm on a solution for reducing header proliferation.

Conclusion

Belive it or not, these are only some of the topics floating around in my head! But I've probably managed to bore everyone enough so I'll call it a day.

I'm always interested in opinions and ideas, especially if they are different from mine. I encourage you to leave a comment if you have something to say.

Bug ID Summary 581352 Console HTTP output should make JSON viewable as objects 586125 “””Copy”” “”Select All”” etc. popup menu item in the Web Console displays keyboard shortcuts 587757 Implement Browser Console 638953 Cannot copy keys or values in web console object inspector 640225 Make Property Panels Editable 688400 Web Console throws a TypeError when trying to display objects with no toString method 688401 Web Console displays “”[null]“” for [Object.create(null)] 711846 Intermittent browser_webconsole_bug_595934_message_categories.js | Test timed out (mostly on WinXP PGO) 725771 xpath use in console returns null elements 744741 console.dir tree is empty with cached messages 748851 Web Developer Console incorrectly displays escaped hex/unicode characters 757897 All code in the web console is considered to be at line 1 774365 Web Console setTimeout can’t find declared functions by name 783499 Web Console should use the debugger API 808370 Use the VariablesView in webconsole 809424 Intermittent browser_webconsole_bug_611795.js | The unknown CSS property warning is displayed only once – Got 1, expected 2 831400 Intermittent devtools/webconsole/test/browser_webconsole_bug_764572_output_open_url.js| Timed out while waiting for: complete to initialize the opening tab 836720 “”InternalError: too much recursion”” not correctly caught 840093 Text in object inspector can’t be copied 841129 Intermittent browser_repeated_messages_accuracy.js | two repeats for the second css warning – Got 1, expected 2 843287 UI improvements for the variables view and sidebar 847349 Variables View should have left arrow do collapse and navigate in separate steps 853568 Intermittentbrowser_webconsole_bug_599725_response_headers.js | we do not have the Content-Type header | Content-Length != 60 – Didn’t expect 60, but got it 855058 Intermittent browser/dom/tests/browser/browser_ConsoleAPITests.js | timer duration is positive 857441 Autocompletion popup is very jumpy while typing text 859170 Trying to select [...] for a very long message printed to the web console will hang Firefox 859818 ConsoleAPIStorage.jsm tries to delete var-declared _consoleStorage 859858 Intermittent browser_bug664688_sandbox_update_after_navigation.js | Timed out while waiting for: window.location.href result is displayed after goBack(), browser_repeated_messages_accuracy.js | Timed out while waiting for: messages displayed 862024 Warning about replaced window.console API shows when content scripts do not change the object 863943 Intermittentbrowser_webconsole_bug_602572_log_bodies_checkbox.js | Test timed out, browser_webconsole_bug_603750_websocket.js | uncaught exception – ReferenceError: ok is not defined 865288 console.log with different object logged from same line number are grouped together 865792 Firefox for Android remote console doesn’t evaluate JS expressions 865871 Object inspector can’t be closed 677930 Style Inspector: make URLs clickable 777681 [rule view] Abbreviate long CSS file names source links 777877 Switch CssRuleView.jsm back to WeakMaps 854948 GCLI context.environment.contentDocument points to doc from when GCLI first opened 855502 we need a proper design for the new checkboxes 855520 “”No element selected”” in the inspector sidebar could use some margins 861908 Destroying Inspector without destroying Toolbox fails. 866381 Inspector shouldn’t break if target is a chrome document 679364 Use either content/devtools/* or content/browser/devtools/* for aliases 851231 Output console.jsm API calls to the browser console 855544 Implement a network monitor 855763 [ResponsiveMode] Intermittent browser_responsiveuiaddcustompreset.js | Test timed out, | Found a tab after previous test timed out: data:text/html,foo,browser_Services.js | Services.prompt is an nsIPromptService 855914 Start using the jetpack loader in devtools 859089 [netmonitor] Under very constrained sizes text in the timeline view is sometimes not rendered 859221 [netmonitor] Sizes are displayed in KB not kb 861197 chrome.manifest files created by the devtools loader should be ignored 723923 Debugger ‘breakpoint list’ GCLI command should have extra nice output 792815 Update promise usage in gcli commands 831720 GCLI needs an appcache command 861192 gcli addon command shows [object Object] instead of a list of addons 865680 devtools.gcli.allowSet is in firefox.js twice 869233 TEST-UNEXPECTED-FAIL | browser_cmd_appcache_valid.js | html output for appcache validate should match successfully. Actual textContent: “”Manifest URI…”” 739031 Check for unused strings not used in translation (e.g. scoped.label) and remove them 740541 Style sheets in iframes are not listed in the style editor 816967 Remotable Style Editor 866264 Intermittent devtools/styleeditor/test/browser_styleeditor_new.js | new editor shows 1 rule after modification – Got 2, expected 1 772119 expose source mapped sources over the remote debugging protocol 818134 Allow multiple debuggers in toolboxes to debug separate tabs 832231 After a reload, breakpoints require multiple resumes to allow execution to continue 837723 [jsdbg2] Debugger should provide a way to get Debugger.Object referents directly 840292 The debugger’s xpcshell tests should use listTabs/attachTab instead of listContexts 843187 Variables view: going down through the properties via keyboard is really broken 849069 relative source map URLs should be resolved according to the spec’s rules 849071 Create some kind of chrome to turn source mapping on/off in the debugger 850738 Add a crap load of telemetry tests 851836 breakpoints[aLocation.line] is undefined after bug 820012 853787 Report errors in nested event loops in xpcshell tests 856232 initialize globalActorFactories upon declaring DebuggerServer 859181 Variables view closes immediately on first open 860349 Intermittent browser_dbg_createChrome.js | application timed out after 330 seconds with no output 861783 Removing watch expression of large object not possible via mouse 861817 Stop using Ctrl+Shift+O shortcut for function searching. 861852 Telemetry data from the remote protocol always appear in the remote buckets 862142 remote debugger doesn’t find any sources for simulator on nightly 862186 remove unused FileUtils import in DebuggerUI.jsm 862360 Use the exported Promise.all instead of reimplementing it 863571 Enable source map xpcshell tests on B2G 864802 The _store on each variables view instance is redundant 864900 The listTabs protocol request can fail if made immediately after a tab is closed 865073 remote debugging protocol: handlers should systematically catch and log exceptions 865328 JS debugger: clean up DebuggerClient.prototype.close 865509 JS debugger: UnsolicitedPauses in dbg-client.jsm should mention ‘exception’ 866278 remote debugging protocol: number packets when dumping 867481 Remote debugging connection screen is broken 839515 Tiny fonts in the remote connection screen 849500 navigate and will-navigate events for remoted targets carry payload that is incompatible with the non-remoted case 851546 Create an Options panel for the toolbox 857082 TabTarget.makeRemote doesn’t need any arguments 859569 Target.makeRemote should attachTab 860101 payload._navPayload is a Window in ‘will-navigate’ event handler 862294 Sidebar should be responsive in docked Toolbox. 862363 Sync the killswitch upon (un)registering of a tool and respect it in Options Panel 862398 Maintain the order of tools in the tab and the options panel list 863600 Intermittent browser_toolbox_options.js on Windows PGO builds | Test timed out 851381 Make the Scratchpad evaluate asynchronously 865803 Unable to save Scratchpads, Style Editor, about:memory, … on OS X 852777 Make Tilt more extensible 862435 Tilt doesn’t open if the Inspector was opened beforehand 855244 Profiler hangs after pressing “”Stop”” if it was started in multiple profiles in different tabs 856797 Typo in “”profilerNotStarted”” in gclicommands.properties 858759 profiler.css in browser/devtools/ should actually be in browser/themes instead 859062 Hide right arrow in the SideMenuWidget 859085 Add tooltips to all labels in the UI that may have cropped values 859308 Hide Blocked in Timings tab until we know what that’s doing 859968 JSONP requests are not parsed into a tree 859980 JSON request not parsed 860055 SideMenuWidget shouldn’t scroll back to the selected item when new requests are coming in 860175 On netmonitor examining the specific details of a network action leads to bad overlapping display 860232 Function callbacks in jsonp responses should be displayed in the Response tab 861487 Sizes are displayed with non-localized decimal separator 862198 Intermittent browser_net_json-long.js | Test timed out, found a tab 864176 Details pane should remember its height when docked on the right 864519 Several Netmonitor tests fail when ECMAScript Internationalization API is enabled 866091 Intermittentdevtools/netmonitor/test/browser_net_prefs-reload.js | Test timed out | Found a tab after previous test timed out: http://example.com/browser/browser/devtools/netmonitor/test/html_simple-test-page.html and tons of fallout

Source: Firefox 22 – 23.

Well, we were “jamun”. This past week we shut off the Jamun project branch of Australis and moved our focus to the UX branch.

What does all this mean?

In a nutshell, this means that the Australis customization rewrite has progressed enough to the point that we feel it is ready to start getting much broader testing. We’re in the final stretch of the project, and we want our changes to reach testers at a faster pace.

Here’s where we stand today:
* Most customization workflows are possible. Not all UI migrations are complete.
* Most polish on the edges isn’t there. It does however look pretty close on Windows and Mac now.
* Some final graphics are missing, but they’re not far away.

Wait, wait, what are we talking about here?

Ah, okay, I’ll take a step back. A while back, in fact, a looooong time ago… there was a presentation from the Firefox UX team about a new browser UI refresh and rewrite of our customization workflows. It turns out that many users don’t know that Firefox is customizable. Some users accidentally customize their browser and don’t know what went wrong. Then, there’s this super-tiny super-expert user group that has figured out how to customize Firefox and they *adore* it.

So, what are we doing about it?

Well, the first thing that we’re doing is making entering and exiting customization of Firefox much easier. No longer will a user have to right-click on a special portion of a toolbar and choose “Customize…”. This was way too hard to find for the vast majority of users. We’ve left that same entry point there, but we’ve also created a very visible “Customize” button.

Sounds good, where is this Customize button though?

Great question! Another goal of Australis is to unify the user experience between Windows, OS X, and Linux. On Windows and Linux, Firefox has an “Application Menu” in the top-left corner of the browser. We’ve moved this menu to the right-side of the navigation toolbar and it will now be visible on all three platforms. We’ve also been hard at work trying to make this menu easy to use and navigate. The Customize button is located at the bottom of this menu.

Another really cool thing about this menu is that it will be customizable. When you enter Customization mode, you’ll be able to add, remove, and rearrange items in the menu as well as items on the toolbars.

Here’s a screenshot of what the Customization mode looks like today on Windows:

As I mentioned earlier, all of this is still very much a “work-in-progress”, so it’s expected that people will find bugs and rough edges. If you’d like to play with it today, you can download the UX Nightly builds and give it a run. The UX Nightly builds will update daily with new changes to the customization.

Please let us know about any bugs that you find by filing a bug in Bugzilla in the Firefox::Theme or Firefox::Toolbars component (and mark the bug as blocking bug 770135). If you don’t feel comfortable doing the above, then just leave a comment on this blog post and I or someone else will file the bug for you.

Tagged: australis, firefox, mozilla, planet-mozilla, usability

I realized today that there is an unfortunate interaction between the proposal for dynamically sized types and closure types. In particular, in the case of the recurring closure, I described the soundness issues that arise in our language when closures are able to recurse.

My solution for this was to make the type system treat a &fn() value the same way it treats &mut T pointers: they would be non-copyable, and when you invoke them, that would be effectively like a “mutable borrow”, meaning that for the duration of the call the original value would become inaccessible. So in short the type system would guarantee that when you call a closure, that same closure is not accessible from any other path in the system, just as we now guarantee that when you mutate a value, that same value is not accessible from any other path in the system.

This is all well and good, and I think this treatment would be largely invisible to the user under common access patterns. However, it does not play well with the proposal for dynamically sized types, because under this proposal all things written &T must behave the same, no matter what T is. This is in fact the whole point of the proposal! But here I want to treat &fn specially.

I’ve been pondering various solutions this morning. I have come up with two possible avenues:

1. Instead of writing &fn() you could write &mut fn(). This is perhaps the “principled” solution, but I consider it rather a non-starter. Writing &fn() for a closure is…tolerable, but &mut fn() is not. It’s verbose and it seems sort of nonsensical (although there is some logic to it, when you consider that calls to the function may mutate the environment and so forth).

2. We go back to the older notation and move sigils for closures after the fn. This actually has some notational perks. For example, rather than writing &fn() we can just write fn() (if there is no sigil, we can default to &). On the minus side, a sendable closure would be written fn~()—but, then again, under the dynamically sized types proposal, sendable closures were going to be written ~fn:Owned(), so is fn~() really so bad?

More details after the fold.

OK, let’s dig into the details a bit more. As anyone who has been following my blog posts probably knows by now, there are many, many use cases for closures. I want to dive into the use cases that are on my mind and elaborate on them. I also want to take this case to write up a bit more thoroughly how I think closures should work, including a few unrelated issues.

Syntax and use cases

Here is a list of use cases to be accommodated:

1. “Higher-order functions”: simple functions like map, fold and so forth. By far the most common use case.
2. “Once functions”: functions that can only execute once. This means that they can move values out of their environment.
3. “Sendable functions”: functions that can be sent between tasks. This means that they only close over “sendable” values (no garbage-collected data or borrowed pointers).
4. “Sendable once functions”: sendable functions that can only execute once. This is what a task body will be.
5. “Const functions”: functions that do not close over mutable state. We don’t make much use of this yet, but I plan to do so in order to achieve lightweight fork-join parallelism a la PJS.

The use cases above seem to me to be the “bread and butter” cases that will arise frequently. I will go over the syntax and give an example for each of those use cases shortly. Interestingly, I think that all of them actually read reasonably well if the sigils are moved after the fn keyword, and in some cases the examples read much better.

However, there are two additional use cases that I have considered in the past which I left out. These use cases become significantly harder to read under the new proposal (though they were always hard to read). Interestingly, I realized while writing this blog post that I think these use cases are no longer terribly important, since both of them can be expressed equally well using objects instead of closures, as I will explain shortly. The two use cases are:

1. “Sendable const functions”: functions that can be sent between tasks and do not close over mutable state. You could safely share such functions between tasks in an ARC (atomically referenced counted container) and execute them multiple times in parallel.
2. “Combinators”: combinator libraries create and return closures that closure over their arguments, which may include borrowed values.

Higher-order functions

Here is an example of a simple higher-order function (with the closure type highlighted):

impl<T:Sized> for [T] { pub fn map<U:Sized>(f: fn(&T) -> U) -> ~[U] { ... } // ^~~~~~~~~~~ }

For contrast, this is &fn(&T) -> U today.

Once functions

Here is an example of a higher-order function that executes at most once:

impl<T:Sized> for Option<T> { pub fn each(f: once fn(&T) -> bool) -> bool { ... } // ^~~~~~~~~~~~~~~~~~~ } }

For contrast, this is &once fn(&T) -> U today.

Sendable functions and sendable once functions

Here is an example of a sendable once function:

fn spawn(f: once fn~()) {...} // ^~~~~~~~~~

The ~ after the fn tells the type system that the environment for this function is allocated using an owned pointer. It also implies a default bound of Owned. The once tells the type system that the function will only execute once.

For contrast, this is ~once fn() today.

Const functions

Here is an example of how I would use a const function to achieve lightweight parallelism:

impl<T:Sized> for [T] { pub fn par_map<U:Sized>(f: fn:Const(&T) -> U) -> bool { ... } // ^~~~~~~~~~~~~~~~~ }

This is a parallel map function. It is similar to the regular map except that its iterations execute in parallel. As a consequence, it demands a fn:Const rather than a fn—the Const bound specifies that all the environmental state must be immutable. This is exactly the “patient parent” or “parallel closures” model that is used in PJS and described in this HotPar paper I wrote.

For contrast, this is &fn:Const() today.

Sendable const functions

Sendable const functions are one of the two cases that I said would become less attractive under the new proposal. They would look something like fn~:Const (vs ~fn:Const today). The newer syntax works and should be available, but it’s hard to read, due I think to the juxtaposition of ~ (which specifies the kind of pointer used for the environment) and the : that begins the bound specifier :Const. If this use case were important, I might be worried that the syntax is too ugly, but when I tried to come up with an example for where this use case would be needed, I realize that time has left the use case behind to some extent.

The primary use case for a sendable const function initially was to allow hashtables to be placed in ARCs—the reason for this was that a HashMap requires closures for for computing the hash function of its argument, and those to share the hashmap (and perform parallel lookups) we had to be sure that the closures would not mutate any state. However, this is somewhat outdated, because hashing and equality comparison today is based on traits rather than closures.

Now, using traits is somewhat limited, because due to coherence it means that any one type can only be hashed in one way, and sometimes you would like to have specialized hashing for specific circumstances. But these use cases can easily be accommodated in three ways:

1. Using newtyped keys (struct MyKey(key)) and defining different implementations for the hashing and equality traits on MyKey.
2. If a newtyped key is not acceptable, you can write a hash table that takes a simple function pointer (extern "Rust" fn) rather than a closure. Function pointers carry no state, but state is rarely needed for equality comparisons.

3. If you really need state, then you can write a specialized trait in lieu of a closure:

   trait HashFuncs<K> { fn hash(&self, k: &K) -> uint; fn eq(&self, k1: &K, k2: &K) -> bool; }

   Now your hashtable can either take a ~HashFuncs object to use for hashing and equality comparison or, if you wish to avoid dynamic dispatch for performance reasons, you can parameterize your hashtable type by the instance of HashFuncs that it should use:

   struct MyHashMap<K,V,F:HashFuncs<K>> { f: F, ... }

Combinators

General purpose combinators are the other case that (might) get less attractive. This is less clear cut. The idea of a combinator library is that you have functions that return functions, and then you can compose these functions into bigger functions. The most common example is a parser combinator, which is a simple way to create inefficient and buggy parsers (ok, that’s unfair, but I couldn’t resist; I’ve had some bad experiences trying to scale up parser combinators—truth is, they are super nice to work with, at least until things go wrong).

Anyway, a typical parser combinator library would begin with a primitive like the following:

fn expect(c: char) -> fn@(&mut ParseState) -> Result<(), Err> { ... } // ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Note that the function returns a closure. We used fn@ because this closure must be allocated on some heap in order for us to return it, and because using the type fn@ (vs say fn~) would allow us to close over managed and other task-local data. So far, I think this example works out fine.

Where things get more complex is if we want to close over borrowed pointers. For example, imagine an expect function that takes a slice:

fn expect_string<'a>(s: &'a str) -> fn@:'a(&mut ParseState) -> Result<(), Err> {...} // ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Here the type system will require that the lifetime 'a of the input slice s appear in the resulting function type, so that it can be sure that the function is not used after the slice is no longer valid. This makes the type more complicated: fn@:'a (vs the also-not-especially-intuitive notation of @'a fn today).

Of course, one could address this problem by having expect_string take a ~str or @str instead of a borrowed string, but in some use cases borrowed pointers may perfect sense. For example, I had once thought to use this pattern to create a combinator library for expressing iteration primitives like enumerate and so forth (similar, experimental work is now underway in the iter module).

Interestingly, just as with sendable const closures, objects and traits can provide an alternative that is ultimately (I think) a better and more readable design anyway. We could rewrite the return type from a closure into a trait:

trait Parser<R> { fn parse(&mut ParseState) -> Result<R,Error>; } fn expect(c: char) -> @Parser<()>; fn expect_string<'a>(s: &'a str) -> @Parser:'a<()>;

Here in the expect_string case I have taken advantage of the fact that object types will also carry bounds similar to closure types. An advantage of this design is that using a trait allows the Parser objects to carry more methods as well.

If we were to extend the example to include an actual combinator, I imagine it would look something like this:

fn or<'a, R>(p1: @Parser:'a<R>, p2: @Parser:'a<R>) -> @Parser:'a<R> {...}

Of course, for maximum efficiency, one would avoid using object types altogether. Then you would just implement Parser directly on the char and &str types, and perhaps write the or combinator like so:

struct or<P1,P2>(P1, P2); impl<R,P1:Parser<R>,P2:Parser<R>> Parser for or<P1,P2> { fn parse(&self, state: &mut ParseState) -> Result<R,Error> { let (ref p1, ref p2) = *self; state.try(); // (*) match p1.parse(state) { Ok(r) => { state.confirm(); Ok(r) } Err(_) => { state.backtrack(); p2.parse() } } } } // (*) Here you see my imperative roots. A true functional // programmer would not use in-place mutation here but rather // clone and return a new parser state. Summary

Another long post mostly targeted at rust devs and myself. Sorry about that. I think the bottom line is that we should move sigils for closures and have them appear after the fn keyword. This makes me sad, because this is how things used to be, and in fact one of the main goals of the dynamically sized types (DST) proposal was to move the sigils in closure types in front. But of course soundness comes first, and I think the general wins of the DST proposal (consistent behavior for all &T, @T, ~T etc) outweigh the need to write fn~ on occasion (I don’t really see much use for fn@).

There is also one final solution I didn’t mention in my initial paragraphs. We could adopt the “principled” solution of using &mut for closures but change the way we notate &mut. I have largely avoided thinking about because I want to avoid destabilizing syntax changes. However, I have toyed around occasion with an idea for reorganizing our types to emphasize ownership and de-emphasize mutability, which goes in this direction. I may indulge myself and write it up at some point. Still, I largely consider this a non-starter.

Adopting the “move sigils in back” proposal does have another casualty, though. There has been some talk of figuring out ways to make @ and ~ less special (as in, allowing user-defined pointer types like RefCounted<T> that are on equal footing). The DST proposal is clearly a step in that direction. Moving the sigils backwards on fn types is, well, a step backward, because closures would always be allocated using a limited set of allocators (stack, ~, or @).

In an odd way, finding this interaction makes me feel good. I’ve been concerned that the DST proposal seemed too easy, which meant we weren’t thinking hard enough about it. But there is another reason as well: I have also been concerned that closure types were becoming a bit too… special, particularly with regard to copyability. Basically I’ve been concerned that although the syntax for a borrowed closure was &fn, borrowed closures didn’t really behave like & pointers—without the DST proposal, this was something that we could safely enforce as part of the type system, but it’s still confusing for users. So I think the DST proposal forces us to be more honest, and that’s a good thing all around.

In my previous post, I talked about the moving of some files for Firefox 21. Due to some major bugs related to partner builds, the distribution directory was moved back. So you no longer need to move distribution files into the browser directory. So here's a summary of what has changed:

• defaults/preferences -> browser/defaults/preferences
• defaults/profile -> browser/defaults/profile
• extensions -> browser/extensions
• searchplugins -> browser/searchplugins

I've also been asked about the file override.ini - this file should now be placed in the browser subdirectory as well.

Remember, Firefox 21 is scheduled to be released tomorrow.

I have an opinion on the usage of Mercurial at Mozilla: it stinks.

Here's why.

The server is configured poorly

Our Mozilla server, hg.mozilla.org, is currently running Mercurial 2.0.2. In terms of Mercurial features, stability, and performance, we are light years behind.

You know that annoying phases configuration you need to set when pushing to Try? That's because the server isn't new enough to tell the client the same thing the configuration option does. It will be fixed when the server is upgraded to 2.1+.

Furthermore, we are running the server over NFS, which introduces known badness, including slowness.

I believe we blame Mercurial for issues that would go away if we configured the Mercurial server properly.

Fortunately, it appears the upgrade to 2.5 is near and I've heard we're moving from NFS to local disk storage as part of that. This should go a long way to making the server better. The upgrade can't happen soon enough.

User education is poor

I think a lot of people are ignorant on the features and abilities of Mercurial.

I commonly hear people are dissatisfied with the behavior of their Mercurial client. They encounter performance issues, bugs, corruption, etc. Nine times out of ten this is due to running an old Mercurial release. Just last Friday someone on my team asked me about weird behavior involving file case. My first question: what version of Mercurial are you using? He was running 2.0.2. I told him to upgrade to 2.5+. It fixed his problem. If you aren't running Mercurial 2.5 or newer, upgrade immediately.

I've heard people say we should switch to Git because Git has feature X. Most of the time, Mercurial has these features. Unfortunately, people just don't realize it. When I point them at Mercurial's extensions list their eyes light up and they thank me for making their lives easier.

I think a problem is a lot of new Mozilla contributors knew Git before and only pick up the bare essentials of Mercurial that allow them to land patches. They prefer Git because it is familiar and just don't bother to pick up Mercurial. The potential of Mercurial is thus lost on them.

Perhaps we should have a brown bag and/or better documentation on getting the most out of Mercurial?

The branching model is far from ideal

For Gecko/Firefox development, we maintain separate repositories for the trunk and release branches. This introduces all kinds of annoying.

We should not have separate repositories for central, inbound, aurora, beta, release, etc. We should be using some combination of branches and bookmarks and have all the release heads in one repository, just like how the GitHub mirror is configured.

As an experiment, I created a unified Mercurial repository. Each current repository is tracked as a bookmark (there are instructions for reproducing this). Unfortunately, the web interface isn't showing bookmarks (perhaps because the version of Mercurial is too old?), so you'll have to clone the repository to play around. Just run hg bookmarks and e.g. hg up aurora after cloning. Warning: I'm not actively synchronizing this repository, so don't rely on it being up to date.

A Mercurial contributor (who is familiar with Mozilla's development model) suggested we use Mercurial branches for every Gecko release (20, 21, 22, etc). I think this and other uses of branches and bookmarks are ideas worth exploring.

We're failing to harness the extensibility

Gecko/Firefox has a complicated code lifecycle and landing process. This could be significantly streamlined if we fully harnessed and embraced the extensibility of Mercurial. While there are some Mozilla-centric extensions (details in my recent post), I don't think they are well known nor used.

I think Mozilla should embrace the functionality of extensions like these (whether they be for Mercurial, Git, or something else) and invest resources in improving the workflows for all developers. Until these tools are obviously superior and advertised, I believe many developers will unknowingly continue to toil without them. And, it's likely hurting our ability to attract and retain new contributors as well.

Conclusion

Mozilla's current usage of Mercurial is far from ideal. It's no wonder people don't like Mercurial (and why some want to switch to Git).

Fortunately, little has to do with shortcomings of Mercurial itself (at least with newer versions). If you want to know why Mercurial isn't working too well for Gecko/Firefox development, most of the problems are self-inflicted or the solutions reside within each of us. Time will tell if we as a community have the will to address these issues.

I spent Friday and Saturday on git-merge, an unconference on git. Thursday was developer day, for core contributors to git itself, and libgit2/jgit. I didn’t go there. Friday was “user” day, and Saturday was hackday. I figured it might be useful to go to the userday, and turned out, it was. It wasn’t all that much about users at all. It was strictly unconference, so people would take the stage and give a quick lightening talk, and later in the day we had a few breakout sessions. The most user-like people were folks migrating to git just now, and they had a good deal of folks to talk to in the breakout session.

The rest of Friday was actually library and tool hackers talking about what they’re doing. There are notes on github.com/git-merge/user-day, but I do want to hightlight a few.

imerge is probably the most interesting to gecko hackers. It allows you to merge or rebase intensive histories with conflicts incrementally, with tons of automation. It even allows to do test runs on the merges it does automatically. If you ever resolved the same conflict 10 times in one rebase, this is for you.

libgit2 was under the hood of many, with core contributors of the library itself being there, plus the maintainers of at least the C# and the python bindings. There was also a good deal of tooling based on jgit, a pure-java implementation of git.

Much debate on java vs not, actually. And Cmd-F1. Little conflict between git and hg.

I also got to enjoy the github backend talk by vmg, with Ernie, Bert, and smoke through chimneys. I had seen a recording of it before, but it was well worth it seeing it live.

I joined the breakout session about teaching git, and had the pleasure to be in a group that tought git in various parts of the globe. Yes, that might be relevant to me, so that was good exercise. I talked with Scott Schacon about localized version of the git book, and localized github docs, too.

Given that I had the chance to talk and co-hack with all those tooling guys, I decided to drop by on Saturday for the hack day. I wanted to take the opportunity to talk about my weird repository rewrite questions, and so I did.

Saturday was great. I only got 20 lines of code written, but I finally understood what git is about in the back-end. There’s loads of hackery that you can do, and funny enough, both I and Ben ended up hacking on a repository rewrite algorithm, which helped me a lot, both talking about the structure of the code, as well as vetting the approach. His code in C# is actually in a PR. Worth a look for people that want to see how to hack tooling on top of that binding. I tried the same in python, and succeeded to some extent. David Ibáñez helped me a great deal. But it also showed me the difference between the C# API and the python bindings. If only mono was easy to get up on the mac.

On the conference itself, it was set up at the Radisson Blu next to the Berliner Dom, which is a nice venue for that size. Wifi worked, food and beer were there. It’s sad that many people claimed tickets and didn’t show up. Now you know what you missed, and what you made other people miss. Bad bunny, no chocolates! Thanks again for Jen for setting things up great.

Thanks to all the folks at git-merge for making it a great event. See you in Berlin…

There’s always a lag behind new releases of PHP and releases of packages for operating systems such as Ubuntu. This lag time means that you could be kept from upgrading to the latest and greatest PHP for a year or more, unless you use an outside repository like Dotdeb. Of course, even when using one [...]

After several days of migraines and with an upcoming intense week in the form of a Mozilla All-Hands meeting, this week I’m going to be focusing on inputs rather than outputs.

This is likely to be my only blog post of the week (apart from my weeknote) and I’ll be a lot lighter in my use of Twitter / Google+ / Facebook.

If you want to make me smile while I’m ensuring I don’t get burned out, how about donating to #LettingGrow? That would be awesome.

Image CC BY Pink Sherbert Photography

Some things made with 3D printers could possibly be made more easily using other manufacturing methods – although printing them yourself is often more fun. But sometimes you come across something which would be fairly tricky to make otherwise, unless perhaps you had high-end equipment like a laser cutter.

Cots, in the UK at least, often have a “drop side” – a side which is normally at the standard height, but can be lowered to make it easier to insert or remove a child. The side is usually held up by some kind of safety clip at each end, such that when you raise the side, the clips automatically engage, and when you want to lower it, you need to lift the side a little and pull both clips out of the way from the outside. This prevents a child pulling off an escape :-)

One of the two cots we have used to belong to my wife’s brother. It has such clips, and long ago one of them broke. He made a flat replacement out of cut perspex, but it has also recently broken. This is a problem because our changing table fits on top of the cot, and if the sides aren’t the same height, the table can’t be used and we are having to change William and John with the mat on the floor! So I thought I’d try and print one.

As you can see from the right of the photo below, the existing clip is a rather specific shape, and would have been hard to produce by many other methods without a lot of trial and error. Many parts have to be exactly the right angle and length or it won’t open, or close, or pass the cot side, or whatever. I took a scan of the existing clip (which was a bit blurry, as the clip is not flat) and traced around it in LibreCAD. I then saved that trace as a DXF file, and wrote an OpenSCAD file to make a new clip, based on a linear extrusion of the outline, plus (on later attempts) two extra blocks for a stop and a handle. Here are my efforts:

The first was too small, the second marginally too big (and the blocks not aligned right), and the third, like Little Bear’s porridge, was just right :-) Well, it suffices anyway, although the hook could do with being 1mm smaller for easy passage of the side.

Here’s the clip in situ, supporting the side of the cot:

And this photo, beautifully modelled by my wife, shows how it fits when the side is down:

I wanted to give my one cent about the GNOME project, and where I think it could be successful. It would be two cents if I were actually involved in any constructive manner, but I am not. So it is one cent.

Ever since I started contributing to GNOME, the looming questions have been mobile, web, and social. Every keynote at GUADEC has tugged us in that direction, or promised to “reboot” the effort. If it is Big Board and Mugshot, Pyro Desktop, Telepathy and the collaboration it was supposed to bring, the countless OpenedHand and Nokia innovations, etc. We have all been running around like a chicken with its head cut off ever since I remember, trying to capture the essence of these new trends and remain relevant.

We failed.

Apple revolutionized not just mobile, but reinvented the mainstream computing form factor. Facebook made “social” ubiquitous, and Google is doing what it is doing to the web. In the meantime, I never gained any following on Last.fm from all those years of scrobbling music with Rhythmbox and Banshee, I never got an opportunity to use Telepathy tubes with any real live person, and apparently my Mugshot profile is gone. My eyes also got tired of squinting at XTerm on my N900.

Last year in Berlin, the lack of direction was apparent. Almost every keynote that I remember was given by one kind of designer or seo company. Somewhere along the line we confused design with leadership. At least there wasn’t as much self delusion about our bright future on mobile.

But there is a way out of this rut. And it requires acknowledging our weaknesses and exploiting our strengths.

Our weak areas are apparent: We are not mobile and we are very far from it. We will never achieve any significant social critical mass, we have had limited successes in embracing web technologies, but the web will always be a better web. Deploying “apps” is a nightmare.

Our strengths are pretty obvious too: In the last few years we successfully refreshed the desktop work flow and our entire framework. We support many productivity and authoring tools. We created a distraction free environment that lets users get work done. We run on commodity hardware. We are free. We have a windowed multitasking environment. We work really well with a screen, mouse and keyboard (not to be taken for granted, look at all the awkward Android tablet keyboard combos out there). More than one web browser supports us. There is a more than fully functional office suite that works well with us. Etc.

So instead of aspiring to be in every consumer gadget out there, I think we should aspire to be the work horse of choice for every content creator out there. This includes mobile/web developers, graphic designers, artists, bloggers, video bloggers, authors, journalists, podcast producers, and every other kind of content creator that makes the mobile web and social such a vital space for the rest of the world. We need to refocus on the desktop.

Let’s leave the mission of bringing free software to mobile and the web to others. Other groups are doing a great job there. They are in their element; let’s remain in ours. We should focus on the production end of the New Media pipeline.

Projects such as The GIMP, PiTiVi, Anjuta, Blender and Libre Office should be our bread and butter. We should strive to stay ahead of the curve on the authoring end. We should document and support Android and Unity development. The Wacom tablet support that landed recently is a good example of what we could be doing.

It feels like GNOME is being maintained by a skeleton crew, and a shrinking set of corporate stakeholders. I think it is time to be realistic about what we could excel at, and go there. We don’t have to be on every existing form factor to achieve world domination. The cloud, and all these cheap new gadgets have lowered the barrier to access. We could lower the barrier of authorship, and enable people to create new and rich content.

Normally, a small book like this (193 pages) would be a quick read for me, but this book took me literally months. Not, I hasten to add, because of any problems with the book or the writing style, that was all fine. The problem was that this book uncovered a bunch of things I am personally working through. I found myself reading a few pages, highlighting some lines, then walking away thinking. Repeat a few times a week. Occasionally, I’d go back and re-read entire chapters.

For me, bragging has negative connotations and is something I avoid like the plague. Stereotypes of obnoxious, pretentious people, loudly telling all within range just how great they are. The very last thing I ever want to be. Whether that is cultural, learned from family, something I developed myself growing up, or a mixture, I don’t know. But it is part of who I am. This book is all about encouraging people to find a comfortable place in between these extremes. As Peggy is quick to note, this means different things to different people, so you need to pay attention to what is authentic for you, as that authenticity is important. People have generations of experience spotting fakes, and worst of all, deep down, you’ll know you are faking it too.

Because of the book title, it took several people pushing to get me to even start reading this book. Chapter#1 opened with a line that stopped me dead in my tracks.

“Myth#1: A job well done speaks for itself.”

I’ve always thought that if I did a good job, or handled a tricky situation well, people would notice. If I solved some complex problem, that people would understand the complexity, understand the importance of the achievement and appreciate the work. In those circumstances, having others recognize and complement the achievement was fine, but any attempt on my part to “brag” about my work would in some way “cheapen the victory”. After reading this book, I now think that is *sometimes* true but not always true. While the people working beside me in the same trenches, working side-by-side with me on the problem might understand the scale of the accomplishment, most people simply don’t know the details. Over time, people might eventually notice that a recurring problem hasn’t happened in a while, or they might simply forget about a previously-annoying problem because it hasn’t happened in a while… but they’d never stop and wonder why. Another common trend is for people to not notice one problem is fixed, but instead notice that a different problem has “appeared”. Oh, and meanwhile, people don’t know what you are working on. Over time, this becomes frustrating for everyone. After reading this book, I’ve learned that I need to make sure I inform people of the work I’m doing, and why it’s important to them. I don’t need to go into all the complexities of the project, unless they ask for more details, but it’s important to make sure others are aware of my work, and the impact it has on them and their work.

I found this a tough read, yet super worth the time. And, yes, I strongly recommend it.

“It ain’t bragging if you done it” (Dizzy Dean)

A few days ago a vertical video went viral of the student Jeff Bliss telling his teacher off for being a bad teacher who just hands out materials without explaining anything.

And we all applauded him for his gall and his eagerness to learn and to point out the obvious flaws in the education system. Teachers are not paid enough, are under more stress to be seen as someone who has students with good grades rather than understanding and have to deliver course materials they don’t believe in but are forced to go through as those are the ones that are easy to mark.

Terrible, isn’t it? So why do people in our little world of web development constantly and voluntarily become this bad, bored and ineffective teacher?

What am I talking about? The thing I mentioned in large detail in my talk at BaconConf this year but here it is for the generation who wants things in 140 chars or as a cute image with large letters on it:

Whenever you answer a question of another developer with “just use $x” you breed an expert idiot. You did not teach them anything, you showed a way out of learning.

In my ongoing task to de-clutter my life I just un-subscribed from several communities on Google+, namely the HTML5 community and the JavaScript one. Not because I am not interested in these matters any more, quite the opposite: because I care much about these communities and all I found there is frustration and annoyance. Nearly every single question new developers have is answered in three ways:

• Use jQuery – here is a plugin
• Just Google for it
• You’ll need to use framework $x / JavaScript and/or HTML5 is not good enough for that

None of these answers are satisfactory if you really want to help someone who needs to solve a problem and learn how to repeat the solution in the future. The latter in most cases is a plain lie and shows that you are blaming a technology for your lack of interest in it.

What gets answered far too quickly is the “how” – how to achieve a massively complex result (which yet has to be proven to be really necessary) without thinking about it or understanding the solution that you apply. We assume that is enough and that we are doing something good – we let a new developer have a positive experience of having something achieved very quickly and that will obviously entice him or her to learn more and go explore in more detail later on.

That is not necessarily the case. We showed people a shortcut and how to focus on the outcome and hope the step where they understand what they are doing comes later. Sadly in a lot of cases this never comes but it fills people with fake confidence that gets shattered once they have their first job interview in a real company who cares about what they build.

If you want to teach people, make them understand the “why” and let them find their own “how”. That is much harder work, but also much more rewarding when you see them grow and explore.

We do not teach people how to write by copying and pasting the style of other authors. We tell them about similes, metaphors, rhetoric devices, orthography and grammar rules. Why bother with that? We could just show them TXTSPK and explain that anyone who knows English will understand what they try to convey. The reason why we do it is that we teach the fun of playing with language and finding your own style.

“But I don’t have time for that – I just want to help someone solving their problem”

Is a very common, admirable, but misguided idea. What you do is advertise the solution that made most sense to you as you already solved the problem. You steal the learning experience away from the other person and the way we learn is our most personal asset and differs vastly from person to person.

If you don’t want to really teach and see people grow and learn on their own terms, please stop spouting truisms and “best quick solutions” in places where people come to learn. If all they want is for you to solve their issue, they should hire you to do so for them.

Don’t be the grumpy teacher you learned to first despise and later on pity. We can do better on the web.

So I've now built my first real application using AngularJS. It's a fun side-project which my wife and I use to track what we spend money on. It's not a work project but it's also not another Todo list application. In fact, the application existed before as a typical jQuery app. So, I knew exactly what I needed to build but this time trying to avoid jQuery as much as I possibly could.

The first jQuery based version is here and although I'm hesitant to share this beginner-creation here's the AngularJS version

The following lists were some stumbling block and other things that stumped me. Hopefully by making this list it might help others who are also new to AngularJS and perhaps the Gods of AngularJS can see what confuses beginners like me.

1. AJAX doesn't work like jQuery

Similar to Backbone, I think, the default thing is to send the GET, POST request with the data the body blob. jQuery, by default, sends it as application/x-www-form-urlencoded. I like that because that's how most of my back ends work (e.g. request.GET.get('variable') in Django). I ended up pasting in this (code below) to get back what I'm familiar with:

module.config(function ($httpProvider) { $httpProvider.defaults.transformRequest = function(data){ if (data === undefined) { return data; } return $.param(data); }; $httpProvider.defaults.headers.post['Content-Type'] = '' + 'application/x-www-form-urlencoded; charset=UTF-8'; }); 2. App/Module configuration confuses me

The whole concept of needing to define the code as an app or a module confused me. I think it all starts to make sense now. Basically, you don't need to think about "modules" until you start to split distinct things into separate files. To get started, you don't need it. At least not for simple applications that just have one chunk of business logic code.

Also, it's confusing why the the name of the app is important and why I even need a name.

3. How to do basic .show() and .hide() is handled by the data

In jQuery, you control the visibility of elements by working with the element based on data. In AngularJS you control the visibility by tying it to the data and then manipulate the data. It's hard to put your finger on it but I'm so used to looking at the data and then decide on elements' visibility. This is not an uncommon pattern in a jQuery app:

<p class="bench-press-question"> <label>How much can you bench press?</label> <input name="bench_press_max"> </p> if (data.user_info.gender == 'male') { $('.bench-press-question input').val(response.user_info.bench_press_max); $('.bench-press-question').show(); }

In AngularJS that would instead look something like this:

<p ng-show="male"> <label>How much can you bench press?</label> <input name="bench_press_max" ng-model="bench_press_max"> </p>

if (data.user_info.gender == 'male') { $scope.male = true; $scope.bench_press_max = data.user_info.bench_press_max; }

I know this can probably be expressed in some smarter way but what made me uneasy is that I mix stuff into the data to do visual things.

4. How do I use controllers that "manage" the whole page?

I like the ng-controller="MyController" thing because it makes it obvious where your "working environment" is as opposed to working with the whole document but what do I do if I need to tie data to lots of several places of the document?

To remedy this for myself I created a controller that manages, basically, the whole body. If I don't, I can't manage scope data that is scattered across totally different sections of the page.

I know it's a weak excuse but the code I ended up with has one massive controller for everything on the page. That can't be right.

5. setTimeout() doesn't quite work as you'd expect

If you do this in AngularJS it won't update as you'd expect.

<p class="status-message" ng-show="message">{{ message }}</p>

$scope.message = 'Changes saved!'; setTimout(function() { $scope.message = null; }, 5 * 1000);

What you have to do, once you know it, is this:

function MyController($scope, $timeout) { ... $scope.message = 'Changes saved!'; $timeout(function() { $scope.message = null; }, 5 * 1000); }

It's not too bad but I couldn't see this until I had Googled some Stackoverflow questions.

6. Autocompleted password fields don't update the scope

Due to this bug when someone fills in a username and password form using autocomplete the password field isn't updating its data.

Let me explain; you have a username and password form. The user types in her username and her browser automatically now also fills in the password field and she's ready to submit. This simply does not work in AngularJS yet. So, if you have this code...:

<form> <input name="username" ng-model="username" placeholder="Username"> <input type="password" name="password" ng-model="password" placeholder="Password"> <a class="button button-block" ng-click="submit()">Submit</a> </form>

$scope.signin_submit = function() { $http.post('/signin', {username: $scope.username, password: $scope.password}) .success(function(data) { console.log('Signed in!'); }; return false; };

It simply doesn't work! I'll leave it to the reader to explore what available jQuery-helped hacks you can use.

7. Events for selection in a <select> tag is weird

This is one of those cases where readers might laugh at me but I just couldn't see how else to do it.
First, let me show you how I'd do it in jQuery:

$('select[name="choice"]').change(function() { if ($(this).val() == 'other') { // the <option value="other">Other...</option> option was chosen } });

Here's how I solved it in AngularJS:

$scope.$watch('choice', function(value) { if (value == 'other') { // the <option value="other">Other...</option> option was chosen } });

What's also strange is that there's nothing in the API documentation about $watch.

8. Controllers "dependency" injection is, by default, dependent on the controller's arguments

To have access to modules like $http and $timeout for example, in a controller, you put them in as arguments like this:

function MyController($scope, $http, $timeout) { ...

It means that it's going to work equally if you do:

function MyController($scope, $timeout, $http) { // order swapped ...

That's fine. Sort of. Except that this breaks minification so you have to do it this way:

var MyController = ['$scope', '$http', '$timeout', function($scope, $http, $timeout) { ...

Ugly! The first form depends on the interpreter inspecting the names of the arguments. The second form depends on the modules as strings.

The more correct way to do it is using the $inject. Like this:

MyController.$inject = ['$scope', '$http', '$timeout']; function MyController($scope, $http, $timeout) { ...

Still ugly because it depends on them being strings. But why isn't this the one and only way to do it in the documentation? These days, no application is worth its salt if it isn't minify'able.

9. Is it "angular" or "angularjs"?

Googling and referring to it "angularjs" seems to yield better results.

This isn't a technical thing but rather something that's still in my head as I'm learning my way around.

In conclusion

I'm eager to write another blog post about how fun it has been to play with AngularJS. It's a fresh new way of doing things.

AngularJS code reminds me of the olden days when the HTML no longer looks like HTML but instead some document that contains half of the business logic spread all over the place. I think I haven't fully grasped this new way of doing things.

From hopping around example code and documentation I've seen some outrageously complicated HTML which I'm used to doing in Javascript instead. I appreciate that the HTML is after all part of the visual presentation and not the data handling but it still stumps me every time I see that what used to be one piece of functionality is now spread across two places (in the javascript controller and in the HTML directive).

I'm not givin up on AngularJS but I'll need to get a lot more comfortable with it before I use it in more serious applications.

My first experience with Mercurial (Firefox development) was very unpleasant. Coming from Git, I thought Mercurial was slow and perhaps even more awkward to use than Git. I frequently encountered repository corruption that required me to reclone. I thought the concept of a patch queue was silly compared to Git branches. It was all extremely frustrating and I dare say a hinderance to my productivity. It didn't help that I was surrounded by a bunch of people who had previous experience with Git and opined about every minute difference.

Two years later and I'm on much better terms with Mercurial. I initially thought it might be Stockholm Syndrome, but after reflection I can point at specific changes and enlightenments that have reshaped my opinions.

Newer versions of Mercurial are much better

I first started using Mercurial in the 1.8 days and thought it was horrible. However, modern releases are much, much better. I've noticed a steady improvement in the quality and speed of Mercurial in the last few years.

If you aren't running 2.5 or later (Mercurial 2.6 was released earlier this month), you should take the time to upgrade today. When you upgrade, you should of course read the changelog and upgrade notes so you can make the most of the new features.

Proper configuration is key

For my workflow, the default configuration of Mercurial out of the box is... far from optimal. There are a number of basic changes that need to be made to satisfy my expectations for a version control tool.

I used to think this was a shortcoming with Mercurial: why not ship a powerful and useful environment out of the box? But, after talking to a Mercurial core contributor, this is mostly by design. Apparently a principle of the Mercurial project is that the CLI tool (hg) should be simple by default and should minimize foot guns. They view actions like rebasing and patch queues as advanced and thus don't have them enabled by default. Seasoned developers may scoff at this. But, I see where Mercurial is coming from. I only need to refer everyone to her first experience with Git as an example of what happens when you don't aim for simplicity. (I've never met a Git user who didn't think it overly complicated at first.)

Anyway, to get the most out of Mercurial, it is essential to configure it to your liking, much like you install plugins or extensions in your code editor.

Every person running Mercurial should go to http://mercurial.selenic.com/wiki/UsingExtensions and take the time to find extensions that will make your life better. You should also run hg help hgrc to view all the configuration options. There is a mountain of productivity wins waiting to be realized.

For reference, my ~/.hgrc. Worth noting are some of the built-in externsions I've enabled:

• color - Colorize terminal output. Clear UX win.
• histedit - Provides git rebase --interactive behavior.
• pager - Feed command output into a pager (like less). Clear UX win.
• progress - Draw progress bars on long-running operations. Clear UX win.
• rebase - Ability to easily rebase patches on top of other heads. This is a basic feature of patch management.
• transplant - Easily move patches between repositories, branches, etc.

If I were on Linux, I'd also use the inotify extension, which installs filesystem watchers so operations like hg status are instantaneous.

In addition to the built-in extensions, there are a number of 3rd party extensions that improve my Mozilla workflow:

• mqext - Automatically commit to your patch queue when you qref, etc. This is a lifesaver. If that's not enough, it suggests reviewers for your patch, suggests a bug component, and let's you find bugs touching the files you are touching.
• trychooser - Easily push changes to Mozilla's Try infrastructure.
• qimportbz - Easily import patches from Bugzilla.
• bzexport - Easily export patches to Bugzilla.

I'm amazed more developers don't use these simple productivity wins. Could it be that people simply don't realize they are available?

Mozilla has a bug tracking easier configuration of the user's Mercurial environment. My hope is one day people simply run a single command and get a Mozilla-optimized Mercurial environment that just works. Along the same vein, if your extensions are out of date, it prompts you to update them. This is one of the benefits of a unified developer tool like mach: you can put these checks in one place and everyone can reap the benefits easily.

Mercurial is extensible

The major differentiator from almost every other version control system (especially Git) is the ease and degree to which Mercurial can be extended and contorted. If you take anything away from this post it should be that Mercurial is a flexible and agile tool.

If you want to change the behavior of a built-in command, you can write an extension that monkeypatches that command. If you want to write a new command, you can of course do that easily. You can have extensions interact with one another - all natively. You can even override the wire protocol to provide new capabilities to extend how peers communicate with one another. You can leverage this to transfer additional metadata or data types. This has nearly infinite potential. If that's not enough, it's possible to create a new branching/development primitive through just an extension alone! If you want to invent Git-style branches with Mercurial, you could do that! It may require client and server support, but it's possible.

Mercurial accomplishes this by being written (mostly) in Python (as opposed to C) and by having a clear API on which extensions can be built. Writing extensions in Python is huge. You can easily drop into the debugger to learn the API and your write-test loop is much smaller.

By contrast, most other version control systems (including Git) require you to parse output of commands (this is the UNIX piping principle). Mercurial supports this too, but the native Python API is so much more powerful. Instead of parsing output, you can just read the raw values from a Python data structure. Yes please.

Since I imagine a lot of people at Mozilla will be reading this, here are some ways Mozilla could leverage the extensibility of Mercurial:

• Command to create try pushes (it exists - see above).
• Record who pushed what when (we have this - it's called the pushlog).
• Command to land patches. If inbound1 is closed, automatically rebase on inbound2. etc. This could even be monkeypatched into hg push so pushes to inbound are automatically intercepted and magic ensues.
• Record the automation success/fail status against individual revisions and integrate with commands (e.g. only pull up to the most recent stable changeset).
• Command to create a review request for a patch or patch queue.
• Command to assist with reviews. Perhaps a reviewer wants to make minor changes. Mercurial could download and apply the patch(es), wait for your changes, then reupload to Bugzilla (or the review tool) automatically.
• Annotating commits or pushes with automation info (which jobs to run, etc).
• Find Bugzilla component for patch (it exists - see above).
• Expose custom protocol for configuring automation settings for a repository or a head. e.g. clients (with access) could reconfigure PGO scheduling, coalescing, etc without having to involve RelEng - useful for twigs and lesser used repositories.
• So much more.

Essentially, Mercurial itself could become the CLI tool code development centers around. Whether that is a good idea is up for debate. But, it can. And that says a lot about the flexibility of Mercurial.

Future potential of Mercurial

When you consider the previous three points, you arrive at a new one: Mercurial has a ton of future potential. The fact that extensions can evolve vanilla Mercurial into something that resembles Mercurial in name only is a testament to this.

When I sat down with a Mercurial core contributor, they reinforced this. To them, Mercurial is a core library with a limited set of user-facing commands forming the stable API. Since core features (like storage) are internal APIs (not public commands - like Git), this means they aren't bound to backwards compatibility and can refactor internals as needed and evolve over time without breaking the world. That is a terrific luxury.

An example of this future potential is changeset evolution. If you don't know what that is, you should because it's awesome. One of the things they figured out is how to propagate rebasing between clones!

Comparing to Git

Two years ago I would have said I would never opt to use Mercurial over Git. I cannot say that today.

I do believe Git still has the advantage over Mercurial in a few areas:

• Branch management. Mercurial branches are a non-starter for light-weigh work. Mercurial bookmarks are kinda-sorta like Git branches, but not quite. I really like aspects of Git branches. Hopefully changeset evolution will cover the remaining gaps and more.
• Patch conflict management. Git seems to do a better job of resolving patch conflicts. But, I think this is mostly due to Mercurial's patch queue extension not using the same merge code as built-in commands (this is a fixable problem).
• Developer mind share and GitHub. The GitHub ecosystem makes up for many of Git's shortcomings. Bitbucket isn't the same.

However, I believe Mercurial has the upper hand for:

• Command line friendliness. Git's command line syntax is notoriously awful and the concepts can be difficult to master.
• Extensibility. It's so easy to program custom workflows and commands with Mercurial. If you want to hack your version control system, Mercurial wins hands down. Where Mercurial embraces extensibility, I couldn't even find a page listing all the useful Git extensions!
• Open source culture. Every time I've popped into the Mercurial IRC channel I've had a good experience. I get a response quickly and without snark. Git by contrast, well, let's just say I'd rather be affiliated with the Mercurial crowd.
• Future potential. Git is a content addressable key-value store with a version control system bolted on top. Mercurial is designed to be a version control system. Furthermore, Mercurial's code base is much easier to hack on than Git's. While Git has largely maintained feature parity in the last few years, Mercurial has grown new features. I see Mercurial evolving faster than Git and in ways Git cannot.

It's worth calling out the major detractors for each.

I think Git's major weakness is its lack of extensibility and inability to evolve (at least currently). Git will need to grow a better extensibility model with better abstractions to compete with Mercurial on new features. Or, the Git community will need to be receptive to experimental features living in the core. All of this will require some major API breakage. Unfortunately, I see little evidence this will occur. I'm unable to find a vision document for the future of Git, a branch with major new features, or interesting threads on the mailing list. I tried to ask in their IRC channel and got crickets.

I think Mercurial's greatest weakness is lack of developer mindshare. Git and GitHub are where it's at. This is huge, especially for projects wanting collaboration.

Of all those points, I want to stress the extensibility and future potential of Mercurial. If hacking your tools to maximize potential and awesomeness is your game, Mercurial wins. End of debate. However, if you don't want to harness these advantages, then I think Git and Mercurial are mostly on equal footing. But given the rate of development in the Mercurial project and relative stagnation of Git (I can't name a major new Git feature in years), I wouldn't be surprised if Mercurial's feature set obviously overtakes Git's in the next year or two. Mind share will of course take longer and will likely depend on what hosting sites like GitHub and Bitbucket do (I wouldn't be surprised if GitHub rebranded as CodeHub or something some day). Time will tell.

Extending case study

I have removed the case study that appeared in the original article because as Mike Hommey observed in the comments, it wasn't a totally accurate comparison. I don't believe the case study significantly added much to the post, so I likely won't write a new one.

Conclusion

From where I started with Mercurial, I never thought I'd say this. But here it goes: I like Mercurial.

I started warming up when it became faster and more robust in recent versions in the last few years. When I learned about its flexibility and the fundamentals of the project and thus its future potential, I became a true fan.

It's easy to not like Mercurial if you are a new user coming from Git and are forced to use a new tool. But, once you take the time to properly configure it and appreciate it for what it is and what it can be, Mercurial is easy to like.

I think Mercurial and Git are both fine version control systems. I would happily use either one for a new project. If the social aspects of development (including encouraging new contributors) were important to me, I would likely select Git and GitHub. But, if I wanted something just for me or I was a large project looking for a system that scales and is flexible or was looking to the future, I'd go with Mercurial.

Mercurial is a rising star in the version control world. It's getting faster and better and enabling others to more easily innovate through powerful extensions. The future is bright for this tool.

Old school add-ons were developed with a similar mindset, which is that creating new windows is ok, and using XUL this became an easy to-do common practice. The times have changed, and opening new windows is no longer considered acceptable practice, opening tabs is now the preferred user experience.

Now, Firefox has a huge cast of popular, restart addled, old school mannered add-ons, which is a problem. So how do we retrofit these geezers?

I see three parts that need to be addressed:

1. New XUL windows (windows created by the add-on)
2. XUL overlays (content added to windows created by Fx or another add-on)
3. Back-end logic

I plan to write more about all three topics later, because the real solutions require more words and work than the shortcuts will.

Shortcuts Chrome.manifest

Because of Bug 559306 we can now include the old school add-on’s chrome.manifest file in jetpacks. There are some limitations however, for more information on that see this page on the chrome.manifest in restartless add-ons. For instance, overlays will not be regestered, resource uris, components, and more. One can register a chrome uri however, with a skin and locale.

This means that add-ons which implement XUL windows can continue to use them as a Jetpack.

These windows should be redesigned however, but this is no longer a bar to entry, and the resdesign of these windows can be done in a separate step from the conversion to Jetpack.

Also, remember that any windows open which are associated to an add-on should be closed when the add-on unloads, if you do not do this then your add-on will not pass a full review on addons.mozilla.org.

Browser UI

Again, there are a few useful third party modules which handle most common use cases, and if they do not cover your use case then they should provide some direction how to implement them.

I’ve got a module in mind that should help with converting overlays, it’s a messy prototype right now though, so we’ll see how that goes..

Components

There are a few useful third party modules out already which handle things like registering custom uris (this includes about: uris, resource: uris, and whatever:), content policies, and in many cases there are better alternatives. For instance I’ve seen quite a few add-ons register components just in order to hear their ‘startup’ event, this is no longer necessary, with a Jetpack you can simply call require('self').loadReason in main.js.