Nicholas Nethercote is on vacation, so this fortnight's MemShrink report is a guest post written by Andrew McCreight.

Images

Joe Drew made it so that on B2G images being displayed on the active page aren't locked, allowing them to be discarded when there is memory pressure. This is important to display image-heavy pages like Pinterest on B2G without OOM crashes.

Andreas Gal added a preference to control the size of the canvas image cache. The use of canvas with large images was causing OOM crashes for the B2G Gallery app. The limit is set to 10mb for B2G, and remains unlimited on desktop.

Leak fixes

Scoobidiver noticed a large increase in empty crashes on Firefox 23. Previously, these have been identified as being caused by leaks of virtual address space. Scoobidiver and Benjamin Smedberg investigated, found a possible culprit, and saw that the number of empty crashes went down to its previous level, or even lower, after some followup work for that bug was landed.

Timothy Nikkel fixed a leak caused by some recent changes to reflow-on-zoom.

Justin Lebar investigated a leak involving long-running automated testing on B2G and determined that it was the same layers leak previously found and fixed, but not backported to B2G18, by new contributor Christophe Mourand.

Randell Jesup fixed a WebRTC leak.

Miscellaneous

Brian Hackett reduced the memory usage of IonMonkey compilation, which should help on things like JS games that intensively run a lot of JS.

Nicholas Nethercote wrote a patch to change how decommitted GC arenas are shown in about:memory. Decommitted arenas don't actually use physical memory, so they need to be accounted for differently.

Boris Zbarsky reduced the memory usage of CSS in certain cases.

Mozilla Battles Snoop Software Sold Under Its Trademark
Reason
Last week, Mozilla took an important step in the fight against the proliferation of pervasive surveillance technologies by sending a cease and desist letter to Gamma International, demanding Gamma stop using Mozilla's trademark. Gamma makes the ...

Hello Folks. If you have been using Lightning 2.3b1 with a localized version of Thunderbird, you probably had troubles with undefined entities. This happened due to a glitch with the Mozilla localization dashboard, the wrong changeset was used.

I have just uploaded Lightning 2.3b2, which should fix the issue. As always, you can download beta versions from the addons.mozilla.org site in the “Developer Channel” box. If you have previously installed a beta, you should get the update via automatic updates.

https://addons.mozilla.org/en-US/thunderbird/addon/lightning/#install-beta

Money-making
As a non-profit, Mozilla has access to four types of revenue: grants, gifts, donations, and earned income. Our earned income streams come through Firefox, which leaves the other three – grants, gifts, and donations – to be driven by our fundraising strategy (the focus of this post).

So far, and while we can always get better, we’re pretty good at securing grants. We’ve built meaningful working relationships with large philanthropies and government agencies, which contribute approximately $5M each year to Webmaker, Open Badges, and our policy work.

We don’t have a major gifts program (as yet; more on this later in the year).

And, while we’ve spent a fair amount of time building a small dollar donations program, it’s definitely not what it could be. Last year, we generated around $750K from e-mails to more than 500K people and an end-of-year presence in the main Firefox channels.

There are many and valid reasons for this result. Fundraising as a large-scale social enterprise is challenging and I believe we’ve done well to get where we are. But given our reach and the importance of the Web there is massive room for improvement.

NOW: Help us get better at fundraising. Tell us what you think about Mozilla, our mission, and our fundraising.

Solving Small Dollar Fundraising
Building an effective small dollar program is important for several reasons:

• It’s the most stable form of revenue available to the Foundation: small amounts of funding from a large and diverse group of donors;
• It mitigates risk by distributing external dependence and increasing revenue diversity; and
• It’s a meaningful contribution path for a large number of Mozillians who don’t have the time or capacity to code.

Grants and (eventually) gifts will always be crucial to the health and growth of Mozilla. They allow us to work with amazing partners, test new ideas, and launch new products. But building a sustainable small dollar program is our top development priority for 2013.

Enter the Study
Good fundraising – like many things – involves constantly questioning and challenging assumptions. We’ve used the first part of year to hold conversations to gain perspective and input from the Mozilla community: people who care about our mission and want to help it succeed.

We’ve asked what people think about Mozilla. Why they have chosen to become involved. Whether they think being a non-profit is central to what we do. Whether fundraising should be a part of that. And, if yes, how we can make it more effective. (Get involved and be heard.)

The Story So Far
The discussions have been amazing. We’re learning a lot about Mozilla and why our community and staff devote their time to the cause. And we’re also seeing themes emerge around how people view Mozilla, our mission, and our fundraising:

Being a non-profit matters. It’s the foundation of our brand, what differentiates our products in the market, and the source of a lot of pride. Not having external shareholders is empowering. But people resist being identified as a charity. The sense is that ‘charity’ carries connotations of need and behaviours that don’t apply to Mozilla.

Tension between being ‘mission-driven’ and ‘product-driven’. There’s a healthy, though challenging, day-to-day tension between the ‘why’ of our work, the mission, and the ‘what’, shipping products. That tension is surfacing more as we enter new markets with different operating cultures. And in terms of fundraising, the tension impacts decisions like whether to direct site traffic to fundraising or product marketing campaigns.

Fundraising has been a black box. Very little is understood about how or why we raise money. Most of the participants in the study wanted to learn more: to have a chance to engage and help shape our fundraising. So we, as the development team, need to get better at sharing our work. (This post you’re reading and this post and these posts and this dashboard are our first attempts to fix this.)

Donating is considered a more accessible contribution path than coding. Mozilla’s competitive advantage is and has always been its community. We don’t have the financial strength or employee base to go head-to-head with our competitors. When we win, it’s because of the people who contribute to our work. As our mission gains importance – as more things move to the Web – we will need to attract contributors from outside our usual channels. People who don’t code, but have other sources of expertise that can advance the mission (see the groundswell of educators gathering around Webmaker). Donating is seen as an easy yet meaningful way to become a Mozillian.

Online fundraising isn’t the only way to raise money. To date, small dollar fundraising and online fundraising have been synonymous. But as we scale Webmaker events around the world, as we get more people into more rooms, and as ReMo continues to grow and kick ass, we increasingly have opportunities to raise money in-person. We need to look away from the Lost Ark of online fundraising to new ways to engage directly with donors.

Donors want to know where their money goes. In line with trends across the non-profit sector, people want to understand the specific impact of their donations. Crowdfunding, social media, and micro-lending platforms have led donors to expect a direct relationship with the recipients of their support. We can do more to draw the line between a $30 donation and a scientist, teacher, or teenager learning how to express themselves on the Web. (We also need to get our act together around an annual donor report, most likely as a fork of the yearly State of Mozilla.)

We should be good at being a non-profit. One of the most interesting themes was the sense that if we’re going to be a non-profit, we should be good at it. That we should leverage all the advantages – volunteers, movement building, partnerships, activism, fundraising, etc. – that come along with it. Not at the expense of our strength – building meaningful products – but as a way of pursuing our mission and expressing our brand.

The study is also surfacing a healthy amount of skepticism:

• “Why do we need my $30 when we have all the Google revenue?”
• “I delete all those e-mails. I don’t think they’re effective.”

Confusion:

• “Doesn’t everyone get paid from the Google deal?”
• “Why do we even have a Foundation?”

And resistance:

• “We shouldn’t be asking people for money. It’s annoying and unnecessary.”
• “We need people closing bugs, not giving money.”

Perhaps the best finding so far is that many of people who work for Mozilla also donate to the mission. This means that the people who experience Mozilla on a daily basis continue to believe in the organization and its work. As a fundraiser, I know that one of the first things major donors ask is what percentage of employees also donate to the mission. So we’re in good shape.

What Next?
We want to talk to more people. But as we can’t take everyone to lunch (sadly!), we’ve put together a survey. Please take 5 minutes and let us know what you think about Mozilla, our mission, and our fundraising. This is your chance to help us rock. We hope you’ll participate.

What Will Happen From There
In early June, we will:

1. Post the final results and analysis of the study;
2. Use the results to shape a new fundraising strategy; and
3. Work to engage every Mozillian – including you – in what we hope will be our most effective end-of-year campaign ever.

So far we’re thrilled at the number of people who actually want to step up and help Mozilla raise money. The challenge is on us, as a team, to make sure you can and to do so with pride.

Filed under: Mozilla, Pitch Geek

Almost two years, I experimented with using the dominant color of a favicon to give a small icon a colorful background. And over the past week, I wrote some patches to incorporate this design into Firefox for Android!

The simple algorithm I wrote long ago was done in JS with canvas, but to use this in our native Android UI, it’s simplest to just do it in Java. Luckily, we already had a dominant color utility method in the tree, but creating a background and border with different saturation levels was trickier than I thought it would be. To solve this problem, I gave the ImageView a background drawable with a solid white interior and a gray border, then applied a transparent version of the dominant color as a color filter. This worked pretty well once I figured out which PorterDuff mode to use, but it made me appreciate the simplicity of CSS.

When testing with various icons, I found that our dominant color algorithm could use some improvement. Our Java algorithm is different than the one I experimented with in JS mainly because it uses the HSV color model as opposed to RGB. Instead of counting every distinct color, we split the range of hues into different bins and find the bin that holds the most colors. For the winning bin, we compute the average H, S, and V values within that bin, and return that as the dominant color. To make sure we only return colorful colors, we ignore transparent pixels, as well as pixels that are close enough to black or white. This simple algorithm may not be perfect, but it works pretty well for us, especially for small favicons.

As an interesting bit of history, after I blogged about my dominant color experiment, Mardak wrote an add-on that used an improved version of my JS snippet. At the time, Wes saw this add-on and incorporated the code into XUL Fennec to create icons for homescreen shortcuts. During the Fennec Native rewrite, Wes reimplemented this feature in Java, and that’s the code I found myself in last week. And for those interested in a more robust solution implemented in JS, there’s actually a toolkit service that does this now.

Yesterday afternoon, I spotted a sentence in a Planet Mozilla Projects page which shocked me to no end.  Benoit Jacob was advocating the end of MathML in Mozilla code.  The thread has attracted a lot of responses, and the tone largely opposed to his proposal.

Personally, I’m opposed to it as well.  MathML is one of those frontiers which has immense unexplored potential.  Can you imagine writing e-mails to instructors with inline mathematics formulae, or including equations in an instant messaging chat?  I can and have imagined exactly that for as long as I can remember.  I tried once upon a time to bring MathML into Mozilla Composer with my Abacus project, but determined it was too hard and too hacky to be a true solution.

This is precisely why I’ve been working on my prototype XML editor, Verbosio.  It’s supposed to be a complete rewrite of how we create and edit web pages.  The idea is that a language like MathML is simply a Mozilla add-on to the editor.  Unfortunately I’ve been buried with both full-time work and college to make any real progress on my Verbosio project on my own.

I’ve said for years that I didn’t want to attract a larger audience on an unproven principle.  Maybe that’s the wrong decision in this open-source Mozilla community.  While I still believe in the idea, I’ve become my own bottleneck.  It’s far past time for me to swallow my pride and admit that.

What I need to continue development is some help – and I don’t care how junior that help is, as long as they’re capable of writing JavaScript and willing to learn.  Two to five people who can work with me by e-mail and are patient can achieve far, far more than I can on my own.  I can train other engineers in this technology.  I can teach and explain what I’m trying to do and why at a deep level.

We’ve seen major improvements to browsers over the last five years:  HTML 5 form controls, audio and video, faster JavaScript performance, etc.  All of these areas are attractive.  Editing web pages?  Not so much – except to me.  The ability to write efficiently is still as important as the ability to read efficiently.

So, if you’re a budding JavaScript developer who wants to get into something experimental with someone who won’t quit on the idea, please leave a comment.  I should’ve asked you years ago.

The OpenStreetMap project has launched an all-in-JavaScript map editor called "iD" this week:



While we at Mozilla know you can do a lot of good things in JS these days - after all, we're even launching our own phone OS building fully on HTML+JS, and we have been using more and more JS code to power key functionality in our browsers and other products over the years - it's great to see that complex things like editing maps can be done fully in JS and available for all platforms now, while previously it took proprietary and availability-limited technologies like Flash or Java to do the same thing.

Great work, OpenStreetMap guys!


(And yes, as a contributor to OpenStreetMap and even OSMF member, I am biased, but free and open map data on the web fits Mozilla philosophy pretty well anyhow...)

The OpenStreetMap project has launched an all-in-JavaScript map editor called "iD" this week:



While we at Mozilla know you can do a lot of good things in JS these days - after all, we're even launching our own phone OS building fully on HTML+JS, and we have been using more and more JS code to power key functionality in our browsers and other products over the years - it's great to see that complex things like editing maps can be done fully in JS and available for all platforms now, while previously it took proprietary and availability-limited technologies like Flash or Java to do the same thing.

Great work, OpenStreetMap guys!


(And yes, as a contributor to OpenStreetMap and even OSMF member, I am biased, but free and open map data on the web fits Mozilla philosophy pretty well anyhow...)

Currently, Firefox Nightly makes all plugins click-to-play. Except that it doesn’t – the latest version of Flash is exempt and I, like a good net citizen, have the latest version. I disabled Flashblock in order to test CTP… is there any way of getting CTP on ALL, as in all, plugins except by re-enabling Flashblock? Do Flashblock and CTP interact well?

I spend about half my time working for Mozilla working on a new, open learning standard for Web Literacy. The other half of the time I’m evangelising Open Badges in the UK and Europe. Unsurprisingly, with the latter a lot of the same questions come up time and time again. These are legitimate concerns and curiosities that people have, so I thought it would be a good idea to have a URL I could point them towards.

Are Open Badges ‘transferable’?

It depends what you mean. Open Badges are issued with a learner’s individual identifier ‘baked’ into it. So if you try and take my badge and put it in your backpack, it’s not going to work. It’ll be rejected.

If, on the other hand, you talking about the ‘portability’ of badges then, absolutely, that’s exactly what we’re aiming for. Multiple badge backpacks, a completely open and decentralised system, and learner sovereignty. The learner earns badges from issuers and then chooses where to host and display them.

Why is Mozilla interested in creating a system for credentialing learning?

We’re a non-profit that believes in the Web. We believe that it’s a fantastic platform for innovation – but only if it’s open, democratic and built upon standards. Because learning today happens anywhere, including on the Web, we want a credentialing system that can bypass the ‘gatekeepers’ to learning. We want better ways to credential experiences, knowledge, interest and skills.

Are all Open Badges public?

They can be. By default when they’re issued, Open Badges are private and can only be seen by an earner who has accepted the badge and placed it in their badge backpack. Once added to a collection (named by the learner) they can optionally be made public and displayed across the Web.

What’s the difference between a ‘digital’ badge and an ‘Open’ badge?

It’s very simple, but with fairly profound consequences. An Open Badge is a digital image that has metadata ‘baked’ into it. So in the same way that you bake ingredients together to make a cake, so you bake a badge. And again, just as you can’t then remove an ingredient from the baked cake so you can’t change an Open Badge once it’s been ‘baked’.

Does Mozilla ‘police’ Open Badges?

We’re looking after the ‘plumbing’ of the Open Badges Infrastructure (OBI). Our focus is upon the technical standards underpinning the whole ecosystem, not the pedagogical or social validity of badges. Some Open Badges will be frivolous and playful. Others will be rigorous and pedagogically sound. All of them will be technically valid badges. The value of a badge comes through a mixture of the reputation of the issuer and the rigour of the criteria for obtaining the badge.

What happens if I invest time in Open Badges and then Mozilla pull the plug?

We’re a non-profit who work (radically) in the open alongside the community. The OBI is a Mozilla product, but it’s more of a model where we’re the lead developers and advocates than having something than can be ‘pulled’. We’re committed to OBI for the long-haul, but even if we were all on several planes that crashed the Open Source community could still develop the infrastructure.

How can Mozilla maintain the quality of Open Badges?

‘Quality’ is an interesting word. Another variation of this question is How can Mozilla guarantee equivalency between badges? The short answer is, of course, that we can’t. That’s because we’re the ones developing the technical standard, but not those that are developing all of the badges within the ecosystem.

The OBI is a platform for innovation. We’ve already seen many high-quality badges that have been produced by lots of different organisations. But, of course, there will be poor badges. The value of the badge doesn’t come through how difficult it is to issue them, but upon the rigour of what you have to do to get them, and the evidence they point to. That’s within the metadata in the badge itself.

CC BY Kyle Bowen

One of the newer metadata fields that’s available within the OBI is a field that allows you to enter the URL of which standard you’re aligning to. So whether it’s a badge that aligns with the Common Core or the Web Literacy standard, there’s something you can point to as a common reference point. The ‘endorsement’ functionality that we’re working upon could then allow organisations to endorse certain badges as being good/valid representations of that standard.

What’s the quickest way of getting started issuing Open Badges?

There’s broadly three ways to start issuing badges. The first is to use a third-party badge issuing platform such as badg.us, forallbadges or openbadges.me. This is the easiest, but the URLs in the metadata of the badge point towards that third-party platform over which you have no control.

The second way to issue badges is to use a plugin for a popular Content Management System or learning platform such as WordPress, Drupal or Moodle. Doing this means that you don’t have to do any coding but the URLs in the Open Badges point back to your domain.

The third way is the most complex and involves being (or hiring) a developer and using Mozilla’s onboarding documentation to build your own badge issuing platform or plugin. Apparently it’s not that hard, but I haven’t tried it.

What happens when there’s millions of Open Badges in the ecosystem and everyone has thousands of them?

Well, first of all that will be awesome! The great thing about Open Badges is that the learner is always in control. That means you can choose which badges to display for what purpose. So, if you want to show all of your gamer and photography badges on Facebook and your professional badges on your online portfolio, you can.

The other thing to remember is that an Open Badge does not stand alone, but is part of a wider ecosystem of value. One of the best ways of imagining this is through badge-based learning pathways. In the same way that you collect cheeses/pies in Trivial Pursuit, so badges can work together to unlock a larger, meta-level badge. Once you’ve unlocked your competency-level badge, it would point back to the five skill-level badges of which it’s comprised.

How can we trust an Open Badge? How do I know someone hasn’t just bought one?

Both very good questions. A combination of the Criteria URL and the Evidence URL should help with this, I think. The (compulsory) Criteria URL states what the earner had to do in order to be issued the badge, and the (optional, but to my mind very important) Evidence URL points to work done in order to get the badge. This is anything that can be displayed on the Web – images, text, videos, etc.

Do people buy qualifications now? Of course they do. Will people attempt to (and sometimes be successful in purchasing) Open Badges? Almost definitely. But the difference between traditional qualifications and credentials, and Open Badges is that the latter leave a breadcrumb trail of evidence. My Great Uncle built his entire adult life on a the claim that he attended Oxford University. After his death we found this to be false. That wouldn’t really have been possible in a badge-based system. He would have been found out very quickly!

Why are Open Badges any more than stickers? Aren’t they just extrinsic motivators?

As stated above, the value of an Open Badge comes through the metadata contained. Learning design is the hard part of creating an ecosystem of badges; it’s the 90% of the iceberg you don’t see. So, of course Open Badges can be used to extrinsically motivate. But, like all credentialing systems, if designed well then they can also promote intrinsic motivation.*

*My rejection of intrinsic/extrinsic motivation as a binary will have to wait for another blog post…

How can issuers ensure their badges are taken as seriously as possible by institutions/employers?

An Open Badge is a metadata-infused credential. Whether badges are taken seriously depends on how trustworthy, relevant and useful they are to others. That’s a function of the reputation of the badge issuer but also on the rigour of the Criteria URL. What did the individual have to do to get the badge? Was that worth doing? Is there an Evidence URL pointing to what that individual actually did?

It’s a fact of life that people like (and trust) good-looking things so it’s worth spending some time on the visual design of the badge. But that’s the tip of the iceberg: it’s the learning design, the partnerships and the thinking through how individuals can ‘level up’ that’s important. DigitalMe have a great CC-licensed badge canvas resource to help you think through some of these things.

Finally, it’s worth having a useful way to display badges to institutions and employers. Purdue University, for example, have an iPad app that students can use to show their badges at interview. Badges can also be displayed on pretty much any kind of website, including e-portfolios and wikis.

Why would I want an Open Badge instead of a degree?

This is the $64,000 question, but one that misses the point in the short term. Often when a new technology comes along we think in terms of either/or. In practice, however, it’s more and/and/and. How can we use Open Badges to credential those things that we think are important but we don’t currently have a way of capturing? How can we make credentialing more granular? How can we make learning more personalised through badge-based ‘playlists’ or ‘pathways’? These are the questions which interest us a lot more than ‘Can X replace Y?’

Have you got any more questions? Ask away below! (or on Twitter / Facebook / Google+) If I get enough I’ll probably do another one of these in a few weeks’ time.

Header image CC BY Bilal Kamoon

This is a ‘how to’ for battery threshold setting on Lenovo laptops with Windows 8 system when your battery recharges fully after the system restart.

I’ve recently updated my ancient Lenovo laptop to Windows 8.  To prolong lifespan of my battery I had to setup again the charge thresholds to not start charging the battery sooner then bellow 5% and stop up at 100% using PowerManager 6.36 with 1.66.0.22 PowerManager driver.

However, after system restart the battery started to charge fully every time – a way to significantly shorten it’s lifespan.

To fix it I did some search.  There are forum posts how to fix charge thresholds manually.  However, it didn’t work for me.  Changing ChargeStartControl, ChargeStartPercentage, ChargeStopControl, ChargeStopPercentage registry keys under HKEY_LOCAL_MACHINE\SOFTWARE\Lenovo\PWRMGRV\Data didn’t help.

The correct registry settings are located elsewhere:

HKEY_CURRENT_USER\Software\Lenovo\PWRMGRV\Data\<eleven alphanums code>\

The “code” seems to be something random and looks e.g. like 1983HD83HM7. This is just an example, it will for sure be different on your machine!

You may need to manually create any missing DWORD values, in my case those are (in hex):
ChargeStartControl: 0×00000001
ChargeStartPercentage: 0×00000004
ChargeStopControl: 0×00000001
ChargeStopPercentage: 0×00000064

After you do the changes, restart your machine.

These settings then produce the following status that persists after system restart:

I have been hard at work getting our H.264 support on Windows Vista and later hardware accelerated using DXVA2. I'm happy to say that this finally landed in Firefox 23 Nightly builds. With this patch we'll use the GPU to accelerate H.264 video decoding when possible on Windows, which greatly reduces our CPU and power usage.

If you spot a bug in H.264 video in Firefox Nightly builds, please file a bug in Core: Audio/Video. Bonus points if you toggle the pref "media.windows-media-foundation.use-dxva" to false, reload the video, and tell me whether the bug still happens!

Notes:

1. H.264/AAC/MP3 support on Windows 7 and later is shipping in Release builds in Firefox 21 next week, Vista gets it in Firefox 22.
2. Edwin Flores is working on our H.264/AAC/MP3 support on Mac and Linux.
3. I'm currently working on getting MP3 support for Windows XP using DirectShow, it will probably land next cycle (Firefox 24).

Among the build systems peer, I am very much a newcomer. Despite working with Thunderbird for over 5 years, I've only grown to understand the comm-central build system in gory detail in the past year. Most of my work before then was in trying to get random projects working; understanding it more thoroughly is a result of attempting to eliminate the need for comm-central to maintain its own build system. The goal of moving our build system description to a series of moz.build files has made this task much more urgent.

At a high level, the core of the comm-central build system is not a single build system but rather three copies of the same build system. In simple terms, there's a matrix on two accesses: which repository does the configuration of code (whose config.status invokes it), and which repository does the build (whose rules.mk is used). Most code is configured and built by mozilla-central. That comm-central code which is linked into libxul is configured by mozilla-central but built by comm-central. tier_app is configured and built by comm-central. This matrix of possibilities causes interesting bugs—like the bustage caused by the XPIDLSRCS migration, or issues I'm facing working with xpcshell manifests—but it is probably possible to make all code get configured by mozilla-central and eliminate several issues for once and all.

With that in mind, here is a step-by-step look at how the amorphous monster that is the comm-central build system works:

python client.py checkout

And comm-central starts with a step that is unknown in mozilla-central. Back when everyone was in CVS, the process of building started with "check out client.mk from the server, set up your mozconfig, and then run make -f client.mk checkout." The checkout would download exactly the directories needed to build the program you were trying to build. When mozilla-central moved to Mercurial, the only external projects in the tree that Firefox used were NSPR and NSS, both of which were set up to pull from a specific revision. The decision was made to import NSPR and NSS as snapshots on a regular basis, so there was no need for the everyday user to use this feature. Thunderbird, on the other hand, pulled in the LDAP code externally, as well as mozilla-central, while SeaMonkey also pulls in the DOM inspector, Venkman, and Chatzilla as extensions. Importing a snapshot was not a tenable option for mozilla-central, as it updates at an aggressive rate, so the functionality of checkout was ported to comm-central in a replacement python fashion.

./configure [comm-central]

The general purpose of configure is to discover the build system and enable or disable components based on options specified by the user. This results in a long list of variables which is read in by the build system. Recently, I changed the script to eliminate the need to port changes from mozilla-central. Instead, this script reads in a few variables and tweaks them slightly to produce a modified command line to call mozilla-central's configure...

./configure [mozilla-central]

... which does all the hard work. There are hooks in the configure script here to run a few extra commands for comm-central's need (primarily adding a few variables and configuring LDAP). This is done by running a bit of m4 over another file and invoking that as a shell script; the m4 is largely to make it look and feel "like" autoconf. At the end of the line, this dumps out all of the variables to a file called config.status; how these get configured in the script is not interesting.

./config.status [mozilla/comm-central]

But config.status is. At this point, we enter the mozbuild world and things become very insane; failure to appreciate what goes on here is a very good way to cause extended bustage for comm-central. The mozbuild code essentially starts at a directory and exhaustively walks it to build a map of all the code. One of the tasks of comm-central's configure is to alert mozbuild to the fact that some of our files use a different build system. We, however, also carefully hide some of our files from mozbuild, so we run another copy of config.status again to add in some more files (tier_app, in short). This results in our code having two different notions of how our build system is split, and was never designed that way. Originally, mozilla-central had no knowledge of the existence of comm-central, but some changes made in the Firefox 4 timeframe suddenly required Thunderbird and SeaMonkey to link all of the mailnews code into libxul, which forced this contorted process to come about.

make

Now that all of the Makefiles have bee generated, building can begin. The first directory entered is the top of comm-central, which proceeds to immediately make all of mozilla-central. How mozilla-central builds itself is perhaps an interesting discussion, but not for this article. The important part is that partway through building, mozilla-central will be told to make ../mailnews (or one of the other few directories). Under recursive make, the only way to "tell" which build system is being used is by the directory that the $(DEPTH) variable is pointing to, since $(DEPTH)/config/config.mk and $(DEPTH)/config/rules/mk are the files included to get the necessary rules. Since mozbuild was informed very early on that comm-central is special, the variables it points to in comm-central are different from those in mozilla-central—and thus comm-central's files are all invariably built with the comm-central build system despite being built from mozilla-central.

However, this is not true of all files. Some of the files, like the chat directory are never mentioned to mozilla-central. Thus, after the comm-central top-level build completes building mozilla-central, it proceeds to do a full build under what it thinks is the complete build system. It is here that later hacks to get things like xpcshell tests working correctly are done. Glossed over in this discussion is the fun process of tiers and other dependency voodoo tricks for a recursive make.

The future

With all of the changes going on, this guide is going to become obsolete quickly. I'm experimenting with eliminating one of our three build system clones by making all comm-central code get configured by mozilla-central, so that mozbuild gets a truly global view of what's going on—which would help not break comm-central for things like eliminating the master xpcshell manifest, or compiling IDLs in parallel. The long-term goal, of course, is to eliminate the ersatz comm-central build system altogether, although the final setup of how that build system works out is still not fully clear, as I'm still in the phase of "get it working when I symlink everything everywhere."

Among the build systems peer, I am very much a newcomer. Despite working with Thunderbird for over 5 years, I've only grown to understand the comm-central build system in gory detail in the past year. Most of my work before then was in trying to get random projects working; understanding it more thoroughly is a result of attempting to eliminate the need for comm-central to maintain its own build system. The goal of moving our build system description to a series of moz.build files has made this task much more urgent.

At a high level, the core of the comm-central build system is not a single build system but rather three copies of the same build system. In simple terms, there's a matrix on two accesses: which repository does the configuration of code (whose config.status invokes it), and which repository does the build (whose rules.mk is used). Most code is configured and built by mozilla-central. That comm-central code which is linked into libxul is configured by mozilla-central but built by comm-central. tier_app is configured and built by comm-central. This matrix of possibilities causes interesting bugs—like the bustage caused by the XPIDLSRCS migration, or issues I'm facing working with xpcshell manifests—but it is probably possible to make all code get configured by mozilla-central and eliminate several issues for once and all.

With that in mind, here is a step-by-step look at how the amorphous monster that is the comm-central build system works:

python client.py checkout

And comm-central starts with a step that is unknown in mozilla-central. Back when everyone was in CVS, the process of building started with "check out client.mk from the server, set up your mozconfig, and then run make -f client.mk checkout." The checkout would download exactly the directories needed to build the program you were trying to build. When mozilla-central moved to Mercurial, the only external projects in the tree that Firefox used were NSPR and NSS, both of which were set up to pull from a specific revision. The decision was made to import NSPR and NSS as snapshots on a regular basis, so there was no need for the everyday user to use this feature. Thunderbird, on the other hand, pulled in the LDAP code externally, as well as mozilla-central, while SeaMonkey also pulls in the DOM inspector, Venkman, and Chatzilla as extensions. Importing a snapshot was not a tenable option for mozilla-central, as it updates at an aggressive rate, so the functionality of checkout was ported to comm-central in a replacement python fashion.

./configure [comm-central]

The general purpose of configure is to discover the build system and enable or disable components based on options specified by the user. This results in a long list of variables which is read in by the build system. Recently, I changed the script to eliminate the need to port changes from mozilla-central. Instead, this script reads in a few variables and tweaks them slightly to produce a modified command line to call mozilla-central's configure...

./configure [mozilla-central]

... which does all the hard work. There are hooks in the configure script here to run a few extra commands for comm-central's need (primarily adding a few variables and configuring LDAP). This is done by running a bit of m4 over another file and invoking that as a shell script; the m4 is largely to make it look and feel "like" autoconf. At the end of the line, this dumps out all of the variables to a file called config.status; how these get configured in the script is not interesting.

./config.status [mozilla/comm-central]

But config.status is. At this point, we enter the mozbuild world and things become very insane; failure to appreciate what goes on here is a very good way to cause extended bustage for comm-central. The mozbuild code essentially starts at a directory and exhaustively walks it to build a map of all the code. One of the tasks of comm-central's configure is to alert mozbuild to the fact that some of our files use a different build system. We, however, also carefully hide some of our files from mozbuild, so we run another copy of config.status again to add in some more files (tier_app, in short). This results in our code having two different notions of how our build system is split, and was never designed that way. Originally, mozilla-central had no knowledge of the existence of comm-central, but some changes made in the Firefox 4 timeframe suddenly required Thunderbird and SeaMonkey to link all of the mailnews code into libxul, which forced this contorted process to come about.

make

Now that all of the Makefiles have bee generated, building can begin. The first directory entered is the top of comm-central, which proceeds to immediately make all of mozilla-central. How mozilla-central builds itself is perhaps an interesting discussion, but not for this article. The important part is that partway through building, mozilla-central will be told to make ../mailnews (or one of the other few directories). Under recursive make, the only way to "tell" which build system is being used is by the directory that the $(DEPTH) variable is pointing to, since $(DEPTH)/config/config.mk and $(DEPTH)/config/rules/mk are the files included to get the necessary rules. Since mozbuild was informed very early on that comm-central is special, the variables it points to in comm-central are different from those in mozilla-central—and thus comm-central's files are all invariably built with the comm-central build system despite being built from mozilla-central.

However, this is not true of all files. Some of the files, like the chat directory are never mentioned to mozilla-central. Thus, after the comm-central top-level build completes building mozilla-central, it proceeds to do a full build under what it thinks is the complete build system. It is here that later hacks to get things like xpcshell tests working correctly are done. Glossed over in this discussion is the fun process of tiers and other dependency voodoo tricks for a recursive make.

The future

With all of the changes going on, this guide is going to become obsolete quickly. I'm experimenting with eliminating one of our three build system clones by making all comm-central code get configured by mozilla-central, so that mozbuild gets a truly global view of what's going on—which would help not break comm-central for things like eliminating the master xpcshell manifest, or compiling IDLs in parallel. The long-term goal, of course, is to eliminate the ersatz comm-central build system altogether, although the final setup of how that build system works out is still not fully clear, as I'm still in the phase of "get it working when I symlink everything everywhere."

A zase rok v pr..li

 

Lo?ský ro?ník Million Marihuana March 2012 se moc nelišil, jen možná více lepšího vegan a vegetariánského jídla.

 

Longtime readers of The Sober Build Engineer may find today’s XKCD amusing1.

I get asked a lot of times why this blog tends to rely heavily on footnotes; it turns out it’s mostly a historical footnote2.

The original incarnation of this blog started back when I was one of Mozilla Corporation’s two3 full-time release engineers. Since a lot of the community communication regarding builds came through that blog at the time, a lot of the funnier content4 had to be relegated down to the footnotes.

And then it just sort of snowballed from there; it turns out people found them amusing and missed them; they’re also a great tool for citations, without being too intrusive.

So yeah… they’re here to stay… for now.

_______________
1 Thanks to Redfive for calling it out!?
2 See what I did there??
3 And then only one… and then two again?
4 Or stuff that were inside jokes anyway?

This is my regularly scheduled post summarizing performance work from the past two weeks. Alternate title: Vlad’s Big Bowl of Performance Chilli

The Performance team had its first monthly status meeting. We decided on projects and set goals & timelines: wiki. The next meeting is on Thursday, June 6th @ 11am PDT (Vidyo room “Performance”), people from other teams who are working on related projects will be invited.

Main thread I/O continues to be a major source of Firefox jank. To illustrate this point, I ran Nightly 23 with its profile stored on an SD card and captured a screen recording. The results were not pretty, as Firefox hung repeatedly during common actions (see blog post). Patrick McManus posted a band-aid patch (bug 868441) that will allow Firefox to by-pass the network cache when locking in the network cache is taking too long. The long-term solution is a network cache re-design. Aaron Klotz and Joel Maher are working on detecting when new sources of main-thread I/O are added to the code in our test environment (bug 644744).

Drew Willcoxon wrote a patch to capture page thumbnails (for about:home) in the background (bug 841495). Once it’s hooked up, this will move the thumbnailing operation off the main thread and will allow Firefox to take snapshots of sites loaded without cookies to avoid capturing sensitive data.

Other fixes:

• bug 852467: nsDisableOldMaxSmartSizePrefEvent runs on the gecko main thread, blocks for long periods of time
• bug 649216: Remove unnecessary delay when clicking tab close buttons sequentially
• bug 699331: Reduce impact of font name enumeration at startup

The team blogged about progress on their longer-term projects:

• Julian Seward: Profiler backend news
• Avi Halchmi: Tabstrip Animation Progress
• Irving Reid: Add-On Manager Progress
• David Teller: Project “Async & Responsive” Progress

We’ve been doing some strategy and planning work for Open Badges and the general badging work, and one thing that became clear was that we have some implicit values and principles that we have carried and/or want to carry with us into the future work.

We decided to write them down and turns out, that process alone was valuable for our strategy work. They have already been useful as a guide on our strategy and a litmus test for possible directions or opportunities. And that’s after just throwing them up on the fly. We want feedback and thoughts so that we can refine and agree on these, and in the process, set a direction that we can all work towards together.

Open Badges Values / Principles:

• Empower the learner. The end game is about helping learners improve their lives, get credit for what they do, and give them the data/ammunition necessary to do the things that they want to do. There are other ways we’ve talked about this - redefining learning, rethinking accreditation, but ultimately its about putting the learner in the driver’s seat.
• Agency. This is similar to the above and is specifically about control. The learner should control their data. They should control the interactions around that data. They should be able to collect and share any badges they want, even “smaller” or social ones that might mean something to them. They should decide who sees badges or what stories they want to tell about themselves (through the badges). 
• Open. This is a loaded word, but its important in every meaning of the word. Badges should remain open in that anyone should be able to issue them. Many ask to restrict what can be badged so that its easier to establish equivalencies but that means we are restricting the possibilities for learners. The onus is on us to figure out how to make sense of that data. There should also be tools to support badging that are free and open source. As mentioned before, no proprietary or closed system should control the badges, the learner should. Open, open, open.
• Interoperable. A single badge might carry some value in some contexts, but a group of badges that tell a more complete story about a learner is so much more powerful. Especially when those badges are earned across experiences. This requires that badges be interoperable. This requires that badges align with the open standard. If we can have consensus at that lower level, then anyone can build tools on top of badges to make them more useable, more shareable, more valuable, etc.
• Distributed. We are working towards a more distributed ecosystem of recognition. That means each touchpoint in the ecosystem should be distributed - issuing, validating/endorsing, sharing, using badges, etc. Badges should be and go where the user is, and the badge information and value should follow. 
• Credible. We think badges can be the real deal - can lead to real results like jobs and credit and advancement. We need to continually think about what gets badges to these standards without squelching the other features of badges. I have some thoughts on that here.
• Flexible/Innovative. (or “Weird.”) At the same time, we need to “keep digital badges weird”. We shouldn’t force all badges to be a one level or for one particular goal, we should build tools and frameworks to allow for innovative uses for badges.
• Community-driven. The community is gold. We can’t do this alone, you can’t do this alone. We are stronger together and a community that shares resources and findings, vets ideas and builds this stuff together is the community that wins. Our community is the lifeblood of the badges work and we need to codesign our future together. (*hugs!*)
• Something we are proud of. We are those feel-goody people that want to be proud of what we do. This means both not being evil, and also producing high quality stuff. On the former, I think we’re doing pretty well already but there is real risk of closed solutions segmenting or threatening the ecosystem and we should fight against this. On the latter, from the conceptual framework and the whitepapers, to the software and technical framework, to the toolkits and implementations, we want to walk away proud. There is a lot that we are proud of but turns out that this is pretty challenging to do all the time when there are so many moving pieces. But its a standard that we should all hold ourselves to and find ways to get there together. 

What are we actively working against?

This list looks like the opposites of all of the above, but a few highlights:

• Data about the learner not for the learner. In our recent offsite, @iamjessklein had a revelation that most, if not all, of the data about learning out there is not for the learner. That’s really broken. 
• Spy-ware. There’s a surge of attention around scouring the web to determine things about individuals or ‘score’ them, and then selling that information to employers. The individuals probably have no idea that this is happening. There is certainly some value in some cases, like the one in this recent NYT article, where some unsuspecting individual is rewarded for previous work or interaction with a job offer. But in most cases, its just spying and making decisions about people without giving them a chance to have their say. Badges should be all about giving people their say - letting them tell the story that they want to tell, but in an evidence-based, verified way.
• Replicating accreditation. A centralized system or body for judging or OKing badges would be bad for badges. If we are embracing open and distributed, as I hope we are, we need to find and open and distributed way to build trust and assurance into badges. I’ve written more about this here [referenced above].
• Closed and siloed. If badges do not meet the open standard or are stored in a system that is closed, we lose the real power of the ecosystem. To empower the learners, we need to let them have access to the broader ecosystem, craft their own pathways and write their own stories without predetermining the set they can work from or the constraints they are under.

We have a real opportunity for change and impact with the badges work, but we feel that the values/principles are important pieces for realizing that opportunity. As you can see from the list above, these principles do not define the solutions - there are still a lot of things to answer or figure out, but knowing the guiding principles can in some cases make those answers easy or keep us on course as we start to tackle them.

I’m sure the list is too long - maybe 3-5 is the magic number. Let us know what you think. Comment below or join us on the community call tomorrow to dig in.

-E

Webdev, as well as many other teams at Mozilla, create awesome web services. I think we need to be creating even more services, so that we can isolate certain responsibilities and reuse them throughout our community and internal tools.

One missing piece is a consistent way to secure our web services.

Let’s look at the Mozillians list users API. It was written by Giorgos Logiotatidis, who is awesome and has been kicking all sorts of ass on the Mozillians codebase.

This API allows you to lock down your website based on if someone is a vouched Mozillian. You can check to see if they are in the group “staff”, etc. Awesome stuff!

How does one get to use this API? How is it secured? The API uses a very simple authentication scheme.

1. You register an API key for your application
2. You send your App Name and App Key in your API call
3. You make your web service calls over HTTPS

Essentially, we’re sending the password over SSL. We can do better!

I try not to promote Golden Hammers, but Eran Hammer’s new protocol seems quite appropriate.

Eran has created an Authentication system called Hawk, that is exactly built for this use case. Erin lead the OAuth specification and is now developing a couple new authentication systems. Hawk and (the still under development) Oz.

Ben Adida suggested using Hawk in a recent bug related to PeterB‘s MozLDAP web service.

So, what is Hawk?

According to the README, Hawk is an HTTP authentication scheme using a message authentication code (MAC) algorithm to provide partial HTTP request cryptographic verification.

So Hawk is a standard way for clients to sign their requests and verify that the response is from the intended server. It allows servers a standard way to verify a request is authentic and to digitally sign it’s response.

Much like the Mozillians API, there is an App ID and a App Secret, but the secret is never sent over the wire. This makes Hawk much better than Basic Digest or an ad hoc method where the secret is exposed.

What’s this partial in “partial HTTP request cryptographic verification” business? Only certain aspects of the HTTP request / response are used in the signing process; making it compatible with HTTP proxies and the real world internet.

The cryptographic verification includes the HTTP method, request URI, host, and optionally the request payload.

Signing involves calculating a request MAC value. There are libraries that hide the complexities of signing and verifying requests.

In addition to not sending the secret, there are more defense in depth features we can use:

• Timestamp – expires requests
• Nonce – mitigates replay attacks

This all sounds rather complicated, but good library support makes using Hawk easy. In my next post, I’ll show examples.

At first blush, Hawk seems like a good system to standardize Mozilla web services on.

What do you think?

• Do you think we should standardize API authentication?
• Do you have a different golden hammer?

In my last post “Proposal for Securing Mozilla Web Services with Hawk”, I recommended using Hawk.

I know what your saying…

“Okay, armchair architect, I’ve read Hawk’s README, it sounds great, but there is no Django or Python port.”

PyHawk is a new Python port of the Node.js Hawk codebase.

It’s here today and seems to work. It is interoperable with the latest version of Hawk (0.13).

The following API is TBD, because I want to work with you to make it more elegant, but here is a the current usage…

Clients

A website can consume web services that are locked down with Hawk as follows:

Server

Similarly, it is fairly easy to lock down your web service APIs.

Conclusion

Hawk is pretty darn simple, right? Be sure to read the original project for caveats and best usage.

So this homely API works, but I need your help:

• My Python is rusty, help make the code beautiful through Issues or Pull Requests
• I need you to add it to your Python pet projects and kick the tires
• Let’s make the API elegant
• More/better unit tests
• Read the docs, Travis, and other polishing

Who’s in? Let’s play in the PyHawk repo.