This is our weekly gathering of Mozilla'a Web QA team filled with discussion on our current and future projects, ideas, demos, and fun facts.
This is my third Firefox release as release manager, and the fifth that I’ve followed closely from the beginning to the end of the release cycle. (31 and 36 as QA lead; 39, 43, and 46 as release manager.) This time I felt more than usually okay with things, even while there was a lot of change in our infrastructure and while we started triaging and following even more bugs than usual. No matter how on top of things I get, there is still chaos and things still come up at the last minute. Stuff breaks, and we never stop finding new issues!
I’m not going into all the details because that would take forever and would mostly be me complaining or blaming myself for things. Save it for the post-mortem meeting. This post is to record my feeling of accomplishment from today.
During the approximately 6 week beta cycle of Firefox development we release around 2 beta versions per week. I read through many bugs nominated as possibly important regressions, and many that need review and assessment to decide if the benefit of backporting warrants the risk of breaking something else.
During this 7 week beta cycle I have made some sort of decision about at least 480 bugs. That usually means that I’ve read many more bugs, since figuring out what’s going on in one may mean reading through its dependencies, duplicates, and see-alsos, or whatever someone randomly mentions in comment 45 of 96.
And today I got to a point I’ve never been at near the end of a beta cycle: Zarro Boogs found!
This is what Bugzilla says when you do a query and it returns 0. I think everyone likes saying (and seeing) “Zarro Boogs”. Its silliness expresses the happy feeling you get when you have burned down a giant list of bugs.
This particular query is for bugs that anyone at all has nominated for the release management team to pay attention to.
Here is the list of requests for uplift (or backporting, same thing) to the mozilla-beta repo:
Yes!! Also zarro boogs.
Since we build our release candidate a week (or a few days) from the mozilla-release repo, I check up on requests to uplift there too:
PEAK ZARRO BOOGS.
For the bugs that are unresolved and that I’m still tracking into the 46 release next week, it’s down to 4: Two fairly high volume crashes that may not be actionable yet, one minor issue in a system addon that will be resolved in a planned out-of-band upgrade, and one web compatibility issue that should be resolved soon by an external site. Really not bad!
Our overall regression tracking has a release health dashboard on displays in many Mozilla offices. Blockers, 0. Known new regressions that we are still working on and haven’t explicitly decided to wontfix: 1. (But this will be fixed by the system addon update once 46 ships.) Carryover regressions: 41; about 15 of them are actually fixed but not marked up correctly yet. The rest are known regressions we shipped with already that still aren’t fixed. Some of those are missed uplift opportunities. We will do better in the next release!
In context, I approved 196 bugs for uplift during beta, and 329 bugs for aurora. And, we fix several thousands of issues in every release during the approx. 12 week development cycle. Which ones of those should we pay the most attention to, and which of those can be backported? Release managers act as a sort of Maxwell’s Demon to let in only particular patches …
Will this grim activity level for the past 7 weeks and my current smug feeling of being on top of regression burndown translate to noticeably better “quality”… for Firefox users? That is hard to tell, but I feel hopeful that it will over time. I like the feeling of being caught up, even temporarily.
Here I am with drink in hand on a sunny afternoon, toasting all the hard working developers, QA testers, beta users, release engineers, PMs, managers and product folks who did most of the actual work to fix this stuff and get it firmly into place in this excellent, free, open source browser. Cheers!Related posts:Kiva lending and people with disabilitiesBugzilla hijinks, Tuesday March 5
Some time around 4 weeks ago, a few of us got together to investigate what it would take to implement the Electron API on top of Gecko. Electron consists of two parts: a Node environment with a few additional Node modules, and a lightweight embedding API for opening windows that point to a local or remote web page in order to display UI. Project Positron tries to create an Electron compatible runtime built on Mozilla technology stack, that is, Gecko and SpiderMonkey.
While a few of my colleagues are busy working on Positron itself, I have been working on SpiderNode, which is intended to be used in Positron to implement the Node part of the Electron API. SpiderNode has been changing rapidly since 3 weeks ago when I made the initial commit.
SpiderNode is still in its early days, and is not yet complete. As such, we still can’t link the Node binary successfully since we’re missing quite a few V8 APIs, but we’re making rapid progress towards finishing the V8 APIs used in Node. If you’re curious to look at the parts of the V8 API that have been implemented so far, check out the existing tests for spidershim.
I have tried to fix the issues that new contributors to SpiderNode may face. As things stand right now, you should be able to clone the repository and build it on Linux and OS X (note that as I said earlier we still can’t link the node binary, so the build won’t finish successfully, see README.md for more details). We have continuous integration set up so that we don’t regress the current state of the builds and tests. I have also written some documentation that should help you get started!
Please see the current list of issues if you’re interested to contribute to SpiderNode. Note that SpiderNode is under active development, so if you’re considering to contribute, it may be a good idea to get in touch with me to avoid working on something that is already being worked on!
mconley livehacks on real Firefox bugs while thinking aloud.
I post these updates every 3 weeks to inform add-on developers about the status of the review queues, add-on compatibility, and other happenings in the add-ons world.The Review Queues
In the past 3 weeks, 902 add-ons were reviewed:
- 846 (94%) were reviewed in fewer than 5 days.
- 27 (3%) were reviewed between 5 and 10 days.
- 29 (3%) were reviewed after more than 10 days.
There are 73 listed add-ons awaiting review.
You can read about the recent improvements in the review queues here.
If you’re an add-on developer and are looking for contribution opportunities, please consider joining us. Add-on reviewers get invited to Mozilla events and earn cool gear with their work. Visit our wiki page for more information.Compatibility Communications
Most of you should have received an email from us about the future compatibility of your add-ons. You can use the compatibility tool to enter your add-on ID and get some info on what we think is the best path forward for your add-on.
To ensure long-term compatibility, we suggest you start looking into WebExtensions, or use the Add-ons SDK and try to stick to the high-level APIs. There are many XUL add-ons that require APIs that aren’t available in either of these options, which is why we’re also asking you to fill out this survey, so we know which APIs we should look into adding to WebExtensions.
The compatibility blog post for 47 is up. The bulk validation will be run soon. Make sure that the compatibility metadata for your add-on is up to date, so you don’t miss these checks.
As always, we recommend that you test your add-ons on Beta and Firefox Developer Edition to make sure that they continue to work correctly. End users can install the Add-on Compatibility Reporter to identify and report any add-ons that aren’t working anymore.Extension Signing
The wiki page on Extension Signing has information about the timeline, as well as responses to some frequently asked questions. The current plan is to remove the signing override preference in Firefox 47 (updated from 46).
This is the sumo weekly call We meet as a community every Wednesday 17:00 - 17:30 UTC The etherpad is here: https://public.etherpad-mozilla.org/p/sumo-2016-04-20
When I started writing my very own password generation extension I didn’t know much about the security aspects. In theory, any hash function should do in order to derive the password because hash functions cannot be reversed, right? Then I started reading and discovered that one is supposed to use PBKDF2. And not just that, you had to use a large number of iterations. But why?Primary threat scenario: Giving away your master password
That’s the major threat with password generators: some website manages to deduce your master password from the password you used there. And once they have the master password they know all your other passwords as well. But how can this happen if hash functions cannot be reversed? Problem is, one can still guess your master password. They will try “password” as master password first — nope, this produces a different password for their site. Then they will try “password1” and get a match. Ok, now they know that your master password is most likely “password1” (it could still be something else but that’s quite unlikely).
Of course, a number of conditions have to be met for this scenario. First, a website where you have an account should be malicious — or simply leak its users database which isn’t too unlikely. Second, they need to know the algorithm you used to generate your password. However, in my case everybody knows now that I’m using Easy Passwords, no need to guess. And even for you it’s generally better if you don’t assume that they won’t figure out. And third, your master password has to be guessable within “finite” time. Problem is, if people start guessing passwords with GPUs most passwords fall way too quickly.
So, how does one address this issue? First, the master password clearly needs to be a strong one. But choosing the right hashing algorithm is also important. PBKDF2 makes guessing hard because it is computationally expensive — depending on the number of iterations generating a single password might take a second. A legitimate user won’t notice this delay, somebody who wants to test millions of guesses however will run out of time pretty quickly.
There are more algorithms, e.g. bcrypt and scrypt are even better. However, none of them found its way into Firefox so far. Since Easy Passwords is using the native (fast) PBKDF2 implementation in Firefox it can use a very high number of iterations without creating noticeable delays for the users. That makes guessing master passwords impractical on current hardware as long as the master password isn’t completely trivial.
Finally, it’s a good measure to use a random salt when hashing passwords — different salts would result in different generated passwords. A truly random salt would usually be unknown to potential attackers and make guessing master passwords impossible. However, that salt would also make recreating passwords on a different device complicated, one would need to back up the salt from the original device and transfer it to the new one. So for Easy Passwords I chose a compromise: the salt isn’t really random, instead the user-defined password name is used as salt. While an attacker will normally be able to guess the password’s name, it still makes his job significantly more complicated.What about other password generators?
In order to check my assumptions I looked into what the other password generators were doing. I found more than twenty password generator extensions for Firefox, and most of them apparently didn’t think much about hashing functions. You have to keep in mind that none of them gained significant traction, most likely due to usability issues. The results outlined in the table below should be correct but I didn’t spend much time figuring out how these extensions work. For a few of them I noticed issues beyond their choice of a hashing algorithm, for others I might have missed these issues.Extension User count Hashing algorithm Security Password Hasher 2491 SHA1 Very weak PwdHash 2325 HMAC+MD5 Very weak1 Hash Password Generator 291 Custom (same as Magic Password Generator) Very weak Password Maker X 276 SHA256/SHA1/MD4/MD5/RIPEMD160, optionally with HMAC Very weak masterpassword for Firefox 155 scrypt, cost parameter 32768, user-defined salt Medium2 uPassword 115 SHA1 Very weak vPass Password Generator 88 TEA, 10 iterations Weak Passwordgen For Firefox 1 77 SHA256 Very weak Phashword 57 SHA-1 Very weak Passera 52 SHA-512 Very weak My Password 51 MD5 Very weak HashPass Firefox 48 MD5/SHA1/SHA256/SHA512 Very weak UniPass 33 SHA-256, 4,096 iterations Weak RndPhrase 29 CubeHash Very weak PasswordProtect 28 SHA1, 10,000 iterations Weak PswGen Toolbar v2.0 24 SHA512 Very weak UniquePasswordBuilder Addon 13 scrypt, cost factor 1024 by default Strong3 hash0 9 PBKDF2+HMAC+SHA256, 100,000 iterations, random salt Very strong4 MS Password Generator 9 SHA1 Very weak Vault 9 PBKDF2+HMAC+SHA1, 8 iterations, fixed salt Weak BPasswd2 8 bcrypt, 64 iterations by default, user-defined salt Weak5 Persistent "Magic" Password Generator 8 MurmurHash Very weak BPasswd 7 bcrypt, 64 iterations Weak SecPassGen 2 PBKDF2+HMAC+SHA1, 10,000 iterations by default Weak6 Magic Password Generator ? Custom Very weak
1 The very weak hash function isn’t even the worst issue with PwdHash. It also requires you to enter the master password into a field on the web page. The half-hearted attempts to prevent the website from stealing that password are easily circumvented.
2 Security rating for masterpassword downgraded because (assuming that I understand the approach correctly) scrypt isn’t being applied correctly. The initial scrypt hash calculation only depends on the username and master password. The resulting key is combined with the site name via SHA-256 hashing then. This means that a website only needs to break the SHA-256 hashing and deduce the intermediate key — as long as the username doesn’t change this key can be used to generate passwords for other websites. This makes breaking scrypt unnecessary, security rating is still “medium” however because the intermediate key shouldn’t be as guessable as the master password itself.
3 Security rating for UniquePasswordBuilder downgraded because of low default cost factor which it mistakenly labels as “rounds.” Users can select cost factor 16384 manually which is very recommendable.
4 hash0 actually went as far as paying for a security audit. Most of the conclusions just reinforced what I already came up with by myself, others were new (e.g. the pointer to window.crypto.getRandomValues() which I didn’t know before).
5 BPasswd2 allows changing the number of iterations, anything up to 2100 goes (the Sun will die sooner than this calculation completes). However, the default is merely 26 iterations which is a weak protection, and the extension neither indicates that changing the default is required nor does it give useful hints towards choosing a better value.
6 Security rating for SecPassGen downgraded because the master password is stored in Firefox preferences as clear text.Additional threats: Shoulder surfing & Co.
Websites aren’t the only threat however, one classic being somebody looking over your shoulder and noting your password. Easy Passwords addresses this by never showing your passwords: it’s either filling in automatically or copying to clipboard so that you can paste it into the password field yourself. In both scenarios the password never become visible.
And what if you leave your computer unattended? Easy Password remembers your master password once it has been entered, this is an important usability feature. The security concerns are addressed by “forgetting” the master password again after a given time, 10 minutes by default. And, of course, the master password is never saved to disk.Usability vs. security: Validating master password
There is one more usability feature in Easy Password with the potential to compromise security. When you mistype your master password Easy Passwords will notify you about it. That’s important because otherwise wrong passwords will get generated and you won’t know why. But how does one validate the master password without storing it?
My initial idea was storing a SHA hash of the master password. Then I realized that it opens the primary threat scenario again: somebody who can get their hands on this SHA hash (e.g. by walking past your computer when it is unattended) can use it to guess your master password. Only store a few characters of the SHA hash? Better but it will still allow an attacker who has both this SHA hash and a generated password to throw away a large number of guesses without having to spend time on calculating the expensive PBKDF2 hash. Wait, why treat this hash differently from other passwords at all?
And that’s the solution I went with. When the master password is set initially it is used to generate a new password with a random salt, using the usual PBKDF2 algorithm. Then this salt and the first two characters of the password are stored. The two characters are sufficient to recognize typos in most cases. They are not sufficient to guess the master password however. And they won’t even provide a shortcut when guessing based on a known generated password — checking the master password hash is just as expensive as checking the generated password itself.Encrypting legacy passwords
One requirement for Easy Passwords was dealing with “legacy passwords,” meaning existing passwords that cannot be changed for some reason. Instead of generating, these passwords would have to be stored securely. Luckily, there is a very straightforward solution: the PBKDF2 algorithm can be used to generate an encryption key. The password is then encrypted with AES-256.
My understanding is that AES-encrypted data currently cannot be decrypted without knowing the encryption key. And the encryption key is derived using the same algorithm as Easy Passwords uses for generating passwords, so the security of stored passwords is identical to that of generated ones. The only drawback of such legacy passwords currently seems to be a more complicated backup approach, also moving the password from one device to another is no longer trivial.Phishing & Co.
Password generators will generally protect you nicely against phishing: a phishing website can look exactly like the original, a password generator will still produce a different password for it. But what about malicious scripts injected into a legitimate site? These will still be able to steal your password. On the bright side, they will only compromise your password for a single website.
Question is, how do malicious scripts get to run there in the first place? One option are XSS vulnerabilities, not much can be done about those. But there are also plenty of websites showing password fields on pages that are transmitted unencrypted (plain HTTP, not HTTPS). These can then be manipulated by an attacker who is in the same network as you. The idea is that Easy Passwords could warn in such cases in future. It should be possible to disable this warning for websites that absolutely don’t support HTTPS, but for others it will hopefully be helpful. Oh, and did I recommend using Enforce Encryption extension already?
Finally, there is the worst-case scenario: your computer could be infected with a password sniffer. This is really bad because it could intercept your master password. Then again, it could also intercept all the individual passwords as you log into the respective websites, it will merely take a bit longer. I think that there is only one effective solution here: just don’t get infected.Other threats?
There are probably more threats to consider that I didn’t think of. It might also be that I made a mistake in my conclusions somewhere. So feel free to post your own thoughts in the comments.
The local osm community in Benin is trying to buy a high-res image of the capital to better map it. They need around 2500€ and have reached 50%. 1′“ days left 5,10, 20 euros would help.
Details http://www.ulule.com/imagerie-cotonou/ .
We’re still trying to figure out what to do with Firefox users on Windows XP.
One option I’ve heard is: Can we just send a Mozillian to each of these users’ houses with a fresh laptop and training in how to migrate apps and data?
( No, we can’t. For one, we can’t uniquely identify who and where these users are (this is by design). For two, even if we could, the Firefox Windows XP userbase is too geographically diverse (as I explained in earlier posts) for “meatspace” activities like these to be effective or efficient. For three, this could be kinda expensive… though, so is supporting extra Operating Systems in our products. )
We don’t have the advertising spend to reach all of these users in the real world, but we do have access to their computers in their houses… so maybe we can inform them that way?
Well, we know we can inform people through their browsers. We have plenty of data from our fundraising drives to that effect… but what do we say?
Can we tell them that their computer is unsafe? Would they believe us if we did?
Can we tell them that their Firefox will stop updating? Will they understand what we mean if we did?
Do these users have the basic level of technical literacy necessary to understand what we have to tell them? And if we somehow manage to get the message across about what is wrong and why, what actions can we recommend they take to fix this?
This last part is the first thing I’m thinking about, as it’s the most engineer-like question: what is the optimal upgrade strategy for these users? Much more concrete to me than trying to figure out wording, appearance, and legality across dozens of languages and cultures.
Well, we could instruct them to upgrade to Linux. Except that it wouldn’t be an upgrade, it’d be a clean wipe and reinstall from scratch: all the applications would be gone and all of their settings would reset to default. All the data on their machines would be gone unless they could save it somewhere else, and if you imagine a user who is running Windows XP, you can easily imagine that they might not have access to a “somewhere else”. Also, given the average level of technical expertise, I don’t think we can make a Linux migration simple enough for most of these users to understand. These users have already bought into Windows, so switching them away is adding complexity no matter how simplistic we could make it for these users once the switch was over.
We could instruct them to upgrade to Windows 7. There is a clear upgrade path from XP to 7 and the system requirements of the two OSes are actually very similar. (Which is, in a sincere hat-tip to Microsoft, an amazing feat of engineering and commitment to users with lower-powered computers) Once there, if the user is eligible for the Windows 10 upgrade, they can take that upgrade if they desire (the system requirements for Windows 10 are only _slightly_ higher than Windows 7 (10 needs some CPU extensions that 7 doesn’t), which is another amazing feat). And from there, the users are in Microsoft’s upgrade path, and out of the clutches of the easiest of exploits, forever. There are a lot of benefits to using Windows 7 as an upgrade path.
There are a few problems with this:
- Finding copies of Windows 7: Microsoft stopped selling copies of Windows 7 years ago, and these days the most reliable way to find a copy is to buy a computer with it already installed. Mozilla likely isn’t above buying computers for everyone who wants them (if it has or can find the money to do so), but software is much easier to deliver than hardware, and is something we already know how to do.
- Paying for copies of Windows 7: Are we really going to encourage our users to spend money they may not have on upgrading a machine that still mostly-works? Or is Mozilla going to spend hard-earned dollarbucks purchasing licenses of out-of-date software for everyone who didn’t or couldn’t upgrade?
- Windows 7 has passed its mainstream support lifetime (extended support’s still good until 2020). Aren’t we just replacing one problem with another?
- Windows 7 System Requirements: Windows XP only needed a 233MHz processor, 64MB of RAM, and 1.5GB of HDD. Windows 7 needs 1GHz, 1GB, and 16GB.
All of these points are problematic, but that last point is at least one I can get some hard numbers for.
We don’t bother asking users how big their disk drives are, so I can’t detect how many users are cannot meet Windows 7’s HDD requirements. However, we do measure users’ CPU speeds and RAM sizes (as these are important for sectioning performance-related metrics. If we want to see if a particular perf improvement is even better on lower-spec hardware, we need to be able to divvy users up by their computers’ specifications).
So, at first this seems like a breeze: the question is simply stated and is about two variables that we measure. “How many Windows XP Firefox users are Stuck because they have CPUs slower than 1GHZ or RAM smaller than 1GB?”
But if you thought that for more than a moment, you should probably go back and read my posts about how Data Science is hard. It turns out that getting the CPU speed on Windows involves asking the registry for data, which can fail. So we have a certain amount of uncertainty.
So, after crunching the data and making some simplifying assumptions (like how I don’t expect the amount of RAM or the speed of a user’s CPU to ever decrease over time) we have the following:
Between 40% and 53% of Firefox users running Windows XP are Stuck (which is to say, they can’t be upgraded past Windows XP because they fail at least one of the requirements).
That’s some millions of users who are Stuck no matter what we do about education, advocacy, and software.
Maybe we should revisit the “Mozillians with free laptops” idea, after all?
How do we know when we are entering a new computing era? One signal is a reemergence of grassroots innovation. Early in a computing era most technical development resources are still focused on sustaining the mature applications and use cases from the waning era or on exploiting attractive transitional technologies.
The first explores of the technologies of a new era are rebels and visionaries operating at the fringes. These explores naturally form grassroots organizations for sharing and socializing their ideas and accomplishments. Such grassroots organizations serve as incubators for the the technologies and leaders of the next era.
The HomeBrew Computing Club was a grassroots group out of which emerged many leaders of the Personal Computing Era. Now, as the Ambient Computing Era progresses, we see grassroots organizations such as the Nodebots movement and numerous collaborative GitHub projects serving a similar role.
Weekly project updates from the Mozilla Connected Devices team.