Mozilla Nederland LogoDe Nederlandse

Abonneren op feed Mozilla planet
Planet Mozilla -
Bijgewerkt: 21 uur 34 min geleden

Tarek Ziadé: Consistent Load Balancing

ma, 16/05/2016 - 19:50

This is an interesting problem to look at. Let's say you have 1 million users and 5 storage servers. You are building an application that needs to pick one of the 5 storage servers depending on the user id.

You want your app to do it as consistently as possible. e.g. a given user should always land on the same backend.

When a request comes in:

  • each request is identified by a unique user id.
  • the app should always call the same storage server given that id.
  • you want all your users to be distributed equally across your storage servers.
  • when a storage server is removed from the list, you want users to stick with the servers they initially got. For users on the removed server, they should be dispatched equally on other servers.
  • when a server is added, you want the minimal numbers of users to be moved around.
  • The application is stateless about this, so when I deploy a new node and give it the list of the storage servers, it should be able to start distributing users among them without prior knowledge.

Point 4 and 6 discards a simple round-robin solution.

The solution to that problem is to build a deterministic function that projects a user id into the space composed of the servers. (yeah well, clustering I guess.)

There are two known algorithms to do that. The consistent hashing algorithm and the rendezvous hashing.

Consistent Hashing

Consistent Hashing is a hashing that can be used to minimize the shuffling of users when a server is removed or added.

This is how it's implemented:

  • each server name is converted into a unique number
  • that number is projected on an modulo interval (a circle)
  • every user is also converted into a unique number and projected on the circle
  • the server that's the closest to the user is picked

If you want nice drawing go here.

This is an elegant solution because removing a server keeps the rest stable, and adding one server shuffles a minimal number of users.

The conversion from a name to a integer is key here: you have to be deterministic but in the same time try to have the numbers randomly and kind-of-evenly distributed on the circle.

Here's how you can do it using MD5:

import hashlib def hash(key): return long(hashlib.md5(key).hexdigest(), 16)

Using a classical hash like MD5 gives us the random part, but depending on the server name you might well end up with two servers that are very close to each other on the circle

And the result will be that when the users are converted into numbers, a very small amount of users will go to some servers.

One solution to fix that is to add replicas: for each server, instead of projecting a single number on the circle, we'll project 100. For example, "server1" becomes "server1:1", "server1:2", .., "server1:100" and those values are transformed into numbers.

Using replicas is very efficient to make sure users are spread evenly.


The other algorithm is called RendezVous and is based on a similar idea where servers are converted into numbers with a hash.

The difference is that instead of projecting servers and their replicas on a circle, the algorithm uses weights. To find which server a user should use, for each combination of server and user, a number is created with a classical hash function.

The server that's picked is the one with the highest number.

The Python code looks like this:

def get_server(user): high_score = -1 winner = None for server in server: score = hash(server + user) if score > high_score: high_score, winner = score, ip elif score == high_score: high_score, winner = score, max(server, winner) return winner

The advantage of this method is that you don't have to create replicas to worry about distribution. In fact, according to my tests, RendezVous is doing a better job than Consistent Hashing for distributing users.

One key decision is to decide which hashing algorithm you want to use.

It's all about the hashing!

Both RendezVous and Consistent hashing are using a classical hashing function to convert the servers and users into numbers - and picking one was not obvious.

I came across this amazing stackexchange post that shed some light on different hashing, their strengths and weaknesses. You should read it, you will learn a lot.

The take away for my experiment was that some hashing functions are doing a better job at randomizing values in the hash space. Some very simple functions are also colliding quite often, which can be a problem.

So I tried a bunch of them and benchmarked them.

It's interesting to note that I had much better results for my use case using RendezVous & sha256 than RendezVous & Murmur, when the latter is usually what people use with RendezVous.

I ended up removing Murmur from my tests, the results where too bad.

Anyways, here's the full implementation I did, based on snippets I found here and there, and the result:

The gist

And the winnner is : RendezVous and sha256

Of course, that entirely depends on how many servers & users you have.

Categorieën: Mozilla-nl planet

Chris Cooper: RelEng & RelOps Weekly highlights - May 16, 2016

ma, 16/05/2016 - 19:18




<marked as spam><deleted>

Modernize infrastructure:

The ongoing work to get TaskCluster building Firefox on Windows reached an important milestone with our first Mozharness based build going green in the TreeHerder dashboard. This represents underlying effort in documenting the dependency chain for Windows builders and producing simple manifests that give greater transparency to changes in this area.

Alin, Amy, and Van brought 192 new OS X 10.10.5 Mac minis online. This should eliminate our Yosemite backlog and allow us to enable more e10s tests.

Q, catlee, and jmaher migrated the first batch of Windows 7 tests to AWS this week. Currently we’re running these suites for all branches of gecko 49 and higher: Web platform tests + reftests, gtest, cppunit, jittest, jsreftest, crashtest. This will reduce our reliance on hardware and allow us to scale dynamically. They are still working on greening larger sets of tests which are more sensitive to their operating environment. Once we have moved a significant portion of tests, we can add additional e10s tests on w7 as well.

Improve Release Pipeline:

In the interest of making progress on migrating Nightly builds to TaskCluster, we had a meeting last week to discuss the security requirements around our nightly release process. Based on the discussions in that meeting, Aki is now iterating on a “one graph” solution for Nightlies (as opposed to a two-graph approach where signing is separate). If this approach works, i.e. we can’t find major security holes in the proposed model, it will simplify our process greatly.

Improve CI Pipeline:

We have achieved our first deprecation milestone in the TaskCluster migration by turning off Linux 64-bit debug builds/tests for aurora/trunk branches in buildbot. These bugs are only generated in TaskCluster now.

Aki added scriptworker to readthedocs:


We had a planned tree closing window over the weekend where, amongst other things, we upgraded our database hosts. It went smoothly, so you probably didn’t even notice.

Our new buildduty associates from Taipei were able to successfully loan machines to developers for the first time last week. Welcome aboard!


Lots of releases published or in-flight: Firefox 47 betas 3-5, Fennec 47.0b4, and Thunderbird 38.8.0 and 45.1.0. As always, you can find more details in our post-mortem wiki pages:

See you next week, freelance writer!

Categorieën: Mozilla-nl planet

Will Kahn-Greene: Anatomy of a class/function decorator and context manager

ma, 16/05/2016 - 18:00

Over the weekend, I wanted to implement something that acted as both a class and function decorator, but could also be used as a context manager. I needed this flexibility for overriding configuration values making it easier to write tests. I wanted to use it in the following ways:

  1. as a function decorator:

    @config_override(DEBUG='False') def test_something(): ...
  2. as a class decorator that would decorate all methods that start with test_:

    @config_override(DEBUG='False') class TestSomething: def test_something(self): ...
  3. as a context manager that allowed for multiple layer of overriding:

    def test_something(): with config_override(DEBUG='False'): with config_override(SOMETHING_ELSE='ou812'): ...

This kind of need comes up periodically, but infrequently enough that I forget how I wrote it the last time around.

This post walks through how I structured it.

Read more… (3 mins to read)

Categorieën: Mozilla-nl planet

Giorgos Logiotatidis: Takis - A util that blocks until a port is open.

ma, 16/05/2016 - 15:20

Over at Mozilla's Engagement Engineering we use Docker to ship our websites. We build the docker images in CI and then we run tests against them. Our tests usually need a database or a cache server which you can get it running simply with a single command:

docker run -d mariadb

The problem is that this container will take some time to initialize and become available to accept connections. Depending on what your test and how you run your tests this delay can cause a test failure to due database connection timeouts.

We used to wait on executing our tests with sleep command but that -besides being an ugly hack- will not always work. For example you may set sleep timeout to 10 seconds and due to CI server load database initialization takes 11 seconds. And nobody wants a non-deterministic test suite.

Meet Takis. Takis checks once per second if a host:port is open. Once it's open, it just returns. It blocks the execution of your pipeline until services become available. No messages or other output to get in the way of your build logs. No complicated configuration either: It reads CHECK_PORT and optionally CHECK_HOST environment variables, waits and eventually returns.

Takis is build using Go and it's fully statically linked as Adriaan explains in this intriguing read. You can download it and directly use it in your scripts

~$ wget ~$ chmod +x takis ~$ CHECK_PORT=3306 ./takis

or use it's super small Docker image

docker run -e CHECK_PORT=3306 -e giorgos/takis

For example here's how we use it to build Snippets Service in TravisCI:

script: - docker run -d --name mariadb -e MYSQL_ALLOW_EMPTY_PASSWORD=yes -e MYSQL_DATABASE=snippets mariadb:10.0 # Wait mariadb to initialize. - docker run --link mariadb:db -e CHECK_PORT=3306 -e CHECK_HOST=db giorgos/takis - docker run --env-file .env --link mariadb:db mozorg/snippets:latest coverage run ./ test

My colleague Paul also build urlwait, a Python utility and library with similar functionality that can be nicely added to your docker-compose workflow to fight the same problem. Neat!

Categorieën: Mozilla-nl planet

Daniel Stenberg: curl user poll 2016

ma, 16/05/2016 - 10:25

It is time for our annual survey on how you use curl and libcurl. Your chance to tell us how you think we’ve done and what we should do next. The survey will close on midnight (central European time) May 27th, 2016.

If you use curl or libcurl from time to time, please consider helping us out with providing your feedback and opinions on a few things:

It’ll take you a couple of minutes and it’ll help us a lot when making decisions going forward. Thanks a lot!

The poll is hosted by Google and that short link above will take you to:

Categorieën: Mozilla-nl planet

This Week In Rust: This Week in Rust 130

ma, 16/05/2016 - 06:00

Hello and welcome to another issue of This Week in Rust! Rust is a systems language pursuing the trifecta: safety, concurrency, and speed. This is a weekly summary of its progress and community. Want something mentioned? Tweet us at @ThisWeekInRust or send us an email! Want to get involved? We love contributions.

This Week in Rust is openly developed on GitHub. If you find any errors in this week's issue, please submit a PR.

This week's edition was edited by: Vikrant and llogiq.

Updates from Rust Community News & Blog Posts New Crates & Project Updates
  • Announcing RustFest Berlin 2016 (17-18 September 2016).
  • hydrogen. Multithreaded, non-blocking Linux server framework in Rust.
  • cargo-profiler. Cargo subcommand to profile binaries.
  • cargo clippy subcommand for convenient production of annoying warnings.
  • rfmt. Rust source code formatter.
  • RustDT 0.6.0 is out with improved build target support and experimental auto-check.
Crate of the Week

This week's Crate of the Week is cargo-profiler, which lets us profile our code directly from cargo using a variety of tools, notably valgrind's callgrind and cachegrind. Thanks to kbknapp for the suggestion!

Submit your suggestions for next week!

Call for Participation

Always wanted to contribute to open-source projects but didn't know where to start? Every week we highlight some tasks from the Rust community for you to pick and get started!

Some of these tasks may also have mentors available, visit the task page for more information.

If you are a Rust project owner and are looking for contributors, please submit tasks here.

Updates from Rust Core

132 pull requests were merged in the last two weeks.

New Contributors
  • billyevans
  • bnewbold
  • Brian Green
  • Cristian Oliveira
  • Dan Fockler
  • Geordon Worley
  • Haiko Schol
  • mrmiywj
  • Pavel Sountsov
  • silvo38
  • Stefan Schindler
  • Steven Burns
Approved RFCs

Changes to Rust follow the Rust RFC (request for comments) process. These are the RFCs that were approved for implementation this week:

Final Comment Period

Every week the team announces the 'final comment period' for RFCs and key PRs which are reaching a decision. Express your opinions now. This week's FCPs are:

No RFCs are currently in final comment period.

New RFCs Upcoming Events

If you are running a Rust event please add it to the calendar to get it mentioned here. Email Erick Tryzelaar or Brian Anderson for access.

fn work(on: RustProject) -> Money

No jobs listed for this week.

Tweet us at @ThisWeekInRust to get your job offers listed here!

Quote of the Week

No quote was selected for QotW.

Submit your quotes for next week!

Categorieën: Mozilla-nl planet

Karl Dubost: [worklog] Make Web sites simpler.

ma, 16/05/2016 - 03:21

Not a song this week, but just a documentary to remind me that some sites are overly complicated and there are strong benefits and resilience in chosing a solid simple framework for working. Not that it makes easier the work. I think it's even the opposite, it's basically harder to make a solid simple Web site. But that the cost is beneficial on the longterm. Tune of the week: The Depth of simplicity in Ozu's movie.

Webcompat Life

Progress this week:

Today: 2016-05-16T10:12:01.879159 354 open issues ---------------------- needsinfo 3 needsdiagnosis 109 needscontact 30 contactready 55 sitewait 142 ----------------------

In my journey in getting the contactready and needscontact lower, we are making progress. You are welcome to participate

Londong agenda.

Reorganizing a bit the wiki so it better aligns with our current work. In Progress.

Good news on the front of appearance in CSS.

The CSSWG just resolved that "appearance: none" should turn checkbox & radio <input> elements into a normal non-replaced element.

Learning on how to do mozregression

We are looking at creating a mechanism similar to Opera browser.js into Firefox. Read and participate to the discussion.

Webcompat issues

(a selection of some of the bugs worked on this week).

Reading List Follow Your Nose TODO
  • Document how to write tests on using test fixtures.
  • ToWrite: rounding numbers in CSS for width
  • ToWrite: Amazon prefetching resources with <object> for Firefox only.


Categorieën: Mozilla-nl planet

Nick Desaulniers: What's in a Word?

ma, 16/05/2016 - 02:58

Recently, there some was some confusion between myself and a coworker over the definition of a “word.” I’m currently working on a blog post about data alignment and figured it would be good to clarify some things now, that we can refer to later.

Having studied computer engineering and being quite fond of processor design, when I think of a “word,” I think of the number of bits wide a processor’s general purpose registers are (aka word size). This places hard requirements on the largest representable number and address space. A 64 bit processor can represent 264-1 (1.8x1019) as the largest unsigned long integer, and address up to 264-1 (16 EiB) different addresses in memory.

Further, word size limits the possible combinations of operations the processor can perform, length of immediate values used, inflates the size of binary files and memory needed to store pointers, and puts pressure on instruction caches.

Word size also has implications on loads and stores based on alignment, as we’ll see in a follow up post.

When I think of 8 bit computers, I think of my first microcontroller: an Arduino with an Atmel AVR processor. When I think of 16 bit computers, I think of my first game console, a Super Nintendo with a Ricoh 5A22. When I think of 32 bit computers, I think of my first desktop with Intel’s Pentium III. And when I think of 64 bit computers, I think modern smartphones with ARMv8 instruction sets. When someone mentions a particular word size, what are the machines that come to mind for you?

So to me, when someone’s talking about a 64b processor, to that machine (and me) a word is 64b. When we’re referring to a 8b processor, a word is 8b.

Now, some confusion.

Back in my previous blog posts about x86-64 assembly, JITs, or debugging, you might have seen me use instructions that have suffixes of b for byte (8b), w for word (16b), dw for double word (32b), and qw for quad word (64b) (since SSE2 there’s also double quadwords of 128b).

Wait a minute! How suddenly does a “word” refer to 16b on a 64b processor, as opposed to a 64b “word?”

In short, historical baggage. Intel’s first hit processor was the 4004, a 4b processor released in 1971. It wasn’t until 1979 that Intel created the 16b 8086 processor.

The 8086 was created to compete with other 16b processors that beat it to the market, like the Zilog Z80 (any Gameboy emulator fans out there? Yes, I know about the Sharp LR35902). The 8086 was the first design in the x86 family, and it allowed for the same assembly syntax from the earlier 8008, 8080, and 8085 to be reassembled for it. The 8086’s little brother (8088) would be used in IBM’s PC, and the rest is history. x86 would become one of the most successful ISAs in history.

For backwards compatibility, it seems that both Microsoft’s (whose success has tracked that of x86 since MS-DOS and IBM’s PC) and Intel’s documentation refers to words still as being 16b. This allowed 16b PE32+ executables to be run on 32b or even 64b newer versions of Windows, without requiring recompilation of source or source code modification.

This isn’t necessarily wrong to refer to a word based on backwards compatibility, it’s just important to understand the context in which the term “word” is being used, and that there might be some confusion if you have a background with x86 assembly, Windows API programming, or processor design.

So the next time someone asks: why does Intel’s documentation commonly refer to a “word” as 16b, you can tell them that the x86 and x86-64 ISAs have maintained the notion of a word being 16b since the first x86 processor, the 8086, which was a 16b processor.

Side Note: for an excellent historical perspective programming early x86 chips, I recommend Michael Abrash’s Graphics Programming Black Book. For instance he talks about 8086’s little brother, the 8088, being a 16b chip but only having an 8b bus with which to access memory. This caused a mysterious “cycle eater” to prevent fast access to 16b variables, though they were the processor’s natural size. Michael also alludes to alignment issues we’ll see in a follow up post.

Categorieën: Mozilla-nl planet

Mark Côté: BMO's database takes a leap forward

ma, 16/05/2016 - 02:52

For historical reasons (or “hysterical raisins” as gps says) that elude me, the BMO database has been in (ughhh) Pacific Time since it was first created. This caused some weirdness on every daylight savings time switch (particularly in the fall when 2:00-3:00 am technically occurs twice), but not enough to justify the work in fixing it (it’s been this way for close to two decades, so that means lots of implicit assumptions in the code).

However, we’re planning to move BMO to AWS at some point, and their standard db solution (RDS) only supports UTC. Thus we finally had the excuse to do the work, and, after a bunch of planning, developing, and reviewing, the migration happened yesterday without issues. I am unreasonably excited by this and proud to have witnessed the correction of this egregious violation of standard db principles 18 years after BMO was originally deployed.

Thanks to the BMO team and the DBAs!

Categorieën: Mozilla-nl planet

The Servo Blog: This Week In Servo 63

ma, 16/05/2016 - 02:30

In the last week, we landed 156 PRs in the Servo organization’s repositories.

Emily Dunham has been working on an org-wide report of the contributor metrics. It’s looking good already, but there is still a lot of fun python work to do on the scripts, for those who want to make sure we’re not counting things incorrectly!

Planning and Status

Our overall roadmap and quarterly goals are available online.

This week’s status updates are here.

Notable Additions
  • nox fixed all the warnings on Windows
  • mbrubeck added support for vertical-align in geckolib
  • paul made Servo no longer always quit when the Escape key is pressed
  • larsberg added initial builder support for more “zealous” SpiderMonkey GC configurations
  • jack improved our build time by adding ccache support to skia and azure
  • larsberg enabled caching on Windows builds, taking them down from 50 minutes to 15
  • kaksmet made text-align: justify safe for incremental layout
  • ms2ger added support for Gecko’s string atoms in rust-selectors
  • ddefisher implemented several WebGL functions
  • pcwalton improved window resize performance on MacOS
  • rzambre implemented the ability to log profiling output to a CSV file
  • shinglyu added support for *-reverse flex directions for flexbox
  • mmatyas enabled the use of NEON instructions on ARM and AArch64
  • dzbarsky implemented the readPixels WebGL API
  • asajeffrey made the WebDriver server wait for pages to load before continuing
  • jdm fixed a garbage collection hazard
  • connorgbrewster merged the redundant-yet-awkwardly-intertwined Page and BrowsingContext types
  • kichjang implemented text/plain form encoding
  • broesamle fixed a panick on reddit
New Contributors Get Involved

Interested in helping build a web browser? Take a look at our curated list of issues that are good for new contributors!


A trace of loading’s 356 web fonts using the snazzy new HTML-based profiler:

Profiler HTML output

Servo’s new application icon:

Servo application in the OS X dock

Categorieën: Mozilla-nl planet

The Rust Programming Language Blog: One year of Rust

ma, 16/05/2016 - 02:00

Rust is a language that gives you:

It’s a language for writing highly reliable, screamingly fast software—and having fun doing it.

And yesterday, Rust turned one year old.

Rust in numbers

A lot has happened in the last 365 days:

  • 11,894 commits by 702 contributors added to the core repository;
  • 88 RFCs merged;
  • 18 compiler targets introduced;
  • 9 releases shipped;
  • 1 year of stability delivered.

On an average week this year, the Rust community merged two RFCs and published 53 brand new crates. Not a single day went by without at least one new Rust library hitting the central package manager. And Rust topped the “most loved language” in this year’s StackOverflow survey.

Speaking of numbers: we recently launched a survey of our own, and want to hear from you whether you are an old hat at Rust, or have never used it.

One place where our numbers are not where we want them to be: community diversity. We’ve had ongoing local outreach efforts, but the Rust community team will soon be launching a coordinated, global effort following the Bridge model (e.g. RailsBridge). If you want to get involved, or have other ideas for outreach, please let the community team know.

Rust in production

This year saw more companies betting on Rust. Each one has a story, but two particularly resonated.

First, there’s Dropbox. For the last several years, the company has been secretively working on a move away from AWS and onto its own infrastructure. The move, which is now complete, included developing custom-build hardware and the software to drive it. While much of Dropbox’s back-end infrastructure is historically written in Go, for some key components the memory footprint and lack of control stood in the way of achieving the server utilization they were striving for. They rewrote those components in Rust. In the words of Jamie Turner, a lead engineer for the project, “the advantages of Rust are many: really powerful abstractions, no null, no segfaults, no leaks, yet C-like performance and control over memory.”

Second, there’s Mozilla. They’ve long been developing Servo as a research browser engine in Rust, but their first production Rust code shipped through a different vehicle: Firefox. In Firefox 45, without any fanfare, Rust code for mp4 metadata parsing went out to OSX and 64-bit Linux users; it will hit Windows in version 48. The code is currently running in test mode, with its results compared against the legacy C++ library: 100% correctness on 1 billion reported executions. But this code is just the tip of the iceberg: after laying a lot of groundwork for Rust integration, Firefox is poised to bring in significant amounts of new Rust code, including components from Servo—and not just in test mode.

We’re hearing similar stories from a range of other shops that are putting Rust into production: Rust helps a team punch above its weight. It gives many of the same benefits as traditional systems languages while being more approachable, safer and often more productive.

These are just a few stories of Rust in production, but we’d love to hear yours!

Rust, improved

Of course, Rust itself hasn’t been standing still. The focus in its first year has been growing and polishing its ecosystem and tooling:

There’s a lot more to say about what’s happened and what’s coming up in the Rust world—over the coming months, we’ll be using this blog to say it.

Rust in community

It turns out that people like to get together and talk Rust. We had a sold out RustCamp last August, and several upcoming events in 2016:

  • September 9-10, 2016: the first RustConf in Portland, OR, USA;
  • September 17, 2016: RustFest, the European community conference, in Berlin, Germany;
  • October 27-18, 2016: Rust Belt Rust, a Rust conference in Pittsburgh, PA, USA;
  • 71 Rust-related meetup groups worldwide.

And that’s no surprise. From a personal perspective, the best part about working with Rust is its community. It’s hard to explain quite what it’s like to be part of this group, but two things stand out. First, its sheer energy: so much happens in any given week that This Week in Rust is a vital resource for anyone hoping to keep up. Second, its welcoming spirit. Rust’s core message is one of empowerment—you can fearlessly write safe, low-level systems code—and that’s reflected in the community. We’re all here to learn how to be better programmers, and support each other in doing so.

There’s never been a better time to get started with Rust, whether through attending a local meetup, saying hello in the users forum, watching a talk, or reading the book. No matter how you find your way in, we’ll be glad to have you.

Happy birthday, Rust!

Categorieën: Mozilla-nl planet

Andreas Tolfsen: Evaluating scripts using Marionette

zo, 15/05/2016 - 16:44

One roadblock using Marionette as the backend for WebDriver automation has been its lacking execute script primitives: It has been impossible to modify the DOM as all scripts were evaluated in sandboxes.

As of last week, I finally managed to land a patch that makes evaluated scripts have lasting side-effects in the document. This means that scripts such as = 42 or = "pink" will take effect on the DOM. This is a requirement for WebDriver conformance, and a major inhibitor for people trying to replace FirefoxDriver with Marionette in their automation.

The patch also refactors the way we do script evaluation in Marionette. Six distinct and different script evaluation implementations have been merged into one, and sandbox management, which is still needed for our Gecko specific automation, is more reliable.

Triggering the lasting side-effects behaviour happens through the absence of a "sandboxName" parameter on the JSON object passed to the GeckoDriver#executeScript function. This lets us retain backwards compatibility for the sandboxed script behaviour for the Mozilla-specific Marionette Python client. However, if you’re using the geckodriver HTTP frontend for Marionette, this will trigger the correct WebDriver compatible behaviour out of the box.

Categorieën: Mozilla-nl planet