This is an interesting problem to look at. Let's say you have 1 million users and 5 storage servers. You are building an application that needs to pick one of the 5 storage servers depending on the user id.
You want your app to do it as consistently as possible. e.g. a given user should always land on the same backend.
When a request comes in:
- each request is identified by a unique user id.
- the app should always call the same storage server given that id.
- you want all your users to be distributed equally across your storage servers.
- when a storage server is removed from the list, you want users to stick with the servers they initially got. For users on the removed server, they should be dispatched equally on other servers.
- when a server is added, you want the minimal numbers of users to be moved around.
- The application is stateless about this, so when I deploy a new node and give it the list of the storage servers, it should be able to start distributing users among them without prior knowledge.
Point 4 and 6 discards a simple round-robin solution.
The solution to that problem is to build a deterministic function that projects a user id into the space composed of the servers. (yeah well, clustering I guess.)
There are two known algorithms to do that. The consistent hashing algorithm and the rendezvous hashing.Consistent Hashing
Consistent Hashing is a hashing that can be used to minimize the shuffling of users when a server is removed or added.
This is how it's implemented:
- each server name is converted into a unique number
- that number is projected on an modulo interval (a circle)
- every user is also converted into a unique number and projected on the circle
- the server that's the closest to the user is picked
If you want nice drawing go here.
This is an elegant solution because removing a server keeps the rest stable, and adding one server shuffles a minimal number of users.
The conversion from a name to a integer is key here: you have to be deterministic but in the same time try to have the numbers randomly and kind-of-evenly distributed on the circle.
Here's how you can do it using MD5:import hashlib def hash(key): return long(hashlib.md5(key).hexdigest(), 16)
Using a classical hash like MD5 gives us the random part, but depending on the server name you might well end up with two servers that are very close to each other on the circle
And the result will be that when the users are converted into numbers, a very small amount of users will go to some servers.
One solution to fix that is to add replicas: for each server, instead of projecting a single number on the circle, we'll project 100. For example, "server1" becomes "server1:1", "server1:2", .., "server1:100" and those values are transformed into numbers.
Using replicas is very efficient to make sure users are spread evenly.RendezVous
The other algorithm is called RendezVous and is based on a similar idea where servers are converted into numbers with a hash.
The difference is that instead of projecting servers and their replicas on a circle, the algorithm uses weights. To find which server a user should use, for each combination of server and user, a number is created with a classical hash function.
The server that's picked is the one with the highest number.
The Python code looks like this:def get_server(user): high_score = -1 winner = None for server in server: score = hash(server + user) if score > high_score: high_score, winner = score, ip elif score == high_score: high_score, winner = score, max(server, winner) return winner
The advantage of this method is that you don't have to create replicas to worry about distribution. In fact, according to my tests, RendezVous is doing a better job than Consistent Hashing for distributing users.
One key decision is to decide which hashing algorithm you want to use.It's all about the hashing!
Both RendezVous and Consistent hashing are using a classical hashing function to convert the servers and users into numbers - and picking one was not obvious.
I came across this amazing stackexchange post that shed some light on different hashing, their strengths and weaknesses. You should read it, you will learn a lot.
The take away for my experiment was that some hashing functions are doing a better job at randomizing values in the hash space. Some very simple functions are also colliding quite often, which can be a problem.
So I tried a bunch of them and benchmarked them.
It's interesting to note that I had much better results for my use case using RendezVous & sha256 than RendezVous & Murmur, when the latter is usually what people use with RendezVous.
I ended up removing Murmur from my tests, the results where too bad.
Anyways, here's the full implementation I did, based on snippets I found here and there, and the result:
And the winnner is : RendezVous and sha256
Of course, that entirely depends on how many servers & users you have.
REQUEST FOR URGENT BUSINESS RELATIONSHIP
FIRST, I MUST SOLICIT YOUR STRICTEST CONFIDENCE IN THIS MIGRATION. THIS IS BY VIRTUE OF ITS NATURE AS BEING UTTERLY CONFIDENTIAL AND ‘TOP SECRET’. I AM SURE AND HAVE CONFIDENCE OF YOUR ABILITY AND RELIABILITY TO HANDLE A TASK OF THIS GREAT MAGNITUDE INVOLVING A PENDING MIGRATION REQUIRING MAXIMUM CONFIDENCE.
WE ARE TOP OFFICIAL OF THE RELENG TEAM WHO ARE INTERESTED IN MIGRATION OF TASKS INTO TASKCLUSTER WITH JOBS WHICH ARE PRESENTLY TRAPPED IN BUILDBOT. IN ORDER TO COMMENCE THIS BUSINESS WE SOLICIT YOUR ASSISTANCE TO ENABLE US MIGRATE INTO YOUR TASKGRAPH THE SAID TRAPPED JOBS.
<marked as spam><deleted>
The ongoing work to get TaskCluster building Firefox on Windows reached an important milestone with our first Mozharness based build going green in the TreeHerder dashboard. This represents underlying effort in documenting the dependency chain for Windows builders and producing simple manifests that give greater transparency to changes in this area.
Alin, Amy, and Van brought 192 new OS X 10.10.5 Mac minis online. This should eliminate our Yosemite backlog and allow us to enable more e10s tests.
Q, catlee, and jmaher migrated the first batch of Windows 7 tests to AWS this week. Currently we’re running these suites for all branches of gecko 49 and higher: Web platform tests + reftests, gtest, cppunit, jittest, jsreftest, crashtest. This will reduce our reliance on hardware and allow us to scale dynamically. They are still working on greening larger sets of tests which are more sensitive to their operating environment. Once we have moved a significant portion of tests, we can add additional e10s tests on w7 as well.
Improve Release Pipeline:
In the interest of making progress on migrating Nightly builds to TaskCluster, we had a meeting last week to discuss the security requirements around our nightly release process. Based on the discussions in that meeting, Aki is now iterating on a “one graph” solution for Nightlies (as opposed to a two-graph approach where signing is separate). If this approach works, i.e. we can’t find major security holes in the proposed model, it will simplify our process greatly.
Improve CI Pipeline:
We have achieved our first deprecation milestone in the TaskCluster migration by turning off Linux 64-bit debug builds/tests for aurora/trunk branches in buildbot. These bugs are only generated in TaskCluster now.
Aki added scriptworker to readthedocs: http://scriptworker.readthedocs.io/en/latest/
We had a planned tree closing window over the weekend where, amongst other things, we upgraded our database hosts. It went smoothly, so you probably didn’t even notice.
Our new buildduty associates from Taipei were able to successfully loan machines to developers for the first time last week. Welcome aboard!
Lots of releases published or in-flight: Firefox 47 betas 3-5, Fennec 47.0b4, and Thunderbird 38.8.0 and 45.1.0. As always, you can find more details in our post-mortem wiki pages:
See you next week, freelance writer!
Over the weekend, I wanted to implement something that acted as both a class and function decorator, but could also be used as a context manager. I needed this flexibility for overriding configuration values making it easier to write tests. I wanted to use it in the following ways:
as a function decorator:@config_override(DEBUG='False') def test_something(): ...
as a class decorator that would decorate all methods that start with test_:@config_override(DEBUG='False') class TestSomething: def test_something(self): ...
as a context manager that allowed for multiple layer of overriding:def test_something(): with config_override(DEBUG='False'): with config_override(SOMETHING_ELSE='ou812'): ...
This kind of need comes up periodically, but infrequently enough that I forget how I wrote it the last time around.
This post walks through how I structured it.
Read more… (3 mins to read)
Over at Mozilla's Engagement Engineering we use Docker to ship our websites. We build the docker images in CI and then we run tests against them. Our tests usually need a database or a cache server which you can get it running simply with a single command:docker run -d mariadb
The problem is that this container will take some time to initialize and become available to accept connections. Depending on what your test and how you run your tests this delay can cause a test failure to due database connection timeouts.
We used to wait on executing our tests with sleep command but that -besides being an ugly hack- will not always work. For example you may set sleep timeout to 10 seconds and due to CI server load database initialization takes 11 seconds. And nobody wants a non-deterministic test suite.
Meet Takis. Takis checks once per second if a host:port is open. Once it's open, it just returns. It blocks the execution of your pipeline until services become available. No messages or other output to get in the way of your build logs. No complicated configuration either: It reads CHECK_PORT and optionally CHECK_HOST environment variables, waits and eventually returns.
Takis is build using Go and it's fully statically linked as Adriaan explains in this intriguing read. You can download it and directly use it in your scripts~$ wget https://github.com/glogiotatidis/takis/raw/master/bin/takis ~$ chmod +x takis ~$ CHECK_PORT=3306 ./takis
or use it's super small Docker imagedocker run -e CHECK_PORT=3306 -e CHECK_HOST=database.example.com giorgos/takis
For example here's how we use it to build Snippets Service in TravisCI:script: - docker run -d --name mariadb -e MYSQL_ALLOW_EMPTY_PASSWORD=yes -e MYSQL_DATABASE=snippets mariadb:10.0 # Wait mariadb to initialize. - docker run --link mariadb:db -e CHECK_PORT=3306 -e CHECK_HOST=db giorgos/takis - docker run --env-file .env --link mariadb:db mozorg/snippets:latest coverage run ./manage.py test
It is time for our annual survey on how you use curl and libcurl. Your chance to tell us how you think we’ve done and what we should do next. The survey will close on midnight (central European time) May 27th, 2016.
If you use curl or libcurl from time to time, please consider helping us out with providing your feedback and opinions on a few things:
It’ll take you a couple of minutes and it’ll help us a lot when making decisions going forward. Thanks a lot!
The poll is hosted by Google and that short link above will take you to:
Hello and welcome to another issue of This Week in Rust! Rust is a systems language pursuing the trifecta: safety, concurrency, and speed. This is a weekly summary of its progress and community. Want something mentioned? Tweet us at @ThisWeekInRust or send us an email! Want to get involved? We love contributions.
- Introducing Helix. Rust + Ruby, without the glue.
- Taking Rust everywhere with rustup. Push-button cross-compilation with rustup.
- Non-lexical lifetimes: Adding the outlives relation.
- Deep Learning in Rust: a walk in the park
- Rust for Node.js developers - Part 1 - Introduction to Rust.
- Writing documentation in Rust.
- Fancy Rust development with Emacs.
- Quick check: does your code work? Using QuickCheck to do property based testing.
- Autoconf for Rust projects.
- This week in Servo 62.
- This week in Rust docs 3.
- This week in intermezzOS 1. intermezzOS is a learning operating system with a companion book, written in Rust.
- [video] Ferris makes emulators: Episode 8. Live stream of Ferris developing a N64 emulator in Rust (also on Twitch).
- Announcing RustFest Berlin 2016 (17-18 September 2016).
- hydrogen. Multithreaded, non-blocking Linux server framework in Rust.
- cargo-profiler. Cargo subcommand to profile binaries.
- cargo clippy subcommand for convenient production of annoying warnings.
- rfmt. Rust source code formatter.
- RustDT 0.6.0 is out with improved build target support and experimental auto-check.
This week's Crate of the Week is cargo-profiler, which lets us profile our code directly from cargo using a variety of tools, notably valgrind's callgrind and cachegrind. Thanks to kbknapp for the suggestion!
Always wanted to contribute to open-source projects but didn't know where to start? Every week we highlight some tasks from the Rust community for you to pick and get started!
Some of these tasks may also have mentors available, visit the task page for more information.
- [easy] rust: Participate in 2016 State of Rust Survey.
- [easy] redox: Participate in Redox Survey.
- [easy] clippy: Participate in Rust Clippy Survey.
- [easy] rust: Add error explanations for all error codes.
- [medium] rustup: Make rustup default x86_64-unknown-linux-gnu do something smarter.
- [easy] servo: Remove --no-ssl option.
If you are a Rust project owner and are looking for contributors, please submit tasks here.Updates from Rust Core
132 pull requests were merged in the last two weeks.
- custom panic runtimes (finally implements RFC #1513 – embedded Rust users rejoice!)
- cache ast-ty-to-ty again
- AtomicBools are now 8 bits wide (used to surprisingly be one ptr wide)
- reduce LLVM IR bloat in simple switches for large enums
- reduce LLVM IR bloat on CFG simplification
- split the type context into a local/global one for some memory savings
- MIR: CFG block merging
- MIR: allow mutable slicing of arbitrarily long arrays
- MIR: Prettier MIR pretty printing
- more MIR blockers fixed
- save-analysis format changes
- correct temp drop scheduling order
- don't use env::current_exe with libbacktrace
- Better derived impls for large (mostly) C-like enums
- std::sync::OnceState now public (though unstable)
- normalize trait refs before use
- const_eval can now cast byte slices to bare ptrs
- earlier name resolution
- allow repr attribute on single variant enum
- `impl (Partial)Ord for Cell+RefCell where T: (Partial)Ord
- integer atomic types
- fix gdb pretty printing
- the playpen now uses the new error format
- rustdoc no longer skips blanket impls
- make lifetime error reporting less confusing
- There were a lot of documentaion improvements, too numerous to mention each.
- Brian Green
- Cristian Oliveira
- Dan Fockler
- Geordon Worley
- Haiko Schol
- Pavel Sountsov
- Stefan Schindler
- Steven Burns
Changes to Rust follow the Rust RFC (request for comments) process. These are the RFCs that were approved for implementation this week:
- RFC 1358: Add #[repr(align = "N")].
- RFC 1492: Permit the .. pattern fragment in more contexts.
- Amendment to RFC 1214: All but the last field of a tuple must be Sized.
No RFCs are currently in final comment period.New RFCs
- 5/17. Moscow Rust Conference.
- 5/18. Rust Meetup Taipei #2.
- 5/18. Rust Community Team Meeting at #rust-community on irc.mozilla.org.
- 5/18. Rust Berlin: Rust 1.0 Anniversary Meetup.
- 5/24. Rust Brisbane Meetup #1.
- 5/24. Rust Barcelona - Cross compiling for ARM linux targets + accessing the GPIO in a Raspberry Pi.
- 5/25. Rust Community Team Meeting at #rust-community on irc.mozilla.org.
- 5/25. Boston Rust Meetup w/ Steve Klabnik and Andrew Gallant.
- 5/26. Rust DC - Inaugural Meetup.
No jobs listed for this week.
Tweet us at @ThisWeekInRust to get your job offers listed here!Quote of the Week
No quote was selected for QotW.
Not a song this week, but just a documentary to remind me that some sites are overly complicated and there are strong benefits and resilience in chosing a solid simple framework for working. Not that it makes easier the work. I think it's even the opposite, it's basically harder to make a solid simple Web site. But that the cost is beneficial on the longterm. Tune of the week: The Depth of simplicity in Ozu's movie.Webcompat Life
Progress this week:Today: 2016-05-16T10:12:01.879159 354 open issues ---------------------- needsinfo 3 needsdiagnosis 109 needscontact 30 contactready 55 sitewait 142 ----------------------
In my journey in getting the contactready and needscontact lower, we are making progress. You are welcome to participate
Reorganizing a bit the wiki so it better aligns with our current work. In Progress.
Good news on the front of appearance in CSS.
The CSSWG just resolved that "appearance: none" should turn checkbox & radio <input> elements into a normal non-replaced element.
Learning on how to do mozregression
We are looking at creating a mechanism similar to Opera browser.js into Firefox. Read and participate to the discussion.Webcompat issues
(a selection of some of the bugs worked on this week).
- Pizza being unresponsive or more to the point being slow. Issue with React not being reactive? When I see the markup, I start crying. Why do we hurt ourselves like this?
- Finally closed the Firefox Gmail on Android issue. Happiness.
- irresponsible responsive: 353 requests 25 Mo…. The number of insane Web sites be on mobile or desktop… It's like the hummer SUV to drive around your neighborhood to buy milk.
- OffShore Panama CSS
- Intent to Use Counter: Everything
- My URL isn’t your URL
- Cache-Control: immutable. Good stuff.
- Compression, Anti-virus, MITM and Facebook: recipes for disaster.
- Being where the people are: Vendor Prefixes are dead but with them mass author involvement in early stage specifications. The history of Grid shows that it is incredibly difficult to get people to do enough work to give helpful feedback with something they can’t use – even a little bit – in production.
- Document how to write tests on webcompat.com using test fixtures.
- ToWrite: rounding numbers in CSS for width
- ToWrite: Amazon prefetching resources with <object> for Firefox only.
Recently, there some was some confusion between myself and a coworker over the definition of a “word.” I’m currently working on a blog post about data alignment and figured it would be good to clarify some things now, that we can refer to later.
Having studied computer engineering and being quite fond of processor design, when I think of a “word,” I think of the number of bits wide a processor’s general purpose registers are (aka word size). This places hard requirements on the largest representable number and address space. A 64 bit processor can represent 264-1 (1.8x1019) as the largest unsigned long integer, and address up to 264-1 (16 EiB) different addresses in memory.
Further, word size limits the possible combinations of operations the processor can perform, length of immediate values used, inflates the size of binary files and memory needed to store pointers, and puts pressure on instruction caches.
Word size also has implications on loads and stores based on alignment, as we’ll see in a follow up post.
When I think of 8 bit computers, I think of my first microcontroller: an Arduino with an Atmel AVR processor. When I think of 16 bit computers, I think of my first game console, a Super Nintendo with a Ricoh 5A22. When I think of 32 bit computers, I think of my first desktop with Intel’s Pentium III. And when I think of 64 bit computers, I think modern smartphones with ARMv8 instruction sets. When someone mentions a particular word size, what are the machines that come to mind for you?
So to me, when someone’s talking about a 64b processor, to that machine (and me) a word is 64b. When we’re referring to a 8b processor, a word is 8b.
Now, some confusion.
Back in my previous blog posts about x86-64 assembly, JITs, or debugging, you might have seen me use instructions that have suffixes of b for byte (8b), w for word (16b), dw for double word (32b), and qw for quad word (64b) (since SSE2 there’s also double quadwords of 128b).
Wait a minute! How suddenly does a “word” refer to 16b on a 64b processor, as opposed to a 64b “word?”
The 8086 was created to compete with other 16b processors that beat it to the market, like the Zilog Z80 (any Gameboy emulator fans out there? Yes, I know about the Sharp LR35902). The 8086 was the first design in the x86 family, and it allowed for the same assembly syntax from the earlier 8008, 8080, and 8085 to be reassembled for it. The 8086’s little brother (8088) would be used in IBM’s PC, and the rest is history. x86 would become one of the most successful ISAs in history.
For backwards compatibility, it seems that both Microsoft’s (whose success has tracked that of x86 since MS-DOS and IBM’s PC) and Intel’s documentation refers to words still as being 16b. This allowed 16b PE32+ executables to be run on 32b or even 64b newer versions of Windows, without requiring recompilation of source or source code modification.
This isn’t necessarily wrong to refer to a word based on backwards compatibility, it’s just important to understand the context in which the term “word” is being used, and that there might be some confusion if you have a background with x86 assembly, Windows API programming, or processor design.
So the next time someone asks: why does Intel’s documentation commonly refer to a “word” as 16b, you can tell them that the x86 and x86-64 ISAs have maintained the notion of a word being 16b since the first x86 processor, the 8086, which was a 16b processor.
Side Note: for an excellent historical perspective programming early x86 chips, I recommend Michael Abrash’s Graphics Programming Black Book. For instance he talks about 8086’s little brother, the 8088, being a 16b chip but only having an 8b bus with which to access memory. This caused a mysterious “cycle eater” to prevent fast access to 16b variables, though they were the processor’s natural size. Michael also alludes to alignment issues we’ll see in a follow up post.
For historical reasons (or “hysterical raisins” as gps says) that elude me, the BMO database has been in (ughhh) Pacific Time since it was first created. This caused some weirdness on every daylight savings time switch (particularly in the fall when 2:00-3:00 am technically occurs twice), but not enough to justify the work in fixing it (it’s been this way for close to two decades, so that means lots of implicit assumptions in the code).
However, we’re planning to move BMO to AWS at some point, and their standard db solution (RDS) only supports UTC. Thus we finally had the excuse to do the work, and, after a bunch of planning, developing, and reviewing, the migration happened yesterday without issues. I am unreasonably excited by this and proud to have witnessed the correction of this egregious violation of standard db principles 18 years after BMO was originally deployed.
Thanks to the BMO team and the DBAs!
In the last week, we landed 156 PRs in the Servo organization’s repositories.
Emily Dunham has been working on an org-wide report of the contributor metrics. It’s looking good already, but there is still a lot of fun python work to do on the scripts, for those who want to make sure we’re not counting things incorrectly!Planning and Status
This week’s status updates are here.Notable Additions
- nox fixed all the warnings on Windows
- mbrubeck added support for vertical-align in geckolib
- paul made Servo no longer always quit when the Escape key is pressed
- larsberg added initial builder support for more “zealous” SpiderMonkey GC configurations
- jack improved our build time by adding ccache support to skia and azure
- larsberg enabled caching on Windows builds, taking them down from 50 minutes to 15
- kaksmet made text-align: justify safe for incremental layout
- ms2ger added support for Gecko’s string atoms in rust-selectors
- ddefisher implemented several WebGL functions
- pcwalton improved window resize performance on MacOS
- rzambre implemented the ability to log profiling output to a CSV file
- shinglyu added support for *-reverse flex directions for flexbox
- mmatyas enabled the use of NEON instructions on ARM and AArch64
- dzbarsky implemented the readPixels WebGL API
- asajeffrey made the WebDriver server wait for pages to load before continuing
- jdm fixed a garbage collection hazard
- connorgbrewster merged the redundant-yet-awkwardly-intertwined Page and BrowsingContext types
- kichjang implemented text/plain form encoding
- broesamle fixed a panick on reddit
Interested in helping build a web browser? Take a look at our curated list of issues that are good for new contributors!Screenshot
A trace of loading CNN.com’s 356 web fonts using the snazzy new HTML-based profiler:
Servo’s new application icon:
Rust is a language that gives you:
- uncompromising performance and control;
- prevention of entire categories of bugs, including classic concurrency pitfalls;
- ergonomics that often rival languages like Python and Ruby.
It’s a language for writing highly reliable, screamingly fast software—and having fun doing it.
And yesterday, Rust turned one year old.
Rust in numbers
A lot has happened in the last 365 days:
- 11,894 commits by 702 contributors added to the core repository;
- 88 RFCs merged;
- 18 compiler targets introduced;
- 9 releases shipped;
- 1 year of stability delivered.
On an average week this year, the Rust community merged two RFCs and published 53 brand new crates. Not a single day went by without at least one new Rust library hitting the central package manager. And Rust topped the “most loved language” in this year’s StackOverflow survey.
Speaking of numbers: we recently launched a survey of our own, and want to hear from you whether you are an old hat at Rust, or have never used it.
One place where our numbers are not where we want them to be: community diversity. We’ve had ongoing local outreach efforts, but the Rust community team will soon be launching a coordinated, global effort following the Bridge model (e.g. RailsBridge). If you want to get involved, or have other ideas for outreach, please let the community team know.Rust in production
This year saw more companies betting on Rust. Each one has a story, but two particularly resonated.
First, there’s Dropbox. For the last several years, the company has been secretively working on a move away from AWS and onto its own infrastructure. The move, which is now complete, included developing custom-build hardware and the software to drive it. While much of Dropbox’s back-end infrastructure is historically written in Go, for some key components the memory footprint and lack of control stood in the way of achieving the server utilization they were striving for. They rewrote those components in Rust. In the words of Jamie Turner, a lead engineer for the project, “the advantages of Rust are many: really powerful abstractions, no null, no segfaults, no leaks, yet C-like performance and control over memory.”
Second, there’s Mozilla. They’ve long been developing Servo as a research browser engine in Rust, but their first production Rust code shipped through a different vehicle: Firefox. In Firefox 45, without any fanfare, Rust code for mp4 metadata parsing went out to OSX and 64-bit Linux users; it will hit Windows in version 48. The code is currently running in test mode, with its results compared against the legacy C++ library: 100% correctness on 1 billion reported executions. But this code is just the tip of the iceberg: after laying a lot of groundwork for Rust integration, Firefox is poised to bring in significant amounts of new Rust code, including components from Servo—and not just in test mode.
We’re hearing similar stories from a range of other shops that are putting Rust into production: Rust helps a team punch above its weight. It gives many of the same benefits as traditional systems languages while being more approachable, safer and often more productive.
These are just a few stories of Rust in production, but we’d love to hear yours!Rust, improved
Of course, Rust itself hasn’t been standing still. The focus in its first year has been growing and polishing its ecosystem and tooling:
Ecosystem. The standard library has steadily expanded, with growth focused on filesystem access, networking, time, and collections APIs—and dramatically better documentation coverage. There’s good support for working with C libraries via the libc, winapi, and gcc crates. And new libraries for low-level async io, easy parallelism, lock-free data structures, Rails-like object-relational mapping, regular expressions, and several parsing libraries, including html5ever, a unique HTML5 parser that leverages Rust’s macro system to make the code resemble the spec as closely as possible. These are just scratching the surface, of course, and ecosystem growth, curation and coherence—particularly around async IO and the web stack—will continue to be a major focus in the coming year.
Platforms and targets. Rust’s memory footprint is not much bigger than C’s, which makes it ideal for using in all kinds of places. Over the last year, Rust gained the ability to work directly with the native MSVC toolchain on Windows, to target musl (thereby creating a binary that can be used with zero dependencies on any variety of Linux), to target Android and ARM devices, and many more platforms. The new rustup tool makes it a breeze to manage and compile to these various targets. As of Rust 1.6, you can use Rust without its full standard library, limiting to a core library that does not require any OS services (and hence is suitable for writing OSes in Rust). Finally, there are an increasing number of libraries for embedding Rust code into other contexts, like node.js, Ruby and Go.
Tools. Because Rust looks just like C on the outside, it’s instantly usable with a wide range of existing tools; it works out of the box with lldb, gdb, perf, valgrind, callgrind, and many, many more. Our focus has been to enrich the experience for these tools by adding Rust-specific hooks and workflows. Another major priority is providing full IDE support, in part by providing daemonized services from the compiler; we made good progress on that front this year, and thanks to the Racer project, numerous IDE plugins are already providing some semantic support for Rust. At the same time, the rustfmt code formatting tool has matured to the point that the Rust community is ready to produce an official style. And the beating heart of Rust’s workflow, Cargo, gained numerous abilities this year, most notably the install subcommand.
Compiler. We’ve seen some across-the-board improvements to compile times, and now offer parallelized code generation for further speedups. But the biggest wins will come from the ongoing work on incremental compilation, which will minimize the amount of work the needed when recompiling code after editing it. A vital step here was the move to a custom intermediate representation, which has many other benefits as well. Another focus has been errors, including detailed explanations of most errors, and ongoing work to improve the “at a glance” readability of errors. Expect to hear more on both fronts soon.
Core language. We’ve kept one list purposefully short this year: growth in the core language. While we have some important features in the pipeline (like improved error handling, more flexible borrowing rules and specialization), Rust users by and large are happy with the core language and prefer the community to focus on the ecosystem and tooling.
There’s a lot more to say about what’s happened and what’s coming up in the Rust world—over the coming months, we’ll be using this blog to say it.Rust in community
It turns out that people like to get together and talk Rust. We had a sold out RustCamp last August, and several upcoming events in 2016:
- September 9-10, 2016: the first RustConf in Portland, OR, USA;
- September 17, 2016: RustFest, the European community conference, in Berlin, Germany;
- October 27-18, 2016: Rust Belt Rust, a Rust conference in Pittsburgh, PA, USA;
- 71 Rust-related meetup groups worldwide.
And that’s no surprise. From a personal perspective, the best part about working with Rust is its community. It’s hard to explain quite what it’s like to be part of this group, but two things stand out. First, its sheer energy: so much happens in any given week that This Week in Rust is a vital resource for anyone hoping to keep up. Second, its welcoming spirit. Rust’s core message is one of empowerment—you can fearlessly write safe, low-level systems code—and that’s reflected in the community. We’re all here to learn how to be better programmers, and support each other in doing so.
There’s never been a better time to get started with Rust, whether through attending a local meetup, saying hello in the users forum, watching a talk, or reading the book. No matter how you find your way in, we’ll be glad to have you.
Happy birthday, Rust!
One roadblock using Marionette as the backend for WebDriver automation has been its lacking execute script primitives: It has been impossible to modify the DOM as all scripts were evaluated in sandboxes.
As of last week, I finally managed to land a patch that makes evaluated scripts have lasting side-effects in the document. This means that scripts such as window.foo = 42 or document.documentElement.style.background = "pink" will take effect on the DOM. This is a requirement for WebDriver conformance, and a major inhibitor for people trying to replace FirefoxDriver with Marionette in their automation.
The patch also refactors the way we do script evaluation in Marionette. Six distinct and different script evaluation implementations have been merged into one, and sandbox management, which is still needed for our Gecko specific automation, is more reliable.
Triggering the lasting side-effects behaviour happens through the absence of a "sandboxName" parameter on the JSON object passed to the GeckoDriver#executeScript function. This lets us retain backwards compatibility for the sandboxed script behaviour for the Mozilla-specific Marionette Python client. However, if you’re using the geckodriver HTTP frontend for Marionette, this will trigger the correct WebDriver compatible behaviour out of the box.sny.no/a