mozilla

Mozilla Nederland LogoDe Nederlandse
Mozilla-gemeenschap

Mike Hoye: Planet: Secure For Now

Mozilla planet - sn, 08/04/2017 - 04:25

Elevation

This is a followup to a followup – hopefully the last one for a while – about Planet. First of all, I apologize to the community for taking this long to resolve it. It turned out to have a lot more moving parts than were visible at first, and I didn’t know enough about the problem’s context to be able to solve it quickly. I owe a number of people an apology for that, first among them Ehsan who originally brought it to my attention.

The root cause of the problem was that HTTPlib2 in Python 2.x doesn’t – and apparently will never – support Server Name Indication, an important part of Transport Layer Security on shared hosts. This is probably not a big deal for anyone who doesn’t need to make legacy web-facing Python utilities interact securely with modernity, but… well. It me, as the kids say. Here we are.

For some context, our particular SSL problems manifested themselves with error messages like “Error urllib2 Python. SSL: TLSV1_ALERT_INTERNAL_ERROR ssl.c:590” behind the scenes and “internal error” in Planet proper, and I think it’s fair to feel like those messages are less than helpful. I also – no slight on my colleagues in this – don’t have a lot of say in the infrastructure Planet is running on, and it’s equally fair to say I’m not much of a programmer. Python feature-backporting is kind of a rodeo too, and I had a hard time mapping from “I’m using this version of Python on this OS” to “therefore, I have these tools available to me.” Ultimately this combination of OS constraints, library opacity and learning how (and if, where and when) SSL works (or doesn’t, and why) while working in the dated idioms of a language I only half-know didn’t add up to the smoothest experience.

I had a few options open to me, or at least I thought I did. Refactoring for Python 3.x was a non-starter, but I spent far more time than I should have trying to rewrite Planet to work directly with Requests. That turned out to be harder than I’d expected, largely because Planet code has a lot of expectations all over it about HTTPlib2 and how it behaves. I mistakenly thought re-engineering that behavior would be straightforward, and I definitely wasn’t expecting the surprising number of rusty edge cases I’d run into when my assumptions hit the real live web.

Partway through this exercise, in a curious set of coincidences, Mike Connor and I were talking about an old line – misquoted by John F. Kennedy as “Don’t ever take a fence down until you know the reason why it was put up” – by G. K. Chesterton, that went:

In the matter of reforming things, as distinct from deforming them, there is one plain and simple principle; a principle which will probably be called a paradox. There exists in such a case a certain institution or law; let us say, for the sake of simplicity, a fence or gate erected across a road. The more modern type of reformer goes gaily up to it and says, “I don’t see the use of this; let us clear it away.” To which the more intelligent type of reformer will do well to answer: “If you don’t see the use of it, I certainly won’t let you clear it away. Go away and think. Then, when you can come back and tell me that you do see the use of it, I may allow you to destroy it.”

Infrastructure

One nice thing about ancient software is that it builds up these fences; they look like cruft, like junk you should tear out and throw away, until you really, really understand that your code, and you, are being tested. That conversation reminded me of this blog post from Joel Spolsky, about The Worst Thing you can do with software, which smelled suspiciously like what I was right in the middle of doing.

There’s a subtle reason that programmers always want to throw away the code and start over. The reason is that they think the old code is a mess. And here is the interesting observation: they are probably wrong. The reason that they think the old code is a mess is because of a cardinal, fundamental law of programming:

It’s harder to read code than to write it.

This is why code reuse is so hard. This is why everybody on your team has a different function they like to use for splitting strings into arrays of strings. They write their own function because it’s easier and more fun than figuring out how the old function works.

As a corollary of this axiom, you can ask almost any programmer today about the code they are working on. “It’s a big hairy mess,” they will tell you. “I’d like nothing better than to throw it out and start over.”

Why is it a mess?

“Well,” they say, “look at this function. It is two pages long! None of this stuff belongs in there! I don’t know what half of these API calls are for.”

[…] I know, it’s just a simple function to display a window, but it has grown little hairs and stuff on it and nobody knows why. Well, I’ll tell you why: those are bug fixes. One of them fixes that bug that Nancy had when she tried to install the thing on a computer that didn’t have Internet Explorer. Another one fixes that bug that occurs in low memory conditions. Another one fixes that bug that occurred when the file is on a floppy disk and the user yanks out the disk in the middle. That LoadLibrary call is ugly but it makes the code work on old versions of Windows 95.

Each of these bugs took weeks of real-world usage before they were found. The programmer might have spent a couple of days reproducing the bug in the lab and fixing it. If it’s like a lot of bugs, the fix might be one line of code, or it might even be a couple of characters, but a lot of work and time went into those two characters.

When you throw away code and start from scratch, you are throwing away all that knowledge. All those collected bug fixes. Years of programming work.

The first one of these fences I hit was when I discovered that HTTPlib2.Response objects are (somehow) case-insensitive dictionaries because HTTP headers, per spec, are case-insensitive (though normal Python dictionaries very much not, even though examining Response objects with basic tools like “print” makes them look just like they’re a perfectly standard python dict(), nothing to see here move along. Which definitely has this kind of a vibe to it.) Another was hitting what might be a bug in Requests, where usually it gives you “200” as the HTTP “Everything’s Fine” response, which Python will happily and silently turn into the integer HTTPlib2 is expecting, but sometimes gives you “200 OK” which: kaboom.

On the bright side, I did get to spend a few minutes reminiscing fondly to myself about working with Dave Humphrey way back in the day; in hindsight he warned me about this kind of thing when we were working through a similar problem. “It’s the Web. You get whatever you get, whenever you get it, and you’ll like it.”

I was mulling over all of this earlier this week when I decided to take the best (and also worst, and also last) available option: I threw out everything I’d done up to that point and just started lying to the program until it did what I wanted.

This gist is the meat of that effort; the rest of it (swap out the HTTPlib2 calls for Requests and update your error handling) is straightforward, and running in production now. It boils down to taking a Requests object, giving it an imaginary friend, and then standing it on that imaginary friend’s shoulders, throwing a trenchcoat over it and telling it to act like a grownup. The content both calls returns is identical but the supplementary data – headers, response codes, etc – isn’t, so using this technique as a shim potentially makes Requests a drop-in replacement for HTTPlib2. On the off chance that you’re facing the same problems Planet was facing, I hope it’s useful to you.

Again, I apologize for the delay in sorting this out, and thank you for your patience.

Categorieën: Mozilla-nl planet

Aaron Klotz: Asynchronous Plugin Initialization: Requiem

Mozilla planet - fr, 07/04/2017 - 23:00

My colleague bsmedberg is going to be removing asynchronous plugin initialization in bug 1352575. Sadly the feature never became solid enough to remain enabled on release, so we cut our losses and cancelled the project early in 2016. Now that code is just a bunch of dead weight. With the deprecation of non-Flash NPAPI plugins in Firefox 52, our developers are now working on simplifying the remaining NPAPI code as much as possible.

Obviously the removal of that code does not prevent me from discussing some of the more interesting facets of that work.

Today I am going to talk about how async plugin init worked when web content attempted to access a property on a plugin’s scriptable object, when that plugin had not yet completed its asynchronous initialization.

As described on MDN, the DOM queries a plugin for scriptability by calling NPP_GetValue with the NPPVpluginScriptableNPObject constant. With async plugin init, we did not return the true NPAPI scriptable object back to the DOM. Instead we returned a surrogate object. This meant that we did not need to synchronously wait for the plugin to initialize before returning a result back to the DOM.

If the DOM subsequently called into that surrogate object, the surrogate would be forced to synchronize with the plugin. There was a limit on how much fakery the async surrogate could do once the DOM needed a definitive answer – after all, the NPAPI itself is entirely synchronous. While you may question whether the asynchronous surrogate actually bought us any responsiveness, performance profiles and measurements that I took at the time did indeed demonstrate that the asynchronous surrogate did buy us enough additional concurrency to make it worthwhile. A good number of plugin instantiations were able to complete in time before the DOM had made a single invocation on the surrogate.

Once the surrogate object had synchronized with the plugin, it would then mostly act as a pass-through to the plugin’s real NPAPI scriptable object, with one notable exception: property accesses.

The reason for this is not necessarily obvious, so allow me to elaborate:

The DOM usually sets up a scriptable object as follows:

this.__proto__.__proto__.__proto__
  • Where this is the WebIDL object (ie, content’s <embed> element);
  • Whose prototype is the NPAPI scriptable object;
  • Whose prototype is the shared WebIDL prototype;
  • Whose prototype is Object.prototype.

NPAPI is reentrant (some might say insanely reentrant). It is possible (and indeed common) for a plugin to set properties on the WebIDL object from within the plugin’s NPP_New.

Suppose that the DOM tries to access a property on the plugin’s WebIDL object that is normally set by the plugin’s NPP_New. In the asynchronous case, the plugin’s initialization might still be in progress, so that property might not yet exist.

In the case where the property does not yet exist on the WebIDL object, JavaScript fails to retrieve an “own” property. It then moves on to the first prototype and attempts to resolve the property on that. As outlined above, this prototype would actually be the async surrogate. The async surrogate would then be in a situation where it must absolutely produce a definitive result, so this would trigger synchronization with the plugin. At this point the plugin would be guaranteed to have finished initializing.

Now we have a problem: JS was already examining the NPAPI scriptable object when it blocked to synchronize with the plugin. Meanwhile, the plugin went ahead and set properties (including the one that we’re interested in) on the WebIDL object. By the time that JS execution resumes, it would already be looking too far up the prototype chain to see those new properties!

The surrogate needed to be aware of this when it synchronized with the plugin during a property access. If the plugin had already completed its initialization (thus rendering synchronization unnecessary), the surrogate would simply pass the property access on to the real NPAPI scriptable object. On the other hand, if a synchronization was performed, the surrogate would first retry the WebIDL object by querying for the WebIDL object’s “own” properties, and return the own property if it now existed. If no own property existed on the WebIDL object, then the surrogate would revert to its “pass through all the things” behaviour.

If I hadn’t made the asynchronous surrogate scriptable object do that, we would have ended up with a strange situation where the DOM’s initial property access on an embed could fail non-deterministically during page load.

That’s enough chatter for today. I enjoy blogging about my crazy hacks that make the impossible, umm… possible, so maybe I’ll write some more of these in the future.

Categorieën: Mozilla-nl planet

Andrew Overholt: Quantum work

Mozilla planet - fr, 07/04/2017 - 21:43

Quantum Curling

Last week we had a work week at Mozilla’s Toronto office for a bunch of different projects including Quantum DOM, Quantum Flow (performance), etc. It was great to have people from a variety of teams participate in discussions and solidify (and change!) plans for upcoming Firefox releases. There were lots of sessions going on in parallel and I wasn’t able to attend them all but some of the results were written up by the inimitable Ehsan in his fourth Quantum Flow newsletter.

Near the end of the week, Ehsan gave an impromptu walkthrough of the Gecko profiler. I’m planning on taking some of the tips he gave and that were discussed and put them onto the documentation for the profiler. If you’re interested in helping, please let me know!

The photo above is of us going curling at the High Park Curling Club. It was a lot of fun and I was happy that only one other person had ever curled before so it was a unique experience for almost everyone!

Categorieën: Mozilla-nl planet

Will Kahn-Greene: Everett v0.9 released and why you should use Everett

Mozilla planet - fr, 07/04/2017 - 16:00
What is it?

Everett is a Python configuration library.

Configuration with Everett:

  • is composeable and flexible
  • makes it easier to provide helpful error messages for users trying to configure your software
  • supports auto-documentation of configuration with a Sphinx autocomponent directive
  • supports easy testing with configuration override
  • can pull configuration from a variety of specified sources (environment, ini files, dict, write-your-own)
  • supports parsing values (bool, int, lists of things, classes, write-your-own)
  • supports key namespaces
  • supports component architectures
  • works with whatever you're writing--command line tools, web sites, system daemons, etc

Everett is inspired by python-decouple and configman.

v0.9 released!

This release focused on overhauling the Sphinx extension. It now:

  • has an Everett domain
  • supports roles
  • indexes Everett components and options
  • looks a lot better

This was the last big thing I wanted to do before doing a 1.0 release. I consider Everett 0.9 to be a solid beta. Next release will be a 1.0.

Why you should take a look at Everett

At Mozilla, I'm using Everett 0.9 for Antenna which is running in our -stage environment and will go to -prod very soon. Antenna is the edge of the crash ingestion pipeline for Mozilla Firefox.

When writing Antenna, I started out with python-decouple, but I didn't like the way python-decouple dealt with configuration errors (it's pretty hands-off) and I really wanted to automatically generate documentation from my configuration code. Why write the same stuff twice especially where it's a critical part of setting Antenna up and the part everyone will trip over first?

Here's the configuration documentation for Antenna:

http://antenna.readthedocs.io/en/latest/configuration.html#application

Here's the index which makes it easy to find things by component or by option (in this case, environment variables):

http://antenna.readthedocs.io/en/latest/genindex.html

When you configure Antenna incorrectly, it spits out an error message like this:

1 <traceback omitted, but it'd be here> 2 everett.InvalidValueError: ValueError: invalid literal for int() with base 10: 'foo' 3 namespace=None key=statsd_port requires a value parseable by int 4 Port for the statsd server 5 For configuration help, see https://antenna.readthedocs.io/en/latest/configuration.html

So what's here?:

  • Block 1 is the traceback so you can trace the code if you need to.
  • Line 2 is the exception type and message
  • Line 3 tells you the namespace, key, and parser used
  • Line 4 is the documentation for that specific configuration option
  • Line 5 is the "see also" documentation for the component with that configuration option

Is it beautiful? No. [1] But it gives you enough information to know what the problem is and where to go for more information.

Further, in Python 3, Everett will always raise a subclass of ConfigurationError so if you don't like the output, you can tailor it to your project's needs. [2]

First-class docs. First-class configuration error help. First-class testing. This is why I created Everett.

If this sounds useful to you, take it for a spin. It's almost a drop-in replacement for python-decouple [3] and os.environ.get('CONFIGVAR', 'default_value') style of configuration.

Enjoy!

[1]I would love some help here--making that information easier to parse would be great for a 1.0 release. [2]Python 2 doesn't support exception chaining and I didn't want to stomp on the original exception thrown, so in Python 2, Everett doesn't wrap exceptions. [3]python-decouple is a great project and does a good job at what it was built to do. I don't mean to demean it in any way. I have additional requirements that python-decouple doesn't do well and that's where I'm coming from. Where to go for more

For more specifics on this release, see here: http://everett.readthedocs.io/en/latest/history.html#april-7th-2017

Documentation and quickstart here: https://everett.readthedocs.org/en/v0.9/

Source code and issue tracker here: https://github.com/willkg/everett

Categorieën: Mozilla-nl planet

Karl Dubost: [worklog] Edition 062. AOL, Etsy and redbubble

Mozilla planet - fr, 07/04/2017 - 11:05
webcompat life
  • Working on a replacement for our minutes script.
  • Booking flights to June meeting in San Francisco and thinking about my strategy with regards to data. Each time I book flights I can't help thinking about the absurdity of airlines and booking strategy.
  • My server for this site and a couple of others have changed place. For the last 15 years, my 1U Dell was hosted at W3C MIT. Thank you. I moved to Gandi in the meantime, but I'm exploring also something else.
webcompat issues webcompat.com dev

Otsukare!

Categorieën: Mozilla-nl planet

Patrick Cloke: “Adding Two-Factor Authentication to Django (with django-allauth)” Lightning Talk

Mozilla planet - fr, 07/04/2017 - 03:14

A bit late on this article, but better late than never! Back on October 27th, 2016 I gave a talk at Django Boston entitled “Adding Two-Factor Authentication to Django (with django-allauth)”. It was a ~20 minute talk on integrating the django-allauth-2fa package into a Django project. The package (which I …

Categorieën: Mozilla-nl planet

Air Mozilla: Reps Weekly Meeting Apr. 06, 2017

Mozilla planet - to, 06/04/2017 - 18:00

Reps Weekly Meeting Apr. 06, 2017 This is a weekly call with some of the Reps to discuss all matters about/affecting Reps and invite Reps to share their work with everyone.

Categorieën: Mozilla-nl planet

Air Mozilla: Reps Weekly Meeting Apr. 06, 2017

Mozilla planet - to, 06/04/2017 - 18:00

Reps Weekly Meeting Apr. 06, 2017 This is a weekly call with some of the Reps to discuss all matters about/affecting Reps and invite Reps to share their work with everyone.

Categorieën: Mozilla-nl planet

Patrick McManus: On The Merits of QUIC for HTTP

Mozilla planet - to, 06/04/2017 - 15:19

I am often asked why the Internet HTTP community is working on an IETF based QUIC when HTTP/2 (RFC 7540) is less than 2 years old. There are good answers! This work is essentially part two of a planned evolution to improve speed, reliability, security, responsiveness, and architecture. These needs were understood when HTTP/2 was standardized but the community decided to defer larger architectural changes to the next version. That next version is arriving now in the form of QUIC.

Flashback to HTTP/2 Development
HTTP/2 development consciously constrained its own work in two ways. One was to preserve HTTP/1 semantics, and the other was to limit the work to what could be done within a traditional TLS/TCP stack.

The choice to use TLS/TCP in HTTP/2 was not a forgone conclusion. During the pre-HTTP/2 bakeoff phase there was quite a bit of side channel chatter about whether anyone would propose a different approach such as the SCTP/UDP/[D]TLS stack RTC DataChannels was contemporaneously considering for standardization.

In the end folks felt that experience and running code were top tier properties for the next revision of HTTP. That argued for TCP/TLS. Using the traditional stack lowered the risk of boiling the ocean, improved the latency of getting something deployed, and captured the meaningful low hanging fruit of multiplexing and priority that HTTP/2 was focused on.

Issues that could not be addressed with TCP/TLS 1.2 were deemed out of scope for that round of development. The mechanisms of ALPN and Alternative Services were created to facilitate successor protocols.

I think our operational experience has proven HTTP/2's choices to be a good decision. It is a big win for a lot of cases, it is now the majority HTTPS protocol, and I don't think we could have done too much better within the constraints applied.

The successor protocol turns out to be QUIC. It is a good sign for the vibrancy of HTTP/2 that the QUIC charter explicitly seeks to map HTTP/2 into its new ecosystem. QUIC is taking on exactly the items that were foreseen when scoping HTTP/2.

This post aims to highlight the benefits of QUIC as it applies to the HTTP ecosystem. I hope it is useful even for those that already understand the protocol mechanics. It, however, does not attempt to fully explain QUIC. The IETF editor's drafts or Jana's recent tutorial might be good references for that.

Fixing the TCP In-Order Penalty
The chief performance frustration with HTTP/2 happens during higher than normal packet loss. The in-order property of TCP spans multiplexed HTTP messages. A single packet loss in one message prevents subsequent unrelated messages from being delivered until the loss is repaired. This is because TCP delays received data in order to provide in-order delivery of the whole stream.

For a simple example imagine images A and B each delivered in two parts in this order: A1, A2, B1, and B2. If only A1 were to suffer a packet loss under TCP that would also delay A2, B1, and B2. While image A is unavoidably damaged by this loss, image B is also impacted even though all of its data was successfully transferred the first time.

This is something that was understood during the development of RFC 7540 and was correctly identified as a tradeoff favoring connections with lower loss rates. Our community has seen some good data on how this has played out "in the wild" recently from both Akamai and Fastly. For most HTTP/2 connections this strategy has been beneficial, but there is a tail population that actually regresses in performance compared to HTTP/1 under high levels of loss because of this.

QUIC fixes this problem through multistreaming onto one connection in a way very familiar to 7540 but with an added twist. It also gives each stream its own ordering context analagous to a TCP sequence number. These streams can be delivered independently to the application because in-order only applies to each stream instead of the whole connection in QUIC.
I believe fixing this issue is the highest impact feature of QUIC.
Starting Faster
HTTP/2 starts much faster than HTTP/1 due to its ability to send multiple requests in the first round trip and its superior connection management results in fewer connection establishments. However, new connections using the TCP/TLS stack still incur 2 or 3 round trips of delay before any HTTP/2 data can be sent.
In order to address this QUIC eschews layers in favor of a component architecture that allows sending encrypted application data immediately. QUIC still uses TLS for security establishment and has a transport connection concept, but these components are not forced into layers that require their own round trips to be initialized. Instead the transport session, the HTTP requests, and the TLS context are all combined into the first packet flight when doing session resumption (i.e. you are returning to a server you have seen before). A key part of this is integration with TLS 1.3  and in particular the 0-RTT (aka Early Data) handshake feature.

The HTTP/2 world, given enough time, will be able to capture some of the same benefits using both TLS 1.3 and TCP Fast Open. Some of that work is made impractical by dependencies on Operating System configurations and the occasional interference from middleboxes unhappy with TCP extensions.

However, even at full deployment of TLS 1.3 and TCP Fast Open, that approach will lag QUIC performance because QUIC can utilize the full flight of data in the first round trip while Fast Open limits the amount of data that can be carried to the roughly 1460 bytes available in a single TCP SYN packet. That packet also needs to include the TLS Client Hello and HTTP SETTINGS information along with any HTTP requests. That single packet runs out of room quickly if you need to encode more than one request or any message body. Any excess needs to wait a round trip.

Harmonizing with TLS
When used with HTTP/1 and HTTP/2, TLS generally operates as a simple pipe. During encryption cleartext streams of bytes go in one side and a stream of encrypted bytes come out the other and are then fed to TCP. The reverse happens when decrypting. Unfortunately, the TLS layer operates internally on multi-byte records instead of a byte stream and the mismatch creates a significant performance problem.

The records can be up to 64KB and a wide variety of sizes are used in practice. In order to enforce data integrity, one of the fundamental security properties of TLS, the entire record must be received before it can be decoded. When the record spans multiple packets a problem similar to the "TCP in-order penalty" discussed earlier appears.

A loss to any packet in the record delays decoding and delivery of the other correctly delivered packets while the loss is repaired. In this case the problem is actually a bit worse as any loss impacts the whole record not just the portion of the stream following the loss. Further, because application delivery of the first byte of the record is always dependent on the receipt of the last byte of the record simple serialization delays or common TCP congestion-control stalls add latency to application delivery even with 0% packet loss.

The 'obvious fix' of placing an independent record in each packet turns out to work much better with QUIC than TCP. This is because TCP's API is a simple byte stream. Applications, including TLS, have no sense of where packets begin or end and have no reliable control over it. Furthermore, TCP proxies or even HTTP proxies commonly rearrange TCP packets while leaving the byte stream in tact (a proof of the practical value of end to end integrity protection!).

Even the absurd-um solution of 1 byte records does not work because the record overhead creates multibyte sequences that will still span packet boundaries. Such a naive approach would also drown in its own overhead.

QUIC shines here by using its component architecture rather than the traditional layers. The QUIC transport layer receives plaintext from the application and consults its own transport information regarding packet numbers, PMTU, and the TLS keying information. It combines all of this to form the encrypted packets that can be decrypted atomically with the equivalent of one record per UDP packet. Intermediaries are unable to mess with the framing because even the transport layer is integrity protected in QUIC during this same process - a significant security bonus! Any loss events will only impact delivery of the lost packet.

No More TCP RST Data Loss
As many HTTP developers will tell you, TCP RST is one of the most painful parts of the existing ecosystem. Its pain comes in many forms, but the data loss is the worst.

The circumstances for an operating system generating a RST and how they respond to them can vary by implementation. One common scenario is a server close()ing a connection that has received another request that the HTTP server has not yet read and is unaware of. This is a routine case for HTTP/1 and HTTP/2 applications. Most kernels will react to the closing of a socket with unconsumed data by sending a RST to the peer.

That RST will be processed out of order when received. In practice this means if the original client does a recv() to consume ordinary data that was sent by the server before the server invoked close() the client will incur an unrecoverable failure if the RST has also already arrived and that data cannot ever be read. This is true even if the kernel has sent a TCP ack for it! The problem gets worse when combined with larger TLS record sizes as often the last bit of data is what is needed to decode the whole record and substantial data loss of up to 64KB occurs.

The QUIC RST equivalent is not part of the orderly shutdown of application streams and it is not expected to ever force the loss of already acknowledged data.
Better Responsiveness through Buffer Management
The primary goal of HTTP/2 was the introduction of multiplexing into a single connection and it was understood that you cannot have meaningful multiplexing without also introducing a priority scheme. HTTP/1 illustrates the problem well - it multiplexes the path through unprioritized TCP parallelism which routinely gives poor results. The final RFC contained both multiplexing and priority mechanisms which for the most part work well.

However, successful prioritization requires you to buffer before serializing the byte stream into TLS and TCP because once sent to TCP those messages cannot be reordered in the case of higher priority data presenting itself.  Unfortunately high latency TCP, requires a significant amount of buffering at the socket layer in order to run as fast as possible. These two competing interests make it difficult to judge how much buffering an HTTP/2 sender should use. While there are some Operating System specific oracles that give some clues, TCP itself does not provide any useful guidance to the application for reasonably sizing its socket buffers.

This combination has made it challenging for applications to determine the appropriate level of socket buffering and in turn they sometimes have overbuffered in order to make TCP run at line rate. This results in poor responsiveness to the priority schedule and the inability for a server to recognize individual streams being canceled (which happens more than you may think) because they have already been buffered.

The blending of transport and application components creates the opportunity for QUIC implementations to do a better job on priority. They do this by buffering application data with its priority information outside of the transmission layer. This allows the late binding of the packet transmission to the data that is highest priority at that moment.

Relatedly, whenever a retransmission is required QUIC retransmits the original data in one or more new packets (with new packet identifiers) instead of retransmitting a copy of the lost packet as TCP does. This creates an opportunity to reprioritize, or even drop canceled streams, during retransmission. This compares favorably to TCP which is sentenced to retransmitting the oldest (and perhaps now irrelevant) data first due to its single sequence number and in-order properties.
UDP means Universal DePloyment
QUIC is not inherently either a user space or kernel space protocol - it is quite possible to deploy it in either configuration. However, UDP based applications are often deployed in userspace configurations and do not require special configurations or permissions to run there. It is fair to expect a number of user space based QUIC implementations.

Time will tell exactly what that looks like, but I anticipate it will be a combination of self-updating evergreen applications such as web servers and browsers and also a small set of well maintained libraries akin to the role openssl plays in distributions.

Decoupling functionality traditionally performed by TCP from the operating system creates an opportunity for deploying software faster, updating it more regularly, and iterating on its algorithms in a tight loop. The long replacement and maintenance schedules of operating systems, sometimes measured in decades, create barriers to deploying new approaches to networking.

This new freedom applies both QUIC itself, but also to some of its embedded techniques that have equivalents in the TCP universe that have traditionally been difficult to deploy. Thanks to user space distribution packet pacing, fast open, and loss discovery improvements like RACK will see greater deployment than ever before.

Userspace will mean faster evolution in networking and greater distribution of the resulting best practices.
  ---

The IETF QUIC effort is informed by Google's efforts on its own preceding protocol. While it is not the same effort it does owe a debt to a number of Google folk. I'm not privy to all of the internal machinations at G but, at the inevitable risk of omitting an important contribution, it is worth calling out Jim Roskind, Jana Iyengar, Ian Swett, Adam Langley, and Ryan Hamilton both for their work and their willingness to evangelize and discuss it with me. Thanks! We're making a better Internet together.

This post was originally drafted as a post to the IETF HTTP Working Group by Patrick McManus ,
Categorieën: Mozilla-nl planet

The Mozilla Blog: It’s Time for Open Citations

Mozilla planet - to, 06/04/2017 - 14:26
Mozilla is signing on to the Initiative for Open Citations (I4OC). We believe open data is integral to a healthy Internet — and to a healthy society.

 

Today, Mozilla is announcing support for the Initiative for Open Citations (I4OC), an effort to make citation data from scholarly publications open and freely accessible.

We’re proud to stand alongside the Wikimedia Foundation, the Public Library of Science and a network of other like-minded institutions, publishers and researchers who believe knowledge should be free from restrictions. We want to create a global, public web of citation data — one that empowers teaching, learning, innovation and progress.

Currently, much of the citation data in scholarly publications is not easily accessible. From geology and chemistry journals to papers on psychology, the citations within are often subject to restrictive and confusing licenses which limit discovery and dissemination of published research. Further, citation data is often not machine readable — meaning we can’t use computer programs to parse the data.

Mozilla understands that in some cases, scholarly publications themselves must be protected or closed in order to respect proprietary ecosystems and business models. But citations are snippets of knowledge that allow everyone to engage with, evaluate and build upon ideas. When citations are inaccessible, the flow of knowledge stalls. Innovation is chilled. The results are damaging.

At Mozilla, we believe openness is a core component of a healthy Internet and a healthy society. Whether open citations or open source code, openness allows people to learn from each other, share ideas and foster collaboration.

I4OC details

I4OC seeks to uphold this openness in the realm of scholarly research. Specifically, I4OC calls for citation data that is structured, separable and open. That means:

Structured ensures citation data is presented in a universal, machine-readable format. This empowers computer programs to unpack and draw connections between different research areas.

Separable ensures citation data can be accessed and analyzed without the need to comb through entire journal articles or books. This allows people to navigate research easily, the same way people navigate the web.

Open ensures citation data is free to access and free to reuse. This allows anyone — students, teachers, entrepreneurs, autodidacts — to benefit.

I4OC is asking scholarly publishers to deposit their citations in Crossref and enable reference distribution. Crossref is a nonprofit organization developing infrastructure and services for scholarly publishers.

We’re already seeing progress. Since the start of I4OC in 2016, the percentage of publishers sharing citations has jumped from 1% of Crossref’s 35 million articles (that include citation data) to over 40%.

That’s heartening, but there’s still a long way to go. We want that number to reach 100%. We want openness to become the norm, not the exception. That’s why Mozilla is supporting I4OC. And that’s why Mozilla is committed to teaching and defending openness online. Want to learn more? Read Mozilla’s Internet Health Report to see how openness makes the Internet exceptional. And sign our petition to open up access to federally-funded research in the U.S.

Photo: Sofia University library // Anastas Tarpanov // CC BY-SA 2.0

The post It’s Time for Open Citations appeared first on The Mozilla Blog.

Categorieën: Mozilla-nl planet

Niko Matsakis: Rayon 0.7 released

Mozilla planet - to, 06/04/2017 - 06:00

We just released Rayon 0.7. This is a pretty exciting release, because it marks the official first step towards Rayon 1.0. In addition, it marks the first release where Rayon’s parallel iterators reach “feature parity” with the standard sequential iterators! To mark the moment, I thought I’d post the release notes here on the blog:

This release marks the first step towards Rayon 1.0. For best performance, it is important that all Rayon users update to at least Rayon 0.7. This is because, as of Rayon 0.7, we have taken steps to ensure that, no matter how many versions of rayon are actively in use, there will only be a single global scheduler. This is achieved via the rayon-core crate, which is being released at version 1.0, and which encapsulates the core schedule APIs like join(). (Note: the rayon-core crate is, to some degree, an implementation detail, and not intended to be imported directly; it’s entire API surface is mirrored through the rayon crate.)

We have also done a lot of work reorganizing the API for Rayon 0.7 in preparation for 1.0. The names of iterator types have been changed and reorganized (but few users are expected to be naming those types explicitly anyhow). In addition, a number of parallel iterator methods have been adjusted to match those in the standard iterator traits more closely. See the “Breaking Changes” section below for details.

Finally, Rayon 0.7 includes a number of new features and new parallel iterator methods. As of this release, Rayon’s parallel iterators have officially reached parity with sequential iterators – that is, every sequential iterator method that makes any sense in parallel is supported in some capacity.

New features and methods
  • The internal Producer trait now features fold_with, which enables better performance for some parallel iterators.
  • Strings now support par_split() and par_split_whitespace().
  • The Configuration API is expanded and simplified:
    • num_threads(0) no longer triggers an error
    • you can now supply a closure to name the Rayon threads that get created by using Configuration::thread_name.
    • you can now inject code when Rayon threads start up and finish
    • you can now set a custom panic handler to handle panics in various odd situations
  • Threadpools are now able to more gracefully put threads to sleep when not needed.
  • Parallel iterators now support find_first(), find_last(), position_first(), and position_last().
  • Parallel iterators now support rev(), which primarily affects subsequent calls to enumerate().
  • The scope() API is now considered stable (and part of rayon-core).
  • There is now a useful rayon::split function for creating custom Rayon parallel iterators.
  • Parallel iterators now allow you to customize the min/max number of items to be processed in a given thread. This mechanism replaces the older weight mechanism, which is deprecated.
  • sum() and friends now use the standard Sum traits
Breaking changes

In the move towards 1.0, there have been a number of minor breaking changes:

  • Configuration setters like Configuration::set_num_threads() lost the set_ prefix, and hence become something like Configuration::num_threads().
  • Configuration getters are removed
  • Iterator types have been shuffled around and exposed more consistently:
    • combinator types live in rayon::iter, e.g. rayon::iter::Filter
    • iterators over various types live in a module named after their type, e.g. rayon::slice::Windows
  • When doing a sum() or product(), type annotations are needed for the result since it is now possible to have the resulting sum be of a type other than the value you are iterating over (this mirrors sequential iterators).
Experimental features

Experimental features require the use of the unstable feature. Their APIs may change or disappear entirely in future releases (even minor releases) and hence they should be avoided for production code.

  • We now have (unstable) support for futures integration. You can use Scope::spawn_future or rayon::spawn_future_async().
  • There is now a rayon::spawn_async() function for using the Rayon threadpool to run tasks that do not have references to the stack.
Contributors

Thanks to the following people for their contributions to this release:

  • @Aaronepower
  • @ChristopherDavenport
  • @bluss
  • @cuviper
  • @froydnj
  • @gaurikholkar
  • @hniksic
  • @leodasvacas
  • @leshow
  • @martinhath
  • @mbrubeck
  • @nikomatsakis
  • @pegomes
  • @schuster
  • @torkleyy
Categorieën: Mozilla-nl planet

William Lachance: Easier reproduction of intermittent test failures in automation

Mozilla planet - wo, 05/04/2017 - 22:14

As part of the Stockwell project, I’ve been hacking on ways to make it easier for developers to diagnose failure of our tests in automation. It’s often very difficult to reproduce an intermittent we see in Treeherder locally since the environment is so different, so figuring out workflows that support debugging of problems remotely is essential.

One option that rolled out last year was the so-called one-click loaner, which enabled developers to sign out an virtual machine instance identical to the ones used to run unit tests (at least if the tests are running on Taskcluster, which is increasingly often the case), then execute their particular case with whatever extra debugging options they would find useful. This is a big step forward, but it’s still quite a bit of hassle, since it requires a bunch of manual work on the part of the developer to interact with the instance.

What if we could just re-run the particular test an arbitrary number of times with whatever options we wanted, simply by clicking on a few buttons on Treeherder? I’ve been exploring this for the first few months of 2017 and I’ve come up with a prototype which I think is ready for people to start playing with.

The user interface to this is pretty straightforward. Just find a job you want to retrigger in Treeherder:

Then select the ’…’ option in the panel below and press “Custom Action…”:

You should get a small piece of JSON to edit, which corresponds to the configuration for the retriggered job:

The main field to edit is “path”. You should set this to the name of the test you want to try retriggering. For example dom/animation/test/css-transitions/test_animation-ready.html. You can also set custom Firefox preferences and environment variables, to turn on different types of debugging.

Unfortunately as usual with a new feature at Mozilla, there are a bunch of limitations and caveats:

  • This depends on functionality that’s only in Taskcluster, so buildbot jobs are exempt.
  • No support for Android yet. In combination with the above limitation, this implies that this functionality only works on Linux (at least until other platforms are moved to Taskcluster, which probably isn’t that far off).
  • Browser chrome tests failing in mysterious ways if run repeatedly (bug 1347654)
  • Only reftest and mochitest are currently supported. XPCShell support is blocked by the lack of support in its harness for running a job repeatedly (bug 1347696). Web Platform Tests need the requisite support in mozharness for just setting up the tests without running them — the same issue that prevents us from debugging such tests with a one-click loaner (bug 1348833).

Aside from fixing the above limitations, the following features would also be really nifty to have:

  • Ability to trigger a custom job as part of a try push (i.e. not needing to retrigger off an existing job)
  • Run these jobs under rr, and provide a way to login and interactively debug when the problem is actually reproduced.

I am actually in the process of moving to another team @ Mozilla (more on that in another post), so I probably won’t have a ton of time to work on the above — but I’d be happy to help anyone who’s interested in developing this idea further.

A special shout out to the Taskcluster team for helping me with the development of this feature: in particular the action task implementation from Jonas Finnemann Jensen that made it possible to implement this in the first place.

Categorieën: Mozilla-nl planet

Air Mozilla: The Joy of Coding - Episode 96

Mozilla planet - wo, 05/04/2017 - 19:00

The Joy of Coding - Episode 96 mconley livehacks on real Firefox bugs while thinking aloud.

Categorieën: Mozilla-nl planet

Manish Goregaokar: You're Doing It Wrong

Mozilla planet - wo, 05/04/2017 - 18:37

“You’re doing it wrong”

A common refrain in issue trackers and discussion forums everywhere. In isolation, it’s a variant of RTFM – give a non-answer when someone wants help, and bounce them back to a manual or docs which they probably have already read. Not very helpful, and not useful to anyone. Of course, one can accompany it with a nice explanation of how to do it right; “You’re doing it wrong” isn’t always a bad thing :)

Especially when it comes to programming languages, but in general in the context of any programming tool or library, “you’re doing it wrong” is almost always due to a “bad” mental model. The person, whilst learning, has built a mental model of how the tool works, but this doesn’t accurately reflect reality. Other times, it does reflect reality, but it does not reflect the mental model of the maintainers (there can be multiple valid ways of looking at something!), which leads to an impedance mismatch when reading docs or error messages.

In other cases, “doing it wrong” is a case of the XY problem, where the user has problem X, and think they can solve it with solution Y, and end up asking how they can achieve Y. This happens pretty often — folks may be approaching your technology with prior experience with related things that work differently, and may think the same idioms apply.

When I was at WONTFIX, someone who had done support work in the past mentioned that one thing everyone learns in support is “the user is always wrong …. and it’s not their fault!”.

This is a pretty good template for an attitude to approach “doing it wrong” questions about your technology on online forums as well. And this doesn’t just benefit the users who ask questions, this attitude can benefit your technology!

Back when I used to be more active contributing to the Rust compiler, I also used to hang out in #rust a lot, and often answer newbie questions (now #rust-beginners exists too, and I hang out in both, but I don’t really actively participate as much). One thing I learned to do was probe deeper into why people hit that confusion in the first place. It’s almost always a “bad” mental model. Rust is rarely the first programming language folks learn, and people approach it with preconceptions about how programming works. This isn’t unique to Rust, this happens any time someone learns a language with a different paradigm — learning C or C++ after doing a GCd language, learning a functional language after an imperative one, statically typed after dynamic, or one of the many other axes by which programming languages differ.

Other times, it’s just assumptions they made when reading between the lines of whatever resource they used to learn the language.

So, anyway, folks often have a “bad” mental model. If we are able to identify that model and correct it, we have saved that person from potentially getting confused at every step in the future. Great!

With a tiny bit more effort, however, we can do one step better. Not for that person, but for ourselves! We can probe a bit more and try to understand what caused them to obtain that mental model. And fix the docs so that it never happens again! Of course, not everyone reads the docs, but that’s what diagnostics are for (in the case of errors). They’re a tool to help us nudge the user towards the right mental model, whilst helping them fix their immediate problem. Rust has for a long time had pretty great diagnostics, with improvements happening all the time1. I think this is at least in part due to the attitude of the folks in #rust; always trying to figure out how to preempt various confusions they see.

It’s a good attitude to have. I hope more folks, both in and out of the Rust community, approach “You’re doing it wrong” cases like that.

  1. Diagnostics issues are often the easiest way to contribute to the compiler itself, so if you want to contribute, I suggest starting there. Willing to mentor!

Categorieën: Mozilla-nl planet

Manish Goregaokar: You're Doing It Wrong

Mozilla planet - wo, 05/04/2017 - 18:37

“You’re doing it wrong”

A common refrain in issue trackers and discussion forums everywhere. In isolation, it’s a variant of RTFM – give a non-answer when someone wants help, and bounce them back to a manual or docs which they probably have already read. Not very helpful, and not useful to anyone. Of course, one can accompany it with a nice explanation of how to do it right; “You’re doing it wrong” isn’t always a bad thing :)

Especially when it comes to programming languages, but in general in the context of any programming tool or library, “you’re doing it wrong” is almost always due to a “bad” mental model. The person, whilst learning, has built a mental model of how the tool works, but this doesn’t accurately reflect reality. Other times, it does reflect reality, but it does not reflect the mental model of the maintainers (there can be multiple valid ways of looking at something!), which leads to an impedance mismatch when reading docs or error messages.

In other cases, “doing it wrong” is a case of the XY problem, where the user has problem X, and think they can solve it with solution Y, and end up asking how they can achieve Y. This happens pretty often — folks may be approaching your technology with prior experience with related things that work differently, and may think the same idioms apply.

When I was at WONTFIX, someone who had done support work in the past mentioned that one thing everyone learns in support is “the user is always wrong …. and it’s not their fault!”.

This is a pretty good template for an attitude to approach “doing it wrong” questions about your technology on online forums as well. And this doesn’t just benefit the users who ask questions, this attitude can benefit your technology!

Back when I used to be more active contributing to the Rust compiler, I also used to hang out in #rust a lot, and often answer newbie questions (now #rust-beginners exists too, and I hang out in both, but I don’t really actively participate as much). One thing I learned to do was probe deeper into why people hit that confusion in the first place. It’s almost always a “bad” mental model. Rust is rarely the first programming language folks learn, and people approach it with preconceptions about how programming works. This isn’t unique to Rust, this happens any time someone learns a language with a different paradigm — learning C or C++ after doing a GCd language, learning a functional language after an imperative one, statically typed after dynamic, or one of the many other axes by which programming languages differ.

Other times, it’s just assumptions they made when reading between the lines of whatever resource they used to learn the language.

So, anyway, folks often have a “bad” mental model. If we are able to identify that model and correct it, we have saved that person from potentially getting confused at every step in the future. Great!

With a tiny bit more effort, however, we can do one step better. Not for that person, but for ourselves! We can probe a bit more and try to understand what caused them to obtain that mental model. And fix the docs so that it never happens again! Of course, not everyone reads the docs, but that’s what diagnostics are for (in the case of errors). They’re a tool to help us nudge the user towards the right mental model, whilst helping them fix their immediate problem. Rust has for a long time had pretty great diagnostics, with improvements happening all the time1. I think this is at least in part due to the attitude of the folks in #rust; always trying to figure out how to preempt various confusions they see.

It’s a good attitude to have. I hope more folks, both in and out of the Rust community, approach “You’re doing it wrong” cases like that.

  1. Diagnostics issues are often the easiest way to contribute to the compiler itself, so if you want to contribute, I suggest starting there. Willing to mentor!

Categorieën: Mozilla-nl planet

Mozilla Addons Blog: Add-on Compatibility for Firefox 54

Mozilla planet - wo, 05/04/2017 - 18:00

If you haven’t yet, please read our roadmap to Firefox 57.

Firefox 54 will be released on June 13th. Here’s the list of changes that went into this version that can affect add-on compatibility. There is more information available in Firefox 54 for Developers, so you should also give it a look.

General
  • Remove -moz-appearance. This doesn’t apply to CSS sheets loaded using a chrome:// URL, but it does affect inline CSS styles in XUL and JavaScript code.
XPCOM and Modules WebExtensions

Let me know in the comments if there’s anything missing or incorrect on these lists. If your add-on breaks on Firefox 54, I’d like to know.

The automatic compatibility validation and upgrade for add-ons on AMO will happen in a few weeks, so keep an eye on your email if you have an add-on listed on our site with its compatibility set to Firefox 53.

The post Add-on Compatibility for Firefox 54 appeared first on Mozilla Add-ons Blog.

Categorieën: Mozilla-nl planet

Air Mozilla: Weekly SUMO Community Meeting Apr. 05, 2017

Mozilla planet - wo, 05/04/2017 - 18:00

Weekly SUMO Community Meeting Apr. 05, 2017 This is the sumo weekly call

Categorieën: Mozilla-nl planet

Air Mozilla: Weekly SUMO Community Meeting Apr. 05, 2017

Mozilla planet - wo, 05/04/2017 - 18:00

Weekly SUMO Community Meeting Apr. 05, 2017 This is the sumo weekly call

Categorieën: Mozilla-nl planet

Robert O'Callahan: Rust Optimizations That C++ Can't Do

Mozilla planet - wo, 05/04/2017 - 07:20

Update dbaupp on Reddit pointed out that this particular example does not support the point I'm trying to make. I'll explain why below. I followed up with a new post with a better example.

Consider the Rust code

pub struct Foo {
v1: u64,
v2: u64,
}
pub fn foo(t: &Foo) -> u64 {
t.v1 + t.v2
}

The stable Rust compiler, in Release mode, generates

_ZN8rust_out3foo17h154ee47e2fd80a9eE:
leaq (%rdi,%rsi), %rax
retq

Hang on! foo is supposed to take a reference, so shouldn't it take a pointer as its parameter and do two loads to get the values of v1 and v2? Instead it receives the values of v1 and v2 in %rdi/%rsi and adds them using leaq as a three-operand "add" instruction. The Rust compiler has decided (correctly) that even though you wrote a reference, in this case it's more efficient to just pass the parameter by value. This is good to know as a Rust developer; I can use references freely without worrying about whether it would be more efficient to pass the value directly.

C++ compilers can't do this optimization, for a couple of reasons. One reason is that C++ has a standard ABI that dictates how parameters are passed; it requires that references are always passed as pointers. A more interesting reason is that in general this optimization is not safe in C++. An immutable Rust reference promises that the referenced value does not change (except for some special cases involving UnsafeCell which are easily detected by the compiler), but not so for C++ const references; a "const" value can be modified through other mutable references, so copies induced by pass-by-value could change program behavior. For a simple case like this a C++ compiler could do some analysis to show that can't happen, but those analyses don't scale (e.g. if foo called, and was called by, unknown code, all bets are off).

Here Rust's strong control over aliasing is enabling improved optimization, not just improved safety. Update Actually as dbaupp points out, this optimization example is not specific to Rust. In fact, LLVM is doing it via local analysis (the sort I mentioned above), and clang++ produces the same results for the equivalent C++ code. In fact Rust in general would need to preserve references because you can convert a reference to a raw pointer and compare them for equality.

It's easy to find other examples, such as improved code motion and common subexpression elimination. I think with more study of large Rust programs, more investment in the Rust compiler, and tighter control over unsafe Rust, we will probably discover that Rust enables many more interesting optimizations.

One of the promises of hair-shirt functional languages was that referential transparency and other strong semantic guarantees would enable extremely aggressive optimizations, which would recover the performance lost by moving away from the stateful semantics of hardware. That didn't really work out, but with Rust we might get the aggressive optimizations on top of semantics that are already very efficient.

Categorieën: Mozilla-nl planet

Robert O'Callahan: Rust Optimizations That C++ Can't Do (Version 2)

Mozilla planet - wo, 05/04/2017 - 07:12

My last post was wrong in the details, so let me show a better example. Rust code:

pub extern fn foo(v: &u64, callback: fn()) -> u64 {
let mut sum = 0;
for _ in 0..100 {
sum += *v;
callback();
}
sum
}C++ version here.

The inner loop of the Rust version turns into:

.LBB0_1:
inc ebx
call r14
add r12, r15
cmp ebx, 100
jl .LBB0_1sum is in r12 and the evaluation of *v has been hoisted out of the loop with the result in r15. In the compiled C++ code the value has not been hoisted: .LBB0_1: # =>This Inner Loop Header: Depth=1
add rbx, qword ptr [r15]
call r14
dec ebp
jne .LBB0_1

Of course the performance difference in this case would be negligible, but if *v was replaced with a more complicated expression, the difference would increase. (No, I don't know why the Rust code counts up instead of counting down like C++ to get slightly better code; someone should fix that.)

Here it's really clear that the semantics of Rust are making this optimization possible. In C++ v could be a reference to a global variable which is modified by the callback function, in which case hoisting the load would be incorrect.

Several people pointed out that with LTO/WPO a C++ compiler might be able to determine that such cases don't happen, in this case that no function passed into callback directly or indirectly modifies v. That's what I meant in my previous post by "a C++ compiler could do some analysis to show that can't happen" ... but as I continued, "those analyses don't scale". The larger and more complicated your code gets, the more likely it is that static analysis will fail to prove the invariants you care about. Sometimes, due to dynamic linking or run-time code generation, relevant code just isn't available to be analyzed. In other cases the analysis algorithms aren't sophisticated enough to prove what you care about (often because no such algorithm is known), or adequate algorithms exist but they were too expensive to deploy. Perhaps the worst cases for sophisticated global analysis are when the property you care about isn't even true, because you accidentally violated it in some part of the program, but you'll never know, because the only effect of that accident is that performance got a little bit worse somewhere else. (E.g. with this example if you accidentally change one of the callback functions to indirectly modify v, your hypothetical WPO compiler does a slightly worse job and you won't know why.)

What we need are language-level invariants that apply globally and support aggressive optimization using only local analysis. C++ has some, but Rust's are stronger. We also need those invariants to be checked and enforced, not just "it's undefined behavior if you do this"; (safe) Rust wins again on that.

Categorieën: Mozilla-nl planet

Pages