mozilla

Mozilla Nederland LogoDe Nederlandse
Mozilla-gemeenschap

Air Mozilla: Mozilla Weekly Project Meeting, 24 Apr 2017

Mozilla planet - mo, 24/04/2017 - 20:00

Mozilla Weekly Project Meeting The Monday Project Meeting

Categorieën: Mozilla-nl planet

Air Mozilla: Mozilla Weekly Project Meeting, 24 Apr 2017

Mozilla planet - mo, 24/04/2017 - 20:00

Mozilla Weekly Project Meeting The Monday Project Meeting

Categorieën: Mozilla-nl planet

David Burns: Harassment of Open Source Maintainers or Contributors

Mozilla planet - mo, 24/04/2017 - 13:27

On Friday I had the unfortunate pleasure of taking the brunt on an unhappy Selenium user. Their issue? My team said that a release of GeckoDriver would happen when we are confident in the code. They said that was not professional. They started by telling me that they contribute to Mozilla and this is not acceptable for them as a customer.

Below is a break down of why I took exception to this:

  • My team was being extremely professional. Software, by its very nature, has bugs but we try minimize the amount of bugs we ship. To do this we don't set release dates, we set certain objectives. My team is relatively small compared to the user group it needs to service so we need to triage bugs, fix code. We have both groups inside and outside of Mozilla. By saying we can only release when it is ready is going to be the best we can do.
  • Please don't ever tell open source maintainers you are their customer unless you are paying for support and you have a contract with SLAs. So that there is no issue with definition of customer I suggest you look at Merriam Webster's definition. It says "one that purchases a commodity or service". Mozilla, just like Google, Microsoft, and Apple, are working on WebDriver to help web developers. There is no monetary benefit from doing this. The same goes for the Selenium project. The work and products are given freely.
  • And finally, and this goes for any F/OSS project even if it comes from large corporations like Google or Facebook, never make demands. Ask how you can help instead. If you disagree with the direction of the project, fork it. Make your own project. They have given everything away for free. Take it, make it better for whatever better means for you.

Now, even after explaining this, the harassment continued. It has lead to that user being blocked on social media for me and my team as well as them being blocked on Github. I really dislike blocking people because I know when they approach us they are frustrated but taking that frustration out on my team doesn't help anyone. If you continue, after being warned, you will be blocked. This is not a threat, this a promise.

Next time you feel frustrated with open source ask the maintainers if you can donate time/money/resources to make their lives easier. Don't be the moron that people will instantly block.

Categorieën: Mozilla-nl planet

Firefox Nightly: Release Notes for Nightly

Mozilla planet - mo, 24/04/2017 - 11:33

release notes for NightlyEvery day, multiple changesets are merged or backed out on mozilla-central and every day we compile a new version of Firefox Nightly based on these changes so as to provide builds that our core community can use, test and report feedback on.

This is why we historically don’t issue release notes for Nightly, it is hard to maintain release notes for software to gets a new release every day. However, knowing what happens, what’s new, what should be tested, has always been a recurring request from our community over the years.

So as to help with this legitimate request, we set up a twitter account that regularly informs about significant new features, and we also have the great “These weeks in Firefox” posts by Mike Conley every two weeks. These new communication channels certainly did improve things for our community over the last year.

We are now going a step further and we just started maintaining real release notes for Nightly at this address: Release Notes for Firefox Nightly

But what does it mean to have release notes for a product released every day?

It means that in the context of Project Dawn, we have started monitoring all the commits landing on mozilla-central so as to make sure changes that would merit a mention in Firefox final release notes are properly documented. This is something that we used to do with the Aurora channel, we are just doing it for Nightly instead and we do that several times a week.

Having release notes for Nightly of course means that those are updated continuously and that we only document features that have not been merged yet to Beta. We also do not intend to document unstable features or features currently hidden behind a preference flag in about:config.

The focus today is Firefox Desktop, but we will  also  produce release notes for Firefox Nightly for Android at a later stage, once we have polished the process for Desktop.

Categorieën: Mozilla-nl planet

Anthony Ricaud: On the utility of filing bugs

Mozilla planet - mo, 24/04/2017 - 10:36

During my five years working at Mozilla, I’ve been known to ask people to file bugs when they encountered an issue. Most of the time, the answer was that they didn’t have time to do so and it was useless. I think it is actually very valuable. You get to learn from that experience: how to file actionable bugs, getting deeper knowledge into a specification, maybe a workaround for the problem.

A recent example

Three weeks ago, at work, we launched a new design for the website header. We got some reports that the logo was missing in Firefox on some pages. After investigation, we discovered that Firefox (and also Edge) had a different behaviour with SVG’s <use xlink:href> on pages with a <base> element. We fixed it right away by using an absolute URL for our logo. But we also filed bugs against Gecko and Edge. As part of filing those bugs, I found the change in the SVG specification clarifying how it should be handled. Microsoft fixed the issue in less than two weeks. Mozilla fixed it in less than three weeks.

In October this year1, all browsers should behave the same way in regard to that issue. And a four year old workaround will be obsolete. We will be able to remove the code that we had to introduce. Less code, yeah!

I hope this will convince you that filing bugs has an impact. You can learn more on how to file actionable bugs. If you’d like an easier venue to file bugs when browsers are incompatible, the WebCompat project is a nice place to start.

  1. Firefox 55 should be released on August 8 and the next Edge should be released in September (maybe even earlier, I’m not clear on Edge’s release schedule) 

Categorieën: Mozilla-nl planet

The Servo Blog: This Week In Servo 99

Mozilla planet - mo, 24/04/2017 - 02:31

In the last week, we landed 127 PRs in the Servo organization’s repositories.

By popular request, we added a ZIP archive link to the Servo nightlies for Windows users.

Planning and Status

Our overall roadmap is available online, including the overall plans for 2017. Q2 plans will appear soon; please check it out and provide feedback!

This week’s status updates are here.

Notable Additions
  • hiikezoe corrected the animation behaviour of pseudo-elements in Stylo.
  • UK992 added some auto cleanup mechanisms for TravisCI.
  • Manishearth implemented system font support in Stylo.
  • glennw added groove and ridged border support to WebRender.
  • bholley converted simple CSS selectors and combinators to use inline storage for improved performance.
  • MortimerGoro implemented the missing GetShaderPrecisionFormat WebGL API.
  • sbwtw corrected the behaviour of CSS’ calc API in certain cases.
  • metajack removed the DOMRectList API.
  • BorisChious extended CSS transition support to shorthand properties.
  • nox improved the parsing of the background-size CSS property.
  • avadacatavra added support for creating Rust-based extensions of the C++ JSPrincipals API for SpiderMonkey.
  • kvark avoided a panic in WebRender encountered when using it through Firefox.
  • paulrouget clamped mouse scrolling to a single dimension at a time.
  • Gankro added IPC overhead profiling to WebRender.
  • stshine improved the inline size calculation for inline block layout.
  • mrobinson fixed several problems with laying out absolute positioned blocks.
  • canaltinova implemented support for the -moz-transform CSS property for Stylo.
  • MortimerGoro modernized the infrastructure surrounding Android builds.
New Contributors

Interested in helping build a web browser? Take a look at our curated list of issues that are good for new contributors!

Categorieën: Mozilla-nl planet

Wladimir Palant: How bad is a buffer overflow in an Emscripten-compiled application?

Mozilla planet - snein, 23/04/2017 - 10:57

Emscripten allows compiling C++ code to JavaScript. It is an interesting approach allowing porting large applications (games) and libraries (crypto) to the web relatively easily. It also promises better performance and memory usage for some scenarios (something we are currently looking into for Adblock Plus core). These beneficial effects largely stem from the fact that the “memory” Emscripten-compiled applications work with is a large uniform typed array. The side-effect is that buffer overflows, use-after-free bugs and similar memory corruption mistakes are introduced to JavaScript that was previously safe from them. But are these really security-relevant?

Worst-case scenario are obviously memory corruption bugs that can be misused in order to execute arbitrary code. At the first glance, this don’t seem to be possible here — even with Emscripten the code is still running inside the JavaScript sandbox and cannot escape. In particular, it can only corrupt data but not change any code because code is kept separately from the array serving as “memory” to the application. Then again, native applications usually cannot modify code either due to protection mechanisms of modern processors. So memory corruption bugs are typically abused by manipulating function pointers such as those found on the stack.

Now Emscripten isn’t working with return pointers on the stack. I could identify one obvious place where function pointers are found: virtual method tables. Consider the following interface for example:

class Database { virtual User* LookupUser(char* userName) = 0; virtual bool DropTable(char* tableName) = 0; ... };

Note how both methods are declared with the virtual keyword. In C++ this means that the methods should not be resolved at compile time but rather looked up when the application is running. Typically, that’s because there isn’t a single Database class but rather multiple possible implementations for the Database interface, and it isn’t known in advance which one will be used (polymorphism). In practice this means that each subclass of the Database interface will have a virtual method table with pointers to its implementations of LookupUser and DropTable methods. And that’s the memory area an attacker would try to modify. If the virtual method table can be changed in such a way that the pointer to LookupUser is pointing to DropTable instead, in the next step the attacker might make the application try to look up user "users" and the application will inadvertently remove the entire table.

There are some limitations here coming from the fact that function pointers in Emscripten aren’t actual pointers (remember, code isn’t stored in memory so you cannot point to it). Instead, they are indexes in the function table that contains all functions with the same signature. Emscripten will only resolve the function pointer against a fixed function table, so the attacker can only replace a function pointer by a pointer to another function with the same signature. Note that the signature of the two methods above is identical as far as Emscripten is concerned: both have an int-like return value (as opposed to void, float or double), both have an int-like value as the first parameter (the implicit this pointer) and another int-like value as the second parameter (a string pointer). Given that most types end up as an int-like values, you cannot really rely on this limitation to protect your application.

But the data corruption alone can already cause significant security issues. Consider for example the following memory layout:

char incomingMessage[256]; bool isAdmin = false;

If the application fails to check the size of incoming messages properly, the data will overflow into the following isAdmin field and the application might allow operations that aren’t safe. It is even possible that in some scenarios confidential data will leak, e.g. with this memory layout:

char response[256]; char sessionToken[32];

If you are working with zero-terminated strings, you should be really sure that the response field will always contain the terminating zero character. For example, if you are using some moral equivalent of the _snprintf function in Microsoft Visual C++ you should always check the function return value in order to verify that the buffer is large enough, because this function will not write the terminating zero when confronted with too much data. If the application fails to check for this scenario, an attacker might trick it into producing an overly large response, meaning that the secret sessionToken field will be sent along with the response due to missing terminator character.

These are the problematic scenarios I could think of, there might be more. Now all this might be irrelevant for your typical online game, if you are only concerned about are cheaters then you likely have bigger worries — cheaters have much easier ways to mess with code that runs on their end. A website on the other hand, which might be handling data from a third-party site (typically received via URL or window.postMessage()), is better be more careful. And browser extensions are clearly endangered if they are processing website data via Emscripten-compiled code.

Categorieën: Mozilla-nl planet

Niko Matsakis: Unification in Chalk, part 2

Mozilla planet - snein, 23/04/2017 - 06:00

In my previous post, I talked over the basics of how unification works and showed how that “mathematical version” winds up being expressed in chalk. I want to go a bit further now and extend that base system to cover associated types. These turn out to be a pretty non-trival extension.

What is an associated type?

If you’re not a Rust programmer, you may not be familiar with the term “associated type” (although many langages have equivalents). The basic idea is that traits can have type members associated with them. I find the most intuitive example to be the Iterator trait, which has an associated type Item. This type corresponds to kind of elements that are produced by the iterator:

trait Iterator { type Item; fn next(&mut self) -> Option<Self::Item>; }

As you can see in the next() method, to reference an associated type, you use a kind of path – that is, when you write Self::Item, it means “the kind of Item that the iterator type Self produces”. I often refer to this as an associated type projection, since one is “projecting out”1 the type Item.

Let’s look at an impl to make this more concrete. Consider the type std::vec::IntoIter<T>, which is one of the iterators associated with a vector (specifically, the iterator you get when you invoke vec.into_iter()). In that case, the elements yielded up by the iterator are of type T, so we have an impl like:

impl<T> Iterator for IntoIter<T> { type Item = T; fn next(&mut self) -> Option<T> { ... } }

This means that if we have the type IntoIter<i32>::Item, that is equivalent to the type i32. We usually call this process of converting an associated trait projection (IntoIter<i32>::Item) into the type found in the impl normalizing the type.

In fact, this IntoIter<i32>::Item is a kind of shorthand; in particular, it didn’t explicitly state what trait the type Item is defined in (it’s always possible that IntoIter<i32> implements more than one trait that define an associated type called Item). To make things fully explicit, then, one can use a fully qualified path like this:

<IntoIter<i32> as Iterator>::Item ^^^^^^^^^^^^^ ^^^^^^^^ ^^^^ | | | | | Associated type name | Trait Self type

I’ll use these fully qualified paths from here on out to avoid confusion.

Integrating associated types into our type system

In this post, we will extend our notion of types to include associated type projections:

T = ?X // type variables | N<T1, ..., Tn> // "applicative" types | P // "projection" types (new in this post) P = <T as Trait>::X

Projection types are quite different from the existing “applicative” types that we saw before. The reason is that they introduce a kind of “alias” into the equality relationship. With just applicative types, we could always make progress at each step: that is, no matter what two types were being equated, we could always break the problem down into simpler subproblems (or else error out). For example, if we had Vec<?T> = Vec<i32>, we knew that this could only be true if ?T == i32.

With associated type projections, this is not always true. Sometimes we just can’t make progress. Imagine, for example, this scenario:

<?X as Iterator>::Item = i32

Here we know that ?X is some kind of iterator that yields up i32 elements: but we have no way of knowing which iterator it is, there are many possibilities. Similarly, imagine this:

<?X as Iterator>::Item = <T as Iterator>::Item

Here we know that ?X and T are both iterators that yield up the same sort of items. But this doesn’t tell us anything about the relationship between ?X and T.

Normalization constraints

To handle associated types, the basic idea is that we will introduce normalization constraints, in addition to just having equality constraints. A normalization constraint is written like this:

<IntoIter<i32> as Iterator>::Item ==> ?X

This constraint says that the associated type projection <IntoIter<i32> as Iterator>::Item, when normalized, should be equal to ?X (a type variable). As we will see in more detail in a bit, we’re going to then go and solve those normalizations, which would eventually allow us to conclude that ?X = i32.

(We could use the Rust syntax IntoIter<i32>: Iterator<Item=?X> for this sort of constraint as well, but I’ve found it to be more confusing overall.)

Processing a normalization constraint is very simple to processing a standard trait constraint. In fact, in chalk, they are literally the same code. If you recall from my first Chalk post, we can lower impls into a series of clauses that express the trait that is being implemented along with the values of its associated types. In this case, if we look at the impl of Iterator for the IntoIter type:

impl<T> Iterator for IntoIter<T> { type Item = T; fn next(&mut self) -> Option<T> { ... } }

We can translate this impl into a series of clauses sort of like this (here, I’ll use the notation I was using in my first post):

// Define that `IntoIter<T>` implements `Iterator`, // if `T` is `Sized` (the sized requirement is // implicit in Rust syntax.) Iterator(IntoIter<T>) :- Sized(T). // Define that the `Item` for `IntoIter<T>` // is `T` itself (but only if `IntoIter<T>` // implements `Iterator`). IteratorItem(IntoIter<T>, T) :- Iterator(IntoIter<T>).

So, to solve the normalization constraint <IntoIter<i32> as Iterator>::Item ==> ?X, we translate that into the goal IteratorItem(IntoIter<i32>, ?X), and we try to prove that goal by searching the applicable clauses. I sort of sketched out the procedure in my first blog post, but I’ll present it in a bit more detail here. The first step is to “instantiate” the clause by replacing the variables (T, in this case) with fresh type variables. This gives us a clause like:

IteratorItem(IntoIter<?T>, ?T) :- Iterator(IntoIter<?T>).

Then we can unify the arguments of the clause with our goals, leading to two unification equalities, and combine that with the conditions of the clause itself, leading to three things we must prove:

IntoIter<?T> = IntoIter<i32> ?T = ?X Iterator(IntoIter<?T)

Now we can recursively try to prove those things. To prove the equalities, we apply the unification procedure we’ve been looking at. Processing the first equation, we can simplify because we have two uses of IntoIter on both sides, so the type arguments must be equal:

?T = i32 // changed this ?T = ?X Iterator(IntoIter<?T>)

From there, we can deduce the value of ?T and do some substitutions:

i32 = ?X Iterator(IntoIter<i32>)

We can now unify ?X with i32, leaving us with:

Iterator(IntoIter<i32>)

We can apply the clause Iterator(IntoIter<T>) :- Sized(T) using the same procedure now, giving us two fresh goals:

IntoIter<i32> = IntoIter<?T> Sized<?T>

The first unification will yield (eventually):

Sized<i32>

And we can prove this because this is a built-in rule for Rust (that is, that i32 is sized).

Unification as just another goal to prove

As you can see in the walk through in the previous section, in a lot of ways, unification is “just another goal to prove”. That is, the basic way that chalk functions is that it has a goal it is trying to prove and, at each step, it tries to simplify that goal into subgoals. Often this takes place by consulting the clauses that we derived from impls (or that are builtin), but in the case of equality goals, the subgoals are constructed by the builtin unification algorithm.

In the previous post, I gave various pointers into the implementation showing how the unification code looks “for real”. I want to extend that explanation now to cover associated types.

The way I presented things in the previous section, unification flattens its subgoals into the master list of goals. But in fact, for efficiency, the unification procedure will typically eagerly process its own subgoals. So e.g. when we transform IntoIter<i32> = IntoIter<?T>, we actually just invoke the code to equate their arguments immediately.

The one exception to this is normalization goals. In that case, we push the goals into a separate list that is returned to the caller. The reason for this is that, sometimes, we can’t make progress on one of those goals immediately (e.g., if it has unresolved type variables, a situation we’ve not discussed in detail yet). The caller can throw it onto a list of pending goals and come back to it later.

Here are the various cases of interest that we’ve covered so far

Fallback for projection

Thus far we showed how projection proceeds in the “successful” case, where we manage to normalize a projection type into a simpler type (in this case, <IntoIter<i32> as Iterator>::Item into i32). But sometimes we want to work with generics we can’t normalize the projection any further. For example, consider this simple function, which extracts the first item from a non-empty iterator (it panics if the iterator is empty):

fn first<I: Iterator>(iter: I) -> I::Item { iter.next().expect("iterator should not be empty") }

What’s interesting here is that we don’t know what I::Item is. So imagine we are given a normalization constraint like this one:

<I as Iterator>::Item ==> ?X

What type should we use for ?X here? What chalk opts to do in cases like this is to construct a sort a special “applicative” type representing the associated item projection. I will write it as <Iterator::Item><I>, for now, but there is no real Rust syntax for this. It basically represents “a projection that we could not normalize further”. You could consider it as a separate item in the grammar for types, except that it’s not really semantically different from a projection; it’s just a way for us to guide the chalk solver.

The way I think of it, there are two rules for proving that a projection type is equal. The first one is that we can prove it via normalization, as we’ve already seen:

IteratorItem(T, X) ------------------------- <T as Iterator>::Item = X

The second is that we can prove it just by having all the inputs be equal:

T = U --------------------------------------------- <T as Iterator>::Item = <U as Iterator>::Item

We’d prefer to use the normalization route, because it is more flexible (i.e., it’s sufficient for T and U to be equal, but not necessary). But if we can definitively show that the normalization route is impossible (i.e., we have no clauses that we can use to normalize), then we we opt for this more restrictive route. The special “applicative” type is a way for chalk to record (internally) that for this projection, it opted for the more restrictive route, because the first one was impossible.

(In general, we’re starting to touch on Chalk’s proof search strategy, which is rather different from Prolog, but beyond the scope of this particular blog post.)

Some examples of the fallback in action

In the first() function we saw before, we will wind up computing the result type of next() as <I as Iterator>::Item. This will be returned, so at some point we will want to prove that this type is equal to the return type of the function (actually, we want to prove subtyping, but for this particular type those are the same thing, so I’ll gloss over that for now). This corresponds to a goal like the following (here I am using the notation I discussed in my first post for universal quantification etc):

forall<I> { if (Iterator(I)) { <I as Iterator>::Item = <I as Iterator>::Item } }

Per the rules we gave earlier, we will process this constraint by introducing a fresh type variable and normalizing both sides to the same thing:

forall<I> { if (Iterator(I)) { exists<?T> { <I as Iterator>::Item ==> ?T, <I as Iterator>::Item ==> ?T, } } }

In this case, both constraints will wind up resulting in ?T being the special applicative type <Iterator::Item><I>, so everything works out successfully.

Let’s briefly look at an illegal function and see what happens here. In this case, we have two iterator types (I and J) and we’ve used the wrong one in the return type:

fn first<I: Iterator, J: Iterator>(iter_i: I, iter_j: J) -> J::Item { iter_i.next().expect("iterator should not be empty") }

This will result in a goal like:

forall<I, J> { if (Iterator(I), Iterator(J)) { <I as Iterator>::Item = <J as Iterator>::Item } }

Which will again be normalized and transformed as follows:

forall<I, J> { if (Iterator(I), Iterator(J)) { exists<?T> { <I as Iterator>::Item ==> ?T, <J as Iterator>::Item ==> ?T, } } }

Here, the difference is that normalizing <I as Iterator>::Item results in <Iterator::Item><I>, but normalizing <J as Iterator>::Item results in <Iterator::Item><J>. Since both of those are equated with ?T, we will ultimately wind up with a unification problem like:

forall<I, J> { if (Iterator(I), Iterator(J)) { <Iterator::Item><I> = <Iterator::Item><J> } }

Following our usual rules, we can handle the equality of two applicative types by equating their arguments, so after that we get forall<I, J> I = J – and this clearly cannot be proven. So we get an error.

Termination, after a fashion

One final note, on termination. We do not, in general, guarantee termination of the unification process once associated types are involved. Rust’s trait matching is turing complete, after all. However, we do wish to ensure that our own unification algorithms don’t introduce problems of their own!

The non-projection parts of unification have a pretty clear argument for termination: each time we remove a constraint, we replace it with (at most) simpler constraints that were all embedded in the original constraint. So types keep getting smaller, and since they are not infinite, we must stop sometime.

This argument is not sufficient for projections. After all, we replace a constraint like <T as Iterator>::Item = U with an equivalent normalization constraint, where all the types are the same:

<T as Iterator>::Item ==> U

The argument for termination then is that normalization, if it terminates, will unify U with an applicative type. Moreover, we only instantiate type variables with normalized types. Now, these applicative types might be the special applicative types that Chalk uses internally (e.g., <IteratorItem><T>), but it’s an applicative type nontheless. When that applicative type is processed later, it will therefore be broken down into smaller pieces (per the prior argument). That’s the rough idea, anyway.

Contrast with rustc

I tend to call the normalization scheme that chalk uses lazy normalization. This is because we don’t normalize until we are actually equating a projection with some other type. In constrast, rustc uses an eager strategy, where we normalize types as soon as we “instantiate” them (e.g., when we took a clause and replaced its type parameters with fresh type variables).

The eager strategy has a number of downsides, not the least of which that it is very easy to forget to normalize something when you were supposed to (and sometimes you wind up with a mix of normalized and unnormalized things).

In rustc, we only have one way to represent projections (i.e., we don’t distinguish the “projection” and “applicative” version of <Iterator::Item><T>). The distinction between an unnormalized <T as Iterator>::Item and one that we failed to normalize further is made simply by knowing (in the code) whether we’ve tried to normalize the type in question or not – the unification routines, in particular, always assume that a projection type implies that normalization wouldn’t succeed.

A note on terminology

I’m not especially happy with the “projection” and “applicative” terminology I’ve been using. Its’s what Chalk uses, but it’s kind of nonsense – for example, both <T as Iterator>::Item and Vec<T> are “applications” of a type function, from a certain perspective. I’m not sure what’s a better choice though. Perhaps just “unnormalized” and “normalized” (with types like Vec<T> always being immediately considered normalized). Suggestions welcome.

Conclusion

I’ve sketched out how associated type normalization works in chalk and how it compares to rustc. I’d like to change rustc over to this strategy, and plan to open up an issue soon describing a strategy. I’ll post a link to it in the internals comment thread once I do.

There are other interesting directions we could go with associated type equality. For example, I was pursuing for some time a strategy based on congruence closure, and even implemented (in ena) an extended version of the algorithm described here. However, I’ve not been able to figure out how to combine congruence closure with things like implication goals – it seems to get quite complicated. I understand that there are papers tackling this topic (e.g, Selsam and de Moura), but haven’t yet had time to read it.

Comments?

I’ll be monitoring [the internals thread] for comments and discussion. =)

Footnotes
  1. Projection is a very common bit of jargon in PL circles, though it typically refers to accessing a field, not a type. As far as I can tell, no mainstream programmer uses it. Ah well, I’m not aware of a good replacement.

Categorieën: Mozilla-nl planet

Daniel Stenberg: Fewer mallocs in curl

Mozilla planet - sn, 22/04/2017 - 20:26

Today I landed yet another small change to libcurl internals that further reduces the number of small mallocs we do. This time the generic linked list functions got converted to become malloc-less (the way linked list functions should behave, really).

Instrument mallocs

I started out my quest a few weeks ago by instrumenting our memory allocations. This is easy since we have our own memory debug and logging system in curl since many years. Using a debug build of curl I run this script in my build dir:

#!/bin/sh export CURL_MEMDEBUG=$HOME/tmp/curlmem.log ./src/curl http://localhost ./tests/memanalyze.pl -v $HOME/tmp/curlmem.log

For curl 7.53.1, this counted about 115 memory allocations. Is that many or a few?

The memory log is very basic. To give you an idea what it looks like, here’s an example snippet:

MEM getinfo.c:70 free((nil)) MEM getinfo.c:73 free((nil)) MEM url.c:294 free((nil)) MEM url.c:297 strdup(0x559e7150d616) (24) = 0x559e73760f98 MEM url.c:294 free((nil)) MEM url.c:297 strdup(0x559e7150d62e) (22) = 0x559e73760fc8 MEM multi.c:302 calloc(1,480) = 0x559e73760ff8 MEM hash.c:75 malloc(224) = 0x559e737611f8 MEM hash.c:75 malloc(29152) = 0x559e737a2bc8 MEM hash.c:75 malloc(3104) = 0x559e737a9dc8 Check the log

I then studied the log closer and I realized that there were many small memory allocations done from the same code lines. We clearly had some rather silly code patterns where we would allocate a struct and then add that struct to a linked list or a hash and that code would then subsequently add yet another small struct and similar – and then often do that in a loop.  (I say we here to avoid blaming anyone, but of course I myself am to blame for most of this…)

Those two allocations would always happen in pairs and they would be freed at the same time. I decided to address those. Doing very small (less than say 32 bytes) allocations is also wasteful just due to the very large amount of data in proportion that will be used just to keep track of that tiny little memory area (within the malloc system). Not to mention fragmentation of the heap.

So, fixing the hash code and the linked list code to not use mallocs were immediate and easy ways to remove over 20% of the mallocs for a plain and simple ‘curl http://localhost’ transfer.

At this point I sorted all allocations based on size and checked all the smallest ones. One that stood out was one we made in curl_multi_wait(), a function that is called over and over in a typical curl transfer main loop. I converted it over to use the stack for most typical use cases. Avoiding mallocs in very repeatedly called functions is a good thing.

Recount

Today, the script from above shows that the same “curl localhost” command is down to 80 allocations from the 115 curl 7.53.1 used. Without sacrificing anything really. An easy 26% improvement. Not bad at all!

But okay, since I modified curl_multi_wait() I wanted to also see how it actually improves things for a slightly more advanced transfer. I took the multi-double.c example code, added the call to initiate the memory logging, made it uses curl_multi_wait() and had it download these two URLs in parallel:

http://www.example.com/ http://localhost/512M

The second one being just 512 megabytes of zeroes and the first being a 600 bytes something public html page. Here’s the count-malloc.c code.

First, I brought out 7.53.1 and built the example against that and had the memanalyze script check it:

Mallocs: 33901 Reallocs: 5 Callocs: 24 Strdups: 31 Wcsdups: 0 Frees: 33956 Allocations: 33961 Maximum allocated: 160385

Okay, so it used 160KB of memory totally and it did over 33,900 allocations. But ok, it downloaded over 512 megabytes of data so it makes one malloc per 15KB of data. Good or bad?

Back to git master, the version we call 7.54.1-DEV right now – since we’re not quite sure which version number it’ll become when we release the next release. It can become 7.54.1 or 7.55.0, it has not been determined yet. But I digress, I ran the same modified multi-double.c example again, ran memanalyze on the memory log again and it now reported…

Mallocs: 69 Reallocs: 5 Callocs: 24 Strdups: 31 Wcsdups: 0 Frees: 124 Allocations: 129 Maximum allocated: 153247

I had to look twice. Did I do something wrong? I better run it again just to double-check. The results are the same no matter how many times I run it…

33,961 vs 129

curl_multi_wait() is called a lot of times in a typical transfer, and it had at least one of the memory allocations we normally did during a transfer so removing that single tiny allocation had a pretty dramatic impact on the counter. A normal transfer also moves things in and out of linked lists and hashes a bit, but they too are mostly malloc-less now. Simply put: the remaining allocations are not done in the transfer loop so they’re way less important.

The old curl did 263 times the number of allocations the current does for this example. Or the other way around: the new one does 0.37% the number of allocations the old one did…

As an added bonus, the new one also allocates less memory in total as it decreased that amount by 7KB (4.3%).

Are mallocs important?

In the day and age with many gigabytes of RAM and all, does a few mallocs in a transfer really make a notable difference for mere mortals? What is the impact of 33,832 extra mallocs done for 512MB of data?

To measure what impact these changes have, I decided to compare HTTP transfers from localhost and see if we can see any speed difference. localhost is fine for this test since there’s no network speed limit, but the faster curl is the faster the download will be. The server side will be equally fast/slow since I’ll use the same set for both tests.

I built curl 7.53.1 and curl 7.54.1-DEV identically and ran this command line:

curl http://localhost/80GB -o /dev/null

80 gigabytes downloaded as fast as possible written into the void.

The exact numbers I got for this may not be totally interesting, as it will depend on CPU in the machine, which HTTP server that serves the file and optimization level when I build curl etc. But the relative numbers should still be highly relevant. The old code vs the new.

7.54.1-DEV repeatedly performed 30% faster! The 2200MB/sec in my build of the earlier release increased to over 2900 MB/sec with the current version.

The point here is of course not that it easily can transfer HTTP over 20 Gigabit/sec using a single core on my machine – since there are very few users who actually do that speedy transfers with curl. The point is rather that curl now uses less CPU per byte transferred, which leaves more CPU over to the rest of the system to perform whatever it needs to do. Or to save battery if the device is a portable one.

On the cost of malloc: The 512MB test I did resulted in 33832 more allocations using the old code. The old code transferred HTTP at a rate of about 2200MB/sec. That equals 145,827 mallocs/second – that are now removed! A 600 MB/sec improvement means that curl managed to transfer 4300 bytes extra for each malloc it didn’t do, each second.

Was removing these mallocs hard?

Not at all, it was all straight forward. It is however interesting that there’s still room for changes like this in a project this old. I’ve had this idea for some years and I’m glad I finally took the time to make it happen. Thanks to our test suite I could do this level of “drastic” internal change with a fairly high degree of confidence that I don’t introduce too terrible regressions. Thanks to our APIs being good at hiding internals, this change could be done completely without changing anything for old or new applications.

(Yeah I haven’t shipped the entire change in a release yet so there’s of course a risk that I’ll have to regret my “this was easy” statement…)

Caveats on the numbers

There have been 213 commits in the curl git repo from 7.53.1 till today. There’s a chance one or more other commits than just the pure alloc changes have made a performance impact, even if I can’t think of any.

More?

Are there more “low hanging fruits” to pick here in the similar vein?

Perhaps. We don’t do a lot of performance measurements or comparisons so who knows, we might do more silly things that we could stop doing and do even better. One thing I’ve always wanted to do, but never got around to, was to add daily “monitoring” of memory/mallocs used and how fast curl performs in order to better track when we unknowingly regress in these areas.

Addendum, April 23rd

(Follow-up on some comments on this article that I’ve read on hacker news, Reddit and elsewhere.)

Someone asked and I ran the 80GB download again with ‘time’. Three times each with the old and the new code, and the “middle” run of them showed these timings:

Old code:

real    0m36.705s user    0m20.176s sys     0m16.072s

New code:

real    0m29.032s user    0m12.196s sys     0m12.820s

The server that hosts this 80GB file is a standard Apache 2.4.25, and the 80GB file is stored on an SSD. The CPU in my machine is a core-i7 3770K 3.50GHz.

Someone also mentioned alloca() as a solution for one of the patches, but alloca() is not portable enough to work as the sole solution, meaning we would have to do ugly #ifdef if we would want to use alloca() there.

Categorieën: Mozilla-nl planet

About:Community: Revitalize participation by understanding our communities

Mozilla planet - fr, 21/04/2017 - 23:37

As part of the bigger Open Innovation strategy project on how openness can better drive Mozilla products and technologies, during the next few months we will be conducting research about our communities and contributors.

We want to take a detailed, data-driven look into our communities and contributors: who we are, what we’re doing, what our motivations are and how we’re connected.

Who: Understanding the people in our communities
  • How many contributors are there in the Mozilla community.
  • Who are we? (how diverse is our community?)
  • Where are we? (geography, groups, projects)
What: Understanding what people are doing
  • What are we doing? (contributing with)
  • What are our skillsets?
  • How much time we’re able to devote to the project.
  • The tools we use.
  • Why do people contribute? (motivations)
  • What blocks people from contributing?
  • What other projects do we contribute to?
  • What other organisations are we connected to?
  • How much do people want to get involved?
Why: Understanding why people contribute
  • What are people’s’ motivations.
  • What are the important factors in contributing for Mozilla (ethical, moral, technological etc).
  • Is there anything Mozilla can do that will lead volunteers to contribute more?
  • For people who have left the project:why do they no longer contribute?)
How & Where: Understanding the shape of our communities and our people’s networks
  • What are the different groups and communities.
  • Who’s inside each group (regional and functional).
  • What is the overlap between people in groups?
  • Which groups have the most overlap, which have the least? (not just a static view, but also over time)
  • How contributors are connected to each other? (related with the “where”)
  • How are our contributors connected to other projects, Mozilla etc

In order to answer all these questions, we have divided the work in three major areas.

Contributors and Contributions Data Analysis

Analyzing past quantitative data about contributions and contributors (from sources like Bugzilla, Github, Mailing Lists, and other sources) to identify patterns and draw conclusions about contributors, contributions and communities.

Communities and Contributors survey

Designing and administering a qualitative survey to as many active contributors as possible (also trying to survey people who have stopped contributing to Mozilla) to get a full view of our volunteers (demographics), motivations, which communities people identify with, and their experience with Mozilla. We’ll use this to identify patterns in motivations.

Insights

We’ll bring together the conclusions and data from both of the above components to articulate a set of insights and recommendations that can be a useful input to the Open Innovation Strategy project.

In particular, one aim that we have is to cross reference individuals from the Mozillians Survey and Data Analysis to better understand — on aggregate — how things like motivations and identity relate to contribution.

Our commitments

In all of this work we are handling data with the care you would expect from Mozilla, in line with our privacy policy and in close consultation with Mozilla’s legal and trust teams.

Additionally, we realize that we at Mozilla often ask for people’s time to provide feedback and you may have recently seen other surveys. Also, we have run research projects of this sort in the past without following up with a clear plan of action. This project is different. It’s more extensive than anything we’ve done, it is connected a much larger project to shape Mozilla’s strategy with respect to open practices, and we will be publishing the results and data.

We would like to know your feedback/input about this project, its scope and implementation:

  • Are we missing any areas/topics we should get information about our communities?
  • Which part do you feel it’s more relevant?
  • Where do you think communities can engage to provide more value to the work we are going to do?
  • Any other ideas we are not thinking about?

Please let us know in this discourse topic.

Thanks everyone!

Categorieën: Mozilla-nl planet

Firefox UX: Ratings and reviews on add-ons.mozilla.org

Mozilla planet - fr, 21/04/2017 - 23:00

Hello!

My name is Philip Walmsley, and I am a Senior Visual Designer on the Firefox UX team. I am also one of the people tasked with making addons.mozilla.org (or, “AMO”) a great place to list and find Firefox extensions and themes.

There are a lot of changes happening in the Firefox and Add-ons ecosystem this year (Quantum, Photon, Web Extensions, etc.), and one of them is a visual and functional redesign of AMO. This has been a long time coming! The internet has progressed in leaps and bounds since our little site was launched many years ago, and it’s time to give it some love. We’ve currently got a top-to-bottom redesign in the works, with the goal of making add-ons more accessible to more users.

I’m here to talk with you about one part of the add-ons experience: ratings and reviews. We have found a few issues with our existing approach:

  • The 5-star rating system is flawed. Star ratings are arbitrary on a user by user basis, and it leads to a muddling of what users really think about an add-on.
  • Some users just want to leave a rating and not write a review. Sometimes this is referred to as “blank page syndrome,” sometimes a user is just in a time-crunch, sometimes a user might have accessibility issues. Forcing users to do both leads to glib, unhelpful, and vague reviews.
  • On that note, what if there was a better way to get reviews from users that may not speak your native tongue? What if instead of writing a review, a user had the option to select tags or qualities describing their experience with an add-on? This would greatly benefit devs (‘80% of the global community think my extension is “Easy to use”!’) and other users (‘80% of the global community believe this extension is “Easy to use”!’).
  • We don’t do a very good job of triaging users actual issues: A user might love an extension but have an (unbeknownst to them) easily-solved technical problem. Instead of leaving a negative 1-star review for this extension that keeps acting weird, can we guide that user to the developer or Mozilla support?
  • We also don’t do a great job of facilitating developer/user communication within AMO. Wouldn’t it be great if you could rectify a user’s issue from within the reviews section on your extension page, changing a negative rating to a positive one?

So, as you can see, we’ve got quite a few issues here. So let’s simplify and tackle these one-by-one: Experience, Tags, Triage.

So many feels

Experience

Someone is not familiar with Lisa Hanawalt

The star rating has its place. It is very useful in systems where the rating you leave is relevant to you and you alone. Your music library, for example: you know why you rate one song two stars and another at four. It is a very personal but very arbitrary way of rating something. Unfortunately, this rating system doesn’t scale well when more than one person is reviewing the same thing: If I love something but rate it two stars because it lacks a particular feature, what does that mean to other users or the overall aggregated rating? It drags down the review of a great add-on, and as other users scan reviews and see 2-stars, they might leave and try to find something else. Not great.

What if instead of stars, we used emotions?

Some of you might have seen these in airports or restrooms. It is a straightforward and fast way for a group of people to indicate “Yep, this restroom is sparkling and well-stocked, great experience.” Or “Someone needs to get in here with a mop, PRONTO.” It changes throughout the day, and an attendant can address issues as they arise. Or, through regular maintenance, they can achieve a happy face rating all day.

What if we applied this method to add-ons? What if the first thing we asked a user once they had used an extension for a day or so was: “How are you enjoying this extension?” and presented them with three faces: Grinning, Meh, and Sad. At a very high level, this gives users and developers a clear, overall impression of how people feel about using this add-on (“90% grinning face for this extension? People must like it, let’s give it a try.”).

So! A user has contributed some useful rating data, which is awesome. At this point, they can leave the flow and continue on their merry way, or we can prompt them to quickly leave a few more bits of even MORE useful review data…

Tags

Not super helpful

Writing a review is hard. Let me rephrase that: Writing a good review is hard. It’s easy to fire off something saying “This add-on is just ok.” It’s hard to write a review explaining in detail why the add-on is “just ok.” Some (read: most) users don’t want to write a detailed review, for many reasons: time, interest, accessibility, etc. What if we provided a way for these users to give feedback in a quick and straightforward way? What if, instead of staring down a blank text field, we displayed a series of tags or descriptors based on the emotion rating the user just gave?

For example, I just clicked a smiling face to review an extension I’m enjoying. Right after that, a grid of tags with associated icons pops up. Words like “fast”, “stable”, “easy to use”, well-designed”, fun”, etc. I liked the speed of this extension, so I click “fast” and “stable” and submit my options. And success: I have submitted two more pieces of data that are useful to devs and users. Developers can find out what users like about their add-on, and users can see what other users are thinking before committing to downloading. We can pop up different tags based on the emotion selected: if a user taps Meh or Sad, we can pop up tags to find out why the user selected that initially. The result is actionable review data that can is translated across all languages spoken by our users! Pretty cool.

Triage

Finally, we reach triage. Once a user submits tag review data, we can present them with a few more options. If a user is happy with this extension and wants to contribute even more, we can present them with an opportunity to write a review, or share it with friends, or contact the developer personally to give them kudos. If a user selected Meh, we could suggest reading some developer-provided documentation, contacting support, or writing a review. If the user selected Sad, we’d show them developer or Mozilla support, extension documentation, file a bug/issue, or write a review. That way we can make sure a user gets the help they need, and we can avoid unnecessary poor reviews. All of these options will also be available on the add-on page as well, so a user always has access to these different actions. If a user leaves a review expressing frustration with an add-on, devs will be able to reply to the review in-line, so other users can see issues being addressed. Once a dev has responded, we will ask the user if this has solved their problem and if they’d like to update their review.

We’ve covered a lot here! Keep in mind that this is still in the early proposal stage and things will change. And that’s good; we want to change this for the better. Is there anything we’ve missed? Other ideas? What’s good about our current rating and review flow? What’s bad? We’d love constructive feedback from AMO users, extension developers, and theme artists.

Please visit this Discourse post to continue the discussion, and thanks for reading!

Philip (@pwalm)
Senior Visual Designer, Firefox UX

Ratings and reviews on add-ons.mozilla.org was originally published in Firefox User Experience on Medium, where people are continuing the conversation by highlighting and responding to this story.

Categorieën: Mozilla-nl planet

Air Mozilla: Webdev Beer and Tell: April 2017

Mozilla planet - fr, 21/04/2017 - 20:00

 April 2017 Once a month web developers across the Mozilla community get together (in person and virtually) to share what cool stuff we've been working on in...

Categorieën: Mozilla-nl planet

Mozilla Open Policy & Advocacy Blog: Dutch court ruling puts net neutrality in question

Mozilla planet - fr, 21/04/2017 - 17:21

On Thursday, April 20th a Rotterdam Court ruled that T-Mobile’s zero rated service “Data Free Music” is legal. The court declared that the Dutch net neutrality law, which prohibits zero rating, is not in accordance with the EU net neutrality law that Brussels lawmakers passed last year.

Zero rating is bad for the long term health of the internet. By disrupting the level playing field and allowing discrimination, zero rating poses a threat to users, competition, and opportunity online.

The Netherlands has been a model to the world in protecting net neutrality. It’s alarming to see these vital protections for users, competition, and opportunity online struck down.

The power and potential of the Internet is greatest when users can access the full diversity of the open Internet, not just some parts of it. We urge the Authority for Consumers & Markets (ACM) to appeal this decision swiftly, and we hope that higher courts will restore the Internet’s level playing field.

The post Dutch court ruling puts net neutrality in question appeared first on Open Policy & Advocacy.

Categorieën: Mozilla-nl planet

Air Mozilla: WorldBots Meetup 4/20/17

Mozilla planet - fr, 21/04/2017 - 04:00

WorldBots Meetup 4/20/17 WorldBots Meetup 2017-04-20 19:00 - 21:00 We're throwing the first World Bot Meetup! International experts from all over the world will talk about the culture,...

Categorieën: Mozilla-nl planet

Cameron Kaiser: The аррӏе bites back

Mozilla planet - fr, 21/04/2017 - 04:00
I've received a number of inquiries about whether TenFourFox will follow the same (essentially wontfix) approach of Firefox for dealing with those international domain names that happen to be whole-script homographs. The matter was forced recently by one enterprising sort who created just this sort of double using Cyrillic characters for https://www.аррӏе.com/, which depending on your font and your system setup, may look identical to https://www.apple.com/ (the site is a proof of concept only).

The circulating advice is to force all IDNs to be displayed in punycode by setting network.IDN_show_punycode to true. This is probably acceptable for most of our users (the vast majority of TenFourFox users operate with a Latin character set), but I agree with Gerv's concern in that Bugzilla entry that doing so disadvantages all other writing systems that are not Latin, so I don't feel this should be the default. That said, I also find the current situation unacceptable and doing nothing, or worse relying on DNS registrars who so far don't really care about anything but getting your money, similarly so. While the number of domains that could be spoofed in this fashion is probably small, it is certainly greater than one, and don't forget that they let the proof-of-concept author register his spoof!

Meanwhile, I'm not sure what the solution right now should be other than "not nothing." Virtually any approach, including the one Google Chrome has decided to take, will disadvantage non-Latin scripts (and the Chrome approach has its own deficiencies and is not IMHO a complete solution to the problem, nor was it designed to be). It would be optimal to adopt whatever solution Firefox eventually decides upon for consistency if they do so, but this is not an issue I'd like to sit on indefinitely. If you use a Latin character set as your default language, and/or you don't care if all domains will appear in either ASCII or punycode, then go ahead and set that pref above; if you don't, or consider this inappropriate, stay tuned. I'm thinking about this in issue 384.

By the way, TenFourFox "FPR0" has been successfully uploaded to Github. Build instructions to follow and the first FPR1 beta should be out in about two to three weeks. I'm also cogitating over a blog post discussing not only us but other Gecko forks (SeaMonkey, Pale Moon, etc.) which for a variety of reasons don't want to follow Mozilla into the unclear misty haze of a post-XUL world. To a first approximation our reasons are generally technical and theirs are primarily philosophical, but we both end up doing some of the same work and we should talk about that as an ecosystem. More later.

Categorieën: Mozilla-nl planet

Air Mozilla: WorldBots Meetup 4/20/17

Mozilla planet - fr, 21/04/2017 - 04:00

WorldBots Meetup 4/20/17 WorldBots Meetup 2017-04-20 19:00 - 21:00 We're throwing the first World Bot Meetup! International experts from all over the world will talk about the culture,...

Categorieën: Mozilla-nl planet

Air Mozilla: Reps Weekly Meeting Apr. 20, 2017

Mozilla planet - to, 20/04/2017 - 18:00

Reps Weekly Meeting Apr. 20, 2017 This is a weekly call with some of the Reps to discuss all matters about/affecting Reps and invite Reps to share their work with everyone.

Categorieën: Mozilla-nl planet

Air Mozilla: Reps Weekly Meeting Apr. 20, 2017

Mozilla planet - to, 20/04/2017 - 18:00

Reps Weekly Meeting Apr. 20, 2017 This is a weekly call with some of the Reps to discuss all matters about/affecting Reps and invite Reps to share their work with everyone.

Categorieën: Mozilla-nl planet

QMO: Firefox 54 Beta 3 Testday, April 28th

Mozilla planet - to, 20/04/2017 - 16:20

Hello Mozillians,

We are happy to let you know that Friday, April 28th, we are organizing Firefox 54 Beta 3 Testday. We’ll be focusing our testing on the following new features: Net Monitor MVP and Download Panel UX Redesign.

Check out the detailed instructions via this etherpad.

No previous testing experience is required, so feel free to join us on #qa IRC channel where our moderators will offer you guidance and answer your questions.

Join us and help us make Firefox better!

See you on Friday!

Categorieën: Mozilla-nl planet

Mozilla Localization (L10N): Localizing Firefox in Barcelona

Mozilla planet - to, 20/04/2017 - 15:23

We were thrilled to start the year’s localization (l10n) community workshops in Barcelona at the end of March 2017! Thanks to the help of the ever dedicated Alba and Benny the workshop was fun, productive, and filled with amazing Catalonian food.

This workshop aimed to gather together core and active localizers from twenty-one l10n communities scattered throughout the southern parts of Western and Eastern Europe. Unlike the 2016 l10n hackathons, this was the first time we brought these twenty-one communities together to share experiences, ideas, and hack on Mozilla l10n projects together.

The workshop was held at Betahaus, a local co-working space in Barcelona located in Villa de Grácia, Barcelona. The space was great for both large group presentations and small group breakouts. We had room to move around, brainstorm on whiteboards, and play our favorite icebreaker game, spectograms.

All of the l10n-drivers were present for this workshop (another first) and many gave presentations on their main projects. Localizers got a look into new developments with L20n, Pontoon, and Pootle. We also had a glimpse into cross-channel localization for Firefox and how localizers can prepare for it to come in June.

Following tradition, l10n communities came to the workshop with specific goals to accomplish while there. While together, these communities were able to complete around 75% of their goals. These goals largely surrounded addressing the question of localization quality and testing, but also included translating strings for Mozilla products, web sites, and planning for recruiting new localizers.

We couldn’t think of being in Barcelona without taking advantage of participating in a cultural activity as a group. Alba was kind enough to guide the whole group through the city on Saturday night and show us some of the most prominent sites, like Sagrada Familia (which happened to be the most popular site among the l10n communities).

On Sunday, the l10n communities and drivers gathered around four different tables to discuss four different topics in 30-minute chunks of time. Every 30 minutes, Mozillians moved to a different table to discuss the topic assigned to that table. These topics included localization quality, style guides, recruiting new localizers, and mentoring new localizers. It was a great opportunity for both veteran and new localizers to come together and share their experience with each topic and ideas on how to take new approaches to each. Sure, it was a bit chaotic, but everyone was flexible and willing to participate, which made it a good experience nevertheless.

For more info about the workshop (including the official Spotify playlist of the workshop), visit the event’s wiki page here. ¡Hasta luego!

More pictures from the event:

Categorieën: Mozilla-nl planet

Pages