Mozilla Nederland LogoDe Nederlandse

Air Mozilla: German speaking community bi-weekly meeting

Mozilla planet - to, 27/08/2015 - 21:00

German speaking community bi-weekly meeting

Categorieën: Mozilla-nl planet

Andrew Halberstadt: Looking beyond Try Syntax

Mozilla planet - to, 27/08/2015 - 20:03

Today marks the 5 year anniversary of try syntax. For the uninitiated, try syntax is a string that you put into your commit message which a parser then uses to determine the set of builds and tests to run on your try push. A common try syntax might look like this:

try: -b o -p linux -u mochitest -t none

Since inception, it has been a core part of the Mozilla development workflow. For many years it has served us well, and even today it serves us passably. But it is almost time for try syntax to don the wooden overcoat, and this post will explain why.

A brief history on try syntax

In the old days, pushing to try involved a web interface called sendchange.cgi. Pushing is probably the wrong word to use, as at no point did the process involve version control. Instead, patches were uploaded to the web service, which in turn invoked a buildbot sendchange with all the required arguments. Like today try server was often overloaded, sometimes taking over 4 hours for results to come back. Unlike today there was no way to pick and choose which builds and tests you wanted, every try push ran the full set.

The obvious solution was to create a mechanism for people to do that. It was while brainstorming this problem that ted, bhearsum and jorendorff came up with the idea of encoding this information in the commit message. Try syntax was first implemented by lsblakk in bug 473184 and landed on August 27th, 2010. It was a simple time; the list of valid builders could fit into a single 30 line config file; Fennec still hadn't picked up full steam; and B2G wasn't even a figment of anyone's wildest imagination.

It's probably not a surprise to anyone that as time went on, things got more complicated. As more build types, platforms and test jobs were added, the try syntax got harder to memorize. To help deal with this, lsblakk created the trychooser syntax builder just a few months later. In 2011, pbiggar created the trychooser mercurial extension (which was later forked and improved by sfink). These tools were (and still are) the canonical way to build a try syntax string. Little has changed since then, with the exception of the mach try command that chmanchester implemented around June 2015.

One step forward, two steps back

Since around 2013, the number of platforms and test configurations have grown at an unprecendented rate. So much so, that the various trychooser tools have been perpetually out of date. Any time someone got around to adding a new job to the tools, two other jobs had already sprung up in its place. Another problem caused by this rapid growth, was that try syntax became finicky. There were a lot of edge cases, exceptions to the rule and arbitrary aliases. Often jobs would mysteriously not show up when they should, or mysteriously show up when they shouldn't.

Both of those problems were exacerbated by the fact that the actual try parser code has never had a definite owner. Since it was first created, there have never been more than 11 commits in a year. There have been only two commits to date in 2015.

Two key insights

At this point, there are two things that are worth calling out:

  1. Generating try strings from memory is getting harder and harder, and for many cases is nigh impossible. We rely more and more on tools like trychooser.
  2. Try syntax is sort of like an API on which these tools are built on top of.

What this means is that primary generators of try syntax have shifted from humans to tools. A command line encoded in a commit message is convenient if you're a human generating the syntax manually. But as far as tooling goes, try syntax is one god awful API. Not only do the tools need to figure out the magic strings, they need to interact with version control, create an empty commit and push it to a remote repository.

There is also tooling on the other side of the see saw, things that process the try syntax post push. We've already seen buildbot's try parser but taskcluster has a separate try parser as well. This means that your try push has different behaviour, depending on whether the jobs are scheduled in buildbot or taskcluster. There are other one off tools that do some try syntax parsing as well, including but not limited to try tools in mozharness, the try re-trigger bot and the AWSY dashboard. These tools are all forced to share and parse the same try syntax string, so they have to be careful not to step on each other's toes.

The takeaway here is that for tools, a string encoded as a commit message is quite limiting and a lot less convenient than say, calling a function in a library.

Despair not, young Padawan

So far we've seen how try syntax is finicky, how the tools that use it are often outdated and how it fails as an API. But what is the alternative? Fortunately, over the course of 2015 a lot of progress has been made on projects that for the first time, give us a viable alternative to try syntax.

First and foremost, is mozci. Mozci, created by armenzg and adusca, is a tool that hooks into the build api (with early support for taskcluster as well). It can do things like schedule builds and tests against any arbitrary pushes, and is being used on the backend for tools like adusca's try-extender with integration directly into treeherder planned.

Another project that improves the situation is taskcluster itself. With taskcluster, job configuration and scheduling all lives in tree. Thanks to bhearsum's buildbot bridge, we can even use taskcluster to schedule jobs that still live in buildbot. There's an opportunity here to leverage these new tools in conjunction with mozci to gain complete and total control over how jobs are scheduled on try.

Finally I'd like to call out mach try once more. It is more than a thin wrapper around try syntax that handles your push for you. It actually lets you control how the harness gets run within a job. For now this is limited to test paths and tags, but there is a lot of potential to do some cool things here. One of the current limiting factors is the unexpressiveness of the try syntax API. Hopefully this won't be a problem too much longer. Oh yeah, and mach try also works with git.

A glimpse into the crystal ball

So we have several different projects all coming together at once. The hard part is figuring out how they all tie in together. What do we want to tackle first? How might the future look? I want to be clear that none of this is imminent. This is a look into what might be, not what will be.

There are two places we mainly care about scheduling jobs on try.

First imagine you push your change to try. You open up treeherder, except no jobs are scheduled. Instead you see every possible job in a distinct greyed out colour. Scheduling what you want is as simple as clicking the desired job icons. Hold on a sec, you don't have to imagine it. Adusca already has a prototype of what this might look like. Being able to schedule your try jobs this way has a huge benefit: you don't need to mentally correlate job symbols to job names. It's as easy as point and click.

Second, is pushing a predefined set of jobs to try from the command line, similar to how things work now. It's often handy to have the try command for a specific job set in your shell history and it's a pain to open up treeherder for a simple push that you've memorized and run dozens of times. There are a few improvements we can do here:

  • We can move the curses ui feature of the hg trychooser extension into mach try.
  • We can use mozci to automatically keep the known list of jobs up to date. This is useful for things like generating the curses ui on the fly, validation and tab completion.
  • We can use mozci + taskcluster + buildbot bridge to provide a much more expressive API for scheduling jobs. For example, you could easily push a T-style try run.
  • We can expand some of the functionality in mach try for controlling how the harnesses are run, for example we could use it to enable some of the debugging features of the harness while investigating test failures.

Finally for those who are stuck in their ways, it should still be possible to have a "classic try syntax" front-end to the new mozci backend. As large as this change sounds, it could be mostly transparent to the user. While I'm certainly not a fan of the current try syntax, there's no reason to begrudge the people who are.

Closing words

Try syntax has served us well for 5 long years. But it's almost time to move on to something better. Soon a lot of new avenues will be open and tools will be created that none of us have thought of yet. I'd like to thank all of the people mentioned in this post for their contributions in this area and I'm very excited for what the future holds.

The future is bright, and change is for the better.

Categorieën: Mozilla-nl planet

Air Mozilla: Web QA Weekly Meeting

Mozilla planet - to, 27/08/2015 - 18:00

Web QA Weekly Meeting This is our weekly gathering of Mozilla'a Web QA team filled with discussion on our current and future projects, ideas, demos, and fun facts.

Categorieën: Mozilla-nl planet

Air Mozilla: Reps weekly

Mozilla planet - to, 27/08/2015 - 17:00

Reps weekly Weekly Mozilla Reps call

Categorieën: Mozilla-nl planet

Mike Taylor: Dynamically updating <meta viewport> in the year 2015.

Mozilla planet - to, 27/08/2015 - 07:00

18 months after writing the net-ward-winning article Dynamically updating <meta viewport> in the year 2014, I wrote some patches for Firefox for Android to make it possible to update a page's existing meta[name=viewport] element's content attribute and have the viewport be updated accordingly.

So when version 43 ships (at some point in 2015), code like this will work in more places than it did in 2014:

if(screen.width < 760) { viewport = document.querySelector("meta[name=viewport]"); viewport.setAttribute('content', 'width=768'); } if(screen.width > 760) { viewport = document.querySelector("meta[name=viewport]"); viewport.setAttribute('content', 'width=1024'); }

I'll just go ahead and accept the 2015 netaward now, thanks for the votes everyone, wowowow.

Categorieën: Mozilla-nl planet

Ian Bicking: Conway’s Corollary

Mozilla planet - to, 27/08/2015 - 07:00

Conway’s Law states:

organizations which design systems are constrained to produce designs which are copies of the communication structures of these organizations

I’ve always read this as an accusation: we are doomed to recreate the structure of our organizations in the structure of software projects. And further: projects cannot become their True Selves, cannot realize the most superior design, unless the organization is itself practically structureless. That only without the constraints of structure can the engineer make the truly correct choices. Michelangelo sculpted from marble, a smooth and uniform stone, not from an aggregate, where any hit with the chisel might reveal only the chaotic structure and fault lines of the rock and not his vision.

But most software is built, not revealed. I’m starting to believe that Conway’s observation is a corollary, not so clearly cause-and-effect. Maybe we should work with it, not struggle against it. (With age I’ve lost the passion for pointless struggle.) It’s not that developers can’t imagine a design that goes contrary to the organizational structure, it’s that they can’t ship those designs. What we’re seeing is natural selection. And when through force of will such a design is shipped, that it survives and is maintained depends on whether that the organization changed in the process, whether a structure was created to support that design.

A second skepticism: must a particular construction and modularity of code be paramount? Code is malleable, and its modularity is for the purpose of humans. Most of what we do disappears anyway when the machine takes over – functions are inlined, types erased, the pieces become linked, and the machine doesn’t care one whit about everything we’ve done to make the software comprehensible. Modularity is to serve our purposes. And sometimes organization structure serves a purpose; we change it to meet goals, and we shouldn’t assume the people who change it are just busybodies. But those changes are often aspirational, and so those changes are setting ourselves up for conflict as the new structure probably does not mirror the software design.

If the parts of an organization (e.g. teams, departments, or subdivisions) do not closely reflect the essential parts of the product, or if the relationship between organizations do not reflect the relationships between product parts, then the project will be in trouble… Therefore: Make sure the organization is compatible with the product architecture – Coplien and Harrison

So change the architecture! There’s more than one way to resolve these tensions.

A last speculation: as described in the Second System Effect we see teams rearchitect systems with excessive modularity and abstraction. Maybe because they remember all these conflicts, they remember all the times organizational structure and product motivations didn’t match architecture. The team makes an incorrect response by creating an architecture that can simultaneously embody all imagined organizational structures, a granularity that embodies not just current organizational tensions but also organizational boundaries that have come and gone. But the value is only in predicting future changes in structure, and only then if you are lucky.

Maybe we should look at Conway’s Law as a prescription: projects should only have hard boundaries where there are organizational boundaries. Soft boundaries and definitions still exist everywhere: just like we give local variables meaningful names (even though outside the function no one can tell the difference), we might also create abstractions and modularity that serve immediate and concrete purposes. But they should only be built for the moment and the task at hand. Extra effort should be applied to being ready to refactor in the future, not predicting and embodying those predictions in present modularity. Perhaps this is another rephrasing of Agile and YAGNI. Code is a liability, agency over that code is an asset.

Categorieën: Mozilla-nl planet

Air Mozilla: Bay Area Rust Meetup August 2015

Mozilla planet - to, 27/08/2015 - 04:00

Bay Area Rust Meetup August 2015 The SF Rust Meetup for August.

Categorieën: Mozilla-nl planet

Jonathan Griffin: Engineering Productivity Update, August 26, 2015

Mozilla planet - to, 27/08/2015 - 01:47

It’s PTO season and many people have taken a few days or week off.  While they’re away, the team continues making progress on a variety of fronts.  Planning also continues for GoFaster and addon-signing, which will both likely be significant projects for the team in Q4.


Treeherder: camd rolled out a change which collapses chunked jobs on Treeherder, reducing visual noise.  In the future, we plan on significantly increasing the number of chunks of many jobs in order to reduce runtimes, so this change makes that work more practical.  See camd’s blog post.  emorley has landed a change which allows TaskCluster job errors that occur outside of mozharness to be properly handled by Treeherder.

Automatic Starring: jgraham has developed a basic backend which supports recognizing simple intermittent failures, and is working on integrating that into Treeherder; mdoglio is landing some related database changes. ekyle has received sheriff training from RyanVM, and plans to use this to help improve the automated failure recognition algorithm.

Perfherder and Performance Testing: Datazilla has finally been decommissioned (R.I.P.), in favor of our newest performance analysis tool, Perfherder.  A lot of Talos documentation updates have been made at, including details about how we perform calculations on data produced by Talos.  wlach performed a useful post-mortem of Eideticker, with several takeaways which should be applicable to many other projects.

MozReview and Autoland: There’s a MozReview meetup underway, so expect some cool updates next time!

TaskCluster Support: ted has made a successful cross-compiled OSX build using TaskCluster!  Take it for a spin.  More work is needed before we can move OSX builds from the mac mini builders to the cloud.

Mobile Automation: gbrown continues to make improvements on the new |mach emulator| command which makes running Android tests locally on emulator very simple.

General Automation: run-by-dir is live on opt mochitest-plain; debug and ASAN coming soon.  This reduces test “bleed-through” and makes it easier to change chunking.  adusca, our Outreachy intern, is working to integrate the try extender into Treeherder.  And ahal has merged the mozharness “in-tree” configs with the regular mozharness config files, now that mozharness lives in the tree.

Firefox Automation: YouTube ad detection has been improved for firefox-media-tests by maja, which fixes the source of the top intermittent failure in this suite.

Bughunter: bc has got asan-opt builds running in production, and is working on gtk3 support. gps has enabled syntax highlighting in hgweb, and has added a new JSON API as well.  See gps’ blog post.

The Details Treeherder Perfherder/Performance Testing
  • talos cleanup and preparation to move in-tree
  • perfherder database cleanup in progress for simpler and more optimized queries. This is mainly preparatory work for making perfherder capable of managing/starring performance alerts, but as a bonus perfherder compare view should load virtually instantly once this is finished. 
  • most talos wiki docs are updated:
TaskCluster Support Mobile Automation
  •  [gbrown] Working on “mach emulator” support: wip can download and run 2.3, 4.3, or x86 emulator images. Integrating with other mach commands like “install” and “mochitest”.
  •  [gbrown] Updated mochitest manifests to run most dom/media mochitests on Android 4.3 (under review, bug 1189784)
Firefox and Media Automation
  • [maja_zf] Improved ad detection on YouTube for firefox-media-tests, which fixes our top intermittent failure for long-running playback tests.
General Automation
  •  run-by-dir is live for mochitest-plain (opt only); debug is coming soon, followed by ASAN.
  • Mozilla CI tools is moving from using BuildAPI as the scheduling entry point to use TaskCluster’ scheduling. This work will allow us to schedule a graph of buildbot jobs and their dependencies in one shot.
  • adusca is integrating into treeherder the ability to extend the jobs run for any push. This is based on the prototype. Follow along in
  • Git was deployed to the test machines. This is necessary to make the Firefox UI update tests work on them.
  • [ahal] merge mozharness in-tree configs with the main mozharness configs
  • Bug fixes to the ETL – fix bad lookups on hg repo, mostly l10n builds 
  • More error reporting on ETL – Structured logging has changed a little, handle the new variations, and be more elegant when it comes to unknowns, an complain when there is non-conformance.
  • Some work on adding hg `repo` table – acts as a cache for ETL, but can be used to calculate ‘per push’ statistics on OrangeFactor data.
  • Added Talos to the `perf` table – used the old Datazilla ETL code to fill the ES cluster.  This may speed up extracting the replicates, for exploring the behaviour of a test.
  • Enable deep queries – Effectively performing SQL join on Elasticsearch – first attempt did too much refactoring.  Second attempt is simpler, but still slogging through all the resulting test breakage WebDriver
  • Updated 
  • [ahal] helped review and finish contributor patch for switching marionette_client from optparse to argparse
  • Corrected UUID used for session ID and element IDs
  • Updated dispatching of various marionette calls in Gecko
  • [bc] Have asan-opt builds running in production. Finalizing patch. Still need to build gtk3 for rhel6 32bit in order to stop using custom builds and support opt in addition to debug.
  • Updated the hierarchical burndowns to EPM’s important metabugs that track features 
  • More config changes

Categorieën: Mozilla-nl planet

Support.Mozilla.Org: SUMO Questions Day this Thursday, 27 August 2015

Mozilla planet - to, 27/08/2015 - 00:00

The summer holidays are now over  so it’s time to start organizing a new SUMO Day!

What are SUMO Days?

A SUMO Day is that time of the week where everybody who loves doing support, contributors, admins, moderators gather together and try and answer all the incoming questions on the Mozilla support forums. This is a 24 hour event, we will start early during European mornings  and finish late during US Pacific evenings.

We are also hanging out having fun and helping each other in #sumo on IRC.

I want to participate! Where do I start?

Just create an account and then take some time to help with unanswered questions. We have an etherpad ready with all the details plus additional tips and resources.

If you get stuck with questions that are too difficult feel free to ping us on IRC #sumo or ask for help on the contributors forums.


SUMO Day will be moderated by madasan (EU morning/afternoon), marksc (EU afternoon/US morning), guigs (US morning/afternoon). We can always use more people to help moderate through the day so if you would like to do this just add your name in the etherpad!

What does it mean to be a SUMO Day moderator?

It’s easy! Just check out the forums and monitor incoming questions. Don’t forget to hang out on IRC on #sumo and the contributor forums and chat with the other SUMO Day participants about possible solutions to questions. As a moderator you also help out contributors who are stuck with difficult questions and need help.

Screensharing experiment

During this SUMO Day some of us will experiment with helping users via screen sharing. This is only open to senior contributors and forum moderators so if you’re one of them and you would like to participate please PM madasan.

 We’re trying to answer each and every incoming question on the support forum on Thursday so please join us. The more the merrier!


See you online and happy SUMO Day!

Categorieën: Mozilla-nl planet

Air Mozilla: Project Beehive: A HW/SW co-designed stack for runtime and architectural research.

Mozilla planet - to, 27/08/2015 - 00:00

 A HW/SW co-designed stack for runtime and  architectural research. In this talk we will present an overview of our recent research efforts focusing on Hw/Sw co-designed platform for heterogeneous many-core architectural research. The presented...

Categorieën: Mozilla-nl planet

Will Kahn-Greene: pytest-wholenodeid addon: v0.2 released!

Mozilla planet - wo, 26/08/2015 - 22:00
What is it?

pytest-wholenodeid is a pytest addon that shows the whole node id on failure rather than just the domain part. This makes it a lot easier to copy and paste the entire node id and re-run the test.

v0.2 released!

I wrote it in an hour today to make it easier to deal with test failures. Then I figured I'd turn it into a real project so friends could use it. Now you can use it, too!

I originally released v0.1 (the first release) and then noticed on PyPI that the description was a mess, so I fixed that and released v0.2.

To install:

pip install pytest-wholenodeid

It runs automatically. If you want to disable it temporarily, pass the --nowholeid argument to pytest.

More details on exactly what it does on the PyPI page.

If you use it and find issues, write up an issue in the issue tracker.

Categorieën: Mozilla-nl planet

Mozilla Community Ops Team: Weekly Update 2015-08-26

Mozilla planet - wo, 26/08/2015 - 20:36
Discourse Discourse UX improvements (@Leo, @yousef)

There are some changes to Discourse that should be made to make it more suitable to Mozillian’s needs

  • Status [In Progress]: See SSO update below. We can still use help researching and building the plugins that we need.
SSO (@Leo)

To improve the login experience for people using Discourse within Mozilla, bridge the gap in various ways between our different instances (e.g. single username across instances), and integrate better with Mozilla wore widely (with Mozillians integration, etc.)

  • Status [In Progress]: Still working on initial version of SSO server, currently working on finishing touches
Discourse Documentation (@Kensie)

To make Discourse more user friendly for Mozillians, we need some good documentation on how to use it

  • Status [In Progress]: Added a couple docs based on questions that came up during the week. Still need people to ask questions so we can answer them.

Putting all Discourse instances on one infrastructure, automated with Ansible and CloudFormation. This will help us keep the many Discourse instances we have secure, up to date and running common plugins easily; at scale. Also saves $$$ while allowing all of our instances to be HA.

  • Status [In Progress]: Turns out this isn’t quite production ready so we’re going to use our staging servers as a test-bed to iron out issues
MoFo Discourse migrations (@Yousef)

Migrating the Webmaker, Science and Hive Discourse instances to MECHADISCOURSE. This provides the teams with more stable Infra for their Discourse instances.

  • Status [In Progress]: Leo is currently implementing Webmaker login for the Teach The Web Discourse
Ansible (@Tanner)

Config management, initializes servers, will be used with MECHADISCOURSE as its first “big” project. Makes it 100x easier to set up servers.

  • Status [Done!]: Production-ready, Jenkins has been set up so jobs can be triggered on-demand.
Monitoring (@Tanner)

we need to set up a robust monitoring solution for our sites.

  • Status [In Progress]: Will be using Nagios. Need to write checks and config for Nagios, and then deploy the NPRE agent to servers.
Community Hosting (@Tanner, @yousef) Audit

We need to understand which sites are being actively used and which no longer need hosting, or need different hosting than they currently have

  • Status [In Progress]: Michael Buluma has started work on defining a MVP (minimum viable product) for a community website.

We will be moving away from OVH to simplify community hosting and save money.

  • Status [Stalled]: Waiting for progress on Participation Infrastructure side
Documentation (@Kensie) Discourse documentation (see above) Wiki update

Our wiki pages our out of date, and shouldn’t be under IT anymore

  • Status [In Progress]: Michael Buluma has started working on this.
Confluence (@Kensie)

Links to JIRA, will use it to help with project management, decision tracking.

  • Status [In Progress]: Help from Atlassian experts would be very welcome!
Matrix (@Leo)

Communication protocol which attempts to bind various different ones together – could possibly be used by us as a Telegram-esque IRC bouncer. Discussion and link to planning pad here.

  • Status [In Progress]: Started to investigate it, finding answers to various questions
MozFest Participation (@Kensie)

We are looking at ways our team can support MozFest, and planning session proposals that would be interesting to MozFest

Online Forum for Participants (@Tanner)

We are offering our services to host a Discourse instance for MozFest

  • Status [In Progress]: Putting up a Discourse instance as a PoC
Session Proposals (@Kensie)
  • Status [In Progress]: We have several proposals to submit, ideally by Friday (deadline is Monday).
  • Crowd didn’t work as hoped. It messed with a lot of plugins for Jenkins that relied on usernames, so ldap might work better.
Contribution Opportunities

Recap of contribution opportunities from status updates and ongoing contribution opportunities:

  • Discourse
    • Research/coding customizations
    • Documenting how to use Discourse/need questions to answer
    • Ansible expertise welcome
  • Monitoring
    • Nagios experts/mentors welcome
    • Community Hosting
    • Research MVP for community sites
  • Documentation
    • Discourse (see above)
    • Need writers to help drive wiki update
    • Atlassian experts welcome to help with Confluence/JIRA organization
Categorieën: Mozilla-nl planet

Air Mozilla: Product Coordination Meeting

Mozilla planet - wo, 26/08/2015 - 20:00

Product Coordination Meeting Duration: 10 minutes This is a weekly status meeting, every Wednesday, that helps coordinate the shipping of our products (across 4 release channels) in order...

Categorieën: Mozilla-nl planet

Mozilla: moet Thunderbird end-to-end-encryptie krijgen? -

Nieuws verzameld via Google - wo, 26/08/2015 - 17:16

Mozilla: moet Thunderbird end-to-end-encryptie krijgen?
End-to-end-encryptie zorgt ervoor dat e-mail volledig versleuteld is en alleen door de afzender en ontvanger kan worden gelezen. Opensourceontwikkelaar Mozilla vraagt zich nu af of het end-to-end-encryptie aan e-maiclient Thunderbird moet toevoegen.

Categorieën: Mozilla-nl planet

Air Mozilla: Bugzilla Development Meeting

Mozilla planet - wo, 26/08/2015 - 16:00

Bugzilla Development Meeting Help define, plan, design, and implement Bugzilla's future!

Categorieën: Mozilla-nl planet

Tantek Çelik: Vacation Mode @Yahoo? How About Evening Mode, Original Content Mode, and Walkie Talkies With Texting?

Mozilla planet - wo, 26/08/2015 - 15:47

Called it. I recently asked “When did you last eat without using digital devices at all?” and proposed a “dumb camera mode” where you could only take/review photos/videos and perhaps also write/edit text notes on your otherwise “smart” phone that usually made you dumber through notification distractions.

Five days later @Yahoo published a news article titled: “The One Feature Every Smartphone Needs: Vacation Mode” — something I’m quite familiar with, having recently completed a one week Alaska cruise during which I was nearly completely off the grid.

Evening Mode rather than Vacation Mode

Despite the proposals in the Yahoo article, I still think a “dumb” capture/view mode would still be better on a vacation, where all you could do with your device was capture photos/text/GPS and subsequently view/edit what you captured. Even limited notifications distract and detract from a vacation.

However, the idea of “social media updates only from people you’re close to, either geographically or emotionally” would be useful when not on vacation. I'd use that as an Evening Mode most nights.

Original content rather than “shares”

In addition, the ability to filter and only see “original content — no shared news stories on Facebook, no retweets on Twitter” would be great as reading prioritization — I only have a minute, show me only original content, or show me original content from the past 24h before any (re)shares/bookmarks etc.

This strong preference to prioritize viewing original content is I think what has moved me to read my Instagram feed, and in contrast nearly ignore my Twitter feed / home page, as well as actively avoid Facebook’s News Feed.

Ideally I’d use an IndieWeb reader, but they too have yet to find a way to distinguish original content posts in contrast to bookmarks or brief quotes / commentary / shares of “news” articles.

Tame your inbox? No, vacation should mean no inbox

The Yahoo article suggests: “tame your inbox in the same fashion, showing messages from your important contacts as they arrive but hiding everything else” and completely misses the point of disconnecting from all inbox stress while on vacation.

SMS smart phone texting frustrations vs stress-free iPod

While I was on the Alaska cruise, other members of my family did txt/SMS each other a bit, but due to the unreliability of the shipboard cell tower, it was more frustrating to them than not.

With my iPod, I completely opted out of all such electronic text comms, and thus never stressed about checking my device to coordinate.

IRL coordination FTW

Instead I coordinated as I remember doing as a kid (and even teenager) — we made plans when we were together, about the next time and place we would meetup, and our general plans for the day. Then we’d adjust our plans by having *in-person* conversations whenever we next saw each other.

Or if we needed to find each other, we would wander around the ship, to our staterooms, the pool decks, the buffet, the gym, knowing that it was a small enough world that we’d likely run into each other, which we did several times.

During the entire trip there was only one time that I lost touch with everyone and actually got frustrated. But even that just took a bit longer of a ship search. Of course even for that situation there are solutions.

Walkie Talkies!

My nephews and niece used walkie-talkies that their father brought on board, and that actually worked in many ways better than anyone’s fancy smart phones.

Except walkie-talkies can be a bit intrusive.

Walkie Texting?

My question is:

If walkie-talkies can send high quality audio back and forth in broadcast mode, why can’t they broadcast short text messages to everyone else on that same “channel” as well?

Then I found this on Amazon: TriSquare eXRS TSX300-2VP 900MHz FHSS Digital Two-Way Radio Two 2-way radios

  • Digital Two-Way Radio
  • spread spectrum and encrypted
  • text mesaging between radios

(Discontinued by Manufacturer)

Anybody have one or a similar two-way radio that also supports texting?

Or would it be possible to do peer-to-peer audio/texting purely in software on smart “phones” peer-to-peer over bluetooth or wifi without having to go through a central router/tower?

That would seem ideal for a weekend road trip, say to Tahoe, or to the desert, or perhaps even for camping, again, maybe in the desert, like when you choose to escape from the rest of civilization for a week or more.

Categorieën: Mozilla-nl planet

Nicholas Nethercote: What does the OS X Activity Monitor’s “Energy Impact” actually measure?

Mozilla planet - wo, 26/08/2015 - 02:37

Activity Monitor is a tool in Mac OS X that shows a variety of real-time process measurements. It is well-known and its “Energy Impact” measure (which was added in Mac OS X 10.9) is often consulted by users to compare the power consumption of different programs. Apple support documentation specifically recommends it for troubleshooting battery life problems, as do countless articles on the web.

However, despite its prominence, the exact meaning of the “Energy Impact” measure is unclear. In this blog post I use a combination of code inspection, measurements, and educated guesses to hypothesize how it is computed in Mac OS X 10.9 and 10.10.

What is known about “Energy Impact”?

The following screenshot shows the Activity Monitor’s “Energy” tab.

There are no units given for “Energy Impact” or “Avg Energy Impact”.

The Activity Monitor documentation says the following.

Energy Impact: A relative measure of the current energy consumption of the app. Lower numbers are better.

Avg Energy Impact: The average energy impact for the past 8 hours or since the Mac started up, whichever is shorter.

That is vague. Other Apple documentation says the following.

The Energy tab of Activity Monitor displays the Energy Impact of each open app based on a number of factors including CPU usage, network traffic, disk activity and more. The higher the number, the more impact an app has on battery power.

More detail, but still vague. Enough so that various other  people have wondered what it means. The most precise description I have found says the following.

If my recollection of the developer presentation slide on App Nap is correct, they are an abstract unit Apple created to represent several factors related to energy usage meant to compare programs relatively.

I don’t believe you can directly relate them to one simple unit, because they are from an arbitrary formula of multiple factors.

[…] To get the units they look at CPU usage, interrupts, and wakeups… track those using counters and apply that to the energy column as a relative measure of an app.

This sounds plausible, and we will soon see that it appears to be close to the truth.

A detour: top

First, a necessary detour. top is a program that is similar to Activity Monitor, but it runs from the command-line. Like Activity Monitor, top performs periodic measurements of many different things, including several that are relevant to power consumption: CPU usage, wakeups, and a “power” measure. To see all these together, invoke it as follows.

top -stats pid,command,cpu,idlew,power -o power -d

(A non-default invocation is necessary because the wakeups and power columns aren’t shown by default unless you have an extremely wide screen.)

It will show real-time data, updated once per second, like the following.

PID COMMAND %CPU IDLEW POWER 50300 firefox 12.9 278 26.6 76256 plugin-container 3.4 159 11.3 151 coreaudiod 0.9 68 4.3 76505 top 1.5 1 1.6 76354 Activity Monitor 1.0 0 1.0

The PID, COMMAND and %CPU columns are self-explanatory.

The IDLEW column is the number of package idle exit wakeups. These occur when the processor package (containing the cores, GPU, caches, etc.) transitions from a low-power idle state to the active state. This happens when the OS schedules a process to run due to some kind of event. Common causes of wakeups include scheduled timers going off and blocked I/O system calls receiving data.

What about the POWER column? top is open source, so its meaning can be determined conclusively by reading the powerscore_insert_cell function in the source code. (The POWER measure was added to top in OS X 10.9.0 and the code has remain unchanged all the way through to OS X 10.10.2, which is the most recent version for which the code is available.)

The following is a summary of what the code does, and it’s easier to understand if the %CPU and POWER computations are shown side-by-side.

|elapsed_us| is the length of the sample period |used_us| is the time this process was running during the sample period %CPU = (used_us * 100.0) / elapsed_us POWER = if is_a_kernel_process() 0 else ((used_us + IDLEW * 500) * 100.0) / elapsed_us

The %CPU computation is as expected.

The POWER computation is a function of CPU and IDLEW. It’s basically the same as %CPU but with a “tax” of 500 microseconds for each wakeup and an exception for kernel processes. The value of this function can easily exceed 100 — e.g. a program with zero CPU usage and 3,000 wakeups per second will have a POWER score of 150 — so it is not a percentage. In fact, POWER is a unitless measure because it is a semi-arbitrary combination of two measures with incompatible units.

Back to Activity Monitor and “Energy Impact” MacBook Pro running Mac OS X 10.9.5

First, I did some measurements with a MacBook Pro with an i7-4960HQ processor running Mac OS X 10.9.5.

I did extensive testing with a range of programs: ones that trigger 100% CPU usage; ones that trigger controllable numbers of idle wakeups; ones that stress the memory system heavily; ones that perform frequent disk operations; and ones that perform frequent network operations.

In every case, Activity Monitor’s “Energy Impact” was the same as top‘s POWER measure. Every indication is that the two are computed identically on this machine.

For example, consider the data in the following table,  The data was gathered with a small test program that fires a timer N times per second; other than extreme cases (see below) each timer firing causes an idle platform wakeup.

----------------------------------------------------------------------------- Hz CPU ms/s Intr Pkg Idle Pkg Power Act.Mon. top ----------------------------------------------------------------------------- 2 0.14 2.00 1.80 2.30W 0.1 0.1 100 4.52 100.13 95.14 3.29W 5 5 500 9.26 499.66 483.87 3.50W 25 25 1000 19.89 1000.15 978.77 5.23W 50 50 5000 17.87 4993.10 4907.54 14.50W 240 240 10000 32.63 9976.38 9194.70 17.61W 485 480 20000 66.66 19970.95 17849.55 21.81W 910 910 30000 99.62 28332.79 25899.13 23.89W 1300 1300 40000 132.08 37255.47 33070.19 24.43W 1610 1650 50000 160.79 46170.83 42665.61 27.31W 2100 2100 60000 281.19 58871.47 32062.39 29.92W 1600 1650 70000 276.43 67023.00 14782.03 31.86W 780 750 80000 304.16 81624.60 258.22 35.72W 43 45 90000 333.20 90100.26 153.13 37.93W 40 42 100000 363.94 98789.49 44.18 39.31W 38 38

The table shows a variety of measurements for this program for different values of N. Columns 2–5 are from powermetrics, and show CPU usage, interrupt frequency, and package idle wakeup frequency, respectively. Column 6 is Activity Monitor’s “Energy Impact”, and column 7 is top‘s POWER measurement. Column 6 and 7 (which are approximate measurements) are identical, modulo small variations due to the noisiness of these measurements.

MacBook Air running Mac OS X 10.10.4

I also tested a MacBook Air with an i5-4250U processor running Mac OS X 10.10.4. The results were substantially different.

----------------------------------------------------------------------------- Hz     CPU ms/s   Intr        Pkg Idle   Pkg Power Act.Mon. top -----------------------------------------------------------------------------      2     0.21        2.00       2.00     0.63W   0.0    0.1    100     6.75       99.29      96.69     0.81W   2.4    5.2    500    22.52      499.40     475.04     1.15W   10     25   1000    44.07      998.93     960.59     1.67W   21     48   3000   109.71     3001.05    2917.54     3.80W   60    145   5000    65.02     4996.13    4781.43     3.79W   90    230   7500   107.53     7483.57    7083.90     4.31W   140    350  10000   144.00     9981.25    9381.06     4.37W   190    460

The results from top are very similar to those from the other machine. But Activity Monitor’s “Energy Impact” no longer matches top‘s POWER measure. As a result it is much harder to say with confidence what “Energy Impact” represents on this machine. I tried tweaking the previous formula so that the idle wakeup “tax” drops from 500 microseconds to 180 or 200 microseconds and that gives results that appear to be in the ballpark but don’t match exactly. I’m a bit skeptical whether Activity Monitor is doing all its measurements at the same time or not. But it’s also quite possible that other inputs have been added to the function that computes “Energy Impact”.

What about “Avg Energy Impact”?

What about the “Avg Energy Impact”? It seems reasonable to assume it is computed in the same way as “Energy Impact”, but averaged over a longer period. In fact, we already know that period from the Apple documentation that says it is the “average energy impact for the past 8 hours or since the Mac started up, whichever is shorter.”

Indeed, when the Energy tab of Activity Monitor is first opened, the “Avg Energy Impact” column is empty and the title bar says “Activity Monitor (Processing…)”. After a few seconds the “Avg Energy Impact” column is populated with values and the title bar changes to “Activity Monitor (Applications in last 8 hours)”. If you have top open during those 5–10 seconds can you see that systemstats is running and using a lot of CPU, and so presumably the measurements are obtained from it.

systemstats is a program that runs all the time and periodically measures, among other things, CPU usage and idle wakeups for each running process (visible in the “Processes” section of its output.) I’ve done further tests that indicate that the “Avg Energy Impact” is almost certainly computed using the same formula as “Energy Impact”. The difference is that the the measurements are from the past 8 hours of wake time — i.e. if a laptop is closed for several hours and then reopened, those hours are not included in the calculation — as opposed to the 1, 2 or 5 seconds of wake time used for “Energy Impact”.

battery status menu

Even more prominent than Activity Monitor is OS X’s battery status menu. When you click on the battery icon in the OS X menu bar you get a drop-down menu which includes a list of “Apps Using Significant Energy”.

Screenshot of the OS X battery status menu

How is this determined? When you open this menu for the first time in a while it says “Collecting Power Usage Information” for a few seconds, and if you have top open during that time you see that, once again, systemstats is running and using a lot of CPU. Furthermore, if you click on an application name in the menu Activity Monitor will be opened and that application’s entry will be highlighted. Based on these facts it seems reasonable to assume that “Energy Impact” is again being used to determine which applications show up in the battery status menu.

I did some more tests (on my MacBook Pro running 10.9.5) and it appears that once an energy-intensive application is started it takes about 20 or 30 seconds for it to show up in the battery status menu. And once the application stops using high amounts of energy I’ve seen it take between 4 and 10 minutes to disappear. The exception is if the application is closed, in which case it disappears immediately.

Finally, I tried to determine the significance threshold. It appears that a program with an “Energy Impact” of roughly 20 or more will eventually show up as significant, and programs that have much higher “Energy Impact” values tend to show up more quickly.

All of these battery status menu observations are difficult to make reliably and so should be treated with caution. They may also be different in OS X 10.10. It is clear, however, that the window used by the battery status menu is measured in seconds or minutes, which is much less than the 8 hour window used for “Avg Energy Impact”.

An aside: systemstats is always running on OS X. The particular invocation used for the long-running instance — the one used by both Activity Monitor and the battery status menu — takes the undocumented --xpc flag. When I tried running it with that flag I got an error message saying “This mode should only be invoked by launchd”. So it’s hard to know how often it’s making measurements. The output from vanilla command-line invocations indicate it’s about every 10 minutes.

But it’s worth noting that systemstats has a -J option which causes the CPU usage and wakeups for child processes to be attributed to their parents. It seems likely that the --xpc option triggers the same behaviour because the Activity Monitor does not show “Avg Energy Impact” for child processes (as can be seen in the screenshot above for the login, bash and vim processes that are children of the Terminal process). This hypothesis also matches up with the battery status menu, which never shows child processes. One consequence of this is that if you ssh into a Mac and run a power-intensive program from the command line it will not show up in Activity Monitor’s energy tab or the battery status menu, because it’s not attributable to a top-level process such as Terminal! Such processes will show up in top and in Activity Monitor’s CPU tab, however.

How good a measure is “Energy Impact”?

We’ve now seen that “Energy Impact” is used widely throughout OS X. How good a measure is it?

The best way to measure power consumption is to actually measure power consumption. One way to do this is to use an ammeter, but this is difficult. Another way is to measure how long it takes for the battery to drain, which is easier but slow and requires steady workloads. Alternatively, recent Intel hardware provides high-quality estimates of processor and memory power consumption that are relatively easy to obtain.

These approaches all have the virtue of measuring or estimating actual power consumption (i.e. Watts). But the big problem is that they are machine-wide measures that cannot be used on a per-process basis. This is why Activity Monitor uses several proxy measures — ones that correlate with power consumption — which can be measured on a per-process basis. “Energy Impact” is a hybrid of at least two different proxy measures: CPU usage and wakeup frequency.

The main problem with this is that “Energy Impact” is an exaggerated measure. Look at the first table above, with data from the 10.9.5 machine. The variation in the “Pkg Power” column — which shows the package power from the above-mentioned Intel hardware estimates — is vastly smaller than the variation in the “Energy Impact” measurements. For example, going from 1,000 to 10,000 wakeups per second increases the package power by 3.4x, but the “Energy Impact” increases by 9.7x, and the skew gets even worse at higher wakeup frequencies. “Energy Impact” clearly weights wakeups too heavily. (In the second table, with data from the 10.10.4 machine, the weight given to wakeups is less, but still too high.)

Also, in the first table “Energy Impact” actually decreases when the timer frequency gets high enough. Presumably this is because the timer interval is so short that the OS has trouble putting the package into a idle power state. This leads to the absurd result that firing a timer at 1,000 Hz has about the same “Energy Impact” value as firing one at 100,000 Hz, when the package power of the latter is about 7.5x higher.

Having said all that, it’s understandable why Apple uses formulations of this kind for “Energy Impact”.

  • CPU usage and wakeup frequency are probably the two most important factors affecting a process’s power consumption, and they are factors that can be measured on a per-process basis.
  • Having a single measure makes things easy for users; evaluating the relative important of multiple measures is more difficult.
  • The exception for kernel processes (which always have an “Energy Impact” of 0) avoids OS X itself being blamed for high power consumption. This makes a certain amount of sense — it’s not like users can close the kernel — while also being somewhat misleading.

If I were in charge of Apple’s Activity Monitor product, I’d do two things.

  1. I would compute a new formula for “Energy Impact”. I would measure the CPU usage, wakeup frequency (and any other inputs) and actual power consumption for a range of real-world programs, on a range of different Apple machines. From this data, hopefully a reasonably accurate model could be constructed. It wouldn’t be perfect, and it wouldn’t need to be perfect, but it should be possible to come up with something that reflects actual power consumption better than the existing formulations. Once formulated, I would then test the new version against synthetic microbenchmarks, like the ones I used above, to see how it holds up. Given the choice between accurately modelling real-world applications and accurately modelling synthetic microbenchmarks, I would definitely favour the former.
  2. I would publicly document the formula that is used so that developers can actually tell how their applications are being evaluated, and can optimize for that measure. You may think “but then developers will be optimizing for a synthetic measure rather than a real one” and you’d be right. That’s an inevitable consequence of giving a synthetic measure such prominence, and all the more reason for improving it.

“Energy Impact” is a flawed measure of an application’s power consumption. Nonetheless, it’s what many people use at this moment to evaluate the power consumption of OS X applications, so it’s worth understanding. And if you are an OS X application developer who wants to reduce the “Energy Impact” of your application, it’s clear that it’s best to focus first on reducing wakeup frequency, and then on reducing CPU usage.

Because Activity Monitor is closed source code I don’t know if I’ve characterized “Energy Impact” exactly correctly. The evidence given above indicates that I am close on 10.9.5, but not as close on 10.10.4. I’d love to hear if anybody has evidence that either corroborates or contradicts the conclusions I’ve made here. Thank you.

Categorieën: Mozilla-nl planet

James Long: The Seasonal Blog Redux

Mozilla planet - wo, 26/08/2015 - 02:00

It's that time of year again! The weeds are growing, the air is thick and stagnant, and I just deployed another refactoring of my blog. "Why does he keep working on his blog," you're thinking, "when I could do all of that with a static-site generator like Jekyll?"

Writing my own blogging engine has been one of the best decisions I've made. Having a side project that I actually use and get value from is a great place to implement my own ideas, or try out new libraries. Every now and then it's fun to throw it back in the furnance, get it hot, and start shaping it with new ideas.

A blog is a great litmus test for new libraries (remember, I have an admin site behind this). You have to deal with routing, forms, interfacing with things like the CodeMirror editor, server-side rendering, async data fetching, and more. I feel like it really hits most of the pain-points of big client-side apps, even if it's a relatively small project. The only thing it doesn't stress is a complex shape of data: the data I get back from the server is pretty simple, and and more complex apps would need something better to handle complex data.

But even then, contrasting my simple code with more complex solutions makes it really clear why they are solved that way. Take GraphQL for example; I definitely don't need it, but there are a few places in my code that would obviously be way more complex if my data was more complex, and it's clear what GraphQL is trying to solve.

Last time I completely rewrote my blog, I learned about react-router, Webpack (with babel integration), server-side rendering (universal apps), Docker, and various aspects of React.

This time, I learned about Redux, immutable-js, and having a fully snapshot-able app state.

What do I mean by snapshot? My entire app state (even component local state) lives as a nested tree with a single root. I can simply serialize that root, and load it in later to see the app exactly how it was a that point in time. Here's a fun trick to show you what I mean: copy all of this text, press cmd+shift+k and paste it in. That's my admin interface with 2 errors; you're seeing it exactly at that point.

Redux What?

Redux is library that complements React and manages application state. It provides a simple workflow for updating application state and allowing React components to subscribe to state changes. While it borrows ideas from Elm, Flux, and various fancy-sounding abstractions, it's actually quite simple.

It embraces an idea currently bubbling up in the UI community: make state explicit and immutable, use pure functions as much as possible, and push all side effects to the edge of your app. In fact, the entire state exists as a single atom: a deeply nested JS object that contains everything you need to render the current UI.

This seems radical, but it's the right way to do things.

  1. Your frontend is made up of simple pure functions that take inputs and return outputs. This makes it extremely easy to test, rationalize about, and do things like hot-reloading. Separating state from code just makes things simpler.

  2. Your state exists as a single object that is never mutated. Normally it's a JS object, but it could be an immutable.js object or even a client-side database. Thats right, putting state in one place means you could even use a database for state. That's not even the best part: with a single atom and immutability, you can easily snapshot and resume the app at any point in time!

Redux provides the ability for the UI to subscribe to changes to specific parts of the app state. Generally only top-level components in the UI select state from the global app state atom, and most components are pure: they simply receive data and render it.

The library has roots in flux, Facebooks original library for handling state. The main similarity is you dispatch actions to change state. An action is simply a JavaScript object with a type field and any other fields as arguments. These actions are dispatched across all registered "reducers", which are functions that take state and an action and return new state: (state, action) -> newState. All new states are grouped together into a new single atom app state.

The real-world is grey and misty like a London street. You can't use pure functions and a global app state atom for everything. Asynchronous code is inherently side-effecting, but by isolating it to a specific part of your app, the rest of the world doesn't have to be bothered with things such as promises or observables. Updating the app state and rendering the UI is completely synchronous, but "async action creators" are functions which have the ability to dispatch multiple actions over time.

Local state is obviously desirable in certain situations, although it's less important than you think. UIs tend to require global state: many different parts of the UI need access to the same data. However, local state is important mainly for performance reasons. We are not out of luck though: we can get local state back by scoping part of the global app state atom to single components, as CircleCI did.

The frontend space is super interesting these days, and there's a lot to talk about. Follow me as I blog more about what I learned rewriting my blog with these ideas. I'll walk through specific techniques in my blog's code dealing with:

  • Using immutable.js for app state
  • Integrating Redux with react-router
  • Data fetching and asynchronous action creators
  • Server-side rendering
  • Local state

Feel free to peruse my blog's code in the meantime.

Categorieën: Mozilla-nl planet

Giorgio Maone: WebExtensions FAQ

Mozilla planet - wo, 26/08/2015 - 01:36

WebExtensions are making some people happy, some people angry, many people ask questions.
Some of the answers can be found here, more to come as add-on developers keep discussing this hot topic.
My favourite one: No, your add-ons' ability and your own creativity won't be limited by the new API.

Categorieën: Mozilla-nl planet

Software Freedom Day Utrecht

Mozilla-NL nieuws - wo, 26/08/2015 - 00:06
Date: zaterdag, 19 september, 2015 - 10:00 tot 17:00Wie: OpenbaarTags: Community

Over de hele wereld organiseren allerlei communities op deze dag een Software Freedom Day. De NLLGG doet dit jaar uiteraard ook weer mee, met verschillende sessies en andere manieren om Linux en open source software in de spotlight te zetten.

meer informatie:

Flickr tags: #Softwarefreedomday2015
Categorieën: Mozilla-nl planet