Mozilla-nl planet
John Ford: Tooltool: a way to help deploy development tools to CI machines in an agile way
One fairly common problem faced by teams at Mozilla is getting their tools plugged into the Release Engineering continuous testing infrastructure. In the past, the workflow has been for the team to file a bug that required a Release Engineer to figure out how to build, package and deploy a new set of tools. This is not the most scalable approach. During our recent Release Engineering workweek, we brainstormed on how to improve scaling and agility by empowering developers to help build and deploy new tools for our CI machines.
Tooltool basicsTooltool is a client side program written in Python that uses a file manifest in concert with HTTP servers to materialize large binary payloads for use in a job. The manifests are JSON files which list details of individual files. Each file is represented in the JSON by a dictionary with the keys “filename”, “digest”, “size” and “algorithm”. An example is located here. The current JSONEncoder and JSONDecoder derived classes only understand how to work with these keys. Making the JSON encoder and decoder work with an extensible version of the manifests should not be difficult.
Tooltool fetch APIWhen Tooltool needs to download a file from the file server it does so by creating a URL. In the current implementation, there is a single base url that is used to construct the URLs that are to be fetched. A good change to make would be to convert this single URL to a list of possible base urls. The address of the file to fetch is generated using a string concatenation of the base URL, a slash, the hashing algorightm’s name, another slash and the full hash value represented in hexadecimal. Given a base url of “http://files.r.us:8080/tooltool” and a file that has a SHA512 hash value of 0123456789abcdef, Tooltool will try to fetch “http://files.r.us:8080/tooltool/sha512/0123456789abcdef”.
There is no server component for Tooltool yet. As a result, the file server is currently implemented as a simple directory on an HTTP host that has the correct directory structure for responding to requests. I’ve started working on a server side component but it isn’t finished. The server side component would allow for easy uploads by developers, easy listing of contents on the server and a way to store files in a nicer way on the server’s file system.
Using TooltoolTooltool has four commands presently: list, validate, add and fetch. There are global options and command arguments. All terminal interactions after the option parser finishes is done through the Python logging API. The default is to print logging.INFO and higher messages. Currently, the following global options exist:
- -q/--quiet tells Tooltool to print only logging.ERROR and higher messages
- -v/--verbose specifies to print logging.INFO and higher
- -m/--manifest <file> instructs Tooltool to reference a manifest file located at the specified path
- -d/--algorithm <algorithm> instructs Tooltool to use the specified algorithm
- -o/--overwrite tells Tooltool to overwrite a local file if the filename matches the manifest but the hash value is different to the manifest
- --url specifies the base url to be used for remote operations
The two most basic commands list a manifest and validate the local files against the manifest. The list command lists out all of the files in the manifest as well as whether they are present and/or valid. The return code from listing is zero unless there was an error in listing the files. Absent or invalid files will still result in an exit code of zero if there was no error in the listing process. The validate command is used to check if all the files in the manifest are present and valid. The exit code for validating is zero if all files in the manifest are present and their hash matches the manifest. It is non-zero if any file is missing locally or the file does not have the same hash as the manifest.
Listing and validating a Tooltool manifest. Note the exit codes
FetchingThe fetch command is used to materialize files locally. Before fetching a file from a remote host, Tooltool will validate local files which match the filename specified in the manifest. The default behaviour when a local file matches the filename but not the hash value is to exit with a non-zero exit code. If –o or –overwrite is specified on the command line Tooltool will overwrite the local file without confirmation with the remote file. The local file will be truncated as soon as Tooltool attempts to start writing the remote file locally.
Fetching a file that exists locally with contents differing from manifest. Note the difference in behaviour when using --overwrite
AddingFiles are added to manifests with the add command. This command looks for a manifest using either the default name of manifest.tt or by the value specified by the -m/–manifest command line option. If the manifest does not exist already on the file system, a new one will be created to store the first file given. An error message will be displayed if a file is added a second time, regardless of whether or not the local contents are the same as what is in the manifest. There is currently no logic to remove a file from a manfiest or overwrite a manifest entry.
- Creating a Tooltool manifest, adding a file to it, trying to re-add file and listing contents. Note that even though ‘a’ didn’t change, tooltool didn’t allow it to be added a second time or overwrite the manifest’s entry
Tooltool is a generic lookaside cache. My intention is to keep it as general as possible by not including logic to deal with payloads. We currently include a bootstrap script in the b2g manifests that understands how to take the rest of the payload and set up a working toolchain. Using a bootstrap script means that Tooltool can be tool agnostic while still allowing complex operations on the fetched tools. A standardized bootstrap script name and interface that is called by Tooltool might make sense. I’d also like to finish the server-side component that has an interface to upload files with and a method for storing files in a less obtuse method than just mirroring the API paths. A command for removing files from a manifest would be helpful, as would the ability to update a manifest with the contents of the directory as they exist right now. Another really cool feature would be the ability to configure a system wide cache directory where files are downloaded to. Once the server component is working, I’d like to add a way for servers to automatically sync their file stores when they are asked for a file that exists on another server but not locally.
Tooltool is GPLv2 and the source is available on github. The best way to contribute code is to send a pull request on github. Things are more likely to be merged if they pass the whole test suite (make check), have tests, improve Tooltool and don’t make Tooltool overly specific.
Meeting Notes from the Mozilla community: Mobile Meeting Minutes: 2012-05-16
- Wednesdays – 9:30am Pacific, 12:30pm Eastern, 16:30 UTC
- Dial-in: conference# 95312
- US/International: +1 650 903 0800 x92 Conf# 95312
- US toll free: +1 800 707 2533 (pin 369) Conf# 95312
- Canada: +1 416 848 3114 x92 Conf# 95312
- irc.mozilla.org #mobile for backchannel
- Warp Core Vidyo Room
- Next merge is 2012-06-05
- Beta
- Fx14 Beta 1 went live in the Market on 2012-05-15
- Fx14 Beta 2 build was tagged/built on 2012-05-15 and planned to be in Market on 2012-05-18
- Monitoring Beta 1
- Now that Fx14 Beta 1 is in the Market (\o/) we are monitoring feedback and crash-stats closely. Check out the reddit thread and the reviews from the Market as examples.
- Add-ons
- We have add-ons listed on AMO that are just not compatible with Firefox Mobile (Native). We are going through the list, installing and checking the code (where possible) to verify which ones actually work in Firefox Mobile (Native). We also need to touch base with several of the add-on developers to see about getting compatible updates for Firefox Mobile (Native).
Suggested format:
- What did you do last week?
- What are working on this week?
- Anything blocking you?
Please keep your update to under 2 minutes!
James W. (snorp)- Last week
- Started working on fullscreen support for Flash once again, made a little progress
- This week
- More work on the above, only remaining problem is an issue on ICS
- Last week:
- bug 752939 – touch events have wrong coordinates on tab switch
- bug 753334 – audit code related to java.nio.Buffer and fix up inconsistencies
- bug 751262 – potential memory leak using JNI buffers
- bug 753845 – race condition when switching tabs
- worked on bug 748384 – RTL pages don’t work correctly
- landed bug 749384 – nytimes slow to respond to stop a fling on touchdown
- Next week:
- Take a look at bug 753525 and dependencies (clicking on things is hard) and see what we can do
- Blockers:
- none
Last week:
- Trying to test Bug 745340 Improve disk cache smart sizing for mobile – led to…
- Bug 755324 Out of memory after 225 page loads
- Bug 725094 Robocop error handling
- Bug 755556 Robocop: upgrade to robotium-solo-3.2.1.jar
This week:
- More Robocop improvements
- Last week
- Working on getting bug 607417 landed – was bounced due to;
- bug 753784 – mask layers are broken with tiled textures/npot in pot textures
- bug 753742 – odd drawing problems on pages with iframes and overflow:hidden
- This week
- bug 753742 – odd drawing problems on pages with iframes and overflow:hidden (fixed, pending review)
- bug 753784 – mask layers are broken with tiled textures/npot in pot textures
- Interviewing, London office events
- Last Week
- Fixed bug 743468 – Entering accented characters on HKB causes mobile.twitter.com to eat subsequent input characters -> But this caused some regressions
- Fixed bug 742267 – URL content is cleared when entering a new character from the hardware keyboard
- This Week
- Fixing bug 751864 – Typing character followed by space adds the same character another time on ICS
- Fixing bug 751513 – Typing characters in the contenteditable div causes the whole line to be deleted
- Last week:
- Loads of Profile Migration bugs (752907, 753175, 746860, 752492, 752444, 754224 and more)
- This week:
- More Profile Migration bugs (755383)
- Tests for Profile Migration & BrowserProvider
- No Blockers
- Done
- bug 753625 – Fennec can get into a session restore crash loop
- Investigated bug 719694 – Video’s do not play on mobile Vimeo
- Mentoring bug 741655 – Add more controls to download manager
- WIP bug 586885 – show search suggestions when entering text in awesome bar
- Next
- bug 752245 – Interactions on the Google Play website are slow
- bug 747388 – .part files from failed downloads are not removed from sdcard/download
- Last Week:
- Implemented a horizontal progress bar.
- Tests on regression pending.
- Fixed onApplicationPause() to work properly – bug 751690
- Implemented a horizontal progress bar.
- This Week:
- Posted 2 different patches for showing menu items properly on first time – bug 753200
- Dependent on menu items from addons.
- Trying to do a proper restart on locale change – bug 732572
- Android silently kills Fennec on second try.
- Android widget is shaping up well – http://cl.ly/0Q2D1o3p0j0Q2J103s1x
- Require newer resources. ICS (XHDPI) devices add more padding, need more XHDPI devices to test.
- Custom menu is shaping up well – http://cl.ly/22381N0m3A003U1i0940
- Posted 2 different patches for showing menu items properly on first time – bug 753200
- Blockers:
- Frustration in fixing doRestart() for locale change. :(
Last week:
- Vacation
This week:
- bug 732052 – XUL Scale (video scrubber) elements should support touch events
- Webapps stuff
- Prompt service stuff (tests, and some features needed for random bugs)
Done:
- bug 752681 – Make official XUL Fennec builds install on xlarge tablets only
- bug 754947 – Use the default search engine for keyword.URL searches in Fennec
- bug 754637 – Don’t zoom in to list items or blockquotes on double-tap
- bug 740878 – Remove dead code from mobile/android
- hung out on reddit
Next:
- bug 707571 – user-scalable property of viewport meta tag is ignored
- PTO next week
Done:
- Profile migrator issues
- Migrate bookmark types correctly – bug 754276
- Crash from badly migrated bookmark types – bug 754286
- Desktop bookmarks folder shows up when it shouldn’t – bug 753534
- Sync account credentials fail to migrate correctly – bug 752514
- Awesome screen tab titles get cut off in some locales – bug 753880
- SessionStore bug that caused tab count to fail to update – bug 752688
- Add a way to remove history entries – bug 671131
- Fixed up ContentPermissionPrompt.js (and wrote some tests for our geolocation notifications) – bug 729485
Next:
- Continue looking into fixing up our notifications (and look into adding tests for them!) – bug 739757
- Blockers as they come up
- Last Week:
- Worked on outstanding font inflation release blockers.
- This Week:
- bug 749186 : Crash in nsFontInflationData::FindFontInflationDataFor() (turns out this wasn’t fixed. :< )
- bug 747267 : Very bad font inflation when filing a new bug
Last week:
- presented at DMCA hearings
- then flew to London
- various London office launch events
- reviews, interviews, triage etc. etc.
This week
- fly home
- sleep
Done:
- Triage and Reviews
- Landing patches, but usually not my own
Next:
- Make sure non one is blocked on betaN blockers
- Keep momentum going on any Fx15 work
- Code?
- collating beta UX feedback, making a list
- Finish tweaks to tab menu designs
- Work with bnicholson on search suggest
- Work with sriram on Widget / Custom Menu
- Reader mode assets and designs for reading list
- This week
- Welcome intern, Eric Wei (irc: xwei) to Mobile QA! He’ll be helping with shipping Fennec Native, robocop automation, and other mobile magic.
- Beta testday this friday. Please, please! come and camp out in #testday to help with testing and questions.
- Qualifying Beta 2
- Ping QA if you want bugs looked at or add qawanted. we’re prioritizing testing around migration, flash, gfx-related, and stability issues.
Note: Socorro team is ramping up for b2g, rapid beta, and some other projects.
- Socorro bugs:
- bug 726385 Please create skiplist implementation for the Java field
- bug 727286 – Incorrect messaging when failing to submit a crash report
- bug 672606 – Aggregate numbers and topcrashes for Nightly and Aurora channels based on build ID date instead of crash date
- Skiplist:
- bug 749608 – Add dvm* to skiplist for Android
- closed off as invalid
- bug 749608 – Add dvm* to skiplist for Android
- Breakpad Integration bugs:
- bug 750348 A large number of devices are not showing up with the device identifiers
- bug 738168 Moving Fennec to the SDCard will prevent crash reports from being copied to the profile, ie no crash report will be reported to Socorro
- bug 732629 crash report failed to send due to : javax.net.ssl.SSLException: Not trusted server certificate
- Please see Platform Meeting Notes for Stability report
- Only a couple of questions on SUMO so far–a good sign for the docs, so we’ll start L10n in earnest today
- Google Play reviews summary, 330 reviews of Beta in first 24 hours; our rating based on those reviews is a solid 4.0 stars. 160 5-star reviews already!
- Themes: fast, wow, flash, memory, smooth, scrolling, sexy, this is now my default browser
- Projections: if we continue at this rate, we could see a 4.0 star overall rating by the end of beta (considerations for tablet distribution strategy?)
- Analysis: will look at specific device trends today, thunderbolt, defy, and tablets
- Requests: gestures and swipe to remove tabs
Meeting Notes from the Mozilla community: Firefox/Gecko Delivery Meeting Minutes: 2012-05-16
« previous week | index | next week »
Planning Meeting Details
- Wednesdays – 11:00am PDT, 18:00 UTC
- Mountain View Offices: Warp Core Conference Room
- Toronto Offices: Finch Conference Room
- irc.mozilla.org #planning for backchannel
- (the developer meeting takes place on Tuesdays)
Video/Teleconference Details – NEW
- 650-903-0800 or 650-215-1282 x92 Conf# 95312 (US/INTL)
- 1-800-707-2533 (pin 369) Conf# 95312 (US)
- Vidyo Room: Warp Core
- Vidyo Guest URL
Contents
- 1 Actions from Last Week
- 2 Schedule & Progress on Upcoming Releases
- 3 Feedback Summary
- 4 UX & User Research
- 5 Market Insights
- 6 Marketing, Press & Public Reaction
- 7 Questions, Comments, FYI
- 8 Actions this week
- Product team working on defining all the “basecamp” requirements.
- Basecamp is a subset of the k9o work.
- Planning to have something ready to communicate in the next few days.
- Work on larger k9o goals proceeding.
- Triage 2x a week but looking at moving times for better participation. Announcement will go out on the list.
- Merge day may move one day earlier than ship date to decouple necessary work and meet our Aurora/Beta target ship by the end of the week. See dev.planning thread. This will not affect any release dates.
- FF13 beta 4 will be released on Friday
Bug 754133 – Set background of standalone images (the image itself, not the whole page) to white
Firefox Mobile- Fennec Native 14 beta 1 is now live on Google Play
- We expect our next beta of Fennec Native (released as early as 5/17) to be multi-locale with the same 13 localizations as XUL Fennec previously
- Day 1 Adoption
- Installs: 785,103 (+3.45% from yesterday)
- Active installs: 209,076 (+14.32%)
- Total Beta ADIs: 39,224 (+66.34%)
- ADIs on 14.0: 15,345 (39% of total ADIs)
- Beta is out! \o/
- Please remember you can have only 1 native sync per android device, so no you cannot have Aurora & Beta syncing at the same time. Read Why Here
- Regular deployment happening.
Release (1.7 -> Firefox 12, 13)
- Released SDK 1.7 yesterday
- mixed bouquet of new features and important bug fixes
- css-only page-mods, l10n pluralization goodness
- configurable contentURL property for panel and widget!
- many bug fixes
Stabilization (1.8 -> Firefox 13, 14)
- on-track for release Tuesday June 26th
- first beta release either today or tomorrow?
Development (1.9 -> Firefox 14, 15)
- Firefox 15 / memory leak issues mitigated by latest khuey patch
- still planning to spin up a campaign to get older extensions re-packed, because its the right thing to do
- more plannig for this will happen at the Addons work week, June 4.
- started landing ‘Spring Cleaning’ items yesterday, aiming to land loader in Firefox
- Focused on Basecamp
- Other work still proceeding, but some of it is pushed back: Persona beta moved to July. Should have no impact on other products.
- Firefox Desktop 14 (Aurora) – landed
- Looking to back out this functionality – bug 750936
- Native Install
- Chromeless launch of apps
- Mozillian Marketplace Release is out! Thanks for the feedback so far!
- Please be mindful that this is an internal release and of alpha quality. Early access is limited to Mozillians and partners for testing, so please do not tweet, communicate or share screenshots, features or details about the Marketplace for now.
- Download Firefox Desktop Nightly (Fx15)
- Have a look at the wiki to learn more about the testing procedure.
- If you run into any hiccups, you can report them to apps-feedback@mozilla.com.
- Once you’re done, there’s also an online survey for you to fill out.
- Firefox Desktop 15 (Nightly)
- Working on fixing bugs – mostly fit and polish to fit Kilimanjaro requirements
- Firefox Mobile 15 (Nightly) – in progress
- Work started in Fx14 continues in 15.
- The team is shoring up the bugs to be in better shape for K9o implementation.
- Proposed marketplace bundling with Fennec is an open issue. See bug 738545
- Apps in the Cloud
- AITC client continues
- AITC Server – Completed
Firefox 12 issues summary:
We currently have a 5% increase in issues over Firefox 11. This seems to be a combination of issues including:
Firefox 12 is slow – Thread was originally about a gfx card issue, but I think it turned into a pile on thread https://support.mozilla.org/en-US/questions/926621
[1]
[2]
Foxit Reader causing Hotmail issues: Solution is to rollback to an older version of Foxit. This combined with the Adobe Reader issue is sticky.
[3]
https://support.mozilla.org/en-US/questions/926740
[4]
[5]
PDF Issues on Mac – Reader update https://support.mozilla.org/en-US/questions/775819
[6]
[7]
[8]
Firefox 12 consumes too much memory – https://support.mozilla.org/en-US/questions/927004
[9]
[10]
[11]
Copy/Paste Image – Fixed – https://bugzilla.mozilla.org/show_bug.cgi?id=749527
[12]
[13]
Firefox 12 incorrectly caching pages – Bug filed – https://bugzilla.mozilla.org/show_bug.cgi?id=748647
[14]
Incompatible Norton Toolbar: Need to update norton 360 and norton internet security suite.
[15]
[16]
FF12 praise summary, down 6% from FF11:
Mobile- Top three dissatisfiers (startup speed, flash support, and memory consumption) are now fixed in Firefox for Android Beta!!
- Google Play reviews summary, 330 reviews of Beta in first 24 hours; our rating based on those reviews is a solid 4.0 stars. 160 5-star reviews already!
- Themes: fast, wow, flash, memory, smooth, scrolling, sexy, this is now my default browser
- “This is now the fastest, smoothest browser in AndroidLand.”
- “I absolutely love FF Sync, uploaded to my phone in seconds.”
- “Great Improvement I thought firefox will never support flash. It is the only reason that I didn’t use as my default browser. At first look it’s amazing. Definitely deserve five stars. And finally thanks for dev team for your hard work.”
- “Beautiful This update is simply amazing! I am thoroughly impressed. Firefox just beat out dolphin for my go to mobile browsing.”
- [apps integration] awesomebar/apps integration concept sketching (Madhava)
- [android] Collating beta1 UX feedback etherpad here (Madhava)
- [android] Finishing tab menu designs for next version; finalizing designs for search suggest; widget; custom menu (Ian)
- [android] blog post on visual design (patryk)
- [metro] Incorporating feedback from last video sketch; preparing another one around interactions of basic browser usage (yuan)
- [B2G Firefox] (Larissa)
- v1 tabs interaction spec
- revising start page spec; starting on details of sharing and browser settings
- [sign-into-browser / ID integration] (Zhenshuo)
- [desktop] more ideas for preferences simplification (Zhenshuo)
- After initially requiring users to pay hundreds of dollars to receive security fixes for vulnerabilities in Flash Professional, Illustrator, and other products, Adobe has relented and will now provide them for free
- A fairly significant security issue was found in Safari’s handling of input in the URL bar; a fix is expected shortly
- Code recently landed that integrates all of a users’ Google Web History with their Chrome browsing history
- Google posted a web developer introduction, with video to its Web Intents implementation
- Tab syncing across devices has now landed in the Chrome stable channel.
- The default extensions for Chrome have been updated to include Google Docs
- Microsoft posted a detailed summary, with video of how social sharing of links and other content from IE10 will work on Windows 8.
- Senate Judiciary Committee staffers plan to take a look at allegations that Microsoft has made it difficult for competing Web browsers to run on a certain version of Windows, an aide told a political blog
- A leaked version of the upcoming Preview Release of IE10 shows an improved score on html5test.com
- There are rumours that Opera will be porting its Opera Mobile browser to the upcoming Windows Phone platform
- Opera Mini was also released for the Samsung Bada platform
- CSS Variables are coming to WebKit
- The tab-size property is now supported, and work has begun on the Media Capture API
Summary below, full update here and in your inbox.
- Updated Android version distribution numbers are available
- Google’s Motorola acquisition expected to be completed soon
- Android 5.0, code-named Jelly Bean, rumoured to be released in fall with multiple flagship devices
- Baidu to be releasing Yi-based devices
- 56.1% of smartphone shipments in Q1 featured Android
- Samsung confirmed as top mobile phone vendor by shipments in Q1
- Transition: Laura Forrest will be the new PMM for Desktop, with full transition by 6/18. Desktop is in excellent hands!
- In the meantime, Laura Mesa and Grace will continue leading through the next Desktop updates, including Aurora & Beta.
- Working through how this may/may not effect Beta and Aurora outbound.
- Otherwise, materials and blog posts on schedule for next update.
Press
Firefox for Android Preps for Prime Time
Mozilla updates Firefox for Android with Adobe Flash support
Firefox 14 beta for Android released
Hands on with Mozilla Firefox OS Boot to Gecko[25]
Questions, Comments, FYI- myk: I keep hearing people talk about this thing called “Basecamp”, and I’m pretending to be a volunteer contributor; where/when can I find out more information about it?
- Response: Basecamp details are being defined right now and the working team will send out an update on dev.planning by the end of next week. For any immediate questions, feel free to message me. (clee@mozilla.com)
- ally/jen to come back with an answer on how we feel about AitC as a blocker to the FF15 apps work
- matt/alex to figure out why our adobe blocklist didn’t eliminate the reader bustage
Ryan Merkley: Thanks Dave.
Today, the University of Waterloo announced they’ve hired Dave Wallace to take on the role of CIO. It’s a loss for the City of Toronto, where Dave has served as CIO for the past 5 years.
I worked with Dave at City Hall, where he distinguished himself with his willingness to try new things, and a desire to go above and beyond. He helped implement the city’s 24-hour 311 call-centre, and the Toronto open data project, which now lives at Toronto.ca/open. Each time he was presented with opportunities, I found him eager to explore and listen.
When I approached him about opening up the City of Toronto’s data, he was excited along with his staff. He listened intently as Mark Surman and I pitched the idea. He came to Open Data Camp, brought his staff, and spent the day talking and listening. He put resources behind the project. He built a team around it and we worked together to implement it. He sold it to his colleagues. Without him, it wouldn’t have happened.
Action in government happens when three things come together: political leadership, public permission, and bureaucratic will. Dave not only provided the internal drive, but he shared what he learned with other municipalities and created opportunities for them to collaborate, so things like licensing and terms of service didn’t have to be written from scratch each time a new city wanted to put their data online.
In a city where public servants are accused more often than lauded, Dave’s one of the good guys. I wish him well at UW (my alma mater) — they’re lucky to have him.
Dave, you’re leaving a wonderful legacy here in Toronto. From one open data geek to another, thank you.
Mitchell Baker: Upcoming: A Year in Europe
Mozilla is an increasingly global community. This is important to the success of our mission. If we hope to have a world of openness and opportunity for all we should be building centers of gravity in many different locales. Silicon Valley in California is still the center of a big chunk of the Internet industry, but Mozilla’s commitment to the Internet as a global public resource means we in particular focus on building leaders in many other places.
With this in mind my family and I have decided to get ourselves out of California for a bit. We’re planning to move to Barcelona next September for a year. Barcelona is not only in the heart of Europe, it’s much closer to the middle east and Africa, and it’s no further from the east coast of Latin America than California. (Although getting to Asia may be a longer trip.) I expect to be able to spend much more time with many more local Mozilla communities.
This is a change of geography, not of commitment to Mozilla. I expect to spend more time meeting Mozillians and more time focusing on project dynamics. I want to strengthen the ability for local leaders to become regional and global leaders in Mozilla.
I also expect to spend more time representing Mozilla to governments, policy-makers and other organizations interested in Mozilla and the Internet. By being located in Europe, we will be able to give more support to the critical issues being discussed in that region. I will also stay involved in our product efforts, as these are so key to have we achieve our mission. Perhaps I’ll find the time to do some of the writing that would be so helpful.
I’m not (yet???) a Spanish-speaker, so I will undoubtedly spend a bunch of time off-balance and trying to figure out how basic things work.
September will be here soon. We’re excited!
Upcoming: A Year in Europe
Mozilla is an increasingly global community. This is important to the success of our mission. If we hope to have a world of openness and opportunity for all we should be building centers of gravity in many different locales. Silicon Valley in California is still the center of a big chunk of the Internet industry, but Mozilla’s commitment to the Internet as a global public resource means we in particular focus on building leaders in many other places.
With this in mind my family and I have decided to get ourselves out of California for a bit. We’re planning to move to Barcelona next September for a year. Barcelona is not only in the heart of Europe, it’s much closer to the middle east and Africa, and it’s no further from the east coast of Latin America than California. (Although getting to Asia may be a longer trip.) I expect to be able to spend much more time with many more local Mozilla communities.
This is a change of geography, not of commitment to Mozilla. I expect to spend more time meeting Mozillians and more time focusing on project dynamics. I want to strengthen the ability for local leaders to become regional and global leaders in Mozilla.
I also expect to spend more time representing Mozilla to governments, policy-makers and other organizations interested in Mozilla and the Internet. By being located in Europe, we will be able to give more support to the critical issues being discussed in that region. I will also stay involved in our product efforts, as these are so key to have we achieve our mission. Perhaps I’ll find the time to do some of the writing that would be so helpful.
I’m not (yet???) a Spanish-speaker, so I will undoubtedly spend a bunch of time off-balance and trying to figure out how basic things work.
September will be here soon. We’re excited!
David Boswell: An invitiation to participate on every page
There is now a link in the footer of www.mozilla.org that invites people to contribute to that page.
We were originally thinking of this as a way to reach out to webdev volunteers, but it became clear that there are many others ways to contribute to a page including translating, designing, writing and testing.
Inviting people to participate on every page could be very powerful. Many people don’t know they can contribute to Mozilla so they wouldn’t think to look for our Get Involved page.
We’ll be able to use the Get Involved dashboard that the Metrics team recently created to see if this help us connect with more potential volunteers. If so, we could look at adding this invitation to participate on every page of every site in the Mozilla universe.
Benoit Girard: Dev Tip: Debugging optimized code without a clobber – Rebuilding a module without optimization
Sometimes you have an optimized build for whatever reason (say you’re doing a lot of profiling) but optimizations make non trivial debugging impossible. You don’t have an up to date build without optimization so you whine, start a non optimize build and start looking at bugzilla for 20 mins.
Frankenstein optimized/non optimized build to the rescue! Simply add:
CXXFLAGS += -O0 -g
to the Makefile for the module(s) you’re interested in debugging, for me it was gfx/layers/Makefile.in.
How does this work? Well optimizations are done at the object level and each object file are built to follow the ABI. As long as the ABI is followed, and it really really should, then you can expect this to work without any problems.
Disclaimer: This isn’t supported! If you have problems then do a clobber build.
Irina Sandu: Android and mobile browsing insights – Week 20
Every week I post an overview on what’s been happening in the mobile (browsing) world and is relevant to Mozilla.
- Updated Android version distribution numbers are available
- Google’s Motorola acquisition expected to be completed soon
- Android 5.0, code-named Jelly Bean, rumoured to be released in fall with multiple flagship devices
- Baidu to be releasing Yi-based devices
- 56.1% of smartphone shipments in Q1 featured Android
- Samsung confirmed as top mobile phone vendor by shipments in Q1
Updated Android version distribution numbers put Gingerbread at 64% of the market, with API level 9 at 0.5% and level 10 at 63.9%, Froyo (level 8) at 20.8% and Ice Cream Sandwich at almost 5%, with most of it on API level 15.
Google declared that it expects its acquisition of Motorola to be completed soon, before the first half of the year. After having passed regulatory approval in the US and the US, the company is now waiting from go-ahead from the authorities in China. The closing of the deal will likely raise more concerns over competition inside the Android ecosystem.
Details about the upcoming version of Android, version 5.0 code-named Jelly Bean, have emerged.The launch is rumoured to happen before Thanksgiving and to feature more OEMs that will produce the version’s flagship devices. It is to be expected that they will also be featured for sale in Google’s Play Store.
Baidu, China’s incumbent search engine, is set to release a series of new mobile devices based on its Yi mobile platform, a fork of Android, that was announced in September 2011. Dell is reported to be the company’s hardware partner for this venture in a country which recently surpassed the US as the fastest growing market for new Android & iOs activations.
The smartphone platform market was further dominated by Android in Q1 smartphone shipments, where Google’s ecosystem captured 56.1% of the market with 81 million units, up from 36.4% of the market and 36.3 million units in the same period of 2011. Apple’s platform was on the second place with 33 million and 22.9% of the market, also up from 16 million and 16.9% marketshare in Q1 of 2011. Other platforms with significant shares are Symbian with 8.6% of the market, followed by BB OS with 6.9%, Bada with 2.7% and Windows Mobile and Windows Phone 7 which together account for 1.9%.
Q1 results of phone and smartphone shipments are out, confirming Samsung as having taken the lead as the top mobile phone vendor with 86 million units and 20.7% of the market, up from 68 million and 16.1% of the market the previous year. On second place there is Nokia, with 83 million and 19.8% marketshare, which is on a downward trend from its Q1 2011 result of 107 million corresponding to a 25.1% marketshare. Apple takes 3rd place with 33 million and 7.9% of the market, up from 16 million and 3.9%. Further down the top there are ZTE, LG, Huawei, RIM, Motorola, Sony and HTC with significant shares of the market.
David Boswell: Grow Mozilla discussion this Thursday
“I’m a professional product manager and if you need help I would be delighted to join.” — from message posted on Get Involved page
Are you interested in helping people get involved with Mozilla, like this person who wants to help with product management? Then join us to discuss community building at Mozilla.
- Meeting time: Thursday, May 17 at 10:00 AM Pacific
- Topics: Localizing Get Involved page and increasing community building skills with Mozilla Brain Builders
- Video: David Boswell’s vidyo room (link for guest access is on agenda)
- Audio: 1-800-707-2533, pin: 369, conf: 9634#
- Room: Get to do Choppa in Mountain View 3rd floor
- Back channel: irc.mozilla.org#mozillians
Note that the video and audio information has changed from previous meetings.
If you have a question you’d like to ask the group, please feel free to edit the agenda on the wiki.
Lawrence Mandel: Sign in to Telemetry with Persona
I’m pleased to report some user visible progress from the performance and metrics work week. Sign in to the Telemetry dashboard now uses Persona (aka, BrowserID).
No special permission is required. The Telemetry dashboard is open to all. Don’t have a Persona account? No problem. Click the sign in button to be prompted to create a free account.
This change is now live. You can try it yourself by visiting the Telemetry dashboard at http://mzl.la/telemetrydash.
Tagged: mozilla, telemetry
Web FWD: Scout Sightings: Joseph Somogyi Reports
Berlin-based WebFWD Scout Joseph Somogyi is keeping busy. He will present WebFWD next week on May 25 at LinuxTag, the biggest Linux and Open Source exhibition in Europe, full of workshops, camps and many conference tracks running in parallel. Linux Tag, a 4-day government-sponsored event for every kind of Open Source, is expecting a host of exhibitors and attendees.
Joseph also found time recently to share a pointer to The Architecture of Open Source Applications, Vol I-II (2012), edited by Amy Brown and Greg Wilson, and available online in its entirety. Chapters are written by a collection of open source heroes, including one called Firefox Release Engineering, by Mozillians Chris AtLee, Lukas Blakk, John O’Duinn, and Armen Zambrano Gasparnian.
Joseph quotes:
“Architects look at thousands of buildings during their training, and study critiques of those buildings written by masters. In contrast, most software developers only ever get to know a handful of large programs well—usually programs they wrote themselves—and never study the great programs of history. As a result, they repeat one another’s mistakes rather than building on one another’s successes.
Our goal is to change that. In these two books, the authors of four dozen open source applications explain how their software is structured, and why. What are each program’s major components? How do they interact? And what did their builders learn during their development? In answering these questions, the contributors to these books provide unique insights into how they think.”
Other sightings: These closing links come from startup country, Silicon Valley, where a couple high profile acquisition plays triggered plenty of noise and some thoughtful postings earlier this spring:
- High Scalability - The Instagram Architecture Facebook Bought for a Cool Billion Dollars
- Learning from Slideshare’s Founders - NextWala
Tom Schuster: Short update of what the JS team is at
We actually wanted to enabled Incremental GC on Nightly, but again we had some fallout and it had to be backed out again. Bill thinks it should reland at the end of the week.
We are happy to welcome Benjamin Peterson, who is going to join us this summer as an intern working on SpiderMonkey’s ES6 support. Benjamin is an active python contributor. He has already started implementing rest parameters.
Till Schneidereit, (a fellow German, finally!) started picking up some GC related bugs, thank you and feel welcome.
In an effort to reduce the memory usage of average JavaScript applications (MemShrink \o/), we came to the conclusion that it is okay to throw away JIT code compiled by Jäger on every Garbage Collection run. Unfortunately this doesn’t work very well for animation heavy scripts like games, where recompiling would introduce long pauses. Brian fixed that.
Jason showed us how to use the new Debugger API to debug JavaScript code running in Firefox.
David Mandelin and me blogged about the SpiderMonkey API (JSAPI), and what needs to change, C++ yeah!
The DataView object landed, thanks to the work of Steve.
Luke just finished a patch that is going to speed up the handling of some function parameters/variables. Besides blocking more IonMonkey performance improvements, it already showed 10% better scores on the v8 early-boyer benchmark. (Bug 659577)
Jan has been working on chunked compilation which should help IonMonkey with very large scripts. But because this is a very broad change and the Ion team likes to focus on stabilizing, fixing crashes and test failures first, this is going to land after the initial release. Luckily these kind of large scripts are uncommon for normal JavaScript, but they are often found in Emscripten compiled code. JägerMonkey (+TI) which has chunked compilation is still going to help those scripts.
Edit: Republished because of some tumblr problems.
Les Orchard: Please Do Learn To Code
TL;DR: I think coding is an essential skill for modern humans surrounded by code and machines that run it. Please learn to code.
I disagree with Jeff Atwood on "Please don't learn to code":
The "everyone should learn to code" movement isn't just wrong because it falsely equates coding with essential life skills like reading, writing, and math. I wish. It is wrong in so many other ways.
In fact, I do regard coding as an essential modern skill. Yes, right alongside reading, writing, and 'rithmatic. At least this part of the post had me nodding:
I suppose I can support learning a tiny bit about programming just so you can recognize what code is, and when code might be an appropriate way to approach a problem you have. But I can also recognize plumbing problems when I see them without any particular training in the area.
Visible pipesLuckily, pipes are not as occult as code. If you go into a basement or open the door under a sink, you can see them and follow where they go. That's some training, albeit informal or self-directed.
I'm not sure how you'd get exposed to code in the same way: View Source in a browser used to be a decent start on the web, but that's less helpful lately. Code elsewhere has typically been hard to get at, Open Source notwithstanding.
Still, I'd bet there are people in the world for whom running water comes from magic and drains into magic. Where magic means: "I never thought about it, never needed to, and am sometimes vaguely afraid of it."
For that class of homeowner, the kitchen floods when the sink springs a leak, until an expert arrives. It's not the end of the world, and plenty of people get by just fine like that. But, I certainly wouldn't agree that "Please don't learn about pipes" is good advice in general.
Learning by doingAdmittedly, "learn to plumb" isn't the same as "learn about pipes". But, is there a difference between "learn to code" and "learn about code"? I don't think so. Like writing, code doesn't seem like a thing that's easy to learn about without doing it.
When I write "coder", I generally mean this: Someone who is capable of encoding his or her intent and decision process into a form that can drive a CPU to perform tasks on his or her behalf.
That's a very broad definition, but it implies a lot. First, you have to realize that you can make a CPU can do things on your behalf--it's okay, you won't break it. It's a tool made by humans and you as a human can understand it. Then, you need a notion of algorithmic thinking, in order to formulate your intent and reasoning in a form that a CPU can execute. These are not natural or intuitive things.
I agree with that it's good to "recognize what code is, and when code might be an appropriate way to approach a problem". But, if you've never made a CPU do your bidding, it's easy to see it as a mysterious black box with a will of its own--possibly malicious or at least capricious.
And, if you've never worked to force your thoughts into to the confoundingly literal and common senseless constraints of computer programming, it's hard to even imagine how code works. If you can't imagine how it works, how do you work it into your mental toolkit for getting things done.
Learn to code, and a lot of other things get dragged into your head along the way.
Who needs all these coders?And then, there were these bits from "Please Don't Learn to Code":
It assumes that adding naive, novice, not-even-sure-they-like-this-whole-programming-thing coders to the workforce is a net positive for the world.
It implies that there's a thin, easily permeable membrane between learning to program and getting paid to program professionally.
This is looking at work from the wrong angle. It isn't about getting paid to program, so much as coding to be good at your job.
I'm not talking about Java-heads who live all day in Eclipse. I'm talking about the Excel-head who used to rock VBA macros, who maybe just started playing with Google Apps Script. I have no idea how popular Google Apps Script might or might not be, but I've seen some crazy amazing things done in VBA by sales and account reps who'd punch you in the nose if you called them geeks.
As far as I can tell, the future of work is heading toward more work with greater volumes of information and data. Should professional programers be the only people in an organization who know how to apply computational power to solve problems? Maybe the vendors will sweep in, clean up all the cybercrud, and get the real work done for us.
Programming should not be a priesthoodConsider writing: there's a lot to learn and it used to be a thing done only by a few scribes. But, people today get a lot of mileage out of just sticky notes and email. Sure, improving your grammar and learning how to structure an essay can help in many, many ways. But, you don't need to be a professional writer to be a professional who uses written language.
The same can apply for coding. The problem, though, is that the sticky-notes-and-email level of competence barely exists or is near impossible to access. So, not only do I think we need more coders--we also need more tools that support coding and make coding more accessible. I think we should support professionals who use code.
More than that, I think we should encourage and support humans who code. I really do consider coding next to reading, writing, and math. I don't think we can all rely on someone else to write the perfect app for the work we'll need to do in the future. I expect the successful people will be those who can apply Taco Bell programming to reams of data and find answers. We'll need to ride bicycles, not tricycles.
Gervase Markham: Mobile Market Reports
“thinkinsights” and Google have released some fascinating data about smartphone usage, gathered from detailed consumer surveys.
All the presentation reports (right hand column) have roughly the same format. Why not download the report for your country, and the one for Brazil (the launch country for B2G) and see how different things are there compared to where you live?
Compared to the UK, in Brazil:
- Far fewer people have smartphones (14% vs 51%)
- Smartphones are used less often (40% vs 59% daily)
- They are used on-the-go less often (64% vs 86%) – poorer network coverage?
- Social networking is proportionately more popular than email
- Smartphone gaming is significantly less popular (39% vs 62%)
- They use fewer apps, even free ones (14 vs 23 installed)
- Almost all smartphone users are urban
- Cohabitation is significantly less common (20% vs 11%)
Thoughts: reading between the lines, network coverage is poorer and data-on-the-go is harder to find. We need to make sure B2G phones and apps are solid in absence of a good network connection. Also, the phone will be the only computing device for many users.
EC houdt Microsoft in de gaten in nieuwe browseroorlog - Webwereld
Techzine
EC houdt Microsoft in de gaten in nieuwe browseroorlog
Webwereld
De Europese Commissie zegt de discussie in de Verenigde Staten te hebben opgemerkt, waar Mozilla heeft geklaagd dat Microsoft zijn eigen browser voortrekt bij het gebruik op de nieuwe tabletversie van zijn besturingssysteem Windows 8.
Onderzoek naar browsers in Windows RTTelegraaf.nl
EU blijft Microsoft in de gaten houdenHet Laatste Nieuws
VS onderzoekt concurrentievervalsing in Windows 8NU.nl
Security.nl -Automatisering Gids -Techzine
alle 14 nieuwsartikelen »
Wim Benes: Mozilla bij Open Overheid Congres
Open standaarden, Open source software en Open data: drie
losse onderwerpen die in de praktijk enorm verweven zijn, nu in één congres! Kom op 31 mei naar het Open Overheid Congres, georganiseerd door ECP, Forum Standaardisatie, ICTU, VKA, TNO en OSSLO.
Laat u bijpraten over de stand van zaken en ontmoet uitvoerders van beleid, beleidsmakers, IT-verantwoordelijken van overheidsorganisaties en marktpartijen. Vernieuwende denkrichtingen en praktijkvoorbeelden zullen u inspireren!
Open Overheid congres
Donderdag 31 mei 2012
12.00 – 17.00 uur (incl. lunch)
Beatrix Theater te Utrecht
Nieuwe bètaversie Mozilla Firefox: nieuwe interface, Flash-ondersteuning - Androidworld
Androidics
Nieuwe bètaversie Mozilla Firefox: nieuwe interface, Flash-ondersteuning
Androidworld
Mozilla vraagt, aangezien het om een testversie gaat, om eventuele problemen te melden via deze pagina. Momenteel is de nieuwe bèta alleen verkrijgbaar in het Engels, maar vertalingen zouden op korte termijn moeten volgen.
Firefox voor Android krijgt nieuwe gui (gallery)Webwereld
alle 3 nieuwsartikelen »
Gervase Markham: Welcome To Life
An under-3-minute video short story by Tom Scott. “A science fiction story about what you see when you die. Or: the Singularity, ruined by lawyers.”
Nicholas Nethercote: MemShrink progress, week 47–48
The main news in the past two weeks has been about Kyle Huey’s patch that prevents most chrome-to-content leaks, which are the most common kind of add-on leak. Testing showed it worked beautifully, but caused a knock-on leak in add-ons built with old versions (1.3 and earlier) of the Add-on SDK. (This received a lot of attention.) Kyle then made a slight tweak that fixed that knock-on leak. So we’re currently still on track for Firefox 15 being “the one that fixes add-on leaks”.
For completeness, here are the add-ons that we know were temporarily affected by that knock-on leak: Wallflower, Visual Hashing, Translate This!, Easy YouTube Video Downloader. They (and probably quite a few others) are all working fine again now.
Here’s a quote from an email that one user sent to Kyle this week.
Firefox was leaking about 1.5GB per hour for me. It started with Firefox 3. I tracked it down to Ghostery and NoScript, but even without those addons it leaked about 500MB per hour of browsing.
GC and CC times got up into the 10 second range. Ugly. Really really ugly! And this is on top of the line massively overclocked hardware, too. I had to install a new addon to add a restart button to Firefox, because Firefox froze solid after hitting 2GB of memory usage. I also patched it after every update to allow up to 4GB, buying a little more time…
Then your patch comes along and solves it all… you are awesome man – totally awesome!
Another user — one who uses the leaky Autopager add-on — commented on Kyle’s blog.
Certainly, before this fix I would find that Firefox often became sluggish (input lag, slow paint operations, less than silky smooth scroll animations) as the memory usage built up. It’s hard to say how much various factors contributed to the whole, but GC pauses did undoubtedly cause the scroll animation stuttering.
Restarting was the cure. I haven’t noticed the same symptoms since, and while I haven’t had enough chance to make a conclusive judgement, the signs certainly seem to be good.
I have a full tab strip more often than not, and Fx set to load tabs from last time. This is offset by the wonderful and elegantly simple tabs on demand feature. I’m running a 2 year old laptop with 4GB ram.
And while we’re on the topic, here’s a comment from my blog.
Opened my firefox today, 30+ Tabs (only counting the ones in the active group, the others aren’t loaded), using just little more than 330 MB of RAM. A year ago, with Firefox 4, this would have been impossible. Keep it going!
Good times.
The following add-ons had zombie compartments fixed: Youtube Ratings Preview, SPDY Indicator It’s likely these leaks would have been fixed by Kyle’s change, but since Firefox 15 won’t be released until August 28, it’s good that they’ve been fixed now. (Indeed, the AMO review policy still requires that add-ons not cause zombie compartments with the current release of Firefox; that policy may be revisited once Firefox 15 is released.)
Compartment-per-globalThe other big news is that compartment-per-global (CPG) landed, thanks to the work of various people, especially Luke Wagner and Bobby Holley. Bobby explained what this means and explored some of the consequences.
CPG will allow lots of things within Firefox to become simpler and faster. The main disadvantage is, unfortunately, increased memory consumption, as can be seen on areweslimyet.com. (Thanks to Luke, this increase was less than it could have been.) This is mostly due to more fragmentation in the JavaScript heap — we now have many more compartments, and each 4KB heap arena cannot be shared between compartments, so there are many more partially empty arenas present.
You might think this would make me bang my head against the wall in frustration, but it doesn’t. That’s because even if I ignore the many non-MemShrink-related benefits of CPG, there are two big MemShrink-related ones.
First, CPG will enable per-tab memory reporting, something that users have been requesting for years.
Second, CPG will lead to much more detail in about:memory and about:compartments. For example, Nils Maier has written a patch that makes it obvious all the JavaScript modules that have been loaded. Another example: Justin Dolske found that plusone.google.com was doing something silly (constantly creating new iframes?) that caused huge numbers of compartments to be created; without CPG I think all those globals would have been lumped into a single compartment and the problem would have been much less obvious. More information in about:memory will lead to more diagnosis of existing problems — particularly leaks of various kinds — in both Firefox and websites.
Memory ReportingKevin Locke tweaked the JS memory reporters so that more compartments are distinguished, instead of being lumped together. This was his first Mozilla patch — well done, Kevin!
Nathan Froyd improved the coverage of the layout memory reporters. This significantly reduces “heap-unclassified” for huge pages like the single-page version of the HTML5 spec.
Bug countsHere are the current bug counts.
- P1: 22 (-2/+3)
- P2: 83 (-6/+4)
- P3: 105 (-5/+6)
- Unprioritized: 3 (-1/+3)
Mostly bouncing around at the moment.
