mozilla

### Honza Bambas: Building mozilla code directly from Visual Studio IDE

Mozilla.org - vr, 29/11/2013 - 01:17

Yes, it’s possible!  With a single key press you can build and have a nice list of errors in the Error List window, clickable to get to the bad source code location easily.  It was a fight, but here it is.  Tested with Visual Studio Express 2013 for Windows Desktop, but I believe this all can be adapted to any version of the IDE.

• Create a shell script, you will (have to) use it every time to start Visual Studio from mozilla-build’s bash prompt:

export MOZ__INCLUDE=$INCLUDE export MOZ__LIB=$LIB
export MOZ__LIBPATH=$LIBPATH export MOZ__PATH=$PATH
export MOZ__VSINSTALLDIR=$VSINSTALLDIR # This is for standard installation of Visual Studio 2013 Desktop, alter the paths to your desired/installed IDE version cd "/c/Program Files (x86)/Microsoft Visual Studio 12.0/Common7/IDE/" ./WDExpress.exe & • Create a solution ‘mozilla-central’ located at the parent directory where your mozilla-central repository clone resides. Say you have a structure like C:\Mozilla\mozilla-central, which is the root source folder where you find .hg, configure.in and all the modules’ sub-dirs. Then C:\Mozilla\ is the parent directory. • In that solution, create a Makefile project ‘mozilla-central’, again located at the parent directory. It will, a bit unexpectedly, be created where you probably want it – in C:\Mozilla\mozilla-central. • Let the Build Command Line for this project be (use the multi-line editor to copy & paste: combo-like arrow on the right, then the <Edit…> command): call "$(MOZ__VSINSTALLDIR)\VC\bin\vcvars32.bat"
set INCLUDE=$(MOZ__INCLUDE) set LIB=$(MOZ__LIB)
set LIBPATH=$(MOZ__LIBPATH) set PATH=$(MOZ__PATH)
set MOZCONFIG=c:\optional\path\to\your\custom\mozconfig
cd $(SolutionDir) python mach --log-no-times build binaries Now when you make a modification to a C/C++ file just build the ‘mozilla-central’ project to run the great build binaries mach feature and quickly build the changes right from the IDE. Compilation and link errors as well as warnings will be nicely caught in the Error List. BE AWARE: There is one problem – when there is a typo/mistake in an exported header file, it’s opened as a new file in the IDE from _obj/dist/include location. When you miss that and modify that file it will overwrite on next build! (I’ll ask Bas Schouten as Chris Pearce suggests if there is some solution.) With these scripts you can use the Visual Studio 2013 IDE but build with any other version of VC++ of your choice. It’s independent, just run the start-up script from different VS configuration mozilla-build prompt. I personally also create projects for modules (like /netwerk, /docshell, /dom) I often use. Just create a Makefile project located at the source root directory with name of the module directory. The project file will then be located in the module – I know, not really what one would expect. Switch Solution Explorer for that project to show all files, include them all in the project, and you are done. Few other tweaks: • Assuming you properly use an object dir, change the Output Directory to point e.g. to$(SolutionDir)\<your obj dir>\$(Configuration)\. Similarly, set the Intermediate Directory to <your obj dir>\$(Configuration)\. The logging and other crap won’t then be created in your source repository.
^.*\.vcproj.*
^.*\.vcxproj.*
.sln$.suo$
.ncb$.sdf$
.opensdf$to your custom hg ingnore file to prevent the Visual Studio project and solution files interfere with Mercurial. Same suggested for git, if you prefer it. Note: you cannot use this for a clobbered build because of an undisclosed Python Windows-specific bug. See here why. Do clobbered builds from a console, or you may experiment with clobber + configure from a console and then build from the IDE. Categorieën: Mozilla-nl planet ### Tantek Çelik: Homebrew Website Club Newsletter Volume 1 Issue 1 Mozilla.org - vr, 29/11/2013 - 00:00 Edited by Tantek Çelik, 2013-11-28 Are you building your own website? Indie reader? Personal publishing web app? Or some other digital magic-cloud proxy? If so, you might like to come to a gathering of people with likeminded interests. Exchange information, swap ideas, talk shop, help work on a project, whatever... This announcement, accompanying blog post, and event note brought nine of us together on short notice in the 7th floor main meeting area at Mozilla's San Francisco office. After brief introductions we went around the room in a "broadcast" phase. Everyone introduced themselves and what personal website successes and challenges they were experiencing. All already had a personal website of some sort, yet also expressed a yearning for something more. Opinions and passion were generally dominated by user-centered perspectives, about giving users (especially themselves) control over their own content/narrative, and focusing on user experience first. Four in the group actively post content on their own site (typically in a "blog" format), and three more on secondary domains, Blogspot, or Tumblr. Two in the group already had personal website tweeting up and running using the POSSE method (Publish on your Own Site, Syndicate Elsewhere). And one had an ownCloud setup working with an SSL certificate. We got into a short dispute over whether to focus on public or private content first until it was pointed out that public first is simpler and can inform private content design. There was a PESOS vs. POSSE debate, especially for quantified self / fitness data. Many in the group conveyed a general lamenting of the lack of support for Activity Streams in services and devices, until one participant noted he'd built a proxy that turns interactions from Facebook, Twitter, G+ (e.g. comments, likes) into Activity Streams. Frustrations were shared about services that show promise yet have odd awkwardnesses like Path and Mint. On the open source side, concerns were raised about monoculture and especially the open source community default culture of assuming one codebase to rule them all. There was much praise for the ease of use, beauty, and customization of Tumblr, especially as a good bar to compare efforts to build personal websites and provide user interfaces for future indieweb onboarding experienes. Despite their beauty or convenience, there was a sense that Tumblr, Blogger, and other content hosting silos will all rot. We split up into small groups as part of the "peer-to-peer" part of the meeting. Kevin Marks did an excellent job of live tweeting a lot of the conversation and posted a summary on his site while at the meeting! At 20:00 we closed the meeting and announced that the next meeting would be in two weeks: NEXT MEETING WEDNESDAY, 2013-12-04, 18:30 at Mozilla's First Floor Common Area, Embarcadero & Harrison, San Francisco, CA. Are you building your own website? Indie reader? Personal publishing web app? Or some other digital magic-cloud proxy? If so, come on by and join a gathering of people with likeminded interests. Bring your friends that want to start a personal web site. Exchange information, swap ideas, talk shop, help work on a project, whatever... This newsletter is placed into the public domain with a CC0 dedication. (with apologies to Homebrew Computer Club Newsletter number one ) Categorieën: Mozilla-nl planet ### William Lachance: mozregression now supports inbound builds Mozilla.org - do, 28/11/2013 - 19:14 Just wanted to send out a quick note that I recently added inbound support to mozregression for desktop builds of Firefox on Windows, Mac, and Linux. For the uninitiated, mozregression is an automated tool that lets you bisect through builds of Firefox to find out when a problem was introduced. You give it the last known good date, the last known bad date and off it will go, automatically pulling down builds to test. After each iteration, it will ask you whether this build was good or bad, update the regression range accordingly, and then the cycle repeats until there are no more intermediate builds. Previously, it would only use nightlies which meant a one day granularity — this meant pretty wide regression ranges, made wider in the last year by the fact that so much more is now going into the tree over the course of the day. However, with inbound support (using the new inbound archive) we now have the potential to get a much tighter range, which should be super helpful for developers. Best of all, mozregression doesn’t require any particularly advanced skills to use which means everyone in the Mozilla community can help out. For anyone interested, there’s quite a bit of scope to improve mozregression to make it do more things (FirefoxOS support, easier installation…). Feel free to check out the repository, the issues list (I just added an easy one which would make a great first bug) and ask questions on irc.mozilla.org#ateam! Categorieën: Mozilla-nl planet ### Frédéric Harper: Firefox OS loves at the Athens App Days Mozilla.org - do, 28/11/2013 - 18:30 Click to see full size Yesterday I was invited to help support the Athens App Days. I did the first technical talk of the day, and my goal was to excited developers about the platform, and to show them all the possibilities they have for building their application. I was quite impressed by the dedication of developers during all the hackathon: they were hard at work to get a chance to win one of the amazing prizes we had! Firefox OS – The platform you deserve – Athens App Days – 2013-11-27 from Frédéric Harper As usual, there is also a recording of my presentation. I hope you enjoyed the presentation, and let me know if you still need help with the development of your Firefox OS app: I can’t wait to see them in the marketplace! It was a real pleasure to be part of this event, and of course, to visit Athens for the first time. -- Firefox OS loves at the Athens App Days is a post on Out of Comfort Zone from Frédéric Harper Related posts: 1. Firefox OS, the platform you deserve – Budapest, Hungary Yesterday, we were doing a Firefox OS workshop in Budapest,... 2. Firefox OS in Guadalajara, Mexico I usually do a blog post after my presentations, and... 3. Firefox OS tools & Web APIs in Krakow Today I did two presentations at the Firefox Krakow workshop.... Categorieën: Mozilla-nl planet ### Aki Sasaki: LWR (job scheduling) part ii: a high level overview Mozilla.org - do, 28/11/2013 - 03:43 compute farm I think of all the ideas we've brainstormed, the one I'm most drawn to is the idea that our automation infrastructure shouldn't just be a build farm feeding into a test farm. It should be a compute farm, capable of running a superset of tasks including, but not restricted to, builds and tests. Once we made that leap, it wasn't too hard to imagine the compute farm running its own maintenance tasks, or doing its own dependency scheduling. Or running any scriptable task we need it to. This perspective also guides the schematics; generic scheduling, generic job running. This job only happens to be a Firefox desktop build, a Firefox mobile l10n repack, or a Firefox OS emulator test. This graph only happens to be the set of builds and tests that we want to spawn per-checkin. But it's not limited to that. dependency graph (cf.) Currently, when we detect a new checkin, we kick off new builds. When they successfully upload, they create new dependent jobs (tests), in a cascading waterfall scheduling method. This works, but is hard to predict, and it doesn't lend itself to backfilling of unscheduled jobs, or knowing when the entire set of builds and tests have finished. Instead, if we create a graph of all builds and tests at the beginning, with dependencies marked, we get these nice properties: • Scheduling changes can be made, debugged, and verified without actually needing to hook it up into a full system; the changes will be visible in the new graph. • It becomes much easier to answer the question of what we expect to run, when, and where. • If we initially mark certain jobs in the graph as inactive, we can backfill those jobs very easily, by later marking them as active. • We are able to create jobs that run at the end of full sets of builds and tests, to run analyses or cleanup tasks. Or "smoketest" jobs that run before any other tests are run, to make sure what we're testing is worth testing further. Or "breakpoint" jobs that pause the graph before proceeding, until someone or something marks that job as finished. • If the graph is viewable and editable, it becomes much easier to toggle specific jobs on or off, or requeue a job with or without changes. Perhaps in a web app. web app The dependency graph could potentially be edited, either before it's submitted, or as runtime changes to pending or re-queued jobs. Given a user-friendly web app that allows you to visualize the graph, and drill down into each job to modify it, we can make scheduling even more flexible. • TryChooser could go from a checkin-comment-based set of flags to a something viewable and editable before you submit the graph. Per-job toggles, certainly (just mochitest-3 on windows64 debug, please, but mochitest-2 through 4 on the other platforms). • If the repository + revision were settable fields in the web app, we could potentially get rid of the multi-headed Try repository altogether (point to a user repo and revision, and build from there). • Some project branches might not need per-checkin or nightly jobs at all, given a convenient way to trigger builds and tests against any revision at will. • Given the ability to specify where the job logic comes from (e.g., mozharness repo and revision), people working on the automation itself can test their changes before rolling them out, especially if there are ways to send the output of jobs (job status, artifact uploads, etc.) to an alternate location. This vastly reduces the need for a completely separate "staging" area that quickly falls out of date. Faster iteration on automation, faster turnaround. community job status One feature we lost with the Tinderbox EOL was the ability for any community member to contribute job status. We'd like to get it back. It's useful for people to be able to set up their own processes and have them show up in TBPL, or other status queries and dashboards. Given the scale we're targeting, it's not immediately clear that a community member's machine(s) would be able to make a dent in the pool. However, other configurations not supported by the compute farm would potentially have immense value: alternate toolchains. Alternate OSes. Alternate hardware, especially since the bulk of the compute farm will be virtual. Run your own build or test (or other job) and send your status to the appropriate API. As for LWR dependency graphs potentially triggering community-run machines: if we had jobs that are useful in aggregate, like a SETI at home communal type job, or intermittent test runner/crasher type jobs, those could be candidates. Or if we wanted to be able to trigger a community alternate-configuration job from the web app. Either a pull-not-push model, or a messaging model where community members can set up listeners, could work here. Since we're talking massive scale, if the jobs in question are runnable on the compute farm, perhaps the best route would be contributing scripts to run. Releng-as-a-Service. Releng-as-a-Service Release Engineering is a bottleneck. I think Ted once said that everyone needs something from RelEng; that's quite possibly true. We've been trying to reverse this trend by empowering others to write or modify their own mozharness scripts: the A-team, :sfink, :gaye, :graydon have all been doing so. More bandwidth. Less bottleneck. We've already established that compute load on a small subset of servers doesn't work as well as moving it to the massively scalable compute farm. This video on leadership says the same thing, in terms of people: empowering the team makes for more brain power than bottlenecking the decisions and logic on one person. Similarly, empowering other teams to update their automation at their own pace will scale much better than funneling all of those tasks into a single team. We could potentially move towards a BYOS (bring your own script) model, since other teams know their workflow, their builds, their tests, their processes better than RelEng ever could. :catlee's been using the term Releng-as-a-Service for a while now. I think it would scale. I would want to allow for any arbitrary script to run on our compute farm (within the realms of operational-, security-, and fiscal- sanity, of course). Comparing talos performance numbers looking for regressions? Parsing logs for metrics? Trying to find patterns in support feedback? Have a whole new class of thing to automate? Run it on the compute farm. We'll help you get started. But first, we have to make it less expensive and complex to schedule arbitrary jobs. This is largely what we talked about, on a high level, both during our team week and over the years. A lot of this seems very blue sky. But we're so much closer to this as a reality than we were when I was first brainstorming about replacing buildbot, 4-5 years ago. We need to work towards this, in phases, while also keeping on top of the rest of our tasks. In part 1, I covered where we are currently, and what needs to change to scale up. In part 3, I'm going to go into some hand-wavy LWR specifics, including what we can roll out in phase 1. In part 4, I'm going to drill down into the dependency graph. Then I'm going to start writing some code. comments Categorieën: Mozilla-nl planet ### Soledad Penades: A few drawings from CascadiaJS 2013 Mozilla.org - do, 28/11/2013 - 00:39 I wasn’t in condition to draw anything the first day of the conference, with being jetlagged and speaking too. But on the second day I was able to sketch five of the speakers for the day before I got exhausted again (that probably coinciding with the fact that my body clock was probably starting to shout at me that it was late in the night –in the London night, 8 timezones away– and it wanted me to SLEEP). You’ll notice that the quality of the drawings quickly degenerates, and I’m quite sorry because the last two speakers I drew were amazing and I don’t feel like I rendered them as accurately as I’d wanted to. Maybe next time, friends, maybe next time! :-) So there we go: Charles Bihis “When dealing with money, use integers” (video) David Bruant “In the ideal world the specs come first but in the real world they come last” (video) Matthew Bergman “Why should you trust me to store your data without reading it?” (video) C J Silverio “Time passed, and Moore’s Law did its thing… as it always does!” (video) Raquel Vélez (and her Batbot) “After working with robots for a while you start giving them names and having conversations…” (video) Categorieën: Mozilla-nl planet ### Christian Heilmann: Help me write a Developer Evangelism/Advocacy guide Mozilla.org - wo, 27/11/2013 - 23:45 A few years ago now, I spent two afternoons to write down all I knew back then about Developer Evangelism/Advocacy and created the Developer Evangelism Handbook. It has been a massive success in terms of readers and what I hear from people has helped a lot of them find a new role in their current company or a new job in others. I know for a fact that the handbook is used in a few companies as training material and I am very happy about that. I also got thanked by a lot of people not in a role like this learning something from the handbook. This made me even happier. With the role of developer evangelist/advocat being rampant now and not a fringe part of what we do in IT I think it is time to give the handbook some love and brush it up to a larger “Guide to Developer Evangelism/Advocacy” by re-writing parts of it and adding new, more interactive features. For this, I am considering starting a Kickstarter project as I will have to do that in my free-time and I see people making money with things they learned. Ads on the page do not cut it – at all (a common issue when you write content for people who use ad-blockers). That’s why I want to sound the waters now to see what you’d want this guide to be like to make it worth you supporting me. In order to learn, I put together a small survey about the Guide and I’d very much appreciate literally 5 minutes of your time to fill it out. No, you can’t win an iPad or a holiday in the Carribean, this is a legit survey. Loading… Let’s get this started, I’d love to hear what you think. Got comments? Please tell me on Google+ or Facebook or Twitter. Categorieën: Mozilla-nl planet ### Niko Matsakis: Thoughts on DST, Part 3 Mozilla.org - wo, 27/11/2013 - 21:06 After posting part 2 of my DST series, I realized that I had focusing too much on the pure “type system” aspect and ignoring some of the more…mundane semantics, and in particular the impact of monomorphization. I realize now that – without some further changes – we would not be able to compile and execute the second proposal (which I will dub statically sized typed (SST) from here on out). Let me first explain the problem and then show how my first thoughts on how it might be addressed. This is part 3 of a series: The problem The problem with the SST solution becomes apparent when you think about how you would compile a dereference *rc of a value rc that has type exists N. RC<[int, ..N]> (written long-hand). Typing this dereference is relatively straightforward, but when you think about the actual code that we generate, things get more complicated. In particular, imagine the Deref impl I showed before: impl<T> Deref<T> for RC<T> { fn deref<'a>(&'a self) -> &'a T { &*self.data } } The problem here is that the way monomorphization currently works, there will be a different impl generated for RC<[int, ..2]> and RC<[int, ..3]> and RC<[int, ..4]> and so on. So if we actually try to generate code, we’ll need to know which of those versions of deref we ought to call. But all we know that we have a RC<[int, ..N]> for some unknown N, which is not enough information. What’s frustrating of course is that it doesn’t actually matter which version we call – they all generate precisely the same code, and in fact they would generate the same code regardless of the type T. In some cases, as an optimization, LLVM or the backend might even collapse these functions into one, since the code is identical, but we have no way at present to guarantee that it would do so or to ensure that the generated code is identical. A solution One possible solution for this would be to permit users to mark type parameters as erased. If a type parameter T is marked erased, the compiler would enforce distinctions that guarantee that the generated code will be the same no matter what type T is bound to. This in turn means the code generator can guarantee that there will only be a single copy of any function parameterized over T (presuming of course that the function is not parameterized over other, non-erased type parameters). If we apply this notion, then we might rewrite our Deref implementation for RC as follows: impl<erased T> Deref<T> for RC<T> { fn deref<'a>(&'a self) -> &'a T { &*self.data } } It would be illegal to perform the following actions on an erased parameter T: • Drop a value of type T – that would require that we know what type T is so we can call the appropriate destructor. • Assign to an lvalue of type T – that would require dropping the previous value • Invoke methods on values of type T – in other words, erased parameters can have no bounds. • Take an argument of type T or have a local variable of type T – that would require knowing how much space to allocate on the stack • Probably a few other things. But maybe that erases too much…? For the most part those restrictions are ok, but one in particular kind of sticks in my craw: how can we handle drops? For example, imagine we have a a value like RC<[~int]>. If this gets dropped, then we’ll need to recursively free all of the ~int values that are contained in the vector. Presumably this is handled by having RC<T> invoking the appropriate “drop glue” (Rust-ese for destructor) for its type T – but if T is erased, we can’t know which drop glue to run. And if T is not erased, then when RC<[~int]> is dropped, we won’t know whether to run the destructor for RC<[~int, ..5]> or RC<[~int, ..6]> etc. And – of course – it’s wildly wasteful to have distinct destructors for each possible length of an array. Erased is the new unsized? This erased annotation should of course remind you of the unsized annotation in DST. The two are very similar: they guarantee that the compiler can generate code even in ignorance of the precise characteristics of the type in question. The difference is that, with unsized, the compile was still generating code specific to each distinct instantiation of the parameter T, it’s just that one valid instantiation would be an unsized type [U] (that is, exists N. [U, ..N]). The compiler knew it could always find the length for any instance of [U] and thus could generate drop glue and so on. So perhaps the solution is not to have erased, which says “code generation knows nothing about T, but rather some sort of partial erasure (similar to the way that we erase lifetimes from types at code generation, and thus can’t the code generator can’t distinguish the lifetimes of two borrowed pointers). Conclusion This naturally throws a wrench in the works. I still lean towards the SST approach, but we’ll have to find the correct variation on erased that preserves enough type info to run destructors but not so much as to require distinct copies of the same function for every distinct vector length. And it seems clear that we don’t get SST “for free” with no annotation burden at all on smart pointer implementors. As a positive, having a smarter story about type erasure will help cut down on code duplication caused by monomorphization. UPDATE: I realize what I’m writing here isn’t enough. To actually drop a value of existential type, we’ll need to make use of the dynamic info – i.e., the length of the vector, or the vtable for the object. So it’s not enough to say that the type parameter is erased during drop – or rather drop can’t possibly work with the type parameter being erased. However, what is somewhat helpful is that user-defined drops are always a “shallow” drop. In other words, it’s the compiler’s job (typically) to drop the fields of an object. And the compiler knows the length of the array etc. In any case, I thnk with some effort, we can make this work, but it’s not as simple as erasing type parameters – we have to be able to tweak the drop protocol, or perhaps convert “partially erased” type parameters into a dynamic value (that would be the length, vtable, or just () for non-existential types) that can be used to permit calls to drop and so on. Categorieën: Mozilla-nl planet ### Peter Bengtsson: Welcome to the world, Wish List Granted Mozilla.org - wo, 27/11/2013 - 19:10 I built something. It's called Wish List Granted. It's a mash-up using Amazon.com's Wish List functionality. What you do is you hook up your Amazon wish list onto wishlistgranted.com and pick one item. Then you share that page with friends and familiy and they can then contribute a small amount each. When the full amount is reached, Wish List Granted will purchase the item and send it to you. The Rules page has more details if you're interested. The problem it tries to solve is that you have friends would want something and even if it's a good friend you might be hesitant to spend$50 on a gift to them. I'm sure you can afford it but if you have many friends it gets unpractical. However, spending \$5 is another matter. Hopefully Wish List Granted solves that problem.

Wish List Granted started as one of those insomnia late-night project. I first wrote a scraper using pyQuery then a couple of Django models and views and then tied it up by integrating Balanced Payments. It was actually working on the first night. Flawed but working start to finish.

When it all started, I used Persona to require people to authenticate to set up a Wish List. After some thought I decided to ditch that and use "email authentication" meaning they have to enter an email address and click a secure link I send to them.

One thing I'm very proud of about Wish List Granted is that it does NOT store any passwords, any credit cards or any personal shipping addresses. Despite being so totally void of personal data I thought it'd look nicer if the whole site is on HTTPS.

Categorieën: Mozilla-nl planet

### Tantek Çelik: How To Lose Your Data In Apple iOS Notes In Five Easy Steps

Mozilla.org - wo, 27/11/2013 - 18:38

Or, why is it so hard to get syncing of simple text notes right?

1. Create a note in Notes on your iOS device (e.g. iPod) with some text like "Note to self: do not save anything valuable in Notes".
2. Connect it to your Mac and (Sync) with iTunes, disconnect
3. Open Mail.app and delete the note on your laptop
4. Add something to the note on your iOS device, e.g. "Note 2: Well maybe I'll try adding something anyway."
5. Reconnect them and (Sync) with iTunes again, disconnect

At this point you've lost the entire note on both devices, including additions you made to it on your iOS device after you deleted the copy of it on your laptop.

If you're doubly unfortunate, iTunes has also thrown away the previous backup for your iOS device, keeping only the most recent "backup" for your iOS device, with, you guessed it, the deletion of that note.

The very act of attempting to backup your iOS device by explicitly syncing it with iTunes, was responsible for losing your data.

iTunes treated the copy of the note on your laptop (or rather the tombstone left in place after you deleted it) as more authoritative than the note on your iOS device - ignoring the fact that you explicitly added to the note on your iOS device after (time-wise) you deleted the copy of it in Mail.app.

iTunes treated the data you added as less important than your older act of deletion and threw away your data.

What should it have done instead? How should it have resolved a seeming conflict between a deletion and an addition, both of which happened after the most recent sync?

Principles

There are a couple of user interface principles to consider here. Quoting from Apple's own OSX Human Interface Guidelines:

Forgiveness

People need to feel that they can try things without damaging the system or jeopardizing their data. Create safety nets, such as the Undo and Revert to Saved commands, so that people will feel comfortable learning and using your product.

Warn users when they initiate a task that will cause irreversible loss of data.

Using iTunes sync jeopardized data. It removed the safety net of the previous iOS device backup. There is no command to Undo a sync or Revert to notes from before. iTunes did not warn before it itself caused an irreversible loss of data.

Next principle, same source:

User Control The key is to provide users with the capabilities they need while helping them avoid dangerous, irreversible actions. For example, in situations where the user might destroy data accidentally, you should always provide a warning, but allow the user to proceed if they choose.

iTunes provided no warning before an irreversible deletion of the note on the iOS device which yes, did destroy data.

There are several approaches that Apple could have taken, any one of these would have been better than the irrecoverable deletion that occurred.

• Treat the later adding to the note as the user intending to never have deleted the note in the first place and recreate it on the laptop.
• If a note is edited in one place and deleted in another (in any time order), treat the edit as more important than the deletion.
• Keep a browsable backup of any deleted notes
• Provide the ability to undo a sync
• Provide the ability to recover deleted notes, on either device
Why not iOS7 and MacOS Mavericks

Perhaps you're wondering what versions of operating systems (and iTunes) were used: iOS 6, MacOS 10.7, iTunes 10.

Why not upgrade to iOS 7 and MacOS Mavericks (and thus iTunes 11) ?

Because then you lose the ability to sync directly between your devices. As noted previously:

[...] If you use OS X Mavericks v.10.9 or later, your contacts, calendars, and other info are updated on your computers and iOS devices via iCloud. [...]

If Apple can't get syncing right between just two devices/destinations:

iOS device <---> Mac laptop

Why should anyone have any expectation that they can get it right among three?

iOS device <---> iCloud <---> Mac laptop

Towards Indie Note Editing And Sync

This episode has illustrated that we cannot trust even Apple to either follow its own user interface guidelines, or implement those guidelines in its own software, nor can we trust Apple's syncing solutions to not lose our data.

I am now investigating alternatives (preferably open source) for:

• editing simple text notes on a mobile (e.g. iOS) device
• syncing them with a mac laptop
• possibly editing them in both places
• syncing them again without loss of data

Preferably without having to use "the cloud" (otherwise known as the internet or the web). That being said, perhaps an open source indie web sync solution could be another path forward. If I could sync any number of my own devices either with each other or directly with my own personal web site, that might work too.

Suggestions welcome. Some discussion already documented on indiewebcamp.com/sync.

Categorieën: Mozilla-nl planet

### Ian Bicking: Live Programming, Walkabout.js

Mozilla.org - wo, 27/11/2013 - 18:30

There’s a number of “live programming” environments used for education. Khan Academy is one example. In it, you write code on the left hand side, and you immediately see the result on the right hand side. You don’t hit “save” or “run” — it’s just always running.

There are a lot of nice features to this. There’s the feedback cycle: everything always happens. Or, if you get something wrong, it distinctly doesn’t happen. It’s similar to the static analysis we so often use — from the simplest case of syntax highlighting (which often finds syntax errors) to code lint tools, type checking or Intelli-sense. Live coding takes this further and makes execution itself somewhat static.

One of the nice parts about actually running the code is that you aren’t relying on static analysis, which is always limited. The only thorough analysis is to model the program’s execution by executing the program. Not to mention it allows the programmer to detect bugs that just cause the program to do the wrong thing, or to be incomplete, but not clearly incorrect, not in error. For instance, in the Khan example I make the shapes transparent:

No static analysis could tell me that this produces an unattractive picture of a person. Proponents of static analysis tend to have a limited concept of “bug” that doesn’t include this sort of problem.

To imagine what live execution might look like when applied more dramatically, you might want to check out Learnable Programming by Bret Victor. Underlying all his mockups is the expectation that the code is being run and analyzed at all times.

That’s all cool… except you can’t just run code all the time. It works for code that produces basically the same output every time it is run, that requires no input, that isn’t reactive or interactive. This is all true for Processing.js programs which Khan Academy and the other live programming environments I’ve seen use (and Khan Academy even disables random numbers to ensure consistency). Processing.js is focused on drawing pictures, and drawing via code is okay, but… it doesn’t excite me. What excites me about code is its emergent properties, how the execution of the program evolves. When you write interesting code you can enable things you didn’t realize, things that you won’t realize until you explore that same code. What happens when you interact with it in a new order? What happens when you give it new input? When a program always produces the same output it makes me feel like the program could be substituted by its output. Who needs to program a drawing when you can just use a drawing program?

I was thinking about these things when I was looking at Waterbear, which is a graphical/pluggable-tile programming language (very similar to Scratch).

A nice aspect of that sort of language is that you are forced to think in terms of the AST instead of text, because all those tiles are the AST. You also get a menu of everything the language can do, including its primitive functions.

With the language laid out like that, I saw that most of it was nice and static and deterministic. Control structures are deterministic: if COND then IFTRUE else IFFALSE always executes the same code given the same input. Most everything is: appending to a list always produces the same result, adding numbers always produces the same result. The list of the non-deterministic building blocks of a program is really small.

And this is exciting! If you can find all the non-deterministic parts of a program and come up with a range of viable results to plug in (i.e., mock) then you can run more-or-less the entire program. And the more I think about it, the more I realize that the list of non-deterministic parts can be quite small for many programs.

For instance, consider this program:

import random def guesser(): number = random.randint(1, 10) print("I'm thinking of a number between 1 and 10") while True: i = input() try: i = int(i) except ValueError: print("Please enter a number") continue if i == number: print("You win!") break elif i < number: print("Too small!") else: print("Too large!")

This is a simple program, but it can execute in lots of ways. There’s two non-deterministic parts: random.randint() and input(). The first can be made deterministic by seeding the random number generator with a known value (and the program can be exercised with multiple runs with multiple seeds). The second is trickier. We know input() returns a string that the user inputs, one line long. But if you throw random strings at the program you won’t get something very interesting. So we need just a little more help, a suggestion of what the person might return. E.g., input.suggest_returns = lambda: str(random.randint(-1, 11)) — it’s still valid that it can return anything, but we’ll be able to best exercise the program with those inputs. We still don’t have a smart player for our game, but it’s something.

This approach to exercising code is exciting because it’s basically automatic: you write your program, and if you are using primitives that have been setup for mocking, then it’s testable. You can build tools around it, the tools can find cases where things go wrong and replay those specific cases for the programmer until they are fixed.

It’s still a challenge to actually get deep into the program: the primitives often don’t express the expectation. For instance in this guessing program it’s valid to enter “one”, but it’s not not very interesting. If you are testing something interactive you might have a Cancel button that undoes a bunch of inputs; while it’s worth hitting Cancel every so often, generally it’s not interesting, even anti-interesting.

But with these thoughts in mind I was naturally drawn to the browser. A browser Javascript program is handy because it has very specific and a fairly limited set of primitives. Nearly everything that’s not deterministic would be considered part of the DOM, which includes not just the HTML page but also (at least in the terminology used by browser insiders) includes all the browser-specific functions exposed to content.

In the case of a browser program, the program tends to be fairly reactive: much of what happens is the program listening for events. This means much of the logic of the program is invoked from the outside. This is helpful because (with some effort) we can detect those listeners, and figure out what events the program is actually interested in (since something like a click can happen anywhere, but usually to no effect). Then you must also filter out handlers that apply to something that is not at the moment possible, for instance a click handler on an element that is not visible.

Trying to exercise a program is not the same as actually confirming the program did the right thing. This testing practice will reward the program that is littered with asserts. Asserts can’t be statically examined, and in that way they are worse than static types, but they can address things that can’t be statically described.

I believe there is a term for this concept: generative testing (for example, some slides from a presentation. Most of what I’ve seen under that name involves relatively small examples, with explicitly defined domains of input and output. I’m proposing to do this at the scale of an application, not a routine; to define inputs as any non-deterministic query or listener; and to define failure as some inline assertion error or warning.

Let’s Do It…?

With this in mind I created a library: Walkabout.js. This either uses the evidence jQuery leaves about bound event handlers, or it can use source code rewriting to track event handlers (tracking event handlers is harder than I would like). From this list it can create a list of plausible actions that can take place, seeing what elements might be clicked, hovered over, selected, etc., filtering out elements that aren’t visible, and so on. Then it uses a pseudo-random number generator to select an action, while checking for uncaught exceptions or warnings written to the console.

The library isn’t complete in what it mocks out, but that’s just a matter of doing more work. It’s a little harder to mock out server interaction, because there’s easy no way to know what exactly to expect back from the server — though if the server is deterministic (and the server’s state can be reset each run) then it’s okay to use it without mocking. Nothing deterministic need be mocked including external components.

There’s a lot I’d like to change about Walkabout.js’s code itself (my opinions on Javascript have changed since I first wrote it), but I worry I get ahead of myself by doing another round of development on it right now. There’s non-trivial tooling required to use this tool, and I need to find a larger environment where it can make sense. Or at least I want to find that environment, because I think the result will be more compelling.

Another big task to consider is how to actually explore the program in depth. It’s easy to come up with really boring, long, useless sequences of actions. Open dialog, close dialog x 100. Enter text, clear text x 10. Hitting some control that terminates the application is only interesting once. And though computers are fast they aren’t so fast they can spend most of their time doing completely useless things. I want my failures now!

To explore an application in depth we need to effectively search the application, using the range of possible inputs. The first idea for scoring a result that I thought of is code coverage: if you are reaching new code, then you are doing something interesting. Then the tooling becomes even more heavy-weight, you have to do code coverage and constantly track it to find productive avenues. Then a second, simpler idea: look for new sets of available inputs. If there’s a new button to click or new fields to interact with, then we’ve probably accomplished something. Continue to explore from that point forward. This option requires only the tooling we already have!

Why Are We Doing This Again?

In addition to just thinking about “live programming” I think this can be a great testing tool in general. And generally I’m suspicious of programming tools that are only applicable to toy/educational programming environments.

A common alternative approach to what I describe is to record user input, and then replay it as a test. It’s like random testing, only instead of a random number generator you have a person. This is basically a refinement of the standard practice of creating a script for a functional test that exercises your full application.

If you’ve used this approach you’ve probably found it annoying. Because it is. When you replay a recording and it doesn’t work, what is more likely: the application is broken, or you deliberately changed the application in a way that affects how the recording replays? In my experience 9 times out of 10 it’s the latter. We spend too much time fixing test failures that are not bugs.

The beauty of the generative approach is that it responds to your changes. It takes your program as it is, not as you might wish it to be. It runs the actions that are valid with this code, not some past version of your code. And the “tests” aren’t expected input and output, they are assertions, and those assertions live right with the code and stay updated with that code. If we care about testing, why don’t we include testing in the code itself? If you want to entertain various possible inputs why not suggest what you are expecting directly in the code?

Once you are exercising the code, you can also learn a lot more about the code at runtime. What kinds of object are assigned to a particular variable? How are pieces of code linked? What is the temporally related code? Given code coverage, you could isolate patterns that exercise a particular line of code. Having found a bug, you also have a script to reach that bug. Having made a change, you could identify past scripts that reach that changed area, giving you a chance to dive into the effect of that change. Many of these kinds of tools would be valid in a general sense, but require a well-exercised program to be useful — because most software tooling doesn’t include a “do lots of stuff” option we’re holding ourself back when it comes to runtime analysis.

So what do you think?

If you want to give it a really quick/rough try, go here, grab the bookmarklet, and go to a single-page app and try it out. It might do silly things, or nothing, but maybe it’ll do something interesting?

Categorieën: Mozilla-nl planet

### Karl Dubost: I like it hardcoded, but not in my code (SFW)

Mozilla.org - wo, 27/11/2013 - 18:05

This is a simple reminder to the Web Developer Community. Hardcoded strings are bad, even for a quick hack to solve an issue that you will forget the day after tomorrow. One of the most recent examples is a site which after being contacted has modified its user agent detection code so Firefox OS would receive the mobile content instead of the desktop content.

So this is what you get when you access the Web site with Firefox OS 1.0. The browser is redirected to http://mobile.lepoint.fr/

Request:

GET / HTTP/1.1 Accept: */* Accept-Encoding: gzip, deflate, compress Host: www.lepoint.fr User-Agent: Mozilla/5.0 (Mobile; rv:18.0) Gecko/18.0 Firefox/18.0

Response:

HTTP/1.1 301 Moved Permanently Content-Encoding: gzip Content-Length: 40987 Content-Type: text/html Date: Wed, 27 Nov 2013 16:47:53 GMT Location: http://mobile.lepoint.fr/ Server: Apache/2.2.25 (Unix) PHP/5.2.17 Vary: User-Agent,Accept-Encoding X-Powered-By: PHP/5.2.17

And this is what you get when you access the Web site with Firefox 1.1. The browser gets the desktop version of the content.

Request:

GET / HTTP/1.1 Accept: */* Accept-Encoding: gzip, deflate, compress Host: www.lepoint.fr User-Agent: Mozilla/5.0 (Mobile; rv:18.1) Gecko/18.1 Firefox/18.1

Response:

HTTP/1.1 200 OK Content-Encoding: gzip Content-Length: 40642 Content-Type: text/html Date: Wed, 27 Nov 2013 16:48:58 GMT Server: Apache/2.2.25 (Unix) PHP/5.2.17 Vary: User-Agent,Accept-Encoding X-Powered-By: PHP/5.2.17

So those with an eagle eye will notice that the only difference in between those two is the version number from 18.0 to 18.1. As a matter of fact you can try any number different from 18.0 and you will get the desktop version.

What could you do?
1. identify s/mobi/i
2. remove ipad from the lot

In that way you will maximize the effect by allowing our friends like Opera Mobile, etc. and you will avoid the iPad Tablets that have a Mobile string.

Otsukare!

Categorieën: Mozilla-nl planet

### Alex Vincent: Enjoy the silence? Not so much.

Mozilla.org - wo, 27/11/2013 - 17:02

For a long time, I’ve been wondering why no one besides spammers was responding to my blog.  After all, comments are one of the main features of a blog, and without it, a blog is just a bully pulpit.  Finally someone got through to me and let me know that comments were broken on my blog entirely.

*sound of a headdesk echoing across the Internet*

Wow.  Just, wow.  I had no idea.  I’m very sorry about that… and I can say that they’re fixed now.  (Thanks a lot, old silently busted WordPress theme!)

Categorieën: Mozilla-nl planet

### Pete Moore: Weekly review 2013-11-27

Mozilla.org - wo, 27/11/2013 - 15:57
Accomplishments & status:

Working on this with Simone. We managed to get emulator builds running on Mac!

This screenshot is taken from bld-lion-r5-087.build.releng.scl3.mozilla.com. We got this working manually, and are currently in the process of moving all the steps into puppet, and creating new builders to trigger the builds, and mozharness scripts to implement the build steps. We managed to get this working by, among other things, upgrading the OS from 10.7.2 to 10.7.4. We are currently validating whether we can get them working without upgrading the OS, since this would impact other builders running on the same 10.7 darwin builders. We have created a separate git repo for tracking our activity on this project, see: https://github.com/petemoore/b2g_emulator_build_darwin Please also see our blog on this work: http://petemoore.tumblr.com/post/66705593417/building-b2g-emulator-on-darwin

Blocked/Waiting on:

To look at over the next week:

Continue with this work on the OS X emulator builds.

Areas to develop:

To be discussed.

Quarterly goal tracking:

Notes / Actions:
Categorieën: Mozilla-nl planet

### Niko Matsakis: Thoughts on DST, Part 2

Mozilla.org - wo, 27/11/2013 - 12:36

In the previous post I elaborated a bit on DSTs and how they could be created and used. I want to look a bit now at an alternate way to support the combination of vector types and smart pointers (e.g., RC<[uint]>). This approach avoids the use of DSTs. We’ll see that it also addresses some of the rough patches of DST, but doesn’t work quite as well for object types.

This is part 2 of a series:

1. Examining DST.
2. Examining an alternative formulation of the current system.
3. …and then a twist.
4. Further complications.
Existential types, take 2

Previously I showed how a type like [T] could be interpreted as an existential type like exists N. [T, ..N]. In this post, I explore the idea that we most the exists qualifier to a different level. So ~[T], for example, would be interpreted as exists N. ~[T, ..N] rather than ~(exists N. [T, ..N]). Naturally the same existential treatment can be applied to objects. So &Trait is formalized as exists T:Trait. &T.

This is in a way very similar to what we have today. In particular, there are no dynamically sized types: [T] is not a type on its own, but rather a kind of shorthand that “globs onto” the enclosing pointer type. However, as we proceed I’ll outline a couple of points where we can generalize and improve upon on what we have today; this is because, today, a ~[T] value is considered a type all its own that is totally distinct from a ~[T, ..N], rather than being an existential variant. This has implications for how we build vectors and for our ability to smoothly support user-defined pointer types.

Now, when I say shorthand, does that imply that users could write out a full existential type? Not necessarily, and probably not in the initial versions. Perhaps in the future. I am thinking of more of a mental shorthand, as instruction for how to think about a type like &Trait or &[T].

Representing existentials

Moving the existential qualifier outside of the pointer simplifies the story about representation and coercion. An existential type like is always represented as two words:

repr(exists N. U) == (repr(U), uint) repr(exists T:Trait. U) == (repr(U), vtable)

So, for example, in the type ~[T] == exists N. ~[T, ..N], the representation would be (pointer, length), where pointer is a pointer to a [T, ..N]. Of course, we don’t know what N is, but that doesn’t matter, because it doesn’t affect the pointer. We can therefore adjust our definition repr slightly to codify the fact that we don’t know – nor care – about the precise values of N or T:

repr(exists N. U) == (repr(U[N => 0]), uint) repr(exists T:Trait. U) == (repr(U[T => ()]), vtable)

Here I just substituted 0 for N and () for T. This makes sense since the compiler doesn’t really know what those values are at compilation time. It also implies that we cannot create an existential unless it’s safe to ignore N and T – e.g., exists N. [T, ..N] would be illegal, since there is no pointer indirection, and hence knowing N is crucial to knowing the representation of [T, ..N].

This definition probably looks pretty similar to what I had before but it’s different in a crucial way. In particular, there are no more fat pointers – rather there are existential types that abstract over pointers. The length or vtable are part of the representation of the existential type, not the pointer. Let me explain the implications of this by example. Imagine our RC type that we had before:

struct RC<T> { priv data: *T, priv ref_count: uint, }

If we have a RC<[int, ..3]> instance, its representation will be (pointer, ref_count). But if we coerce it to RC<[int]>, its representation will be ((pointer, ref_count), length). Note that RC pointer itself is unchanged: it’s just embedded in a tuple.

Embedding the RC value in a tuple is naturally simpler than what we had before, which had to kind of rewrite the RC value to insert the length in the middle. But it’s also just plain more expressive. For example, consider the example of a custom allocator smart pointer that includes some headers on the allocation before the data itself (I introduced this type in part 1):

With DST, a type like MyAlloc1<[int]> is not even expressible because the type parameter T is not found behind a pointer sigil and thus T couldn’t be bound to an unsized type. Even if we could overcome that, we could not have coerced a MyAlloc1<[int,..3]> to a MyAlloc<[int]> because we couldn’t “convert” the representation of MyAlloc1 to make data.payload fat pointer. But all of this poses no problem under the existential scheme: if we represent MyAlloc1<[int, ..3]> as (pointer), the representation of MyAlloc1<[int]> is just ((pointer), length). This in turn implies that it should be possible to support the C-like inline arrays that I described before, though some future extensions will be required.

What does this scheme mean in practice?

For users, this scheme will feel pretty similar to what we have today, except that some odd discrepancies like ~[1, 2, 3] vs ~([1, 2, 3]) go away.

In general, the only legal operation we would permit on an existential type like RC<[int]> or ~[int] is to dereference it. The compiler automatically propagates the existential-ness over to the result of the dereference. That means that *rc where rc has type RC<[int]> would have type &[int] – or, more explicitly, exists N. RC<[int, ..N]> is dereferenced to exists N. &[int, ..N]. In formal terms, this is a combined pack-and-unpack operation. I’ll discuss this in part 3 of this series. The special case would be &[T], for which we can define indexing – and this would also perform the bounds check. Object types (&Trait, RC<Trait>) would be similar except that they would only permit dereferencing and method calls.

For people implementing smart pointers, this scheme has a straightforward story. No special work is required to make a smart pointer compatible with vector or trait types: after all, at the time that a smart pointer instance is created, we always have full knowledge of the type being allocated. So if the user writes new(RC) [1, 2, 3] (employing the overloadable new operator we are discussing), that corresponds to creating an instance of RC<[int, ..3]>. [int, ..3] is just a normal type with a known size, like any other.

“Case study”: Ref-counted pointers

Really, the only thing that distinguish a “smart pointer” from any other type is that it overloads * (and possibly integrates with new). I’ve got another post planned on the details of these mechanisms, but let’s look at overloading * a bit here to see how it interacts with existential types. The deref operator traits would look something like this:

trait Deref<T> { // Equivalent of &*self operation fn deref<'a>(&'a self) -> &'a T; } trait MutDeref<T> { // Equivalent of &mut *self operation fn mut_deref<'a>(&'a mut self) -> &'a mut T; }

Here is how we might implement define an RC type and implement the Deref trait:

struct RC<T> { priv data: *T, priv ref_count: uint, } impl<T> Deref<T> for RC<T> { fn deref<'a>(&'a self) -> &'a T { &*self.data } }

Note that RC doesn’t implement the MutDeref trait. This is because RC pointers can’t make any kind of uniqueness guarantees. If you want a ref-counted pointer to mutable data you can compose one using the newly created Cell and RefCell types, which offer dynamic soundness checks (e.g., RC<Cell<int>> would be a ref-counted mutable integer). In any case, I don’t have the space to delve into more detail on mutability control in the face of aliasing here – it would make a good topic for a future post as we’ve been working on a design there that offers a better balance than today’s @mut and is smart-pointer friendly.

As I said before, the RC implementation does not make any mention whatsoever of vectors or arrays or anything similar. It’s defined over all types T, and that includes [int, ..3]. Nothing to see here folks, move along.

The compiler will invoke the user-defined deref operator both for explicit derefs (the * operator) and auto-derefs (field access, method call, indexing). Consider the following example:

fn sum(rc: RC<[int]>) -> int { let mut sum = 0; let l = rc.len(); // (1) for i in range(0, l) { sum += rc[i]; // (2) } }

Here, autoderef will be employed at two points. First, in the call rc.len(), the pointer rc will be autoderef’d to a &[int] while searching for a len() method (see the type rules below for how this works). len() is defined for a &[int] type, and so the call succeeds. Similarly in the access rc[i], the indexing operator will autoderef rc to &[int] in its search for something indexable. Since &[int] is indexable, the call succeeds. The important point here is that the RC type itself only supports deref; the indexing operations etc come for free because &[int] is indexable.

UPDATE: Thinking on this a bit more I realized an obvious complication. Without knowing the value of N, we can’t actually know which monomorphized variant of deref to invoke. This matters if the value of N affects the layout of fields and so on. There are various solutions to this – for example, only permitting existential construction when the types are laid out such that the value of N is immaterial for anything besides bounds checking, or perhaps including a vtable rather than a length – but it is definitely a crimp in the plan. Seems obvious in retrospect. Well, more thought is warranted.

Comparing DST to this approach

I currently favor this approach – which clearly needs a confusing acronym! – over DST. It seems simpler overall and the ability to coerce from arbitrary pointer types into existential types is very appealing. It is a shame that it doesn’t address the issues with object types that I mentioned in the previous post but there are workarounds there (better factoring for traits intended to be used as objects, essentially).

This will not address the new user confusion that &[T] is valid syntax even though [T] is not a type, but I think it does offer a new way to better explain that discrepancy: &[T] is short for &[T, ..N] for an unknown N, and thus [T, ..N] is the memory’s actual type.

Categorieën: Mozilla-nl planet

### Alex Vincent: Should I switch to GitHub? Should I mirror?

Mozilla.org - wo, 27/11/2013 - 10:37

I’m in a dilemma.  Enough Mozilla community members have asked me, “why don’t you switch to GitHub?  There’s a larger JS community over there.”  I can’t exactly ignore that, considering that I really do need help.  And yet… I like SourceForge.  But not enough to be alone in the desert.

What do you think, Mozillians?  I know you’re busy, but how strongly do you prefer GitHub or SourceForge over the other?

UPDATE:  I just discovered comments were broken… fixed.

Categorieën: Mozilla-nl planet

### Niko Matsakis: Thoughts on DST, Part 1

Mozilla.org - wo, 27/11/2013 - 03:52

In the past, I’ve been quite the champion of dynamically sized types (DST). Specifically what this means is that things like [T] and Trait would be “types” in the Rust type system. Lately I’ve been investing a lot of effort thinking through the ramifications of offering better support for smart pointers, and in particular how this interacts with dynamically sized types, and I am no longer persuaded that DST offer the best way forward. I’m a bit unsure, though, and the topic is complicated, so I wanted to stop and write up a short series of posts laying out my thought process thus far. This post will describe what it would mean to offer DST in more detail. I don’t plan to give a lot of Rust background, since there’s enough to talk about.

This is part 1 of a series:

Vectors vs braces

I am assuming that we adopt the Vec vs ~[T] proposal that originated on Reddit. The basic idea is to have the builtin vector notation [T] be used for fixed-length vectors, whereas growable vectors would be implemented with a library type like Vec<T>. This slices the gordian knot that, to be growable, you need a pointer, length, and capacity (hence sizeof::<~[T]>() would be 3 words) but otherwise a pointer and a length will suffice (hence sizeof::<&[T]>() would be 2 words). Since ~[T] would not be growable, all “pointers-to-[T]” can be 2 words in length.

Interpreting DSTs as existential types

Informally, a type like [T] means “0 or more instances of T laid out sequentially in memory”. Formally, we can describe that type like exists N. [T, ..N] – this is called an existential type and it means “this memory can be described as the type [T, ..N] for some value of N, but we don’t know what that value of N is”.

For now, I’m going to limit my discussion to vector types, because I think the interactions are more clear, but everything applies equally well to “trait types”. For example, the type Trait can be described as exists T:Trait. T, which is read “an instance of some type T that implements Trait, but we don’t know what that type T is”.

Anyway, so back to vector types. Now imagine that we have a value slice of type &[T] – armed with existential types, we can interpret this sa &(exists N. [T, ..N]). That is, “a borrowed pointer to some memory containing some number of T instances”. Of course, before we can do anything useful with slice, we kind of need to know how many instances of T are present: otherwise if we had an expression like slice[i], we’d have no way to know whether the index i was in bounds or not.

To address this problem, we say that pointers to a DST (i.e., &[T], ~[T], and *[T]) are all fat pointers, meaning that they are represented as two words: the data pointer and a length. We also impose limitations that ensures that DSTs only appear behind one of their built-in pointer types.

This does not mean that you will never see a DST anyplace else. For example, I could write a struct definition like RC (for reference-counted data):

struct RC<T> { priv data: *T, priv ref_count: uint, }

Now, a type like RC<[int]> would be legal – this is because, when the structure definition is “expanded out”, the DST [int] appears behind a * sigil (i.e., the type of data will be *[int]).

This setup implies that a type like RC<[int, ..3]> and a type like RC<[int]> will have different sizes; we’ll see later that this is a bit of a complication in some regards. The reason that the sizes are different is that [int, ..3] has a statically known size, and hence *[int, ..3] is a thin pointer. This means that RC<[int, ..3]> is represented in memory as two words: (pointer, ref_count). [int], in contrast, requires a fat pointer, and thus the layout of RC<[int]> is ((pointer, length), ref_count).

Constructing instances of DSTs via coercion

So how do we get an instance of a type that includes a DST? Let’s start by discussing ~[int] and then extend to RC<[int]>.

My preferred approach is to extend the “casting” approach that we use for creating objects. For now, let me just discuss this in terms of the built-in

Recall that an object type like ~Writer is creating by repackaging an existing pointer of some type ~Foo, where Foo implements Writer, with a vtable. In other words, we convert the static knowledge of the precise type (Foo) into dynamic knowledge (the vtable).

You can imagine using a similar process to obtain a ~[int]. For example, suppose we created a vector v like v = ~[1, 2, 3] – this would have type ~[int, ..3]. This vector v could then be coerced into a ~[int] by “forgetting” the length and moving it to a dynamic value. That is, our thin pointer v can be converted into a fat poiner (v, 3). This is exactly analogous to the object creation case: the static knowledge about the length has been moved into a dynamic value.

Using this coercion approach addresses one of the annoying inconsistencies we suffer with today. That is, today, the expression ~[1, 2, 3] allocates a ~[int] of length 3, but the expression ~([1, 2, 3]) allocates a ~[int, ..3]. That is, there is a special case in the parser where ~[...] is not treated as the composition of the ~ operator with a [...] expression but rather as a thing in and of itself. This is consistent with our current approach to the types, where ~[int] is a single, indivisible unit, but it is not particularly consistent with a DST approach, nor it is particularly elegant.

Another problem with the current “special allocation form” approach is that it doesn’t extend to objects, unless we want to force object creation to allocate a new pointer. That was how we used to do things, but we found that re-packaging an existing pointer is much cleaner and opens up more coding patterns. (For example, I could take as input an &[int, ..3] and then pass it to a helper routine that expects a &[int].)

Extending coercion to user-defined smart pointer types

OK, in the previous section I showed coercion is an elegant option for creating DSTs. But can we extend it to user-defined smart pointer types like RC? The answer is mostly yes, though not without some complications. The next post will cover an alternative interpretation of RC<[int]> that works more smoothly.

The challenge here is that the memory layouts for a type like RC<[int, ..3]> and a type like RC<[int]> are quite different. As I showed before, the former is simply (data, refcount), but the latter is ((data, length), refcount). Still, we are coercing the pointer, which means that we’re allowed to change the representation. So you can imagine the compiler would emit code to move each field of the RC<[int, ..3]> into its proper place, inserting the length as needed.

For a simple type like RC, the compiler adaption is always possible, but it won’t work when the data for the smart pointer is itself located behind another pointer. For example, imagine a smart pointer that connects to some other allocation scheme, in which the allocation is always associated with a header allocated in some fixed location:

Here the *T which must be coerced from a *[int, ..3] into a *[int] is located behind another pointer. We clearly can’t adapt this type.

Limitation: DSTs must appear behind a pointer

That example is rather artificial, but it points the way at another limitation. It’s easy to imagine a special allocator that inserts a header before the payload itself. If we attemped to model this header explicitly in the types, it might look like the following:

This is basically the same as before but the type of payload has changed. As before, we can’t hope to coerce this type, but this is for an even more fundamental reason: because the T type doesn’t appear directly behind a * pointer, it couldn’t even be instantiated with an unsized type to begin with.

This same principle extends to another use case, one where DSTs seem like they offer a good approach, but in fact they do not. A common C idiom is to have a structure that is coupled with an array; because the length of this array does not change, the structure and the array are allocated in one chunk. As an example, let’s look briefly at functional trees, where the shape doesn’t change once the trees are construted; in such a case, we might want to allocate the node and its array of children all together.

There are many ways to encode this in C but this is my personal preferred one, because it is quite explicit:

struct Tree { int value; int num_children; } Tree **children(Tree *t) { return (Tree**) (t + 1); }

Whenever we allocate a new tree node, we make sure to allocate enough space not only for the Tree fields but for the array of children:

Tree *new_tree(int value, int num_children) { size_t children_size = sizeof(Tree*) * num_children; size_t parent_size = sizeof(Tree); Tree *parent = (Tree*) malloc(parent_size + children_size); parent->value = value; parent->num_children = num_children; parentset(children(parent), 0, children_size); return parent; }

And I can easily iterate over a subtree like so:

int sum(Tree *parent) { int r = parent->value; for (int i = 0; i < parent->num_children; i++) sum(children(parent)[i]); }

This is all nice but of course horribly unsafe. Can we create a safe Rust equivalent?

You might at first think that I could write a type like:

struct Tree { value: uint, num_children: uint, children: [RC<Tree>], // Refcounting works great for trees! }

But of course this structure wouldn’t be permitted, since a DST like [RC<Tree>] must appear behind a pointer. And of course Rust also has no idea that num_children is the length of children.

This is not to say that there is no way to handle this familiar C pattern, but it’s not clear how DST supports it. The next scheme I discuss offers a clearer path.

Impls and DSTs

One big argument in favor of DST is that it permits impls over types like [T] and Trait. This promises to eliminate a lot of boilerplate. But when I looked into it in more detail, I found the story wasn’t quite that simple.

Implementing for vector types is not that useful.

In my initial post, I gave the example of implementing ToStr, pointing out that without DST, you need a lot of impls to handle the “boilerplate” cases:

impl<T:ToStr> ToStr for ~T { ... } impl<'a, T:ToStr> ToStr for &'a T { ... } impl<T:ToStr> ToStr for ~[T] { ... } impl<'a, T:ToStr> ToStr for &'a [T] { ... }

Whereas with DST things are more composable:

impl<T:ToStr> ToStr for ~T { ... } impl<'a, T:ToStr> ToStr for &'a T { ... } impl<T:ToStr+Sized> ToStr for [T] { ... }

This point is still valid, but the question is, how far does this get you? When I started experimenting with other traits, I found that implementing on [T] often didn’t work out so well.

For example, consider the Map trait:

trait Map<K,V> { fn insert(&mut self, key: K, value: V); fn get<'a>(&'a self, key: K) -> &'a V; }

Imagine I wanted to implement the Map trait on association lists (vector of pairs). I’d prefer to implement on the type [(K,V)] because that would

impl<K:Eq,V> Map<K,V> for [(K,V)] { fn insert(&mut self, key: K, value: V) { // Here: self has type &mut [(K,V)] self.push((key, value)); // ERROR. } fn get<'a>(&'a self, key: K) -> &'a V { // Here: self has type &[(K,V)] for &(ref k, ref v) in self.iter() { if k == key { return v; } } fail!("Bad key"); } }

The problem here lies in the insert() method, where I find that I cannot push onto a slice.

But implementing for object types is useful.

On the other hand, because able to write an impl over a type like Trait is quite useful. Let me elaborate. Currently an object type like &Trait or ~Trait is not automatically considered to implement the interface Trait. This is because it is not always possible. Consider the following example:

trait Message { fn send(~self); fn increment(&mut self); fn read(&self) -> int; }

Now imagine that we were trying to implement this trait for an object type such as &Message. We’re going to run into problems because the object type bakes in a particular pointer type – in this case, & – and thus we are not able to implement the methods send() or increment():

impl Message for &Message { fn send(~self) { // Argh! self has type ~&Message, but I need // a ~Message. } fn increment(&mut self) { // Argh! self has type &mut &Message, but I need // an &mut Message. } fn read(&self) -> int { // OK, self has type &&Message, I can // transform that to an &Message and call read(): (*self).read(); } }

Thanks to inherited mutability, what would work is to implement Message for ~Message:

impl Message for ~Message { fn send(~self) { // OK, self has type ~~Message. A bit silly but workable. (*self).send() } fn increment(&mut self) { // self has type &mut ~Message. Inherited mutability // implies that the ~Message itself is thus mutable. (*self).increment(); } fn read(&self) -> int { // self has type &~Message. We can read it. (*self).read(); } }

Note that while this will compile, the type of send() is rather inefficient. We wind up with a double allocation. (I leave as an exercise to the reader to imagine what will happen when we extend self types to include things like self: RC<Self> and so on.)

If we are limited to implementing traits for object types, then, I think one must be careful when designing traits to be used as objects. You should avoid mixing sigils and instead have a series of base traits that are combined. And you should avoid ~self and prefer by-value self (modulo issue 10672).

trait ReadMessage { fn read(&self) -> int; } trait WriteMessage { fn increment(&mut self); } trait Message : ReadMessage+WriteMessage { fn send(self); }

Now we can implement ReadMessage for &Message, WriteMessage for &mut Message, and Message for ~Message (or other smart pointer types that convey ownership), no problem.

DST offers a way out

Alternatively, under a DST system, we could implement the original Message trait once for all object types:

impl Message for Message { fn send(~self) { /* self: ~Message, OK! */ } fn increment(&mut self) { /* self: &mut Message, OK! */ } fn read(&self) -> int { /* self: &Message, OK! */ } }

We could probably just have the compiler implement this automatically, even. One catch is that we could not support a by-value self method like fn send(self). This is mildly hostile to user-defined smart pointers since ownership transfer via object type methods would really be limited to ~ pointers.

Conclusion

None yet, wait for the thrilling part 2!

Categorieën: Mozilla-nl planet

### Mozilla plans tour to guide Firefox users when Australis comes around - Ghacks Technology News

Nieuws verzameld via Google - di, 26/11/2013 - 09:22

Ghacks Technology News

Mozilla plans tour to guide Firefox users when Australis comes around
Ghacks Technology News
No matter how you feel about it, it is clear that Mozilla will go forward with it. What may happen along the way is that Mozilla will make adjustments to it, depending on user feedback. Australis is a major redesign of the browser. It is only natural ...
Firefox OS on the Fast TrackMidsize Insider
Firefox To Feature A New Streamlined, Rounded LookTechtree.com

alle 3 nieuwsartikelen »
Categorieën: Mozilla-nl planet

### Mozilla, BBC Team Up To Improve The Web - WebProNews

Nieuws verzameld via Google - ma, 25/11/2013 - 20:55

Mozilla, BBC Team Up To Improve The Web
WebProNews
Mozilla chimed in with its own statement on the signing of an MOU with the BBC by saying that it will continue to work with the broadcaster “on open technical standards, Web literacy, education projects and a number of other shared initiatives.” The ...

en meer »
Categorieën: Mozilla-nl planet

### The BBC and Mozilla formalize partnership for web skills in the UK

Mozilla Blog - ma, 25/11/2013 - 07:28

Paula Le Dieu at the MOU signing with the BBC.

Today we’re excited to announce the formalization of our partnership with the BBC to support free and open internet technologies.

While the BBC and Mozilla have been working together on digital skills for some time—the BBC’s Connected Studios regularly run their workshops at Mozilla’s London community space, and we are working together on a number of digital learning initiatives that involve both Open Badges and Webmaker projects—this is the first time Mozilla and the BBC have formalized the relationship with a memorandum of understanding.

Mozilla will join three other organizations, the Open Data Institute, the Open Knowledge Foundation, and the Europeana Foundation to support free and open internet technologies. Mozilla and the BBC will continue to work together on open technical standards, web literacy, education projects, and a number of other shared initiatives. The agreements precede a year of programming and events that support Web and Digital Creativity across the UK that the BBC will kick off in 2015.

You can read more about the Memorandum of Understanding over at the BBC website.

Categorieën: Mozilla-nl planet