De Nederlandse
Mozilla-gemeenschap

This got a bit delayed due to $DAYJOB interfering with my important hacking and writing time (darn having to make a living), but Firefox 111 is out. As usual you'll need to deal with bug 1775202 either with this patch — but without the line containing desktop_capture/desktop_capture_gn, since that's been gone since the latest WebRTC update — or put --disable-webrtc in your .mozconfig if you don't need WebRTC. The workaround adding #pragma GCC diagnostic ignored "-Wnonnull" to js/src/irregexp/imported/regexp-parser.cc for optimized builds fortunately was addressed by bug 1810584, so you no longer need it, and the browser otherwise builds and works with the PGO-LTO patch for Firefox 110 and the .mozconfigs from Firefox 105.

It’s well known that my blog has some issues. The category links don’t work. It renders oddly on mobile. And maybe Safari, too? The Rust snippets are not colored. The RSS feed is apparently not advertised properly in the metadata. It’s published via a makefile instead of some hot-rod CI/CD script, and it uses jekyll instead of whatever the new hotness is.1 Being a programmer, you’d think I could fix this, but I am intimidated by HTML, CSS, and Github Actions. Hence this call for help: I’d like to hire someone to “tune up” the blog, a combination of fixing the underlying setup and also the visual layout. This post will be a rough set of things I have in mind, but I’m open to suggestions. If you think you’d be up for the job, read on.

Desiderata2

In short, I am looking for a rad visual designer who also can do the technical side of fixing up my jekyll and CI/CD setup.

Specific works item I have in mind:

• Syntax highlighting
• Make it look great on mobile and safari
• Fix the category links
• Add RSS feed into metadata and link it, whatever is normal
• CI/CD setup so that when I push or land a PR, it deploys automatically
• “Tune up” the layout, but keep the cute picture!3

Bonus points if you can make the setup easier to duplicate. Installing and upgrading Ruby is a horrible pain and I always forget whether I like rbenv or rubyenv or whatever better. Porting over to Hugo or Zola would likely be awesome, so long as links and content can be preserved. I do use some funky jekyll plugins, though I kind of forgot why. Alternatively maybe something with docker?

Current blog implementation

The blog is a jekyll blog with a custom theme. Sources are here:

• https://github.com/nikomatsakis/babysteps
• https://github.com/nikomatsakis/nikomatsakis-babysteps-theme

Deployment is done via rsync at present.

Interested?

Send me an email with your name, some examples of past work, any recommendations etc, and the rate you charge. Thanks!

1. On the other hand, it has that super cute picture of my daughter (from around a decade ago, but still…). And the content, I like to think, is decent. ↩

2. I have a soft spot for wacky plurals, and “desiderata” might be my fave. I heard it first from a Dave Herman presentation to TC39 and it’s been rattling in my brain ever since, wanting to be used. ↩

3. Ooooh, I always want nice looking tables like those wizards who style github have. How come my tables are always so ugly? ↩

Git-cinnabar is a git remote helper to interact with mercurial repositories. It allows to clone, pull and push from/to mercurial remote repositories, using git.

Get it on github.

These release notes are also available on the git-cinnabar wiki.

What’s new since 0.5.11?

• Full rewrite of the Python parts of git-cinnabar in Rust.
• Push performance is between twice and 10 times faster than 0.5.x, depending on scenarios.
• Based on git 2.38.0.
• git cinnabar fetch now accepts a --tags flag to fetch tags.
• git cinnabar bundle now accepts a -t flag to give a specific bundlespec.
• git cinnabar rollback now accepts a --candidates flag to list the metadata sha1 that can be used as target of the rollback.
• git cinnabar rollback now also accepts a --force flag to allow any commit sha1 as metadata.
• git cinnabar now has a self-update subcommand that upgrades it when a new version is available. The subcommand is only available when building with the self-update feature (enabled on prebuilt versions of git-cinnabar).
• Disabled inexact copy/rename detection, that was enabled by accident.

What’s new since 0.6.0rc2?

• Fixed use-after-free in metadata initialization.
• Look for the new location of the CA bundle in git-windows 2.40.

March 31, or “three thirty-one,” is something of a talisman in the Mozilla community. It’s the date that, back in 1998, Mozilla first came into being — the date that we open-sourced the Netscape code for the world to use.

This year, “three thirty-one” is especially meaningful: It’s Mozilla’s 25 year anniversary.

A lot has changed since 1998. Mozilla is no longer just a bold idea. We’re a family of organizations — a nonprofit, a public benefit-corporation, and others — that builds products, fuels movements, and invests in responsible tech.

And we’re no longer a small group of engineers in Netscape’s Mountain View office. We’re technologists, researchers, and activists located around the globe — not to mention tens of thousands of volunteers.

But if a Mozillian from 1998 stepped into a Mozilla office (or joined a Mozilla video call) in 2023, I think they’d quickly feel something recognizable. A familiar spirit, and a familiar set of values.

When Mozilla open-sourced our browser code 25 years ago, the reason was the public interest: We wanted to spark more innovation, more competition, and more choice online. Technology in the public interest has been our manifesto ever since — whether releasing Firefox 1.0 in 2004, or launching Mozilla.ai earlier this year.

Right now, technology in the public interest seems more important than ever before. The internet today is deeply entwined with our personal lives, our professional lives, and society at large. The internet today is also flawed. Centralized control reduces choice and competition. A focus on “engagement” magnifies outrage, and bad actors are thriving.

Right now — and over the next 25 years — Mozilla can do something about this.

Mozilla’s mission and principles are evergreen, and we will continue to evolve to meet the needs and challenges of the modern internet. How people use the internet will change over time, but the need for innovative products that give individuals agency and choice on the internet is a constant. Firefox has evolved from a faithful and efficient render of web pages on PCs to a cross-platform agent that acts on behalf of the individual, protecting them from bad actors and surveillance capitalists as they navigate the web. Mozilla has introduced new products, such as Firefox Relay and Mozilla VPN, to keep people’s identity protected and activity private as they use the internet. Mozilla is contributing to healthy public discourse, with Pocket enabling discovery of amazing content and the mozilla.social Mastodon instance supporting decentralized, community-driven social media.

We’re constantly exploring ways to apply new technologies so that people feel the benefits in their everyday lives, as well as inspire others to responsibly innovate on behalf of humanity. As AI emerges as a core building block for the future of computing, we’ll turn our attention in that direction and ask: How can we make products and technologies like machine learning work in the public interest? We’ve already started this work via Mozilla.ai, a new Mozilla organization focusing on a trustworthy, independent, and open-source AI ecosystem. And via the Responsible AI Challenge, where we’re convening (and funding) bright people and ambitious projects building trustworthy AI.

And we will continue to champion public policy that keeps the internet healthy. There is proposed legislation around the world that seeks to maintain the internet in the public interest: the Platform Accountability and Transparency Act (PATA) in the U.S., the Digital Services Act (DSA) in the EU. Mozilla has helped shape these laws, and we will continue to follow along closely with their implementation and enforcement.

On this “three thirty-one,” I’m realistic about the challenges facing the internet. But I’m also optimistic about Mozilla’s potential to address them. And I’m looking forward to another 25 years of not just product, but also advocacy, philanthropy, and policy in service of a better internet.

In Firefox 110, users now have the ability to control which third-party DLLs are allowed to load into Firefox processes.

Let’s talk about what this means and when it might be useful.

What is third-party DLL injection?

On Windows, third-party products have a variety of ways to inject their code into other running processes. This is done for a number of reasons; the most common is for antivirus software, but other uses include hardware drivers, screen readers, banking (in some countries) and, unfortunately, malware.

Having a DLL from a third-party product injected into a Firefox process is surprisingly common – according to our telemetry, over 70% of users on Windows have at least one such DLL! (to be clear, this means any DLL not digitally signed by Mozilla or part of the OS).

Most users are unaware when DLLs are injected into Firefox, as most of the time there’s no obvious indication this is happening, other than checking the about:third-party page.

Unfortunately, having DLLs injected into Firefox can lead to performance, security, or stability problems. This is for a number of reasons:

• DLLs will often hook into internal Firefox functions, which are subject to change from release to release. We make no special effort to maintain the behavior of internal functions (of which there are thousands), so the publisher of the third-party product has to be diligent about testing with new versions of Firefox to avoid stability problems.
• Firefox, being a web browser, loads and runs code from untrusted and potentially hostile websites. Knowing this, we go to a lot of effort to keep Firefox secure; see, for example, the Site Isolation Security Architecture and Improved Process Isolation. Third-party products may not have the same focus on security.
• We run an extensive number of tests on Firefox, and third-party products may not test to that extent since they’re probably not designed to work specifically with Firefox.

Indeed, our data shows that just over 2% of all Firefox crash reports on Windows are in third-party code. This is despite the fact that Firefox already blocks a number of specific third-party DLLs that are known to cause a crash (see below for details).

This also undercounts crashes that are caused indirectly by third-party DLLs, since our metrics only look for third-party DLLs directly in the call stack. Additionally, third-party DLLs are a bit more likely to cause crashes at startup, which are much more serious for users.

Firefox has a third-party injection policy, and whenever possible we recommend third parties instead use extensions to integrate into Firefox, as this is officially supported and much more stable.

Why not block all DLL injection by default?

For maximum stability and performance, Firefox could try to block all third-party DLLs from being injected into its processes. However, this would break some useful products like screen readers that users want to be able to use with Firefox. This would also be technically challenging and it would probably be impossible to block every third-party DLL, especially third-party products that run with higher privilege than Firefox.

Since 2010, Mozilla has had the ability to block specific third-party DLLs for all Windows users of Firefox. We do this only as a last resort, after trying to communicate with the vendor to get the underlying issue fixed, and we tailor it as tightly as we can to make Firefox users stop crashing. (We have the ability to only block specific versions of the DLL and only in specific Firefox processes where it’s causing problems). This is a helpful tool, but we only consider using it if a particular third-party DLL is causing lots of crashes such that it shows up on our list of top crashes in Firefox.

Even if we know a third-party DLL can cause a crash in Firefox, there are times when the functionality that the DLL provides is essential to the user, and the user would not want us to block the DLL on their behalf. If the user’s bank or government requires some software to access their accounts or file their taxes, we wouldn’t be doing them any favors by blocking it, even if blocking it would make Firefox more stable.

Giving users the power to block injected DLLs

With Firefox 110, users can block third-party DLLs from being loaded into Firefox. This can be done on the about:third-party page, which already lists all loaded third-party modules. The about:third-party page also shows which third-party DLLs have been involved in previous Firefox crashes; along with the name of the publisher of the DLL, hopefully this will let users make an informed decision about whether or not to block a DLL. Here’s an example of a DLL that recently crashed Firefox; clicking the button with a dash on it will block it:

Here’s what it looks like after blocking the DLL and restarting Firefox:

If blocking a DLL causes a problem, launching Firefox in Troubleshoot Mode will disable all third-party DLL blocking for that run of Firefox, and DLLs can be blocked or unblocked on the about:third-party page as usual.

How it works

Blocking DLLs from loading into a process is tricky business. In order to detect all DLLs loading into a Firefox process, the blocklist has to be set up very early during startup. For this purpose, we have the launcher process, which creates the main browser process in a suspended state. Then it sets up any sandboxing policies, loads the blocklist file from disk, and copies the entries into the browser process before starting that process.

The copying is done in an interesting way: the launcher process creates an OS-backed file mapping object with CreateFileMapping(), and, after populating that with blocklist entries, duplicates the handle and uses WriteProcessMemory() to write that handle value into the browser process. Ironically, WriteProcessMemory() is often used as a way for third-party DLLs to inject themselves into other processes; here we’re using it to set a variable at a known location, since the launcher process and the browser process are run from the same .exe file!

Because everything happens so early during startup, well before the Firefox profile is loaded, the list of blocked DLLs is stored per Windows user instead of per Firefox profile. Specifically, the file is in %AppData%\Mozilla\Firefox, and the filename has the format blocklist-{install hash}, where the install hash is a hash of the location on disk of Firefox. This is an easy way of keeping the blocklist separate for different Firefox installations.

Detecting and blocking DLLs from loading

To detect when a DLL is trying to load, Firefox uses a technique known as function interception or hooking. This modifies an existing function in memory so another function can be called before the existing function begins to execute. This can be useful for many reasons; it allows changing the function’s behavior even if the function wasn’t designed to allow changes. Microsoft Detours is a tool commonly used to intercept functions.

In Firefox’s case, the function we’re interested in is NtMapViewOfSection(), which gets called whenever a DLL loads. The goal is to get notified when this happens so we can check the blocklist and forbid a DLL from loading if it’s on the blocklist.

To do this, Firefox uses a homegrown function interceptor to intercept calls to NtMapViewOfSection() and return that the mapping failed if the DLL is on the blocklist. To do this, the interceptor tries two different techniques:

• On the 32-bit x86 platform, some functions exported from a DLL will begin with a two-byte instruction that does nothing (mov edi, edi) and have five one-byte unused instructions before that. (either nop or int 3)  For example: nop nop nop nop nop DLLFunction: mov edi, edi (actual function code starts here)

  If the interceptor detects that this is the case, it can replace the five bytes of unused instructions with a jmp to the address of the function to call instead. (since we’re on a 32-bit platform, we just need one byte to indicate a jump and four bytes for the address) So, this would look like

  jmp <address of Firefox patched function> DLLFunction: jmp $-5 # encodes in two bytes: EB F9 (actual function code starts here)

  When the patched function wants to call the unpatched version of DLLFunction(), it simply jumps 2 bytes past the address of DLLFunction() to start the actual function code.

• Otherwise, things get a bit more complicated. Let’s consider the x64 case. The instructions to jump to our patched function require 13 bytes: 10 bytes for loading the address into a register, and 3 bytes to jump to that register’s location. So the interceptor needs to move at least the first 13 bytes worth of instructions, plus enough to finish the last instruction if needed, to a trampoline function. (it’s known as a trampoline because typically code jumps there, which causes a few instructions to run, and then jumps out to the rest of the target function). Let’s look at a real example. Here’s a simple function that we’re going to intercept, first the C source (Godbolt compiler explorer link): int fn(int aX, int aY) { if (aX + 1 >= aY) { return aX * 3; } return aY + 5 - aX; }

  and the assembly, with corresponding raw instructions. Note that this was compiled with -O3, so it’s a little dense:

  fn(int,int): lea eax,[rdi+0x1] # 8d 47 01 mov ecx,esi # 89 f1 sub ecx,edi # 29 f9 add ecx,0x5 # 83 c1 05 cmp eax,esi # 39 f0 lea eax,[rdi+rdi*2] # 8d 04 7f cmovl eax,ecx # 0f 4c c1 ret # c3

  Now, counting 13 bytes from the beginning of fn() puts us in the middle of the lea eax,[rdi+rdi*2] instruction, so we’ll have to copy everything down to that point to the trampoline.

  The end result looks like this:

  fn(int,int) (address 0x100000000): # overwritten code mov r11, 0x600000000 # 49 bb 00 00 00 00 06 00 00 00 jmp r11 # 41 ff e3 # leftover bytes from the last instruction # so the addresses of everything stays the same # We could also fill these with nop’s or int 3’s, # since they won’t be executed .byte 04 .byte 7f # rest of fn() starts here cmovl eax,ecx # 0f 4c c1 ret # c3 Trampoline (address 0x300000000): # First 13 bytes worth of instructions from fn() lea eax,[rdi+0x1] # 8d 47 01 mov ecx,esi # 89 f1 sub ecx,edi # 29 f9 add ecx,0x5 # 83 c1 05 cmp eax,esi # 39 f0 lea eax,[rdi+rdi*2] # 8d 04 7f # Now jump past first 13 bytes of fn() jmp [RIP+0x0] # ff 25 00 00 00 00 # implemented as jmp [RIP+0x0], then storing # address to jump to directly after this # instruction .qword 0x10000000f Firefox patched function (address 0x600000000): <whatever the patched function wants to do>

  If the Firefox patched function wants to call the unpatched fn(), the patcher has stored the address of the trampoline (0x300000000 in this example). In C++ code we encapsulate this in the FuncHook class, and the patched function can just call the trampoline with the same syntax as a normal function call.

  This whole setup is significantly more complicated than the first case; you can see that the patcher for the first case is only around 200 lines long while the patcher that handles this case is more than 1700 lines long! Some additional notes and complications:

  • Not all instructions that get moved to the trampoline can necessarily stay exactly the same. One example is jumping to a relative address that didn’t get moved to the trampoline – since the instruction has moved in memory, the patcher needs to replace this with an absolute jump. The patcher doesn’t handle every kind of x64 instruction (otherwise it would have to be much longer!), but we have automated tests to make sure we can successfully intercept the Windows functions that we know Firefox needs.
  • We specifically use r11 to load the address of the patched function into because according to the x64 calling convention, r11 is a volatile register that is not required to be preserved by the callee.
  • Since we use jmp to get from fn() to the patched function instead of ret, and similarly to get from the trampoline back into the main code of fn(), this keeps the code stack-neutral. So calling other functions and returning from fn() all work correctly with respect to the position of the stack.
  • If there are any jumps from later in fn() into the first 13 bytes, these will now be jumping into the middle of the jump to the patched function and bad things will almost certainly happen. Luckily this is very rare; most functions are doing function prologue operations in their beginning, so this isn’t a problem for the functions that Firefox intercepts.
  • Similarly, in some cases fn() has some data stored in the first 13 bytes that are used by later instructions, and moving this data to the trampoline will result in the later instructions getting the wrong data. We have run into this, and can work around it by using a shorter mov instruction if we can allocate space for a trampoline that’s within the first 2 GB of address space. This results in a 10 byte patch instead of a 13 byte patch, which in many cases is good enough to avoid problems.
  • Some other complications to quickly mention (not an exhaustive list!):
    • Firefox also has a way to do this interception across processes. Fun!
    • Trampolines are tricky for the Control Flow Guard security measure: since they are legitimate indirect call targets that do not exist at compile time, it requires special care to allow Firefox patched functions to call into them.
    • Trampolines also involve some more fixing up for exception handling, as we must provide unwind info for them.

If the DLL is on the blocklist, our patched version of NtMapViewOfSection() will return that the mapping fails, which causes the whole DLL load to fail. This will not work to block every kind of injection, but it does block most of them.

One added complication is that some DLLs will inject themselves by modifying firefox.exe’s Import Address Table, which is a list of external functions that firefox.exe calls into. If one of these functions fails to load, Windows will terminate the Firefox process. So if Firefox detects this sort of injection and wants to block the DLL, we will instead redirect the DLL’s DllMain() to a function that does nothing.

Final words

Principle 4 of the Mozilla Manifesto states that “Individuals’ security and privacy on the internet are fundamental and must not be treated as optional”, and we hope that this will give Firefox users the power to access the internet with more confidence. Instead of having to choose between uninstalling a useful third-party product and having stability problems with Firefox, now users have a third option of leaving the third-party product installed and blocking it from injecting into Firefox!

As this is a new feature, if you have problems with blocking third-party DLLs, please file a bug. If you have issues with a third-party product causing problems in Firefox, please don’t forget to file an issue with the vendor of that product – since you’re the user of that product, any report the vendor gets means more coming from you than it does coming from us!

More information

• Technical documentation about DLL blocklisting
• An Empirical Study of DLL Injection Bugs in the Firefox Ecosystem (.pdf)

Special thanks to David Parks and Yannis Juglaret for reading and providing feedback on many drafts of this post and Toshihito Kikuchi for the initial prototype of the dynamic blocklist.

The post Letting users block injected third-party DLLs in Firefox appeared first on Mozilla Hacks - the Web developer blog.

I’ve been thinking about async closures and how they could work once we have static async fn in trait. Somewhat surprisingly to me, I found that async closures are a strong example for where async transformers could be an important tool. Let’s dive in! We’re going to start with the problem, then show why modeling async closures as “closures that return futures” would require some deep lifetime magic, and finally circle back to how async transformers can make all this “just work” in a surprisingly natural way.

Sync closures

Closures are omnipresent in combinator style APIs in Rust. For the purposes of this post, let’s dive into a really simple closure function, call_twice_sync:

fn call_twice_sync(mut op: impl FnMut(&str)) { op("Hello"); op("Rustaceans"); }

As the name suggests, call_twice_sync invokes its argument twice. You might call it from synchronous code like so:

let mut buf = String::new(); call_twice_sync(|s| buf.push_str(s));

As you might expect, after this code executes, buf will have the value "HelloRustaceans". (Playground link, if you’re curious to try it out.)

Async closures as closures that return futures

Suppose we want to allow the closure to do async operations, though. That won’t work with call_twice_sync because the closure is a synchronous function:

let mut buf = String::new(); call_twice_sync(|s| s.push_str(receive_message().await)); // ----- ERROR

Given that an async function is just a sync function that returns a future, perhaps we can model an async clousure as a sync closure that returns a future? Let’s try it.

fn call_twice_async<F>(op: impl FnMut(&str) -> F) where F: Future<Output = ()>, { op("Hello").await; op("Rustaceans").await; }

This compiles. So far so good. Now let’s try using it. For now we won’t even use an await, just the same sync code we tried before:

// Hint: won't compile async fn use_it() { let mut buf = String::new(); call_twice_async(|s| async { buf.push_str(s); }); // ----- Return a future }

Wait, what’s this? Lo and behold, we get an error, and a kind of intimidating one:

error: captured variable cannot escape `FnMut` closure body --> src/lib.rs:13:26 | 12 | let mut buf = String::new(); | ------- variable defined here 13 | call_twice_async(|s| async { buf.push_str(s); }); | - ^^^^^^^^---^^^^^^^^^^^^^^^ | | | | | | | variable captured here | | returns an `async` block that contains a reference to a captured variable, which then escapes the closure body | inferred to be a `FnMut` closure | = note: `FnMut` closures only have access to their captured variables while they are executing... = note: ...therefore, they cannot allow references to captured variables to escape

So what is this all about? The last two lines actually tell you, but to really see it you have to do a bit of desugaring.

Futures capture the data they will use

The closure tries to construct a future with an async block. This async block is going to capture a reference to all the variables it needs: in this case, s and buf. So the closure will become something like:

|s| MyAsyncBlockType { buf, s }

where MyAsyncBlockType implements Future:

struct MyAsyncBlockType<'b> { buf: &'b mut String, s: &'b str, } impl Future for MyAsyncBlockType<'_> { type Output = (); fn poll(..) { ... } }

The key point here is that the closure is returning a struct (MyAsyncBlockType) and this struct is holding on to a reference to both buf and s so that it can use them when it is awaited.

Closure signature promises to be finished

The problem is that the FnMut closure signature actually promises something different than what the body does. The signature says that it takes an &str – this means that the closure is allowed to use the string while it executes, but it cannot hold on to a reference to the string and use it later. The same is true for buf, which will be accessible through the implicit self argument of the closure. But when the closure return the future, it is trying to create references to buf and s that outlive the closure itself! This is why the error message says:

= note: `FnMut` closures only have access to their captured variables while they are executing... = note: ...therefore, they cannot allow references to captured variables to escape

This is a problem!

Add some lifetime arguments?

So maybe we can declare the fact that we hold on to the data? It turns out you almost can, but not quite, and making an async closure be “just” a sync closure that returns a future would require some rather fundamental extensions to Rust’s trait system. There are two variables to consider, buf and s. Let’s begin with the argument s.

An aside: impl Trait capture rules

Before we dive more deeply into the closure case, let’s back up and imagine a top-level function that returns a future:

fn push_buf(buf: &mut String, s: &str) -> impl Future<Output = ()> { async move { buf.push_str(s); } }

If you try to compile this code, you’ll find that it does not build (playground):

error[E0700]: hidden type for `impl Future<Output = ()>` captures lifetime that does not appear in bounds --> src/lib.rs:4:5 | 3 | fn push_buf(buf: &mut String, s: &str) -> impl Future<Output = ()> { | ----------- hidden type `[async block@src/lib.rs:4:5: 6:6]` captures the anonymous lifetime defined here 4 | / async move { 5 | | buf.push_str(s); 6 | | } | |_____^ | help: to declare that `impl Future<Output = ()>` captures `'_`, you can introduce a named lifetime parameter `'a` | 3 | fn push_buf<'a>(buf: &'a mut String, s: &'a str) -> impl Future<Output = ()> + 'a { | ++++ ++ ++ ++++

impl Trait values can only capture borrowed data if they explicitly name the lifetime. This is why the suggested fix is to use a named lifetime 'a for buf and s and declare that the Future captures it:

fn push_buf<'a>(buf: &'a mut String, s: &'a str) -> impl Future<Output = ()> + 'a

If you desugar this return position impl trait into an explicit type alias impl trait, you can see the captures more clearly, as they become parameters to the type. The original (no captures) would be:

type PushBuf = impl Future<Output = ()>; fn push_buf<'a>(buf: &'a mut String, s: &'a str) -> PushBuf

and the fixed version would be:

type PushBuf<'a> = impl Future<Output = ()> + 'a fn push_buf<'a>(buf: &'a mut String, s: &'a str) -> PushBuf<'a> From functions to closures

OK, so we just saw how we can define a function that returns an impl Future, how that future will wind up capturing the arguments, and how that is made explicit in the return type by references to a named lifetime 'a. We could do something similar for closures, although Rust’s rather limited support for explicit closure syntax makes it awkward. I’ll use the unimplemented syntax from RFC 3216, you can see the workaround on the playground if that’s your thing:

type PushBuf<'a> = impl Future<Output = ()> + 'a async fn test() { let mut c = for<'a> |buf: &'a mut String, s: &'a str| -> PushBuf<'a> { async move { buf.push_str(s) } }); let mut buf = String::new(); c(&mut buf, "foo").await; }

(Side note that this is an interesting case for the “currently under debate” rules around defining type alias impl trait.)

Now for the HAMMER

OK, so far so grody, but we’ve shown that indeed you could define a closure that returns a future and it seems like things would work. But now comes the problem. Let’s take a look at the call_twice_async function – i.e., instead of looking at where the closure is defined, we look at the function that takes the closure as argument. That’s where things get tricky.

Here is call_twice_async, but with the anonymous lifetime given an explicit name 'a:

fn call_twice_async<F>(op: impl for<'a> FnMut(&str) -> F) where F: Future<Output = ()>,

Now the problem is this: we need to declare that the future which is returned (F) might capture 'a. But F is declared in an outer scope, and it can’t name 'a. In other words, right now, the return type F of the closure op must be the same each time the closure is called, but to get the semantics we want, we need the return type to include a different value for 'a each time.

If Rust had higher-kinded types (HKT), you could do something a bit wild, like this…

fn call_twice_async<F<'_>>(op: impl for<'a> FnMut(&'a str) -> F<'a>) // ----- HKT where for<'a> F<'a>: Future<Output = ()>,

but, of course, we don’t have HKT (and, cool as they are, I don’t think that’s a good fit for Rust right now, it would bust our complexity barrier in my opinion and then some without near enough payoff).

Short of adding HKT or some equivalent, I believe the option workaround is to use a dyn type:

fn call_twice_async(op: impl for<'a> FnMut(&'a str) -> Box<dyn Future<Output = ()> + 'a>)

This works today (and it is, for example, what moro does to resolve exactly this problem). Of course that means that the closure has to allocate a box, instead of just returning an async move. That’s a non-starter.

So we’re kind of stuck. As far as I can tell, modeling async closures as “normal closures that happen to return futures” requires one of two unappealing options

• extend the language with HKT, or possibly some syntactic sugar that ultimately however desugars to HKT
• use Box<dyn> everywhere, giving up on zero cost futures, embedded use cases, etc.

More traits, less problems

But wait, there is another way. Instead of modeling async closures using the normal Fn traits, we could define some async closure traits. To keep our life simple, let’s just look at one, for FnMut:

trait AsyncFnMut<A> { type Output; async fn call(&mut self, args: A) -> Self::Output; }

This is identical to the [sync FnMut] trait, except that call is an async fn. But that’s a pretty important difference. If we desugar the async fn to one using impl Trait, and then to GATs, we can start to see why:

trait AsyncFnMut<A> { type Output; type Call<'a>: Future<Output = Self::Output> + 'a; fn call(&mut self, args: A) -> Self::Call<'_>; }

Notice the Generic Associated Type (GAT) Call. GATs are basically the Rusty way to do HKTs (if you want to go deeper, I wrote a comparison series which may help; back then we called them associated type constructors, not GATs). Essentially what has happened here is that we moved the “HKT” into the trait definition itself, instead of forcing the caller to have it.

Given this definition, when we try to write the “call twice async” function, things work out more smoothly:

async fn call_twice_async<F>(mut op: impl AsyncFnMut(&str)) { op.call("Hello").await; op.call("World").await; }

Try it out on the playground, though note that we don’t actually support the () sugar for arbitrary traits, so I wrote impl for<'a> AsyncFnMut<&'a str, Output = ()> instead.

Connection to trait transformers

The translation between the normal FnMut trait and the AsyncFnMut trait was pretty automatic. The only thing we did was change the “call” function to async. So what if we had an async trait transformer, as was discussed earlier? Then we only have one “maybe async” trait, FnMut:

#[maybe(async)] trait FnMut<A> { type Output; #[maybe(async)] fn call(&mut self, args: A) -> Self::Output; }

Now we can write call_twice either sync or async, as we like, and the code is virtually identical. The only difference is that I write impl FnMut for sync or impl async FnMut for async:

fn call_twice_sync<F>(mut op: impl FnMut(&str)) { op.call("Hello"); op.call("World"); } async fn call_twice_async<F>(mut op: impl async FnMut(&str)) { op.call("Hello").await; op.call("World").await; }

Of course, with a more general maybe-async design, we might just write this function once, but that’s separate concern. Right now I’m only concerned with the idea of authoring traits that can be used in two modes, but not necessarily with writing code that is generic over which mode is being used.

Final note: creating the closure in a maybe-async world

When calling call_twice, we could write |s| buf.push_str(s) or async |s| buf.push_str(s) to indicate which traits it implements, but we could also infer this from context. We already do similar inference to decide the type of s for example. In fact, we could have some blanket impls, so that every F: FnMut also implements F: async FnMut; I guess this is generally true for any trait.

Conclusion

My conclusions:

• Nothing in this discussion required or even suggested any changes to the underlying design of async fn in trait. Stabilizing the statically dispatched subset of async fn in trait should be forwards compatible with supporting async closures. :tada:
• The “higher-kinded-ness” of async closures has to go somewhere. In stabilizing GATs, in my view, we’ve committed to the path that it should go into the trait definition (vs HKT, which would push it to the use site). The standard “def vs use site” tradeoffs apply here, I think: def sites often feel simpler and easier to understand, but are less flexible. I think that’s fine.
• Async trait transformers feel like a great option here that makes async closures work just like you would expect.

As the value of being connected to the internet increases, the need to make internet access available and affordable to all citizens becomes ever more urgent.  But how many people actually have access to the internet? Statistics are often quoted about what percentage of the world has access to the internet but those numbers are inevitably fuzzy, relying on varying definitions of internet access and varying levels of reporting quality.  In order to understand the true extent of the internet, we need to start with the basics, the physical infrastructure that carries the internet around the world.

The first step in developing a solid foundation on which to understand the growth of the internet are terrestrial fibre optic networks, the high-capacity backbones that carry thousands of terabits of internet traffic every day across vast distances. Fibre optic networks are the “deep water ports” of the internet, offering orders of magnitude of greater communication capacity than any other access technology. Research has shown that just living close to a fibre optic network is positively correlated with higher employment levels.

But fibre networks come in all sorts of shapes and sizes, from underground to aerial, from a few fibre strands to hundreds, from old technology to new. We can’t map the full extent of the fibre optic network infrastructure underpinning the internet without a common framework for describing them. And that is where the Open Fibre Data Standard comes in.  It provides a common framework for describing terrestrial fibre optic infrastructure that can be used by operators, researchers, governments, and regulators.

Building on Open Data principles, Mozilla has partnered with the World Bank, the International Telecommunications Union, Mozilla Corporation, the Internet Society, Liquid Intelligent Technologies, CSquared, and Digital Council Africa to support the development of the Open Fibre Data Standard and to promote its adoption. Thanks to financial support from the World Bank, the Open Data Services Cooperative (ODSC), a consultancy with extensive experience and expertise in Open Data standard development, was contracted to provide technical assistance in the development of the standard.

The Open Fibre Data Standard has gone through extensive development and consultation.  Beginning with an Alpha and Beta release, it is now in release 0.2.  Mozilla staff will travel to Ghana, Kenya and India in March and April of 2023 to build technical capacity for operators and regulators in the implementation of the standard and to support participation in its further development.

The post Open Fibre Data Standard: Understanding the True Extent of the Internet appeared first on Open Policy & Advocacy.

Clutter isn’t just material stuff scattered about your floor and shelves. Clutter can consume us in digital form, too — from an overabundance of browser bookmarks and open tabs to navigating a world wide web that’s littered with junk. The right browser extension, however, can really help clean things up… 

Tranquility Reader

Want to sweep away all the mess around your website reading material, like images, ads, and links to other content you don’t care about? Tranquility Reader does just that. Hit the extension’s toolbar button and presto — everything but the words disappear. 

The extension offers a few other nice features as well, like the ability to save content offline for later reading, customizable font sizes and color, add annotations to saved pages, and more.

Turn Off the Lights for Firefox

Clear out everything on your desktop except the video you want to watch. With the flick of a button, Turn Off the Lights for Firefox fades out everything on screen except your video player. 

More than just a fancy light switch, the extension offers a bunch of great customization features, including… 

• Auto HD for YouTube
• Mouse-wheel volume control
• Atmosphere lighting or background imagery
• Lots of fun fade out/in visual effects

Unhook

Speaking of uncluttering your video experience, Unhook elegantly removes a bunch of distracting YouTube elements. 

Enjoy YouTube with more breathing room once Unhook removes all those “rabbit hole” temptations like related or recommended videos, trending content, homepage suggestions, user comments, and much more. Unhook features 20+ customization options. 

Minimal Theme for Twitter

Streamline your social media. Minimal Theme for Twitter offers a few new looks for a stripped down scrolling experience. 

Great features let you do things like… 

• Customize the dimensions of Twitter’s timeline and interface buttons 
• Remove promoted posts
• Hide unwanted elements like search bar, Tweet button, view counts & more
• “Writer” mode shows just the tweet composer, removing all other interface distractions

Link Gopher

Have you ever kept a web page open for days or maybe even weeks just because it contains a bunch of links you’ve been meaning to save and organize in some fashion, some day? Link Gopher can help. 

Simple but such a time saver — with a single click Link Gopher grabs all links from within a web page and sorts them in a new tab, so you can easily copy/paste anywhere for proper organization. Any duplicate links are automatically removed. 

OneTab

This is like an emergency escape hatch when you find yourself overwhelmed with open browser tabs. Click the toolbar button and OneTab grabs all of your open links and lists them on a single scrollable page. You’ll save a lot of CPU and memory once your previously open tabs go dormant on a listing page. 

OneTab is a simple and effective way of dealing with sudden tab overload. But if you need more robust tab management tools, you might be interested to check out these other great tab manager extensions. 

We hope some of these extensions help you achieve a less cluttered, more serene online experience. If you’d like to keep exploring, you can find thousands of extensions on addons.mozilla.org. 

YouTube wants you to experience YouTube in very prescribed ways. But with the right browser extension, you’re free to alter YouTube to taste. Change the way the site looks, behaves, and delivers your favorite videos. 

Enhancer for YouTube

With dozens of customization features, Enhancer for YouTube has the power to dramatically reorient the way you watch videos. 

While a bunch of customization options may seem overwhelming, Enhancer for YouTube actually makes it very simple to navigate its settings and select just your favorite features. You can even choose which of your preferred features will display in the extension’s easy access interface that appears just beneath the video player.

<figcaption class="wp-element-caption">Enhancer for YouTube offers easy access controls just beneath the video player.</figcaption>

Key features… 

• Customize video player size 
• Change YouTube’s look with a dark theme
• Volume booster
• Ad blocking (with ability to whitelist channels you OK for ads)
• Take quick screenshots of videos
• Change playback speed
• Set default video quality from low to high def
• Shortcut configuration

Return YouTube Dislike

Do you like the Dislike? YouTube removed the display that revealed the number of thumbs-down Dislikes a video has, but with Return YouTube Dislike you can bring back the brutal truth. 

“Does exactly what the name suggests. Can’t see myself without this extension. Seriously, bad move on YouTube for removing such a vital tool.”

Firefox user OFG

“i have never smashed 5 stars faster.”

Firefox user 12918016 YouTube High Definition

Though its primary function is to automatically play all YouTube videos in their highest possible resolution, YouTube High Definition has a few other fine features to offer. 

In addition to automatic HD, YouTube High Definition can…

• Customize video player size
• HD support for clips embedded on external sites
• Specify your ideal resolution (4k – 144p)
• Set a preferred volume level 
• Also automatically plays the highest quality audio

YouTube NonStop

So simple. So awesome. YouTube NonStop remedies the headache of interrupting your music with that awful “Video paused. Continue watching?” message. 

Works on YouTube and YouTube Music. You’re now free to navigate away from your YouTube tab for as long as you like and not fret that the rock will stop rolling. 

YouTube Audio

Another simple but great extension for music fans, YouTube Audio disables the video broadcast and just streams audio to save you a ton of bandwidth. 

This is an essential extension if you have limited internet bandwidth and only want the music anyway. Click YouTube Audio’s toolbar button to mute the video stream anytime you like. Also helps preserve battery life. 

Unhook: Remove YouTube Recommended Videos & Comments

Instant serenity for YouTube! Unhook lets you strip away unwanted distractions like the promotional sidebar, endscreen suggestions, trending tab, and much more. 

More than two dozen customization options make this an essential extension for anyone seeking escape from YouTube rabbit holes. You can even hide notifications and live chat boxes. 

“This is the best extension to control YouTube usage, and not let YouTube control you.”

Firefox user Shubham Mandiya PocketTube

If you subscribe to a lot of YouTube channels PocketTube is a fantastic way to organize all your subscriptions by themed collections. 

Group your channel collections by subject, like “Sports,” “Cooking,” “Cat videos” or whatever. Other key features include…

• Add custom icons to easily identify your channel collections
• Customize your feed so you just see videos you haven’t watched yet, prioritize videos from certain channels, plus other content settings
• Integrates seamlessly with YouTube homepage 
• Sync collections across Firefox/Android/iOS using Google Drive and Chrome Profiler

<figcaption class="wp-element-caption">PocketTube keeps your channel collections neatly tucked away to the side. </figcaption> AdBlocker for YouTube

It’s not just you who’s noticed a lot more ads lately. Regain control with AdBlocker for YouTube. 

The extension very simply and effectively removes both video and display ads from YouTube. Period. Enjoy a faster, more focused YouTube. 

SponsorBlock

It’s a terrible experience when you’re enjoying a video or music on YouTube and you’re suddenly interrupted by a blaring ad. SponsorBlock solves this problem in a highly effective and original way. 

Leveraging the power of crowd sourced information to locate where—precisely— interruptive sponsored segments appear in videos, SponsorBlock learns where to automatically skip sponsored segments with its ever growing database of videos. You can also participate in the project by reporting sponsored segments whenever you encounter them (it’s easy to report right there on the video page with the extension). 

SponsorBlock can also learn to skip non-music portions of music videos and intros/outros, as well. If you’d like a deeper dive of SponsorBlock we profiled its developer and open source project on Mozilla Distilled. 

We hope one of these extensions enhances the way you enjoy YouTube. Feel free to explore more great media extensions on addons.mozilla.org.