mozilla

Mozilla Nederland LogoDe Nederlandse
Mozilla-gemeenschap

Support.Mozilla.Org: What’s Up with SUMO – 1st December

Mozilla planet - do, 01/12/2016 - 20:20

Greetings, SUMO Nation!

This is it! The start of the last month of the year, friends :-) We are kicking off the 12th month of 2016 with some news for your reading pleasure. Dig in!

Welcome, new contributors!

If you just joined us, don’t hesitate – come over and say “hi” in the forums!

Contributors of the week

We salute all of you!

Don’t forget that if you are new to SUMO and someone helped you get started in a nice way you can nominate them for the Buddy of the Month! SUMO Community meetings Community Platform Social
  • Reminder: Army of Awesome (as a “community trademark”) is going away. Please reach out to the Social Support team or ask in #sumo for more information.
  • Remember, you can contact Sierra (sreed@), Elisabeth (ehull@), or Rachel (guigs@) to get started with Social support. Help us provide friendly help through the likes of @firefox, @firefox_es, @firefox_fr and @firefoxbrasil on Twitter and beyond :-)
Support Forum
  • If you see Firefox users asking about the “zero day exploit”, please let them know that “We have been made aware of the issue and are working on a fix.  We will have more to say once the fix has been shipped.” (some media context here)
  • A polite reminder: please do not delete posts without an explanation to the poster in the forums, it can be frustrating to both the author and the user in the thread – thank you!
Knowledge Base & L10n Firefox
  • for iOS
    • Firefox for iOS 6.0 coming up your way before the end of this year, we hear! :-)

…and that’s it for today! So, what are you usually looking forward to about December? For me it would be the snow (not the case any more, sadly) and a few special dishes… Well, OK, also the fun party on the last night of the year in our calendar! Let us hear from you about your December picks – tell us in the forums or in the comments!

Categorieën: Mozilla-nl planet

Mitchell Baker: Julie Hanna Joins the Mozilla Corporation Board of Directors

Mozilla planet - do, 01/12/2016 - 20:18

This post was originally posted on the Mozilla.org website.

Julie Hanna, new Mozilla Corporation Board member

Julie Hanna, new Mozilla Corporation Board member

Today, we are very pleased to announce the latest addition to the Mozilla Corporation Board of Directors – Julie Hanna. Julie is the Executive Chairman for Kiva and a Presidential Ambassador for Global Entrepreneurship and we couldn’t be more excited to have her joining our Board.

Throughout this year, we have been focused on board development for both the Mozilla Foundation and the Mozilla Corporation boards of directors. We envisioned a diverse group who embodied the same values and mission that Mozilla stands for. We want each person to contribute a unique point of view. After extensive conversations, it was clear to the Mozilla Corporation leadership team that Julie brings exactly the type of perspective and approach that we seek.

Born in Egypt, Julie has lived in various countries including Jordan and Lebanon before finally immigrating to the United States. Julie graduated from the University of Alabama at Birmingham with a B.S. in Computer Science. She currently serves as Executive Chairman at Kiva, a peer-peer lending pioneer and the world’s largest crowdlending marketplace for underserved entrepreneurs. During her tenure, Kiva has scaled its reach to 190+ countries and facilitated nearly $1 billion dollars in loans to 2 million people with a 97% repayment rate. U.S. President Barack Obama appointed Julie as a Presidential Ambassador for Global Entrepreneurship to help develop the next generation of entrepreneurs. In that capacity, her signature initiative has delivered over $100M in capital to nearly 300,000 women and young entrepreneurs across 86 countries.

Julie is known as a serial entrepreneur with a focus on open source. She was a founder or founding executive at several innovative technology companies directly relevant to Mozilla’s world in browsers and open source. These include Scalix, a pioneering open source email/collaboration platform and developer of the most advanced AJAX application of its time, the first enterprise portal provider 2Bridge Software, and Portola Systems, which was acquired by Netscape Communications and become Netscape Mail.

She has also built a wealth of experience as an active investor and advisor to high-growth technology companies, including sharing economy pioneer Lyft, Lending Club and online retail innovator Bonobos. Julie also serves as an advisor to Idealab, Bill Gross’ highly regarded incubator which has launched dozens of IPO-destined companies.

Please join me in welcoming Julie Hanna to the Mozilla Board of Directors.

Mitchell

Background:

Twitter: @JulesHanna

High-res photo

 

Categorieën: Mozilla-nl planet

Julie Hanna Joins the Mozilla Corporation Board of Directors

Mitchell Baker - do, 01/12/2016 - 20:18

This post was originally posted on the Mozilla.org website.

Julie Hanna, new Mozilla Corporation Board member

Julie Hanna, new Mozilla Corporation Board member

Today, we are very pleased to announce the latest addition to the Mozilla Corporation Board of Directors – Julie Hanna. Julie is the Executive Chairman for Kiva and a Presidential Ambassador for Global Entrepreneurship and we couldn’t be more excited to have her joining our Board.

Throughout this year, we have been focused on board development for both the Mozilla Foundation and the Mozilla Corporation boards of directors. We envisioned a diverse group who embodied the same values and mission that Mozilla stands for. We want each person to contribute a unique point of view. After extensive conversations, it was clear to the Mozilla Corporation leadership team that Julie brings exactly the type of perspective and approach that we seek.

Born in Egypt, Julie has lived in various countries including Jordan and Lebanon before finally immigrating to the United States. Julie graduated from the University of Alabama at Birmingham with a B.S. in Computer Science. She currently serves as Executive Chairman at Kiva, a peer-peer lending pioneer and the world’s largest crowdlending marketplace for underserved entrepreneurs. During her tenure, Kiva has scaled its reach to 190+ countries and facilitated nearly $1 billion dollars in loans to 2 million people with a 97% repayment rate. U.S. President Barack Obama appointed Julie as a Presidential Ambassador for Global Entrepreneurship to help develop the next generation of entrepreneurs. In that capacity, her signature initiative has delivered over $100M in capital to nearly 300,000 women and young entrepreneurs across 86 countries.

Julie is known as a serial entrepreneur with a focus on open source. She was a founder or founding executive at several innovative technology companies directly relevant to Mozilla’s world in browsers and open source. These include Scalix, a pioneering open source email/collaboration platform and developer of the most advanced AJAX application of its time, the first enterprise portal provider 2Bridge Software, and Portola Systems, which was acquired by Netscape Communications and become Netscape Mail.

She has also built a wealth of experience as an active investor and advisor to high-growth technology companies, including sharing economy pioneer Lyft, Lending Club and online retail innovator Bonobos. Julie also serves as an advisor to Idealab, Bill Gross’ highly regarded incubator which has launched dozens of IPO-destined companies.

Please join me in welcoming Julie Hanna to the Mozilla Board of Directors.

Mitchell

Background:

Twitter: @JulesHanna

High-res photo

 

Categorieën: Mozilla-nl planet

The Mozilla Blog: Julie Hanna Joins the Mozilla Corporation Board of Directors

Mozilla planet - do, 01/12/2016 - 20:17

Today, we are very pleased to announce the latest addition to the Mozilla Corporation Board of Directors – Julie Hanna. Julie is the Executive Chairman for Kiva and a Presidential Ambassador for Global Entrepreneurship and we couldn’t be more excited to have her joining our Board.

Throughout this year, we have been focused on board development for both the Mozilla Foundation and the Mozilla Corporation boards of directors. We envisioned a diverse group who embodied the same values and mission that Mozilla stands for. We want each person to contribute a unique point of view. After extensive conversations, it was clear to the Mozilla Corporation leadership team that Julie brings exactly the type of perspective and approach that we seek.

Born in Egypt, Julie has lived in various countries including Jordan and Lebanon before finally immigrating to the United States. Julie graduated from the University of Alabama at Birmingham with a B.S. in Computer Science. She currently serves as Executive Chairman at Kiva, a peer-peer lending pioneer and the world’s largest crowdlending marketplace for underserved entrepreneurs. During her tenure, Kiva has scaled its reach to 190+ countries and facilitated nearly $1 billion dollars in loans to 2 million people with a 97% repayment rate. U.S. President Barack Obama appointed Julie as a Presidential Ambassador for Global Entrepreneurship to help develop the next generation of entrepreneurs. In that capacity, her signature initiative has delivered over $100M in capital to nearly 300,000 women and young entrepreneurs across 86 countries.

Julie is known as a serial entrepreneur with a focus on open source. She was a founder or founding executive at several innovative technology companies directly relevant to Mozilla’s world in browsers and open source. These include Scalix, a pioneering open source email/collaboration platform and developer of the most advanced AJAX application of its time, the first enterprise portal provider 2Bridge Software, and Portola Systems, which was acquired by Netscape Communications and become Netscape Mail.

She has also built a wealth of experience as an active investor and advisor to high-growth technology companies, including sharing economy pioneer Lyft, Lending Club and online retail innovator Bonobos. Julie also serves as an advisor to Idealab, Bill Gross’ highly regarded incubator which has launched dozens of IPO-destined companies.

Please join me in welcoming Julie Hanna to the Mozilla Board of Directors.

Mitchell

You can read Julie’s message about why she’s joining Mozilla here.

Background:

Twitter: @JulesHanna

High-res photo (photo credit: Chris Michel)

Categorieën: Mozilla-nl planet

Julie Hanna Joins the Mozilla Corporation Board of Directors

Mozilla Blog - do, 01/12/2016 - 20:17

Today, we are very pleased to announce the latest addition to the Mozilla Corporation Board of Directors – Julie Hanna. Julie is the Executive Chairman for Kiva and a Presidential Ambassador for Global Entrepreneurship and we couldn’t be more excited to have her joining our Board.

Throughout this year, we have been focused on board development for both the Mozilla Foundation and the Mozilla Corporation boards of directors. We envisioned a diverse group who embodied the same values and mission that Mozilla stands for. We want each person to contribute a unique point of view. After extensive conversations, it was clear to the Mozilla Corporation leadership team that Julie brings exactly the type of perspective and approach that we seek.

Born in Egypt, Julie has lived in various countries including Jordan and Lebanon before finally immigrating to the United States. Julie graduated from the University of Alabama at Birmingham with a B.S. in Computer Science. She currently serves as Executive Chairman at Kiva, a peer-peer lending pioneer and the world’s largest crowdlending marketplace for underserved entrepreneurs. During her tenure, Kiva has scaled its reach to 190+ countries and facilitated nearly $1 billion dollars in loans to 2 million people with a 97% repayment rate. U.S. President Barack Obama appointed Julie as a Presidential Ambassador for Global Entrepreneurship to help develop the next generation of entrepreneurs. In that capacity, her signature initiative has delivered over $100M in capital to nearly 300,000 women and young entrepreneurs across 86 countries.

Julie is known as a serial entrepreneur with a focus on open source. She was a founder or founding executive at several innovative technology companies directly relevant to Mozilla’s world in browsers and open source. These include Scalix, a pioneering open source email/collaboration platform and developer of the most advanced AJAX application of its time, the first enterprise portal provider 2Bridge Software, and Portola Systems, which was acquired by Netscape Communications and become Netscape Mail.

She has also built a wealth of experience as an active investor and advisor to high-growth technology companies, including sharing economy pioneer Lyft, Lending Club and online retail innovator Bonobos. Julie also serves as an advisor to Idealab, Bill Gross’ highly regarded incubator which has launched dozens of IPO-destined companies.

Please join me in welcoming Julie Hanna to the Mozilla Board of Directors.

Mitchell

You can read Julie’s message about why she’s joining Mozilla here.

Background:

Twitter: @JulesHanna

High-res photo

Categorieën: Mozilla-nl planet

Air Mozilla: Connected Devices Weekly Program Update, 01 Dec 2016

Mozilla planet - do, 01/12/2016 - 19:45

Connected Devices Weekly Program Update Weekly project updates from the Mozilla Connected Devices team.

Categorieën: Mozilla-nl planet

Mozilla Patches Firefox Zero Day Used to Unmask Tor Browser Users - Threatpost

Nieuws verzameld via Google - do, 01/12/2016 - 18:46

Threatpost

Mozilla Patches Firefox Zero Day Used to Unmask Tor Browser Users
Threatpost
According to Daniel Veditz, who leads Mozilla's security team, Firefox users should have their browsers automatically updated at some point over the next 24 hours. If they'd rather not wait, users can download the updated versions – Firefox 50.0.2 ...
Mozilla and Tor release urgent update for Firefox 0-day under active attackArs Technica
Mozilla and Tor issue patches for Firefox flaw exposing Tor usersComputerWeekly.com
Mozilla and Tor Warn of Critical Firefox Vulnerability, Urge Users to UpdateMac Rumors
ZDNet -BleepingComputer -Inquirer
alle 46 nieuwsartikelen »
Categorieën: Mozilla-nl planet

The Mozilla Blog: State of Mozilla 2015 Annual Report

Mozilla planet - do, 01/12/2016 - 17:40

We just released our State of Mozilla annual report for 2015. This report highlights key activities for Mozilla in 2015 and includes detailed financial documents.

Mozilla is not your average company. We’re a different kind of organization – a nonprofit, global community with a mission to ensure that the internet is a global public resource, open and accessible to all.

I hope you enjoy reading and learning more about Mozilla and our developments in products, web technologies, policy, advocacy and internet health.

 

 

Categorieën: Mozilla-nl planet

State of Mozilla 2015 Annual Report

Mozilla Blog - do, 01/12/2016 - 17:40

We just released our State of Mozilla annual report for 2015. This report highlights key activities for Mozilla in 2015 and includes detailed financial documents.

Mozilla is not your average company. We’re a different kind of organization – a nonprofit, global community with a mission to ensure that the internet is a global public resource, open and accessible to all.

I hope you enjoy reading and learning more about Mozilla and our developments in products, web technologies, policy, advocacy and internet health.

 

 

Categorieën: Mozilla-nl planet

Air Mozilla: Webinar 2: What is Equal Rating.

Mozilla planet - do, 01/12/2016 - 17:16

 What is Equal Rating. Overview of Equal Rating

Categorieën: Mozilla-nl planet

Mozilla Open Innovation Team: The Problem with Privacy in IoT

Mozilla planet - do, 01/12/2016 - 17:11

Every year Mozilla hosts DinoTank, an internal pitch platform, and this year instead of pitching ideas we focused on pitching problems. To give each DinoTank winner the best possible start, we set up a design sprint for each one. This is the first installment of that series of DinoTank sprints…

The Problem

I work on the Internet of Things at Mozilla but I am apprehensive about bringing most smart home products into my house. I don’t want a microphone that is always listening to me. I don’t want an internet connected camera that could be used to spy on me. I don’t want my thermostat, door locks, and light bulbs all collecting unknown amounts of information about my daily behavior. Suffice it to say that I have a vague sense of dread about all the new types of data being collected and transmitted from inside my home. So I pitched this problem to the judges at DinoTank. It turns out they saw this problem as important and relevant to Mozilla’s mission. And so to explore further, we ran a 5 day product design sprint with the help of several field experts.

Brainstorming

A team of 8 staff members was gathered in San Francisco for a week of problem refinement, insight gathering, brainstorming, prototyping, and user testing. Among us we had experts in Design Thinking, user research, marketing, business development, engineering, product, user experience, and design. The diversity of skillsets and backgrounds allowed us to approach the problem from multiple different angles, and through our discussion several important questions arose which we would seek to answer by building prototypes and putting them in front of potential consumers.

The Solution

After 3 days of exploring the problem, brainstorming ideas and then them narrowing down, we settled on a single product solution. It would be a small physical device that plugs into the home’s router to monitor the network activity of local smart devices. It would have a control panel that could be accessed from a web browser. It would allow the user to keep up to date through periodic status emails, and only in critical situations would it notify the user’s phone with needed actions. We mocked up an end-to-end experience using clickable and paper prototypes, and put it in front of privacy aware IoT home owners.

What We Learned

Surprisingly, our test users saw the product as more of an all inclusive internet security system rather than a IoT only solution. One of our solutions focused more on ‘data protection’ and here we clearly learned that there is a sense of resignation towards large data collection, with comments like “Google already has all my data anyway.”

Of the positive learnings, the mobile notifications really resonated with users. And interestingly — though not surprisingly — people became much more interested in the privacy aspects of our mock-ups when their children were involved in the conversation.

Next Steps

The big question we were left with was: is this a latent but growing problem, or was this never a problem at all? To answer this, we will tweak our prototypes to test different market positioning of the product as well as explore potential audiences that have a larger interest in data privacy.

My Reflections

Now, if I had done this project without DinoTank’s help, I probably would have started by grabbing a Raspberry Pi and writing some code. But instead I learned how to take a step back and start by focusing on people. Here I learned about evaluating a user problem, sussing out a potential solution, and testing its usefulness in front of users. And so regardless of what direction we now take, we didn’t waste any time because we learned about a problem and the people whom we could reach.

If you’re looking for more details about my design sprint you can find the full results in our report. If you would like to learn more about the other DinoTank design sprints, check out the tumblr. And if you are interested in learning more about the methodologies we are using, check out our Open Innovation Toolkit.

People I’d like to thank:

Katharina Borchert, Bertrand Neveux, Christopher Arnold, Susan Chen, Liz Hunt, Francis Djabri, David Bialer, Kunal Agarwal, Fabrice Desré, Annelise Shonnard, Janis Greenspan, Jeremy Merle and Rina Jensen.

The Problem with Privacy in IoT was originally published in Mozilla Open Innovation on Medium, where people are continuing the conversation by highlighting and responding to this story.

Categorieën: Mozilla-nl planet

​Mozilla puts new money to use fighting for 'internet health' - CNET

Nieuws verzameld via Google - do, 01/12/2016 - 17:06

CNET

Mozilla puts new money to use fighting for 'internet health'
CNET
If you've heard of Mozilla, it's probably because you've used its Firefox browser. Now the nonprofit wants to expand out of technology into politics. Mozilla is marshaling public support for political positions, like backing net neutrality, defending ...
Mozilla, With Revenue Up 28%, Maps An Ambitious AgendaForbes

alle 2 nieuwsartikelen »
Categorieën: Mozilla-nl planet

Myk Melez: Mozilla and Node.js

Mozilla planet - do, 01/12/2016 - 17:03

Recently the Node.js Foundation announced that Mozilla is joining forces with IBM, Intel, Microsoft, and NodeSource on the Node.js API. So what’s Mozilla doing with Node? Actually, a few things…

You may already know about SpiderNode, a Node.js implementation on SpiderMonkey, which Ehsan Akhgari announced in April. Ehsan, Trevor Saunders, Brendan Dahl, and other contributors have since made a bunch of progress on it, and it now builds successfully on Mac and Linux and runs some Node.js programs.

Brendan additionally did the heavy lifting to build SpiderNode as a static library, link it with Positron, and integrate it with Positron’s main process, improving that framework’s support for running Electron apps. He’s now looking at opportunities to expose SpiderNode to WebExtensions and to chrome code in Firefox.

Meanwhile, I’ve been analyzing the Node.js API being developed by the API Working Group, and I’ve also been considering opportunities to productize SpiderNode for Node developers who want to use emerging JavaScript features in SpiderMonkey, such as WebAssembly and Shared Memory.

If you’re a WebExtension developer or Firefox engineer, would you use Node APIs if they were available to you? If you’re a Node programmer, would you use a Node implementation running on SpiderMonkey? And if so, would you require Node.js Addons (i.e. native modules) to do so?

Categorieën: Mozilla-nl planet

Mozilla, With Revenue Up 28%, Maps An Ambitious Agenda - Forbes

Nieuws verzameld via Google - do, 01/12/2016 - 17:01

CNET

Mozilla, With Revenue Up 28%, Maps An Ambitious Agenda
Forbes
One of Silicon Valley's most idiosyncratic players — the Mozilla Foundation — just reported a 28% leap in revenue for 2015, to $421 million, with cash flow more than tripling, to about $80 million. Further increases appear likely this year. Those ...
Mozilla puts new money to use fighting for 'internet health'CNET

alle 2 nieuwsartikelen »
Categorieën: Mozilla-nl planet

Air Mozilla: Reps Weekly Meeting Dec. 01, 2016

Mozilla planet - do, 01/12/2016 - 17:00

Reps Weekly Meeting Dec. 01, 2016 This is a weekly call with some of the Reps to discuss all matters about/affecting Reps and invite Reps to share their work with everyone.

Categorieën: Mozilla-nl planet

Patch komt uit voor Firefox-lek dat Tor-gebruikers kan ontmaskeren - Tweakers

Nieuws verzameld via Google - do, 01/12/2016 - 15:50

NU.nl

Patch komt uit voor Firefox-lek dat Tor-gebruikers kan ontmaskeren
Tweakers
Mozilla heeft een patch uitgebracht voor het Firefox-lek dat gebruikt kan worden om Tor-gebruikers te ontmaskeren. Het team achter de Tor-browser heeft de patch in zijn browser overgenomen en een nieuwe versie uitgebracht. Tor Mozilla laat weten dat ...
Noodpatch voor aangevallen lek in Firefox en Tor BrowserSecurity.nl

alle 8 nieuwsartikelen »
Categorieën: Mozilla-nl planet

FBI-Hacker involviert?: Mozilla stopft gefährliche Firefox-Lücke - n-tv.de NACHRICHTEN

Nieuws verzameld via Google - do, 01/12/2016 - 14:22

n-tv.de NACHRICHTEN

FBI-Hacker involviert?: Mozilla stopft gefährliche Firefox-Lücke
n-tv.de NACHRICHTEN
Im weit verbreiteten Browser Firefox klafft eine Sicherheitslücke, die es Hackern ermöglicht, über präparierte Webseiten Schadcode auf Windows-Rechner einzuschleusen. Mozilla hat schnell nach Bekanntwerden der Schwachstelle ein Update veröffentlicht, ...
Firefox 50.0.2: Mozilla behebt 0-Day-Sicherheitslückesoeren-hentzschel.at
Firefox 50.0.2: Mozilla stopft kritische Lücke - Update sofort installierenCHIP Online
Firefox: Warum Sie den Mozilla-Browser sofort updaten solltenSTERN
PC-Welt -inFranken.de -t-online.de
alle 61 nieuwsartikelen »
Categorieën: Mozilla-nl planet

Daniel Pocock: Using a fully free OS for devices in the home

Mozilla planet - do, 01/12/2016 - 14:11

There are more and more devices around the home (and in many small offices) running a GNU/Linux-based firmware. Consider routers, entry-level NAS appliances, smart phones and home entertainment boxes.

More and more people are coming to realize that there is a lack of security updates for these devices and a big risk that the proprietary parts of the code are either very badly engineered (if you don't plan to release your code, why code it properly?) or deliberately includes spyware that calls home to the vendor, ISP or other third parties. IoT botnet incidents, which are becoming more widely publicized, emphasize some of these risks.

On top of this is the frustration of trying to become familiar with numerous different web interfaces (for your own devices and those of any friends and family members you give assistance to) and the fact that many of these devices have very limited feature sets.

Many people hail OpenWRT as an example of a free alternative (for routers), but I recently discovered that OpenWRT's web interface won't let me enable both DHCP and DHCPv6 concurrently. The underlying OS and utilities fully support dual stack, but the UI designers haven't encountered that configuration before. Conclusion: move to a device running a full OS, probably Debian-based, but I would consider BSD-based solutions too.

For many people, the benefit of this strategy is simple: use the same skills across all the different devices, at home and in a professional capacity. Get rapid access to security updates. Install extra packages or enable extra features if really necessary. For example, I already use Shorewall and strongSwan on various Debian boxes and I find it more convenient to configure firewall zones using Shorewall syntax rather than OpenWRT's UI.

Which boxes to start with?

There are various considerations when going down this path:

  • Start with existing hardware, or buy new devices that are easier to re-flash? Sometimes there are other reasons to buy new hardware, for example, when upgrading a broadband connection to Gigabit or when an older NAS gets a noisy fan or struggles with SSD performance and in these cases, the decision about what to buy can be limited to those devices that are optimal for replacing the OS.
  • How will the device be supported? Can other non-technical users do troubleshooting? If mixing and matching components, how will faults be identified? If buying a purpose-built NAS box and the CPU board fails, will the vendor provide next day replacement, or could it be gone for a month? Is it better to use generic components that you can replace yourself?
  • Is a completely silent/fanless solution necessary?
  • Is it possibly to completely avoid embedded microcode and firmware?
  • How many other free software developers are using the same box, or will you be first?
Discussing these options

I recently started threads on the debian-user mailing list discussing options for routers and home NAS boxes. A range of interesting suggestions have already appeared, it would be great to see any other ideas that people have about these choices.

Categorieën: Mozilla-nl planet

Giorgos Logiotatidis: Taco Bell Parallel Programming

Mozilla planet - do, 01/12/2016 - 14:02

While working on migrating support.mozilla.org away from Kitsune (which is a great community support platform that needs love, remember that internet) I needed to convert about 4M database rows of a custom, Markdown inspired, format to HTML.

The challenge of the task is that it needs to happen as fast as possible so we can dump the database, convert the data and load the database onto the new platform with the minimum possible time between the first and the last step.

I started a fresh MySQL container and started hacking:

Load the database dump

Kitsune's database weights about 35GiB so creating and loading the dump is a lengthy procedure. I used some tricks taken from different places with most notable ones:

  • Set innodb_flush_log_at_trx_commit = 2 for more speed. This should not be used in production as it may break ACID compliance but for my use case it's fine.

  • Set innodb_write_io_threads = 16

  • Set innodb_buffer_pool_size=16G and innodb_log_file_size=4G. I read that the innodb_log_file_size is recommended to be 1/4th of innodb_buffer_pool_size and I set the later based on my available memory.

Loading the database dump takes about 60 minutes. I'm pretty sure there's room for improvement there.

Extra tip: When dumping such huge databases from production websites make sure to use a replica host and mysqldump's --single-transaction flag to avoid locking the database.

Create a place to store the processed data

Kitsune being a Django project I created extra fields named content_html in the Models with markdown content, generated the migrations and run them against the db.

Process the data

An AWS m4.2xl gives 8 cores at my disposal and 32GiB of memory, of which 16 I allocated to MySQL earlier.

I started with a basic single core solution::

for question in Question.objects.all(): question.content_html = parser.wiki_2_html(question.content) question.save()

which obviously does the job but it's super slow.

Transactions take a fair amount of time, what if we could bundle multiple saves into one transaction?

def chunks(count, nn=500): """Yield successive n-sized chunks from l.""" offset = 0 while True: yield (offset, min(offset+nn, count)) offset += nn if offset > count: break for low, high in chunks(Question.objects.count()): with transaction.atomic(): for question in Question.objects.all().limit[low:high]: question.content_html = parser.wiki_2_html(question.content) question.save()

This is getting better. Increasing the chunk size to 20000 items in the cost of more RAM used produces faster results. Anything above this value seems to require about the same time to complete.

Tried pypy and I didn't get better results so I defaulted to CPython.

Let's add some more cores into the mix using Python's multiprocessing library.

I created a Pool with 7 processes (always leave one core outside the Pool so the system remains responsive) and used apply_async to generate the commands to run by the Pool.

results = [] it = Question.objects.all() number_of_rows = it.count() pool = mp.Pool(processes=7) [pool.apply_async(process_chunk), (chunk,), callback=results.append) for chunk in chunks(it)] sum_results = 0 while sum_results < number_of_rows: print 'Progress: {}/{}'.format(sum_results, number_of_rows) sum_results = sum(results) sleep(1)

Function process_chunk will process, save and return the number of rows processed. Then apply_async will append this number to results which is used in the while loop to give me an overview of what's happening while I'm waiting.

So far so good, this is significantly faster. It took some tries before getting this right. Two things to remember when dealing with multiprocess and Django are:

  • ./manage.py shell won't work. I don't know why but I went ahead and wrote a standalone python script, imported django and run django.setup().

  • When a process forks, Django's database connection which was already created by that time, needs to be cleared out and get re-created for every process. First thing process_chunk does is db.connections.close_all(). Django will take care re-creating when needed.

OK I'm good to hit the road -I thought- and I launched the process with all the rows that needed parsing. As the time goes by I see the memory usage to increase dramatically and eventually the kernel would kill my process to free up memory.

It seems that the queries would take too much memory. I set the Pool to shutdown and start a new process on every new chunk with maxtasksperchild=1 which helped a bit but again, the farther in the process the more the memory usage. I tried to debug the issue with different Django queries and profiling (good luck with that on a multiprocess program) and I failed. Eventually I needed to figure out a solution before it's too late, so back to the drawing board.

Process the data, take two

I read this interesting blog post the other day named Taco Bell Programming where Ted is claiming that many times you can achieve the desired functionality just by rearranging the Unix tool set, much like Taco Bell is producing its menu by rearranging basic ingredients.

What you win with Taco Bell Programming is battle-tested tools and throughout documentation which should save you time from actual coding and time debugging problems already solved.

I took a step back and re-thought by problem. The single core solution was working just fine and had no memory issues. What if I could find a program to paralellize multiple runs? And that tool (obviously) exists, it's GNU Parallel.

In the true spirit of other GNU tools, Parallel has a gazillion command line arguments and can do a ton of things related to parallelizing the run of a list of commands.

I mention just the most important to me at the moment:

  1. Read from command line a list of commands
  2. Show progress and provide ETA
  3. Limit the run to a number of cores
  4. Retry failed jobs, resume runs and book overall keeping.
  5. Send jobs to other machines (I wish I had the time to test that, amazing)
Prepare the input to Parallel

I reverted to the original one core python script and refactored it a bit so I can call it using python -c. I also removed the chunk generation code since I'll do that elsewhere

def parse_to_html(it, from_field, to_field): with transaction.atomic(): for p in it: setattr(p, to_field, parser.wiki_to_html(getattr(p, from_field))) p.save()

Now to process all questions I can call this thing using

$ echo "import wikitohtml; it = Question.objects.all(); wikitohtml.parse_to_html(it, 'content', 'content_html')" | python -

Then I wrote a small python script to generate the chunks and print out commands to be later used by Parallel

CMD = '''echo "import wikitohtml; it = wikitohtml.Question.objects.filter(id__gte={from_value}, id__lt={to_value}); wikitohtml.parse_to_html(it, 'content', 'content_html')" | python - > /dev/null''' for i in range(0, 1200000, 10000): print CMD.format(from_value=i, to_value=i+step)

I wouldn't be surprised if Parallel can do the chunking itself but in this case it's easier for me to fine tune it using Python.

Now I can process all questions in parallel using

$ python generate-cmds.py | parallel -j 7 --eta --joblog joblog

So everything is working now in parallel and the memory leak is gone!

But I'm not done yet.

Deadlocks

I left the script running for half an hour and then I started seeing MySQL aborted transactions that failed to grab a lock. OK that's should be an easy fix by increasing the wait lock time with SET innodb_lock_wait_timeout = 5000; (up from 50). Later I added --retries 3 in Parallel to make sure that if anything failed it would get retried.

That actually made things worse as it introduced everyone's favorite issue in parallel programming, deadlocks. I reverted the MySQL change and looked deeper. Being unfamiliar with Kitsune's code I was not aware that the model.save() methods are doing a number of different things, including saving other objects as well, e.g. Answer.save() also calls Question.save().

Since I'm only processing one field and save the result into another field which is unrelated to everything else all the magic that happens in save() can be skipped. Besides dealing with the deadlock this can actually get us a speed increase for free.

I refactored the python code to use Django's update() which directly hits the database and does not go through save().

def parse_to_html(it, from_field, to_field, id_field='id'): with transaction.atomic(): for p in it: it.filter(**{id_field: getattr(p, id_field)}).update(**{to_field: parser.wiki_to_html(getattr(p, from_field))})

Everything works and indeed update() did increase things a lot and solved the deadlock issue. The cores are 100% utilized which means that throwing more CPU power into the problem would buy more speed. The processing of all 4 million rows takes now about 30 minutes, down from many many hours.

Magic!

Categorieën: Mozilla-nl planet

Firefox zero-day: Mozilla, Tor issue critical patches to block active attacks - ZDNet

Nieuws verzameld via Google - do, 01/12/2016 - 12:29

BleepingComputer

Firefox zero-day: Mozilla, Tor issue critical patches to block active attacks
ZDNet
The payload only works against Windows systems running Firefox and the Tor Browser, although the vulnerability also exists on the Firefox for macOS and Linux. Image: Mozilla. Mozilla and Tor have released patches for Firefox and the Firefox-based Tor ...
Mozilla and Tor release urgent update for Firefox 0-day under active attackArs Technica
Mozilla and Tor issue patches for Firefox flaw exposing Tor usersComputerWeekly.com
Mozilla and Tor Warn of Critical Firefox Vulnerability, Urge Users to UpdateMac Rumors
BleepingComputer -NDTV -Infosecurity Magazine
alle 37 nieuwsartikelen »
Categorieën: Mozilla-nl planet

Pagina's