Wondering why all of your Firefox add-ons broke last week? You’re not the only one, and Firefox’s chief technology officer, Eric Rescorla, has penned a blog post going into detail about the add-on-pocalypse explaining what happened and how Firefox will sidestep it in the future.
In the blog post, which you can read on Mozilla’s blog, Rescoria confirmed what Mozilla had already said: an expired security certificate was to blame for felling thousands of Firefox add-ons. If your add-ons broke, and if they are still broken, you’ve been affected by this issue.
“Recently, Firefox had an incident in which most add-ons stopped working,” said Rescoria. “This was due to an error on our end: we let one of the certificates used to sign add-ons expire which had the effect of disabling the vast majority of add-on.”
Related: Best VPN 2019
The disaster was that it the Mozilla team noticed it at 6pm Pacific Time on Friday evening. Most of the team were leaving the office for the weekend. The issue wasn’t widespread at this stage, because installs for add-ons are checked “about every 24 hours” although the exact time is different for each user.
With each add-on check, Firefox would initiate the check and quickly find that the relevant signing certificate has expired. Firefox would then kill all of the add-ons signed by that certificate. In this case, this led to literally thousands being killed.
That’s the how and the why. The team immediately got to work, and Rescorla said in the blog post that they shipped a fix at 2:44AM, “or after less than 9 hours, and then it took another 6-12 hours before most of our users had it.”
Throw in another few hours until Firefox performed its next check, and you can see why the issue took a little while to resolve. Rescorla claims this is “quite good from a standing start.” in terms of a fix, although this will likely not be too much of a comfort for the thousands of users who had all of their add-ons disabled.
Related: Best Free VPN 2019
While the issue is resolved for most people, some users with older versions of Firefox still haven’t received a fix. For these users, Mozilla says the only option is to upgrade to the latest version of the browser.
A formal post-mortem will appear next week, but in the meantime, Rescorla adds: “First, I want to say that the team here did amazing work: they built and shipped a fix in less than 12 hours from the initial report. As someone who sat in the meeting where it happened, I can say that people were working incredibly hard in a tough situation and that very little time was wasted.
With that said, obviously this isn’t an ideal situation and it shouldn’t have happened in the first place. We clearly need to adjust our processes both to make this and similar incidents it less likely to happen and to make them easier to fix.”
The blog post on Mozilla’s site dives really deep into the causes and processes of the issue, including the steps taken to fix it. It’s worth a read if you want to know more.
Editorial independence
Editorial independence means being able to give an unbiased verdict about a product or company, with the avoidance of conflicts of interest. To ensure this is possible, every member of the editorial staff follows a clear code of conduct.
Professional conduct
We also expect our journalists to follow clear ethical standards in their work. Our staff members must strive for honesty and accuracy in everything they do. We follow the IPSO Editors’ code of practice to underpin these standards.
Editorial independence
Editorial independence means being able to give an unbiased verdict about a product or company, with the avoidance of conflicts of interest. To ensure this is possible, every member of the editorial staff follows a clear code of conduct.
Professional conduct
We also expect our journalists to follow clear ethical standards in their work. Our staff members must strive for honesty and accuracy in everything they do. We follow the IPSO Editors’ code of practice to underpin these standards.
