When the certificate for Mozilla Add-ons went expired, the infrastructure disabled Firefox add-ons for millions of users.
And at the same time, the issue also prevented those affected users from re-activating or re-installing the browser extensions, essentially making Firefox 'extension-less'.
While the issue didn't affect all Firefox users, but the result was enough to have affected all versions of Firefox, old and new, and Stable and Nightly. The issue also impacts the Tor Browser, which supports Firefox add-ons.
As expected, the issue quickly triggered complaints by users and developers on the web, where they all expressed their concerns on Twitter, Reddit and others.
In response, Mozilla, the the not-for-profit organization behind the Firefox browser, acknowledged the issue, and at the time said that: "We're sorry that there is currently an issue where existing and new add-ons are failing to run or be installed on Firefox."
"We know what the issue is and are working hard to restore add-on functionality to Firefox as soon as possible."
"We'll continue to provide updates via our Twitter channels. Please bear with us while we get the problem fixed," the browser maker said.
The problem happened because all Firefox add-ons are digitally signed since 2016, when the organization released Firefox 48.
This mechanism was introduced to prevent bad actors from spreading and distributing malware through Firefox add-ons. Essentially, the strategy centralizes all add-ons management operations by trying all extensions to Mozilla's very own infrastructure.
But what happened here was that, the certificate that signed those add-ons expired. As a result, add-ons that were signed after 2016, were disabled.
The easy way to counter this issue, would be to visit the about:config section on Firefox, and set the xpinstall.signatures.required to false.
This would disable the extension signing mechanism in Firefox, the system through which the browser verifies that the installed extension is one that's been installed from the Mozilla Add-ons repository, the place where the signing certificate had expired.
Another possible way to resolve it, as recommended by many Firefox users, would be to turn system clocks before Mozilla's certificate expired, which was on May 4, 2019, 12:00AM UTC. But this solution was not advised as it may prevent other local apps that depend on accurate system clock from running properly.
Firefox quickly rolled out a hotfix to re-enable affected add-ons, saying that the fix would be applied in the background.
According to Firefox CTO Eric Rescorla in a blog post:
"Second, we need a mechanism to be able to quickly push updates to our users even when -- especially when -- everything else is down."
"Finally, we'll be looking more generally at our add-on security architecture to make sure that it's enforcing the right security properties at the least risk of breakage."
Rescorla added that the browser maker has considered using a Firefox point release to change the date used to validate the expired certificate. It also has considered generating a replacement certificate that was valid as it worked out how to push it to existing Firefox users.
And responding to criticism on how long it took Mozilla to push a fix, Rescorla said the response was "quite good", saying that the Firefox team did "amazing work" in creating a fix in less than 12 hours from it first being reported.
"Second, developing the system add-on takes some time. It's conceptually very simple, but even simple programs require taking some care, and we really wanted to make sure we didn't make things worse."
"As someone who sat in the meeting where it happened, I can say that people were working incredibly hard in a tough situation and that very little time was wasted."