On Thursday 29th September 2021 something happened to the security infrastructure of the Internet which has had far-reaching implications despite not being well publicised or well understood.
Like many organisations, Bookswarm uses Let’s Encrypt to generate digital certificates for all the websites we host. Let’s Encrypt is apparently used to secure over 45 million websites around the world.
Having an SSL certificate is important for security but also for SEO, as Google penalises sites without them. We have therefore added SSL to all our sites as standard for a number of years. When you see a padlock next to a website’s URL you know it has a valid SSL certificate.
Here’s a basic summary of what happened on Thursday, courtesy of TechCrunch:
One of the largest providers of HTTPS certificates, Let’s Encrypt, saw its root certificate expire this week — meaning you might need to upgrade your devices to prevent them from breaking.
Let’s Encrypt, a free-to-use nonprofit, issues certificates that encrypt the connections between your devices and the wider internet, ensuring that nobody can intercept and steal your data in transit. Millions of websites alone rely on Let’s Encrypt. But, as warned by security researcher Scott Helme, the root certificate that Let’s Encrypt currently uses — the IdentTrust DST Root CA X3 — was set to expire on September 30. After expiry, computers, devices and web clients — such as browsers — will no longer trust certificates that have been issued by this certificate authority.
For the overwhelming majority of website users, there is nothing to worry about and September 30 will be business as usual. Older devices, however, could run into some trouble.
The impact was wider than predicted, affecting not just users of older devices but many web-based services, such as Xero and Shopify. At Bookswarm, we immediately started to notice problems falling into two categories:
1. Server-based authentication issues
A number of WordPress websites that used premium plugins stopped working properly – many of these plugins ‘phone home’ to check for updates and confirm they have a valid license.
We also saw problems on sites which had any kind of integration with other third-party services, as they were no longer being trusted by those third parties.
Most of these issues didn’t affect the end user experience of the site – but a few definitely did.
After much investigation and experimentation, we discovered that a full reboot of the server resolved these issues – possibly because it caused the servers in question to stop using the old, expired root certificate and replace it with a newer one.
We have now rebooted all our servers and feel like we have successfully solved these problems. However any client who notices unusual issues or strange error messages should contact us in the usual ways, as it may still be related.
2. Older client devices no longer trust Let’s Encrypt
Users with older devices now receive errors when trying to access websites secured via Let’s Encrypt certificates. Unlike the first issue, this one is outside Bookswarm’s control, as the problem is on the device being used, not the server. We have found some suggested workarounds for Windows and macOS here but we can’t confirm their effectiveness.
Any Bookswarm client who can’t access their website since Thursday should consider the possibility that this issue is affecting them – which can be verified by successfully accessing the site on a different device, or checking via a service like downforeveryoneorjustme.com.
The good news is that the majority of Internet users will still be able to access your website succesfully; the bad news is, you may need a device update to resolve the issues.
If device-based workarounds don’t work, then switching to an alternative certificate provider is the only other choice that we are aware of. This has a cost involved – and while it may enable you to gain access to your own website, it won’t help with the 45 million others!
The morals of this story…
- The Internet is more fragile than we realise
- There’s still a lot of power in turning things off and on again!