Meta explained the specific technical reason that some Quest headsets were bricked late last month.
If you missed it, in December some Quest headsets, most commonly Quest 2 headsets, stopped booting successfully, getting stuck in what’s known as a “boot loop”. The issue occurred after an automatic firmware update for headsets which hadn’t been updated in a long time, where the difference between the previously installed version and the latest update was significant.
By the end of the month, Meta had acknowledged the issue and told customers “in most cases, you are now able to use your device normally”. For devices bricked, the company started offering free replacements, even out of warranty.
Ian Hamilton
Now, three weeks later, Meta’s VP of Quest and Horizon OS Mark Rabkin is detailing exactly what happened.
Rabkin says the issue was a “rare race condition in AOSP R/W ext4 filesystem that corrupts files + a key security patch that prevents OS rollback”, saying that the bug has been there for four years but hasn’t been an issue until December. Further, according to Rabkin, a security patch prevented the OS from rolling back to a working version.
Bug’s been in there 4 years but never been hit by us before Dec 🤷♂️. Race conditions are ouch.
— Mark Rabkin (@mrabkin) January 20, 2025
AOSP is the Android Open Source Project, the open-source core of Meta’s Horizon OS that runs on Quest headsets, ext4 is the Linux file system that Android uses, and R/W refers to reading and writing from the file system at the same time. A race condition is an issue which can occur in multithreaded software when different parts of the code running alongside each other execute in an unexpected order, leading to an unanticipated outcome.
Essentially, Rabkin is claiming that a bug in how Android’s file system handles reading and writing at the same time in a very specific and rare condition caused the issue, and a security patch intended to prevent rolling back to the previous OS version left this faulty update in place.
David Heaney
It’s the latest in a history of outages from Meta, and for those affected, far more serious than any past issue. Back in 2018 every Oculus Rift stopped working due to an expired certificate. That problem took Meta (then Facebook) more than 12 hours to fix. More recently, every internet-connected Quest headset stopped working for around an hour in March 2024 due to a server error, leading to Meta vowing to make Quest headsets “more resilient” to server issues going forward.
The bootloader of consumer Quest headsets is locked, meaning you can’t manually flash a custom operating system, and Meta doesn’t allow you to downgrade to an older Horizon OS version either. These decisions leave the devices vulnerable to an issue like this, even though the conditions needed for one to arise are exceedingly rare. To be seen as a serious consumer electronics company on par with Apple, Meta may have to reconsider some of its policies on software updates, and build up the kind of trust that is significantly eroded by being left with a brick on Christmas day.
