Hi everyone. OSGrid is back online. We took advantage of the downtime today to do some grid maintenance we were originally going to schedule for a few weeks from now. We moved the database backend for the central grid services (except assets) to a new pair of db servers. Our backend dbs had grown so large, and the disks on the machine serving them were sufficiently slow, that running the backups would effectively take down the inventory server for up to an hour. So we had disabled the automatic, nightly db backups because they were causing inventory problems for users. But that meant we didn’t have automatic, nightly db backups anymore. We had to backup the dbs by hand when an admin was available to babysit the grid servers while it ran and kill the backup if they started causing problems.
The fallout from that is that the most recent inventory backup we have is from June 11. While it appears most users were unaffected by the inventory problems yesterday, some were. If your inventory is missing and you have an Inventory ARchive (IAR) you can just load it up in a region you’re in and your inventory will be restored. If you don’t have an IAR to restore from, we can restore your inventory to as it was the date of the most recent backup, June 11. We apologize for the inconvenience we know this will cause. To have your inventory restored from June 11, please send your full avatar name via email to info@osgrid.org. Depending on the number of people affected this may take us a day or two to complete the restoration for them.
The good news is now that we’ve migrated our db services to a new pair of hosts, we’re able, and have already started, making multiple db backups per day. This will allow us to minimize impact from any sort of catastrophic failure in the future, and help us to recover more quickly.
It’s not clear at this point what caused the inventory erasure for some users, but we may never really know the answer. OpenSim is still alpha quality software, and unexpected things can and will happen. That’s not to say we couldn’t have done a better job with backups; because OSGrid is 100% donation driving we try to be as thrifty as possible, but in this case we should’ve spent the money earlier when we disabled automatic backups to quickly get us to a place where we could turn them back on. But today we have the benefit of hindsight.
We apologize for the downtime and for the inventory troubles for those affected, thank you for your patience, and thank you for using OSGrid.
-Dave
