Asset Cluster Maintenance Complete

CLUSTER MAINTENANCE

On April 16th, OSG undertook an emergency cluster maintenance to update core cluster software packages to resolve critical bugs.

INVENTORY AND ASSETS

After the cluster maintenance and restart – many of you began noticing “missing asset” errors when rezzing recent items.

From evidence and investigation – these missing asset messages are not related to the updates applied on the 16th.

It appears that for the last month or so – the asset server was unable to write assets to the database, in a way that did *not* cause an exception and alert the asset server of trouble.

Asset updates appear to have been silently discarded by one or both of the asset clusters during that time.

These silent discards may have been “masked” by the various viewer, region, and nginx caches so that these “ghost assets” still appeared viable… until the asset server was restarted for emergency patching.

CHANGES

While investigating what happened, and trying to determine how this was even possible, several changes have been made to the asset cluster to attempt to prevent a repeat of this type of event, or detect and report it IF it starts to happen again.

In response, OSG has changed the database indexing system, added new logging and exception reporting, and additional database write reporting.

Additional asset write-and-verify tests are being implemented going forward.

At this point, the asset cluster appears to be 100% functioning correctly.

GOING FORWARD

Please examine your viewer and region inventory.

Clear viewer caches and reload them, and delete (or move to another folder) any inventory items that report missing assets.

Regions can use the fcache assets, fcache status, and backup commands to check their region cache and restart your region, watching for new or different asset errors.

Any assets which still appear missing may need to be reloaded from their source, or an IAR/OAR made before or during the time the asset database was silently discarding writes.

FINAL NOTE

We feel your pain with these “missing asset” messages, as the OSG admin inventories and assets were not immune and are experiencing the same issues you are.

We deeply apologize for any inventory or asset issues that you have encountered.