Beware the vanity metric

By:
On:
With: 28 Comments

As the hypergrid grows, we should be wary of vanity metrics.

I’m a pretty big fan of the Lean Business model pioneered by Eric Reis of IMVU fame.  The main concept behind Lean is that you start small, release a barebones version of your product, measure carefully, and then modify the product based on feedback. Rinse and repeat until you succeed, give up, or decide things need to radically change.  Critical to that process is knowing what’s the right metric to measure. It’s very tempting to measure what Eric calls “vanity metrics” such as user registrations, or number of posts.

While these metrics sound impressive and make the creator — and investors — feel fuzzy inside, they are ultimately pretty useless metrics for figuring out how to make things better once things begin to go wrong. Metrics like “the average number of posts per user per day” are far more useful, because they let you tap into how your users or customers are actually using the product.

Hypergrid space

As the hypergrid and Opensim in general grows, I’m seeing an awful lot of vanity metrics being tossed around as solid indicators of healthy growth. There’s two in particular I want to call out:

  1. Region Count: It doesn’t matter how many regions there are in the hypergrid or any grid if they’re all abandoned 99.99 percent of the time.
  2. User Count: I know for a fact I have at least two different accounts on OSgrid I’ve abandoned for various reasons (passwords are hard). Maybe they still exist, maybe they don’t.  But one thing is for sure, they sure don’t represent a hyper-engaged user brimming with virtual social vitality.

I’ll give Active Users a pass, even though technically it’s pretty much a vanity metric.  At the very least it gives you a sense of how “active” — a pretty vague concept — a grid has been in the recent past.

This isn’t to say we shouldn’t share vanity metrics. They’re encouraging to look at, and give us a vague sense of what direction the wind is blowing in. However, when it comes to actually running our startups, we should learn to differentiate vanity metrics from useful ones and be honest with ourselves about which is which.

Now, having just poo-poo’ed several of the most popular metrics used to measure the growth of Opensim, it is only be fair to propose some other metrics to use instead.

These metrics are not necessarily for public consumption, but can be important for internal measurement of a grid’s health as a business.

These ones, while certainly not all that glamorous, would give us a better idea of what’s actually going on.

  • Average session length: How long are people actually staying on?
  • Average session region visit count: How many regions are users visiting?
  • New user login frequency: Are new users coming back, and after how long? Is there a fall-off rate?
  • Time-to first friend: How long do users go before they find someone to be friends with?
  • Text traffic breakdown: Which do users use more, region chat, private IM, or group IM?
  • Average download size per region: How much are region owners asking their users to download when they visit?
  • Average session friend encounters: How often, per session does a user end up within chat range of a friend?
  • Average population of teleport destinations:  Do users gravitate toward populated regions, or wander off to empty ones?
  • Time-to non-default avatar: How long does it take a new user to change their default appearance?
  • Region population turnover: How much of a region’s traffic is returning users versus new visitors?
  • Region percentage loaded: How much of a region ends up loading for a visitor during their stay?
  • Average Conversation Size: How many other users do users chat with?
  • Parcel Visit Distribution:  What parcel do region visitors end up visiting the most? For how long?

There’s probably hundreds of other questions we can ask. It all depends on what we’re trying to find out. The important thing is to ask in the first place, and then set out to try and answer them.

Region counts on the 40 largest public OpenSim grids -- but size does not indicate success.
Region counts on the 40 largest public OpenSim grids — but size does not indicate success.

Getting these metrics will probably require the collaboration of grid operators, region owners, and in some cases viewer developers.

What metrics would you like to see the data on?

(This article adapted from a post at MetaPundit and its follow-up article.)

nexusomega@gmail.com'
Latest posts by Alan Tupper (see all)