Update 5/19: AnSky is switching away from Gloebit, possibly to a local currency on the jOpenSimMoney module.
The Gloebit payment services is back up this week, kind of, after more than a month of downtime, but many grids and users are still reporting issues with the payment system.
“We restored the service last weekend,” Gloebit founder and CEO Chris Colosi told Hypergrid Business on Tuesday. “People should be able to access their accounts, buy and sell Gloebits, and cash out.”
However, some issues still remain, he said. “Sending email from the service is still down, so anything that requires email — password resets, confirming an email address, contact-us page — are not back yet.”
There are also some performance issues which are impacting some inworld transactions. he added. “We’re working through items to improve the performance.”
Gloebit is a popular multi-grid payments system, kind of a PayPal for OpenSim. So the outage affected many grids. Equally as upsetting for many was the lack of communication. Statements from the company have been short, and, for the most part, not informative to actual end users.
“We’re back,” Colosi said in a statement on Facebook Monday.
There were no apologies. No explanations of what the company will do in the future to avoid a similar shutdown. No acknowledgement even that it has hurt the users of the service.
During a month-long outage, Colosi has posted eight statements on Facebook, nothing on Twitter, no updates on their website, and no direct contact with the users affected by the loss of the service.
The first note was short and uninformative.
“We are experiencing an outage,” Colosi posted on April 6. “AWS believes our service is up but it is not accessible. We have a support ticket into them.” AWS — Amazon Web Services — is the platform where the Gloebit application and data are hosted.
This was followed by a few short updates over the following four weeks with technical details about the company’s problems getting support from Amazon. There were a few details about how the service is configured, but no information about impact to users, or what the company was doing to mitigate the effects of the outage.
For example, on April 11, Colosi wrote, “We haven’t yet found a way to restore or recreate our old elastic beanstalk environment. We are working on dockerizing our service to remove the dependency on elastic beanstalk and allow us to easily deploy it elsewhere. This outage and the one in December were both caused by elastic beanstalk, so we looking forward to getting off of EB and hopefully restoring a more robust service.”
He was referring to the AWS Elastic Beanstack, an application deployment platform offered by Amazon, and to Docker, which is a way of packaging applications into small virtual containers that can be moved from one cloud to another. While of possible interest to other application engineers, these updates were of little use to actual users of the platform.
Meanwhile, even though Gloebit is supposed to be up, some people are still having issues.
“The balance is okay on the website,” said Panthera Grid owner Kylie Brimmer. “But Gloebit isn’t working property on my grid.” Instead of confirming the payment, it just goes to the website to show the currency balance.
“I posted the issues on Gloebit’s Facebook page and no answers,” he said. “I’m not the only one to have issues.”
The Dynamic Worldz grid still has not seen the service restored, grid owner Danny Cruise told Hypergrid Business. However after further research, the problem only shows up on grids hosted on Windows servers and, more specifically, Windows server 2012 and earlier.
“As I have a mix of servers, all on different versions, I will try and move any of our Gloebit enabled regions onto the newer servers,” he said. “This is a .Net issue and Windows 2012 and earlier do not support the required release.”
Grid owners, users are furious
OpenSim grids use Gloebit to provide payment for their virtual worlds in a way that bring in revenues — they get a cut of all in-world transactions — but removes risks for themselves. Avination, for example, once one of OpenSim’s most successful commercial grids, lost $126,000 to hackers, which contributed to its demise.
For merchants, Gloebit is an easy way to accept payments from stores on multiple grids, and have it go into one centrally-managed wallet, kind of like a PayPal for OpenSim. Instead of trying to figure out how to cash out multiple local currencies, they only had to work with one, relatively easy, online interface.
And for OpenSim residents, Gloebit is a way to keep money safely in an online wallet that follows them from grid to grid to grid. If one of those grids goes down, all the money is still safe — just like your PayPal balance isn’t affected if one of the websites that uses it for payments goes out of business.
If anyone could get a payment system like this to work, it should have been Colosi, who’s best known as the guy who used to run the Second Life Marketplace and the Lindex exchange. Today, Colosi works as a software engineering manager at financial technology firm States Title.
The recent outage — and the series of outages last year — are giving people second thoughts.
“Back when I worked at IBM we used to say that if a bank or financial institution did not recover from an outage within 24 hours, it would be out of business within a year in 99 percent of the cases,” said Xmir grid owner Geir Nøklebye.
Technical issues are sometimes out of your control, said DigiWorldz grid owner Terry Ford. But constant communication with customers is vial.
“Even if the communication is not what they want to hear, at the very least your customers will know someone is aware, has taken responsibility, and they are working towards a resolution,” he told Hypergrid Business. “In the case of the Gloebit outage, many users felt as if they were being avoided, wondered if they would ever get their money, and were upset they weren’t being updated frequently. I believe if you are to provide a service which so many users depend on, you need to have plans, systems, and resources in place in order to resolve all issues, from minor to major. It’s clear in this case, if such a plan existed, it has failed miserably.”
DigiWorldz give users the option of using Gloebit or the local DigiWorldz currency on their regions.
Many users have already asked him to remove Gloebit, he said.
“Some are saying they would return, and others saying they would never use the service again,” he said. “Which is similar to what I’ve heard from other grid owners and users regarding this outage. Hopefully, the lessons learned from the outage will help to make the system more reliable, and will help to mitigate any future issues, but for now, I think many users are lost forever, while others will be willing to try again. I believe if you are to provide a service which so many users depend on, you need to have plans, systems, and resources in place in order to resolve all issues, from minor to major. It’s clear in this case, if such a plan existed, it has failed miserably.”
AviWorlds, OpenSim’s third large grid by number of users, has already switched away.
“We held in-world meetings with our staff and also polled our residents and we have decided to make a switch to Podex,” AviWorlds CEO Josh Boam told Hypergrid Business. “It is not acceptable for a business to operate like this.”
Neverworld, which converted the entire grid to Gloebit a year ago after four years with Podex, has switched back to Podex.
Even before this outage happened, the grid wasn’t happy with Gloebit, grid owner Govega Sachertorte told Hypergrid Business.
“They were very slow in answering any support emails that we sent to them or in providing any assistance,” she said. “We will still offer it for personal regions if they absolutely insist in having it but I give Gloebit a big thumbs down.”
Discovery Grid, one of the top 15 most popular OpenSim grids, is still having problems with the payments platform.
“Gloebit only appears to be back,” said grid owner Rene Vega, also known as Balpien Hammerer in-world. And it isn’t anywhere near even “kind of” back, he added.
“I had to inform my Discovery Grid residents of how badly the Gloebit transaction service is performing,” he told Hypergrid Business. “We are back to long transaction times and transaction failures. This happened in the previous 2nd outage where for several months people were complaining about failed transactions or transactions that took days to resolve.”
He said that he no longer trusts Gloebit to be a reliable service.
“Three outages with apparently no hot backups, no warm backups, seemingly no recovery plan in place, and worse, no transparency by the company makes it impossible to keep my residents apprised of the situation,” he said.
A “hot” backup is when there is a second, exact duplicate of the service that is activated as soon as there are any problems, without any interruptions to users at all. A “warm” backup is one that takes a little longer to start up, but it usually costs a little less to maintain. A “cold” backup is just copies of all the files — to restore, the infrastructure needs to be set up from scratch, which can cause some delays, and is the cheapest of the three options.
All three require a disaster recovery plan in place to switch over to the new system, reroute traffic, and notify all users about any expected downtime. Plus, these disaster preparedness plans need to be practiced regularly to make sure they’re still working, and that employees are well versed on procedures.
Colosi in particular, with a background in the financial services — and is experience at Linden Lab — should have been well versed in disaster planning and recovery procedures.
It’s no surprise that users are angry.
“Most merchants I asked are furious,” said Discovery Grid’s Vega. “And half of them will not go back to Gloebit. The other half will continue because they have a large hypergrid visitor clientèle, so they are stuck with using Gloebit transactions in their regions. Though I have deprioritized the use of Gloebits in my sponsored regions, regions owners still will have the ability to choose which currency they choose for their regions. This is a self-service option for them. Their choice, their risk.”
Discovery Grid has two primary economies, he explained.
“There is the community oriented one where residents use the local currency — DC$ — to trade amongst themselves,” he said. “There are the ‘minstrels,’ visiting entertainers who travel from town to town, grid to grid, performing for people and mostly cashing out their tips. They rarely trade within the grids they visit. I believe the latter activity is better served by using real money transactions mediated by PayPal. Now that PayPal is offering QR code-based payments using their wallet app, the overhead of the typical small payments is quite low.”
Normally, a PayPal transaction costs 30 cents plus 2.9% of the total amount. For a $10 purchase, this adds up to 59 cents, which isn’t too unreasonable. But for a $1, the cost is 33 cents — a third of the total amount. Merchants either need to take the loss, or raise prices to cover the fees.
My comparison, Vega said, the QR code transactions cost 10 cents plus 2.9 percent of the total amount, so for a $1 transactions, the fee is a much more reasonable 13 cents.
“My plan is to integrate PayPal transactions with the local money module so that selected objects, in otherwise local currency regions, will transact in real money and the money event will trigger when a payment completes,” said Vega. “Another plan is to have the local money module allow hypergrid visitors to transact in the local currency, although this will also require Podex to link hypergrid accounts so that they can purchase local currency. I believe these changes will be a reasonable replacement for the unreliable Gloebit.”
He also urged OpenSim users to remember that virtual currency can disappear at any time, even with the best of backups.
“People should always be mindful of their currency balances, no matter which ones, and only keep a balance no higher than they are comfortable losing,” he said.
At first, the risks of running his own currency outweighed the risks of sticking with Gloebit, said AnSky grid owner Andrew Simpson.
But now, trying to keeping Gloebit going is becoming too problematic.
“I’m stopping using Gloebits due to the grid having so many issues with Gloebits are on,” he told Hypergrid Business. “I had to draw the line.”
Instead, he will probably be switching to jOpenSimMoney, he said, in combination with PayPal.
PayPal is a common option for grids for higher-value payments, such as region rentals, or to top up virtual currency wallets. But PayPal is expensive when used for in-world transactions and exposes the buyer’s email address to the seller, which creates privacy worries for some users.
The commercial jOpenSimMoney platform is offered by the team behind the jOpenSimWorld grid and starts at 30 Euros (about US $37) per year for a 20-region grid, with an additional per-region cost for more regions.
German World Grid is staying away from virtual currencies of any kind, grid owner Mickie Bohannes told Hypergrid Business.Â
“Both of the options that exist seem too insecure to me,” he said.
Plus, a virtual currency makes less sense in a community where people want to have content shared for free.
“Despite that, I think it’s a great thing that there are such currency providers,” he said. “My goal is to strengthen OpenSim and to keep people in OpenSim.”
Gloebit alternatives
Podex is the single biggest alternative to Gloebit, but it uses a different operating model.
With Podex, each grid still has its own currency, but Podex provides exchange services to make it easier for residents to buy and sell that currency or to trade it for currencies on other grids. Shopping is much less convenient, though, since merchants have to manage multiple currency accounts and shoppers who teleport to a new grid will have a hard time making purchases there. Gloebit was perfectly positioned for hypergrid shopping — when the service was working, of course.
Another option is to go fully independent, creating a currency system from scratch. That leads to potential security risks, if, say, users buy currency with stolen credit card numbers, redeem it for cash, then the credit cards are cancelled and the payments are reversed, leaving the grid in debt.
AviTron will also continue to use Gloebit, but is actively looking for alternatives, said grid owner Alexsandro Pomposelli.
“JOpenSimMoney is great, but it is a paid one,” he told Hypergrid Business. “It charges a per sim amount.”
The Virtual Temple grid is one that’s been using the jOpenSim money module, grid owner Stefan told Hypergrid Business.
“It works fine and can easily be connected to PayPal for users to buy more currency,” he said. “However it is not multi-grid. I have had thoughts to change over to Gloebit, for the sake of being multi-grid and established.”
However, the platform’s performance over this past year has given him pause. “I didn’t know they have had problems for a year. It’s not good if they haven’t made any apologies or comments on it.”
“”I would wait to change over to Gloebits, and see how they will perform from now,” he added. “But I believe they will solve the issues over time.”
Another option for in-world payments is PayPal.
PayPal’s costs are high when used for in=-world micropayments, but most grids already use it for bigger-ticket things like region purchases.
“I am intending to use PayPal if needed,” said Kashi Takeshi owner of the Virtual Worlds Zone grid. “Gloebit proved to be too unstable to function as a trustworthy money platform for OpenSim.”
Kitely, which has its own Kitely Credits virtual currency, also has an option of PayPal payments on the Kitely Market.
“Virtual currency solutions come and go, and OpenSim has had multiple providers over the years that gained traction on multiple grids but ultimately suffered the same fate,” said Ilan Tochner, Kitely’s co-founder and CEO. “My recommendation for people who wish to buy or sell content without having to rely on the continued operation of such virtual currency systems is to do it directly using real money via their PayPal account. You can use PayPal directly on Kitely Market or using several third-party inworld vendors that support PayPal payments.”
People can also accept donations or rent land using PayPal, he added.
To make it easier for in-world merchants to set up payment systems, there are products available on the Kitely Market.
Ocean Engineering‘s PayPal Rental Box and PayPal Donate Box handle the payments processing, but at a cost of 20 to 30 cents per transaction. Plus, with PayPal, the payment recipient is notified of the payer’s email.
Another option that merchants can consider is sending buyers to the Kitely Market website for the final purchase, said Tochner, by, say, having their product sale sign link directly to the item’s market listing.
“That way when the sign is clicked the relevant page will open in the buyer’s browser and they’ll be able to securely complete the order from there — including in-world delivery using the Kitely Market hypergrid delivery system,” he said.
The process does add a few extra steps to the purchase process.
On the other hand, it also helps merchants serve customers on defunct grids, by allowing them to re-deliver the purchased items to customers’ Kitely avatars, said Tochner.
And there’s also a privacy benefit, he added.
“When people use PayPal on Kitely Market the buyer and the seller don’t see each other’s PayPal information or email,” he said. “Instead, they only share their avatar name, just as they would if they made the transaction in-world.”
ZetaWorlds, 3rd Life, Phantom Rose, Langalf’s Demesne sticking with Gloebit
“The recent outage was vastly mishandled, there is no doubt about it,” said said Vincent Sylvester, CEO of Zetamex Network, an OpenSim hosting provider and operator of the ZetaWorlds grid.
But it’s still the only game in town, he said, since the alternatives are less flexible and less secure, he told Hypergrid Business.
“Seeing the ease at which Gloebits can be sold indicates residents across the metaverse are still purchasing now that the service has returned,” he said. “As we have in the past, we will continue to work closely with Gloebit to aid them in strengthening their service and improving the quality for users.”
As frustrating as the outage was, it illustrates the inherent problems of using Amazon, he added. The company doesn’t provide adequate support and solutions for issues on their network.
And aside from the infrastructure issues, Gloebit’s design and setup, while still needing work, is much better suited for actual commerce than the alternatives.
One of the most popular option for grids creating their own currencies, for example, is the DTL money server for OpenSim, he said. “The disclaimer of ‘play money’ is as prominent as ever and rightly so.”
In addition, the OpenSim user base is forgiving.
“The metaverse has seen grids go down for longer and still somehow retain the majority of their userbase,” he said. “Commerce and money is not the essence of life, thankfully.”
Looking ahead, he recommends that Gloebit find a way to get off of the Amazon platform.
“AWS really lacks the support for smaller scale environments and their systems are so notorious for going down and breaking in random manner that most setups use redundancies and multiple backups to maintain service level,” he said. “For a business operating at larger scales the issues of AWS are less apparent and the cost savings are easier to see. Lower the scale and setups on dedicated hardware and providers more aligned with individuals than corporations are the better choice.”
Sylvester said that he’s in talks with Gloebit to provide further assistance and consulting to help avoid future outages.
“Blaming and pointing fingers only gets you so far,” he said. “Providing help and support to the betterment of everyone involved secures the future of not only Gloebit, but OpenSim as software and community.”
Zetamex Network was one of the launch customers of Gloebit, he added. “We continue to work with them to resolve issues and find a way forward for the service.”
He’s not the only who’s betting on Gloebit.
3rd Life Grid will be sticking with service as well, grid owner Tommy Seetan told Hypergrid Business.
“Our residents that sell items like the fact that residents from other grids can buy from them and take it back to their own grid,” he said. And residents like the fact that they can use their Gloebits on other grids as well.
The only other alternative is Podex, he added. But that platform doesn’t have the same kind of multi-grid functionality as Gloebit does.
As the owner of an independent, non-commercial grid, Phantom Rose, I find Gloebit funds very valuable to me,” said Phantom Rose grid owner Lannorra Sion. “It really is the currency for the metaverse. It allows anyone from any grid to buy things from different creators from different grids. I use it exclusively. If Gloebit does not come back, my only source of buying items I want is through the Kitely Market.  Which would be very disappointing. Kitely Market is great, and I do buy often from them, but it doesn’t carry everything available.”
With Gloebit, she can go shopping at pop-up markets on Discovery Grid, for example. “Many of the merchants will use Gloebits and I will buy things that I have not seen on the Kitely Market,” she said. “I will definitely be using Gloebit once their service is running normally again.”
She has been burned over the years when grids shut down, however, which is one reason that, in addition to owning small bits of land on other grids, she also has her own grid.
“Going forward, I think I will buy Gloebits as I need them, and not keep much as a balance,” she said. “But otherwise, I look forward to using their service again. And I hope they stay in business!”
“I plan to stick with Gloebits,” said Larry Schrupp, owner of the Langalf’s Demesne grid. “There are only a handful of users on my grid, and no real commerce at present, so the down time has been a nuisance only. For us it is more an experimental environment. We use Gloebits in transactions on other grids, and it does not hurt me to support it on my grid. I do not really know of any viable alternatives for a small grid like mine.”
“It is good to see them back, I hope they can go forwards now and hopefully get stronger in the future,” added Enchanted Grid owner Dai Rhys-Owain.
- OpenSim user growth cools down with chilly weather - November 15, 2024
- Kitely adds PBR materials support, larger textures - November 8, 2024
- OSCC 2024 Submission Deadline Approaching - October 19, 2024