US military turns to Halcyon branch of OpenSim

By:
On:
Tagged: , ,
With: 5 Comments

Editor’s note: This article was updated on 1/27/2023 to remove dead links and links to malicious sites.

Editor: The team behind U.S. Army’s MOSES grid have long been cutting-edge contributors to OpenSim, building a grid management system, running scalability tests, and donating content to the OpenSim community. Now they are in the process of moving to a branch of OpenSim called Halcyon, open-sourced last year by InWorldz, OpenSim’s most popular grid.

(Image courtesy Douglas Maxwell.)
(Image courtesy Douglas Maxwell.)

The MOSES project and the OpenSimulator project have diverging goals. Specifically, we place lesser value on backwards compatibility and for us security is a high priority.  I believe the legacy code is holding OpenSim back and slowing true innovation.

Last year, we dissected and profiled the OpenSimulator code and found five major areas for improvement.  These areas include physics, database management, client management, conflicting TCP/UDP network stack, and script engine operations.

Also, there was an attempt to multi-thread pieces of the simulator, but it was done in a way that did more harm than good.  In order for us to correctly assess the behavior of the simulator, the first thing we fixed was the broken frame rate counters and network performance reporting.

We had a lot of trouble getting patches approved and integrated into the OpenSimulator code base last summer, with many of the objections and comments being that the updates might have unexpected effects on existing users.

The engineering team at InWorldz had been working on their Halcyon code base during the same time period.  They saw our posts on OpenSim’s developer channels and contacted me directly.  They had arrived at many of the same conclusions we did and identified many of the same areas of improvement.  They independently worked on scripting engine improvements and database management.

I was particularly impressed with the way they used the Cassandra database. We also did early physics work with PhysX for Windows, focusing our initial efforts on a Linux based PhysX integration and networking.

An actual environment, on the left, compared to its virtual representation on the right. (Image courtesy Douglas Maxwell.)
An actual environment, on the left, compared to its virtual representation on the right. (Image courtesy Douglas Maxwell.)

In a lot of ways, the Halcyon developers found and fixed significant deficiencies that persist in the mainline OpenSim code even today.  Even though the InWorldz team forked from mainline OpenSim a long time ago, their current Halcyon code is quite advanced and it is no surprise to me why their grid is stable and successful.

The MOSES team spent a considerable amount of time examining the Halcyon code base and we compared its differences to the current OpenSimulator.

As it turns out, our MOSES Grid Management tools are a bit more functional than theirs, so we integrated them into Halcyon.  We have MOSES-Halcyon hybrids running on our servers at the moment and we will migrate after we finish updating the Halcyon code to accept IAR files — inventory export files.

Over the past year, we’ve had a number of engineering exchanges with the Halcyon developers.  My team and I have been to San Antonio for meetings with them and they have come to Orlando for technical exchanges at our labs.  The two teams have integrated well and our cultures are in much better alignment than trying to deal with the melee of the OpenSimulator core.  The two teams have a scheduled on-site meeting in Orlando again during the week of August 22.

Where do we go from here?  In short, we are following a professional systems engineering approach to planning a modernized OpenSimulator code base. OpenSimulator is woefully insecure.  The first major co-development effort is to change authentication and begin using web tokens with encryption.

We are also in the planning stages of a detailed design document for the HTML5 client, which will be a browser-based viewer for OpenSim.

Near term, we will pull the entire networking stack out of the simulator.  The simulator will become strictly a compute process.  We are currently working on a network data arbiter that is a glorified firewall that will be a stateless and lightweight web translation layer between the HTML5 clients and the simulator.  This will use Google’s protocol buffers or something similar and allow us to efficiently scale on the simulator side.

This will also be a port reduction strategy and reduce the network footprint of the larger grids.  Any plans for a hypergrid-like functionality that allows teleports between separate grids will need to be completely re-engineered.

It is true that the OpenSimulator code is around a million lines of code, but some of it is fat that should be trimmed.  Unnecessary appendages will be cut or replaced with modern equivalents.  Since we are looking to create a truly distributed simulation engine, any data that is currently transmitted using clear text will be replaced with serialized formats.

Mesh vehicle convoy on the MOSES grid. (Image courtesy Douglas Maxwell.)
Mesh vehicle convoy on the MOSES grid. (Image courtesy Douglas Maxwell.)

Lastly, OAR — region export files — and IAR functionality will be preserved for the foreseeable future.  They are convenient and well done.  We would also like to include support for static meshes in the HTML5 client, so watch out for announcements on that front.  Some adjustments will also need to be made to accommodate the new physics functionality, such as preservation of new types of linked objects.

We will be pushing regular updates to our public Github site, so anyone is able to pitch in and help with development.  We’ve already had a number of people contact me with intense interest in the HTML5 work.

douglas.b.maxwell@hypergridbusiness.com'
Latest posts by Douglas Maxwell (see all)