Vortex Engine V3 and Editor Vertical Slice Complete!
Just a few days ahead of the 2018 SIGGRAPH conference, the last few pieces came into place to allow a complete experience from Editor to Runtime.
When we set out to revamp the engine and build an editor for Vortex, we wanted to provide users of the engine with a way to intuitively assemble and tweak 3D worlds and then run them on the engine without the fuzz of having to rebuild the app every time.
The Vortex Editor moved us along that direction, allowing the user to visually build their world using point and click. The final missing part was the ability to build a self-contained package that wrapped all the project resources and could be distributed.
Enter the Vortex Archive (.vtx) files.
Vortex Archive files wrap all the resources necessary for the engine to load your created 3D world and run it. With this in place, we now have a full end-to-end experience where a world can be authored completely in the Vortex Editor, then packaged into a Vortex Archive, and ultimately run on any of the supported platforms.
Vortex Archive Format (.vtx)
In order to package the scene manifest and all referenced resources, I ended up designing a custom binary file format for the Archive. I used the extension .vtx, although I originally wanted to call these .var files (after Vortex ARchive). .var is however used for temporary files in UNIX systems so I didn’t want to clash with that convention.
The format in its initial version is pretty simple to read and write. The following table shows how the resources are stored inside the archive.
| Size (bytes) | Contents |
|---|---|
| Archive Header | |
| 8 | Archive Version (currently 1.0) |
| 8 | Number of Entries |
| Contents | |
| 8 | Resource Path String Length |
| 8 | Sub-Resource Identifier String Length |
| 8 | Data Lump Size |
| varies | resource path |
| varies | sub-resource path |
| varies | raw file data |
| … | … |
The contents section contains all the resources one after the other. The total number of stored resources is given by the archive header, under the “Number of Entries” field.
I could’ve added a “magic” number at the beginning, but all in all, this is a very simple format that binds everything together.
A Note on Compression
As part of the format definition process, I studied compressing each individual resource using zip. Ultimately, I discarded the idea.
Although zip compression would be beneficial for text resources (such as the scene manifest), at this time the vast majority of resources stored are (already) compressed image files. These are not expected to deflate significantly, so I couldn’t justify the increase in complexity at this time.
I might revisit this in the future as we expand scripting support and provide the ability to write custom shaders.
Vortex Runtime
With a complete workflow from Editor to Engine, it’s now possible to completely build a 3D world on desktop and deploy in any of the supported platforms: Windows, Mac, Linux-based OS’s and mobile devices.
Now, as easy as it is to add the engine to a project, there might be cases where we don’t want to write an app just to be able to run a 3D world. Cases where all we need is a thin layer that loads and runs our archive. For these cases, we’ve decided to add a third project into the mix: the Vortex Runtime.
The Runtime is a small frontend app to the engine that can load a Vortex Archive and play it. It’s a minimal, platform-specific app that wraps the underlying intricacies and provides a consistent interface on which Vortex Archives can be run.
Runtimes can be developed for any platform where the engine is available, enabling authored 3D worlds to be deployed virtually anywhere. An advanced user will probably still want to use C++ in order to stay in control, but for building simple playgrounds in Lua, the Runtime might be all that you need.
I think that this is a powerful concept – and a fun one to explore – being able to determine how much of the engine we can expose through the scripting API before you have to dip into native code.
Conclusion
It’s been a long ride since we initially set off to build a vertical of the Editor and revamp the Engine, but it has been worth it.
With these latest additions, we’ve now got a complete tool that we can grow horizontally. It is a starting point we can use to study the implementation of new rendering techniques, as well as further explore tech related to simulation, physics, compilers, scripting APIs and native platforms.
This moment is the culmination of a lot of hard work (during my free time), but it’s not the end. It is the beginning. Stay tuned for more to come!