Polymorphic Serialization In Unreal Engine

2022-12-08

Intro

Speaking of polymorphism in programming most people would think of C++ virtual function, which is a form of runtime polymorphism.

A related concept that we call polymorphic serialization is also seen a lot in game programming. But somehow it's less talked about. In this post we'll:

Definition

Here's a classical toy example of C++ polymorphism: a base Shape class with derived Rect and Circle.

struct Shape
{
    virtual float Size() = 0;
};

struct Rect : public Shape
{
    float Width;
    float Height;
    float Size() override { return Width * Height; }
};

struct Circle : public Shape
{
    float Radius;
    float Size() override { return 3.14f * Radius * Radius; }
};

Now virtual float Size() is a virtual function which implements runtime polymorphism. The thing we're more interested in is that given a Shape pointer or reference, how can we serialize it with correct type info and its member fields? This process is what we call polymorphic serialization.

Note that there's no out of the box solution with stock C++ and STL. It's actually pretty tricky to implement when you think about it:

The good news is that Unreal Engine supports this out of the box. Let's take a look at how it's done.

Instanced Object

We know that UObject is Unreal Engine's magical class that support reflection, serialization and everything. The first solution here is to take advantage of Instanced Object, which is toggled with UCLASS(DefaultToInstanced):

UCLASS(Abstract, BlueprintType, EditInlineNew, DefaultToInstanced)
class DATACONFIGEXTRA_API UDcBaseShape : public UObject
{
    GENERATED_BODY()
public:

    UPROPERTY(EditAnywhere) FName ShapeName;
};

UCLASS()
class DATACONFIGEXTRA_API UDcShapeBox : public UDcBaseShape
{
    GENERATED_BODY()
public:

    UPROPERTY(EditAnywhere) float Height;
    UPROPERTY(EditAnywhere) float Width;
};

UCLASS()
class DATACONFIGEXTRA_API UDcShapeSquare : public UDcBaseShape
{
    GENERATED_BODY()
public:

    UPROPERTY(EditAnywhere) float Radius;
};

Here're some good docs on the UPROPERTY and UCLASS flags. To use these classes simply setup a UDcBaseShape* property with Instanced flag.

UPROPERTY(EditAnywhere, Instanced)
UDcBaseShape* ShapeField1;

In Unreal Editor it looks like this:

DcBaseShape Property in Unreal Editor

Under the hood instanced children UObjects would be created for each field. These sub objects get serialized along with their parent object and have the same lifetime.

Note that it also works as intended with TArray/TMap, with proper editor and serialization support.

Instanced objects are used in UE5's Modular Game Features plugin. There're sample setups of GameFeatureData in Lyra. One primary DataAsset class looks like this:

UCLASS()
class GAMEFEATURES_API UGameFeatureData : public UPrimaryDataAsset
{
    // ...
    UPROPERTY(EditDefaultsOnly, Instanced, Category="Actions")
    TArray<TObjectPtr<UGameFeatureAction>> Actions;
};

In Lyra the ShooterCore is a instance of of GameFeatureData and the actions setup in the editor looks like this:

Lyra GameFeatureActions ShooterCore

This is an example of UE's way of doing data driven game development. By setting up polymorphic instanced objects you can mix and match these actions and eventually build up modular features without hard code stuff in C++.

Instanced Struct

Instanced objects works but there're obvious drawbacks:

Well turns out there's a solution to all the issues above. Starting with UE5.0 there's a new plugin StructUtils with a struct FInstancedStruct. It does exactly what it's name suggests: same feature as instanced objects but now uses all USTRUCTs.

The class hierarchy above ported struct looks like this:

USTRUCT(BlueprintType)
struct DATACONFIGEXTRA_API FDcStructShapeBase
{
    GENERATED_BODY()
    UPROPERTY(EditAnywhere) FName ShapeName;
};

USTRUCT(BlueprintType)
struct DATACONFIGEXTRA_API FDcStructShapeRectangle : public FDcStructShapeBase
{
    GENERATED_BODY()
    UPROPERTY(EditAnywhere) float Height;
    UPROPERTY(EditAnywhere) float Width;
};

USTRUCT(BlueprintType)
struct DATACONFIGEXTRA_API FDcStructShapeCircle : public FDcStructShapeBase
{
    GENERATED_BODY()
    UPROPERTY(EditAnywhere) float Radius;
};

Note all these structs are all plain USTRUCTs. To use these structs setup a FInstancedStruct property like this:

UPROPERTY(EditAnywhere, meta = (BaseStruct = "DcStructShapeBase"))
FInstancedStruct ShapeField1;

In Unreal Editor it looks almost the same as instanced objects:

DcStructShapeBase InstancedStruct

As you would expect serialization also works as intended. This is almost a drop in replacement for instanced objects for most use cases.

Instanced structs comes along with Mass framework and it's used a lot there. In the City Sample there's an sample of FInstancedStruct in action:

UCLASS(meta=(DisplayName="Assorted Fragments"))
class MASSSPAWNER_API UMassAssortedFragmentsTrait : public UMassEntityTraitBase
{
    // ...
    UPROPERTY(Category="Fragments", EditAnywhere, meta = (BaseStruct = "/Script/MassEntity.MassFragment", ExcludeBaseStruct))
    TArray<FInstancedStruct> Fragments;
};

In the editor the Fragments looks like this:

City AssortedFragments MassCrowdAgentConfig

It also looks the same as the instanced object counter parts. Under the hood it saves some memory but more importantly it gives you peace of mind for cutting the unneeded overhead for UObject.

Polymorphic Serialization with DataConfig

If you haven't noticed yet, we're working on DataConfig which is a serialization framework that supports JSON and MsgPack. At the very start we're determined to support polymorphic serialization as a very typical use case.

In the last DataConfig JSON Asset release we added a feature to dump DataAssets as JSON. The ShooterCore example above would be dumped like this:

// /ShooterCore/ShooterCore
{
    "$type" : "GameFeatureData",
    "Actions" : [
        {
            "$type" : "GameFeatureAction_AddComponents",
            "ComponentList" : [
                {
                    "ActorClass" : "GameStateBase",
                    "ComponentClass" : "/ShooterCore/Accolades/B_EliminationFeedRelay",
                    "bClientComponent" : true,
                    "bServerComponent" : true
                },
                {
                    "ActorClass" : "LyraCharacter",
                    "ComponentClass" : "LyraEquipmentManagerComponent",
                    "bClientComponent" : true,
                    "bServerComponent" : true
                },
                // ...
            ]
        },
        {
            "$type" : "GameFeatureAction_DataRegistry",
            "RegistriesToAdd" : [
                "/ShooterCore/Accolades/AccoladeDataRegistry"
            ]
        }
        // ...
    ]
}

You can see more dump results from Lyra and CitySample here. With DataConfig both instanced object and struct can be serialized and deserialized. For more details:

Finally we have a cool example which takes an extra step over instanced struct: Inline Struct behaves like instanced struct but it stores the struct instance inline. This avoids heap allocation and offers better data locality. The trade off is that the contained struct sizes are limited.

We provide FDcInlineStruct64/FDcInlineStruct128/FDcInlineStruct256/FDcInlineStruct512 which have varying sizes. The deserialization process also verify that the struct class can be contained within the inline buffer. Otherwise it would report the dialgnose message below:

# DataConfig Error: Inline struct too big: BufSize '56', Struct 'DcExtraTestStruct128' Size '128'
- [WideCharDcJsonReader] --> <in-memory>5:38
   3 |        {
   4 |            "InlineField1" : {
   5 |                "$type" : "DcExtraTestStruct128",
     |                                                ^
   6 |                "NameField" : "Foo"
   7 |            },

DataConfig has many other cool samples. Checkout links below if this interests you:

slowburn.dev