DataConfig Core and JSON Asset 1.3 Release

2022-06-20

Intro

We've just released 1.3 version of DataConfig serialization framework and the market plugin DataConfig JSON Asset. Some highlights:

The full changelists are here:

In this post we'll highlights some cool new stuff.

Non Class/Struct Roots

Say you want to use DataConfig to deserialize into a TArray<int>.

FString Fixture = TEXT("[1,2,3,4,5]");
TArray<int> Arr;

There's FFoo::StaticStruct() and UBar::StaticClass(), but there aren't corresponding ones for TArray<> and other containers. How do you get the property?

Well turns out you can create FArrayProperty and others by yourself. This seems to be a pattern used a lot in builtin plugins like 'PythonScriptPlugin' (see PyUtil::FPropertyDef). In this release we have DcPropertyUtils::FDcPropertyBuilder helper to make this easier:

//  create int array property
using namespace DcPropertyUtils;
auto ArrProp = FDcPropertyBuilder::Array(
    FDcPropertyBuilder::Int()
    ).LinkOnScope();

FDcJsonReader Reader{Fixture};
DC_TRY(DcAutomationUtils::DeserializeFrom(&Reader, FDcPropertyDatum(ArrProp.Get(), &Arr)));

Checkout more details here: DataConfig - Non Class/Struct Root

Deserialize From SQLite

Unreal Engine has bundled 'SQLiteCore' with SQLite3 source and a set of helpers. In this release we added an extra example to use DataConfig to directly deserialize from SQLite query into an array of structs:

//  execute query and deserialize
TArray<FDcExtraTestUser> Arr;
UTEST_OK("Extra Sqlite", DcExtra::LoadStructArrayFromSQLite(
    &TestDb,
    TEXT("SELECT * FROM users WHERE title == 'Engineer' ORDER BY id"),
    Arr
    ));

//  equivalent fixture
FString Fixture = TEXT(R"(
    [
        {
            "Id" : 2,
            "Name" : "Mark",
            "Title" : "Engineer"
        },
        {
            "Id" : 3,
            "Name" : "Bob",
            "Title" : "Engineer"
        }
    ]
)");

This is an example of extending DataConfig to support new formats other than builtin JSON/MsgPack. The cool thing is, once you derived FDcReader/FDcWriter and get it working, other things like serialization handlers can be reused and it should just work.

See links below for details:

Inline Structs

Previously we have FDcAnyStruct that can store a heap allocated struct of any type. In this release we added example of FDcInlineStruct64/128/256 that stores the data inline within a fixed size buffer:

FDcInlineStruct64 Inline1;
Inline1.Emplace<FColor>(255, 0, 0, 255);
UTEST_TRUE("Inline Struct", *Inline1.GetChecked<FColor>() == FColor::Red);

Comparing to FDcAnyStruct this get rid of the heap alloc and have better cache locality. What's more exciting is that this inline struct supports polymorphism deserialization, and can be used as a cheap alternative to inline objects. We'll do a blog post on this later.

See: DataConfig - Inline Structs

DcJsonAsset Dump To JSON

Finally we added a set of context menu actions for DataConfig JSON Asset:

JsonAssetActions

And there's a new Dc.DumpDirectory commandlet for batch dumping DataAsset to JSON files on disk.

We tested this feature over Lyra and CitySample and iterated a bit. Here's an excerpt of dumping Lyra ShooterCore's main game feature data.

// /ShooterCore/ShooterCore
{
    "$type" : "GameFeatureData",
    "Actions" : [
        {
            "$type" : "GameFeatureAction_AddComponents",
            "ComponentList" : [
                {
                    "ActorClass" : "GameStateBase",
                    "ComponentClass" : "/ShooterCore/Accolades/B_EliminationFeedRelay",
                    "bClientComponent" : true,
                    "bServerComponent" : true
                },
                {
                    "ActorClass" : "LyraCharacter",
                    "ComponentClass" : "LyraEquipmentManagerComponent",
                    "bClientComponent" : true,
                    "bServerComponent" : true
                },
                // ...

We take special care to make sure it's human readable. You can take a look at more dumped JSON here. See links below for details:

Conclusion

DataConfig 1.3 didn't add much new features instead focus on getting existing stuff fixed and improved. Give it a try if you deal with data and stuff in Unreal Engine.

slowburn.dev