DataConfig Book

Serialization framework for Unreal Engine that just works!

DataConfig is a serialization framework built on top of Unreal Engine's Property System. It aims to be friendly and robust while offering good performance. Notably features:

• Out of the box JSON/MsgPack read write.
• Full support for UPROPERTY()/UCLASS()/USTRUCT()/UENUM().
• Pull/Push style API for verbatim data access and lossless type information.
• Designed as a collection of tools that can be easily extended to support other formats.

Getting Started

• Get the code on github.
• See Examples for examples.
• See Integration for quick integration guide.
• See Design for more context about the project.
• See Extra for more advanced usages.
• See Changes for versioning history.

There's also DataConfig JSON Asset on UE Marketplace. It's a premium plugin for importing JSON to UE data assets.

License

DataConfig is released under free and permissive MIT license. We'd really appreciate to credit us if you find it useful. See License for details.

Examples

Here're some short and quick examples showcasing DataConfig API usage and features. All code shown here can be found in the repo.

Blueprint Nodes

If you integrated the full DataConfig plugin we have Blueprint Nodes for you to quickly try it out.

Create a Blueprint Actor and setup the BeginPlay event like this:

Then place the actor in the level and start play. The JSON string would be print to screen like this.

JSON Deserialization

• DcTestBlurb.cpp

Given the structFDcTestExampleStruct:

// DataConfigTests/Private/DcTestBlurb.h
UENUM()
enum class EDcTestExampleEnum
{
    Foo, Bar, Baz
};

USTRUCT()
struct FDcTestExampleStruct
{
    GENERATED_BODY()
    UPROPERTY() FString StrField;
    UPROPERTY() EDcTestExampleEnum EnumField;
    UPROPERTY() TArray<FColor> Colors;
};

We can deserialize an instance from JSON with the snippet below:

// DataConfigTests/Private/DcTestBlurb.cpp
FString Str = TEXT(R"(
    {
        "StrField" : "Lorem ipsum dolor sit amet",
        "EnumField" : "Bar",
        "Colors" : [
            "#FF0000FF", "#00FF00FF", "#0000FFFF"
        ]
    }
)");

FDcTestExampleStruct Dest;

//  create and setup a deserializer
FDcDeserializer Deserializer;
DcSetupJsonDeserializeHandlers(Deserializer);
Deserializer.AddStructHandler(
    TBaseStructure<FColor>::Get(),
    FDcDeserializeDelegate::CreateStatic(DcExtra::HandlerColorDeserialize)
);

//  prepare context for this run
FDcPropertyDatum Datum(&Dest);
FDcJsonReader Reader(Str);
FDcPropertyWriter Writer(Datum);

FDcDeserializeContext Ctx;
Ctx.Reader = &Reader;
Ctx.Writer = &Writer;
Ctx.Deserializer = &Deserializer;
DC_TRY(Ctx.Prepare());

//  kick off deserialization
DC_TRY(Deserializer.Deserialize(Ctx));

//  validate results
check(Dest.StrField == TEXT("Lorem ipsum dolor sit amet"));
check(Dest.EnumField == EDcTestExampleEnum::Bar);
check(Dest.Colors[0] == FColor::Red);
check(Dest.Colors[1] == FColor::Green);
check(Dest.Colors[2] == FColor::Blue);

Note that EDcTestExampleEnum is deserialized by its name and FColor is deserialized from a html color string like #RRGGBBAA.

Say if we accidentally mistyped the EnumField value:

{
    "StrField" : "Lorem ipsum dolor sit amet",
    "EnumField" : "Far",
}

It would fail gracefully with diagnostics:

# DataConfig Error: Enum name not found in enum type: EDcTestExampleEnum, Actual: 'Far'
- [JsonReader] --> <in-memory>4:25
   2 |    {
   3 |        "StrField" : "Lorem ipsum dolor sit amet",
   4 |        "EnumField" : "Far",
     |                           ^
   5 |        "Colors" : [
   6 |            "#FF0000FF", "#00FF00FF", "#0000FFFF"
- [PropertyWriter] Writing property: (FDcTestExampleStruct)$root.(EEDcTestExampleEnum)EnumField

As bonus we serialize the struct into MsgPack:

FDcSerializer Serializer;
DcSetupMsgPackSerializeHandlers(Serializer);

FDcPropertyDatum Datum(&Dest);
FDcPropertyReader Reader(Datum);
FDcMsgPackWriter Writer;

//  prepare serialize context
FDcSerializeContext Ctx;
Ctx.Reader = &Reader;
Ctx.Writer = &Writer;
Ctx.Serializer = &Serializer;
DC_TRY(Ctx.Prepare());

//  kick off serialization
DC_TRY(Serializer.Serialize(Ctx));

auto& Buffer = Writer.GetMainBuffer();
//  starts withMsgPack FIXMAP(3) header
check(Buffer[0] == 0x83);   

Custom Serialization And Deserialization

DataConfig support custom serialization and deserialization logic by implementing FDcSerializeDelegate/FDcDeserializeDelegate.

In this example, we'd like to convert FColor into #RRGGBBAA and vice versa:

• DcSerdeColor.h
• DcSerdeColor.cpp

// DataConfigExtra/Public/DataConfig/Extra/Deserialize/DcSerDeColor.h
USTRUCT()
struct FDcExtraTestStructWithColor1
{
    GENERATED_BODY()

    UPROPERTY() FColor ColorField1;
    UPROPERTY() FColor ColorField2;
};

// DataConfigExtra/Private/DataConfig/Extra/Deserialize/DcSerDeColor.cpp
FString Str = TEXT(R"(
    {
        "ColorField1" : "#0000FFFF",
        "ColorField2" : "#FF0000FF",
    }
)");

First you'll need to implement a FDcDeserializePredicate delegate to pick out FColor properties:

//  DataConfigExtra/Private/DataConfig/Extra/SerDe/DcSerDeColor.cpp
EDcDeserializePredicateResult PredicateIsColorStruct(FDcDeserializeContext& Ctx)
{
    return DcDeserializeUtils::PredicateIsUStruct<FColor>(Ctx);
}

Then we'll need to implement a FDcDeserializeDelegate to deserialize a FColor. Here we'll do it by writing through R/G/B/A fields by name with the FDcWriter API.

// DataConfigExtra/Private/DataConfig/Extra/Deserialize/DcSerDeColor.cpp
FDcResult HandlerColorDeserialize(FDcDeserializeContext& Ctx)
{
    FDcPropertyDatum Datum;
    DC_TRY(Ctx.Writer->WriteDataEntry(FStructProperty::StaticClass(), Datum));

    FString ColorStr;
    DC_TRY(Ctx.Reader->ReadString(&ColorStr));
    FColor Color = FColor::FromHex(ColorStr);

    FColor* ColorPtr = (FColor*)Datum.DataPtr;
    *ColorPtr = Color;

    return DcOk();
}

Note how we retrieve the hex string, then parse it with FColor::FromHex.

Upon deserializing we'll need to register these pair of delegates to the FDcDeserializer.

// DataConfigTests/Private/DcTestBlurb.cpp
FDcDeserializer Deserializer;
DcSetupJsonDeserializeHandlers(Deserializer);
Deserializer.AddStructHandler(
    TBaseStructure<FColor>::Get(),
    FDcDeserializeDelegate::CreateStatic(DcExtra::HandlerColorDeserialize)
);

And then it's done! It would work recursively on FColor everywhere, like in UCLASS/USTRUCT members, in TArray/TSet and in TMap as key or values.

Note that DataConfig completely separate serialization and deserialization logic. To serialize FColor into #RRGGBBAA string one needs to implement a similar set of methods.

Debug Dump

DcAutomationUtils::DumpToLog() can dump a FDcPropertyDatum to a string representation, in which FDcPropertyDatum is simply a (FProperty, void*) fat pointer tuple that can represent anything in the property system:

• DcTestBlurb.cpp

// DataConfigTests/Private/DcTestBlurb.cpp
FVector Vec(1.0f, 2.0f, 3.0f);
FDcPropertyDatum VecDatum(TBaseStructure<FVector>::Get(), &Vec);

DcAutomationUtils::DumpToLog(VecDatum);

Output would be:

-----------------------------------------
# Datum: 'ScriptStruct', 'Vector'
<StructRoot> 'Vector'
|---<Name> 'X'
|---<Float> '1.000000'
|---<Name> 'Y'
|---<Float> '2.000000'
|---<Name> 'Z'
|---<Float> '3.000000'
<StructEnd> 'Vector'
-----------------------------------------

Additionally we wrapped this into gDcDebug that can be invoked in MSVC immediate window. Calling it during debug would dump into MSVC Output window:

// DataConfigCore/Public/DataConfig/Automation/DcAutomationUtils.h
struct DATACONFIGCORE_API FDcDebug
{
    FORCENOINLINE void DumpStruct(char* StructNameChars, void* Ptr);
    FORCENOINLINE void DumpObject(UObject* Obj);
    FORCENOINLINE void DumpDatum(void* DatumPtr);
};

/// Access `gDcDebugg` in MSVC immediate window:
///
/// - in monolith builds:
/// gDcDebug.DumpObject(Obj)
///
/// - in DLL builds prefix with dll name:
/// ({,,UE4Editor-DataConfigCore}gDcDebug).DumpObject(ObjPtr)

extern FDcDebug gDcDebug;

Here's an animated demo showing dumping the vector above during debug break in MSVC:

The full expression to evaluate is:

({,,UE4Editor-DataConfigCore}gDcDebug).DumpDatum(&VecDatum)

We need DLL name to locate gDcDebug in a non monolith build.

Integration

At the moment it supports these the engine versions below:

• UE 5.5
• UE 5.4
• UE 5.3
• UE 5.2
• UE 5.1
• UE 5.0
• UE 4.27
• UE 4.26
• UE 4.25

Integrate DataConfig Plugin

The easiest way to try out DataConfig is to add it as a plugin into your C++ project. In this section we'll walk through these steps.

Download DataConfig Plugin

The quickest way to try out DataConfig is to download the latest release at DataConfig releases page.

1. Download the zip files on the releases page. Note there're UE4 and UE5 plugin respectively.

2. Unzip it into your Unreal Engine project's Plugin folder. The layout should be like this:

<Your project root>
|- <Your project>.uproject
|- Content
|- Source
|- ...
|- Plugins
   |- DataConfig
      |- DataConfig.uplugin

3. Validate DataConfig plugin is integrated corrrectly.

Generate DataConfig Plugin for UE4/UE5

DataConfig now uses separated uplugin files for UE4 and UE5 so that we can try out new features in UE5 without dropping support for UE4. We bundled scripts to generate clean plugins for UE4 and UE5. This is how the DataConfig releases are built.

git clone https://github.com/slowburn-dev/DataConfig
# requires python 3.6+
python ./DataConfig/Misc/Scripts/make_dataconfig_ue4.py
python ./DataConfig/Misc/Scripts/make_dataconfig_ue54.py
python ./DataConfig/Misc/Scripts/make_dataconfig_ue5.py

DataConfigXX.uplugin

Ultimately we figured that we'll need multiple .uplugin files to support acrosss UE versions:

When packaging for each engine version we rename the one we want and delete the rest.

Manual Steps for UE5 Latest

1. Get a copy of DataConfig repository. Then copy ./DataConfig (where DataConfig.uplugin is located) into your project's Plugins directory.

2. Delete DataConfig4.uplugin, and other DataConfig5X.uplugin.

3. Delete DataConfig/Source/DataConfigHeadless folder. This step is crucial or you your project won't build.

Manual Steps for UE4

1. Get a copy of the repository. Then copy ./DataConfig (where DataConfig.uplugin is located) into your project's Plugins directory.

2. Delete DataConfig.uplugin, then rename DataConfig4.uplugin to DataConfig.uplugin.

3. Delete DataConfig/Source/DataConfigHeadless folder. This step is crucial or you your project won't build.

4. Additionally delete UE5 specific modules.

   • DataConfig/Source/DataConfigEngineExtra5

Validate integration

Follow these steps to ensure DataConfig is properly integrated into your project.

1. Restart your project. There should be a prompt to compile plugin sources. Confirm and wait until your project launches. Then open Settings -> Plugins you should see Data Config listed under Project Editor category.

   

2. The plugin comes with a set of tests. Open menu Window -> Developer Tools -> Session Frontend. Find and run the DataConfig tests and it should all pass.

   

Integrate DataConfigCore Module

DataConfig is packed into a plugin to bundle automation tests with a few assets. You're encouraged to integrate only the DataConfigCore module. It contains all core features with minimal dependencies.

Most projects should has a editor module already setup. In this section we'll go through the steps of integrating DataConfigCore and build it with the project's FooProjectEditor module.

1. Get a copy of this repository. Then copy DataConfig/Source/DataConfigCore into your project's Source directory.

2. Edit FooProjectEditor.Build.cs add add DataConfigCore as an extra module:

   using UnrealBuildTool;
   
   public class FooProjectEditor : ModuleRules
   {
       public FooProjectEditor(ReadOnlyTargetRules Target) : base(Target)
       {
           PublicDependencyModuleNames.AddRange(new string[] { 
   			//...
               "DataConfigCore",	// <- add this
               });
       }
   }
   

3. DataConfig needs to be explicitly initialized before use. Find FooProjectEditor module's start up and shut down methods and setup DataConfig accordingly.

   #include "DataConfig/DcEnv.h"
   #include "DataConfig/Automation/DcAutomationUtils.h"
   
   void FFooProjectEditorModule::StartupModule()
   {
       // ...
       DcStartUp(EDcInitializeAction::SetAsConsole);
       // dump a FVector to try it out
       FVector Vec(1.0f, 2.0f, 3.0f);
       FDcPropertyDatum VecDatum(TBaseStructure<FVector>::Get(), &Vec);
   
       DcAutomationUtils::DumpToLog(VecDatum);
   }
   
   void FFooProjectEditorModule::ShutdownModule()
   {
       // ...
   	DcShutDown();
   }
   

4. Rebuild the project and restart the editor. Open Output Log and you should be able to find the dump results (to filter use Categories -> None ).

   

You can refer to Module Setup for more detailed integration instructions.

Design

This page documents the overall design, goals and reasoning around DataConfig.

Rationale

At the time we started this project we were looking for a JSON parser that:

• Supports a relaxed JSON spec, i.e. comment and trailing comma.
• Supports custom deserialization logic, i.e. deserializes FColor from #RRGGBBAA.
• Supports UE instanced sub objects and polymorphism.

Ultimately we implemented all these in DataConfig. We also tried not to limit this to be a JSON parser but to provide a set of tools for reading-from and writing-to the property system.

If you are an Unreal Engine C++ developers that:

• Looking for alternative JSON reader/writer.
• Looking for MsgPack serializer.
• Looking for a textual configuration format.
• Thinking of implementing custom textual/binary format.
• Write code that deals with FProperty on a daily bases.

You should try DataConfig and it's highly likely DataConfig will fit into your solution.

Manifesto

• Deliver as a quality C++ source library.

  DataConfig should ship with no UI nor tooling code. Users are expected to integrate only DataConfigCore as a source module. We intentionally limit the scope of DataConfig to a "C++ Library". Our users should be proficient UE C++ programmers.

  • DataConfig should ship with good testing and documentation coverage.
  • DataConfig follows idiomatic UE C++ conventions and has no external dependencies.
  • DataConfigCore depends only on Core and CoreUObject and can be used in standalone Program targets.
  • DataConfig API are UObject free and stack friendly.
  • Built-in features serve as examples and sensible defaults. Users are expected to write on their own Reader/Writer/Handlers.

• Runtime performance is not our top priority.

  We expect users to use DataConfig in an offline, editor only scenario. In this use case we favor some other aspects over runtime performance:

  • Idiomatic. We follow Unreal Engine C++ coding conventions and keep core dependency to only Core and CoreUObject.
  • Friendly. When processing invalid data and invalid API usage DataConfig should not crash. It should fail explicitly with detailed context and diagnostics.
  • Small code size / fast compile time. DataConfig tries not to expose template API. TDcJsonReader is explicit instantiated with its definition in private files.
  • Light memory footprint. Our JSON parser does stream parsing and would not construct the loaded JSON document in memory at all.

  With that said DataConfig has decent performance that satisfies common runtime use cases. Do note the subtle differences when doing shipping builds.

• Works with whatever property system supports.

  The idea is that DataConfig supports everything that can be tagged with UCLASS/USTRUCT/UPROPERTY/UENUM macros, which covers the full data model of the property system.

  Fields such as weak object reference and delegates doesn't make much sense to be serialized into textual format. However it turns out supporting the full data model makes it suitable to some other tasks like debug dump and in-memory data wrangling.

  This also means that DataConfig focuses only on reading from and writing into C++ data structures. For example we do not have a DOM or object like API for JSON at all. The only use case DataConfig supports is to deserialize from JSON into native C++ objects.

Acknowledgement

• References serde.rs on API and the SerDe acronym.
• References FullSerializer and OdinSerializer on API.
• JSON parser/writer implementation and test cases references JSON for Modern C++ and RapidJson.
• Integrated nst/JSONTestSuite.
• Integrated kawanet/msgpack-test-suite.

Programming Guides

This section contains doc for programming DataConfig APIs.

DataConfig Data Model

Conceptually the DataConfig data model is defined by 3 C++ types:

1. EDcDataEntry - enum covers every possible data type.
2. FDcReader - methods to read from the data model.
3. FDcWriter - methods to write into the data model.

And that's it. The obvious missing thing is a DOM like object that you can random access and serialize into - we choose to not implement that and it's crucial to understand this to get to know how DataConfig works.

EDcDataEntry

The enum covers all possible types:

// DataConfigCore/Public/DataConfig/DcTypes.h
UENUM()
enum class EDcDataEntry : uint16
{
    None,

    Bool,
    Name,
    String,
    Text,
    Enum,

    Float,
    Double,

    Int8,
    Int16,
    Int32,
    Int64,

    UInt8,
    UInt16,
    UInt32,
    UInt64,

    //  Struct
    StructRoot,
    StructEnd,

    //  Class
    ClassRoot,
    ClassEnd,

    //  Map
    MapRoot,
    MapEnd,

    //  Array
    ArrayRoot,
    ArrayEnd,

    //  Set,
    SetRoot,
    SetEnd,

    //  Optional
    OptionalRoot,
    OptionalEnd,

    //  Reference
    ObjectReference,
    ClassReference,

    WeakObjectReference,
    LazyObjectReference,
    SoftObjectReference,
    SoftClassReference,
    InterfaceReference,

    //  Delegates
    Delegate,
    MulticastInlineDelegate,
    MulticastSparseDelegate,

    //  Field
    FieldPath,

    //  Extra
    Blob,

    //  Extension
    Extension,

    //  End
    Ended,
};

Most enumerators directly maps to a FProperty type:

• EDcDataEntry::Bool - FBoolProperty
• EDcDataEntry::Name - FNameProperty
• EDcDataEntry::String - FStrProperty
• EDcDataEntry::ArrayRoot/ArrayEnd- FArrayProperty

It should've covered all possible FProperty types. There're some additions that has no direct FProperty mapping:

• EDcDataEntry::None - It's maps null in JSON, and it's also used to explicitly represent null object reference.
• EDcDataEntry::Ended - It's a phony type that is returned when there's no more data or reader/writer is in a invalid state.
• EDcDataEntry::Blob - It's an extension to allow direct memory read/write from given fields.
• EDcDataEntry::Extension - It's an extension that allows additional data formats. MsgPack reader/writer uses this to support its extension data types.

FDcReader

FDcReader is the one and only way to read from DataConfig data model. For every enumerator in EDcDataEntry there's a member method on FDcReader to from it.

Here we set up a simple struct trying out the reader methods:

// DataConfigTests/Private/DcTestBlurb.h
USTRUCT()
struct FDcTestExampleSimple
{
	GENERATED_BODY()

	UPROPERTY() FString StrField;
	UPROPERTY() int IntField;
};

// DataConfigTests/Private/DcTestBlurb.cpp
FDcTestExampleSimple SimpleStruct;
SimpleStruct.StrField = TEXT("Foo Str");
SimpleStruct.IntField = 253;

Since we know exactly how the FDcTestExampleSimple looks like we can manually arrange the read calls:

// DataConfigTests/Private/DcTestBlurb.cpp
FDcPropertyReader Reader{FDcPropertyDatum(&SimpleStruct)};

DC_TRY(Reader.ReadStructRoot(&Struct));   // `FDcTestExampleSimple` Struct Root

    DC_TRY(Reader.ReadName(&FieldName));  // 'StrField' as FName
    DC_TRY(Reader.ReadString(&StrValue)); // "Foo STr"

    DC_TRY(Reader.ReadName(&FieldName));  // 'IntField' as FName
    DC_TRY(Reader.ReadInt32(&IntValue));  // 253

DC_TRY(Reader.ReadStructEnd(&Struct));    // `FDcTestExampleSimple` Struct Root

In the example above FDcReader behave like a iterator as each ReadXXX() call emits value and move the internal cursor into the next slot. In case we're reading a unknown structure, we can use FReader::PeekRead() to peek what's coming next.

FDcWriter

FDcWriter is the counter part of writing into the data config model. To write into the example instance above:

DC_TRY(Writer.WriteStructRoot(FDcStructStat{})); // `FDcTestExampleSimple` Struct Root

    DC_TRY(Writer.WriteName(TEXT("StrField")));      // 'StrField' as FName
    DC_TRY(Writer.WriteString(TEXT("Alt Str")));     // "Foo STr"

    DC_TRY(Writer.WriteName(TEXT("IntField")));      // 'IntField' as FName
    DC_TRY(Writer.WriteInt32(233));                  // 233

DC_TRY(Writer.WriteStructEnd(FDcStructStat{}));  // `FDcTestExampleSimple` Struct Root

There's also FDcWriter::PeekRead() to query whether it's possible to write given data type.

Sum Up

DataConfig provide FDcReader and FDcWriter to access the property system. It can be considered as a friendly alternative to the property system API. This is also how we support JSON and MsgPack in an uniform API.

Error Handling

Proper error handling is crucial to implement robust serialization as it needs to deal with unknown user input. DataConfig also provide diagnostic to help users quickly pin down common errors such as typos or missing colons in JSON. Here's an example:

# DataConfig Error: Enum name not found in enum type: EDcTestExampleEnum, Actual: 'Far'
- [JsonReader] --> <in-memory>4:25
   2 |    {
   3 |        "StrField" : "Lorem ipsum dolor sit amet",
   4 |        "EnumField" : "Far",
     |                           ^
   5 |        "Colors" : [
   6 |            "#FF0000FF", "#00FF00FF", "#0000FFFF"
- [PropertyWriter] Writing property: (FDcTestExampleStruct)$root.(EEDcTestExampleEnum)EnumField

Internally DataConfig applies a consistent error handling strategy across all API. User code is expected to follow along.

Returning FDcResult

The gist is that if a method can fail, it should return a FDcResult. It's a simple struct:

// DataConfigCore/Public/DataConfig/DcTypes.h
struct DATACONFIGCORE_API DC_NODISCARD FDcResult
{
    enum class EStatus : uint8
    {
        Ok,
        Error
    };

    EStatus Status;

    FORCEINLINE bool Ok() const
    {
        return Status == EStatus::Ok;
    }
};

//  See FDcReader's methods as example
// DataConfigCore/Public/DataConfig/Reader/DcReader.h
struct DATACONFIGCORE_API FDcReader
{
    //...
    virtual FDcResult ReadBool(bool* OutPtr);
    virtual FDcResult ReadName(FName* OutPtr);
    virtual FDcResult ReadString(FString* OutPtr);
    virtual FDcResult ReadText(FText* OutPtr);
    virtual FDcResult ReadEnum(FDcEnumData* OutPtr);
    //...
};

Then use DC_TRY to call these kinds of functions. The macro itself does early return when result is not Ok:

// DataConfigCore/Public/DataConfig/DcTypes.h
#define DC_TRY(expr)                        \
    do {                                    \
        ::FDcResult Ret = (expr);           \
        if (!Ret.Ok()) {                    \
            return Ret;                     \
        }                                   \
    } while (0)

//  Example of calling methods returning `FDcResult`
// DataConfigExtra/Private/DataConfig/Extra/Deserialize/DcDeserializeColor.cpp
template<>
FDcResult TemplatedWriteColorDispatch<EDcColorDeserializeMethod::WriterAPI>(const FColor& Color, FDcDeserializeContext& Ctx)
{
    DC_TRY(Ctx.Writer->WriteStructRoot(FDcStructStat{ TEXT("Color"), FDcStructStat::WriteCheckName }));

    DC_TRY(Ctx.Writer->WriteName(TEXT("B")));
    DC_TRY(Ctx.Writer->WriteUInt8(Color.B));

    DC_TRY(Ctx.Writer->WriteName(TEXT("G")));
    DC_TRY(Ctx.Writer->WriteUInt8(Color.G));

    DC_TRY(Ctx.Writer->WriteName(TEXT("R")));
    DC_TRY(Ctx.Writer->WriteUInt8(Color.R));

    DC_TRY(Ctx.Writer->WriteName(TEXT("A")));
    DC_TRY(Ctx.Writer->WriteUInt8(Color.A));

    DC_TRY(Ctx.Writer->WriteStructEnd(FDcStructStat{ TEXT("Color"), FDcStructStat::WriteCheckName }));

    return DcOk();
}

This pattern is similar to Outcome and std::expected except we give up using the return value. Return values should be passed through reference or pointers in function arguments.

Diagnostics

When implementing a method that returns FDcResult you have 2 options:

• Return DcOk() on succeed.
• Return DC_FAIL(<Catetory>, <ErrId>) on error.

Examples:

// DataConfigTests/Private/DcTestBlurb.cpp
FDcResult Succeed() {
    // succeed
    return DcOk();
}

FDcResult Fail() {
    // fail !
    return DC_FAIL(DcDCommon, Unexpected1) << "My Custom Message";
}

In the examples above DcDCommon and Unexpected1 are called error category and error id respectively. DcDCommon is a built-in error category:

// DataConfigCore/Public/DataConfig/Diagnostic/DcDiagnosticCommon.h
namespace DcDCommon
{
static const uint16 Category = 0x1;

enum Type : uint16
{
    //...
    Unexpected1,
};

} // namespace DcDCommon

// DataConfigCore/Private/DataConfig/Diagnostic/DcDiagnosticCommon.cpp
namespace DcDCommon
{
static FDcDiagnosticDetail _CommonDetails[] = {
    // ...
    { Unexpected1, TEXT("Unexpected: '{0}'") },
};

Note that we can pipe argument into the diagnostic. The diagnostic reported by invoking Fail() would be like:

* # DataConfig Error: Unexpected: 'My Custom Message'

Conclusion

DataConfig uses FDcResult, DC_TRY, DC_FAIL for error handling. It's lightweight and relatively easy to grasp. There's still some limitations though:

• FDcResult occupied the return position making passing value to parent a bit cumbersome.
• For now we always stop as the first error. There's no plan to support error recovery.

Some closing notes:

• Reported diagnostics get queued. You'll need to call FDcEnv::FlushDiags() to flush them to consumers.
• See DcDiagnosticExtra.h/cpp for how to register user category.
• See DcEditorExtraModule.cpp - FDcMessageLogDiagnosticConsumer for custom diagnostic handler and formatting.

Env

DataConfig put most global state into a stack of FDcEnv:

// DataConfigCore/Public/DataConfig/DcEnv.h
struct DATACONFIGCORE_API FDcEnv
{
    TArray<FDcDiagnostic> Diagnostics;

    TSharedPtr<IDcDiagnosticConsumer> DiagConsumer;

    TArray<FDcReader*> ReaderStack;
    TArray<FDcWriter*> WriterStack;

    bool bExpectFail = false;   // mute debug break

    FDcDiagnostic& Diag(FDcErrorCode InErr);

    void FlushDiags();

    FORCEINLINE FDcDiagnostic& GetLastDiag() 
    {
        checkf(Diagnostics.Num(), TEXT("<empty diagnostics>"));
        return Diagnostics.Last();
    }

    ~FDcEnv();
};

DataConfig needs explicit initializatioon before use. This is done through manually DcStartUp(). There's also a paired DcShutdown() that should be called when DataConfig isn't used anymore. Here's an example:

// DataConfigEngineExtra/Private/DcEngineExtraModule.cpp
void FDcEngineExtraModule::StartupModule()
{
    UE_LOG(LogDataConfigCore, Log, TEXT("DcEngineExtraModule module starting up"));
    DcRegisterDiagnosticGroup(&DcDExtra::Details);
    DcRegisterDiagnosticGroup(&DcDEngineExtra::Details);

    DcStartUp(EDcInitializeAction::Minimal);
    //...
}

void FDcEngineExtraModule::ShutdownModule()
{
    DcShutDown();
    //...
}

The active FDcEnv is accessed by calling global function DcEnv(). Inside the Env:

• Diagnostics: all diagnostics are flushed into env.
• DiagConsumer: diagnostic handler, format and print diagnostic to log or MessageLog or even on screen.
• ReaderStack/WriterStack: used to pass along reader/writer down the callstack. See FScopedStackedReader uses for example.
• ... and everything else.

You can use DcPushEnv() to create new env then destroy it calling DcPopEnv(). At this moment it's mostly used to handle reentrant during serialization. See FDcScopedEnv uses for examples.

Reader/Writer

FDcReader/FDcWriter defines the set of API for accessing DataConfig data model. Here's a check list for implementing a reader/writer:

1. Implement GetID()/ClassID() for RTTI.
2. Implement PeekRead()/PeekWriter() and selected set of ReadXXX()/WriteXXX().
3. Implement FormatDiagnostic() for error reporting.

You should look at builtin implementation for references. Here's some general rules and caveats:

• PeekRead()/PeekWrite() should act like it's side-effect free.

  This means that it's OK to call PeekRead()/PeekWrite() multiple times. In comparison access methods like ReadBool()/WriteBool() consume the data and alternate internal state. Note that under the hood it might do anything. Both returns FDcResult so the peek can fail. The reason behind this is that calling PeekRead()/PeekWrite() is totally optional. In FDcJsonReader::PeekRead() we do parsing and cache the parsed result to follow this convention.

• CastByID() does not respect inheritance hierarchy.

  We have this very minimal RTTI implemetantion that only allow casting to the exact type.

• Implement a subset of the data model.

  The API is designed in a way that it covers the whole Property System. It's also a super set that can express common formats like JSON/MsgPack. For example JSON don't have struct, class or set. It's actually the job of Serializer/Deserializer to convert between these subsets to the property system.

Builtin Reader/Writer

We have 3 major pairs of reader/writers:

• FDcPropertyReader/FDcPropertyWriter - Accesing Unreal Engin property system.
• FDcJsonReader/FDcJsonWriter - JSON support.
• FDcMsgPackReader/FDcMsgPackWriter - MsgPack support.

These are all talked about in details in the formats section. We'll go through other builtin Reader/Writers below.

FDcPipeVisitor and FDcPrettyPrintWriter

FDcPipeVisitor takes a FDcReader and a FDcWriter then start peek-read-write loop until it peeks EDcDataEntry::Ended from reader or an error happens

Then there's FDcPrettyPrintWriter that dumps everything that got write to it as string.

Combining these two we get a way to dump arbitrary FDcReader into a string!. This is how built-in debug dump features are implemented:

// DataConfigCore/Private/DataConfig/Automation/DcAutomationUtils.cpp
void DumpToOutputDevice(...)
{
    //...
    FDcPropertyReader PropReader(Datum);
    FDcPrettyPrintWriter PrettyWriter(Output);
    FDcPipeVisitor PrettyPrintVisit(&PropReader, &PrettyWriter);

    if (!PrettyPrintVisit.PipeVisit().Ok())
        ScopedEnv.Get().FlushDiags();
    //...
}

FDcPipeVisitor is a handy utility that we use it extensively through the code base for various cases. Try FDcPipeVisitor when you got a reader/writer pair.

There's also FNoopWriter takes every write and do nothing with it.

Composition

Reader/Writers can also be composited and nested:

• FDcWeakCompositeWriter is a writer that multiplex into a list of writers. You can combine an arbitrary writer with a FPrettyPrintWriter then get a tracing writer.
• FDcPutbackReader/FPutbackWriter: Reader/writers don't support lookahead. It can only peek next item's type but not value. This class is used to support limited lookahead by putting back read value. We'll see it being used in implementing custom deserializer handlers.

Conclusion

Implement new FDcReader/FDcWriter when you want to support a new file format. You can also write utility reader/writer that composite existing ones.

Serializer/Deserializer

Serializer/Deserializer are built on top of the data model to convert between external format and Unreal Engine property system.

Context

A company class to the deserializer is FDcDeserializeContext:

// DataConfigCore/Public/DataConfig/Deserialize/DcDeserializeTypes.h
struct DATACONFIGCORE_API FDcDeserializeContext
{
    //...
    FDcDeserializer* Deserializer;
    FDcReader* Reader;
    FDcPropertyWriter* Writer;
    //...
};

Note how it takes an FDcReader and a FPropertyWriter - we're deserializing arbitrary format into the property system.

The mirrored version for serializer is FDcSerializeContext.

// DataConfigCore/Public/DataConfig/Serialize/DcSerializeTypes.h
struct DATACONFIGCORE_API FDcSerializeContext
{
    //...
    FDcSerializer* Serializer = nullptr;
    FDcPropertyReader* Reader = nullptr;
    FDcWriter* Writer = nullptr;
    //...
};

Note how it takes an FDcWriter and a FDcPropertyReader - we're serializing data from the property system to arbitrary format.

Since serializer and deserializer have extremely similar APIs, we're showing examples using deserializer below from here.

Handlers

Say that we're deserializing a JSON object into a USTRUCT instance. The FDcJsonReader implements ReadMapRoot()/ReadMapEnd() but doesn't have ReadStructRoot()/ReadStructEnd(). To make the conversion we basically want to map ReadMap() and calls into WriteStruct() calls. This is where handlers come into play:

// DataConfigCore/Public/DataConfig/SerDe/DcDeserializeCommon.inl
FDcResult DcHandlerDeserializeMapToStruct(FDcDeserializeContext& Ctx)
{
    FDcStructAccess Access;
    DC_TRY(Ctx.Reader->ReadMapRoot());
    DC_TRY(Ctx.Writer->WriteStructRootAccess(Access));

    EDcDataEntry CurPeek;
    while (true)
    {
        DC_TRY(Ctx.Reader->PeekRead(&CurPeek));
        if (CurPeek == EDcDataEntry::MapEnd)
            break;

        FName FieldName;
        DC_TRY(Ctx.Reader->ReadName(&FieldName));
        DC_TRY(Ctx.Writer->WriteName(FieldName));

        DC_TRY(DcDeserializeUtils::RecursiveDeserialize(Ctx));
    }

    DC_TRY(Ctx.Reader->ReadMapEnd());
    DC_TRY(Ctx.Writer->WriteStructEnd());

    return DcOk();
}

// DataConfigCore/Private/DataConfig/Deserialize/Handlers/Json/DcJsonCommonDeserializers.cpp
FDcResult HandlerStructRootDeserialize(FDcDeserializeContext& Ctx)
{
    return DcHandlerDeserializeMapToStruct(Ctx);
}

Note that Ctx.Reader is a FDcReader that can be any derived class, while Ctx.Writer is always a FDcPropertyWriter. Deserialize handlers have an uniform signature:

using FDcDeserializeDelegateSignature = FDcResult(*)(FDcDeserializeContext& Ctx);

Deserializer Setup

Note how deserialize handler above doesn't specify when it should be invoked. These info are described in FDcDeserializer:

// DataConfigCore/Public/DataConfig/Deserialize/DcDeserializer.h
struct DATACONFIGCORE_API FDcDeserializer : public FNoncopyable
{
    //...
    FDcResult Deserialize(FDcDeserializeContext& Ctx);

    void AddDirectHandler(FFieldClass* PropertyClass, FDcDeserializeDelegate&& Delegate);
    void AddDirectHandler(UClass* PropertyClass, FDcDeserializeDelegate&& Delegate);
    void AddPredicatedHandler(FDcDeserializePredicate&& Predicate, FDcDeserializeDelegate&& Delegate);
    //...
};

Comparing to FDcDeserializeContext, which describes data needed for a single run, FDcDeserializer contains info on what handlers to execute. Deserializer can also be reused across multiple runs.

We use "direct handlers" to cover common cases:

// DataConfigCore/Private/DataConfig/Deserialize/DcDeserializerSetup.cpp
Deserializer.AddDirectHandler(FArrayProperty::StaticClass(), FDcDeserializeDelegate::CreateStatic(HandlerArrayDeserialize));
Deserializer.AddDirectHandler(FSetProperty::StaticClass(), FDcDeserializeDelegate::CreateStatic(HandlerSetDeserialize));
Deserializer.AddDirectHandler(FMapProperty::StaticClass(), FDcDeserializeDelegate::CreateStatic(HandlerMapDeserialize));

These basically says that "when running into array, set, map properties, use these provided handlers".

Then there's "struct handler" that uses a UStruct that maps a specific class/struct to a handler:

// DataConfigCore/Private/DataConfig/Deserialize/DcDeserializerSetup.cpp
Deserializer.AddStructHandler(TBaseStructure<FGuid>::Get(), FDcDeserializeDelegate::CreateStatic(HandlerGuidDeserialize));
Deserializer.AddStructHandler(TBaseStructure<FColor>::Get(), FDcDeserializeDelegate::CreateStatic(HandlerColorDeserialize));
Deserializer.AddStructHandler(TBaseStructure<FDateTime>::Get(), FDcDeserializeDelegate::CreateStatic(HandlerDateTimeDeserialize));

This means "when running into a FGuid, use these attached handlers". This is run before direct handlers.

Then we have "predicated handler" that get tested very early. This is how we allow custom conversion logic setup for very specific class:

// DataConfigCore/Private/DataConfig/Deserialize/DcDeserializerSetup.cpp
EDcDeserializePredicateResult PredicateIsScalarArrayProperty(FDcDeserializeContext& Ctx)
{
    FProperty* Prop = CastField<FProperty>(Ctx.TopProperty().ToField());
    return Prop && Prop->ArrayDim > 1 && !Ctx.Writer->IsWritingScalarArrayItem()
        ? EDcDeserializePredicateResult::Process
        : EDcDeserializePredicateResult::Pass;
}

// ...
Deserializer.AddPredicatedHandler(
    FDcDeserializePredicate::CreateStatic(PredicateIsScalarArrayProperty),
    FDcDeserializeDelegate::CreateStatic(HandlerArrayDeserialize)
);

By convention the current deserializing property can be retrieved with Ctx.TopProperty(). PredicateIsScalarArrayProperty here checks if it's wring a scalar array with non 1 dimension, if that's the case it would need to treat it like an array.

Note that all registered predicate handler is iterated through on every property, then proceed to handler on first success match or fall through to struct/direct handlers when no match. Use it only when struct/direct handlers doesn't fit.

To recap:

Serializer Setup

Serializer has exactly the same API as deserializer and the semantics are all the same.

Sum Up

Serializer/Deserializer are built on top of Reader/Writer, to convert between Unreal Engine property system and external data formats.

• FDcSerializeContext/FDcDeserializeContext contains data.
• FDcSerializer/FDcDeserializer contains setup.
• Implement FDcDeserializeDelegate/FDcSerializeDelegate and FDcDeserializePredicate/FDcSerializePredicate pair for custom conversion logic.

Formats

This section contains documentation for supported formats.

JSON

JSON is likely the most popular data interchange format. Unreal Engine already supports it with JsonUtilities and some related modules. We provide an alternative implementation along with DataConfig.

JSON Reader

FDcJsonReader is the DataConfig JSON reader:

// DataConfigTests/Private/DcTestBlurb.cpp
FString Str = TEXT(R"(
    {
        "Str":    "Fooo",
        "Number": 1.875,
        "Bool":   true
    } 
)");

FDcJsonReader Reader(Str);

//  calling read methods
FString KeyStr;
FString GotStr;
double GotNumber;
bool GotBool;

DC_TRY(Reader.ReadMapRoot());

    DC_TRY(Reader.ReadString(&KeyStr));
    DC_TRY(Reader.ReadString(&GotStr));

    DC_TRY(Reader.ReadString(&KeyStr));
    DC_TRY(Reader.ReadDouble(&GotNumber));

    DC_TRY(Reader.ReadString(&KeyStr));
    DC_TRY(Reader.ReadBool(&GotBool));

DC_TRY(Reader.ReadMapEnd());

//  validate results
check(GotStr == TEXT("Fooo"));
check(GotNumber == 1.875);
check(GotBool == true);

In the example above we deserialized a JSON object from string. The first and last calls are ReadMapRoot and ReadMapEnd, which are also used to read Unreal's TMap properties. The difference is that UE's TMap is strictly typed while JSON object values can have arbitrary type. This means that if you use FDcPipeVisitor to pipe a FDcJsonReader into a FDcPropertyWriter it won't work.

Remember that DataConfig data model is designed to support conversion between subsets within the data model. As long as you can use FDcReader/FDcWriter API to describe the format you want to serialize you're good to go. Mapping and conversion between these different shapes of reader/writers are handled by deserializers.

Some additional caveats:

• Similar to stock TJsonReader, we provide TDcJsonReader with 2 specializations:
  • Usually you just use FDcJsonReader that reads from FString, TCHAR*.
  • Under the hood there're FDcAnsiJsonReader that reads ANSICHAR string and FDcWideJsonReader that reads WIDECHAR string.
• We support a relaxed superset of JSON:
  • Allow C Style comments, i.e /* block */ and // line .
  • Allow trailing comma, i.e [1,2,3,], .
  • Allow non object root. You can put a list as the root, or even string, numbers.
• Number parsing are delegated to Unreal's built-ins to reduce dependencies. We might change this in the future.
  • Parse numbers: TCString::Atof/Strtoi/Strtoi64

JSON Writer

FDcJsonWriter is the DataConfig JSON writer:

// DataConfigTests/Private/DcTestBlurb.cpp
FDcJsonWriter Writer;

DC_TRY(Writer.WriteMapRoot());

    DC_TRY(Writer.WriteString(TEXT("Str")));
    DC_TRY(Writer.WriteString(TEXT("Fooo")));

    DC_TRY(Writer.WriteString(TEXT("Number")));
    DC_TRY(Writer.WriteFloat(1.875f));

    DC_TRY(Writer.WriteString(TEXT("Bool")));
    DC_TRY(Writer.WriteBool(true));

DC_TRY(Writer.WriteMapEnd());
Writer.Sb.Append(TCHAR('\n'));

FString Str = TEXT(R"(
    {
        "Str" : "Fooo",
        "Number" : 1.875,
        "Bool" : true
    }
)");

//  validate results
check(DcReindentStringLiteral(Str) == Writer.Sb.ToString());
return DcOk();

• Similar to stock TJsonWriter, we provide TDcJsonWriter with 2 specializations:
  • Usually you just use FDcJsonWriter that writes FString, TCHAR*.
  • Under the hood there're FDcAnsiJsonWriter that writes ANSICHAR string and FDcWideJsonWriter that writes WIDECHAR string.
• It takes a Config object that specify formatting settings like indentation size and new lines.
  • FDcPrettyJsonWriter is a type alias that formats indented JSON.
  • FDcCondensedJsonWriter is a type alias that format single line, condensed output.
• FDcJsonWriter owns the output string buffer, in FDcJsonWriter::Sb.
  • By writing to a single writer and appending a new line after each serialization, we can output NDJSON.
  • Our JSON reader is also flexible enough to directly load NDJSON. See corpus benchmark.

JSON Serialize/Deserialize

DataConfig bundles a set of JSON serialize and deserialize handlers, which are all roundtrip-able:

// DataConfigTests/Private/DcTestBlurb.cpp
#include "DataConfig/Deserialize/DcDeserializerSetup.h"

// ...
//  create and setup a deserializer
FDcDeserializer Deserializer;
DcSetupJsonDeserializeHandlers(Deserializer);

//  create and setup a serializer
FDcSerializer Serializer;
DcSetupJsonSerializeHandlers(Serializer);

Schema

JSON types get mapped into DataConfig data model in a very unsurprising way.

Here's an example:

// DataConfigTests/Private/DcTestDeserialize.cpp
FString Str = TEXT(R"(
    {
        "BoolField" : true,
        "NameField" : "AName",
        "StringField" : "AStr",
        "TextField" : "AText",
        "EnumField" : "Tard",

        "FloatField" : 17.5,
        "DoubleField" : 19.375,

        "Int8Field" : -43,
        "Int16Field" : -2243,
        "Int32Field" : -23415,
        "Int64Field" : -1524523,

        "UInt8Field" : 213,
        "UInt16Field" : 2243,
        "UInt32Field" : 23415,
        "UInt64Field" : 1524523,
    }
)");

//  deserialized equivelent

FDcTestStruct1 Expect;
Expect.BoolField = true;
Expect.NameField = TEXT("AName");
Expect.StringField = TEXT("AStr");
Expect.TextField = FText::FromString(TEXT("AText"));
Expect.EnumField = EDcTestEnum1::Tard;

Expect.FloatField = 17.5f;
Expect.DoubleField = 19.375;

Expect.Int8Field = -43;
Expect.Int16Field = -2243;
Expect.Int32Field = -23415;
Expect.Int64Field = -1524523;

Expect.UInt8Field = 213;
Expect.UInt16Field = 2243,
Expect.UInt32Field = 23415;
Expect.UInt64Field = 1524523;

Map

JSON only allow string as object/mapping keys, while in UE TMap<> can use any type. When doing serialization TMap<FString/FName/FText,(TValue)> types would be directly converted to a JSON object:

// DataConfigTests/Public/DcTestDeserialize.h
USTRUCT()
struct FDcTestStruct3
{
    // ...
    UPROPERTY() TMap<FString, FString> StringMap;
};

// DataConfigTests/Public/DcTestDeserialize.cpp
{
    // ...
    "StringMap" : {
        "One": "1",
        "Two": "2", 
        "Three": "3",
    },
}

For other key types it would be serialized as an array of objects:

// DataConfigTests/Public/DcTestSerDe.h
USTRUCT()
struct FDcTestStructMaps
{
    // ...
    UPROPERTY() TMap<FColor, FString> ColorKeyMap;
    UPROPERTY() TMap<EDcTestEnumFlag, FString> EnumFlagsMap;
};

// DataConfigTests/Public/DcTestDeserialize.cpp
{
    "ColorKeyMap" : {
        "#FF0000FF" : "Red",
        "#00FF00FF" : "Green",
        "#0000FFFF" : "Blue"
    },
    "EnumFlagsMap" : [
        {
            "$key" : [],
            "$value" : "None"
        },
        {
            "$key" : [
                "One",
                "Three"
            ],
            "$value" : "One | Three"
        },
        {
            "$key" : [
                "Five"
            ],
            "$value" : "Five"
        }
    ]
}

Enum Flags

UENUM that get marked with Bitflags meta are deserialized from a list of strings:

// DataConfigTests/Public/DcTestDeserialize.h
UENUM(meta = (Bitflags))
enum class EDcTestEnumFlag :uint32
{
    None    = 0,
    One     = (1 << 0),
    Two     = (1 << 1),
    //...
};
ENUM_CLASS_FLAGS(EDcTestEnumFlag);

// DataConfigTests/Private/DcTestDeserialize.cpp
FString Str = TEXT(R"(
    {
        "EnumFlagField1" : [],
        "EnumFlagField2" : ["One", "Three", "Five"],
    }
)");

//  deserialized equivelent
FDcTestStructEnumFlag1 Expect;
Expect.EnumFlagField1 = EDcTestEnumFlag::None;
Expect.EnumFlagField2 = EDcTestEnumFlag::One | EDcTestEnumFlag::Three | EDcTestEnumFlag::Five;

Sub Objects

By default we treat UOBJECT marked with DefaultToInstanced, EditInlineNew and UPROPERTY marked with Instanced as sub object. In this case we'll actually instantiate new object during deserialization, using Ctx.TopObject() as parent:

// DataConfigTests/Public/DcTestProperty.h
UCLASS(BlueprintType, EditInlineNew, DefaultToInstanced)
class UDcBaseShape : public UObject
{
    //...
    UPROPERTY() FName ShapeName;
};

UCLASS()
class UDcShapeBox : public UDcBaseShape
{
    //...
    UPROPERTY() float Height;
    UPROPERTY() float Width;
};

UCLASS()
class UDcShapeSquare : public UDcBaseShape
{
    //...
    UPROPERTY() float Radius;
};

// DataConfigTests/Public/DcTestDeserialize.h
USTRUCT()
struct FDcTestStructShapeContainer1
{
    GENERATED_BODY()

    UPROPERTY() UDcBaseShape* ShapeField1;
    UPROPERTY() UDcBaseShape* ShapeField2;
    UPROPERTY() UDcBaseShape* ShapeField3;
}USTRUCT()
struct FDcEditorExtraTestObjectRefs1
{
    GENERATED_BODY()

    UPROPERTY() UObject* ObjField1;
    UPROPERTY() UObject* ObjField2;
    UPROPERTY() UObject* ObjField3;
    UPROPERTY() UObject* ObjField4;
};

// DataConfigTests/Private/DcTestDeserialize.cpp
FString Str = TEXT(R"(
    {
        "ShapeField1" :  {
            "$type" : "DcShapeBox",
            "ShapeName" : "Box1",
            "Height" : 17.5,
            "Width" : 1.9375
        },
        "ShapeField2" : {
            "$type" : "DcShapeSquare",
            "ShapeName" : "Square1",
            "Radius" : 1.75,
        },
        "ShapeField3" : null
    }
)");

//  deserialized equivelent
UDcShapeBox* Shape1 = NewObject<UDcShapeBox>();
Shape1->ShapeName = TEXT("Box1");
Shape1->Height = 17.5;
Shape1->Width = 1.9375;
Expect.ShapeField1 = Shape1;

UDcShapeSquare* Shape2 = NewObject<UDcShapeSquare>();
Shape2->ShapeName = TEXT("Square1");
Shape2->Radius = 1.75;
Expect.ShapeField2 = Shape2;

Expect.ShapeField3 = nullptr;

Note that criteria for sub object selection can be easily overridden with a new deserialize predicate or alternative FDcPropertyConfig when constructing the reader.

Also see AnyStruct, InlineStruct and InstancedStruct for lighter weight alternatives.

Object and Class Reference

We support multiple ways of referencing a UObject in memory or serialized on disk:

// DataConfigEditorExtra/Private/DataConfig/EditorExtra/Tests/DcTestDeserializeEditor.h
USTRUCT()
struct FDcEditorExtraTestObjectRefs1
{
    GENERATED_BODY()

    UPROPERTY() UObject* ObjField1;
    UPROPERTY() UObject* ObjField2;
    UPROPERTY() UObject* ObjField3;
    UPROPERTY() UObject* ObjField4;
};

// DataConfigEditorExtra/Private/DataConfig/EditorExtra/Tests/DcTestDeserializeEditor.cpp
FString Str = TEXT(R"(
    {
        "ObjField1" : "DcEditorExtraNativeDataAsset'/DataConfig/DcFixture/DcTestNativeDataAssetAlpha.DcTestNativeDataAssetAlpha'",
        "ObjField2" : "/DataConfig/DcFixture/DcTestNativeDataAssetAlpha",
        "ObjField3" : 
        {
            "$type" : "DcEditorExtraNativeDataAsset",
            "$path" : "/DataConfig/DcFixture/DcTestNativeDataAssetAlpha"
        },
        "ObjField4" : null,
    }
)");

//  deserialized equivelent
UDcEditorExtraNativeDataAsset* DataAsset = Cast<UDcEditorExtraNativeDataAsset>(StaticLoadObject(
    UDcEditorExtraNativeDataAsset::StaticClass(),
    nullptr,
    TEXT("/DataConfig/DcFixture/DcTestNativeDataAssetAlpha"),
    nullptr
));

Expect.ObjField1 = DataAsset;
Expect.ObjField2 = DataAsset;
Expect.ObjField3 = DataAsset;
Expect.ObjField4 = nullptr;

In the example above, ObjField1 uses the reference string that can be retrieved in editor context menu:

For ObjField2/ObjField3 relative path to the uasset is used, but without file name suffix.

We also support class reference fields of TSubclassOf<>s:

// DataConfigTests/Private/DcTestDeserialize.h
USTRUCT()
struct FDcTestStructSubClass1
{
    GENERATED_BODY()

    UPROPERTY() TSubclassOf<UStruct> StructSubClassField1;
    UPROPERTY() TSubclassOf<UStruct> StructSubClassField2;
    UPROPERTY() TSubclassOf<UStruct> StructSubClassField3;
};

// DataConfigTests/Private/DcTestDeserialize.cpp
FString Str = TEXT(R"(
    {
        "StructSubClassField1" : null,
        "StructSubClassField2" : "ScriptStruct",
        "StructSubClassField3" : "DynamicClass",
    }
)");

//  deserialized equivelent
FDcTestStructSubClass1 Expect;
Expect.StructSubClassField1 = nullptr;
Expect.StructSubClassField2 = UScriptStruct::StaticClass();
Expect.StructSubClassField3 = UDynamicClass::StaticClass();

Note that these do not support Blueprint classes. The direct reason is that Blueprint depends on Engine module and we'd like not to take dependency on in DataConfigCore.

We do have an example that supports Blueprint classes, see DataConfigEditorExtra - DcDeserializeBPClass.h/cpp

Soft Lazy as String

DcSetupJsonSerializeHandlers()/DcSetupJsonDeserializeHandlers() accepts an enum to setup alternative handlers. For now StringSoftLazy branch would setup special FSoftObjectProperty/FLazyObjectProperty handlers that directly serialize these into string. Comparing to this the default setup would always resolve the indirect reference into memory, which maybe isn't always desirable.

// DataConfigTests/Private/DcTestBlurb.cpp
FDcTestStructRefs1 Source{};
UObject* TestsObject = StaticFindObject(UObject::StaticClass(), nullptr, TEXT("/Script/DataConfigTests"));

Source.SoftField1 = TestsObject;
Source.LazyField1 = TestsObject;

FDcJsonWriter Writer;
DC_TRY(DcAutomationUtils::SerializeInto(&Writer, FDcPropertyDatum(&Source),
[](FDcSerializeContext& Ctx) {
    DcSetupJsonSerializeHandlers(*Ctx.Serializer, EDcJsonSerializeType::StringSoftLazy);
}, DcAutomationUtils::EDefaultSetupType::SetupNothing));

//  serialized result
{
    // ...
    "SoftField1" : "/Script/DataConfigTests",
    "SoftField2" : "",
    "LazyField1" : "C851179E-45A51045-0006AE91-F9B16EC0",
    "LazyField2" : "00000000-00000000-00000000-00000000"
}

Override Config

Sometimes we want to write a single line array/object within a pretty print JSON writer. To implement this we added mechanic to apply an override config in scope:

// DataConfigTests/Private/DcTestJSON2.cpp
DC_TRY(Writer.WriteMapRoot());
{
    DC_TRY(Writer.WriteName("Obj1"));

    FDcScopedTryUseJSONOverrideConfig ScopedOverrideConfig(&Writer);
    DC_TRY(Writer.WriteMapRoot());

        DC_TRY(Writer.WriteString("Foo"));
        DC_TRY(Writer.WriteString("Bar"));

        DC_TRY(Writer.WriteString("Alpha"));
        DC_TRY(Writer.WriteString("Beta"));

    DC_TRY(Writer.WriteMapEnd());
}
/// ...

With FDcScopedTryUseJSONOverrideConfig the following object would be written on a single line:

{
    "Obj1" : {"Foo" : "Bar", "Alpha" : "Beta"},
    // ...
}

This is how UE Core Types serializer handlers are implemented.

Caveats

Here're some closing notes:

• For meta fields like $type it must be the first member, meaning object fields are order dependent. This means that the JSON we're supporting is a super set of standard JSON spec (again).

• Bundled serializers and deserializers are designed to be roundtrip-able. For example in test DataConfig.Core.RoundTrip.JsonRoundtrip1_Default:

  1. Serialize UDcTestRoundtrip1 instance Source into JSON.
  2. Then deserialize JSON above into instance Dest.
  3. Deep-compare Source and Dest. If they're equal them we say it's a roundtrip.

  Note that we carefully picked float and doubles in the test case, as it's tricky to support floating point roundtrip. We might consider supporting this with alternative float parse and format routines.

• There're many data types that can not be deserialized from JSON, for example Delegate/WeakObjectReference. Remember that you always have the option to override or selectively enable handlers to support additional properties that make sense in your context. See DcSetupJsonDeserializeHandlers() body on how handlers are registered. You can skip this method and select the ones you want and provide additional handlers.

• The JSON handlers are designed to NOT read anything during the deserialization. This is crucial since USTRUCT can contain uninitialized fields. For example:

  // DataConfigTests/Private/DcTestBlurb.cpp
  FString Str = TEXT(R"(
      {
          // pass
      } 
  )");
  FDcJsonReader Reader(Str);
  
  FDcTestExampleSimple Dest;
  FDcPropertyDatum DestDatum(&Dest);
  
  DC_TRY(DcAutomationUtils::DeserializeJsonInto(&Reader, DestDatum));
  
  check(Dest.StrField.IsEmpty());
  //  but Dest.IntField contains uninitialized value
  DcAutomationUtils::DumpToLog(DestDatum);
  
  // dump results
  <StructRoot> 'DcTestExampleSimple'
  |---<Name> 'StrField'
  |---<String> ''
  |---<Name> 'IntField'
  |---<Int32> '1689777552' // <- arbitrary value
  <StructEnd> 'DcTestExampleSimple'
  

  This would cause trouble when you try read a pointer field during deserialization. Remember that primitive fields might be uninitialized during deserialization when implementing your own handlers.

• One interesting trait of the pull/push styled API is that FDcJsonReader does not preemptively parse number into double and convert it to int/float later on. When reading a number token it would do the number parsing at call site. If ReadIntX() is called then the number is parsed as integer. If ReadFloat()/ReadDouble() is called the token will be parsed as floating point.

• FDcJsonReader::FinishRead() can be used to check JSON is fully consumed and sound (no trailing tokens, object/array fully closed). This optional by design and not part of FDcReader/FDcWriter API and would make reading more flexible. See NDJSON for more.

MsgPack

MsgPack is an popular binary serialization format. It can be considered as a binary superset of JSON. Unreal Engine already supports Cbor module which is a format which is very similar to MsgPack.

We choose to implement MsgPack as we're more familiar with it and also providing an alternative.

MsgPack Reader/Writer

For the most part MsgPack reader/writer works just like their JSON counterpart. There're just a few additional data types that belongs to this:

Binary

MsgPack directly supports bin format family which directly maps to EDcDataEntry::Blob:

// DataConfigTests/Private/DcTestBlurb.cpp
DC_TRY(Writer.WriteBlob({Bytes, 0}));
TArray<uint8> Arr = {1,2,3,4,5};

FDcMsgPackWriter Writer;
DC_TRY(Writer.WriteBlob(FDcBlobViewData::From(Arr)));
auto& Buf = Writer.GetMainBuffer();

FDcMsgPackReader Reader(FDcBlobViewData::From(Buf));
FDcBlobViewData Blob;
DC_TRY(Reader.ReadBlob(&Blob));

check(Blob.Num == 5);
check(FPlatformMemory::Memcmp(Arr.GetData(), Blob.DataPtr, Blob.Num) == 0);

Extension

MsgPack also supports ext format family which is basically fixed size binary data with a header:

// DataConfigTests/Private/DcTestBlurb.cpp
FDcMsgPackWriter Writer;
DC_TRY(Writer.WriteFixExt2(1, {2, 3}));
auto& Buf = Writer.GetMainBuffer();

FDcMsgPackReader Reader(FDcBlobViewData::From(Buf));
uint8 Type;
FDcBytes2 Bytes;
DC_TRY(Reader.ReadFixExt2(&Type, &Bytes));

check(Type == 1);
check(Bytes.Data[0] == 2);
check(Bytes.Data[1] == 3);

MsgPack Serialize/Deserialize

MsgPack handlers also support multiple setup types:

// DataConfigCore/Public/DataConfig/Serialize/DcSerializerSetup.h
enum class EDcMsgPackSerializeType
{
    Default,
    StringSoftLazy, // Serialize Soft/Lazy references as string
    InMemory,       // Serialize pointer/enum/FName etc as underlying integer values
};

Persistent handlers

The Default and StringSoftLazy options would setup a set of handlers that behaves like their JSON counterparts.

We have a "Property -> Json -> MsgPack -> Json -> Property" roundtrip test setup in DataConfig.Core.RoundTrip.Property_Json_MsgPack_Json_Property test.

In Memory handlers

This is a special set of handlers that only makes sense for binary formats. For example pointers are serialized as memory addresses.

With these handlers all data types can be serialized. Note that serializing stuff as memory address isn't always what you want. These are provided as soft of a reference on how to access various data.

Property

The property system is at the heart of DataConfig: serializer read from property system, while deserializer writes into it. This is directly reflected in the types:

// DataConfigCore/Public/DataConfig/Serialize/DcSerializeTypes.h
struct DATACONFIGCORE_API FDcSerializeContext
{
    // ...
    FDcPropertyReader* Reader = nullptr;
    FDcWriter* Writer = nullptr;
};

// DataConfigCore/Public/DataConfig/Deserialize/DcDeserializeTypes.h
struct DATACONFIGCORE_API FDcDeserializeContext
{
    FDcReader* Reader = nullptr;
    FDcPropertyWriter* Writer = nullptr;
};

Property Datum and Reader/Writer

• DcPropertyDatum.h
• DcPropertyReader.h
• DcPropertyWriter.h

Property reader/writer are usually constructed from a FDcPropertyDatum, which is a "fat pointer" that represents an entry in the property system.

// DataConfigCore/Public/DataConfig/Property/DcPropertyDatum.h
struct DATACONFIGCORE_API FDcPropertyDatum
{
    FFieldVariant Property;
    void* DataPtr;
    //...
}

Recall that DataConfig data model expands to a set of Read/Write calls. In the example below we setup a FDcPropertyReader with a simple struct and dump it out:

//  DataConfigTests/Private/DcTestBlurb2.h
USTRUCT()
struct FDcTestExample2
{
    GENERATED_BODY()

    UPROPERTY() FString StrField;
    UPROPERTY() FString StrArrField[3];

    UPROPERTY() UDcBaseShape* InlineField;
    UPROPERTY() UDcTestClass1* RefField;
    // ...
};

//  DataConfigTests/Private/DcTestBlurb2.cpp
FDcTestExample2 Value;
Value.MakeFixture();

FDcPropertyDatum Datum(&Value);
FDcPropertyReader Reader(Datum);
DcAutomationUtils::DumpToLog(&Reader);

The results are like this:

<StructRoot> 'DcTestExample2'
|---<Name> 'StrField'
|---<String> 'Foo'
|---<Name> 'StrArrField'
|---<ArrayRoot>
|   |---<String> 'One'
|   |---<String> 'Two'
|   |---<String> 'Three'
|---<ArrayEnd>
|---<Name> 'InlineField'
|---<ClassRoot> 'DcShapeSquare'
|   |---<Name> 'Radius'
|   |---<Float> '2.000000'
|   |---<Name> 'ShapeName'
|   |---<Name> 'MyBox'
|---<ClassEnd> 'DcBaseShape'
|---<Name> 'RefField'
|---<ClassRoot> 'DcTestClass1'
|   |---<ObjectReference> '337' 'DcTestClass1_0'
|---<ClassEnd> 'DcTestClass1'
<StructEnd> 'DcTestExample2'

Some caveats:

• Note that DcTestExample2::StrArrField is dumped into a array. Note that DataConfig can't distinguish between Foo and Foo[1] that's a 1 element array. It would be read as a normal field.

• Note that DcTestExample2::InlineField is expanded and DcTestExample2::RefField is read as an reference. This is determined by FDcPropertyConfig::ExpandObjectPredicate, which by default expands:

  • Field marked with UPROPERTY(Instanced)
  • Class marked with UCLASS(DefaultToInstanced, EditInlineNew)

Configuring with FDcPropertyConfig

• DcPropertyTypes.h

Property reader/writer accepts FDcPropertyConfig class for customizing behaviors. By default we've implemented DcSkip metadata that you can mark on given reader/writer.URPOPERTY() and the field would be skipped by given reader/writer:

//  DataConfigTests/Private/DcTestProperty3.h
USTRUCT()
struct FDcTestMeta1
{
    GENERATED_BODY()

    UPROPERTY(meta = (DcSkip)) int SkipField1;
};

Though that DcSkip behavior is enabled by default, you can override this with a custom FPropertyConfig instance. Here's an example of processing only fields with DcTestSerialize meta:

//  DataConfigTests/Private/DcTestProperty4.h
USTRUCT()
struct FDcTestSerializeMeta1
{
	GENERATED_BODY()

	UPROPERTY(meta = (DcTestSerialize)) int SerializedField;
	UPROPERTY() int IgnoredField;
};

//  DataConfigTests/Private/DcTestProperty4.cpp
FDcPropertyConfig Config;
//  only process fields that has `DcTestSerialize` meta
Config.ProcessPropertyPredicate = FDcProcessPropertyPredicateDelegate::CreateLambda([](FProperty* Property)
{
    const static FName TestSerializeMeta = FName(TEXT("DcTestSerialize"));
    return DcPropertyUtils::IsEffectiveProperty(Property)
        && Property->HasMetaData(TestSerializeMeta);
});
Config.ExpandObjectPredicate = FDcExpandObjectPredicateDelegate::CreateStatic(DcPropertyUtils::IsSubObjectProperty);
Ctx.Reader->SetConfig(Config);

Pipe Property Handlers

• DcPropertyPipeSerializers.h
• DcPropertyPipeSerializers.cpp
• DcPropertyPipeDeserializers.h
• DcPropertyPipeDeserializers.cpp

There's a set of deserialize handlers in DcPropertyPipeHandlers namespace. It's used to roundtripping property system objects.

Simply speaking it's equivalent to doing a FDcPipeVisitor pipe visit.

//  DataConfigTests/Private/DcTestBlurb.cpp
//  these two blocks are equivalent
{
    FDcPropertyReader Reader(FromDatum);
    FDcPropertyWriter Writer(ToDatum);
    FDcPipeVisitor RoundtripVisit(&Reader, &Writer);

    DC_TRY(RoundtripVisit.PipeVisit());
}

{
    FDcDeserializer Deserializer;
    DcSetupPropertyPipeDeserializeHandlers(Deserializer);

    FDcPropertyReader Reader(FromDatum);
    FDcPropertyWriter Writer(ToDatum);

    FDcDeserializeContext Ctx;
    Ctx.Reader = &Reader;
    Ctx.Writer = &Writer;
    Ctx.Deserializer = &Deserializer;
    DC_TRY(Ctx.Prepare());

    DC_TRY(Deserializer.Deserialize(Ctx));
}

These are provided as a set of basis to for building custom property wrangling utils. See Field Renamer for example.

Extra

Alongside DataConfigCore we have other modules DataConfigExtra, DataConfigEngineExtra and DataConfigEditorExtra. It have self contained samples built on top of DataConfig framework.

Note that these are not intended to be integrated directly. You can take these as references when implementing custom features.

FColor Serialization/Deserialization

This example has been shown in previous chapter. It's also a benchmark use case for our custom deserialization logic:

• DcSerDeColor.h
• DcSerDeColor.cpp

// DataConfigExtra/Public/DataConfig/Extra/SerDe/DcSerDeColor.h
USTRUCT()
struct FDcExtraTestStructWithColor1
{
    GENERATED_BODY()

    UPROPERTY() FColor ColorField1;
    UPROPERTY() FColor ColorField2;
};

// DataConfigExtra/Private/DataConfig/Extra/SerDe/DcSerDeColor.cpp
FString Str = TEXT(R"(
    {
        "ColorField1" : "#0000FFFF",
        "ColorField2" : "#FF0000FF",
    }
)");

FColor is converted into a #RRGGBBAA hex string. The corresponding handlers looks pretty mirrored.

// DataConfigExtra/Private/DataConfig/Extra/SerDe/DcSerDeColor.cpp
FDcResult HandlerColorDeserialize(FDcDeserializeContext& Ctx)
{
	FDcPropertyDatum Datum;
	DC_TRY(Ctx.Writer->WriteDataEntry(FStructProperty::StaticClass(), Datum));

	FString ColorStr;
	DC_TRY(Ctx.Reader->ReadString(&ColorStr));
	FColor Color = FColor::FromHex(ColorStr);

	FColor* ColorPtr = (FColor*)Datum.DataPtr;
	*ColorPtr = Color;

	return DcOk();
}

FDcResult HandlerColorSerialize(FDcSerializeContext& Ctx)
{
	FDcPropertyDatum Datum;
	DC_TRY(Ctx.Reader->ReadDataEntry(FStructProperty::StaticClass(), Datum));

	FColor* ColorPtr = (FColor*)Datum.DataPtr;
	DC_TRY(Ctx.Writer->WriteString(TEXT("#") + ColorPtr->ToHex()));

	return DcOk();
}

Note how FDcPropertyReader::ReadDataEntry and FDcPropertyWriter::WriteDataEntry retrieves the next property as a FDcPropertyDatum, which allows us to directly manipulate a FColor pointer.

Base64 Blob Serialization/Deserialization

This demonstrates conversion betweenTArray<uint8> and Base64 encoded strings in JSON:

• DcSerDeBase64.h
• DcSerDeBase64.cpp

// DataConfigExtra/Public/DataConfig/Extra/SerDe/DcSerDeBase64.h
USTRUCT()
struct FDcExtraTestStructWithBase64
{
    GENERATED_BODY()

    UPROPERTY(meta = (DcExtraBase64)) TArray<uint8> BlobField1;
    UPROPERTY(meta = (DcExtraBase64)) TArray<uint8> BlobField2;
};

// DataConfigExtra/Private/DataConfig/Extra/SerDe/DcSerDeBase64.cpp
FString Str = TEXT(R"(
    {
        "BlobField1" : "dGhlc2UgYXJlIG15IHR3aXN0ZWQgd29yZHM=",
        "BlobField2" : "",
    }
)");

Note that we're tagging the BlobField with (meta = (DcExtraBase64)) to explicitly show that we' want this member to be converted into Base64.

UE support arbitrary meta data in the meta = () segment. But beware that the meta data is only available when WITH_EDITORDATA flag is defined. In predicate we check for this DcExtraBase64 like this:

// DataConfigExtra/Private/DataConfig/Extra/SerDe/DcSerDeBase64.cpp
EDcDeserializePredicateResult PredicateIsBase64Blob(FDcDeserializeContext& Ctx)
{
	FArrayProperty* ArrayProperty = DcPropertyUtils::CastFieldVariant<FArrayProperty>(Ctx.TopProperty());

	//	check for only TArray<uint8>
	if (ArrayProperty == nullptr)
		return EDcDeserializePredicateResult::Pass;
	if (!ArrayProperty->Inner->IsA<FByteProperty>())
		return EDcDeserializePredicateResult::Pass;

	return ArrayProperty->HasMetaData(TEXT("DcExtraBase64"))
		? EDcDeserializePredicateResult::Process
		: EDcDeserializePredicateResult::Pass;
}

Nested Arrays

UE property system has a limitation that you can't have a property of nested array:

// UHT would error out: The type 'TArray<int32>' can not be used as a value in a TArray  
UPROPERTY() TArray<TArray<int>> Arr2D;

You can workaround this by wrap inner array in a struct. DataConfig however is flexible enough that you can serialize and deserialize nested JSON arrays into your data structures however you want:

• DcSerDeNested.h
• DcSerDeNested.cpp

// DataConfigExtra/Public/DataConfig/Extra/SerDe/DcSerDeNested.h
USTRUCT()
struct FDcExtraTestNested_Vec2
{
    GENERATED_BODY()

    UPROPERTY() TArray<FVector2D> Vec2ArrayField1;
    UPROPERTY() TArray<FVector2D> Vec2ArrayField2;
};

// DataConfigExtra/Private/DataConfig/Extra/SerDe/DcSerDeNested.cpp
// equivalent fixture
FString Str = TEXT(R"(
    {
        "Vec2ArrayField1" :
        [
            {"X": 1.0, "Y": 2.0},
            {"X": 2.0, "Y": 3.0},
            {"X": 3.0, "Y": 4.0},
            {"X": 4.0, "Y": 5.0},
        ],
        "Vec2ArrayField2" :
        [
            [1.0, 2.0],
            [2.0, 3.0],
            [3.0, 4.0],
            [4.0, 5.0],
        ],
    }
)");

The next example we define a simple struct FDcGrid2D and stores the dimension using metadata.

// DataConfigExtra/Public/DataConfig/Extra/SerDe/DcSerDeNested.h
USTRUCT()
struct FDcGrid2D
{
    GENERATED_BODY()
    UPROPERTY() TArray<int> Data;
};

USTRUCT()
struct FDcExtraTestNested_Grid
{
    GENERATED_BODY()
    UPROPERTY(meta=(DcWidth = 2, DcHeight = 2)) FDcGrid2D GridField1;
    UPROPERTY(meta=(DcWidth = 3, DcHeight = 4)) FDcGrid2D GridField2;
};

// DataConfigExtra/Private/DataConfig/Extra/SerDe/DcSerDeNested.cpp
// equivalent fixture
FString Str = TEXT(R"(
    {
        "GridField1" :
        [
            [1,2],
            [3,4],
        ],
        "GridField2" :
        [
            [ 1, 2, 3],
            [ 4, 5, 6],
            [ 7, 8, 9],
            [10,11,12],
        ],
    }
)");

When data dimension doesn't match a diagnostic would be reported:

# DataConfig Error: Nested Grid2D height mismatch, Expect '2'
- [WideCharDcJsonReader] --> <in-memory>7:22
   5 |                    [1,2],
   6 |                    [1,2],
   7 |                    [1,2],
     |                    ^
   8 |                ],
   9 |            }

Writer API Alternatives

Previously we're deserializing FColor by writing into its member fields separately, which is a bit cumbersome. In this case DataConfig do support better alternatives.

Since we know that FColor is POD type we can construct one by filling in correct bit pattern. In this case FDcPropertyWriter allow struct property to be coerced from a blob:

• DcSerDeColor.h
• DcSerDeColor.cpp

//  DataConfigExtra/Private/DataConfig/Extra/SerDe/DcSerDeColor.cpp
template<>
FDcResult TemplatedWriteColorDispatch<EDcColorDeserializeMethod::WriteBlob>(const FColor& Color, FDcDeserializeContext& Ctx)
{
    return Ctx.Writer->WriteBlob({
        (uint8*)&Color, // treat `Color` as opaque blob data
        sizeof(FColor)
    });
}

Alternatively we can get FProperty and data pointer in place and setting the value through Unreal's FProperty API:

//  DataConfigExtra/Private/DataConfig/Extra/SerDe/DcSerDeColor.cpp
template<>
FDcResult TemplatedWriteColorDispatch<EDcColorDeserializeMethod::WriteDataEntry>(const FColor& Color, FDcDeserializeContext& Ctx)
{
    FDcPropertyDatum Datum;
    DC_TRY(Ctx.Writer->WriteDataEntry(FStructProperty::StaticClass(), Datum));

    Datum.CastFieldChecked<FStructProperty>()->CopySingleValue(Datum.DataPtr, &Color);
    return DcOk();
}

Note that we already know that Datum.DataPtr points to a allocated FColor instance. Thus we can simply cast it into a FColor* and directly manipulate the pointer.

//  DataConfigExtra/Private/DataConfig/Extra/SerDe/DcSerDeColor.cpp
template<>
FDcResult TemplatedWriteColorDispatch<EDcColorDeserializeMethod::WritePointer>(const FColor& Color, FDcDeserializeContext& Ctx)
{
    FDcPropertyDatum Datum;
    DC_TRY(Ctx.Writer->WriteDataEntry(FStructProperty::StaticClass(), Datum));

    FColor* ColorPtr = (FColor*)Datum.DataPtr;
    *ColorPtr = Color;  // deserialize by assignment

    return DcOk();
}

Note that these techniques also applies on serialization:

//  DataConfigExtra/Private/DataConfig/Extra/SerDe/DcSerDeColor.cpp
FDcResult HandlerColorSerialize(FDcSerializeContext& Ctx)
{
    FDcPropertyDatum Datum;
    DC_TRY(Ctx.Reader->ReadDataEntry(FStructProperty::StaticClass(), Datum));

    FColor* ColorPtr = (FColor*)Datum.DataPtr;
    DC_TRY(Ctx.Writer->WriteString(TEXT("#") + ColorPtr->ToHex()));

    return DcOk();
}

JsonConverter in DataConfig

UE comes with a handy module JsonUtilities that handles conversion between USTRUCTs and JSON. In this example we've implemented similar functionalities that behaves almost identical to stock FJsonConverter.

• DcJsonConverter.h
• DcJsonConverter.cpp

// DataConfigExtra/Private/DataConfig/Extra/Types/DcJsonConverter.cpp
FString Str = TEXT(R"(
    {
        "strField" : "Foo",
        "nestField" : {
            "strArrayField" : [
                "One",
                "Two",
                "Three"
            ],
            "strIntMapField" : {
                "One": 1,
                "Two": 2,
                "Three": 3
            }
        },
        "intField" : 253,
        "boolField" : true
    }
)");

{
	FDcTestJsonConverter1 Lhs;
	bool LhsOk = DcExtra::JsonObjectStringToUStruct(Str, &Lhs);

    FDcTestJsonConverter1 Rhs;
	bool RhsOk = FJsonObjectConverter::JsonObjectStringToUStruct(Str, &Rhs);
}    
{
	FString Lhs;
	bool LhsOk = DcExtra::UStructToJsonObjectString(Data, Lhs);

	FString Rhs;
	bool RhsOk = FJsonObjectConverter::UStructToJsonObjectString(Data, Rhs);
}

DcExtra::JsonObjectStringToUStruct() body is trivia as it delegates most of the work to DcDeserializer:

// DataConfigExtra/Private/DataConfig/Extra/Types/DcJsonConverter.cpp
bool JsonObjectReaderToUStruct(FDcReader* Reader, FDcPropertyDatum Datum)
{
	FDcResult Ret = [&]() -> FDcResult {
		using namespace JsonConverterDetails;
		LazyInitializeDeserializer();

		FDcPropertyWriter Writer(Datum);

		FDcDeserializeContext Ctx;
		Ctx.Reader = Reader;
		Ctx.Writer = &Writer;
		Ctx.Deserializer = &Deserializer.GetValue();
		DC_TRY(Ctx.Prepare());

		DC_TRY(Deserializer->Deserialize(Ctx));
		return DcOk();
	}();

	if (!Ret.Ok())
	{
		DcEnv().FlushDiags();
		return false;
	}
	else
	{
		return true;
	}
}

The serializing function DcExtra::UStructToJsonObjectString() needs some customization as default DcJsonWriter and DcSerializer handlers behaves a bit different against stock FDcJsonConverter:

• It serialize field names as camelCase.
• It uses platform dependent line endings, that is \r\n on Windows.
• It have subtle new line breaking rules on nested array and object, and on spacing around : token.

The good news is that one can customize these behaviors with DataConfig to match it:

// DataConfigExtra/Public/DataConfig/Extra/Types/DcJsonConverter.h
template<typename InStructType>
static bool UStructToJsonObjectString(const InStructType& InStruct, FString& OutJsonString)
{
    static FDcJsonWriter::ConfigType _JSON_CONVERTER_CONFIG = []
	{
		FDcJsonWriter::ConfigType Config = FDcJsonWriter::DefaultConfig;
		Config.IndentLiteral = TEXT("\t");
		Config.LineEndLiteral = LINE_TERMINATOR;
		Config.LeftSpacingLiteral = TEXT("");
		Config.bNestedArrayStartsOnNewLine = false;
		Config.bNestedObjectStartsOnNewLine = true;
		return Config;
	}();

	FDcJsonWriter Writer(_JSON_CONVERTER_CONFIG);
    ...
}

// DataConfigExtra/Private/DataConfig/Extra/Types/DcJsonConverter.cpp
static FDcResult HandlerStructRootSerializeCamelCase(FDcSerializeContext& Ctx)
{
    ...
	else if (CurPeek == EDcDataEntry::Name)
	{
		FName Value;
		DC_TRY(Ctx.Reader->ReadName(&Value));
		DC_TRY(Ctx.Writer->WriteString(FJsonObjectConverter::StandardizeCase(Value.ToString())));
    }
}

We aim to support flexible serialization and formatting behaviors without modifying DataConfigCore code:

AnyStruct

This is an intermediate example that takes advantage of the flexibility provided by the property system. FDcAnyStruct is a struct that stores a heap allocated USTRUCT of any type while maintaining value semantic on itself. If you're familiar with the concept of variant type, just think of it as a variant type that supports all USTRUCT:

• DcAnyStruct.h
• DcAnyStruct.cpp

// DataConfigExtra/Private/DataConfig/Extra/SerDe/DcSerDeAnyStruct.cpp
//  instantiate from heap allocated structs
FDcAnyStruct Any1 = new FDcExtraTestSimpleStruct1();
Any1.GetChecked<FDcExtraTestSimpleStruct1>()->NameField = TEXT("Foo");

//  supports moving
FDcAnyStruct Any2 = MoveTemp(Any1);
check(!Any1.IsValid());
check(Any2.GetChecked<FDcExtraTestSimpleStruct1>()->NameField == TEXT("Foo"));
Any2.Reset();

//  supports shared referencing
Any2 = new FDcExtraTestSimpleStruct2();
Any2.GetChecked<FDcExtraTestSimpleStruct2>()->StrField = TEXT("Bar");

Any1 = Any2;

check(Any1.DataPtr == Any2.DataPtr);
check(Any1.StructClass == Any2.StructClass);

We then implemented conversion logic between FDcAnyStruct and JSON:

• DcSerDeAnyStruct.h
• DcSerDeAnyStruct.cpp

// DataConfigExtra/Public/DataConfig/Extra/SerDe/DcSerDeAnyStruct.h
USTRUCT()
struct FDcExtraTestWithAnyStruct1
{
    GENERATED_BODY()

    UPROPERTY() FDcAnyStruct AnyStructField1;
    UPROPERTY() FDcAnyStruct AnyStructField2;
    UPROPERTY() FDcAnyStruct AnyStructField3;
};

// DataConfigExtra/Private/DataConfig/Extra/SerDe/DcSerDeAnyStruct.cpp
FString Str = TEXT(R"(
    {
        "AnyStructField1" : {
            "$type" : "DcExtraTestSimpleStruct1",
            "NameField" : "Foo"
        },
        "AnyStructField2" : {
            "$type" : "DcExtraTestStructWithColor1",
            "ColorField1" : "#0000FFFF",
            "ColorField2" : "#FF0000FF"
        },
        "AnyStructField3" : null
    }
)");

Note how the custom FColor <-> "#RRGGBBAA" conversion recursively works within FDcAnyStruct. This should be a good starting point for you to implement your own nested variant types and containers. For more details refer to the implementation of HandlerDcAnyStruct[Serialize/Deserialize].

Inline Struct

In AnyStruct we implemented FDcAnyStruct that can store arbitrary heap allocated USTRUCT. A shortcoming of this approach is that it introduces overhead of heap allocated memory, which also have worse cache locality comparing to stack allocated structs.

In this example we implemented a series of structs FDcInlineStruct64/FDcInlineStruct128/FDcInlineStruct256/FDcInlineStruct512 which stores a USTRUCT inline. Think it as a small buffer optimized version of FDcAnyStruct. These can also be used as a cheap alternative to UCLASS based polymorphism.

• DcInlineStruct.h
• DcInlineStruct.cpp

// DataConfigExtra/Private/DataConfig/Extra/SerDe/DcSerDeInlineStruct.cpp
//  stack allocated usage
FDcInlineStruct64 Inline1;
Inline1.Emplace<FColor>(255, 0, 0, 255);
UTEST_TRUE("Inline Struct", *Inline1.GetChecked<FColor>() == FColor::Red);

//  support copy 
FDcInlineStruct64 Inline2 = Inline1;
UTEST_TRUE("Inline Struct", *Inline2.GetChecked<FColor>() == FColor::Red);

Serialization handlers for inline structs are similar to any struct ones:

• DcSerDeInlineStruct.h
• DcSerDeInlineStruct.cpp

// DataConfigExtra/Public/DataConfig/Extra/SerDe/DcSerDeInlineStruct.h
USTRUCT()
struct FDcExtraTestWithInlineStruct1
{
    GENERATED_BODY()

    UPROPERTY() FDcInlineStruct64 InlineField1;
    UPROPERTY() FDcInlineStruct64 InlineField2;;
};

// DataConfigExtra/Private/DataConfig/Extra/SerDe/DcSerDeInlineStruct.cpp
FString Str = TEXT(R"(
    {
        "InlineField1" : {
            "$type" : "DcExtraTestSimpleStruct1",
            "NameField" : "Foo"
        },
        "InlineField2" : {
            "$type" : "DcExtraTestStructWithColor1",
            "ColorField1" : "#0000FFFF",
            "ColorField2" : "#FF0000FF"
        }
    }
)");

Note that one limitation with inline structs the USTRUCT get put in must have smaller size than the inline struct storage. Deserialize handlers would check for these cases:

LogDataConfigCore: Display: # DataConfig Error: Inline struct too big: BufSize '56', Struct 'DcExtraTestStruct128' Size '128'
LogDataConfigCore: Display: - [WideCharDcJsonReader] --> <in-memory>5:38
   3 |        {
   4 |            "InlineField1" : {
   5 |                "$type" : "DcExtraTestStruct128",
     |                                                ^
   6 |                "NameField" : "Foo"
   7 |            },
LogDataConfigCore: Display: - [DcPropertyWriter] Writing property: (FDcExtraTestWithInlineStruct1)$root.(FDcInlineStruct64)InlineField1
LogDataConfigCore: Display: - [DcSerializer]

InstancedStruct

This example only works with UE5.0+

Starting with UE 5.0 there's a new plugin StructUtils featuring a struct type called FInstancedStruct. It's shares the same idea to previous AnyStruct example, while it has proper asset serialization logic and editor support.

Given a struct hierarchy like this:

// DataConfigExtra/Public/DataConfig/Extra/Types/DcExtraTestFixtures.h
USTRUCT(BlueprintType)
struct FDcStructShapeBase
{
    GENERATED_BODY()
    
    UPROPERTY(EditAnywhere) FName ShapeName;
};

USTRUCT(BlueprintType)
struct FDcStructShapeRectangle : public FDcStructShapeBase
{
    GENERATED_BODY()
    
    UPROPERTY(EditAnywhere) float Height;
    UPROPERTY(EditAnywhere) float Width;
};

USTRUCT(BlueprintType)
struct FDcStructShapeCircle : public FDcStructShapeBase
{
    GENERATED_BODY()
    
    UPROPERTY(EditAnywhere) float Radius;
};

You can use FInstancedStruct specified with a BaseStruct meta to reference to a polymorphism instance:

// DataConfigEngineExtra5/Public/DataConfig/EngineExtra/SerDe/DcSerDeInstancedStruct.h
UPROPERTY(EditAnywhere, meta = (BaseStruct = "/Script/DataConfigExtra.DcStructShapeBase"))
FInstancedStruct InstancedStruct1;

UPROPERTY(EditAnywhere, meta = (BaseStruct = "DcStructShapeBase"))
FInstancedStruct InstancedStruct2;

The best part is that the editor is also working as intended:

Prior to this you'll need to setup Sub Objects for similar behavior, which costs unnecessary overhead.

FInstancedStruct can also be serialized to and from JSON with DataConfig:

• DcSerDeInstancedStruct.h
• DcSerDeInstancedStruct.cpp

// DataConfigExtra5/Public/DataConfig/EngineExtra/SerDe/DcSerDeInstancedStruct.h
USTRUCT()
struct FDcEngineExtra5InstancedStruct1
{
    GENERATED_BODY()

    UPROPERTY() FInstancedStruct InstancedStruct1;
    UPROPERTY() FInstancedStruct InstancedStruct2;
    UPROPERTY() FInstancedStruct InstancedStruct3;
};

// DataConfigExtra5/Private/DataConfig/EngineExtra/SerDe/DcSerDeInstancedStruct.cpp
FString Str = TEXT(R"(
    {
        "InstancedStruct1" : {
            "$type" : "DcExtraTestSimpleStruct1",
            "NameField" : "Foo"
        },
        "InstancedStruct2" : {
            "$type" : "DcExtraTestStructWithColor1",
            "ColorField1" : "#0000FFFF",
            "ColorField2" : "#FF0000FF"
        },
        "InstancedStruct3" : null
    }
)");

Note how the custom FColor <-> "#RRGGBBAA" conversion recursively works within FInstancedStruct.

Field Renamer

DataConfig can also be used to author one-off utility. In this example we implemented DcExtra::DeserializeStructRenaming() that copies data between structural identical data structures, while renaming field names by a user function.

• DcDeserializeRenameStructFieldNames.h
• DcDeserializeRenameStructFieldNames.cpp

// DataConfigExtra/Private/DataConfig/Extra/Deserialize/DcDeserializeRenameStructFieldNames.cpp
//  struct equivelent to this:
FString Str = TEXT(R"(
    {
        "FromName1" : "Foo",
        "FromStructSet1" : 
        [
            {
                "FromStr1" : "One",
                "FromInt1" : 1,
            },
            {
                "FromStr1" : "Two",
                "FromInt1" : 2,
            }
        ]
    }
)");

// ... deserialize with a functor renaming `FromXXX` to `ToXXX`:
UTEST_OK("...", DcExtra::DeserializeStructRenaming(
    FromDatum, ToDatum, FDcExtraRenamer::CreateLambda([](const FName& FromName){
    FString FromStr = FromName.ToString();
    if (FromStr.StartsWith(TEXT("From")))
        return FName(TEXT("To") + FromStr.Mid(4));
    else
        return FromName;
})));

// ... results into a struct equivelent to this: 
FString Str = TEXT(R"(
    {
        "ToName1" : "Foo",
        "ToStructSet1" : 
        [
            {
                "ToStr1" : "One",
                "ToInt1" : 1,
            },
            {
                "ToStr1" : "Two",
                "ToInt1" : 2,
            }
        ]
    }
)");

This takes advantage of the DcPropertyPipeHandlers that simply do verbatim data piping.

The gist is that you should consider DataConfig an option when working with batch data processing within Unreal Engine. We are trying to provide tools to support these use cases.

Access property by path

This example demonstrates that FReader/FWriter can be used standalone, without FDcSerializer/FDcDeserializer.

UE built-in module PropertyPath allow accessing nested object properties by a path like Foo.Bar.Baz:

• DcPropertyPathAccess.h
• DcPropertyPathAccess.cpp

// DataConfigExtra/Private/DataConfig/Extra/Types/DcPropertyPathAccess.cpp
FString Str;
UTEST_TRUE("...", PropertyPathHelpers::GetPropertyValue(Outer, TEXT("StructRoot.Middle.InnerMost.StrField"), Str));
UTEST_TRUE("...", Str == TEXT("Foo"));

UTEST_TRUE("...", PropertyPathHelpers::SetPropertyValue(Outer, TEXT("StructRoot.Middle.InnerMost.StrField"), FString(TEXT("Bar"))));
UTEST_TRUE("...", Outer->StructRoot.Middle.InnerMost.StrField == TEXT("Bar"));

We implemented a pair of methods DcExtra::GetDatumPropertyByPath/SetDatumPropertyByPath with FDcPropertyReader:

// DataConfigExtra/Private/DataConfig/Extra/Types/DcPropertyPathAccess.cpp
UTEST_TRUE("...", CheckStrPtr(GetDatumPropertyByPath<FString>(FDcPropertyDatum(Outer), "StructRoot.Middle.InnerMost.StrField"), TEXT("Foo")));
UTEST_TRUE("...", CheckStrPtr(GetDatumPropertyByPath<FString>(FDcPropertyDatum(Outer), "StructRoot.Arr.0.StrField"), TEXT("Bar0")));
UTEST_TRUE("...", CheckStrPtr(GetDatumPropertyByPath<FString>(FDcPropertyDatum(Outer), "StructRoot.Arr.1.StrField"), TEXT("Bar1")));
UTEST_TRUE("...", CheckStrPtr(GetDatumPropertyByPath<FString>(FDcPropertyDatum(Outer), "StructRoot.NameMap.FooKey.StrField"), TEXT("FooValue")));

UTEST_TRUE("...", SetDatumPropertyByPath<FString>(FDcPropertyDatum(Outer), "StructRoot.Middle.InnerMost.StrField", TEXT("AltFoo")));
UTEST_TRUE("...", SetDatumPropertyByPath<FString>(FDcPropertyDatum(Outer), "StructRoot.Arr.0.StrField", TEXT("AltBar0")));
UTEST_TRUE("...", SetDatumPropertyByPath<FString>(FDcPropertyDatum(Outer), "StructRoot.Arr.1.StrField", TEXT("AltBar1")));
UTEST_TRUE("...", SetDatumPropertyByPath<FString>(FDcPropertyDatum(Outer), "StructRoot.NameMap.FooKey.StrField", TEXT("AltFooValue")));

Comparing to PropertyPathHelpers these new ones support Array and Map, and support USTRUCT roots. We're missing some features like expanding weak/lazy object references but it should be easy to implement.

Remember that we have bundled JSON/MsgPack reader/writers that can also be used standalone.

Deserialize From SQLite Query

Unreal Engine bundles SQLite3 in SQLiteCore plugin and bundled with minimal C++ abstractions. In this example we implemented LoadStructArrayFromSQLite to load a SQLite query into a TArray of structs:

• DcSqlite.h
• DcSqlite.cpp

// DataConfigExtra/Public/DataConfig/Extra/Misc/DcSqlite.h
template<typename TStruct>
FDcResult LoadStructArrayFromSQLite(FSQLiteDatabase* Db, const TCHAR* Query, TArray<TStruct>& Arr)
// ...

With this method we can easily turn SQLite query results into structs that's easy to manipulate with:

// DataConfig/Source/DataConfigExtra/Private/DataConfig/Extra/Misc/DcSqlite.cpp
/// data fixture
FString Statement = TEXT("CREATE TABLE users (id INTEGER NOT NULL,name TEXT, title TEXT)");
bSuccess &= TestDb.Execute(*Statement);
Statement = TEXT("INSERT INTO users (id, name, title) VALUES (1, 'John', 'Manager')");
bSuccess &= TestDb.Execute(*Statement);
Statement = TEXT("INSERT INTO users (id, name, title) VALUES (2, 'Mark', 'Engineer')");
bSuccess &= TestDb.Execute(*Statement);
Statement = TEXT("INSERT INTO users (id, name, title) VALUES (3, 'Bob', 'Engineer')");
bSuccess &= TestDb.Execute(*Statement);
Statement = TEXT("INSERT INTO users (id, name, title) VALUES (4, 'Mike', 'QA')");

//  execute query and deserialize
TArray<FDcExtraTestUser> Arr;
UTEST_OK("Extra Sqlite", DcExtra::LoadStructArrayFromSQLite(
    &TestDb,
    TEXT("SELECT * FROM users WHERE title == 'Engineer' ORDER BY id"),
    Arr
    ));

//  equivalent fixture
FString Fixture = TEXT(R"(
    [
        {
            "Id" : 2,
            "Name" : "Mark",
            "Title" : "Engineer"
        },
        {
            "Id" : 3,
            "Name" : "Bob",
            "Title" : "Engineer"
        }
    ]
)");

For this to work we'll need to implement FSqliteReader which implements FDcReader API so it can be consumed by deserializer. The cool thing is that FSqliteReader works very well with SQLite's step API and can wrap SQLite error reporting into diagnostics that DataConfig can report.

NDJSON

NDJSON is a popular extension to JSON that stores 1 JSON object per line. DataConfig's JSON parser and writer is flexible enough to easily support this use case.

• DcNDJSON.h
• DcNDJSON.cpp

// DataConfigExtra/Public/DataConfig/Extra/Misc/DcNDJSON.h
template<typename TStruct>
DATACONFIGEXTRA_API FDcResult LoadNDJSON(const TCHAR* Str, TArray<TStruct>& Arr)
// ...
template<typename TStruct>
DATACONFIGEXTRA_API FDcResult SaveNDJSON(const TArray<TStruct>& Arr, FString& OutStr)
// ...

With this method we can load a NDJSON string into a struct array and later serialize it back to NDJSON.

// DataConfig/Source/DataConfigExtra/Private/DataConfig/Extra/Misc/DcNDJSON.cpp
FString Str = TEXT(R"(

    { "Name" : "Foo", "Id" : 1, "Type" : "Alpha" }
    { "Name" : "Bar", "Id" : 2, "Type" : "Beta" }
    { "Name" : "Baz", "Id" : 3, "Type" : "Gamma" }

)");

UTEST_OK("Extra NDJSON", LoadNDJSON(*Str, Dest));

FString SavedStr;
UTEST_OK("Extra NDJSON", SaveNDJSON(Dest, SavedStr));

Note that our parser supports common extension to JSON:

• Allow C Style comments, i.e /* block */ and // line .
• Allow trailing comma, i.e [1,2,3,], .
• Allow non object root. You can put a list as the root, or even string, numbers.

Root Object

Many modern JSON like data markup languages allows root level object and arrays, i.e omitting top level braces. This can be done in DataConfig with custom serialize handlers.

• DcSerDeRoot.h
• DcSerDeRoot.cpp

// DataConfigExtra/Private/DataConfig/Extra/SerDe/DcSerDeRoot.cpp
// root object
FString Str = TEXT(R"(

    "Name" : "Foo",
    "Id" : 253,
    "Type" : "Beta"

)");

//  equivalent fixture
FDcExtraSimpleStruct Expect;
Expect.Name = TEXT("Foo");
Expect.Id = 253;
Expect.Type = EDcExtraTestEnum1::Beta;

// DataConfigExtra/Private/DataConfig/Extra/SerDe/DcSerDeRoot.cpp
// root list
FString Str = TEXT(R"(

    "Alpha",
    "Beta",
    "Gamma"

)");

//  equivalent fixture
TArray<EDcExtraTestEnum1> Expect = {
    EDcExtraTestEnum1::Alpha,
    EDcExtraTestEnum1::Beta,
    EDcExtraTestEnum1::Gamma};

This can be a tiny QOL improvement for manually authoring JSON data.

Extra Module Setups

Unreal Engine Modules is how the engine handles its C++ code physical design. You'll need to be pretty familiar with the system to scale up your C++ project.

We split extra samples in two modules. The first is DataConfigExtra which does not depend on Engine/UnrealEd module. It can be built along program target. DataConfigExtra is basically a set of C++ source files bundled and there's no special setup for it. The key setup is to set ModuleRules.bRequiresImplementModule to be false:

• DataConfigExtra.Build.cs

// DataConfigExtra/DataConfigExtra.Build.cs
public class DataConfigExtra : ModuleRules
{
	public DataConfigExtra(ReadOnlyTargetRules Target) : base(Target)
	{
	    bRequiresImplementModule = false;
	    Type = ModuleType.CPlusPlus;
		PCHUsage = PCHUsageMode.UseExplicitOrSharedPCHs;
		PublicDependencyModuleNames.AddRange(
			new string[] {
			"DataConfigCore",
			"Core",
			"CoreUObject",
             // ...
			});
	}
}

The other module is DcEngineExtraModule, which is a more conventional runtime module that introduces DataConfig dependency. We also put samples that depends on Engine and other gameplay system in here.

Most of integration code is in IModuleInterface::StartUpModule/ShutdownModule().

• DcEngineExtraModule.cpp

// DataConfigEngineExtra/Private/DcEngineExtraModule.cpp
void FDcEngineExtraModule::StartupModule()
{
    // ...
	DcRegisterDiagnosticGroup(&DcDExtra::Details);
	DcRegisterDiagnosticGroup(&DcDEngineExtra::Details);

	DcStartUp(EDcInitializeAction::Minimal);
	DcEnv().DiagConsumer = MakeShareable(new FDcMessageLogDiagnosticConsumer());
	// ...
}

void FDcEngineExtraModule::ShutdownModule()
{
	DcShutDown();
    // ...
}

Here's a checklist for integration:

• Register additional diagnostics early.
• Call DcStartUp()/DcShutDonw() pair.
• Register custom diagnostic consumer.

FDcMessageLogDiagnosticConsumer is an example of redirecting diagnostics to the UE Message Log window with its own category.

Dump Asset To Log

In this example we added a Dump Asset To Log menu entry. After enabling it you can see it on the context menu on all asset types:

On clicking it would use the pretty print writer to dump the asset into Output Log:

This is implemented as an editor menu extender:

• DcEditorDumpAssetToLog.cpp

// DataConfigExtra/Private/DataConfig/EditorExtra/Editor/DcEditorDumpAssetToLog.cpp
TSharedRef<FExtender> DumpAssetToLogExtender(const TArray<FAssetData>& SelectedAssets)
{
    // ...
    Extender->AddMenuExtension("GetAssetActions", EExtensionHook::After, TSharedPtr<FUICommandList>(),
        FMenuExtensionDelegate::CreateLambda([Asset](FMenuBuilder& MenuBuilder)
        {
        MenuBuilder.AddMenuEntry(
        // ...
        FUIAction(
            FExecuteAction::CreateLambda([Asset]{
                if (UBlueprint* Blueprint = Cast<UBlueprint>(Asset.GetAsset()))
                {
                    //  dump BP class generated class CDO as it makes more sense
                    DcAutomationUtils::DumpToLog(FDcPropertyDatum(Blueprint->GeneratedClass->ClassDefaultObject));
                }
                else
                {
                    DcAutomationUtils::DumpToLog(FDcPropertyDatum(Asset.GetAsset()));
                }
                }),
                FCanExecuteAction()
            )
        );
        }));
    }
    // ...

When asset is a Blueprint we dump its CDO instead as it makes more sense.

Blueprint Serialization/Deserialization

The Property System is so powerful that you can create new Blueprint Class/Struct, which is equivalent to C++ UCLASS/USTRUCT to some extents. In this example we'll show how to handle these in DataConfig.

The whole blueprint stuff depends on Engine module. This is also why we put related code into DataConfigEngineExtra module.

Blueprint Class and Object references

Blueprint class are stored within UBlueprint typed assets. Note that we automatically unwrap the container in handlers.

• DcSerDeBlueprint.h
• DcSerDeBlueprint.cpp

//DataConfigEngineExtra/Public/DataConfig/EngineExtra/SerDe/DcSerDeBlueprint.h
USTRUCT()
struct FDcEngineExtraTestStructWithBPClass
{
    GENERATED_BODY()

    UPROPERTY() TSubclassOf<UDcTestBPClassBase> ClassField1;
    UPROPERTY() TSubclassOf<UDcTestBPClassBase> ClassField2;
    UPROPERTY() TSubclassOf<UDcTestBPClassBase> ClassField3;
};

USTRUCT()
struct FDcEngineExtraTestStructWithBPInstance
{
    GENERATED_BODY()

    UPROPERTY() UDcTestBPClassBase* InstanceField1;
    UPROPERTY() UDcTestBPClassBase* InstanceField2;
    UPROPERTY() UDcTestBPClassBase* InstanceField3;
};

//DataConfigEngineExtra/Private/DataConfig/EngineExtra/SerDe/DcSerDeBlueprint.cpp
FString Str = TEXT(R"(
    {
        "ClassField1" : null,
        "ClassField2" : "DcTestNativeDerived1",
        "ClassField3" : "/DataConfig/DcFixture/DcTestBlueprintClassBeta"
    }
)");

FString Str = TEXT(R"(
    {
        "InstanceField1" : null,
        "InstanceField2" : "/DataConfig/DcFixture/DcTestBlueprintInstanceAlpha",
        "InstanceField3" : "/DataConfig/DcFixture/DcTestBlueprintInstanceBeta"
    }
)");

Blueprint Class Instance

In this example we roundtrip a Blueprint class instance from JSON:

//DataConfigEngineExtra/Private/DataConfig/EngineExtra/SerDe/DcSerDeBlueprint.cpp
FString Str = TEXT(R"(
    {
        "StrField" : "Foo",
        "BPEnumField" : "Baz",
        "IntField" : 345
    }
)");

You can also reference on how to handle Blueprint enum in this example. It need some special care to convert between int value and the descriptive text set within the editor.

Blueprint Struct Instance

Finally we're combined FColor, FDcAnyStruct and Blueprint struct into one single example:

//DataConfigEngineExtra/Private/DataConfig/EngineExtra/SerDe/DcSerDeBlueprint.cpp
FString Str = TEXT(R"(
    {
        "AnyStructField1" : {
            "$type" : "/DataConfig/DcFixture/DcTestBlueprintStructWithColor",
            "NameField" : "Foo",
            "StrField" : "Bar",
            "IntField" : 123,
            "ColorField" : "#FF0000FF"
        }
    }
)");

Under the hood Blueprint struct mangles its field names. The struct above dumps to something like this:

-----------------------------------------
# Datum: 'UserDefinedStruct', 'DcTestBlueprintStructWithColor'
<StructRoot> 'DcTestBlueprintStructWithColor'
|---<Name> 'NameField_5_97BFF114405C1934C2F33E8668BF1652'
|---<Name> 'Foo'
|---<Name> 'StrField_9_FAA71EFE4896F4E6B1478B9C13B2CE52'
|---<String> 'Bar'
|---<Name> 'IntField_11_3BC7CB0F42439CE2196F7AA82A1AC374'
|---<Int32> '123'
|---<Name> 'ColorField_14_F676BCF245B2977B678B65A8216E94EB'
|---<StructRoot> 'Color'
|   |---<Name> 'B'
|   |---<UInt8> '0'
|   |---<Name> 'G'
|   |---<UInt8> '0'
|   |---<Name> 'R'
|   |---<UInt8> '255'
|   |---<Name> 'A'
|   |---<UInt8> '255'
|---<StructEnd> 'Color'
<StructEnd> 'DcTestBlueprintStructWithColor'
-----------------------------------------

The good news is that DataConfig already got this covered.

Gameplay Tag Serialization/Deserialization

GameplayTags is a built-in runtime module that implements hierarchical tags.

In this example we have roundtrip handlers for FGameplayTag and FGameplayTagContainer.

• DcSerDeGameplayTags.h
• DcSerDeGameplayTags.cpp

// DataConfigEngineExtra/Public/DataConfig/EngineExtra/SerDe/DcSerDeGameplayTags.h
USTRUCT()
struct FDcEngineExtraTestStructWithGameplayTag1
{
    GENERATED_BODY()

    UPROPERTY() FGameplayTag TagField1;
    UPROPERTY() FGameplayTag TagField2;
};

// DataConfigEngineExtra/Private/DataConfig/EngineExtra/SerDe/DcSerDeGameplayTags.cpp
FString Str = TEXT(R"(
    {
        "TagField1" : null,
        "TagField2" : "DataConfig.Foo.Bar"
    }
)");

FGameplayTagContainer converts to a list of strings:

// DataConfigEngineExtra/Public/DataConfig/EngineExtra/Deserialize/DcDeserializeGameplayTags.h
USTRUCT()
struct FDcEngineExtraTestStructWithGameplayTag2
{
    GENERATED_BODY()

    UPROPERTY() FGameplayTagContainer TagContainerField1;
    UPROPERTY() FGameplayTagContainer TagContainerField2;
};

// DataConfigEngineExtra/Private/DataConfig/EngineExtra/Deserialize/DcDeserializeGameplayTags.cpp
FString Str = TEXT(R"(
    {
        "TagContainerField1" : [],
        "TagContainerField2" : [
            "DataConfig.Foo.Bar",
            "DataConfig.Foo.Bar.Baz",
            "DataConfig.Tar.Taz",
        ]
    }
)");

Note that gameplay tag parsing has error reporting built-in. In this case we can pipe it into our diagnostic:

// DataConfigEngineExtra/Private/DataConfig/EngineExtra/Deserialize/DcDeserializeGameplayTags.cpp
static FDcResult _StringToGameplayTag(FDcDeserializeContext& Ctx, const FString& Str, FGameplayTag* OutTagPtr)
{
    FString FixedString;
    FText Err;
    if (!FGameplayTag::IsValidGameplayTagString(Str, &Err, &FixedString))
    {
        return DC_FAIL(DcDEngineExtra, InvalidGameplayTagStringFixErr)
            << Str << FixedString << Err;
    }
    //...
}

In case of a invalid tag it would report the reason and fixed string:

# DataConfig Error: Invalid Gameplay Tag String, Actual: 'DataConfig.Invalid.Tag.', Fixed: 'DataConfig.Invalid.Tag', Error: 'Tag ends with .'
- [JsonReader] --> <in-memory>5:4
   3 |        { 
   4 |            "TagField1" : null, 
   5 |            "TagField2" : "DataConfig.Invalid.Tag." 
     |                          ^^^^^^^^^^^^^^^^^^^^^^^^^
   6 |        } 
   7 |    
- [PropertyWriter] Writing property: (FDcEngineExtraTestStructWithGameplayTag1)$root.(FGameplayTag)TagField2
 [C:\DevUE\UnrealEngine\Engine\Source\Developer\MessageLog\Private\Model\MessageLogListingModel.cpp(73)]

Deserialize Gameplay Abilities

We'll conclude with a concrete user story: populating GameplayAbility and GameplayEffect blueprint from JSON file.

• DcDeserializeGameplayAbility.h
• DcDeserializeGameplayAbility.cpp

Gameplay Ability System is a built-in plugin for building data driven abilities. Users are expected to derived and modify GameplayAbility and GameplayEffect blueprint for custom logic.

Given a JSON like this:

// DataConfig/Tests/Fixture_AbilityAlpha.json
{
    /// Tags
    "AbilityTags" : [
        "DataConfig.Foo.Bar",
        "DataConfig.Foo.Bar.Baz",
    ],
    "CancelAbilitiesWithTag" : [
        "DataConfig.Foo.Bar.Baz",
        "DataConfig.Tar.Taz",
    ],
    /// Costs
    "CostGameplayEffectClass" : "/DataConfig/DcFixture/DcTestGameplayEffectAlpha",
    /// Advanced
    "ReplicationPolicy" : "ReplicateYes",
    "InstancingPolicy" : "NonInstanced",
}

Right click on a GameplayAbility blueprint asset and select Load From JSON， then select this file and confirm. It would correctly populate the fields with the values in JSON, as seen in the pic below:

Most of the logic is in DataConfig/EditorExtra/Deserialize/DcDeserializeGameplayAbility.cpp:

• The context menu is added from GameplayAbilityEffectExtender. There's another handy item named Dump To Log which dumps any blueprint CDO into the log.
• DataConfig deserializer is setup in LazyInitializeDeserializer(). We added custom logic for deserializing FGameplayAttribute from a string like DcTestAttributeSet.Mana.
• We also reused many methods from previous examples to support FGameplayTag deserialization and Blueprint class look up by path.

Blueprint Nodes

We have a set of blueprint nodes that's have similar API as built-in JsonBlueprintUtilities plugin:

Some caveats:

• The Dump/Load Object APIs always expand the root level object, even if it's a external object reference or an asset. These are by default serialized to a string by DataConfig. Also the Self pin only works with these due to some BP limitations.
• Most extra examples are integrated into these nodes so you can try them out without writing C++ code.
  • AnyStruct
  • Base64
  • InlineStruct
  • Bluerprint Class/Struct/Enum
  • Gameplay Tag
  • InstancedStruct
• Error diagnostic is also wired up to message log:

Advanced

This section contains documentation for advanced topics.

Benchmark

We've integrated two benchmarks fixtures from JSON for modern C++ project. In the benchmark we deserialize JSON into C++ structs, then serialize them back to JSON. Then we convert JSON to MsgPack and do identical process.

• canada.json
• corpus.ndjson

See here for instructions to build and run the benchmark.

Here're the results:

• DcBenchmarkFixture1.cpp

On a AMD Ryzen 9 5950X 16-Core Processor 3.40 GHz:

Corpus Json Deserialize: [Shipping] Bandwidth: 94.951(MB/s), Mean: 58.097(ms), Median:58.061(ms), Deviation:0.997
Corpus Json Serialize: [Shipping] Bandwidth: 117.859(MB/s), Mean: 46.805(ms), Median:46.086(ms), Deviation:1.818
Corpus MsgPack Deserialize: [Shipping] Bandwidth: 103.436(MB/s), Mean: 50.531(ms), Median:50.453(ms), Deviation:0.469
Corpus MsgPack Serialize: [Shipping] Bandwidth: 103.586(MB/s), Mean: 50.457(ms), Median:50.342(ms), Deviation:0.614

Canada Json Deserialize: [Shipping] Bandwidth: 73.581(MB/s), Mean: 29.176(ms), Median:29.171(ms), Deviation:0.132
Canada Json Serialize: [Shipping] Bandwidth: 56.439(MB/s), Mean: 38.037(ms), Median:37.882(ms), Deviation:1.050
Canada MsgPack Serialize: [Shipping] Bandwidth: 131.555(MB/s), Mean: 4.441(ms), Median:4.432(ms), Deviation:0.030
Canada MsgPack Deserialize: [Shipping] Bandwidth: 100.450(MB/s), Mean: 5.816(ms), Median:5.816(ms), Deviation:0.024

Some insights on the results:

• Benchmark in Shipping build configuration, otherwise it doesn't make much sense.

• Recall that runtime performance isn't our top priority. We opted for a classic inheritance based API for FDcReader/FDcWriter which means that each read/write step result in a virtual dispatch. This by design would result in mediocre performance metrics. The bandwidth should be in the range of 10~100(MB/s) on common PC setup, no matter how simple the format is.

• MsgPack and JSON has similar bandwidth numbers in the benchmark. However MsgPack has far more tight layout when dealing with numeric data. Note in the Canada fixture MsgPack only takes around 10ms, as this fixture is mostly float number coordinates.

Tips for writing handlers

There're some recurring patterns when writing handlers in DataConfig.

Recursive Deserialize

When deserializing a container like USTRUCT root or TArray you'll need to recursively deserialize children properties. This is wrapped in a single function call:

// DataConfigCore/Private/DataConfig/Deserialize/Handlers/Json/DcJsonStructDeserializers.cpp
DC_TRY(DcDeserializeUtils::RecursiveDeserialize(Ctx));

Internally it would push writer's next property into FDcDeserializeContext::Properties to satisfiy the invariant that FDcDeserializeContext::TopProperty() always points to the current writing property. It would also clear up the top property on return.

Another example is how we pipe deserialize a TMap<>. When at key and value position we simply call this method 2 times:

// DataConfigCore/Private/DataConfig/Deserialize/Handlers/Property/DcPropertyPipeDeserializers.cpp
DC_TRY(Ctx.Reader->ReadMapRoot());
DC_TRY(Ctx.Writer->WriteMapRoot());

EDcDataEntry CurPeek;
while (true)
{
    DC_TRY(Ctx.Reader->PeekRead(&CurPeek));
    if (CurPeek == EDcDataEntry::MapEnd)
        break;

    DC_TRY(DcDeserializeUtils::RecursiveDeserialize(Ctx));
    DC_TRY(DcDeserializeUtils::RecursiveDeserialize(Ctx));
}

DC_TRY(Ctx.Reader->ReadMapEnd());
DC_TRY(Ctx.Writer->WriteMapEnd());

Provide TopObject()

Sometimes deserialization will create new UObject along the way. In this case you'll need to fill in FDcDeserializeContext::Objects so the top one is used for NewObject() calls. For transient objecst you can use GetTransientPackage():

// DataConfigTests/Private/DcTestDeserialize.cpp
Ctx.Objects.Push(GetTransientPackage());

Peek By Value

Sometimes you want to peek the content of the next entry. For example in DcExtra::HandlerBPDcAnyStructDeserialize() we're dealing with a JSON like this:

// DataConfigEngineExtra/Private/DataConfig/EngineExtra/Deserialize/DcDeserializeBPClass.cpp
{
    "AnyStructField1" : {
        "$type" : "/DataConfig/DcFixture/DcTestBlueprintStructWithColor",
        "NameField" : "Foo",
        //...
    }
}

We want to consume the $type key and its value, and then delegate the logic back to the deserializer. The solution here is first to consume the pair. Then we put back a { then replace the reader:

// DataConfigEngineExtra/Private/DataConfig/EngineExtra/Deserialize/DcDeserializeBPClass.cpp
FDcPutbackReader PutbackReader(Ctx.Reader);
PutbackReader.Putback(EDcDataEntry::MapRoot);
TDcStoreThenReset<FDcReader*> RestoreReader(Ctx.Reader, &PutbackReader);

FDcScopedProperty ScopedValueProperty(Ctx);
DC_TRY(ScopedValueProperty.PushProperty());
DC_TRY(Ctx.Deserializer->Deserialize(Ctx));

Beware that Putback only support a limited subset of data types.

Coercion

Readers implements FDcReader::Coercion() which can be used to query if the next value can be coerced into other types.

Here's an example of reading a JSON number as string:

// DataConfigTests/Private/DcTestBlurb2.cpp
FDcJsonReader Reader(TEXT(R"(
    1.234567
)"));

//  check coercion
//  note this is a query and doesn't affect reading at all
bool bCanCoerceToStr;
DC_TRY(Reader.Coercion(EDcDataEntry::String, &bCanCoerceToStr));
check(bCanCoerceToStr);

//  read number token as stjring
//  note here we skipped parsing the number to float
FString NumStr;
DC_TRY(Reader.ReadString(&NumStr));

check(NumStr == TEXT("1.234567"));
return DcOk();

What's cool here is that FDcJsonReader does parsing at the ReadFloat() callsite. So when doing coercion above it actually skipped string to float parsing.

Here's a table of all existing coercion rules:

Some caveats regarding coercion:

• When reading from JSON/MsgPack string can be read as Name/Text for convenient.
• When reading from Property, Array/Struct can be read as a FDcBlobViewData which directly points to the memory span.

UE Core Types Handlers

We bundled some custom serialization handlers for common Unreal Engine types (think FText/FVector/FColor) to improve developer experience. You need to explicitly set it up with your serializer/deserializers.

//  setup core types serializers
FDcSerializer Serializer;
DcSetupCoreTypesSerializeHandlers(Serializer);

//  setup core types deserializers
FDcDeserializer Deserializer;
DcSetupCoreTypesDeserializeHandlers(Deserializer);

These are also used in DataConfig blueprint nodes.

Note that you always can custom these default behavior by writing your own handlers.

FText

FText is used to handle localization in UE and it's quite different from FString. When calling ReadText() on JSON/MsgPack reader we pipe the string through a helper method FTextStringHelper::CreateFromBuffer() which parses and handles NSLOCTEXT()/LOCTABLE() style string.

//  DataConfigTests/Private/DcTestProperty5.cpp
{
    "TextFieldAlpha" : "NSLOCTEXT(\"DataConfig.Tests\", \"BAR_TEST\", \"Localize String Bar\")",
    "TextFieldBeta" : "LOCTABLE(\"/Bogus/Path/To/Table\", \"FOO_TEST\")"
}

The escaping sequences in JSON makes it a bit difficult to use. With the core types handlers we also accept arrays:

//  DataConfigTests/Private/DcTestRoundtrip2.cpp
{
    // [<namespace>, <text-id>, <source>], same as `NSLOCTEXT()`
    "TextFieldAlpha" : [
        "DataConfig.Tests",
        "BAR_TEST",
        "Localize String Bar"
    ],
    // [<string-table-id>, <text-id>], same as `LOCTABLE()`
    "TextFieldBeta" : [
        "/Bogus/Path/To/Table",
        "FOO_TEST"
    ]
}

Compact Presentation

Without any special handling a FVector is serialized as a JSON object like this:

{
    "X" : 1,
    "Y" : 2,
    "Z" : 3,
}

This is fine but sometimes we want it to be more compact. DataConfig now comes with a set of serialize/deserialize handlers that writes the commonly used types in compact form.

With a struct like this:

USTRUCT()
struct DATACONFIGEXTRA_API FDcExtraCoreTypesStruct
{
    GENERATED_BODY()

    UPROPERTY() FGuid GuidField1;
    UPROPERTY() FGuid GuidField2;

    UPROPERTY() FVector2D Vec2Field1;
    UPROPERTY() FVector2D Vec2Field2;

    UPROPERTY() FVector VecField1;
    UPROPERTY() FVector VecField2;

    UPROPERTY() FPlane PlaneField1;
    UPROPERTY() FPlane PlaneField2;

    UPROPERTY() FMatrix MatrixField1;
    UPROPERTY() FMatrix MatrixField2;

    UPROPERTY() FBox BoxField1;
    UPROPERTY() FBox BoxField2;

    UPROPERTY() FRotator RotatorField1;
    UPROPERTY() FRotator RotatorField2;

    UPROPERTY() FQuat QuatField1;
    UPROPERTY() FQuat QuatField2;

    UPROPERTY() FTransform TransformField1;
    UPROPERTY() FTransform TransformField2;

    UPROPERTY() FColor ColorField1;
    UPROPERTY() FColor ColorField2;

    UPROPERTY() FLinearColor LinearColorField1;
    UPROPERTY() FLinearColor LinearColorField2;

    UPROPERTY() FIntPoint IntPointField1;
    UPROPERTY() FIntPoint IntPointField2;

    UPROPERTY() FIntVector IntVectorField1;
    UPROPERTY() FIntVector IntVectorField2;

    UPROPERTY() FDateTime DateTimeField1;
    UPROPERTY() FDateTime DateTimeField2;

    UPROPERTY() FTimespan TimespanField1;
    UPROPERTY() FTimespan TimespanField2;

};

It would be serialized like this:

{
    "GuidField1" : [       1,        2,        3,        4],
    "GuidField2" : [       0,        0,        0,        0],
    "Vec2Field1" : [        0,         1],
    "Vec2Field2" : [        0,         0],
    "VecField1" : [        0,         0,         1],
    "VecField2" : [        0,         0,         0],
    "PlaneField1" : [        1,         2,         3,         4],
    "PlaneField2" : [        0,         0,         0,         0],
    "MatrixField1" : [
        [        1,         0,         0,         0],
        [        0,         1,         0,         0],
        [        0,         0,         1,         0],
        [        0,         0,         0,         1]
    ],
    "MatrixField2" : [
        [        0,         0,         0,         0],
        [        0,         0,         0,         0],
        [        0,         0,         0,         0],
        [        0,         0,         0,         0]
    ],
    "BoxField1" : {
        "Min" : [        0,         0,         0],
        "Max" : [        0,         0,         1],
        "IsValid" : 1
    },
    "BoxField2" : {
        "Min" : [        1,         1,         1],
        "Max" : [        0,         0,        -1],
        "IsValid" : 1
    },
    "RotatorField1" : [        1,         2,         3],
    "RotatorField2" : [        0,         0,         0],
    "QuatField1" : [        1,         2,         3,         4],
    "QuatField2" : [        0,         0,         0,         0],
    "TransformField1" : {
        "Rotation" : [        0,         0,         0,         1],
        "Translation" : [        1,         2,         3],
        "Scale3D" : [        1,         1,         1]
    },
    "TransformField2" : {
        "Rotation" : [        0,         0,         0,         1],
        "Translation" : [        0,         0,         0],
        "Scale3D" : [        1,         1,         1]
    },
    "ColorField1" : "#000000FF",
    "ColorField2" : "#0000FFFF",
    "LinearColorField1" : [        0,         0,         0,         1],
    "LinearColorField2" : [        1,         1,         1,         1],
    "IntPointField1" : [       1,        2],
    "IntPointField2" : [       0,        0],
    "IntVectorField1" : [       1,        2,        3],
    "IntVectorField2" : [       0,        0,        0],
    "DateTimeField1" : "0001.01.01-00.00.00",
    "DateTimeField2" : "1988.07.23-00.00.00",
    "TimespanField1" : "+00:00:00.000",
    "TimespanField2" : "+5.06:07:08.000"
}

This would make large JSON dumps more readable. Deserialize also works and it supports both array and object form.

Optional

This example only works with UE5.4+

UE 5.4 introduced optional property which now allows you to mark TOptional fields as UPROPERTY:

// DataConfigTests54/Private/DcTestUE54.h
USTRUCT()
struct FDcTestOptional
{
    GENERATED_BODY()

    UPROPERTY() TOptional<float> OptionalFloatField1;
    UPROPERTY() TOptional<float> OptionalFloatField2;

    UPROPERTY() TOptional<FString> OptionalStrField1;
    UPROPERTY() TOptional<FString> OptionalStrField2;

    UPROPERTY() TOptional<FDcInnerStruct54> OptionalStructField1;
    UPROPERTY() TOptional<FDcInnerStruct54> OptionalStructField2;
};

It's a useful utility to correctly model your data structure. For example you can use TOptional<uint32> to correctly represent an optional unsigned int, rather than using a special value like UINT32_MAX. DataConfig fully supports optional in the reader writer API:

// DataConfigTests54/Private/DcTestUE54.cpp
TOptional<FString> Source;

// ...
DC_TRY(Writer.WriteOptionalRoot());                 // Optional Root
DC_TRY(Writer.WriteString(TEXT("Some String")));    // Str
DC_TRY(Writer.WriteOptionalEnd());                  // Optional End
// ...
FString ReadStr;
DC_TRY(Reader.ReadOptionalRoot());      // Optional Root
DC_TRY(Reader.ReadString(&ReadStr));    // Str
check(ReadStr == TEXT("Some String"));  
DC_TRY(Reader.ReadOptionalEnd());       // Optional End
// ...
DC_TRY(Writer.WriteOptionalRoot()); //  Optional Root
DC_TRY(Writer.WriteNone());         //  None
DC_TRY(Writer.WriteOptionalEnd());  //  Optional End
// ...
DC_TRY(Reader.ReadOptionalRoot());  // Optional Root
DC_TRY(Reader.ReadNone());          // None
DC_TRY(Reader.ReadOptionalEnd());   // Optional End

Optional also maps perfectly with JSON since any JSON value can be null:

// DataConfigTests54/Private/DcTestUE54.cpp
FString Str = TEXT(R"(

    {
        "OptionalFloatField1" : 17.5,
        "OptionalFloatField2" : null,

        "OptionalStrField1" : "Alpha",
        "OptionalStrField2" : null,

        "OptionalStructField1" : {
            "StrField" : "Beta",
            "IntField" : 42
        },
        "OptionalStructField2" : null
    }

)");

//  equivalent fixture
FDcTestOptional Expect;
Expect.OptionalFloatField1 = 17.5f;
Expect.OptionalStrField1 = TEXT("Alpha");
Expect.OptionalStructField1.Emplace();
Expect.OptionalStructField1->StrField = TEXT("Beta");
Expect.OptionalStructField1->IntField = 42;

Note how it works with nested optional struct.

Non Class/Struct Root

Sometimes you just want to deserialize something into an TArray/TMap/TSet. Then you'll realize that you don't have something corresponding to StaticClass()/StaticStruct() as root to pass to DataConfig.

// DataConfigTests/Private/DcTestBlurb2.cpp
FString Fixture = TEXT("[1,2,3,4,5]");
TArray<int> Arr;

Turns out you can create adhoc FProperty without USTRUCT/UCLASS parents and use them just fine. In DataConfig we've provided DcPropertyUtils::FDcPropertyBuilder to ease this use case.

• DcPropertyUtils.h

// DataConfigTests/Private/DcTestBlurb2.cpp
//  create int array property
using namespace DcPropertyUtils;
auto ArrProp = FDcPropertyBuilder::Array(
    FDcPropertyBuilder::Int()
    ).LinkOnScope();

FDcJsonReader Reader{Fixture};
DC_TRY(DcAutomationUtils::DeserializeFrom(&Reader, FDcPropertyDatum(ArrProp.Get(), &Arr)));

//  validate results
check(Arr.Num() == 5);
check(Arr[4] == 5);

Note that FDcPropertyBuilder would create a heap allocated FProperty and LinkOnScope() returns a TUniquePtr. You might want to cache the properties if used repeatedly.

No MetaData

Some DataConfig behaviors take advantage of property metadata. For example the default DcSkip metadata let DataConfig ignore marked property:

//  DataConfigTests/Private/DcTestProperty3.h
UPROPERTY(meta = (DcSkip)) int SkipField1;

However the metadata would be stripped in packaged builds and FProperty::GetMetaData/HasMetaData() set of methods would be gone. The state is roughly like this:

1. In packaged builds there's no metadata C++ methods nor any data.
2. Program targets can choose whether to keep metadata methods by setting bBuildWithEditorOnlyData in Target.cs. This toggles macro WITH_EDITORONLY_DATA. However the actual metadata would not be auto loaded.
3. In editor metadata works as intended.

In release 1.4.1 we fixed compilation on !WITH_EDITORONLY_DATA. This would allow you to include DataConfigCore as a runtime module and ship it with your project. This is an intended and supported use case.

However with no metadata some default DataConfig behaviors would be different:

• All builtin metas (DcSkip/DcMsgPackBlob) are ignored.
• Bitflag enums (UENUM(meta = (Bitflags))) are ignored. All enums are now treated as scalar.

The good news is that you can workaround this by writing custom serialization handlers. For example for bitflag enums:

• Name your bitflag enums with a prefix like EFlagBar and check for it in predicates.
• Collect a list of bitflag enum names and test for it in predicates.
• On serializing PeekRead for next data entry. If it's a ArrayRoot then treat it as a bitflag enum.

Note that you should be able to implement these without modifying DataConfigCore code.

Automation

One thing we find that's really important when maintaining DataConfig across multiple UE versions, is that proper automation is a must. On this page we document how to run the bundled automation tests.

Note that all instructions shown here are using Windows with a Cmd shell. You'll need to adapt to your system setup.

DataConfigHeadless

We provide a standalone, command target that runs tests on core, non editor DataConfig features. This requires a source build to work, that means downloaded dists from Epic Launcher can not build this target.

1. Get a copy of Unreal Engine source code. Also checkout DataConfig supported UE versions.

2. Run ./GenerateProjectFiles.bat under Unreal Engine root. Note that you don't need to build the editor. This step would build UnrealBuildTool which is enough for the headless target to work.

3. Get a copy of DataConfig repository. At the folder where DataConfig.uplugin and DataConfig4.uplugin resides:

   # UE5
   <PathToUE5SourceBuild>/Engine/Build/BatchFiles/RunUBT -project="%CD%/DataConfig.uplugin" DataConfigHeadless Win64 Debug
   
   # UE4
   <PathToUE4SourceBuild>/Engine/Binaries/DotNET/UnrealBuildTool.exe -project="%CD%/DataConfig4.uplugin" DataConfigHeadless Win64 Debug
   

   Note how UE4/5 uses DataConfig4.uplugin/DataConfig.uplugin respectively.

4. Run the headless target binary:

   %CD%/Binaries/Win64/DataConfigHeadless-Win64-Debug.exe
   

   A success run looks like this:

   LogDataConfigCore: Display: UE Version: 5.1.0, DataConfigCore Version: 1.4.0, 10400
   LogDataConfigCore: Display: Filters: DataConfig.
   -   OK | DataConfig.Core.Property.StackScalarRoots
   -   OK | DataConfig.Extra.SerDe.Base64
   .......
   -   OK | DataConfig.Extra.InlineStructUsage
   -   OK | DataConfig.Core.Property.Primitive1
   Run:  122, Success:  122, Fail:    0
   

Running the benchmarks

DataConfigHeadless accepts command line arguments as filters. The benchmarks runs slower are skipped by default. You can build and run the benchmarks with the commands below:

# remember to build in Shipping
<PathToUE5SourceBuild>Engine/Build/BatchFiles/RunUBT -project="%CD%/DataConfig.uplugin" DataConfigHeadless Win64 Shipping
%CD%/Binaries/Win64/DataConfigHeadless-Win64-Shipping.exe DataConfigBenchmark

Build and run Linux target with WSL2

UE officially support cross compiling for linux and distribute toolchains on its website. Here we demonstrate how to build the headless target for Linux and run it under WSL2.

1. Setup WSL2 following this guide. You should get the latest LTS Ubuntu. Run wsl lsb_release -ir to validate it's working.

   > wsl lsb_release -ir
   Distributor ID: Ubuntu
   Release:        22.04
   

2. Install UE cross compile toolchain. Grab the installer and install on your machine.

   • Cross-compile toolchain for UE4
   • Cross-compile toolchain for UE5

   Note that each UE version matches a different toolchain. You can download and install multiple toolchains and select which to use at build time through environment variable.

3. Build the headless target for Linux.

   # UE5.1
   set LINUX_MULTIARCH_ROOT=<PathToToolchains>/v20_clang-13.0.1-centos7
   <PathToUE5SourceBuild>/Engine/Build/BatchFiles/RunUBT -project="%CD%/DataConfig.uplugin" DataConfigHeadless Linux Debug
   
   # UE4
   set LINUX_MULTIARCH_ROOT=<PathToToolchains>/v19_clang-11.0.1-centos7
   <PathToUE4SourceBuild>/Engine/Binaries/DotNET/UnrealBuildTool.exe -project="%CD%/DataConfig4.uplugin" DataConfigHeadless Linux Debug
   

4. Run the headless target through WSL.

   wsl ./Binaries/Linux/DataConfigHeadless-Linux-Debug 
   

   A success run looks like this:

   Using Mimalloc.
   LogInit: Build: ++UE5+Release-5.1-CL-0
   LogInit: Engine Version: 5.1.0-0+++UE5+Release-5.1
   LogInit: Compatible Engine Version: 5.1.0-0+++UE5+Release-5.1
   LogInit: OS: Ubuntu 22.04.1 LTS (5.10.102.1-microsoft-standard-WSL2)
   .......
   LogDataConfigCore: Display: UE Version: 5.1.0, DataConfigCore Version: 1.4.0, 10400
   LogDataConfigCore: Display: Filters: DataConfig.
   -   OK | DataConfig.Core.MsgPack.TestSuite
   -   OK | DataConfig.Core.Property.Blob2
   .......
   -   OK | DataConfig.Core.Serialize.ObjectRef
   -   OK | DataConfig.Core.MsgPack.Extension
   Run:  122, Success:  122, Fail:    0
   LogCore: Engine exit requested (reason: DataConfigHeadless Main Exit)
   LogExit: Preparing to exit.
   LogExit: Object subsystem successfully closed.
   LogExit: Exiting.
   

DcCoreTestsCommandlet

DataConfigHeadless is a Program target that does not depend on Engine and UnrealEd module. It get faster compile and iteration time, but losing functionality to touch any gameplay and editor code.

With the editor target we have a DcCoreTestsCommandlet that can be run through the commandline editor target. On top of that this works with pre-built editor downloaded from Epic Launcher.

1. Integrate DataConfig plugin into your project. You can also find a clean project bundled at Misc/Project.

2. Build the editor target. You can do it in Visual Studio or with commands below:

   # UE5
   # build
   <PathToUE5>/Engine/Build/BatchFiles/RunUBT DcProjectEditor Win64 Development %CD%/DcProject5.uproject -NoHotReload -NoEngineChanges
   # run commandlet
   <PathToUE5>/Engine/Binaries/Win64/UnrealEditor-Cmd.exe %CD%/DcProject5.uproject DataConfigEditorExtra.DcCoreTestsCommandlet
   
   # UE4
   # build
   <PathToUE4>/Engine/Binaries/DotNET/UnrealBuildTool.exe DcProjectEditor Win64 Development %CD%/DcProject4.uproject -NoHotReload -NoEngineChanges
   # run commandlet
   <PathToUE4>/Engine/Binaries/Win64/UE4Editor-Cmd.exe %CD%/DcProject4.uproject DataConfigEditorExtra.DcCoreTestsCommandlet
   

   A success run looks like this:

    [2022.10.25-14.33.52:893][  0]LogTextureFormatETC2: Display: ETC2 Texture loading DLL: TextureConverter.dll
    [2022.10.25-14.33.52:903][  0]LogTargetPlatformManager: Display: Loaded TargetPlatform 'Android'
    [2022.10.25-14.33.52:903][  0]LogTargetPlatformManager: Display: Loaded TargetPlatform 'Android_ASTC'
    [2022.10.25-14.33.52:903][  0]LogTargetPlatformManager: Display: Loaded TargetPlatform 'Android_DXT'
    .......
    [2022.10.25-14.39.30:999][  0]LogDataConfigCore: Display: Filters: DataConfig.
    -   OK | DataConfig.Core.RoundTrip.MsgPack_Persistent_StringSoftLazy
    -   OK | DataConfig.EditorExtra.GameplayEffect
    .......
    -   OK | DataConfig.Core.Property.PropertyBuilder3
    -   OK | DataConfig.EditorExtra.BPObjectInstance
    Run:  129, Success:  129, Fail:    0
   

Running automation in the editor

Lastly you can run and debug automation tests from the editor.

It's the most conventional way to run automation and is well supported in DataConfig.

Unreal Engine Upgrades

DataConfig is committed to support multiple UE versions with no deprecations and warnings. On this page we'll document important upgrade and migration info.

UE5.5

• EAutomationTestFlags is now a enum class type.
• TIsTriviallyDestructible is deprecated over std::is_trivially_destructible_v.
• TFieldPath(OtherPropertyType*) constructor now checks for actual type safety.
• FProperty::ElementSize deprecated over GetElementSize.
• StructUtils plugin is deprecated with all things moved in engine.
• PER_MODULE_BOILERPLATE not needed anymore as UBT handles it automatically.
• FVerseValueProperty renamed to FVValueProperty, also compiled only with WITH_VERSE_VM define.

UE5.4

• New property Optional and VValue are added. We fully support Optional starting by adding EDcDataEntry::OptionalRoot/OptionalEnd and then evantually it works with all DataConfig APIs including JSON/MsgPack serialization and property builder.
• FObjectPtrProperty/FClassPtrProperty are deprecated. It's introduced in 5.0 now removed and alias to FObjectProperty/FClassProperty respectively.
• Defaults to MSVC \W4 flag now which checks for unreachable code. It reports confusing locations and you can set bUseUnity = false in your *.Build.cs module rules to locate whereabout.
• FText internal pointer changed from TSharedRef to TRefCountPtr.

UE5.3

• Introduces BuildSettingsVersion.V4 which now defaults to C++ 20.
• TRemoveConst is deprecated over std::remove_const.
• FScriptDelegate etc now has additional checkers based on threading model and debug/release build. Thus we change how FScriptDelegateAccess works.

UE5.2

• TIsSame is deprecated over std::is_same.
• In Build.cs bEnforceIWYU is changed to enum IWYUSupport.

UE5.1

• UE5.1 deprecates ANY_PACKAGE in favor of a new method FindFirstObject. In DataConfig we provided DcSerdeUtils::FindFirstObject which calls FindObject(ANY_PACKAGE) pre 5.1 and calls FindFirstObject() for 5.1 and onwards.

UE5.0

• New TObjectPtr to replace raw UObject pointers. Turns out this is mostly handled within the engine and is transparent to DataConfig.

• New property types FObjectPtrProperty and FClassPtrProperty are added. They're handled the same as FObjectProperty and FClassProperty respectively.

• FVector now is 3 doubles, and Real data type in Blueprint now is also double. This is also mostly transparent to DataConfig.

• FScriptArrayHelperAccess size changes with a added uint32 ElementAlignment.

• TStringBuilderWithBuffer API changes. At call sites we now do Sb << TCHAR('\n') instead of Sb.Append(TCHAR('\n')) .

UE4

• The oldest version DataConfig supports is UE 4.25, in which it introduces a major refactor that changes UProperty to FProperty. We intended to support UE4 in the foreseeable future, especially when we now have separated uplugin for UE4 and UE5.

Breaking Changes

On this page we'll document breaking changes across versions.

1.7.1

• FDcPropertyWriter will now empty TArray/TSet/TMap on write.

  Previously when running into container property with default values DataConfig will thrash the defaults also leaving the container with incorrect elements count. Now the containers will be properly emptied on write.

  // Example of TArray property with default values.
  USTRUCT()
  struct FDcTestStructWithDefaults
  {
      // ...
      UPROPERTY() TArray<FString> StrArrFieldWithDefault = {"Foo", "Bar", "Baz"};
  };
  

  Though this is a behavior change but it should have minimal impact. See DataConfig.Core.Property.Defaults test for details.

1.7.0

• UE 5.5 deprecated StructUtils plugin, thus we'll need to setup multiple uplugins for UE5. See: DataConfigXX.uplugin

1.6.0

• Added a optional FName to FDcSerializer/FDcDeserializer::AddPredicatedHandler which can be used to identify an entry so one can replace it later like this:

  Ctx.Deserializer->PredicatedDeserializers.FindByPredicate([](auto& Entry){
  return Entry.Name == FName(TEXT("Enum"));
  })->Handler = FDcDeserializeDelegate::CreateStatic(DcEngineExtra::HandlerBPEnumDeserialize);
  

• Removed DcDeserializeUtils/DcSerializeUtils::PredicateIsUStruct. Use FDcSerializer/FDcDeserializer::AddStructHandler instead. It's more accurate and overall gets better performance.

  For an concrete example, change this:

  Ctx.Deserializer->AddPredicatedHandler(
      FDcDeserializePredicate::CreateStatic(DcDeserializeUtils::PredicateIsUStruct<FMsgPackTestSuite>),
      FDcDeserializeDelegate::CreateStatic(HandlerMsgPackTestSuite)
  );
  

  to this:

  Ctx.Deserializer->AddStructHandler(
      TBaseStructure<FMsgPackTestSuite>::Get(),
      FDcDeserializeDelegate::CreateStatic(HandlerMsgPackTestSuite)
  );
  

  and that's done. For more details see: Serializer Deserializer Setup.

• Move many DataConfigEditorExtra content to the new DataConfigEngineExtra runtime module so it can be used at runtime. This is mostly for the blueprint nodes to be available at engine runtime.

1.5.0

• Renamed EDcDataEntry::Nil to EDcDataEntry::None. This is a fix for MacOS build.

Changes

All notable changes to this project will be documented in this file.

1.7.2 - 2024-11-27

• New Better FText deserialize handling.
  • See: Core Types - FText

1.7.1 - 2024-11-20

• BREAKING Empty TArray/TSet/TMap properties on write.
  • See: Breaking - 1.7.1

1.7.0 - 2024-10-2

• Initial UE 5.5 support.
• BREAKING Separate DataConfig54.uplugin and DataConfig.uplugin.
  • See: Breaking - 1.7.0

1.6.2 - 2024-5-18

• Misc Shipping configuration build fixes.

1.6.1 - 2024-4-28

• Fix compilation for UE 5.4.

1.6.0 - 2024-3-31

• BREAKING Removed PredicateIsUStruct. Use struct handlers instead.
  • See: Breaking - 1.6.0
• NEW JSON Blueprint Library Nodes
  • See: Extra - Blueprint Nodes
  • Moved most editor extra samples to engine so it can be used at runtime.
• NEW Struct handlers added to serializer/deserializers.
  • See: Serializer Deserializer Setup
• NEW JSON Writer improvements.
  • Allows override config for inline object/arrays.
    See: JSON - Override Config
  • Add float/int format strings to config.
• NEW UE core types serializers and deserializers.
  • See: UE Core Types
• NEW Property reader/writer heuristic stack overflow detection.
  • This avoids a hard crash when serializing running into infinite loop.
• FIX EditorConfig fixes.
  • Now properly indented with tab just like UE code base.

1.5.0 - 2024-2-20

• BREAKING EDcDataEntry::Nil renamed to EDcDataType::None.
  • This is necessary to fix Mac builds.
• UE 5.4 support.
  • See UE version upgrade 5.4
• NEW Optional support.
  • See Optional

1.4.3 - 2023-8-27

• NEW Extra samples.
  • NDJSON
  • Root Object
  • Nested Arrays
• CHANGE small QoL changes:
  • Fix FDcDiagnostic enum argument captures.
  • Better FDcJsonReader::FinishRead() behavior.
  • Diagnostic message fixes.
  • Relax JSONReader a bit to allow root object/array.
  • Add PredicateIsRootProperty util to select root property.

1.4.2 - 2023-8-1

• UE 5.3 support.
  • See: UE version upgrade 5.3

1.4.1 - 2023-2-15

• Fix compilation on !WITH_EDITORONLY_DATA.
  • See No MetaData.
• UE 5.2 support.
  • See: UE version upgrade 5.2

1.4.0 - 2022-11-17

Checkout blog post "DataConfig 1.4 Released with UE 5.1 Support".

• NEW Support for UE 5.1.
  • See UE version upgrade.
• NEW Extra samples and docs.
  • Advanced - Automation
  • Extra - InstancedStruct
  • Configuring with FPropertyConfig
  • Writing Handlers - Coercion
• CHANGE Use separated uplugin for UE4 and UE5.
  • See Integration
• FIX Core changes and fixes:
  • Fail when FPropertyWriter::WriteObjectReference() takes nullptr. See test: DataConfig.Core.Property.DiagNullObject
  • Additional check for class mismatch for default object deserialize handlers. See test: DataConfig.Core.Deserialize.DiagObjectClassMismatch

1.3.0 - 2022-6-20

Checkout blog post "DataConfig Core and JSON Asset 1.3 Release".

• CHANGE Property reader/writer improvement and fixes
  • Allow reading Array/Set/Map and native array like int arr[5] as root.
    See: Non Class/Struct Root
  • Expand property with ArrayDim > 1 as array.
    See: Property
  • Performance improvement by caching FScript/Array/SetHelper
• NEW Extra samples:
  • Dump Asset To Log
  • Deserialize From SQLite Query
  • Inline Struct
• FIX Core changes and fixes:
  • Fix stale enum property fields serialization crash.
  • Fix TSet/TMap serialization crashes.
  • Fix soft object/class reference serialize to nil when it's not loaded.
  • Fix pipe property class/object handlers.
  • Fix TObjectPtr<> serialialzation and deserialization.
  • Fix PeekReadDataPtr on class property.
  • Removed FScopedStackedReader/FScopedStackedWriter usage.
  • Fix HeuristicVerifyPointer diagnostic.
  • Fix DC_TRY shadowing variable Ret.
  • JSON now support non string keyed TMap<> as [{ "$key": <foo>, "$value": <bar> }].
    See: JSON - Map
  • Fix DcPropertyUtils::DcIsSubObjectProperty()
    Now it only checks for CPF_InstancedReference.
  • Update screenshots to UE5.
    Note that DataConfig still supports from 4.25 and onwards.
  • Update Property docs.

1.2.2 - 2022-4-5

• Support for UE 5.0.0
• Add DebugGetRealPropertyValue for double BP fields.

1.2.1 - 2022-2-23

• Support for UE5 Preview 1.
• FIX Compile fixes for examples on UE 4.25.
• FIX UE 4.25/4.26 editor extra BP serde automation test fixes.
• FIX FDcAnsiJsonWriter writes non ascii char to ? when string contains escaping characters.
  • FDcJsonWriter was unaffected. This only happens to the ansi char writer and only when input has escapes like \t.

1.2.0 - 2022-1-26

Checkout blog post "Introducing DataConfig 1.2".

• NEW Serializer. Previously we only have deserializers.
  • Serializer API mirrors deserializers.
  • Builtin serialization and deserialization handlers are all roundtrip-able.
  • DcDiagnosticDeserialize -> DcDiagnosticSerDe for sharing diagnostics.
  • DcDeserializeUtils -> DcSerDeUtils for sharing code.
• NEW MsgPack reader and writer.
  • Full spec implemented, minus the "Timestamp extension type".
  • Integrate and passes kawanet/msgpack-test-suite.
• NEW JSON writer.
  • With WIDECHAR/ANSICHAR specialization as JSON Reader.
  • Accept config to output pretty or condensed output.
• NEW Builtin metas.
  • DcSkip - skip marked fields
  • DcMsgPackBlob - marked TArray<>/Struct would be read as blob by MsgPack SerDe
• CHANGE Core type changes.
  • FDcReader/FDcWriter changes.
    • FDcStruct/ClassStat renamed to FDcStruct/ClassAccess.
    • ReadStruct/ClassRoot() renamed to ReadStruct/ClassRootAccess.
    • AddReadStruct/ClassRoot() that takes no argument for common use cases.
    • FDcReader::Coercion() now returns a FDcResult
    • [Read/Write]Soft[Object/Class]Reference takes FSoftObjectPtr.
    • RTTI with GetId() and CastById()
  • FDcSerializer/FDcDeserializer changes.
    • DcDiagnosticDeserialize -> DcDiagnosticSerDe for sharing diagnostics.
    • DcDeserializeUtils -> DcSerDeUtils for sharing code.
    • Add handlers to read/write Soft/Lazy references as is, without loading the object.
    • Implicit call Properties.Push() before context Prepare().
    • Removed FDcScopedProperty in favor of DcDeserializeUtils::RecursiveDeserialize() it's more concise.
  • FDcPropertyReader/FDcPropertyWriter changes.
    • When reading class object keys any one with $ will be ignored.
      • previously only allow $type, $path.
      • note that struct by default don't check for these. It's trivia to add the logic if you want to.
    • Add FDcPropertyReader::PeekReadDataPtr matches with PeekProperty
  • FDcJsonReader changes.
    • Remove object key length limit, which was 2048 and it's incorrect.
      • Though FName is capped at 1024, which is a Unreal Engine limit.
    • Fix ReadName() which previously would fail.
    • Fix quoted string parsing/escaping in ParseQuotedString
  • Add EDcDataEntry::Extension.
  • Add FDcPropertyDatum template constructor to directly construct one from a FSturct*.
  • FPrettyPrintWriter now print blobs with hash, previously it's pointer value.
  • DcAutomationUtils::SerializeIntoJson/DeserializeIntoJson -> SerializeInto/DeserializeFrom as we're supporting other formats.
  • Add HeuristicVerifyPointer to check common magic invalid pointers.
• Misc fixes and QOL improvements.
  • Fixed linux build. Now the headless program target cross compiles and runs under wsl.
  • Serializer/Deserializer now also report diagnostics.
  • Better Json reader diagnostic formatting. Now it clamps long lines properly.
  • Add DataConfigEditorExtra.DcCoreTestsCommandlet as tests runner.
  • Add performance benchmark.
  • Restructured DataConfig book for topics on serializer and MsgPack.

1.1.1 - 2021-10-6

• Support for UE 4.27.
• Support for UE 5.

1.1.0 - 2021-4-24

• Integrate nst/JSONTestSuite. Now DcJSONParser pass most of them. Skipped tests are also documented.
• FDcAnsiJsonReader now detect and convert non ASCII UTF8 characters. Previously these characters are dropped.
• Headless test runner pass along parameters to tests, for example DataConfigHeadless-Win64-Debug.exe Parsing -- n_array_just_minus

License

DataConfig is released under MIT License.

MIT License

Copyright (c) 2021-2024 Chen Tao 

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.

Attribution

If you find DataConfig useful in your project, consider credit us in your project with the full license above or the shorter snippets below.

DataConfig <https//slowburn.dev/dataconfig>
MIT License, Copyright (c) 2021-2024 Chen Tao

There's also DataConfig JSON Asset on UE Marketplace. It's a premium plugin for importing JSON to UE data assets.

You can reach us by email to [email protected] or on twitter @slowburndev.

We'll setup a section showcasing projects using DataConfig in the future.