DataConfig Book

Serialization framework for Unreal Engine Property System that just works!

DataConfig is a serialization framework built on top of Unreal Engine's Property System. It aims to be friendly and robust while offering good performance.

Getting Started.

  • See Examples to look at some code.
  • See Integration for quick integration guide.
  • See Design for more context about the project.
  • See Changes for versioning history.

There's also DataConfig JSON Asset on UE Marketplace. It's a premium plugin for importing JSON to UE data assets.

License

DataConfig is released under a free and permissive license. But we really appreciate you to credit us if you find it useful. See License for details

Examples

This page shows some short and quick examples showcasing DataConfig API usage and features. All code shown here can be found in the sources.

JSON Deserialization

This is the example shown on front page. Given the structFDcTestExampleStruct:

// DataConfig/DataConfig/Source/DataConfigTests/Private/DcTestBlurb.h
UENUM()
enum class EDcTestExampleEnum
{
    Foo, Bar, Baz
};

USTRUCT()
struct FDcTestExampleStruct
{
    GENERATED_BODY()
    UPROPERTY() FString StrField;
    UPROPERTY() EDcTestExampleEnum EnumField;
    UPROPERTY() TArray<FColor> Colors;
};

We can deserialize an instance from JSON with the snippet below:

// DataConfig/Source/DataConfigTests/Private/DcTestBlurb.cpp
FString Str = TEXT(R"(
    {
        "StrField" : "Lorem ipsum dolor sit amet",
        "EnumField" : "Bar",
        "Colors" : [
            "#FF0000FF", "#00FF00FF", "#0000FFFF"
        ]
    }
)");

FDcTestExampleStruct Dest;

//  create and setup a deserializer
FDcDeserializer Deserializer;
DcSetupJsonDeserializeHandlers(Deserializer);
Deserializer.AddPredicatedHandler(
    FDcDeserializePredicate::CreateStatic(DcExtra::PredicateIsColorStruct),
    FDcDeserializeDelegate::CreateStatic(DcExtra::HandlerColorDeserialize)
);

//  prepare context for this run
FDcPropertyDatum Datum(FDcTestExampleStruct::StaticStruct(), &Dest);
FDcJsonReader Reader(Str);
FDcPropertyWriter Writer(Datum);

FDcDeserializeContext Ctx;
Ctx.Reader = &Reader;
Ctx.Writer = &Writer;
Ctx.Deserializer = &Deserializer;
Ctx.Properties.Push(Datum.Property);
DC_TRY(Ctx.Prepare());

//  kick off deserialization
DC_TRY(Deserializer.Deserialize(Ctx));

//  validate results
check(Dest.StrField == TEXT("Lorem ipsum dolor sit amet"));
check(Dest.EnumField == EDcTestExampleEnum::Bar);
check(Dest.Colors[0] == FColor::Red);
check(Dest.Colors[1] == FColor::Green);
check(Dest.Colors[2] == FColor::Blue);

Note that EDcTestExampleEnum is deserialized by its name and FColor is deserialized from a html color string like #RRGGBBAA.

Say if we accidentally mistyped the EnumField value:

{
    "StrField" : "Lorem ipsum dolor sit amet",
    "EnumField" : "Far",

It would fail gracefully with diagnostics:

# DataConfig Error: Enum name not found in enum type: EDcTestExampleEnum, Actual: 'Far'
- [JsonReader] --> <in-memory>4:25
   2 |    {
   3 |        "StrField" : "Lorem ipsum dolor sit amet",
   4 |        "EnumField" : "Far",
     |                           ^
   5 |        "Colors" : [
   6 |            "#FF0000FF", "#00FF00FF", "#0000FFFF"
- [PropertyWriter] Writing property: (FDcTestExampleStruct)$root.(EEDcTestExampleEnum)EnumField

Custom Deserialization Logic

To deserialize FColor with #RRGGBBAA we'll need to provide custom logic to the deserializer. First you'll need to implement a FDcDeserializePredicate delegate to pick out FColor:

//	DataConfig/Source/DataConfigExtra/Private/DataConfig/Extra/Deserialize/DcDeserializeColor.cpp
EDcDeserializePredicateResult PredicateIsColorStruct(FDcDeserializeContext& Ctx)
{
	UScriptStruct* Struct = DcPropertyUtils::TryGetStructClass(Ctx.TopProperty());
	return Struct && Struct == TBaseStructure<FColor>::Get()
		? EDcDeserializePredicateResult::Process
		: EDcDeserializePredicateResult::Pass;
}

Then we'll need to implement a FDcDeserializeDelegate to deserialize a FColor. Here we'll do it by writing through R/G/B/A fields by name with the FDcWriter API.

// DataConfig/Source/DataConfigExtra/Private/DataConfig/Extra/Deserialize/DcDeserializeColor.cpp
template<>
FDcResult TemplatedWriteColorDispatch<EDcColorDeserializeMethod::WriterAPI>(const FColor& Color, FDcDeserializeContext& Ctx)
{
	DC_TRY(Ctx.Writer->WriteStructRoot(FDcStructStat{ TEXT("Color"), FDcStructStat::WriteCheckName }));

	DC_TRY(Ctx.Writer->WriteName(TEXT("B")));
	DC_TRY(Ctx.Writer->WriteUInt8(Color.B));

	DC_TRY(Ctx.Writer->WriteName(TEXT("G")));
	DC_TRY(Ctx.Writer->WriteUInt8(Color.G));

	DC_TRY(Ctx.Writer->WriteName(TEXT("R")));
	DC_TRY(Ctx.Writer->WriteUInt8(Color.R));

	DC_TRY(Ctx.Writer->WriteName(TEXT("A")));
	DC_TRY(Ctx.Writer->WriteUInt8(Color.A));

	DC_TRY(Ctx.Writer->WriteStructEnd(FDcStructStat{ TEXT("Color"), FDcStructStat::WriteCheckName }));

	return DcOk();
}

Then we'll need to register these pair of delegates to the FDcDeserializer.

// DataConfig/Source/DataConfigTests/Private/DcTestBlurb.cpp
FDcDeserializer Deserializer;
DcSetupJsonDeserializeHandlers(Deserializer);
Deserializer.AddPredicatedHandler(
    FDcDeserializePredicate::CreateStatic(DcExtra::PredicateIsColorStruct),
    FDcDeserializeDelegate::CreateStatic(DcExtra::HandlerColorDeserialize)
);

And then it's done. It would work recursively on FColor everywhere, like in UCLASS/USTRUCT members, in TArray/TSet and in TMap as key or values.

Writer API Alternatives

In the example above we're deserializing FColor by writing into its member fields separately, which is a bit dumb. In this case DataConfig do support better alternatives.

Since we know that FColor is POD type we can construct one by filling in correct bit pattern. In this case FDcPropertyWriter allow struct property to be coerced from a blob:

//	DataConfig/Source/DataConfigExtra/Private/DataConfig/Extra/Deserialize/DcDeserializeColor.cpp
template<>
FDcResult TemplatedWriteColorDispatch<EDcColorDeserializeMethod::WriteBlob>(const FColor& Color, FDcDeserializeContext& Ctx)
{
	return Ctx.Writer->WriteBlob({
		(uint8*)&Color,	// treat `Color` as opaque blob data
		sizeof(FColor)
	});
}

Alternatively we can get FProperty and data pointer in place and setting the value through Unreal's FProperty API:

// DataConfig/Source/DataConfigExtra/Private/DataConfig/Extra/Deserialize/DcDeserializeColor.cpp
template<>
FDcResult TemplatedWriteColorDispatch<EDcColorDeserializeMethod::WriteDataEntry>(const FColor& Color, FDcDeserializeContext& Ctx)
{
	FDcPropertyDatum Datum;
	DC_TRY(Ctx.Writer->WriteDataEntry(FStructProperty::StaticClass(), Datum));

	Datum.CastFieldChecked<FStructProperty>()->CopySingleValue(Datum.DataPtr, &Color);
	return DcOk();
}

Note that we already know that Datum.DataPtr points to a allocated FColor instance. Thus we can simply cast it into a FColor* and directly manipulate the pointer.

// DataConfig/Source/DataConfigExtra/Private/DataConfig/Extra/Deserialize/DcDeserializeColor.cpp
template<>
FDcResult TemplatedWriteColorDispatch<EDcColorDeserializeMethod::WritePointer>(const FColor& Color, FDcDeserializeContext& Ctx)
{
	FDcPropertyDatum Datum;
	DC_TRY(Ctx.Writer->WriteDataEntry(FStructProperty::StaticClass(), Datum));

	FColor* ColorPtr = (FColor*)Datum.DataPtr;
	*ColorPtr = Color;	// deserialize by assignment

	return DcOk();
}

Debug Dump

DcAutomationUtils::DumpToLog() can dump a FDcPropertyDatum to a string representation, in which FDcPropertyDatum is simply a FProperty + void* pair that can represent anything in the property system:

// DataConfig/Source/DataConfigTests/Private/DcTestBlurb.cpp
FVector Vec(1.0f, 2.0f, 3.0f);
FDcPropertyDatum VecDatum(TBaseStructure<FVector>::Get(), &Vec);

DcAutomationUtils::DumpToLog(VecDatum);

Output would be:

-----------------------------------------
# Datum: 'ScriptStruct', 'Vector'
<StructRoot> 'Vector'
|---<Name> 'X'
|---<Float> '1.000000'
|---<Name> 'Y'
|---<Float> '2.000000'
|---<Name> 'Z'
|---<Float> '3.000000'
<StructEnd> 'Vector'
-----------------------------------------

Additionally we wrapped this into gDcDebug that can be invoked in MSVC immediate window. Calling it during debug would dump into MSVC Output window:

// DataConfig/Source/DataConfigCore/Public/DataConfig/Automation/DcAutomationUtils.h
struct DATACONFIGCORE_API FDcDebug
{
	FORCENOINLINE void DumpStruct(char* StructNameChars, void* Ptr);
	FORCENOINLINE void DumpObject(UObject* Obj);
	FORCENOINLINE void DumpDatum(void* DatumPtr);
};

///	Access `gDcDebugg` in MSVC immediate window:
///
///	- in monolith builds:
///	gDcDebug.DumpObject(Obj)
///
///	- in DLL builds prefix with dll name:
///	({,,UE4Editor-DataConfigCore}gDcDebug).DumpObject(ObjPtr)

extern FDcDebug gDcDebug;

Here's an animated demo showing dumping the vector above during debug break in MSVC:

Examples-DebugDumpVecDatum

The full expression to evaluate is:

({,,UE4Editor-DataConfigCore}gDcDebug).DumpDatum(&VecDatum)

We need DLL name to locate gDcDebug in a non monolith build.

More

There's more detailed examples in DataConfigExtra and DataConfigEditorExtra

Integration

This page shows integration instructions for DataConfig. At the moment it supports these the engine versions below:

  • UE 4.25
  • UE 4.26

Integrate DataConfig Plugin

The easiest way to try out DataConfig is to add it as a plugin into your C++ project. In this section we'll walk through the steps of integrating DataConfig plugin into a empty UE C++ Project.

  1. Get a copy of the repository. Then copy ./DataConfig (where DataConfig.uplugin is located) into your project's Plugins directory.

  2. Delete DataConfig\Source\DataConfigHeadless folder. It has a DataConfigHeadless.Target.cs file which is a hack to build a headless binary during development. This step is crucial or you your project won't build.

  3. Restart your project. There should be a prompt to compile plugin sources. Confirm and wait until your project launches. Then open Settings -> Plugins you should see Data Config listed under Project Editor category.

    Integration-DataConfigPlugin

  4. The plugin comes with a set of tests. Open menu Window -> Developer Tools -> Session Frontend. Find and run the DataConfig tests and it should all pass.

    Integration-DataConfigAutomations

Integrate DataConfigCore Module

DataConfig is packed into a plugin to bundle automation tests with a few assets. You're encouraged to integrate only the DataConfigCore module. It contains all core features with minimal dependencies.

Most projects should has a editor module already setup. In this section we'll go through the steps of integrating DataConfigCore and build it with the project's FooProjectEditor module.

  1. Get a copy of this repository. Then copy DataConfig/Source/DataConfigCore into your project's Source directory.

  2. Edit FooProjectEditor.Build.cs add add DataConfigCore as an extra module:

    using UnrealBuildTool;
    
    public class FooProjectEditor : ModuleRules
    {
        public FooProjectEditor(ReadOnlyTargetRules Target) : base(Target)
        {
            PublicDependencyModuleNames.AddRange(new string[] { 
    			//...
                "DataConfigCore",	// <- add this
                });
        }
    }
    
  3. DataConfig needs to be explicitly initialized before use. Find FooProjectEditor module's start up and shut down methods and setup DataConfig accordingly.

    #include "DataConfig/DcEnv.h"
    #include "DataConfig/Automation/DcAutomationUtils.h"
    
    void FFooProjectEditorModule::StartupModule()
    {
        // ...
        DcStartUp(EDcInitializeAction::SetAsConsole);
        // dump a FVector to try it out
        FVector Vec(1.0f, 2.0f, 3.0f);
        FDcPropertyDatum VecDatum(TBaseStructure<FVector>::Get(), &Vec);
    
        DcAutomationUtils::DumpToLog(VecDatum);
    }
    
    void FFooProjectEditorModule::ShutdownModule()
    {
        // ...
    	DcShutDown();
    }
    
  4. Rebuild the project and restart the editor. Open Output Log and use LogDataConfigCore filter and find the dump output.

    Integration-DataConfigCoreOutput

You can refer to DataConfigEditorExtra module for more detailed integration options.

Design

This page documents the overall design, goals and reasoning around DataConfig.

Rationale

At the time we started this project we're looking for a JSON parser that:

  • Supports a relaxed JSON spec, i.e. comment and trailing comma.
  • Supports custom deserialization logic, i.e. deserialize FColor from #RRGGBBAA.
  • Supports UE4 instanced sub objects and polymorphism.

Eventually we implemented all these in DataConfig. We also tried not limit this to be a JSON parser but deliver as a set of tools for reading-from and writing-to the property system.

If you're Unreal Engine C++ developers that:

  • Looking for alternative JSON parser.
  • Looking for a textual configuration format.
  • Thinking of implementing custom textual/binary format.
  • Regularly dealing with FProperty related code.

You should give DataConfig a try and it's highly likely DataConfig might fit into part of your solution.

Manifesto

  • Deliver as a quality C++ source library.

    DataConfig should ship with no UI nor tooling code. Users are expected to integrate only DataConfigCore as a source module. We're intentionally limiting the scope of DataConfig to be a "C++ Library". Our users should be proficient UE4 C++ programmers.

    • DataConfig should ship with good test and documentation coverage.
    • DataConfig follows idiomatic UE4 C++ conventions and has no external dependency.
    • DataConfigCore only depends on Core and CoreUObject and can be used in standalone Program targets.
    • DataConfig API are UObject free and stack friendly.
    • Built-in features serve as examples and sensible defaults. Users are expected to write on their own Reader/Writer/Handlers.
  • Runtime performance is not top priority.

    We expect users to use DataConfig in a offline, editor only scenario. In this use case we favor some other aspects over runtime performance:

    • Idiomatic. We follow UE4 c++ coding conventions and keep core dependency to only Core and CoreUObject.
    • Friendly. When processing invalid data and invalid API usage DataConfig should not crash. It should fail explicitly with detailed context and diagnostics.
    • Small code size / fast compile time. DataConfig try not to expose template API. TDcJsonReader is explicit instantiated with its definition in private files.
    • Light memory footprint. Our JSON parser do stream parsing and would not construct the loaded JSON document in memory at all.
  • Works with whatever property system supports.

    The idea is that DataConfig supports everything that can be marked with UCLASS/USTRUCT/UPROPERTY/UENUM macros, which covers the full data model of the property system.

    Fields like weak object reference and delegates doesn't make much sense to be serialized into textual format. But turns out supporting the full data model makes DataConfig suitable to some other tasks like debug dump and in-memory data wrangling.

    This also means that DataConfig only focus on reading from and writing into C++ data structures. For example we don't have a DOM or object like API for JSON at all. The only use case DataConfig supports is to deserialize from JSON into native C++ objects.

Programming Guides

This section contains doc for programming DataConfig APIs.

Core Types

This page documents core data types in DataConfigCore

DataConfig Data Model

Conceptually the DataConfig data model is defined by 3 C++ types:

  1. EDcDataEntry - enum covers every possible data type.
  2. FDcReader - methods to read from the data model.
  3. FDcWriter - methods to write into the data model.

And that's it. The obvious missing thing is a DOM like object that you can random access and serialize into - we choose to not implement that and it's crucial to understand this to get to know how DataConfig works.

EDcDataEntry

The enum covers all possible types:

// DataConfig/DataConfig/Source/DataConfigCore/Public/DataConfig/DcTypes.h
UENUM()
enum class EDcDataEntry : uint16
{
	Nil,

	Bool,
	Name,
	String,
	Text,
	Enum,

	Float,
	Double,

	Int8,
	Int16,
	Int32,
	Int64,

	UInt8,
	UInt16,
	UInt32,
	UInt64,

	//	Struct
	StructRoot,
	StructEnd,

	//	Class
	ClassRoot,
	ClassEnd,

	//	Map
	MapRoot,
	MapEnd,

	//	Array
	ArrayRoot,
	ArrayEnd,

	//	Set,
	SetRoot,
	SetEnd,

	//	Reference
	ObjectReference,
	ClassReference,

	WeakObjectReference,
	LazyObjectReference,
	SoftObjectReference,
	SoftClassReference,
	InterfaceReference,

	//	Delegates
	Delegate,
	MulticastInlineDelegate,
	MulticastSparseDelegate,

	//	Field
	FieldPath,

	//	Extension
	Blob,

	//	End
	Ended,
};

Most enumerators directly maps to a FProperty type:

  • EDcDataEntry::Bool - FBoolProperty
  • EDcDataEntry::Name - FNameProperty
  • EDcDataEntry::String - FStrProperty
  • EDcDataEntry::ArrayRoot/ArrayEnd- FArrayProperty

It should've covered all possible FProperty types. Addition to that there're a few that has

  • EDcDataEntry::Nil - It's added to match null in JSON, and it's also used to explicitly represent null object reference.
  • EDcDataEntry::Ended - It's a phony type that is returned when there's no more data or reader/writer is in a invalid state.
  • EDcDataEntry::Blob - It's a extension to allow direct memory read/write from given fields.

FDcReader

FDcReader is the one and only way to read from DataConfig data model. For every enumerator in EDcDataEntry there's a member method on FDcReader to from it.

Here we set up a simple struct trying out the reader methods:

// DataConfig/Source/DataConfigTests/Private/DcTestBlurb.h
USTRUCT()
struct FDcTestExampleSimple
{
	GENERATED_BODY()

	UPROPERTY() FString StrField;
	UPROPERTY() int IntField;
};

// DataConfig/Source/DataConfigTests/Private/DcTestBlurb.cpp
FDcTestExampleSimple SimpleStruct;
SimpleStruct.StrField = TEXT("Foo Str");
SimpleStruct.IntField = 253;

Since we know exactly how the FDcTestExampleSimple looks like we can manually arrange the read calls:

// DataConfig/Source/DataConfigTests/Private/DcTestBlurb.cpp
FDcPropertyReader Reader(FDcPropertyDatum(FDcTestExampleSimple::StaticStruct(), &SimpleStruct));

DC_TRY(Reader.ReadStructRoot(&Struct));   // `FDcTestExampleSimple` Struct Root

    DC_TRY(Reader.ReadName(&FieldName));  // 'StrField' as FName
    DC_TRY(Reader.ReadString(&StrValue)); // "Foo STr"

    DC_TRY(Reader.ReadName(&FieldName));  // 'IntField' as FName
    DC_TRY(Reader.ReadInt32(&IntValue));  // 253

DC_TRY(Reader.ReadStructEnd(&Struct));    // `FDcTestExampleSimple` Struct Root

In the example above FDcReader behave like a iterator as each ReadXXX() call emits value and move the internal cursor into the next slot. In case we're reading a unknown structure, we can use FReader::PeekRead() to peek what's coming next.

FDcWriter

FDcWriter is the counter part of writing into the data config model. To write into the example instance above:

DC_TRY(Writer.WriteStructRoot(FDcStructStat{})); // `FDcTestExampleSimple` Struct Root

    DC_TRY(Writer.WriteName(TEXT("StrField")));      // 'StrField' as FName
    DC_TRY(Writer.WriteString(TEXT("Alt Str")));     // "Foo STr"

    DC_TRY(Writer.WriteName(TEXT("IntField")));      // 'IntField' as FName
    DC_TRY(Writer.WriteInt32(233));                  // 233

DC_TRY(Writer.WriteStructEnd(FDcStructStat{}));  // `FDcTestExampleSimple` Struct Root

There's also FDcWriter::PeekRead() to query whether it's possible to write given data type.

Conclusion

DataConfig provide FDcReader and FDcWriter to access the property system. It can be considered as a friendly alternative to the property system API. It's also how we implemented flexible JSON deserialization, which would be described in later chapters.

Error Handling

Proper error handling is crucial to implement robust serialization as it needs to deal with unknown user input. DataConfig also provide diagnostic to help users quickly pin down common errors like typo or missing colons in JSON. Here's an example:

# DataConfig Error: Enum name not found in enum type: EDcTestExampleEnum, Actual: 'Far'
- [JsonReader] --> <in-memory>4:25
   2 |    {
   3 |        "StrField" : "Lorem ipsum dolor sit amet",
   4 |        "EnumField" : "Far",
     |                           ^
   5 |        "Colors" : [
   6 |            "#FF0000FF", "#00FF00FF", "#0000FFFF"
- [PropertyWriter] Writing property: (FDcTestExampleStruct)$root.(EEDcTestExampleEnum)EnumField

Internally DataConfig is applying a consistent error handling strategy across all API. User code are expected to follow along.

Returning FDcResult

The gist is that if a method can fail, return a FDcResult, which is a simple struct:

// DataConfig/DataConfig/Source/DataConfigCore/Public/DataConfig/DcTypes.h
struct DATACONFIGCORE_API DC_NODISCARD FDcResult
{
    enum class EStatus : uint8
    {
        Ok,
        Error
    };

    EStatus Status;

    FORCEINLINE bool Ok() const
    {
        return Status == EStatus::Ok;
    }
};

//  See FDcReader's methods as example
// DataConfig/Source/DataConfigCore/Public/DataConfig/Reader/DcReader.h
struct DATACONFIGCORE_API FDcReader
{
    //...
    virtual FDcResult ReadBool(bool* OutPtr);
    virtual FDcResult ReadName(FName* OutPtr);
    virtual FDcResult ReadString(FString* OutPtr);
    virtual FDcResult ReadText(FText* OutPtr);
    virtual FDcResult ReadEnum(FDcEnumData* OutPtr);
    //...
};

Then use DC_TRY to call these kinds of functions. The macro itself does early return when result is not ok:

// DataConfig/DataConfig/Source/DataConfigCore/Public/DataConfig/DcTypes.h
#define DC_TRY(expr)                        \
    do {                                    \
        ::FDcResult Ret = (expr);           \
        if (!Ret.Ok()) {                    \
            return Ret;                     \
        }                                   \
    } while (0)

//  Example of calling methods returning `FDcResult`
// DataConfig/Source/DataConfigExtra/Private/DataConfig/Extra/Deserialize/DcDeserializeColor.cpp
template<>
FDcResult TemplatedWriteColorDispatch<EDcColorDeserializeMethod::WriterAPI>(const FColor& Color, FDcDeserializeContext& Ctx)
{
    DC_TRY(Ctx.Writer->WriteStructRoot(FDcStructStat{ TEXT("Color"), FDcStructStat::WriteCheckName }));

    DC_TRY(Ctx.Writer->WriteName(TEXT("B")));
    DC_TRY(Ctx.Writer->WriteUInt8(Color.B));

    DC_TRY(Ctx.Writer->WriteName(TEXT("G")));
    DC_TRY(Ctx.Writer->WriteUInt8(Color.G));

    DC_TRY(Ctx.Writer->WriteName(TEXT("R")));
    DC_TRY(Ctx.Writer->WriteUInt8(Color.R));

    DC_TRY(Ctx.Writer->WriteName(TEXT("A")));
    DC_TRY(Ctx.Writer->WriteUInt8(Color.A));

    DC_TRY(Ctx.Writer->WriteStructEnd(FDcStructStat{ TEXT("Color"), FDcStructStat::WriteCheckName }));

    return DcOk();
}

This pattern is similar to Outcome and std::expected except we give up using the return value. Return values should be passed through reference or pointers in function arguments.

Diagnostics

When implementing a method that returns FDcResult you have 2 options:

  • Return DcOk() on succeed.
  • Return DC_FAIL(<Catetory>, <ErrId>) on error.

Examples:

// DataConfig/Source/DataConfigTests/Private/DcTestBlurb.cpp
FDcResult Succeed() {
    // succeed
    return DcOk();
}

FDcResult Fail() {
    // fail !
    return DC_FAIL(DcDCommon, Unexpected1) << "My Custom Message";
}

In the examples above DcDCommon and Unexpected1 are called error category and error id respectively. DcDCommon is a built-in error category:

// DataConfig/Source/DataConfigCore/Public/DataConfig/Diagnostic/DcDiagnosticCommon.h
namespace DcDCommon
{
static const uint16 Category = 0x1;

enum Type : uint16
{
    //...
    Unexpected1,
};

} // namespace DcDCommon

// DataConfig/Source/DataConfigCore/Private/DataConfig/Diagnostic/DcDiagnosticCommon.cpp
namespace DcDCommon
{
static FDcDiagnosticDetail _CommonDetails[] = {
    // ...
    { Unexpected1, TEXT("Unexpected: '{0}'") },
};

Note that we can pipe argument into the diagnostic. The diagnostic reported by invoking Fail() would be like:

* # DataConfig Error: Unexpected: 'My Custom Message'

Conclusion

DataConfig uses FDcResult, DC_TRY, DC_FAIL for error handling. It's lightweight and relatively easy to grasp. There's still some limitations in this regard:

  • FDcResult occupied the return position making passing value to parent a bit cumbersome.
  • For now we always stop as the first error. There's no plan to support error recovery.

Some closing notes:

  • Reported diagnostics get queued. You'll need to call FDcEnv::FlushDiags() to flush them to consumers.
  • See DcDiagnosticExtra.h/cpp for how to register user category.
  • See DcEditorExtraModule.cpp - FDcMessageLogDiagnosticConsumer for custom diagnostic handler and formatting.

Env

DataConfig put most global state into a stack of FDcEnv:

// DataConfig/Source/DataConfigCore/Public/DataConfig/DcEnv.h
struct DATACONFIGCORE_API FDcEnv
{
    TArray<FDcDiagnostic> Diagnostics;

    TSharedPtr<IDcDiagnosticConsumer> DiagConsumer;

    TArray<FDcReader*> ReaderStack;
    TArray<FDcWriter*> WriterStack;

    bool bExpectFail = false;   // mute debug break

    FDcDiagnostic& Diag(FDcErrorCode InErr);

    void FlushDiags();

    FORCEINLINE FDcDiagnostic& GetLastDiag() 
    {
        checkf(Diagnostics.Num(), TEXT("<empty diagnostics>"));
        return Diagnostics.Last();
    }

    ~FDcEnv();
};

DataConfig needs explicit initializatioon before use. This is done through manually DcStartUp(). There's also a paired DcShutdown() that should be called when DataConfig isn't used anymore. Here's an example:

// DataConfig/Source/DataConfigEditorExtra/Private/DcEditorExtraModule.cpp
void FDcEditorExtraModule::StartupModule()
{
    UE_LOG(LogDataConfigCore, Log, TEXT("DcEditorExtraModule module starting up"));
    DcRegisterDiagnosticGroup(&DcDExtra::Details);
    DcRegisterDiagnosticGroup(&DcDEditorExtra::Details);

    DcStartUp(EDcInitializeAction::Minimal);
    //...
}

void FDcEditorExtraModule::ShutdownModule()
{
    DcShutDown();
    //...
}

The active FDcEnv is accessed by calling global function DcEnv(). Inside the Env:

  • Diagnostics: all diagnostics are flushed into env.
  • DiagConsumer: diagnostic handler, format and print diagnostic to log or MessageLog or even on screen.
  • ReaderStack/WriterStack: used to pass along reader/writer down the callstack. See FScopedStackedReader uses for example.
  • ... and everything else.

You can use DcPushEnv() to create new env then destroy it calling DcPopEnv(). At this moment it's mostly used to handle reentrant during serialization. See FDcScopedEnv uses for examples.

Built-in Reader and Writers

This page briefly walks through FDcReader/FDcWriter classes bundled in DataConfigCore.

FDcPipeVisitor and FDcPrettyPrintWriter

FDcPipeVisitor takes a FDcReader and a FDcWriter then start peek-read-write loop until it peeks EDcDataEntry::Ended from reader or an error happens.

Then there's FDcPrettyPrintWriter that dumps everything write into it as string.

Combining these two we get a way to dump arbitrary FDcReader into a string!. This is how built-in debug dump features are implemented:

// DataConfig/Source/DataConfigCore/Private/DataConfig/Automation/DcAutomationUtils.cpp
void DumpToOutputDevice(...)
{
    //...
    FDcPropertyReader PropReader(Datum);
    FDcPrettyPrintWriter PrettyWriter(Output);
    FDcPipeVisitor PrettyPrintVisit(&PropReader, &PrettyWriter);

    if (!PrettyPrintVisit.PipeVisit().Ok())
        ScopedEnv.Get().FlushDiags();
    //...
}

In following sections we'll see some other usages of the pipe visitor.

FDcPropertyReader/FDcPropertyWriter

This pair of classes is used to access the actual Unreal Property System. Both takes a FDcPropertyDatum to construct:

// DataConfig/Source/DataConfigCore/Public/DataConfig/Property/DcPropertyDatum.h
struct DATACONFIGCORE_API FDcPropertyDatum
{
    FFieldVariant Property;
    void* DataPtr;
    //...
}

It's simply a property plus opaque pointer pair. These constructs are also called "Fat Pointers". Turns out this is enough to represent everything in the Property System. Property reader/writer needs one of these as a root and start reading/writing from there.

A simple use case of these is to roundtrip two objects so that every property in the first one is copied into latter:

// DataConfig/Source/DataConfigTests/Private/DcTestCommon.h
FDcResult DcTestPropertyRoundtrip(...)
{
    FDcPropertyReader Reader(FromDatum);
    FDcPropertyWriter Writer(ToDatum);
    //...
    FDcPipeVisitor RoundtripVisit(&Reader, &Writer);
    return RoundtripVisit.PipeVisit();
}

There's a quirk that you cannot create a FDcPropertyDatum for stack allocated TArray :

void f()
{
    // this is ok
    FDcTestExampleStruct MyStruct;
    FDcPropertyDatum StructDatum(&FDcTestExampleStruct::StaticStruct(), &MyStruct);

    // this is not
    TArray<int> myArr;
    FDcPropertyDatum myDatum(&???, &myArr);
}

We can, however, get datum for member TArray and other fields. We'll see related example in later chapters.

FDcJsonReader

This is the only reader that reads a external textual format. It's also an example showcasing that the DataConfig data model is actually a superset of the property system.

// DataConfig/DataConfig/Source/DataConfigTests/Private/DcTestBlurb.cpp
FString Str = TEXT(R"(
    {
        "Str":    "Fooo",
        "Number": 1.875,
        "Bool":   true
    } 
)");

FDcJsonReader Reader(Str);

//  calling read methods
FString KeyStr;
FString GotStr;
double GotNumber;
bool GotBool;

DC_TRY(Reader.ReadMapRoot());

    DC_TRY(Reader.ReadString(&KeyStr));
    DC_TRY(Reader.ReadString(&GotStr));

    DC_TRY(Reader.ReadString(&KeyStr));
    DC_TRY(Reader.ReadDouble(&GotNumber));

    DC_TRY(Reader.ReadString(&KeyStr));
    DC_TRY(Reader.ReadBool(&GotBool));

DC_TRY(Reader.ReadMapEnd());

//  validate results
check(GotStr == TEXT("Fooo"));
check(GotNumber == 1.875);
check(GotBool == true);

In the example above we deserialized a JSON object from string. The first and the last call is ReadMapRoot and ReadMapEnd, which is also used to read Unreal's TMap properties. The difference is that UE's TMap is strictly typed but JSON object values can have arbitrary type. This means that if you use FDcPipeVisitor to pipe a FDcJsonReader into a FDcPropertyWriter it won't work.

The good news is that DataConfig data model is designed to support these use cases. As long as you can use FDcReader/FDcWriter API to describe the format you want to serialize you're good to go. Mapping and conversion between these different shapes of reader/writers are handled by deserializers.

Some additional caveats:

  • Similar to Unreal's TJsonReader, we provide TDcJsonReader with 2 specializations:
    • FDcJsonReader that reads TCHAR*, FString
    • FDcAnsiJsonReader that reads char*.
  • We're supporting a relaxed superset of JSON:
    • Allow C Style comments, i.e /* block */ and // line .
    • Allow trailing comma, i.e [1,2,3,], .
    • Allow non object root. You can put a list as the root.
  • String parsing and number parsing are delegated to Unreal's built-ins:
    • Parse string: FParse::QuotedString()
    • Parse numbers: TCString::Atof/Strtoi/Strtoi64

Utilities

Finally there're some utility reader/writers for various purposes.

  • FDcNoopWriter - a writer that does literally nothing. Useful to benchmark reader performance.
  • FDcWeakCompositeWriter - a writer that multiplex into a list of writers. Useful to trace writing calls.
  • FDcPutbackReader - FDcReader doesn't support lookahead. It can only peek next item's type but not value. This class is used to support limited lookahead by putting back read value. It's used in implementing custom deserializer handlers.

Conclusion

You should consider implement new FDcReader/FDcWriter when you want to support a new file format. You can also implement utility reader/writer that nest other reader/writers.

Deserializer

In the previous chapter we mentioned that DataConfig data model is a super set of the property system. FDcDeserializer is used to convert different subsets of it into the property system.

Context

A company class to the deserializer is FDcDeserializeContext:

// DataConfig/Source/DataConfigCore/Public/DataConfig/Deserialize/DcDeserializeTypes.h
struct DATACONFIGCORE_API FDcDeserializeContext
{
    //...
    FDcDeserializer* Deserializer;
    FDcReader* Reader;
    FDcPropertyWriter* Writer;
    //...
};

Comparing to FDcPipeVisitor which takes a FDcReader and a FDcWriter, FDcDeserializeContext takes explicitly a FDcPropertyWriter to construct. The deserializer reads from arbitrary reader but writes only into the property system objects.

Handlers

Custom deserialize logic is provided through FDcDeserializeDelegate:

// DataConfig/Source/DataConfigCore/Public/DataConfig/Deserialize/DcDeserializeTypes.h
using FDcDeserializeDelegateSignature = FDcResult(*)(FDcDeserializeContext& Ctx);
DECLARE_DELEGATE_RetVal_OneParam(FDcResult, FDcDeserializeDelegate, FDcDeserializeContext&);

We call these functions handlers. Here's a simple one that deserialize booleans:

// DataConfig/Source/DataConfigCore/Private/DataConfig/Deserialize/Handlers/Json/DcJsonPrimitiveDeserializers.cpp
FDcResult HandlerBoolDeserialize(FDcDeserializeContext& Ctx)
{
    EDcDataEntry Next;
    DC_TRY(Ctx.Reader->PeekRead(&Next));

    if (Next != EDcDataEntry::Bool)
    {
        return DC_FAIL(DcDDeserialize, DataEntryMismatch)
            << EDcDataEntry::Bool << Next;
    }

    bool Value;
    DC_TRY(Ctx.Reader->ReadBool(&Value));
    DC_TRY(Ctx.Writer->WriteBool(Value));

    return DcOk();
}

Note how we propagate errors to the caller by using DC_TRY or fail explicitly by returning DC_FAILwith diagnostic.

Predicates

In many occasions we want to provide custom deserialization logic for a very specific class. The selection process is done through FDcDeserializePredicate:

// DataConfig/Source/DataConfigCore/Public/DataConfig/Deserialize/DcDeserializeTypes.h
using FDcDeserializePredicateSignature = EDcDeserializePredicateResult(*)(FDcDeserializeContext& Ctx);
DECLARE_DELEGATE_RetVal_OneParam(EDcDeserializePredicateResult, FDcDeserializePredicate, FDcDeserializeContext&);

We call these Predicates. Here's an example of selecting the bulit-in FColor:

// DataConfig/Source/DataConfigExtra/Private/DataConfig/Extra/Deserialize/DcDeserializeColor.cpp
EDcDeserializePredicateResult PredicateIsColorStruct(FDcDeserializeContext& Ctx)
{
    UScriptStruct* Struct = DcPropertyUtils::TryGetStructClass(Ctx.TopProperty());
    return Struct && Struct == TBaseStructure<FColor>::Get()
        ? EDcDeserializePredicateResult::Process
        : EDcDeserializePredicateResult::Pass;
}

Similar to handlers it's checking Ctx.TopProperty() and return a EDcDeserializePredicateResult to decide to process or pass.

Deserializer

Finally there's the FDcDeserializer which is just a collection of predicates and handlers. It contains no mutable state as those are put in FDcDeserializeContext:

// DataConfig/Source/DataConfigCore/Public/DataConfig/Deserialize/DcDeserializer.h
struct DATACONFIGCORE_API FDcDeserializer : public FNoncopyable
{
    //...
    FDcResult Deserialize(FDcDeserializeContext& Ctx);

    void AddDirectHandler(FFieldClass* PropertyClass, FDcDeserializeDelegate&& Delegate);
    void AddDirectHandler(UClass* PropertyClass, FDcDeserializeDelegate&& Delegate);
    void AddPredicatedHandler(FDcDeserializePredicate&& Predicate, FDcDeserializeDelegate&& Delegate);
    //...
};

AddDirectHandler() registers handlers for a specific property type. Here's an example of registering the HandlerBoolDeserialize above:

// DataConfig/Source/DataConfigCore/Private/DataConfig/Deserialize/DcDeserializerSetup.cpp
Deserializer.AddDirectHandler(
    FBoolProperty::StaticClass(), 
    FDcDeserializeDelegate::CreateStatic(HandlerBoolDeserialize)
);

AddPredicatedHandler() registers a predicate and handler pair. Here's an example of registering the PredicateIsColorStruct predicate above:

// DataConfig/Source/DataConfigTests/Private/DcTestBlurb.cpp
Deserializer.AddPredicatedHandler(
    FDcDeserializePredicate::CreateStatic(DcExtra::PredicateIsColorStruct),
    FDcDeserializeDelegate::CreateStatic(DcExtra::HandlerColorDeserialize)
);

To start deserialization you need to prepare a FDcDeserializeContext and call FDcDeserializer::Deserialize(Ctx):

// DataConfig/Source/DataConfigTests/Private/DcTestBlurb.cpp
//  prepare context for this run
FDcPropertyDatum Datum(FDcTestExampleStruct::StaticStruct(), &Dest);
FDcJsonReader Reader(Str);
FDcPropertyWriter Writer(Datum);

FDcDeserializeContext Ctx;
Ctx.Reader = &Reader;
Ctx.Writer = &Writer;
Ctx.Deserializer = &Deserializer;
Ctx.Properties.Push(Datum.Property);
DC_TRY(Ctx.Prepare());

//  kick off deserialization
DC_TRY(Deserializer.Deserialize(Ctx));

Tips for writing handlers

There're some recurring patterns when writing deserialization handlers in DataConfig.

Recursive Deserialize

When deserializing a container like USTRUCT root or TArray you'll need to recursively deserialize children properties. Here's how it's done:

// DataConfig/Source/DataConfigCore/Private/DataConfig/Deserialize/Handlers/Json/DcJsonStructDeserializers.cpp
FDcScopedProperty ScopedValueProperty(Ctx);
DC_TRY(ScopedValueProperty.PushProperty());
DC_TRY(Ctx.Deserializer->Deserialize(Ctx));

FDcScopedProperty is used to push writer's next property into FDcDeserializeContext::Properties to satisfiy the invariant that FDcDeserializeContext::TopProperty() always points to the current writing property.

Provide TopObject()

Sometimes deserialization will create new UObject along the way. In this case you'll need to fill in FDcDeserializeContext::Objects so the top one is used for NewObject() calls. For transient objecst you can use GetTransientPackage():

// DataConfig/Source/DataConfigTests/Private/DcTestDeserialize.cpp
Ctx.Objects.Push(GetTransientPackage());

Peek By Value

Sometimes you want to peek the content of the next entry. For example in DcExtra::HandlerBPDcAnyStructDeserialize() we're dealing with a JSON like this:

// DataConfig/Source/DataConfigEditorExtra/Private/DataConfig/EditorExtra/Deserialize/DcDeserializeBPClass.cpp
{
    "AnyStructField1" : {
        "$type" : "/DataConfig/DcFixture/DcTestBlueprintStructWithColor",
        "NameField" : "Foo",
        //...
    }
}

We want to consume the $type key and its value, and then delegate the logic back to the deserializer. The solution here is first to consume the pair. Then we put back a { then replace the reader:

// DataConfig/Source/DataConfigEditorExtra/Private/DataConfig/EditorExtra/Deserialize/DcDeserializeBPClass.cpp
FDcPutbackReader PutbackReader(Ctx.Reader);
PutbackReader.Putback(EDcDataEntry::MapRoot);
TDcStoreThenReset<FDcReader*> RestoreReader(Ctx.Reader, &PutbackReader);

FDcScopedProperty ScopedValueProperty(Ctx);
DC_TRY(ScopedValueProperty.PushProperty());
DC_TRY(Ctx.Deserializer->Deserialize(Ctx));

Beware that Putback only support a limited subset of data types.

Deserialize

This section contains doc for existing deserialize handlers sets.

Deserialize JSON

DataConfig bundles a set of JSON deserialize handlers. Setup by calling DcSetupJsonDeserializeHandlers():

// DataConfig/Source/DataConfigTests/Private/DcTestBlurb.cpp
#include "DataConfig/Deserialize/DcDeserializerSetup.h"

//...
//  create and setup a deserializer
FDcDeserializer Deserializer;
DcSetupJsonDeserializeHandlers(Deserializer);

Basics

JSON types get mapped into DataConfig data model in a very unsurprising way.

JSON TypeDcDataEntry
BooleanBool
NullNil
StringString, Name, Text, Enum
Number(All numerics)
ArrayArray, Set
ObjectClass, Struct, Map

Here's an example:

// DataConfig/Source/DataConfigTests/Private/DcTestDeserialize.cpp
FString Str = TEXT(R"(
    {
        "BoolField" : true,
        "NameField" : "AName",
        "StringField" : "AStr",
        "TextField" : "AText",
        "EnumField" : "Tard",

        "FloatField" : 17.5,
        "DoubleField" : 19.375,

        "Int8Field" : -43,
        "Int16Field" : -2243,
        "Int32Field" : -23415,
        "Int64Field" : -1524523,

        "UInt8Field" : 213,
        "UInt16Field" : 2243,
        "UInt32Field" : 23415,
        "UInt64Field" : 1524523,
    }
)");

//  deserialized equivelent

FDcTestStruct1 Expect;
Expect.BoolField = true;
Expect.NameField = TEXT("AName");
Expect.StringField = TEXT("AStr");
Expect.TextField = FText::FromString(TEXT("AText"));
Expect.EnumField = EDcTestEnum1::Tard;

Expect.FloatField = 17.5f;
Expect.DoubleField = 19.375;

Expect.Int8Field = -43;
Expect.Int16Field = -2243;
Expect.Int32Field = -23415;
Expect.Int64Field = -1524523;

Expect.UInt8Field = 213;
Expect.UInt16Field = 2243,
Expect.UInt32Field = 23415;
Expect.UInt64Field = 1524523;

Enum Flags

UENUM that get marked with Bitflags meta are deserialized from a list of strings:

// DataConfig/Source/DataConfigTests/Public/DcTestDeserialize.h
UENUM(meta = (Bitflags))
enum class EDcTestEnumFlag :uint32
{
    None    = 0,
    One     = (1 << 0),
    Two     = (1 << 1),
    //...
};
ENUM_CLASS_FLAGS(EDcTestEnumFlag);

// DataConfig/Source/DataConfigTests/Private/DcTestDeserialize.cpp
FString Str = TEXT(R"(
    {
        "EnumFlagField1" : [],
        "EnumFlagField2" : ["One", "Three", "Five"],
    }
)");

//  deserialized equivelent
FDcTestStructEnumFlag1 Expect;
Expect.EnumFlagField1 = EDcTestEnumFlag::None;
Expect.EnumFlagField2 = EDcTestEnumFlag::One | EDcTestEnumFlag::Three | EDcTestEnumFlag::Five;

Sub Objects

By default We treat UOBJECT marked with DefaultToInstanced, EditInlineNew and UPROPERTY marked with Instanced as sub object. In this case we'll actually instantiate new object during deserialization, using Ctx.TopObject() as parent:

// DataConfig/Source/DataConfigTests/Public/DcTestProperty.h
UCLASS(BlueprintType, EditInlineNew, DefaultToInstanced)
class UDcBaseShape : public UObject
{
    //...
    UPROPERTY() FName ShapeName;
};

UCLASS()
class UDcShapeBox : public UDcBaseShape
{
    //...
    UPROPERTY() float Height;
    UPROPERTY() float Width;
};

UCLASS()
class UDcShapeSquare : public UDcBaseShape
{
    //...
    UPROPERTY() float Radius;
};

// DataConfig/Source/DataConfigTests/Public/DcTestDeserialize.h
USTRUCT()
struct FDcTestStructShapeContainer1
{
    GENERATED_BODY()

    UPROPERTY() UDcBaseShape* ShapeField1;
    UPROPERTY() UDcBaseShape* ShapeField2;
    UPROPERTY() UDcBaseShape* ShapeField3;
}USTRUCT()
struct FDcEditorExtraTestObjectRefs1
{
    GENERATED_BODY()

    UPROPERTY() UObject* ObjField1;
    UPROPERTY() UObject* ObjField2;
    UPROPERTY() UObject* ObjField3;
    UPROPERTY() UObject* ObjField4;
};

// DataConfig/Source/DataConfigTests/Private/DcTestDeserialize.cpp
FString Str = TEXT(R"(
    {
        "ShapeField1" :  {
            "$type" : "DcShapeBox",
            "ShapeName" : "Box1",
            "Height" : 17.5,
            "Width" : 1.9375
        },
        "ShapeField2" : {
            "$type" : "DcShapeSquare",
            "ShapeName" : "Square1",
            "Radius" : 1.75,
        },
        "ShapeField3" : null
    }
)");

//  deserialized equivelent
UDcShapeBox* Shape1 = NewObject<UDcShapeBox>();
Shape1->ShapeName = TEXT("Box1");
Shape1->Height = 17.5;
Shape1->Width = 1.9375;
Expect.ShapeField1 = Shape1;

UDcShapeSquare* Shape2 = NewObject<UDcShapeSquare>();
Shape2->ShapeName = TEXT("Square1");
Shape2->Radius = 1.75;
Expect.ShapeField2 = Shape2;

Expect.ShapeField3 = nullptr;

Note that the sub object criteria can be easily overridden with a new deserialize predicate or alternative FDcPropertyConfig when constructing the reader.

Object and Class Reference

We support multiple ways of referencing a UObject in memory or serialized on disk:

// DataConfig/Source/DataConfigEditorExtra/Private/DataConfig/EditorExtra/Tests/DcTestDeserializeEditor.h
USTRUCT()
struct FDcEditorExtraTestObjectRefs1
{
    GENERATED_BODY()

    UPROPERTY() UObject* ObjField1;
    UPROPERTY() UObject* ObjField2;
    UPROPERTY() UObject* ObjField3;
    UPROPERTY() UObject* ObjField4;
};

// DataConfig/Source/DataConfigEditorExtra/Private/DataConfig/EditorExtra/Tests/DcTestDeserializeEditor.cpp
FString Str = TEXT(R"(
    {
        "ObjField1" : "DcEditorExtraNativeDataAsset'/DataConfig/DcFixture/DcTestNativeDataAssetAlpha.DcTestNativeDataAssetAlpha'",
        "ObjField2" : "/DataConfig/DcFixture/DcTestNativeDataAssetAlpha",
        "ObjField3" : 
        {
            "$type" : "DcEditorExtraNativeDataAsset",
            "$path" : "/DataConfig/DcFixture/DcTestNativeDataAssetAlpha"
        },
        "ObjField4" : null,
    }
)");

//  deserialized equivelent
UDcEditorExtraNativeDataAsset* DataAsset = Cast<UDcEditorExtraNativeDataAsset>(StaticLoadObject(
    UDcEditorExtraNativeDataAsset::StaticClass(),
    nullptr,
    TEXT("/DataConfig/DcFixture/DcTestNativeDataAssetAlpha"),
    nullptr
));

Expect.ObjField1 = DataAsset;
Expect.ObjField2 = DataAsset;
Expect.ObjField3 = DataAsset;
Expect.ObjField4 = nullptr;

In the example above, ObjField1 is using the reference string that can be retrieved in editor context menu:

Deserialize-CopyReference

For ObjField2/ObjField3 it's using a relative path to the uasset but without file name suffix.

We also support class reference fields of TSubclassOf<>s:

// DataConfig/Source/DataConfigTests/Private/DcTestDeserialize.h
USTRUCT()
struct FDcTestStructSubClass1
{
    GENERATED_BODY()

    UPROPERTY() TSubclassOf<UStruct> StructSubClassField1;
    UPROPERTY() TSubclassOf<UStruct> StructSubClassField2;
    UPROPERTY() TSubclassOf<UStruct> StructSubClassField3;
};

// DataConfig/Source/DataConfigTests/Private/DcTestDeserialize.cpp
FString Str = TEXT(R"(
    {
        "StructSubClassField1" : null,
        "StructSubClassField2" : "ScriptStruct",
        "StructSubClassField3" : "DynamicClass",
    }
)");

//  deserialized equivelent
FDcTestStructSubClass1 Expect;
Expect.StructSubClassField1 = nullptr;
Expect.StructSubClassField2 = UScriptStruct::StaticClass();
Expect.StructSubClassField3 = UDynamicClass::StaticClass();

Note that these do not support Blueprint classes. The direct reason is that Blueprint depends on Engine module and we'd like not to take dependency on in DataConfigCore.

We do have an example that supports Blueprint classes, see DataConfigEditorExtra - DcDeserializeBPClass.h/cpp

Caveats

Here're some closing notes:

  • For meta fields like $type we require it to be the first member, meaning object fields are order dependent. This means that the JSON we're supporting is a super set of standard JSON spec (again).

  • There're many data types that can not be deserialized from JSON, for example Delegate/WeakObjectReference. Remember that you always have the option to override or selectively enable the handlers to support additional property that makes sense in your context. See DcSetupJsonDeserializeHandlers() body how handlers are registered. You can skip this method and select the ones you want and provide additional handlers.

  • The JSON handlers are designed to NOT read anything during the deserialization. This is crucial since USTRUCT can contain uninitialized fields. For example:

    // DataConfig/Source/DataConfigTests/Private/DcTestBlurb.cpp
    FString Str = TEXT(R"(
        {
            // pass
        } 
    )");
    FDcJsonReader Reader(Str);
    
    FDcTestExampleSimple Dest;
    FDcPropertyDatum DestDatum(FDcTestExampleSimple::StaticStruct(), &Dest);
    
    DC_TRY(DcAutomationUtils::DeserializeJsonInto(&Reader, DestDatum));
    
    check(Dest.StrField.IsEmpty());
    //  but Dest.IntField contains uninitialized value
    DcAutomationUtils::DumpToLog(DestDatum);
    
    // dump results
    <StructRoot> 'DcTestExampleSimple'
    |---<Name> 'StrField'
    |---<String> ''
    |---<Name> 'IntField'
    |---<Int32> '1689777552' // <- arbitrary value
    <StructEnd> 'DcTestExampleSimple'
    

    This would cause trouble when you try read a pointer field during deserialization. Remember that primitive fields might be uninitialized during deserialization when implementing your own handlers.

  • One interesting trait of the pull/push styled API is that FDcJsonReader does not preemptively parse number into double and convert it to int/float later on. When reading a number token it would do the number parsing at call site. If ReadIntX() is called then the number is parsed as integer. If ReadFloat()/ReadDouble() is called the token will be parsed as floating point.

Pipe Deserialize

There's a set of deserialize handlers in DcPropertyPipeHandlers namespace. It's used to roundtripping property system objects.

Simply speaking it's equivalent to doing a FDcPipeVisitor pipe visit.

// DataConfig/Source/DataConfigTests/Private/DcTestBlurb.cpp
//  these two blocks are equivalent
{
    FDcPropertyReader Reader(FromDatum);
    FDcPropertyWriter Writer(ToDatum);
    FDcPipeVisitor RoundtripVisit(&Reader, &Writer);

    DC_TRY(RoundtripVisit.PipeVisit());
}

{
    FDcDeserializer Deserializer;
    DcSetupPropertyPipeDeserializeHandlers(Deserializer);

    FDcPropertyReader Reader(FromDatum);
    FDcPropertyWriter Writer(ToDatum);

    FDcDeserializeContext Ctx;
    Ctx.Reader = &Reader;
    Ctx.Writer = &Writer;
    Ctx.Deserializer = &Deserializer;
    Ctx.Properties.Push(FromDatum.Property);
    DC_TRY(Ctx.Prepare());

    DC_TRY(Deserializer.Deserialize(Ctx));
}

These are provided as a set of basis to for building custom property wrangling utils. See DataConfigExtra - DcDeserializeRenameStructFieldNames.h/cpp for example.

Extra

Alongside DataConfigCore we have two other modules DataConfigExtra and DataConfigEditorExtra. It have self contained samples built on top of DataConfig framework.

Note that these are not intended to be integrated directly. You can take these as references when implementing custom features.

DataConfigExtra

Module DataConfigExtra contains examples that doesn't dependent on Engine/UnrealEd.

Deserialize JSON into Struct

There's a built-in method in JsonUtilities module that simply deserialize a JSON string into a struct. In this example we implemented a similar method with almost identical API:

// DataConfig/Source/DataConfigExtra/Private/DataConfig/Extra/Types/DcJsonConverter.cpp
FString Str = TEXT(R"(
    {
        "StrField" : "Foo",
        "IntField" : 253,
        "BoolField" : true
    }
)");

FDcTestJsonConverter1 Lhs;
bool LhsOk = DcExtra::JsonObjectStringToUStruct(Str, &Lhs);

FDcTestJsonConverter1 Rhs;
bool RhsOk = FJsonObjectConverter::JsonObjectStringToUStruct(Str, &Rhs, 0, 0);

Comparing to the stock method DcExtra::JsonObjectStringToUStruct allows relaxed JSON with comments and trailing comma. It would also provide better diagnostics on parse error.

Deserialize FColor

This has been shown multiple times in previous chapters. It's also a benchmark use case for our custom deserialization logic:

// DataConfig/Source/DataConfigExtra/Private/DataConfig/Extra/Deserialize/DcDeserializeColor.cpp
FString Str = TEXT(R"(
    {
        "ColorField1" : "#0000FFFF",
        "ColorField2" : "#FF0000FF",
    }
)");

// deserialized equivelent
FDcExtraTestStructWithColor1 Expect;

Expect.ColorField1 = FColor::Blue;
Expect.ColorField2 = FColor::Red;

Deserialize Base64 string as Blob

In this example we deserialize TArray<uint8> from Base64 strings in JSON:

// DataConfig/Source/DataConfigExtra/Private/DataConfig/Extra/Deserialize/DcDeserializeBase64.cpp
FString Str = TEXT(R"(
    {
        "BlobField1" : "dGhlc2UgYXJlIG15IHR3aXN0ZWQgd29yZHM=",
        "BlobField2" : "",
    }
)");

// deserialized equivelent
FDcExtraTestStructWithBase64 Expect;

const char* Literal = "these are my twisted words";
Expect.BlobField1 = TArray<uint8>((uint8*)Literal, FCStringAnsi::Strlen(Literal));
Expect.BlobField2 = TArray<uint8>();

In the predicate we are checking for Arrays with custom meta data DcExtraBlob:

// DataConfig\Source\DataConfigExtra\Public\DataConfig\Extra\Deserialize\DcDeserializeBase64.h
USTRUCT()
struct FDcExtraTestStructWithBase64
{
    GENERATED_BODY()

    UPROPERTY(meta = (DcExtraBase64)) TArray<uint8> BlobField1;
    UPROPERTY(meta = (DcExtraBase64)) TArray<uint8> BlobField2;
};

UE support arbitrary meta data in the meta = () segment. But beware that the meta data is only available when WITH_EDITORDATA flag is defined.

Deserialize FDcAnyStruct

we've implemented FDcAnyStruct that can be used to store a heap allocated USTRUCT of any type while keep proper value sematic on itself:

// DataConfig/Source/DataConfigExtra/Private/DataConfig/Extra/Deserialize/DcDeserializeAnyStruct.cpp
//  instantiate from heap allocated structs
FDcAnyStruct Any1 = new FDcExtraTestSimpleStruct1();
Any1.GetChecked<FDcExtraTestSimpleStruct1>()->NameField = TEXT("Foo");

//  supports moving
FDcAnyStruct Any2 = MoveTemp(Any1);
check(!Any1.IsValid());
check(Any2.GetChecked<FDcExtraTestSimpleStruct1>()->NameField == TEXT("Foo"));
Any2.Reset();

//  supports shared referencing
Any2 = new FDcExtraTestSimpleStruct2();
Any2.GetChecked<FDcExtraTestSimpleStruct2>()->StrField = TEXT("Bar");

Any1 = Any2;

check(Any1.DataPtr == Any2.DataPtr);
check(Any1.StructClass == Any2.StructClass);

In this example we implemented predicate and handler to support deserializing FDcAnyStruct from a JSON object with a $type field, or a null.

// DataConfig/Source/DataConfigExtra/Private/DataConfig/Extra/Deserialize/DcDeserializeAnyStruct.cpp
FString Str = TEXT(R"(
    {
        "AnyStructField1" : {
            "$type" : "DcExtraTestSimpleStruct1",
            "NameField" : "Foo"
        },
        "AnyStructField2" : {
            "$type" : "DcExtraTestStructWithColor1",
            "ColorField1" : "#0000FFFF",
            "ColorField2" : "#FF0000FF"
        },
        "AnyStructField3" : null
    }
)");

//...

UTEST_TRUE("...", Dest.AnyStructField1.GetChecked<FDcExtraTestSimpleStruct1>()->NameField == TEXT("Foo"));
UTEST_TRUE("...", Dest.AnyStructField2.GetChecked<FDcExtraTestStructWithColor1>()->ColorField1 == FColor::Blue);
UTEST_TRUE("...", Dest.AnyStructField2.GetChecked<FDcExtraTestStructWithColor1>()->ColorField2 == FColor::Red);
UTEST_TRUE("...", !Dest.AnyStructField3.IsValid());

Note how custom FColor deserializing works inside a FDcAnyStruct.

Copying struct while renaming field names

This is an example of using FDcDeserializer with non FDcJsonReader. It uses FDcPropertyReader with the DcPropertyPipeHandlers to do renaming:

// DataConfig/Source/DataConfigExtra/Private/DataConfig/Extra/Deserialize/DcDeserializeRenameStructFieldNames.cpp
//  struct equivelent to this:
FString Str = TEXT(R"(
    {
        "FromName1" : "Foo",
        "FromStructSet1" : 
        [
            {
                "FromStr1" : "One",
                "FromInt1" : 1,
            },
            {
                "FromStr1" : "Two",
                "FromInt1" : 2,
            }
        ]
    }
)");

// ... deserialize with a functor renaming `FromXXX` to `ToXXX`:
UTEST_OK("...", DcExtra::DeserializeStructRenaming(FromDatum, ToDatum, FDcExtraRenamer::CreateLambda([](const FName& FromName){
    FString FromStr = FromName.ToString();
    if (FromStr.StartsWith(TEXT("From")))
        return FName(TEXT("To") + FromStr.Mid(4));
    else
        return FromName;
})));

// ... results into a struct equivelent to this: 
FString Str = TEXT(R"(
    {
        "ToName1" : "Foo",
        "ToStructSet1" : 
        [
            {
                "ToStr1" : "One",
                "ToInt1" : 1,
            },
            {
                "ToStr1" : "Two",
                "ToInt1" : 2,
            }
        ]
    }
)");

Access property by path

UE built-in module PropertyPath allow accessing nested object properties by a path like Foo.Bar.Baz:

// DataConfig/Source/DataConfigExtra/Private/DataConfig/Extra/Types/DcPropertyPathAccess.cpp
FString Str;
UTEST_TRUE("...", PropertyPathHelpers::GetPropertyValue(Outer, TEXT("StructRoot.Middle.InnerMost.StrField"), Str));
UTEST_TRUE("...", Str == TEXT("Foo"));

UTEST_TRUE("...", PropertyPathHelpers::SetPropertyValue(Outer, TEXT("StructRoot.Middle.InnerMost.StrField"), FString(TEXT("Bar"))));
UTEST_TRUE("...", Outer->StructRoot.Middle.InnerMost.StrField == TEXT("Bar"));

We implemented a pair of methods GetDatumPropertyByPath/SetDatumPropertyByPath with FDcPropertyReader:

// DataConfig/Source/DataConfigExtra/Private/DataConfig/Extra/Types/DcPropertyPathAccess.cpp
UTEST_TRUE("...", CheckStrPtr(GetDatumPropertyByPath<FString>(FDcPropertyDatum(Outer), "StructRoot.Middle.InnerMost.StrField"), TEXT("Foo")));
UTEST_TRUE("...", CheckStrPtr(GetDatumPropertyByPath<FString>(FDcPropertyDatum(Outer), "StructRoot.Arr.0.StrField"), TEXT("Bar0")));
UTEST_TRUE("...", CheckStrPtr(GetDatumPropertyByPath<FString>(FDcPropertyDatum(Outer), "StructRoot.Arr.1.StrField"), TEXT("Bar1")));
UTEST_TRUE("...", CheckStrPtr(GetDatumPropertyByPath<FString>(FDcPropertyDatum(Outer), "StructRoot.NameMap.FooKey.StrField"), TEXT("FooValue")));

UTEST_TRUE("...", SetDatumPropertyByPath<FString>(FDcPropertyDatum(Outer), "StructRoot.Middle.InnerMost.StrField", TEXT("AltFoo")));
UTEST_TRUE("...", SetDatumPropertyByPath<FString>(FDcPropertyDatum(Outer), "StructRoot.Arr.0.StrField", TEXT("AltBar0")));
UTEST_TRUE("...", SetDatumPropertyByPath<FString>(FDcPropertyDatum(Outer), "StructRoot.Arr.1.StrField", TEXT("AltBar1")));
UTEST_TRUE("...", SetDatumPropertyByPath<FString>(FDcPropertyDatum(Outer), "StructRoot.NameMap.FooKey.StrField", TEXT("AltFooValue")));

Comparing to PropertyPathHelpers these new ones support Array and Map, and support USTRUCT roots. We're also missing some features like expanding weak/lazy object references but it should be easy to implement.

DataConfigEditorExtra

Module DataConfigEditorExtra contains examples that need to be run in a editor context.

DcEditorExtraModule

This is a good reference of integrating DataConfigCore in a editor module. Here's a checklist:

  • Register additional diagnostics early.
  • Call DcStartUp()/DcShutDonw() pair.
  • Register custom diagnostic consumer.

FDcMessageLogDiagnosticConsumer is an example of redirecting diagnostics into UE Message Log window with its own category.

Deserialize Blueprint Class Instances

The Property System is so powerful that you can create new Blueprint Class, which is equivalent to C++ UCLASS to some extents, within the Blueprint Editor. In this example we implemented deserializing these.

Blueprint Class can be referenced by the blueprint asset path:

// DataConfig/Source/DataConfigEditorExtra/Private/DataConfig/EditorExtra/Deserialize/DcDeserializeBPClass.cpp
FString Str = TEXT(R"(
    {
        //...
        "ClassField3" : "/DataConfig/DcFixture/DcTestBlueprintClassBeta",
    }
)");

UTEST_TRUE("...", Dest.ClassField3->GetFName() == TEXT("DcTestBlueprintClassBeta_C"));

And Blueprint structs can also be deserialized from JSON. We need to rewrite the handler HandlerBPDcAnyStructDeserialize for looking up Blueprint Struct by path:

// DataConfig/Source/DataConfigEditorExtra/Private/DataConfig/EditorExtra/Deserialize/DcDeserializeBPClass.cpp
FString Str = TEXT(R"(
    {
        "AnyStructField1" : {
            "$type" : "/DataConfig/DcFixture/DcTestBlueprintStructWithColor",
            "NameField" : "Foo",
            "StrField" : "Bar",
            "IntField" : 123,
            "ColorField" : "#FF0000FF"
        }
    }
)");

There's a quirk that Blueprint Struct actually mangle its field names. The struct above dumps to something like this:

-----------------------------------------
# Datum: 'UserDefinedStruct', 'DcTestBlueprintStructWithColor'
<StructRoot> 'DcTestBlueprintStructWithColor'
|---<Name> 'NameField_5_97BFF114405C1934C2F33E8668BF1652'
|---<Name> 'Foo'
|---<Name> 'StrField_9_FAA71EFE4896F4E6B1478B9C13B2CE52'
|---<String> 'Bar'
|---<Name> 'IntField_11_3BC7CB0F42439CE2196F7AA82A1AC374'
|---<Int32> '123'
|---<Name> 'ColorField_14_F676BCF245B2977B678B65A8216E94EB'
|---<StructRoot> 'Color'
|   |---<Name> 'B'
|   |---<UInt8> '0'
|   |---<Name> 'G'
|   |---<UInt8> '0'
|   |---<Name> 'R'
|   |---<UInt8> '255'
|   |---<Name> 'A'
|   |---<UInt8> '255'
|---<StructEnd> 'Color'
<StructEnd> 'DcTestBlueprintStructWithColor'
-----------------------------------------

The good news is that DataConfig already got this covered.

Deserialize GameplayTags

GameplayTags is a built-in runtime module that implements hierarchical tags. In this example we implemented deserializing into FGameplayTag from a string.

// DataConfig/Source/DataConfigEditorExtra/Private/DataConfig/EditorExtra/Deserialize/DcDeserializeGameplayTags.cpp
FString Str = TEXT(R"(
    {
        "TagField1" : null,
        "TagField2" : "DataConfig.Foo.Bar"
    }
)");

UTEST_FALSE("...", Dest.TagField1.IsValid());
UTEST_TRUE("...", Dest.TagField2.IsValid());
UTEST_TRUE("...", Dest.TagField2 == UGameplayTagsManager::Get().RequestGameplayTag(TEXT("DataConfig.Foo.Bar")));

We also implemented deserializing FGameplayTagContainer from a list of strings:

// DataConfig/Source/DataConfigEditorExtra/Private/DataConfig/EditorExtra/Deserialize/DcDeserializeGameplayTags.cpp
FString Str = TEXT(R"(
    {
        "TagContainerField1" : [],
        "TagContainerField2" : [
            "DataConfig.Foo.Bar",
            "DataConfig.Foo.Bar.Baz",
            "DataConfig.Tar.Taz",
        ]
    }
)");

UTEST_TRUE("...", Dest.TagContainerField1.Num() == 0);
UTEST_TRUE("...", Dest.TagContainerField2.Num() == 3);
UTEST_TRUE("...", Dest.TagContainerField2.HasTagExact(
    UGameplayTagsManager::Get().RequestGameplayTag(TEXT("DataConfig.Foo.Bar"))
));
UTEST_TRUE("...", Dest.TagContainerField2.HasTagExact(
    UGameplayTagsManager::Get().RequestGameplayTag(TEXT("DataConfig.Foo.Bar.Baz"))
));
UTEST_TRUE("...", Dest.TagContainerField2.HasTagExact(
    UGameplayTagsManager::Get().RequestGameplayTag(TEXT("DataConfig.Tar.Taz"))
));

Note that gameplay tag parsing has error reporting built-in. In this case we can pipe it into our diagnostic:

// DataConfig/Source/DataConfigEditorExtra/Private/DataConfig/EditorExtra/Deserialize/DcDeserializeGameplayTags.cpp
static FDcResult _StringToGameplayTag(FDcDeserializeContext& Ctx, const FString& Str, FGameplayTag* OutTagPtr)
{
    FString FixedString;
    FText Err;
    if (!FGameplayTag::IsValidGameplayTagString(Str, &Err, &FixedString))
    {
        return DC_FAIL(DcDEditorExtra, InvalidGameplayTagStringFixErr)
            << Str << FixedString << Err;
    }
    //...
}

In case of a invalid tag it would report the reason and fixed string:

# DataConfig Error: Invalid Gameplay Tag String, Actual: 'DataConfig.Invalid.Tag.', Fixed: 'DataConfig.Invalid.Tag', Error: 'Tag ends with .'
- [JsonReader] --> <in-memory>5:4
   3 |        { 
   4 |            "TagField1" : null, 
   5 |            "TagField2" : "DataConfig.Invalid.Tag." 
     |                          ^^^^^^^^^^^^^^^^^^^^^^^^^
   6 |        } 
   7 |    
- [PropertyWriter] Writing property: (FDcEditorExtraTestStructWithGameplayTag1)$root.(FGameplayTag)TagField2
 [C:\DevUE\UnrealEngine\Engine\Source\Developer\MessageLog\Private\Model\MessageLogListingModel.cpp(73)]

Deserialize Gameplay Abilities

We'll conclude examples with a concrete user story: populating GameplayAbility and GameplayEffect blueprint from JSON file.

Gameplay Ability System is a built-in plugin for building data driven abilities. Users are expected to derived and modify GameplayAbility and GameplayEffect blueprint for custom logic.

Given a JSON like this:

// DataConfig/Tests/Fixture_AbilityAlpha.json
{
    /// Tags
    "AbilityTags" : [
        "DataConfig.Foo.Bar",
        "DataConfig.Foo.Bar.Baz",
    ],
    "CancelAbilitiesWithTag" : [
        "DataConfig.Foo.Bar.Baz",
        "DataConfig.Tar.Taz",
    ],
    /// Costs
    "CostGameplayEffectClass" : "/DataConfig/DcFixture/DcTestGameplayEffectAlpha",
    /// Advanced
    "ReplicationPolicy" : "ReplicateYes",
    "InstancingPolicy" : "NonInstanced",
}

Right click on a GameplayAbility blueprint asset and select Load From JSON, then select this file and confirm. It would correctly populate the fields with the values in JSON, as seen in the pic below:

DataConfigEditorExtra-LoadJsonIntoAbility

Most of the logic is in DataConfig/EditorExtra/Deserialize/DcDeserializeGameplayAbility.cpp:

  • The context menu is added from GameplayAbilityEffectExtender. There's another handy item named Dump To Log which dumps any blueprint CDO into the log.
  • DataConfig deserializer is setup in LazyInitializeDeserializer(). We added custom logic for deserializing FGameplayAttribute from a string like DcTestAttributeSet.Mana.
  • We also reused many methods from previous examples to support FGameplayTag deserialization and Blueprint class look up by path.

Changes

All notable changes to this project will be documented in this file.

1.1.0 - 2021-4-24

  • Integrate nst/JSONTestSuite. Now DcJSONParser pass most of them. Skipped tests are also documented.
  • FDcAnsiJsonReader now detect and convert non ASCII UTF8 characters. Previously these characters are dropped.
  • Headless test runner pass along parameters to tests, for example DataConfigHeadless-Win64-Debug.exe Parsing -- n_array_just_minus

License

DataConfig is released under MIT License.

MIT License

Copyright (c) 2021 Chen Tao 

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.

Attribution

If you find DataConfig useful in your project, consider credit us in your project with the full license above or the shorter snippets below.

DataConfig <https//slowburn.dev/dataconfig>
MIT License, Copyright (c) 2021 Chen Tao

There's also DataConfig JSON Asset on UE Marketplace. It's a premium plugin for importing JSON to UE data assets.

You can reach us by email to hislowburn@gmail.com or on twitter @slowburndev.

We'll setup a section showcasing projects using DataConfig in the future.