DataConfig Book
Serialization framework for Unreal Engine that just works!
DataConfig is a serialization framework built on top of Unreal Engine's Property System. It aims to be friendly and robust while offering good performance. Notably features:
- Out of the box JSON/MsgPack read write.
- Full support for
UPROPERTY()/UCLASS()/USTRUCT()/UENUM()
. - Pull/Push style API for verbatim data access and lossless type information.
- Designed as a collection of tools that can be easily extended to support other formats.
Getting Started
- Get the code on github.
- See Examples for examples.
- See Integration for quick integration guide.
- See Design for more context about the project.
- See Extra for more advanced usages.
- See Changes for versioning history.
There's also DataConfig JSON Asset on UE Marketplace. It's a premium plugin for importing JSON to UE data assets.
License
DataConfig is released under free and permissive MIT license. We'd really appreciate to credit us if you find it useful. See License for details.
Examples
Here're some short and quick examples showcasing DataConfig API usage and features. All code shown here can be found in the repo.
Blueprint Nodes
If you integrated the full DataConfig plugin we have Blueprint Nodes for you to quickly try it out.
Create a Blueprint Actor and setup the BeginPlay
event like this:
Then place the actor in the level and start play. The JSON string would be print to screen like this.
JSON Deserialization
Given the structFDcTestExampleStruct
:
// DataConfigTests/Private/DcTestBlurb.h
UENUM()
enum class EDcTestExampleEnum
{
Foo, Bar, Baz
};
USTRUCT()
struct FDcTestExampleStruct
{
GENERATED_BODY()
UPROPERTY() FString StrField;
UPROPERTY() EDcTestExampleEnum EnumField;
UPROPERTY() TArray<FColor> Colors;
};
We can deserialize an instance from JSON with the snippet below:
// DataConfigTests/Private/DcTestBlurb.cpp
FString Str = TEXT(R"(
{
"StrField" : "Lorem ipsum dolor sit amet",
"EnumField" : "Bar",
"Colors" : [
"#FF0000FF", "#00FF00FF", "#0000FFFF"
]
}
)");
FDcTestExampleStruct Dest;
// create and setup a deserializer
FDcDeserializer Deserializer;
DcSetupJsonDeserializeHandlers(Deserializer);
Deserializer.AddStructHandler(
TBaseStructure<FColor>::Get(),
FDcDeserializeDelegate::CreateStatic(DcExtra::HandlerColorDeserialize)
);
// prepare context for this run
FDcPropertyDatum Datum(&Dest);
FDcJsonReader Reader(Str);
FDcPropertyWriter Writer(Datum);
FDcDeserializeContext Ctx;
Ctx.Reader = &Reader;
Ctx.Writer = &Writer;
Ctx.Deserializer = &Deserializer;
DC_TRY(Ctx.Prepare());
// kick off deserialization
DC_TRY(Deserializer.Deserialize(Ctx));
// validate results
check(Dest.StrField == TEXT("Lorem ipsum dolor sit amet"));
check(Dest.EnumField == EDcTestExampleEnum::Bar);
check(Dest.Colors[0] == FColor::Red);
check(Dest.Colors[1] == FColor::Green);
check(Dest.Colors[2] == FColor::Blue);
Note that EDcTestExampleEnum
is deserialized by its name and FColor
is deserialized from a html color string like #RRGGBBAA
.
Say if we accidentally mistyped the EnumField
value:
{
"StrField" : "Lorem ipsum dolor sit amet",
"EnumField" : "Far",
}
It would fail gracefully with diagnostics:
# DataConfig Error: Enum name not found in enum type: EDcTestExampleEnum, Actual: 'Far'
- [JsonReader] --> <in-memory>4:25
2 | {
3 | "StrField" : "Lorem ipsum dolor sit amet",
4 | "EnumField" : "Far",
| ^
5 | "Colors" : [
6 | "#FF0000FF", "#00FF00FF", "#0000FFFF"
- [PropertyWriter] Writing property: (FDcTestExampleStruct)$root.(EEDcTestExampleEnum)EnumField
As bonus we serialize the struct into MsgPack:
FDcSerializer Serializer;
DcSetupMsgPackSerializeHandlers(Serializer);
FDcPropertyDatum Datum(&Dest);
FDcPropertyReader Reader(Datum);
FDcMsgPackWriter Writer;
// prepare serialize context
FDcSerializeContext Ctx;
Ctx.Reader = &Reader;
Ctx.Writer = &Writer;
Ctx.Serializer = &Serializer;
DC_TRY(Ctx.Prepare());
// kick off serialization
DC_TRY(Serializer.Serialize(Ctx));
auto& Buffer = Writer.GetMainBuffer();
// starts withMsgPack FIXMAP(3) header
check(Buffer[0] == 0x83);
Custom Serialization And Deserialization
DataConfig support custom serialization and deserialization logic by implementing FDcSerializeDelegate/FDcDeserializeDelegate
.
In this example, we'd like to convert FColor
into #RRGGBBAA
and vice versa:
// DataConfigExtra/Public/DataConfig/Extra/Deserialize/DcSerDeColor.h
USTRUCT()
struct FDcExtraTestStructWithColor1
{
GENERATED_BODY()
UPROPERTY() FColor ColorField1;
UPROPERTY() FColor ColorField2;
};
// DataConfigExtra/Private/DataConfig/Extra/Deserialize/DcSerDeColor.cpp
FString Str = TEXT(R"(
{
"ColorField1" : "#0000FFFF",
"ColorField2" : "#FF0000FF",
}
)");
First you'll need to implement a FDcDeserializePredicate
delegate to pick out FColor
properties:
// DataConfigExtra/Private/DataConfig/Extra/SerDe/DcSerDeColor.cpp
EDcDeserializePredicateResult PredicateIsColorStruct(FDcDeserializeContext& Ctx)
{
return DcDeserializeUtils::PredicateIsUStruct<FColor>(Ctx);
}
Then we'll need to implement a FDcDeserializeDelegate
to deserialize a FColor
. Here we'll do it by writing through R/G/B/A
fields by name with the FDcWriter
API.
// DataConfigExtra/Private/DataConfig/Extra/Deserialize/DcSerDeColor.cpp
FDcResult HandlerColorDeserialize(FDcDeserializeContext& Ctx)
{
FDcPropertyDatum Datum;
DC_TRY(Ctx.Writer->WriteDataEntry(FStructProperty::StaticClass(), Datum));
FString ColorStr;
DC_TRY(Ctx.Reader->ReadString(&ColorStr));
FColor Color = FColor::FromHex(ColorStr);
FColor* ColorPtr = (FColor*)Datum.DataPtr;
*ColorPtr = Color;
return DcOk();
}
Note how we retrieve the hex string, then parse it with FColor::FromHex
.
Upon deserializing we'll need to register these pair of delegates to the FDcDeserializer
.
// DataConfigTests/Private/DcTestBlurb.cpp
FDcDeserializer Deserializer;
DcSetupJsonDeserializeHandlers(Deserializer);
Deserializer.AddStructHandler(
TBaseStructure<FColor>::Get(),
FDcDeserializeDelegate::CreateStatic(DcExtra::HandlerColorDeserialize)
);
And then it's done! It would work recursively on FColor
everywhere, like in UCLASS/USTRUCT
members, in TArray/TSet
and in TMap
as key or values.
Note that DataConfig completely separate serialization and deserialization logic. To serialize FColor
into #RRGGBBAA
string one needs to implement a similar set of methods.
Debug Dump
DcAutomationUtils::DumpToLog()
can dump a FDcPropertyDatum
to a string representation, in which FDcPropertyDatum
is simply a (FProperty
, void*
) fat pointer tuple that can represent anything in the property system:
// DataConfigTests/Private/DcTestBlurb.cpp
FVector Vec(1.0f, 2.0f, 3.0f);
FDcPropertyDatum VecDatum(TBaseStructure<FVector>::Get(), &Vec);
DcAutomationUtils::DumpToLog(VecDatum);
Output would be:
-----------------------------------------
# Datum: 'ScriptStruct', 'Vector'
<StructRoot> 'Vector'
|---<Name> 'X'
|---<Float> '1.000000'
|---<Name> 'Y'
|---<Float> '2.000000'
|---<Name> 'Z'
|---<Float> '3.000000'
<StructEnd> 'Vector'
-----------------------------------------
Additionally we wrapped this into gDcDebug
that can be invoked in MSVC immediate window. Calling it during debug would dump into MSVC Output window:
// DataConfigCore/Public/DataConfig/Automation/DcAutomationUtils.h
struct DATACONFIGCORE_API FDcDebug
{
FORCENOINLINE void DumpStruct(char* StructNameChars, void* Ptr);
FORCENOINLINE void DumpObject(UObject* Obj);
FORCENOINLINE void DumpDatum(void* DatumPtr);
};
/// Access `gDcDebugg` in MSVC immediate window:
///
/// - in monolith builds:
/// gDcDebug.DumpObject(Obj)
///
/// - in DLL builds prefix with dll name:
/// ({,,UE4Editor-DataConfigCore}gDcDebug).DumpObject(ObjPtr)
extern FDcDebug gDcDebug;
Here's an animated demo showing dumping the vector above during debug break in MSVC:
The full expression to evaluate is:
({,,UE4Editor-DataConfigCore}gDcDebug).DumpDatum(&VecDatum)
We need DLL name to locate gDcDebug
in a non monolith build.
Integration
At the moment it supports these the engine versions below:
- UE 5.5
- UE 5.4
- UE 5.3
- UE 5.2
- UE 5.1
- UE 5.0
- UE 4.27
- UE 4.26
- UE 4.25
Integrate DataConfig
Plugin
The easiest way to try out DataConfig is to add it as a plugin into your C++ project. In this section we'll walk through these steps.
Download DataConfig Plugin
The quickest way to try out DataConfig is to download the latest release at DataConfig releases page.
-
Download the zip files on the releases page. Note there're UE4 and UE5 plugin respectively.
-
Unzip it into your Unreal Engine project's
Plugin
folder. The layout should be like this:
<Your project root>
|- <Your project>.uproject
|- Content
|- Source
|- ...
|- Plugins
|- DataConfig
|- DataConfig.uplugin
Generate DataConfig Plugin for UE4/UE5
DataConfig now uses separated uplugin
files for UE4 and UE5 so that we can try out new features in UE5 without dropping support for UE4. We bundled scripts to generate clean plugins for UE4 and UE5. This is how the DataConfig releases are built.
git clone https://github.com/slowburn-dev/DataConfig
# requires python 3.6+
python ./DataConfig/Misc/Scripts/make_dataconfig_ue4.py
python ./DataConfig/Misc/Scripts/make_dataconfig_ue54.py
python ./DataConfig/Misc/Scripts/make_dataconfig_ue5.py
DataConfigXX.uplugin
Ultimately we figured that we'll need multiple .uplugin
files to support acrosss UE versions:
Name | Version |
---|---|
DataConfig4.uplugin | 4.25 - 4.27 |
DataConfig54.uplugin | 5.0 - 5.4 |
DataConfig5.uplugin | 5.5 - Latest |
When packaging for each engine version we rename the one we want and delete the rest.
Manual Steps for UE5 Latest
-
Get a copy of DataConfig repository. Then copy
./DataConfig
(whereDataConfig.uplugin
is located) into your project'sPlugins
directory. -
Delete
DataConfig4.uplugin
, and otherDataConfig5X.uplugin
. -
Delete
DataConfig/Source/DataConfigHeadless
folder. This step is crucial or you your project won't build.
Manual Steps for UE4
-
Get a copy of the repository. Then copy
./DataConfig
(whereDataConfig.uplugin
is located) into your project'sPlugins
directory. -
Delete
DataConfig.uplugin
, then renameDataConfig4.uplugin
toDataConfig.uplugin
. -
Delete
DataConfig/Source/DataConfigHeadless
folder. This step is crucial or you your project won't build. -
Additionally delete UE5 specific modules.
DataConfig/Source/DataConfigEngineExtra5
Validate integration
Follow these steps to ensure DataConfig is properly integrated into your project.
-
Restart your project. There should be a prompt to compile plugin sources. Confirm and wait until your project launches. Then open
Settings -> Plugins
you should see Data Config listed under Project Editor category. -
The plugin comes with a set of tests. Open menu
Window -> Developer Tools -> Session Frontend
. Find and run theDataConfig
tests and it should all pass.
Integrate DataConfigCore
Module
DataConfig is packed into a plugin to bundle automation tests with a few assets. You're encouraged to integrate only the DataConfigCore
module. It contains all core features with minimal dependencies.
Most projects should has a editor module already setup. In this section we'll go through the steps of integrating DataConfigCore
and build it with the project's FooProjectEditor
module.
-
Get a copy of this repository. Then copy
DataConfig/Source/DataConfigCore
into your project'sSource
directory. -
Edit
FooProjectEditor.Build.cs
add addDataConfigCore
as an extra module:using UnrealBuildTool; public class FooProjectEditor : ModuleRules { public FooProjectEditor(ReadOnlyTargetRules Target) : base(Target) { PublicDependencyModuleNames.AddRange(new string[] { //... "DataConfigCore", // <- add this }); } }
-
DataConfig needs to be explicitly initialized before use. Find
FooProjectEditor
module's start up and shut down methods and setup DataConfig accordingly.#include "DataConfig/DcEnv.h" #include "DataConfig/Automation/DcAutomationUtils.h" void FFooProjectEditorModule::StartupModule() { // ... DcStartUp(EDcInitializeAction::SetAsConsole); // dump a FVector to try it out FVector Vec(1.0f, 2.0f, 3.0f); FDcPropertyDatum VecDatum(TBaseStructure<FVector>::Get(), &Vec); DcAutomationUtils::DumpToLog(VecDatum); } void FFooProjectEditorModule::ShutdownModule() { // ... DcShutDown(); }
-
Rebuild the project and restart the editor. Open
Output Log
and you should be able to find the dump results (to filter useCategories -> None
).
You can refer to Module Setup for more detailed integration instructions.
Design
This page documents the overall design, goals and reasoning around DataConfig.
Rationale
At the time we started this project we were looking for a JSON parser that:
- Supports a relaxed JSON spec, i.e. comment and trailing comma.
- Supports custom deserialization logic, i.e. deserializes
FColor
from#RRGGBBAA
. - Supports UE instanced sub objects and polymorphism.
Ultimately we implemented all these in DataConfig. We also tried not to limit this to be a JSON parser but to provide a set of tools for reading-from and writing-to the property system.
If you are an Unreal Engine C++ developers that:
- Looking for alternative JSON reader/writer.
- Looking for MsgPack serializer.
- Looking for a textual configuration format.
- Thinking of implementing custom textual/binary format.
- Write code that deals with
FProperty
on a daily bases.
You should try DataConfig and it's highly likely DataConfig will fit into your solution.
Manifesto
-
Deliver as a quality C++ source library.
DataConfig should ship with no UI nor tooling code. Users are expected to integrate only
DataConfigCore
as a source module. We intentionally limit the scope of DataConfig to a "C++ Library". Our users should be proficient UE C++ programmers.- DataConfig should ship with good testing and documentation coverage.
- DataConfig follows idiomatic UE C++ conventions and has no external dependencies.
DataConfigCore
depends only onCore
andCoreUObject
and can be used in standaloneProgram
targets.- DataConfig API are
UObject
free and stack friendly. - Built-in features serve as examples and sensible defaults. Users are expected to write on their own
Reader/Writer/Handlers
.
-
Runtime performance is not our top priority.
We expect users to use DataConfig in an offline, editor only scenario. In this use case we favor some other aspects over runtime performance:
- Idiomatic. We follow Unreal Engine C++ coding conventions and keep core dependency to only
Core
andCoreUObject
. - Friendly. When processing invalid data and invalid API usage DataConfig should not crash. It should fail explicitly with detailed context and diagnostics.
- Small code size / fast compile time. DataConfig tries not to expose template API.
TDcJsonReader
is explicit instantiated with its definition in private files. - Light memory footprint. Our JSON parser does stream parsing and would not construct the loaded JSON document in memory at all.
With that said DataConfig has decent performance that satisfies common runtime use cases. Do note the subtle differences when doing shipping builds.
- Idiomatic. We follow Unreal Engine C++ coding conventions and keep core dependency to only
-
Works with whatever property system supports.
The idea is that DataConfig supports everything that can be tagged with
UCLASS/USTRUCT/UPROPERTY/UENUM
macros, which covers the full data model of the property system.Fields such as weak object reference and delegates doesn't make much sense to be serialized into textual format. However it turns out supporting the full data model makes it suitable to some other tasks like debug dump and in-memory data wrangling.
This also means that DataConfig focuses only on reading from and writing into C++ data structures. For example we do not have a DOM or object like API for JSON at all. The only use case DataConfig supports is to deserialize from JSON into native C++ objects.
Acknowledgement
- References serde.rs on API and the
SerDe
acronym. - References FullSerializer and OdinSerializer on API.
- JSON parser/writer implementation and test cases references JSON for Modern C++ and RapidJson.
- Integrated nst/JSONTestSuite.
- Integrated kawanet/msgpack-test-suite.
Programming Guides
This section contains doc for programming DataConfig APIs.
DataConfig Data Model
Conceptually the DataConfig data model is defined by 3 C++ types:
EDcDataEntry
- enum covers every possible data type.FDcReader
- methods to read from the data model.FDcWriter
- methods to write into the data model.
And that's it. The obvious missing thing is a DOM like object that you can random access and serialize into - we choose to not implement that and it's crucial to understand this to get to know how DataConfig works.
EDcDataEntry
The enum covers all possible types:
// DataConfigCore/Public/DataConfig/DcTypes.h
UENUM()
enum class EDcDataEntry : uint16
{
None,
Bool,
Name,
String,
Text,
Enum,
Float,
Double,
Int8,
Int16,
Int32,
Int64,
UInt8,
UInt16,
UInt32,
UInt64,
// Struct
StructRoot,
StructEnd,
// Class
ClassRoot,
ClassEnd,
// Map
MapRoot,
MapEnd,
// Array
ArrayRoot,
ArrayEnd,
// Set,
SetRoot,
SetEnd,
// Optional
OptionalRoot,
OptionalEnd,
// Reference
ObjectReference,
ClassReference,
WeakObjectReference,
LazyObjectReference,
SoftObjectReference,
SoftClassReference,
InterfaceReference,
// Delegates
Delegate,
MulticastInlineDelegate,
MulticastSparseDelegate,
// Field
FieldPath,
// Extra
Blob,
// Extension
Extension,
// End
Ended,
};
Most enumerators directly maps to a FProperty
type:
EDcDataEntry::Bool
-FBoolProperty
EDcDataEntry::Name
-FNameProperty
EDcDataEntry::String
-FStrProperty
EDcDataEntry::ArrayRoot/ArrayEnd
-FArrayProperty
It should've covered all possible FProperty
types. There're some additions that has no direct FProperty
mapping:
EDcDataEntry::None
- It's mapsnull
in JSON, and it's also used to explicitly represent null object reference.EDcDataEntry::Ended
- It's a phony type that is returned when there's no more data or reader/writer is in a invalid state.EDcDataEntry::Blob
- It's an extension to allow direct memory read/write from given fields.EDcDataEntry::Extension
- It's an extension that allows additional data formats. MsgPack reader/writer uses this to support itsextension
data types.
FDcReader
FDcReader
is the one and only way to read from DataConfig data model. For every enumerator in EDcDataEntry
there's a member method on FDcReader
to from it.
Here we set up a simple struct trying out the reader methods:
// DataConfigTests/Private/DcTestBlurb.h
USTRUCT()
struct FDcTestExampleSimple
{
GENERATED_BODY()
UPROPERTY() FString StrField;
UPROPERTY() int IntField;
};
// DataConfigTests/Private/DcTestBlurb.cpp
FDcTestExampleSimple SimpleStruct;
SimpleStruct.StrField = TEXT("Foo Str");
SimpleStruct.IntField = 253;
Since we know exactly how the FDcTestExampleSimple
looks like we can manually arrange the read calls:
// DataConfigTests/Private/DcTestBlurb.cpp
FDcPropertyReader Reader{FDcPropertyDatum(&SimpleStruct)};
DC_TRY(Reader.ReadStructRoot(&Struct)); // `FDcTestExampleSimple` Struct Root
DC_TRY(Reader.ReadName(&FieldName)); // 'StrField' as FName
DC_TRY(Reader.ReadString(&StrValue)); // "Foo STr"
DC_TRY(Reader.ReadName(&FieldName)); // 'IntField' as FName
DC_TRY(Reader.ReadInt32(&IntValue)); // 253
DC_TRY(Reader.ReadStructEnd(&Struct)); // `FDcTestExampleSimple` Struct Root
In the example above FDcReader
behave like a iterator as each ReadXXX()
call emits value and move the internal cursor into the next slot. In case we're reading a unknown structure, we can use FReader::PeekRead()
to peek what's coming next.
FDcWriter
FDcWriter
is the counter part of writing into the data config model. To write into the example instance above:
DC_TRY(Writer.WriteStructRoot(FDcStructStat{})); // `FDcTestExampleSimple` Struct Root
DC_TRY(Writer.WriteName(TEXT("StrField"))); // 'StrField' as FName
DC_TRY(Writer.WriteString(TEXT("Alt Str"))); // "Foo STr"
DC_TRY(Writer.WriteName(TEXT("IntField"))); // 'IntField' as FName
DC_TRY(Writer.WriteInt32(233)); // 233
DC_TRY(Writer.WriteStructEnd(FDcStructStat{})); // `FDcTestExampleSimple` Struct Root
There's also FDcWriter::PeekRead()
to query whether it's possible to write given data type.
Sum Up
DataConfig provide FDcReader
and FDcWriter
to access the property system. It can be considered as a friendly alternative to the property system API. This is also how we support JSON and MsgPack in an uniform API.
Error Handling
Proper error handling is crucial to implement robust serialization as it needs to deal with unknown user input. DataConfig also provide diagnostic to help users quickly pin down common errors such as typos or missing colons in JSON. Here's an example:
# DataConfig Error: Enum name not found in enum type: EDcTestExampleEnum, Actual: 'Far'
- [JsonReader] --> <in-memory>4:25
2 | {
3 | "StrField" : "Lorem ipsum dolor sit amet",
4 | "EnumField" : "Far",
| ^
5 | "Colors" : [
6 | "#FF0000FF", "#00FF00FF", "#0000FFFF"
- [PropertyWriter] Writing property: (FDcTestExampleStruct)$root.(EEDcTestExampleEnum)EnumField
Internally DataConfig applies a consistent error handling strategy across all API. User code is expected to follow along.
Returning FDcResult
The gist is that if a method can fail, it should return a FDcResult
. It's a simple struct:
// DataConfigCore/Public/DataConfig/DcTypes.h
struct DATACONFIGCORE_API DC_NODISCARD FDcResult
{
enum class EStatus : uint8
{
Ok,
Error
};
EStatus Status;
FORCEINLINE bool Ok() const
{
return Status == EStatus::Ok;
}
};
// See FDcReader's methods as example
// DataConfigCore/Public/DataConfig/Reader/DcReader.h
struct DATACONFIGCORE_API FDcReader
{
//...
virtual FDcResult ReadBool(bool* OutPtr);
virtual FDcResult ReadName(FName* OutPtr);
virtual FDcResult ReadString(FString* OutPtr);
virtual FDcResult ReadText(FText* OutPtr);
virtual FDcResult ReadEnum(FDcEnumData* OutPtr);
//...
};
Then use DC_TRY
to call these kinds of functions. The macro itself does early return when result is not Ok
:
// DataConfigCore/Public/DataConfig/DcTypes.h
#define DC_TRY(expr) \
do { \
::FDcResult Ret = (expr); \
if (!Ret.Ok()) { \
return Ret; \
} \
} while (0)
// Example of calling methods returning `FDcResult`
// DataConfigExtra/Private/DataConfig/Extra/Deserialize/DcDeserializeColor.cpp
template<>
FDcResult TemplatedWriteColorDispatch<EDcColorDeserializeMethod::WriterAPI>(const FColor& Color, FDcDeserializeContext& Ctx)
{
DC_TRY(Ctx.Writer->WriteStructRoot(FDcStructStat{ TEXT("Color"), FDcStructStat::WriteCheckName }));
DC_TRY(Ctx.Writer->WriteName(TEXT("B")));
DC_TRY(Ctx.Writer->WriteUInt8(Color.B));
DC_TRY(Ctx.Writer->WriteName(TEXT("G")));
DC_TRY(Ctx.Writer->WriteUInt8(Color.G));
DC_TRY(Ctx.Writer->WriteName(TEXT("R")));
DC_TRY(Ctx.Writer->WriteUInt8(Color.R));
DC_TRY(Ctx.Writer->WriteName(TEXT("A")));
DC_TRY(Ctx.Writer->WriteUInt8(Color.A));
DC_TRY(Ctx.Writer->WriteStructEnd(FDcStructStat{ TEXT("Color"), FDcStructStat::WriteCheckName }));
return DcOk();
}
This pattern is similar to Outcome and std::expected except we give up using the return value. Return values should be passed through reference or pointers in function arguments.
Diagnostics
When implementing a method that returns FDcResult
you have 2 options:
- Return
DcOk()
on succeed. - Return
DC_FAIL(<Catetory>, <ErrId>)
on error.
Examples:
// DataConfigTests/Private/DcTestBlurb.cpp
FDcResult Succeed() {
// succeed
return DcOk();
}
FDcResult Fail() {
// fail !
return DC_FAIL(DcDCommon, Unexpected1) << "My Custom Message";
}
In the examples above DcDCommon
and Unexpected1
are called error category and error id respectively. DcDCommon
is a built-in error category:
// DataConfigCore/Public/DataConfig/Diagnostic/DcDiagnosticCommon.h
namespace DcDCommon
{
static const uint16 Category = 0x1;
enum Type : uint16
{
//...
Unexpected1,
};
} // namespace DcDCommon
// DataConfigCore/Private/DataConfig/Diagnostic/DcDiagnosticCommon.cpp
namespace DcDCommon
{
static FDcDiagnosticDetail _CommonDetails[] = {
// ...
{ Unexpected1, TEXT("Unexpected: '{0}'") },
};
Note that we can pipe argument into the diagnostic. The diagnostic reported by invoking Fail()
would be like:
* # DataConfig Error: Unexpected: 'My Custom Message'
Conclusion
DataConfig uses FDcResult
, DC_TRY
, DC_FAIL
for error handling. It's lightweight and relatively easy to grasp. There's still some limitations though:
FDcResult
occupied the return position making passing value to parent a bit cumbersome.- For now we always stop as the first error. There's no plan to support error recovery.
Some closing notes:
- Reported diagnostics get queued. You'll need to call
FDcEnv::FlushDiags()
to flush them to consumers. - See
DcDiagnosticExtra.h/cpp
for how to register user category. - See
DcEditorExtraModule.cpp - FDcMessageLogDiagnosticConsumer
for custom diagnostic handler and formatting.
Env
DataConfig put most global state into a stack of FDcEnv
:
// DataConfigCore/Public/DataConfig/DcEnv.h
struct DATACONFIGCORE_API FDcEnv
{
TArray<FDcDiagnostic> Diagnostics;
TSharedPtr<IDcDiagnosticConsumer> DiagConsumer;
TArray<FDcReader*> ReaderStack;
TArray<FDcWriter*> WriterStack;
bool bExpectFail = false; // mute debug break
FDcDiagnostic& Diag(FDcErrorCode InErr);
void FlushDiags();
FORCEINLINE FDcDiagnostic& GetLastDiag()
{
checkf(Diagnostics.Num(), TEXT("<empty diagnostics>"));
return Diagnostics.Last();
}
~FDcEnv();
};
DataConfig needs explicit initializatioon before use. This is done through manually DcStartUp()
. There's also a paired DcShutdown()
that should be called when DataConfig isn't used anymore. Here's an example:
// DataConfigEngineExtra/Private/DcEngineExtraModule.cpp
void FDcEngineExtraModule::StartupModule()
{
UE_LOG(LogDataConfigCore, Log, TEXT("DcEngineExtraModule module starting up"));
DcRegisterDiagnosticGroup(&DcDExtra::Details);
DcRegisterDiagnosticGroup(&DcDEngineExtra::Details);
DcStartUp(EDcInitializeAction::Minimal);
//...
}
void FDcEngineExtraModule::ShutdownModule()
{
DcShutDown();
//...
}
The active FDcEnv
is accessed by calling global function DcEnv()
. Inside the Env:
Diagnostics
: all diagnostics are flushed into env.DiagConsumer
: diagnostic handler, format and print diagnostic to log orMessageLog
or even on screen.ReaderStack/WriterStack
: used to pass along reader/writer down the callstack. SeeFScopedStackedReader
uses for example.- ... and everything else.
You can use DcPushEnv()
to create new env then destroy it calling DcPopEnv()
. At this moment it's mostly used to handle reentrant during serialization. See FDcScopedEnv
uses for examples.
Reader/Writer
FDcReader/FDcWriter defines the set of API for accessing DataConfig data model. Here's a check list for implementing a reader/writer:
- Implement
GetID()/ClassID()
for RTTI. - Implement
PeekRead()/PeekWriter()
and selected set ofReadXXX()/WriteXXX()
. - Implement
FormatDiagnostic()
for error reporting.
You should look at builtin implementation for references. Here's some general rules and caveats:
-
PeekRead()/PeekWrite()
should act like it's side-effect free.This means that it's OK to call
PeekRead()/PeekWrite()
multiple times. In comparison access methods likeReadBool()/WriteBool()
consume the data and alternate internal state. Note that under the hood it might do anything. Both returnsFDcResult
so the peek can fail. The reason behind this is that callingPeekRead()/PeekWrite()
is totally optional. InFDcJsonReader::PeekRead()
we do parsing and cache the parsed result to follow this convention. -
CastByID()
does not respect inheritance hierarchy.We have this very minimal RTTI implemetantion that only allow casting to the exact type.
-
Implement a subset of the data model.
The API is designed in a way that it covers the whole Property System. It's also a super set that can express common formats like JSON/MsgPack. For example JSON don't have struct, class or set. It's actually the job of Serializer/Deserializer to convert between these subsets to the property system.
Builtin Reader/Writer
We have 3 major pairs of reader/writers:
FDcPropertyReader/FDcPropertyWriter
- Accesing Unreal Engin property system.FDcJsonReader/FDcJsonWriter
- JSON support.FDcMsgPackReader/FDcMsgPackWriter
- MsgPack support.
These are all talked about in details in the formats section. We'll go through other builtin Reader/Writers below.
FDcPipeVisitor
and FDcPrettyPrintWriter
FDcPipeVisitor
takes a FDcReader
and a FDcWriter
then start peek-read-write loop until it peeks EDcDataEntry::Ended
from reader or an error happens
Then there's FDcPrettyPrintWriter
that dumps everything that got write to it as string.
Combining these two we get a way to dump arbitrary FDcReader
into a string!. This is how built-in debug dump features are implemented:
// DataConfigCore/Private/DataConfig/Automation/DcAutomationUtils.cpp
void DumpToOutputDevice(...)
{
//...
FDcPropertyReader PropReader(Datum);
FDcPrettyPrintWriter PrettyWriter(Output);
FDcPipeVisitor PrettyPrintVisit(&PropReader, &PrettyWriter);
if (!PrettyPrintVisit.PipeVisit().Ok())
ScopedEnv.Get().FlushDiags();
//...
}
FDcPipeVisitor
is a handy utility that we use it extensively through the code base for various cases. Try FDcPipeVisitor
when you got a reader/writer pair.
There's also FNoopWriter
takes every write and do nothing with it.
Composition
Reader/Writers can also be composited and nested:
FDcWeakCompositeWriter
is a writer that multiplex into a list of writers. You can combine an arbitrary writer with aFPrettyPrintWriter
then get a tracing writer.FDcPutbackReader/FPutbackWriter
: Reader/writers don't support lookahead. It can only peek next item's type but not value. This class is used to support limited lookahead by putting back read value. We'll see it being used in implementing custom deserializer handlers.
Conclusion
Implement new FDcReader/FDcWriter
when you want to support a new file format. You can also write utility reader/writer that composite existing ones.
Serializer/Deserializer
Serializer/Deserializer are built on top of the data model to convert between external format and Unreal Engine property system.
Context
A company class to the deserializer is FDcDeserializeContext
:
// DataConfigCore/Public/DataConfig/Deserialize/DcDeserializeTypes.h
struct DATACONFIGCORE_API FDcDeserializeContext
{
//...
FDcDeserializer* Deserializer;
FDcReader* Reader;
FDcPropertyWriter* Writer;
//...
};
Note how it takes an FDcReader
and a FPropertyWriter
- we're deserializing arbitrary format into the property system.
The mirrored version for serializer is FDcSerializeContext
.
// DataConfigCore/Public/DataConfig/Serialize/DcSerializeTypes.h
struct DATACONFIGCORE_API FDcSerializeContext
{
//...
FDcSerializer* Serializer = nullptr;
FDcPropertyReader* Reader = nullptr;
FDcWriter* Writer = nullptr;
//...
};
Note how it takes an FDcWriter
and a FDcPropertyReader
- we're serializing data from the property system to arbitrary format.
Since serializer and deserializer have extremely similar APIs, we're showing examples using deserializer below from here.
Handlers
Say that we're deserializing a JSON object into a USTRUCT
instance. The FDcJsonReader
implements ReadMapRoot()/ReadMapEnd()
but doesn't have ReadStructRoot()/ReadStructEnd()
. To make the conversion we basically want to map ReadMap()
and calls into WriteStruct()
calls. This is where handlers come into play:
// DataConfigCore/Public/DataConfig/SerDe/DcDeserializeCommon.inl
FDcResult DcHandlerDeserializeMapToStruct(FDcDeserializeContext& Ctx)
{
FDcStructAccess Access;
DC_TRY(Ctx.Reader->ReadMapRoot());
DC_TRY(Ctx.Writer->WriteStructRootAccess(Access));
EDcDataEntry CurPeek;
while (true)
{
DC_TRY(Ctx.Reader->PeekRead(&CurPeek));
if (CurPeek == EDcDataEntry::MapEnd)
break;
FName FieldName;
DC_TRY(Ctx.Reader->ReadName(&FieldName));
DC_TRY(Ctx.Writer->WriteName(FieldName));
DC_TRY(DcDeserializeUtils::RecursiveDeserialize(Ctx));
}
DC_TRY(Ctx.Reader->ReadMapEnd());
DC_TRY(Ctx.Writer->WriteStructEnd());
return DcOk();
}
// DataConfigCore/Private/DataConfig/Deserialize/Handlers/Json/DcJsonCommonDeserializers.cpp
FDcResult HandlerStructRootDeserialize(FDcDeserializeContext& Ctx)
{
return DcHandlerDeserializeMapToStruct(Ctx);
}
Note that Ctx.Reader
is a FDcReader
that can be any derived class, while Ctx.Writer
is always a FDcPropertyWriter
. Deserialize handlers have an uniform signature:
using FDcDeserializeDelegateSignature = FDcResult(*)(FDcDeserializeContext& Ctx);
Deserializer Setup
Note how deserialize handler above doesn't specify when it should be invoked.
These info are described in FDcDeserializer
:
// DataConfigCore/Public/DataConfig/Deserialize/DcDeserializer.h
struct DATACONFIGCORE_API FDcDeserializer : public FNoncopyable
{
//...
FDcResult Deserialize(FDcDeserializeContext& Ctx);
void AddDirectHandler(FFieldClass* PropertyClass, FDcDeserializeDelegate&& Delegate);
void AddDirectHandler(UClass* PropertyClass, FDcDeserializeDelegate&& Delegate);
void AddPredicatedHandler(FDcDeserializePredicate&& Predicate, FDcDeserializeDelegate&& Delegate);
//...
};
Comparing to FDcDeserializeContext
, which describes data needed for a single run,
FDcDeserializer
contains info on what handlers to execute. Deserializer can also be reused across multiple
runs.
We use "direct handlers" to cover common cases:
// DataConfigCore/Private/DataConfig/Deserialize/DcDeserializerSetup.cpp
Deserializer.AddDirectHandler(FArrayProperty::StaticClass(), FDcDeserializeDelegate::CreateStatic(HandlerArrayDeserialize));
Deserializer.AddDirectHandler(FSetProperty::StaticClass(), FDcDeserializeDelegate::CreateStatic(HandlerSetDeserialize));
Deserializer.AddDirectHandler(FMapProperty::StaticClass(), FDcDeserializeDelegate::CreateStatic(HandlerMapDeserialize));
These basically says that "when running into array, set, map properties, use these provided handlers".
Then there's "struct handler" that uses a UStruct
that maps a specific class/struct
to a handler:
// DataConfigCore/Private/DataConfig/Deserialize/DcDeserializerSetup.cpp
Deserializer.AddStructHandler(TBaseStructure<FGuid>::Get(), FDcDeserializeDelegate::CreateStatic(HandlerGuidDeserialize));
Deserializer.AddStructHandler(TBaseStructure<FColor>::Get(), FDcDeserializeDelegate::CreateStatic(HandlerColorDeserialize));
Deserializer.AddStructHandler(TBaseStructure<FDateTime>::Get(), FDcDeserializeDelegate::CreateStatic(HandlerDateTimeDeserialize));
This means "when running into a FGuid
, use these attached handlers". This is run before direct handlers.
Then we have "predicated handler" that get tested very early. This is how we allow custom conversion logic setup for very specific class:
// DataConfigCore/Private/DataConfig/Deserialize/DcDeserializerSetup.cpp
EDcDeserializePredicateResult PredicateIsScalarArrayProperty(FDcDeserializeContext& Ctx)
{
FProperty* Prop = CastField<FProperty>(Ctx.TopProperty().ToField());
return Prop && Prop->ArrayDim > 1 && !Ctx.Writer->IsWritingScalarArrayItem()
? EDcDeserializePredicateResult::Process
: EDcDeserializePredicateResult::Pass;
}
// ...
Deserializer.AddPredicatedHandler(
FDcDeserializePredicate::CreateStatic(PredicateIsScalarArrayProperty),
FDcDeserializeDelegate::CreateStatic(HandlerArrayDeserialize)
);
By convention the current deserializing property can be retrieved with Ctx.TopProperty()
. PredicateIsScalarArrayProperty
here
checks if it's wring a scalar array with non 1 dimension, if that's the case it would need to treat it like an array.
Note that all registered predicate handler is iterated through on every property, then proceed to handler on first success match or fall through to struct/direct handlers when no match. Use it only when struct/direct handlers doesn't fit.
To recap:
Handler Type | Order | Usage | Execution |
---|---|---|---|
Predicate handler | First | Most flexible | Iteration through all and match first success |
Struct handler | Second | "Is FColor ? " | Direct match |
Direct handler | Last | "Is Map/Array ? " | Direct match |
Serializer Setup
Serializer has exactly the same API as deserializer and the semantics are all the same.
Sum Up
Serializer/Deserializer are built on top of Reader/Writer, to convert between Unreal Engine property system and external data formats.
FDcSerializeContext/FDcDeserializeContext
contains data.FDcSerializer/FDcDeserializer
contains setup.- Implement
FDcDeserializeDelegate/FDcSerializeDelegate
andFDcDeserializePredicate/FDcSerializePredicate
pair for custom conversion logic.
Formats
This section contains documentation for supported formats.
JSON
JSON is likely the most popular data interchange format. Unreal Engine already supports it with JsonUtilities
and some related modules. We provide an alternative implementation along with DataConfig.
JSON Reader
FDcJsonReader
is the DataConfig JSON reader:
// DataConfigTests/Private/DcTestBlurb.cpp
FString Str = TEXT(R"(
{
"Str": "Fooo",
"Number": 1.875,
"Bool": true
}
)");
FDcJsonReader Reader(Str);
// calling read methods
FString KeyStr;
FString GotStr;
double GotNumber;
bool GotBool;
DC_TRY(Reader.ReadMapRoot());
DC_TRY(Reader.ReadString(&KeyStr));
DC_TRY(Reader.ReadString(&GotStr));
DC_TRY(Reader.ReadString(&KeyStr));
DC_TRY(Reader.ReadDouble(&GotNumber));
DC_TRY(Reader.ReadString(&KeyStr));
DC_TRY(Reader.ReadBool(&GotBool));
DC_TRY(Reader.ReadMapEnd());
// validate results
check(GotStr == TEXT("Fooo"));
check(GotNumber == 1.875);
check(GotBool == true);
In the example above we deserialized a JSON
object from string. The first and last calls are ReadMapRoot
and ReadMapEnd
, which are also used to read Unreal's TMap
properties. The difference is that UE's TMap
is strictly typed while JSON object values can have arbitrary type. This means that if you use FDcPipeVisitor
to pipe a FDcJsonReader
into a FDcPropertyWriter
it won't work.
Remember that DataConfig data model is designed to support conversion between subsets within the data model. As long as you can use FDcReader/FDcWriter
API to describe the format you want to serialize you're good to go. Mapping and conversion between these different shapes of reader/writers are handled by deserializers.
Some additional caveats:
- Similar to stock
TJsonReader
, we provideTDcJsonReader
with 2 specializations:- Usually you just use
FDcJsonReader
that reads fromFString, TCHAR*
. - Under the hood there're
FDcAnsiJsonReader
that reads ANSICHAR string andFDcWideJsonReader
that reads WIDECHAR string.
- Usually you just use
- We support a relaxed superset of JSON:
- Allow C Style comments, i.e
/* block */
and// line
. - Allow trailing comma, i.e
[1,2,3,],
. - Allow non object root. You can put a list as the root, or even string, numbers.
- Allow C Style comments, i.e
- Number parsing are delegated to Unreal's built-ins to reduce dependencies. We might change this in the future.
- Parse numbers:
TCString::Atof/Strtoi/Strtoi64
- Parse numbers:
JSON Writer
FDcJsonWriter
is the DataConfig JSON writer:
// DataConfigTests/Private/DcTestBlurb.cpp
FDcJsonWriter Writer;
DC_TRY(Writer.WriteMapRoot());
DC_TRY(Writer.WriteString(TEXT("Str")));
DC_TRY(Writer.WriteString(TEXT("Fooo")));
DC_TRY(Writer.WriteString(TEXT("Number")));
DC_TRY(Writer.WriteFloat(1.875f));
DC_TRY(Writer.WriteString(TEXT("Bool")));
DC_TRY(Writer.WriteBool(true));
DC_TRY(Writer.WriteMapEnd());
Writer.Sb.Append(TCHAR('\n'));
FString Str = TEXT(R"(
{
"Str" : "Fooo",
"Number" : 1.875,
"Bool" : true
}
)");
// validate results
check(DcReindentStringLiteral(Str) == Writer.Sb.ToString());
return DcOk();
- Similar to stock
TJsonWriter
, we provideTDcJsonWriter
with 2 specializations:- Usually you just use
FDcJsonWriter
that writesFString, TCHAR*
. - Under the hood there're
FDcAnsiJsonWriter
that writes ANSICHAR string andFDcWideJsonWriter
that writes WIDECHAR string.
- Usually you just use
- It takes a
Config
object that specify formatting settings like indentation size and new lines.FDcPrettyJsonWriter
is a type alias that formats indented JSON.FDcCondensedJsonWriter
is a type alias that format single line, condensed output.
FDcJsonWriter
owns the output string buffer, inFDcJsonWriter::Sb
.- By writing to a single writer and appending a new line after each serialization, we can output NDJSON.
- Our JSON reader is also flexible enough to directly load NDJSON. See corpus benchmark.
JSON Serialize/Deserialize
DataConfig bundles a set of JSON serialize and deserialize handlers, which are all roundtrip-able:
// DataConfigTests/Private/DcTestBlurb.cpp
#include "DataConfig/Deserialize/DcDeserializerSetup.h"
// ...
// create and setup a deserializer
FDcDeserializer Deserializer;
DcSetupJsonDeserializeHandlers(Deserializer);
// create and setup a serializer
FDcSerializer Serializer;
DcSetupJsonSerializeHandlers(Serializer);
Schema
JSON types get mapped into DataConfig data model in a very unsurprising way.
JSON Type | DcDataEntry |
---|---|
Boolean | Bool |
Null | None |
String | String, Name, Text, Enum |
Number | (All numerics) |
Array | Array, Set |
Object | Class, Struct, Map |
Here's an example:
// DataConfigTests/Private/DcTestDeserialize.cpp
FString Str = TEXT(R"(
{
"BoolField" : true,
"NameField" : "AName",
"StringField" : "AStr",
"TextField" : "AText",
"EnumField" : "Tard",
"FloatField" : 17.5,
"DoubleField" : 19.375,
"Int8Field" : -43,
"Int16Field" : -2243,
"Int32Field" : -23415,
"Int64Field" : -1524523,
"UInt8Field" : 213,
"UInt16Field" : 2243,
"UInt32Field" : 23415,
"UInt64Field" : 1524523,
}
)");
// deserialized equivelent
FDcTestStruct1 Expect;
Expect.BoolField = true;
Expect.NameField = TEXT("AName");
Expect.StringField = TEXT("AStr");
Expect.TextField = FText::FromString(TEXT("AText"));
Expect.EnumField = EDcTestEnum1::Tard;
Expect.FloatField = 17.5f;
Expect.DoubleField = 19.375;
Expect.Int8Field = -43;
Expect.Int16Field = -2243;
Expect.Int32Field = -23415;
Expect.Int64Field = -1524523;
Expect.UInt8Field = 213;
Expect.UInt16Field = 2243,
Expect.UInt32Field = 23415;
Expect.UInt64Field = 1524523;
Map
JSON only allow string as object/mapping keys, while in UE TMap<>
can use any type. When doing serialization TMap<FString/FName/FText,(TValue)>
types would be directly converted to a JSON object:
// DataConfigTests/Public/DcTestDeserialize.h
USTRUCT()
struct FDcTestStruct3
{
// ...
UPROPERTY() TMap<FString, FString> StringMap;
};
// DataConfigTests/Public/DcTestDeserialize.cpp
{
// ...
"StringMap" : {
"One": "1",
"Two": "2",
"Three": "3",
},
}
For other key types it would be serialized as an array of objects:
// DataConfigTests/Public/DcTestSerDe.h
USTRUCT()
struct FDcTestStructMaps
{
// ...
UPROPERTY() TMap<FColor, FString> ColorKeyMap;
UPROPERTY() TMap<EDcTestEnumFlag, FString> EnumFlagsMap;
};
// DataConfigTests/Public/DcTestDeserialize.cpp
{
"ColorKeyMap" : {
"#FF0000FF" : "Red",
"#00FF00FF" : "Green",
"#0000FFFF" : "Blue"
},
"EnumFlagsMap" : [
{
"$key" : [],
"$value" : "None"
},
{
"$key" : [
"One",
"Three"
],
"$value" : "One | Three"
},
{
"$key" : [
"Five"
],
"$value" : "Five"
}
]
}
Enum Flags
UENUM
that get marked with Bitflags
meta are deserialized from a list of strings:
// DataConfigTests/Public/DcTestDeserialize.h
UENUM(meta = (Bitflags))
enum class EDcTestEnumFlag :uint32
{
None = 0,
One = (1 << 0),
Two = (1 << 1),
//...
};
ENUM_CLASS_FLAGS(EDcTestEnumFlag);
// DataConfigTests/Private/DcTestDeserialize.cpp
FString Str = TEXT(R"(
{
"EnumFlagField1" : [],
"EnumFlagField2" : ["One", "Three", "Five"],
}
)");
// deserialized equivelent
FDcTestStructEnumFlag1 Expect;
Expect.EnumFlagField1 = EDcTestEnumFlag::None;
Expect.EnumFlagField2 = EDcTestEnumFlag::One | EDcTestEnumFlag::Three | EDcTestEnumFlag::Five;
Sub Objects
By default we treat UOBJECT
marked with DefaultToInstanced, EditInlineNew
and UPROPERTY
marked with Instanced
as sub object. In this case we'll actually instantiate new object during deserialization, using Ctx.TopObject()
as parent:
// DataConfigTests/Public/DcTestProperty.h
UCLASS(BlueprintType, EditInlineNew, DefaultToInstanced)
class UDcBaseShape : public UObject
{
//...
UPROPERTY() FName ShapeName;
};
UCLASS()
class UDcShapeBox : public UDcBaseShape
{
//...
UPROPERTY() float Height;
UPROPERTY() float Width;
};
UCLASS()
class UDcShapeSquare : public UDcBaseShape
{
//...
UPROPERTY() float Radius;
};
// DataConfigTests/Public/DcTestDeserialize.h
USTRUCT()
struct FDcTestStructShapeContainer1
{
GENERATED_BODY()
UPROPERTY() UDcBaseShape* ShapeField1;
UPROPERTY() UDcBaseShape* ShapeField2;
UPROPERTY() UDcBaseShape* ShapeField3;
}USTRUCT()
struct FDcEditorExtraTestObjectRefs1
{
GENERATED_BODY()
UPROPERTY() UObject* ObjField1;
UPROPERTY() UObject* ObjField2;
UPROPERTY() UObject* ObjField3;
UPROPERTY() UObject* ObjField4;
};
// DataConfigTests/Private/DcTestDeserialize.cpp
FString Str = TEXT(R"(
{
"ShapeField1" : {
"$type" : "DcShapeBox",
"ShapeName" : "Box1",
"Height" : 17.5,
"Width" : 1.9375
},
"ShapeField2" : {
"$type" : "DcShapeSquare",
"ShapeName" : "Square1",
"Radius" : 1.75,
},
"ShapeField3" : null
}
)");
// deserialized equivelent
UDcShapeBox* Shape1 = NewObject<UDcShapeBox>();
Shape1->ShapeName = TEXT("Box1");
Shape1->Height = 17.5;
Shape1->Width = 1.9375;
Expect.ShapeField1 = Shape1;
UDcShapeSquare* Shape2 = NewObject<UDcShapeSquare>();
Shape2->ShapeName = TEXT("Square1");
Shape2->Radius = 1.75;
Expect.ShapeField2 = Shape2;
Expect.ShapeField3 = nullptr;
Note that criteria for sub object selection can be easily overridden with a new deserialize predicate or alternative FDcPropertyConfig
when constructing the reader.
Also see AnyStruct, InlineStruct and InstancedStruct for lighter weight alternatives.
Object and Class Reference
We support multiple ways of referencing a UObject
in memory or serialized on disk:
// DataConfigEditorExtra/Private/DataConfig/EditorExtra/Tests/DcTestDeserializeEditor.h
USTRUCT()
struct FDcEditorExtraTestObjectRefs1
{
GENERATED_BODY()
UPROPERTY() UObject* ObjField1;
UPROPERTY() UObject* ObjField2;
UPROPERTY() UObject* ObjField3;
UPROPERTY() UObject* ObjField4;
};
// DataConfigEditorExtra/Private/DataConfig/EditorExtra/Tests/DcTestDeserializeEditor.cpp
FString Str = TEXT(R"(
{
"ObjField1" : "DcEditorExtraNativeDataAsset'/DataConfig/DcFixture/DcTestNativeDataAssetAlpha.DcTestNativeDataAssetAlpha'",
"ObjField2" : "/DataConfig/DcFixture/DcTestNativeDataAssetAlpha",
"ObjField3" :
{
"$type" : "DcEditorExtraNativeDataAsset",
"$path" : "/DataConfig/DcFixture/DcTestNativeDataAssetAlpha"
},
"ObjField4" : null,
}
)");
// deserialized equivelent
UDcEditorExtraNativeDataAsset* DataAsset = Cast<UDcEditorExtraNativeDataAsset>(StaticLoadObject(
UDcEditorExtraNativeDataAsset::StaticClass(),
nullptr,
TEXT("/DataConfig/DcFixture/DcTestNativeDataAssetAlpha"),
nullptr
));
Expect.ObjField1 = DataAsset;
Expect.ObjField2 = DataAsset;
Expect.ObjField3 = DataAsset;
Expect.ObjField4 = nullptr;
In the example above, ObjField1
uses the reference string that can be retrieved in editor context menu:
For ObjField2/ObjField3
relative path to the uasset
is used, but without file name suffix.
We also support class reference fields of TSubclassOf<>
s:
// DataConfigTests/Private/DcTestDeserialize.h
USTRUCT()
struct FDcTestStructSubClass1
{
GENERATED_BODY()
UPROPERTY() TSubclassOf<UStruct> StructSubClassField1;
UPROPERTY() TSubclassOf<UStruct> StructSubClassField2;
UPROPERTY() TSubclassOf<UStruct> StructSubClassField3;
};
// DataConfigTests/Private/DcTestDeserialize.cpp
FString Str = TEXT(R"(
{
"StructSubClassField1" : null,
"StructSubClassField2" : "ScriptStruct",
"StructSubClassField3" : "DynamicClass",
}
)");
// deserialized equivelent
FDcTestStructSubClass1 Expect;
Expect.StructSubClassField1 = nullptr;
Expect.StructSubClassField2 = UScriptStruct::StaticClass();
Expect.StructSubClassField3 = UDynamicClass::StaticClass();
Note that these do not support Blueprint classes. The direct reason is that Blueprint depends on Engine
module and we'd like not to take dependency on in DataConfigCore
.
We do have an example that supports Blueprint classes, see DataConfigEditorExtra - DcDeserializeBPClass.h/cpp
Soft Lazy as String
DcSetupJsonSerializeHandlers()/DcSetupJsonDeserializeHandlers()
accepts an enum to setup alternative handlers. For now StringSoftLazy
branch would setup special FSoftObjectProperty/FLazyObjectProperty
handlers that directly serialize these into string. Comparing to this the default setup would always resolve the indirect reference into memory, which maybe isn't always desirable.
// DataConfigTests/Private/DcTestBlurb.cpp
FDcTestStructRefs1 Source{};
UObject* TestsObject = StaticFindObject(UObject::StaticClass(), nullptr, TEXT("/Script/DataConfigTests"));
Source.SoftField1 = TestsObject;
Source.LazyField1 = TestsObject;
FDcJsonWriter Writer;
DC_TRY(DcAutomationUtils::SerializeInto(&Writer, FDcPropertyDatum(&Source),
[](FDcSerializeContext& Ctx) {
DcSetupJsonSerializeHandlers(*Ctx.Serializer, EDcJsonSerializeType::StringSoftLazy);
}, DcAutomationUtils::EDefaultSetupType::SetupNothing));
// serialized result
{
// ...
"SoftField1" : "/Script/DataConfigTests",
"SoftField2" : "",
"LazyField1" : "C851179E-45A51045-0006AE91-F9B16EC0",
"LazyField2" : "00000000-00000000-00000000-00000000"
}
Override Config
Sometimes we want to write a single line array/object within a pretty print JSON writer. To implement this we added mechanic to apply an override config in scope:
// DataConfigTests/Private/DcTestJSON2.cpp
DC_TRY(Writer.WriteMapRoot());
{
DC_TRY(Writer.WriteName("Obj1"));
FDcScopedTryUseJSONOverrideConfig ScopedOverrideConfig(&Writer);
DC_TRY(Writer.WriteMapRoot());
DC_TRY(Writer.WriteString("Foo"));
DC_TRY(Writer.WriteString("Bar"));
DC_TRY(Writer.WriteString("Alpha"));
DC_TRY(Writer.WriteString("Beta"));
DC_TRY(Writer.WriteMapEnd());
}
/// ...
With FDcScopedTryUseJSONOverrideConfig
the following object would be written on a single line:
{
"Obj1" : {"Foo" : "Bar", "Alpha" : "Beta"},
// ...
}
This is how UE Core Types serializer handlers are implemented.
Caveats
Here're some closing notes:
-
For meta fields like
$type
it must be the first member, meaning object fields are order dependent. This means that the JSON we're supporting is a super set of standard JSON spec (again). -
Bundled serializers and deserializers are designed to be roundtrip-able. For example in test
DataConfig.Core.RoundTrip.JsonRoundtrip1_Default
:- Serialize
UDcTestRoundtrip1
instanceSource
into JSON. - Then deserialize JSON above into instance
Dest
. - Deep-compare
Source
andDest
. If they're equal them we say it's a roundtrip.
Note that we carefully picked float and doubles in the test case, as it's tricky to support floating point roundtrip. We might consider supporting this with alternative float parse and format routines.
- Serialize
-
There're many data types that can not be deserialized from JSON, for example
Delegate/WeakObjectReference
. Remember that you always have the option to override or selectively enable handlers to support additional properties that make sense in your context. SeeDcSetupJsonDeserializeHandlers()
body on how handlers are registered. You can skip this method and select the ones you want and provide additional handlers. -
The JSON handlers are designed to NOT read anything during the deserialization. This is crucial since
USTRUCT
can contain uninitialized fields. For example:// DataConfigTests/Private/DcTestBlurb.cpp FString Str = TEXT(R"( { // pass } )"); FDcJsonReader Reader(Str); FDcTestExampleSimple Dest; FDcPropertyDatum DestDatum(&Dest); DC_TRY(DcAutomationUtils::DeserializeJsonInto(&Reader, DestDatum)); check(Dest.StrField.IsEmpty()); // but Dest.IntField contains uninitialized value DcAutomationUtils::DumpToLog(DestDatum); // dump results <StructRoot> 'DcTestExampleSimple' |---<Name> 'StrField' |---<String> '' |---<Name> 'IntField' |---<Int32> '1689777552' // <- arbitrary value <StructEnd> 'DcTestExampleSimple'
This would cause trouble when you try read a pointer field during deserialization. Remember that primitive fields might be uninitialized during deserialization when implementing your own handlers.
-
One interesting trait of the pull/push styled API is that
FDcJsonReader
does not preemptively parse number into double and convert it toint/float
later on. When reading a number token it would do the number parsing at call site. IfReadIntX()
is called then the number is parsed as integer. IfReadFloat()/ReadDouble()
is called the token will be parsed as floating point. -
FDcJsonReader::FinishRead()
can be used to check JSON is fully consumed and sound (no trailing tokens, object/array fully closed). This optional by design and not part ofFDcReader/FDcWriter
API and would make reading more flexible. See NDJSON for more.
MsgPack
MsgPack is an popular binary serialization format. It can be considered as a binary superset of JSON. Unreal Engine already supports Cbor
module which is a format which is very similar to MsgPack.
We choose to implement MsgPack as we're more familiar with it and also providing an alternative.
MsgPack Reader/Writer
For the most part MsgPack reader/writer works just like their JSON counterpart. There're just a few additional data types that belongs to this:
Binary
MsgPack directly supports bin format family
which directly maps to EDcDataEntry::Blob
:
// DataConfigTests/Private/DcTestBlurb.cpp
DC_TRY(Writer.WriteBlob({Bytes, 0}));
TArray<uint8> Arr = {1,2,3,4,5};
FDcMsgPackWriter Writer;
DC_TRY(Writer.WriteBlob(FDcBlobViewData::From(Arr)));
auto& Buf = Writer.GetMainBuffer();
FDcMsgPackReader Reader(FDcBlobViewData::From(Buf));
FDcBlobViewData Blob;
DC_TRY(Reader.ReadBlob(&Blob));
check(Blob.Num == 5);
check(FPlatformMemory::Memcmp(Arr.GetData(), Blob.DataPtr, Blob.Num) == 0);
Extension
MsgPack also supports ext format family
which is basically fixed size binary data with a header:
// DataConfigTests/Private/DcTestBlurb.cpp
FDcMsgPackWriter Writer;
DC_TRY(Writer.WriteFixExt2(1, {2, 3}));
auto& Buf = Writer.GetMainBuffer();
FDcMsgPackReader Reader(FDcBlobViewData::From(Buf));
uint8 Type;
FDcBytes2 Bytes;
DC_TRY(Reader.ReadFixExt2(&Type, &Bytes));
check(Type == 1);
check(Bytes.Data[0] == 2);
check(Bytes.Data[1] == 3);
MsgPack Serialize/Deserialize
MsgPack handlers also support multiple setup types:
// DataConfigCore/Public/DataConfig/Serialize/DcSerializerSetup.h
enum class EDcMsgPackSerializeType
{
Default,
StringSoftLazy, // Serialize Soft/Lazy references as string
InMemory, // Serialize pointer/enum/FName etc as underlying integer values
};
Persistent handlers
The Default
and StringSoftLazy
options would setup a set of handlers that behaves
like their JSON counterparts.
We have a "Property -> Json -> MsgPack -> Json -> Property" roundtrip test setup in DataConfig.Core.RoundTrip.Property_Json_MsgPack_Json_Property
test.
In Memory handlers
This is a special set of handlers that only makes sense for binary formats. For example pointers are serialized as memory addresses.
EDcDataEntry | Serialized |
---|---|
Name | [uint32, uint32, int32] |
Text | [void*, void*, uint32] |
ObjectReference, ClassReference | void* |
SoftObjectReference, SoftClassReference | FString or void* |
WeakObjectReference | [int32, int32] |
LazyObjectReference | <uuid as FIXEXT16> |
InterfaceReference | [void*, void*] |
Delegate | [int32, int32, (FName)[uint32, uint32, int32]] |
MulticastInlineDelegate, MulticastSparseDelegate | [(list of <Delegate>)] |
FieldPath | void* |
Enum | uint64 |
With these handlers all data types can be serialized. Note that serializing stuff as memory address isn't always what you want. These are provided as soft of a reference on how to access various data.
Property
The property system is at the heart of DataConfig: serializer read from property system, while deserializer writes into it. This is directly reflected in the types:
// DataConfigCore/Public/DataConfig/Serialize/DcSerializeTypes.h
struct DATACONFIGCORE_API FDcSerializeContext
{
// ...
FDcPropertyReader* Reader = nullptr;
FDcWriter* Writer = nullptr;
};
// DataConfigCore/Public/DataConfig/Deserialize/DcDeserializeTypes.h
struct DATACONFIGCORE_API FDcDeserializeContext
{
FDcReader* Reader = nullptr;
FDcPropertyWriter* Writer = nullptr;
};
Property Datum and Reader/Writer
Property reader/writer are usually constructed from a FDcPropertyDatum
, which is a "fat pointer" that represents an entry in the property system.
// DataConfigCore/Public/DataConfig/Property/DcPropertyDatum.h
struct DATACONFIGCORE_API FDcPropertyDatum
{
FFieldVariant Property;
void* DataPtr;
//...
}
Recall that DataConfig data model expands to a set of Read/Write
calls. In the example below we setup a FDcPropertyReader
with a simple struct and dump it out:
// DataConfigTests/Private/DcTestBlurb2.h
USTRUCT()
struct FDcTestExample2
{
GENERATED_BODY()
UPROPERTY() FString StrField;
UPROPERTY() FString StrArrField[3];
UPROPERTY() UDcBaseShape* InlineField;
UPROPERTY() UDcTestClass1* RefField;
// ...
};
// DataConfigTests/Private/DcTestBlurb2.cpp
FDcTestExample2 Value;
Value.MakeFixture();
FDcPropertyDatum Datum(&Value);
FDcPropertyReader Reader(Datum);
DcAutomationUtils::DumpToLog(&Reader);
The results are like this:
<StructRoot> 'DcTestExample2'
|---<Name> 'StrField'
|---<String> 'Foo'
|---<Name> 'StrArrField'
|---<ArrayRoot>
| |---<String> 'One'
| |---<String> 'Two'
| |---<String> 'Three'
|---<ArrayEnd>
|---<Name> 'InlineField'
|---<ClassRoot> 'DcShapeSquare'
| |---<Name> 'Radius'
| |---<Float> '2.000000'
| |---<Name> 'ShapeName'
| |---<Name> 'MyBox'
|---<ClassEnd> 'DcBaseShape'
|---<Name> 'RefField'
|---<ClassRoot> 'DcTestClass1'
| |---<ObjectReference> '337' 'DcTestClass1_0'
|---<ClassEnd> 'DcTestClass1'
<StructEnd> 'DcTestExample2'
Some caveats:
-
Note that
DcTestExample2::StrArrField
is dumped into a array. Note that DataConfig can't distinguish betweenFoo
andFoo[1]
that's a 1 element array. It would be read as a normal field. -
Note that
DcTestExample2::InlineField
is expanded andDcTestExample2::RefField
is read as an reference. This is determined byFDcPropertyConfig::ExpandObjectPredicate
, which by default expands:- Field marked with
UPROPERTY(Instanced)
- Class marked with
UCLASS(DefaultToInstanced, EditInlineNew)
- Field marked with
Configuring with FDcPropertyConfig
Property reader/writer accepts FDcPropertyConfig
class for customizing behaviors. By default we've implemented DcSkip
metadata that you can mark on given reader/writer.URPOPERTY()
and the field would be skipped by given reader/writer:
// DataConfigTests/Private/DcTestProperty3.h
USTRUCT()
struct FDcTestMeta1
{
GENERATED_BODY()
UPROPERTY(meta = (DcSkip)) int SkipField1;
};
Though that DcSkip
behavior is enabled by default, you can override this with a custom FPropertyConfig
instance. Here's an example of processing only fields with DcTestSerialize
meta:
// DataConfigTests/Private/DcTestProperty4.h
USTRUCT()
struct FDcTestSerializeMeta1
{
GENERATED_BODY()
UPROPERTY(meta = (DcTestSerialize)) int SerializedField;
UPROPERTY() int IgnoredField;
};
// DataConfigTests/Private/DcTestProperty4.cpp
FDcPropertyConfig Config;
// only process fields that has `DcTestSerialize` meta
Config.ProcessPropertyPredicate = FDcProcessPropertyPredicateDelegate::CreateLambda([](FProperty* Property)
{
const static FName TestSerializeMeta = FName(TEXT("DcTestSerialize"));
return DcPropertyUtils::IsEffectiveProperty(Property)
&& Property->HasMetaData(TestSerializeMeta);
});
Config.ExpandObjectPredicate = FDcExpandObjectPredicateDelegate::CreateStatic(DcPropertyUtils::IsSubObjectProperty);
Ctx.Reader->SetConfig(Config);
Pipe Property Handlers
- DcPropertyPipeSerializers.h
- DcPropertyPipeSerializers.cpp
- DcPropertyPipeDeserializers.h
- DcPropertyPipeDeserializers.cpp
There's a set of deserialize handlers in DcPropertyPipeHandlers
namespace. It's used to roundtripping property system objects.
Simply speaking it's equivalent to doing a FDcPipeVisitor
pipe visit.
// DataConfigTests/Private/DcTestBlurb.cpp
// these two blocks are equivalent
{
FDcPropertyReader Reader(FromDatum);
FDcPropertyWriter Writer(ToDatum);
FDcPipeVisitor RoundtripVisit(&Reader, &Writer);
DC_TRY(RoundtripVisit.PipeVisit());
}
{
FDcDeserializer Deserializer;
DcSetupPropertyPipeDeserializeHandlers(Deserializer);
FDcPropertyReader Reader(FromDatum);
FDcPropertyWriter Writer(ToDatum);
FDcDeserializeContext Ctx;
Ctx.Reader = &Reader;
Ctx.Writer = &Writer;
Ctx.Deserializer = &Deserializer;
DC_TRY(Ctx.Prepare());
DC_TRY(Deserializer.Deserialize(Ctx));
}
These are provided as a set of basis to for building custom property wrangling utils. See Field Renamer for example.
Extra
Alongside DataConfigCore
we have other modules DataConfigExtra
, DataConfigEngineExtra
and DataConfigEditorExtra
. It have self contained samples built on top of DataConfig framework.
Note that these are not intended to be integrated directly. You can take these as references when implementing custom features.
FColor Serialization/Deserialization
This example has been shown in previous chapter. It's also a benchmark use case for our custom deserialization logic:
// DataConfigExtra/Public/DataConfig/Extra/SerDe/DcSerDeColor.h
USTRUCT()
struct FDcExtraTestStructWithColor1
{
GENERATED_BODY()
UPROPERTY() FColor ColorField1;
UPROPERTY() FColor ColorField2;
};
// DataConfigExtra/Private/DataConfig/Extra/SerDe/DcSerDeColor.cpp
FString Str = TEXT(R"(
{
"ColorField1" : "#0000FFFF",
"ColorField2" : "#FF0000FF",
}
)");
FColor
is converted into a #RRGGBBAA
hex string. The corresponding handlers looks pretty mirrored.
// DataConfigExtra/Private/DataConfig/Extra/SerDe/DcSerDeColor.cpp
FDcResult HandlerColorDeserialize(FDcDeserializeContext& Ctx)
{
FDcPropertyDatum Datum;
DC_TRY(Ctx.Writer->WriteDataEntry(FStructProperty::StaticClass(), Datum));
FString ColorStr;
DC_TRY(Ctx.Reader->ReadString(&ColorStr));
FColor Color = FColor::FromHex(ColorStr);
FColor* ColorPtr = (FColor*)Datum.DataPtr;
*ColorPtr = Color;
return DcOk();
}
FDcResult HandlerColorSerialize(FDcSerializeContext& Ctx)
{
FDcPropertyDatum Datum;
DC_TRY(Ctx.Reader->ReadDataEntry(FStructProperty::StaticClass(), Datum));
FColor* ColorPtr = (FColor*)Datum.DataPtr;
DC_TRY(Ctx.Writer->WriteString(TEXT("#") + ColorPtr->ToHex()));
return DcOk();
}
Note how FDcPropertyReader::ReadDataEntry
and FDcPropertyWriter::WriteDataEntry
retrieves the next property as a FDcPropertyDatum
, which allows us to directly manipulate a FColor
pointer.
Base64 Blob Serialization/Deserialization
This demonstrates conversion betweenTArray<uint8>
and Base64 encoded strings in JSON:
// DataConfigExtra/Public/DataConfig/Extra/SerDe/DcSerDeBase64.h
USTRUCT()
struct FDcExtraTestStructWithBase64
{
GENERATED_BODY()
UPROPERTY(meta = (DcExtraBase64)) TArray<uint8> BlobField1;
UPROPERTY(meta = (DcExtraBase64)) TArray<uint8> BlobField2;
};
// DataConfigExtra/Private/DataConfig/Extra/SerDe/DcSerDeBase64.cpp
FString Str = TEXT(R"(
{
"BlobField1" : "dGhlc2UgYXJlIG15IHR3aXN0ZWQgd29yZHM=",
"BlobField2" : "",
}
)");
Note that we're tagging the BlobField
with (meta = (DcExtraBase64))
to explicitly show that we' want this member to be converted into Base64.
UE support arbitrary meta data in the meta = ()
segment. But beware that the meta data is only available when WITH_EDITORDATA
flag is defined. In predicate we check for this DcExtraBase64
like this:
// DataConfigExtra/Private/DataConfig/Extra/SerDe/DcSerDeBase64.cpp
EDcDeserializePredicateResult PredicateIsBase64Blob(FDcDeserializeContext& Ctx)
{
FArrayProperty* ArrayProperty = DcPropertyUtils::CastFieldVariant<FArrayProperty>(Ctx.TopProperty());
// check for only TArray<uint8>
if (ArrayProperty == nullptr)
return EDcDeserializePredicateResult::Pass;
if (!ArrayProperty->Inner->IsA<FByteProperty>())
return EDcDeserializePredicateResult::Pass;
return ArrayProperty->HasMetaData(TEXT("DcExtraBase64"))
? EDcDeserializePredicateResult::Process
: EDcDeserializePredicateResult::Pass;
}
Nested Arrays
UE property system has a limitation that you can't have a property of nested array:
// UHT would error out: The type 'TArray<int32>' can not be used as a value in a TArray
UPROPERTY() TArray<TArray<int>> Arr2D;
You can workaround this by wrap inner array in a struct. DataConfig however is flexible enough that you can serialize and deserialize nested JSON arrays into your data structures however you want:
// DataConfigExtra/Public/DataConfig/Extra/SerDe/DcSerDeNested.h
USTRUCT()
struct FDcExtraTestNested_Vec2
{
GENERATED_BODY()
UPROPERTY() TArray<FVector2D> Vec2ArrayField1;
UPROPERTY() TArray<FVector2D> Vec2ArrayField2;
};
// DataConfigExtra/Private/DataConfig/Extra/SerDe/DcSerDeNested.cpp
// equivalent fixture
FString Str = TEXT(R"(
{
"Vec2ArrayField1" :
[
{"X": 1.0, "Y": 2.0},
{"X": 2.0, "Y": 3.0},
{"X": 3.0, "Y": 4.0},
{"X": 4.0, "Y": 5.0},
],
"Vec2ArrayField2" :
[
[1.0, 2.0],
[2.0, 3.0],
[3.0, 4.0],
[4.0, 5.0],
],
}
)");
The next example we define a simple struct FDcGrid2D
and stores the dimension using metadata.
// DataConfigExtra/Public/DataConfig/Extra/SerDe/DcSerDeNested.h
USTRUCT()
struct FDcGrid2D
{
GENERATED_BODY()
UPROPERTY() TArray<int> Data;
};
USTRUCT()
struct FDcExtraTestNested_Grid
{
GENERATED_BODY()
UPROPERTY(meta=(DcWidth = 2, DcHeight = 2)) FDcGrid2D GridField1;
UPROPERTY(meta=(DcWidth = 3, DcHeight = 4)) FDcGrid2D GridField2;
};
// DataConfigExtra/Private/DataConfig/Extra/SerDe/DcSerDeNested.cpp
// equivalent fixture
FString Str = TEXT(R"(
{
"GridField1" :
[
[1,2],
[3,4],
],
"GridField2" :
[
[ 1, 2, 3],
[ 4, 5, 6],
[ 7, 8, 9],
[10,11,12],
],
}
)");
When data dimension doesn't match a diagnostic would be reported:
# DataConfig Error: Nested Grid2D height mismatch, Expect '2'
- [WideCharDcJsonReader] --> <in-memory>7:22
5 | [1,2],
6 | [1,2],
7 | [1,2],
| ^
8 | ],
9 | }
Writer API Alternatives
Previously we're deserializing FColor
by writing into its member fields separately, which is a bit cumbersome. In this case DataConfig do support better alternatives.
Since we know that FColor
is POD type we can construct one by filling in correct bit pattern. In this case FDcPropertyWriter
allow struct property to be coerced from a blob:
// DataConfigExtra/Private/DataConfig/Extra/SerDe/DcSerDeColor.cpp
template<>
FDcResult TemplatedWriteColorDispatch<EDcColorDeserializeMethod::WriteBlob>(const FColor& Color, FDcDeserializeContext& Ctx)
{
return Ctx.Writer->WriteBlob({
(uint8*)&Color, // treat `Color` as opaque blob data
sizeof(FColor)
});
}
Alternatively we can get FProperty
and data pointer in place and setting the value through Unreal's FProperty
API:
// DataConfigExtra/Private/DataConfig/Extra/SerDe/DcSerDeColor.cpp
template<>
FDcResult TemplatedWriteColorDispatch<EDcColorDeserializeMethod::WriteDataEntry>(const FColor& Color, FDcDeserializeContext& Ctx)
{
FDcPropertyDatum Datum;
DC_TRY(Ctx.Writer->WriteDataEntry(FStructProperty::StaticClass(), Datum));
Datum.CastFieldChecked<FStructProperty>()->CopySingleValue(Datum.DataPtr, &Color);
return DcOk();
}
Note that we already know that Datum.DataPtr
points to a allocated FColor
instance. Thus we can simply cast it into a FColor*
and directly manipulate the pointer.
// DataConfigExtra/Private/DataConfig/Extra/SerDe/DcSerDeColor.cpp
template<>
FDcResult TemplatedWriteColorDispatch<EDcColorDeserializeMethod::WritePointer>(const FColor& Color, FDcDeserializeContext& Ctx)
{
FDcPropertyDatum Datum;
DC_TRY(Ctx.Writer->WriteDataEntry(FStructProperty::StaticClass(), Datum));
FColor* ColorPtr = (FColor*)Datum.DataPtr;
*ColorPtr = Color; // deserialize by assignment
return DcOk();
}
Note that these techniques also applies on serialization:
// DataConfigExtra/Private/DataConfig/Extra/SerDe/DcSerDeColor.cpp
FDcResult HandlerColorSerialize(FDcSerializeContext& Ctx)
{
FDcPropertyDatum Datum;
DC_TRY(Ctx.Reader->ReadDataEntry(FStructProperty::StaticClass(), Datum));
FColor* ColorPtr = (FColor*)Datum.DataPtr;
DC_TRY(Ctx.Writer->WriteString(TEXT("#") + ColorPtr->ToHex()));
return DcOk();
}
JsonConverter in DataConfig
UE comes with a handy module JsonUtilities
that handles conversion between USTRUCT
s and JSON. In this example we've implemented similar functionalities that behaves almost identical to stock FJsonConverter
.
// DataConfigExtra/Private/DataConfig/Extra/Types/DcJsonConverter.cpp
FString Str = TEXT(R"(
{
"strField" : "Foo",
"nestField" : {
"strArrayField" : [
"One",
"Two",
"Three"
],
"strIntMapField" : {
"One": 1,
"Two": 2,
"Three": 3
}
},
"intField" : 253,
"boolField" : true
}
)");
{
FDcTestJsonConverter1 Lhs;
bool LhsOk = DcExtra::JsonObjectStringToUStruct(Str, &Lhs);
FDcTestJsonConverter1 Rhs;
bool RhsOk = FJsonObjectConverter::JsonObjectStringToUStruct(Str, &Rhs);
}
{
FString Lhs;
bool LhsOk = DcExtra::UStructToJsonObjectString(Data, Lhs);
FString Rhs;
bool RhsOk = FJsonObjectConverter::UStructToJsonObjectString(Data, Rhs);
}
DcExtra::JsonObjectStringToUStruct()
body is trivia as it delegates most of the work to DcDeserializer
:
// DataConfigExtra/Private/DataConfig/Extra/Types/DcJsonConverter.cpp
bool JsonObjectReaderToUStruct(FDcReader* Reader, FDcPropertyDatum Datum)
{
FDcResult Ret = [&]() -> FDcResult {
using namespace JsonConverterDetails;
LazyInitializeDeserializer();
FDcPropertyWriter Writer(Datum);
FDcDeserializeContext Ctx;
Ctx.Reader = Reader;
Ctx.Writer = &Writer;
Ctx.Deserializer = &Deserializer.GetValue();
DC_TRY(Ctx.Prepare());
DC_TRY(Deserializer->Deserialize(Ctx));
return DcOk();
}();
if (!Ret.Ok())
{
DcEnv().FlushDiags();
return false;
}
else
{
return true;
}
}
The serializing function DcExtra::UStructToJsonObjectString()
needs some customization as default DcJsonWriter
and DcSerializer
handlers behaves a bit different against stock FDcJsonConverter
:
- It serialize field names as
camelCase
. - It uses platform dependent line endings, that is
\r\n
on Windows. - It have subtle new line breaking rules on nested array and object, and on spacing around
:
token.
The good news is that one can customize these behaviors with DataConfig to match it:
// DataConfigExtra/Public/DataConfig/Extra/Types/DcJsonConverter.h
template<typename InStructType>
static bool UStructToJsonObjectString(const InStructType& InStruct, FString& OutJsonString)
{
static FDcJsonWriter::ConfigType _JSON_CONVERTER_CONFIG = []
{
FDcJsonWriter::ConfigType Config = FDcJsonWriter::DefaultConfig;
Config.IndentLiteral = TEXT("\t");
Config.LineEndLiteral = LINE_TERMINATOR;
Config.LeftSpacingLiteral = TEXT("");
Config.bNestedArrayStartsOnNewLine = false;
Config.bNestedObjectStartsOnNewLine = true;
return Config;
}();
FDcJsonWriter Writer(_JSON_CONVERTER_CONFIG);
...
}
// DataConfigExtra/Private/DataConfig/Extra/Types/DcJsonConverter.cpp
static FDcResult HandlerStructRootSerializeCamelCase(FDcSerializeContext& Ctx)
{
...
else if (CurPeek == EDcDataEntry::Name)
{
FName Value;
DC_TRY(Ctx.Reader->ReadName(&Value));
DC_TRY(Ctx.Writer->WriteString(FJsonObjectConverter::StandardizeCase(Value.ToString())));
}
}
We aim to support flexible serialization and formatting behaviors without modifying DataConfigCore
code:
AnyStruct
This is an intermediate example that takes advantage of the flexibility provided by the property system. FDcAnyStruct
is a struct that stores a heap allocated USTRUCT
of any type while maintaining value semantic on itself. If you're familiar with the concept of variant type, just think of it as a variant type that supports all USTRUCT
:
// DataConfigExtra/Private/DataConfig/Extra/SerDe/DcSerDeAnyStruct.cpp
// instantiate from heap allocated structs
FDcAnyStruct Any1 = new FDcExtraTestSimpleStruct1();
Any1.GetChecked<FDcExtraTestSimpleStruct1>()->NameField = TEXT("Foo");
// supports moving
FDcAnyStruct Any2 = MoveTemp(Any1);
check(!Any1.IsValid());
check(Any2.GetChecked<FDcExtraTestSimpleStruct1>()->NameField == TEXT("Foo"));
Any2.Reset();
// supports shared referencing
Any2 = new FDcExtraTestSimpleStruct2();
Any2.GetChecked<FDcExtraTestSimpleStruct2>()->StrField = TEXT("Bar");
Any1 = Any2;
check(Any1.DataPtr == Any2.DataPtr);
check(Any1.StructClass == Any2.StructClass);
We then implemented conversion logic between FDcAnyStruct
and JSON:
// DataConfigExtra/Public/DataConfig/Extra/SerDe/DcSerDeAnyStruct.h
USTRUCT()
struct FDcExtraTestWithAnyStruct1
{
GENERATED_BODY()
UPROPERTY() FDcAnyStruct AnyStructField1;
UPROPERTY() FDcAnyStruct AnyStructField2;
UPROPERTY() FDcAnyStruct AnyStructField3;
};
// DataConfigExtra/Private/DataConfig/Extra/SerDe/DcSerDeAnyStruct.cpp
FString Str = TEXT(R"(
{
"AnyStructField1" : {
"$type" : "DcExtraTestSimpleStruct1",
"NameField" : "Foo"
},
"AnyStructField2" : {
"$type" : "DcExtraTestStructWithColor1",
"ColorField1" : "#0000FFFF",
"ColorField2" : "#FF0000FF"
},
"AnyStructField3" : null
}
)");
Note how the custom FColor <-> "#RRGGBBAA"
conversion recursively works within FDcAnyStruct
. This should be a good starting point for you to implement your own nested variant types and containers. For more details refer to the implementation of HandlerDcAnyStruct[Serialize/Deserialize]
.
Inline Struct
In AnyStruct we implemented FDcAnyStruct
that can store arbitrary heap allocated USTRUCT
. A shortcoming of this approach is that it introduces overhead of heap allocated memory, which also have worse cache locality comparing to stack allocated structs.
In this example we implemented a series of structs FDcInlineStruct64/FDcInlineStruct128/FDcInlineStruct256/FDcInlineStruct512
which stores a USTRUCT inline. Think it as a small buffer optimized version of FDcAnyStruct
. These can also be used as a cheap alternative to UCLASS based polymorphism.
// DataConfigExtra/Private/DataConfig/Extra/SerDe/DcSerDeInlineStruct.cpp
// stack allocated usage
FDcInlineStruct64 Inline1;
Inline1.Emplace<FColor>(255, 0, 0, 255);
UTEST_TRUE("Inline Struct", *Inline1.GetChecked<FColor>() == FColor::Red);
// support copy
FDcInlineStruct64 Inline2 = Inline1;
UTEST_TRUE("Inline Struct", *Inline2.GetChecked<FColor>() == FColor::Red);
Serialization handlers for inline structs are similar to any struct ones:
// DataConfigExtra/Public/DataConfig/Extra/SerDe/DcSerDeInlineStruct.h
USTRUCT()
struct FDcExtraTestWithInlineStruct1
{
GENERATED_BODY()
UPROPERTY() FDcInlineStruct64 InlineField1;
UPROPERTY() FDcInlineStruct64 InlineField2;;
};
// DataConfigExtra/Private/DataConfig/Extra/SerDe/DcSerDeInlineStruct.cpp
FString Str = TEXT(R"(
{
"InlineField1" : {
"$type" : "DcExtraTestSimpleStruct1",
"NameField" : "Foo"
},
"InlineField2" : {
"$type" : "DcExtraTestStructWithColor1",
"ColorField1" : "#0000FFFF",
"ColorField2" : "#FF0000FF"
}
}
)");
Note that one limitation with inline structs the USTRUCT get put in must have smaller size than the inline struct storage. Deserialize handlers would check for these cases:
LogDataConfigCore: Display: # DataConfig Error: Inline struct too big: BufSize '56', Struct 'DcExtraTestStruct128' Size '128'
LogDataConfigCore: Display: - [WideCharDcJsonReader] --> <in-memory>5:38
3 | {
4 | "InlineField1" : {
5 | "$type" : "DcExtraTestStruct128",
| ^
6 | "NameField" : "Foo"
7 | },
LogDataConfigCore: Display: - [DcPropertyWriter] Writing property: (FDcExtraTestWithInlineStruct1)$root.(FDcInlineStruct64)InlineField1
LogDataConfigCore: Display: - [DcSerializer]
InstancedStruct
This example only works with UE5.0+
Starting with UE 5.0 there's a new plugin StructUtils featuring a struct type called FInstancedStruct. It's shares the same idea to previous AnyStruct example, while it has proper asset serialization logic and editor support.
Given a struct hierarchy like this:
// DataConfigExtra/Public/DataConfig/Extra/Types/DcExtraTestFixtures.h
USTRUCT(BlueprintType)
struct FDcStructShapeBase
{
GENERATED_BODY()
UPROPERTY(EditAnywhere) FName ShapeName;
};
USTRUCT(BlueprintType)
struct FDcStructShapeRectangle : public FDcStructShapeBase
{
GENERATED_BODY()
UPROPERTY(EditAnywhere) float Height;
UPROPERTY(EditAnywhere) float Width;
};
USTRUCT(BlueprintType)
struct FDcStructShapeCircle : public FDcStructShapeBase
{
GENERATED_BODY()
UPROPERTY(EditAnywhere) float Radius;
};
You can use FInstancedStruct
specified with a BaseStruct
meta to reference to a polymorphism instance:
// DataConfigEngineExtra5/Public/DataConfig/EngineExtra/SerDe/DcSerDeInstancedStruct.h
UPROPERTY(EditAnywhere, meta = (BaseStruct = "/Script/DataConfigExtra.DcStructShapeBase"))
FInstancedStruct InstancedStruct1;
UPROPERTY(EditAnywhere, meta = (BaseStruct = "DcStructShapeBase"))
FInstancedStruct InstancedStruct2;
The best part is that the editor is also working as intended:
Prior to this you'll need to setup Sub Objects for similar behavior, which costs unnecessary overhead.
FInstancedStruct
can also be serialized to and from JSON with DataConfig:
// DataConfigExtra5/Public/DataConfig/EngineExtra/SerDe/DcSerDeInstancedStruct.h
USTRUCT()
struct FDcEngineExtra5InstancedStruct1
{
GENERATED_BODY()
UPROPERTY() FInstancedStruct InstancedStruct1;
UPROPERTY() FInstancedStruct InstancedStruct2;
UPROPERTY() FInstancedStruct InstancedStruct3;
};
// DataConfigExtra5/Private/DataConfig/EngineExtra/SerDe/DcSerDeInstancedStruct.cpp
FString Str = TEXT(R"(
{
"InstancedStruct1" : {
"$type" : "DcExtraTestSimpleStruct1",
"NameField" : "Foo"
},
"InstancedStruct2" : {
"$type" : "DcExtraTestStructWithColor1",
"ColorField1" : "#0000FFFF",
"ColorField2" : "#FF0000FF"
},
"InstancedStruct3" : null
}
)");
Note how the custom FColor <-> "#RRGGBBAA"
conversion recursively works within FInstancedStruct
.
Field Renamer
DataConfig can also be used to author one-off utility. In this example we implemented DcExtra::DeserializeStructRenaming()
that copies data between structural identical data structures, while renaming field names by a user function.
// DataConfigExtra/Private/DataConfig/Extra/Deserialize/DcDeserializeRenameStructFieldNames.cpp
// struct equivelent to this:
FString Str = TEXT(R"(
{
"FromName1" : "Foo",
"FromStructSet1" :
[
{
"FromStr1" : "One",
"FromInt1" : 1,
},
{
"FromStr1" : "Two",
"FromInt1" : 2,
}
]
}
)");
// ... deserialize with a functor renaming `FromXXX` to `ToXXX`:
UTEST_OK("...", DcExtra::DeserializeStructRenaming(
FromDatum, ToDatum, FDcExtraRenamer::CreateLambda([](const FName& FromName){
FString FromStr = FromName.ToString();
if (FromStr.StartsWith(TEXT("From")))
return FName(TEXT("To") + FromStr.Mid(4));
else
return FromName;
})));
// ... results into a struct equivelent to this:
FString Str = TEXT(R"(
{
"ToName1" : "Foo",
"ToStructSet1" :
[
{
"ToStr1" : "One",
"ToInt1" : 1,
},
{
"ToStr1" : "Two",
"ToInt1" : 2,
}
]
}
)");
This takes advantage of the DcPropertyPipeHandlers
that simply do verbatim data piping.
The gist is that you should consider DataConfig an option when working with batch data processing within Unreal Engine. We are trying to provide tools to support these use cases.
Access property by path
This example demonstrates that FReader/FWriter
can be used standalone, without FDcSerializer/FDcDeserializer
.
UE built-in module PropertyPath
allow accessing nested object properties by a path like Foo.Bar.Baz
:
// DataConfigExtra/Private/DataConfig/Extra/Types/DcPropertyPathAccess.cpp
FString Str;
UTEST_TRUE("...", PropertyPathHelpers::GetPropertyValue(Outer, TEXT("StructRoot.Middle.InnerMost.StrField"), Str));
UTEST_TRUE("...", Str == TEXT("Foo"));
UTEST_TRUE("...", PropertyPathHelpers::SetPropertyValue(Outer, TEXT("StructRoot.Middle.InnerMost.StrField"), FString(TEXT("Bar"))));
UTEST_TRUE("...", Outer->StructRoot.Middle.InnerMost.StrField == TEXT("Bar"));
We implemented a pair of methods DcExtra::GetDatumPropertyByPath/SetDatumPropertyByPath
with FDcPropertyReader
:
// DataConfigExtra/Private/DataConfig/Extra/Types/DcPropertyPathAccess.cpp
UTEST_TRUE("...", CheckStrPtr(GetDatumPropertyByPath<FString>(FDcPropertyDatum(Outer), "StructRoot.Middle.InnerMost.StrField"), TEXT("Foo")));
UTEST_TRUE("...", CheckStrPtr(GetDatumPropertyByPath<FString>(FDcPropertyDatum(Outer), "StructRoot.Arr.0.StrField"), TEXT("Bar0")));
UTEST_TRUE("...", CheckStrPtr(GetDatumPropertyByPath<FString>(FDcPropertyDatum(Outer), "StructRoot.Arr.1.StrField"), TEXT("Bar1")));
UTEST_TRUE("...", CheckStrPtr(GetDatumPropertyByPath<FString>(FDcPropertyDatum(Outer), "StructRoot.NameMap.FooKey.StrField"), TEXT("FooValue")));
UTEST_TRUE("...", SetDatumPropertyByPath<FString>(FDcPropertyDatum(Outer), "StructRoot.Middle.InnerMost.StrField", TEXT("AltFoo")));
UTEST_TRUE("...", SetDatumPropertyByPath<FString>(FDcPropertyDatum(Outer), "StructRoot.Arr.0.StrField", TEXT("AltBar0")));
UTEST_TRUE("...", SetDatumPropertyByPath<FString>(FDcPropertyDatum(Outer), "StructRoot.Arr.1.StrField", TEXT("AltBar1")));
UTEST_TRUE("...", SetDatumPropertyByPath<FString>(FDcPropertyDatum(Outer), "StructRoot.NameMap.FooKey.StrField", TEXT("AltFooValue")));
Comparing to PropertyPathHelpers
these new ones support Array
and Map
, and support USTRUCT
roots. We're missing some features like expanding weak/lazy object references but it should be easy to implement.
Remember that we have bundled JSON/MsgPack reader/writers that can also be used standalone.
Deserialize From SQLite Query
Unreal Engine bundles SQLite3 in SQLiteCore plugin and bundled with minimal C++ abstractions. In this example we implemented LoadStructArrayFromSQLite
to load a SQLite query into a TArray
of structs:
// DataConfigExtra/Public/DataConfig/Extra/Misc/DcSqlite.h
template<typename TStruct>
FDcResult LoadStructArrayFromSQLite(FSQLiteDatabase* Db, const TCHAR* Query, TArray<TStruct>& Arr)
// ...
With this method we can easily turn SQLite query results into structs that's easy to manipulate with:
// DataConfig/Source/DataConfigExtra/Private/DataConfig/Extra/Misc/DcSqlite.cpp
/// data fixture
FString Statement = TEXT("CREATE TABLE users (id INTEGER NOT NULL,name TEXT, title TEXT)");
bSuccess &= TestDb.Execute(*Statement);
Statement = TEXT("INSERT INTO users (id, name, title) VALUES (1, 'John', 'Manager')");
bSuccess &= TestDb.Execute(*Statement);
Statement = TEXT("INSERT INTO users (id, name, title) VALUES (2, 'Mark', 'Engineer')");
bSuccess &= TestDb.Execute(*Statement);
Statement = TEXT("INSERT INTO users (id, name, title) VALUES (3, 'Bob', 'Engineer')");
bSuccess &= TestDb.Execute(*Statement);
Statement = TEXT("INSERT INTO users (id, name, title) VALUES (4, 'Mike', 'QA')");
// execute query and deserialize
TArray<FDcExtraTestUser> Arr;
UTEST_OK("Extra Sqlite", DcExtra::LoadStructArrayFromSQLite(
&TestDb,
TEXT("SELECT * FROM users WHERE title == 'Engineer' ORDER BY id"),
Arr
));
// equivalent fixture
FString Fixture = TEXT(R"(
[
{
"Id" : 2,
"Name" : "Mark",
"Title" : "Engineer"
},
{
"Id" : 3,
"Name" : "Bob",
"Title" : "Engineer"
}
]
)");
For this to work we'll need to implement FSqliteReader
which implements FDcReader
API so it can be consumed by deserializer. The cool thing is that FSqliteReader
works very well with SQLite's step API and can wrap SQLite error reporting into diagnostics that DataConfig can report.
NDJSON
NDJSON is a popular extension to JSON that stores 1 JSON object per line. DataConfig's JSON parser and writer is flexible enough to easily support this use case.
// DataConfigExtra/Public/DataConfig/Extra/Misc/DcNDJSON.h
template<typename TStruct>
DATACONFIGEXTRA_API FDcResult LoadNDJSON(const TCHAR* Str, TArray<TStruct>& Arr)
// ...
template<typename TStruct>
DATACONFIGEXTRA_API FDcResult SaveNDJSON(const TArray<TStruct>& Arr, FString& OutStr)
// ...
With this method we can load a NDJSON string into a struct array and later serialize it back to NDJSON.
// DataConfig/Source/DataConfigExtra/Private/DataConfig/Extra/Misc/DcNDJSON.cpp
FString Str = TEXT(R"(
{ "Name" : "Foo", "Id" : 1, "Type" : "Alpha" }
{ "Name" : "Bar", "Id" : 2, "Type" : "Beta" }
{ "Name" : "Baz", "Id" : 3, "Type" : "Gamma" }
)");
UTEST_OK("Extra NDJSON", LoadNDJSON(*Str, Dest));
FString SavedStr;
UTEST_OK("Extra NDJSON", SaveNDJSON(Dest, SavedStr));
Note that our parser supports common extension to JSON:
- Allow C Style comments, i.e
/* block */
and// line
. - Allow trailing comma, i.e
[1,2,3,],
. - Allow non object root. You can put a list as the root, or even string, numbers.
Root Object
Many modern JSON like data markup languages allows root level object and arrays, i.e omitting top level braces. This can be done in DataConfig with custom serialize handlers.
// DataConfigExtra/Private/DataConfig/Extra/SerDe/DcSerDeRoot.cpp
// root object
FString Str = TEXT(R"(
"Name" : "Foo",
"Id" : 253,
"Type" : "Beta"
)");
// equivalent fixture
FDcExtraSimpleStruct Expect;
Expect.Name = TEXT("Foo");
Expect.Id = 253;
Expect.Type = EDcExtraTestEnum1::Beta;
// DataConfigExtra/Private/DataConfig/Extra/SerDe/DcSerDeRoot.cpp
// root list
FString Str = TEXT(R"(
"Alpha",
"Beta",
"Gamma"
)");
// equivalent fixture
TArray<EDcExtraTestEnum1> Expect = {
EDcExtraTestEnum1::Alpha,
EDcExtraTestEnum1::Beta,
EDcExtraTestEnum1::Gamma};
This can be a tiny QOL improvement for manually authoring JSON data.
Extra Module Setups
Unreal Engine Modules is how the engine handles its C++ code physical design. You'll need to be pretty familiar with the system to scale up your C++ project.
We split extra samples in two modules. The first is DataConfigExtra
which does not depend on Engine/UnrealEd
module. It can be built along program target. DataConfigExtra
is basically a set of C++ source files bundled and there's no special setup for it. The key setup is to set ModuleRules.bRequiresImplementModule
to be false
:
// DataConfigExtra/DataConfigExtra.Build.cs
public class DataConfigExtra : ModuleRules
{
public DataConfigExtra(ReadOnlyTargetRules Target) : base(Target)
{
bRequiresImplementModule = false;
Type = ModuleType.CPlusPlus;
PCHUsage = PCHUsageMode.UseExplicitOrSharedPCHs;
PublicDependencyModuleNames.AddRange(
new string[] {
"DataConfigCore",
"Core",
"CoreUObject",
// ...
});
}
}
The other module is DcEngineExtraModule
, which is a more conventional runtime module that introduces DataConfig dependency. We also put samples that depends on Engine
and other gameplay system in here.
Most of integration code is in IModuleInterface::StartUpModule/ShutdownModule()
.
// DataConfigEngineExtra/Private/DcEngineExtraModule.cpp
void FDcEngineExtraModule::StartupModule()
{
// ...
DcRegisterDiagnosticGroup(&DcDExtra::Details);
DcRegisterDiagnosticGroup(&DcDEngineExtra::Details);
DcStartUp(EDcInitializeAction::Minimal);
DcEnv().DiagConsumer = MakeShareable(new FDcMessageLogDiagnosticConsumer());
// ...
}
void FDcEngineExtraModule::ShutdownModule()
{
DcShutDown();
// ...
}
Here's a checklist for integration:
- Register additional diagnostics early.
- Call
DcStartUp()/DcShutDonw()
pair. - Register custom diagnostic consumer.
FDcMessageLogDiagnosticConsumer
is an example of redirecting diagnostics to the UE Message Log window with its own category.
Dump Asset To Log
In this example we added a Dump Asset To Log
menu entry. After enabling it you can see it on the context menu on all asset types:
On clicking it would use the pretty print writer to dump the asset into Output Log:
This is implemented as an editor menu extender:
// DataConfigExtra/Private/DataConfig/EditorExtra/Editor/DcEditorDumpAssetToLog.cpp
TSharedRef<FExtender> DumpAssetToLogExtender(const TArray<FAssetData>& SelectedAssets)
{
// ...
Extender->AddMenuExtension("GetAssetActions", EExtensionHook::After, TSharedPtr<FUICommandList>(),
FMenuExtensionDelegate::CreateLambda([Asset](FMenuBuilder& MenuBuilder)
{
MenuBuilder.AddMenuEntry(
// ...
FUIAction(
FExecuteAction::CreateLambda([Asset]{
if (UBlueprint* Blueprint = Cast<UBlueprint>(Asset.GetAsset()))
{
// dump BP class generated class CDO as it makes more sense
DcAutomationUtils::DumpToLog(FDcPropertyDatum(Blueprint->GeneratedClass->ClassDefaultObject));
}
else
{
DcAutomationUtils::DumpToLog(FDcPropertyDatum(Asset.GetAsset()));
}
}),
FCanExecuteAction()
)
);
}));
}
// ...
When asset is a Blueprint we dump its CDO instead as it makes more sense.
Blueprint Serialization/Deserialization
The Property System is so powerful that you can create new Blueprint Class/Struct, which is equivalent to C++ UCLASS/USTRUCT
to some extents. In this example we'll show how to handle these in DataConfig.
The whole blueprint stuff depends on Engine
module. This is also why we put related code into DataConfigEngineExtra
module.
Blueprint Class and Object references
Blueprint class are stored within UBlueprint
typed assets. Note that we automatically unwrap the container in handlers.
//DataConfigEngineExtra/Public/DataConfig/EngineExtra/SerDe/DcSerDeBlueprint.h
USTRUCT()
struct FDcEngineExtraTestStructWithBPClass
{
GENERATED_BODY()
UPROPERTY() TSubclassOf<UDcTestBPClassBase> ClassField1;
UPROPERTY() TSubclassOf<UDcTestBPClassBase> ClassField2;
UPROPERTY() TSubclassOf<UDcTestBPClassBase> ClassField3;
};
USTRUCT()
struct FDcEngineExtraTestStructWithBPInstance
{
GENERATED_BODY()
UPROPERTY() UDcTestBPClassBase* InstanceField1;
UPROPERTY() UDcTestBPClassBase* InstanceField2;
UPROPERTY() UDcTestBPClassBase* InstanceField3;
};
//DataConfigEngineExtra/Private/DataConfig/EngineExtra/SerDe/DcSerDeBlueprint.cpp
FString Str = TEXT(R"(
{
"ClassField1" : null,
"ClassField2" : "DcTestNativeDerived1",
"ClassField3" : "/DataConfig/DcFixture/DcTestBlueprintClassBeta"
}
)");
FString Str = TEXT(R"(
{
"InstanceField1" : null,
"InstanceField2" : "/DataConfig/DcFixture/DcTestBlueprintInstanceAlpha",
"InstanceField3" : "/DataConfig/DcFixture/DcTestBlueprintInstanceBeta"
}
)");
Blueprint Class Instance
In this example we roundtrip a Blueprint class instance from JSON:
//DataConfigEngineExtra/Private/DataConfig/EngineExtra/SerDe/DcSerDeBlueprint.cpp
FString Str = TEXT(R"(
{
"StrField" : "Foo",
"BPEnumField" : "Baz",
"IntField" : 345
}
)");
You can also reference on how to handle Blueprint enum in this example. It need some special care to convert between int value and the descriptive text set within the editor.
Blueprint Struct Instance
Finally we're combined FColor
, FDcAnyStruct
and Blueprint struct into one single example:
//DataConfigEngineExtra/Private/DataConfig/EngineExtra/SerDe/DcSerDeBlueprint.cpp
FString Str = TEXT(R"(
{
"AnyStructField1" : {
"$type" : "/DataConfig/DcFixture/DcTestBlueprintStructWithColor",
"NameField" : "Foo",
"StrField" : "Bar",
"IntField" : 123,
"ColorField" : "#FF0000FF"
}
}
)");
Under the hood Blueprint struct mangles its field names. The struct above dumps to something like this:
-----------------------------------------
# Datum: 'UserDefinedStruct', 'DcTestBlueprintStructWithColor'
<StructRoot> 'DcTestBlueprintStructWithColor'
|---<Name> 'NameField_5_97BFF114405C1934C2F33E8668BF1652'
|---<Name> 'Foo'
|---<Name> 'StrField_9_FAA71EFE4896F4E6B1478B9C13B2CE52'
|---<String> 'Bar'
|---<Name> 'IntField_11_3BC7CB0F42439CE2196F7AA82A1AC374'
|---<Int32> '123'
|---<Name> 'ColorField_14_F676BCF245B2977B678B65A8216E94EB'
|---<StructRoot> 'Color'
| |---<Name> 'B'
| |---<UInt8> '0'
| |---<Name> 'G'
| |---<UInt8> '0'
| |---<Name> 'R'
| |---<UInt8> '255'
| |---<Name> 'A'
| |---<UInt8> '255'
|---<StructEnd> 'Color'
<StructEnd> 'DcTestBlueprintStructWithColor'
-----------------------------------------
The good news is that DataConfig already got this covered.
Gameplay Tag Serialization/Deserialization
GameplayTags is a built-in runtime module that implements hierarchical tags.
In this example we have roundtrip handlers for FGameplayTag
and FGameplayTagContainer
.
// DataConfigEngineExtra/Public/DataConfig/EngineExtra/SerDe/DcSerDeGameplayTags.h
USTRUCT()
struct FDcEngineExtraTestStructWithGameplayTag1
{
GENERATED_BODY()
UPROPERTY() FGameplayTag TagField1;
UPROPERTY() FGameplayTag TagField2;
};
// DataConfigEngineExtra/Private/DataConfig/EngineExtra/SerDe/DcSerDeGameplayTags.cpp
FString Str = TEXT(R"(
{
"TagField1" : null,
"TagField2" : "DataConfig.Foo.Bar"
}
)");
FGameplayTagContainer
converts to a list of strings:
// DataConfigEngineExtra/Public/DataConfig/EngineExtra/Deserialize/DcDeserializeGameplayTags.h
USTRUCT()
struct FDcEngineExtraTestStructWithGameplayTag2
{
GENERATED_BODY()
UPROPERTY() FGameplayTagContainer TagContainerField1;
UPROPERTY() FGameplayTagContainer TagContainerField2;
};
// DataConfigEngineExtra/Private/DataConfig/EngineExtra/Deserialize/DcDeserializeGameplayTags.cpp
FString Str = TEXT(R"(
{
"TagContainerField1" : [],
"TagContainerField2" : [
"DataConfig.Foo.Bar",
"DataConfig.Foo.Bar.Baz",
"DataConfig.Tar.Taz",
]
}
)");
Note that gameplay tag parsing has error reporting built-in. In this case we can pipe it into our diagnostic:
// DataConfigEngineExtra/Private/DataConfig/EngineExtra/Deserialize/DcDeserializeGameplayTags.cpp
static FDcResult _StringToGameplayTag(FDcDeserializeContext& Ctx, const FString& Str, FGameplayTag* OutTagPtr)
{
FString FixedString;
FText Err;
if (!FGameplayTag::IsValidGameplayTagString(Str, &Err, &FixedString))
{
return DC_FAIL(DcDEngineExtra, InvalidGameplayTagStringFixErr)
<< Str << FixedString << Err;
}
//...
}
In case of a invalid tag it would report the reason and fixed string:
# DataConfig Error: Invalid Gameplay Tag String, Actual: 'DataConfig.Invalid.Tag.', Fixed: 'DataConfig.Invalid.Tag', Error: 'Tag ends with .'
- [JsonReader] --> <in-memory>5:4
3 | {
4 | "TagField1" : null,
5 | "TagField2" : "DataConfig.Invalid.Tag."
| ^^^^^^^^^^^^^^^^^^^^^^^^^
6 | }
7 |
- [PropertyWriter] Writing property: (FDcEngineExtraTestStructWithGameplayTag1)$root.(FGameplayTag)TagField2
[C:\DevUE\UnrealEngine\Engine\Source\Developer\MessageLog\Private\Model\MessageLogListingModel.cpp(73)]
Deserialize Gameplay Abilities
We'll conclude with a concrete user story: populating GameplayAbility
and GameplayEffect
blueprint from JSON file.
Gameplay Ability System is a built-in plugin for building data driven abilities. Users are expected to derived and modify GameplayAbility
and GameplayEffect
blueprint for custom logic.
Given a JSON like this:
// DataConfig/Tests/Fixture_AbilityAlpha.json
{
/// Tags
"AbilityTags" : [
"DataConfig.Foo.Bar",
"DataConfig.Foo.Bar.Baz",
],
"CancelAbilitiesWithTag" : [
"DataConfig.Foo.Bar.Baz",
"DataConfig.Tar.Taz",
],
/// Costs
"CostGameplayEffectClass" : "/DataConfig/DcFixture/DcTestGameplayEffectAlpha",
/// Advanced
"ReplicationPolicy" : "ReplicateYes",
"InstancingPolicy" : "NonInstanced",
}
Right click on a GameplayAbility
blueprint asset and select Load From JSON
, then select this file and confirm. It would correctly populate the fields with the values in JSON, as seen in the pic below:
Most of the logic is in DataConfig/EditorExtra/Deserialize/DcDeserializeGameplayAbility.cpp
:
- The context menu is added from
GameplayAbilityEffectExtender
. There's another handy item namedDump To Log
which dumps any blueprint CDO into the log. - DataConfig deserializer is setup in
LazyInitializeDeserializer()
. We added custom logic for deserializingFGameplayAttribute
from a string likeDcTestAttributeSet.Mana
. - We also reused many methods from previous examples to support
FGameplayTag
deserialization and Blueprint class look up by path.
Blueprint Nodes
We have a set of blueprint nodes that's have similar API as built-in JsonBlueprintUtilities plugin:
Category | Name | Usage |
---|---|---|
Dump | DataConfig Dump Object To JSON | Dump UObject to a JSON string. |
Dump | DataConfig Dump Value To JSON | Dump arbitrary value to a JSON string. |
Load | DataConfig Load Object From JSON | Load data to UObject from a JSON string. |
Load | DataConfig Load Value From JSON | Load data to arbitrary value from a JSON String |
Some caveats:
- The Dump/Load
Object
APIs always expand the root level object, even if it's a external object reference or an asset. These are by default serialized to a string by DataConfig. Also theSelf
pin only works with these due to some BP limitations. - Most extra examples are integrated into these nodes so you can try them out without writing C++ code.
- Error diagnostic is also wired up to message log:
Advanced
This section contains documentation for advanced topics.
Benchmark
We've integrated two benchmarks fixtures from JSON for modern C++ project. In the benchmark we deserialize JSON into C++ structs, then serialize them back to JSON. Then we convert JSON to MsgPack and do identical process.
See here for instructions to build and run the benchmark.
Here're the results:
On a AMD Ryzen 9 5950X 16-Core Processor 3.40 GHz:
Corpus Json Deserialize: [Shipping] Bandwidth: 94.951(MB/s), Mean: 58.097(ms), Median:58.061(ms), Deviation:0.997
Corpus Json Serialize: [Shipping] Bandwidth: 117.859(MB/s), Mean: 46.805(ms), Median:46.086(ms), Deviation:1.818
Corpus MsgPack Deserialize: [Shipping] Bandwidth: 103.436(MB/s), Mean: 50.531(ms), Median:50.453(ms), Deviation:0.469
Corpus MsgPack Serialize: [Shipping] Bandwidth: 103.586(MB/s), Mean: 50.457(ms), Median:50.342(ms), Deviation:0.614
Canada Json Deserialize: [Shipping] Bandwidth: 73.581(MB/s), Mean: 29.176(ms), Median:29.171(ms), Deviation:0.132
Canada Json Serialize: [Shipping] Bandwidth: 56.439(MB/s), Mean: 38.037(ms), Median:37.882(ms), Deviation:1.050
Canada MsgPack Serialize: [Shipping] Bandwidth: 131.555(MB/s), Mean: 4.441(ms), Median:4.432(ms), Deviation:0.030
Canada MsgPack Deserialize: [Shipping] Bandwidth: 100.450(MB/s), Mean: 5.816(ms), Median:5.816(ms), Deviation:0.024
Some insights on the results:
-
Benchmark in
Shipping
build configuration, otherwise it doesn't make much sense. -
Recall that runtime performance isn't our top priority. We opted for a classic inheritance based API for
FDcReader/FDcWriter
which means that each read/write step result in a virtual dispatch. This by design would result in mediocre performance metrics. The bandwidth should be in the range of10~100(MB/s)
on common PC setup, no matter how simple the format is. -
MsgPack and JSON has similar bandwidth numbers in the benchmark. However MsgPack has far more tight layout when dealing with numeric data. Note in the
Canada
fixture MsgPack only takes around 10ms, as this fixture is mostly float number coordinates.
Tips for writing handlers
There're some recurring patterns when writing handlers in DataConfig.
Recursive Deserialize
When deserializing a container like USTRUCT
root or TArray
you'll need to recursively deserialize children properties. This is wrapped in a single function call:
// DataConfigCore/Private/DataConfig/Deserialize/Handlers/Json/DcJsonStructDeserializers.cpp
DC_TRY(DcDeserializeUtils::RecursiveDeserialize(Ctx));
Internally it would push writer's next property into FDcDeserializeContext::Properties
to satisfiy the invariant that FDcDeserializeContext::TopProperty()
always points to the current writing property. It would also clear up the top property on return.
Another example is how we pipe deserialize a TMap<>
. When at key and value position we simply call this method 2 times:
// DataConfigCore/Private/DataConfig/Deserialize/Handlers/Property/DcPropertyPipeDeserializers.cpp
DC_TRY(Ctx.Reader->ReadMapRoot());
DC_TRY(Ctx.Writer->WriteMapRoot());
EDcDataEntry CurPeek;
while (true)
{
DC_TRY(Ctx.Reader->PeekRead(&CurPeek));
if (CurPeek == EDcDataEntry::MapEnd)
break;
DC_TRY(DcDeserializeUtils::RecursiveDeserialize(Ctx));
DC_TRY(DcDeserializeUtils::RecursiveDeserialize(Ctx));
}
DC_TRY(Ctx.Reader->ReadMapEnd());
DC_TRY(Ctx.Writer->WriteMapEnd());
Provide TopObject()
Sometimes deserialization will create new UObject
along the way. In this case you'll need to fill in FDcDeserializeContext::Objects
so the top one is used for NewObject()
calls. For transient objecst you can use GetTransientPackage()
:
// DataConfigTests/Private/DcTestDeserialize.cpp
Ctx.Objects.Push(GetTransientPackage());
Peek By Value
Sometimes you want to peek the content of the next entry. For example in DcExtra::HandlerBPDcAnyStructDeserialize()
we're dealing with a JSON like this:
// DataConfigEngineExtra/Private/DataConfig/EngineExtra/Deserialize/DcDeserializeBPClass.cpp
{
"AnyStructField1" : {
"$type" : "/DataConfig/DcFixture/DcTestBlueprintStructWithColor",
"NameField" : "Foo",
//...
}
}
We want to consume the $type
key and its value, and then delegate the logic back to the deserializer. The solution here is first to consume the pair. Then we put back a {
then replace the reader:
// DataConfigEngineExtra/Private/DataConfig/EngineExtra/Deserialize/DcDeserializeBPClass.cpp
FDcPutbackReader PutbackReader(Ctx.Reader);
PutbackReader.Putback(EDcDataEntry::MapRoot);
TDcStoreThenReset<FDcReader*> RestoreReader(Ctx.Reader, &PutbackReader);
FDcScopedProperty ScopedValueProperty(Ctx);
DC_TRY(ScopedValueProperty.PushProperty());
DC_TRY(Ctx.Deserializer->Deserialize(Ctx));
Beware that Putback
only support a limited subset of data types.
Coercion
Readers implements FDcReader::Coercion()
which can be used to query if the next value can be coerced into other types.
Here's an example of reading a JSON number as string:
// DataConfigTests/Private/DcTestBlurb2.cpp
FDcJsonReader Reader(TEXT(R"(
1.234567
)"));
// check coercion
// note this is a query and doesn't affect reading at all
bool bCanCoerceToStr;
DC_TRY(Reader.Coercion(EDcDataEntry::String, &bCanCoerceToStr));
check(bCanCoerceToStr);
// read number token as stjring
// note here we skipped parsing the number to float
FString NumStr;
DC_TRY(Reader.ReadString(&NumStr));
check(NumStr == TEXT("1.234567"));
return DcOk();
What's cool here is that FDcJsonReader
does parsing at the ReadFloat()
callsite. So when doing coercion above it actually skipped string to float parsing.
Here's a table of all existing coercion rules:
Reader | From Type | To Type |
---|---|---|
FDcPropertyReader | Array | Blob |
StructRoot | Blob | |
FDcJsonReader | Double | String |
Double | Int8/Int16/Int32/Int64 UInt8/Uint16/UInt32/Uint64 Float | |
String | Name/Text | |
FDcMsgPackReader | String | Name/Text |
Some caveats regarding coercion:
- When reading from JSON/MsgPack string can be read as Name/Text for convenient.
- When reading from Property, Array/Struct can be read as a
FDcBlobViewData
which directly points to the memory span.
UE Core Types Handlers
Without any special handling a FVector
is serialized as a JSON object like this:
{
"X" : 1,
"Y" : 2,
"Z" : 3,
}
This is fine but sometimes we want it to be more compact. DataConfig now comes with a set of serialize/deserialize handlers that writes the commonly used types in compact form:
// setup core types serializers
FDcSerializer Serializer;
DcSetupCoreTypesSerializeHandlers(Serializer);
// setup core types deserializers
FDcDeserializer Deserializer;
DcSetupCoreTypesDeserializeHandlers(Deserializer);
With a struct like this:
USTRUCT()
struct DATACONFIGEXTRA_API FDcExtraCoreTypesStruct
{
GENERATED_BODY()
UPROPERTY() FGuid GuidField1;
UPROPERTY() FGuid GuidField2;
UPROPERTY() FVector2D Vec2Field1;
UPROPERTY() FVector2D Vec2Field2;
UPROPERTY() FVector VecField1;
UPROPERTY() FVector VecField2;
UPROPERTY() FPlane PlaneField1;
UPROPERTY() FPlane PlaneField2;
UPROPERTY() FMatrix MatrixField1;
UPROPERTY() FMatrix MatrixField2;
UPROPERTY() FBox BoxField1;
UPROPERTY() FBox BoxField2;
UPROPERTY() FRotator RotatorField1;
UPROPERTY() FRotator RotatorField2;
UPROPERTY() FQuat QuatField1;
UPROPERTY() FQuat QuatField2;
UPROPERTY() FTransform TransformField1;
UPROPERTY() FTransform TransformField2;
UPROPERTY() FColor ColorField1;
UPROPERTY() FColor ColorField2;
UPROPERTY() FLinearColor LinearColorField1;
UPROPERTY() FLinearColor LinearColorField2;
UPROPERTY() FIntPoint IntPointField1;
UPROPERTY() FIntPoint IntPointField2;
UPROPERTY() FIntVector IntVectorField1;
UPROPERTY() FIntVector IntVectorField2;
UPROPERTY() FDateTime DateTimeField1;
UPROPERTY() FDateTime DateTimeField2;
UPROPERTY() FTimespan TimespanField1;
UPROPERTY() FTimespan TimespanField2;
};
It would be serialized like this:
{
"GuidField1" : [ 1, 2, 3, 4],
"GuidField2" : [ 0, 0, 0, 0],
"Vec2Field1" : [ 0, 1],
"Vec2Field2" : [ 0, 0],
"VecField1" : [ 0, 0, 1],
"VecField2" : [ 0, 0, 0],
"PlaneField1" : [ 1, 2, 3, 4],
"PlaneField2" : [ 0, 0, 0, 0],
"MatrixField1" : [
[ 1, 0, 0, 0],
[ 0, 1, 0, 0],
[ 0, 0, 1, 0],
[ 0, 0, 0, 1]
],
"MatrixField2" : [
[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0]
],
"BoxField1" : {
"Min" : [ 0, 0, 0],
"Max" : [ 0, 0, 1],
"IsValid" : 1
},
"BoxField2" : {
"Min" : [ 1, 1, 1],
"Max" : [ 0, 0, -1],
"IsValid" : 1
},
"RotatorField1" : [ 1, 2, 3],
"RotatorField2" : [ 0, 0, 0],
"QuatField1" : [ 1, 2, 3, 4],
"QuatField2" : [ 0, 0, 0, 0],
"TransformField1" : {
"Rotation" : [ 0, 0, 0, 1],
"Translation" : [ 1, 2, 3],
"Scale3D" : [ 1, 1, 1]
},
"TransformField2" : {
"Rotation" : [ 0, 0, 0, 1],
"Translation" : [ 0, 0, 0],
"Scale3D" : [ 1, 1, 1]
},
"ColorField1" : "#000000FF",
"ColorField2" : "#0000FFFF",
"LinearColorField1" : [ 0, 0, 0, 1],
"LinearColorField2" : [ 1, 1, 1, 1],
"IntPointField1" : [ 1, 2],
"IntPointField2" : [ 0, 0],
"IntVectorField1" : [ 1, 2, 3],
"IntVectorField2" : [ 0, 0, 0],
"DateTimeField1" : "0001.01.01-00.00.00",
"DateTimeField2" : "1988.07.23-00.00.00",
"TimespanField1" : "+00:00:00.000",
"TimespanField2" : "+5.06:07:08.000"
}
This would make large JSON dumps more readable. Deserialize also works and it supports both array and object form.
Optional
This example only works with UE5.4+
UE 5.4 introduced optional property which now allows you to mark TOptional
fields as UPROPERTY
:
// DataConfigTests54/Private/DcTestUE54.h
USTRUCT()
struct FDcTestOptional
{
GENERATED_BODY()
UPROPERTY() TOptional<float> OptionalFloatField1;
UPROPERTY() TOptional<float> OptionalFloatField2;
UPROPERTY() TOptional<FString> OptionalStrField1;
UPROPERTY() TOptional<FString> OptionalStrField2;
UPROPERTY() TOptional<FDcInnerStruct54> OptionalStructField1;
UPROPERTY() TOptional<FDcInnerStruct54> OptionalStructField2;
};
It's a useful utility to correctly model your data structure. For example you can use TOptional<uint32>
to correctly represent an optional unsigned int, rather than using a special value like UINT32_MAX
. DataConfig fully supports optional in the reader writer API:
// DataConfigTests54/Private/DcTestUE54.cpp
TOptional<FString> Source;
// ...
DC_TRY(Writer.WriteOptionalRoot()); // Optional Root
DC_TRY(Writer.WriteString(TEXT("Some String"))); // Str
DC_TRY(Writer.WriteOptionalEnd()); // Optional End
// ...
FString ReadStr;
DC_TRY(Reader.ReadOptionalRoot()); // Optional Root
DC_TRY(Reader.ReadString(&ReadStr)); // Str
check(ReadStr == TEXT("Some String"));
DC_TRY(Reader.ReadOptionalEnd()); // Optional End
// ...
DC_TRY(Writer.WriteOptionalRoot()); // Optional Root
DC_TRY(Writer.WriteNone()); // None
DC_TRY(Writer.WriteOptionalEnd()); // Optional End
// ...
DC_TRY(Reader.ReadOptionalRoot()); // Optional Root
DC_TRY(Reader.ReadNone()); // None
DC_TRY(Reader.ReadOptionalEnd()); // Optional End
Optional also maps perfectly with JSON since any JSON value can be null
:
// DataConfigTests54/Private/DcTestUE54.cpp
FString Str = TEXT(R"(
{
"OptionalFloatField1" : 17.5,
"OptionalFloatField2" : null,
"OptionalStrField1" : "Alpha",
"OptionalStrField2" : null,
"OptionalStructField1" : {
"StrField" : "Beta",
"IntField" : 42
},
"OptionalStructField2" : null
}
)");
// equivalent fixture
FDcTestOptional Expect;
Expect.OptionalFloatField1 = 17.5f;
Expect.OptionalStrField1 = TEXT("Alpha");
Expect.OptionalStructField1.Emplace();
Expect.OptionalStructField1->StrField = TEXT("Beta");
Expect.OptionalStructField1->IntField = 42;
Note how it works with nested optional struct.
Non Class/Struct Root
Sometimes you just want to deserialize something into an TArray/TMap/TSet
. Then you'll realize that you don't have something corresponding to StaticClass()/StaticStruct()
as root to pass to DataConfig.
// DataConfigTests/Private/DcTestBlurb2.cpp
FString Fixture = TEXT("[1,2,3,4,5]");
TArray<int> Arr;
Turns out you can create adhoc FProperty
without USTRUCT/UCLASS
parents and use them just fine. In DataConfig we've provided DcPropertyUtils::FDcPropertyBuilder
to ease this use case.
// DataConfigTests/Private/DcTestBlurb2.cpp
// create int array property
using namespace DcPropertyUtils;
auto ArrProp = FDcPropertyBuilder::Array(
FDcPropertyBuilder::Int()
).LinkOnScope();
FDcJsonReader Reader{Fixture};
DC_TRY(DcAutomationUtils::DeserializeFrom(&Reader, FDcPropertyDatum(ArrProp.Get(), &Arr)));
// validate results
check(Arr.Num() == 5);
check(Arr[4] == 5);
Note that FDcPropertyBuilder
would create a heap allocated FProperty
and LinkOnScope()
returns a TUniquePtr
. You might want to cache the properties if used repeatedly.
No MetaData
Some DataConfig behaviors take advantage of property metadata. For example the default DcSkip
metadata let DataConfig ignore marked property:
// DataConfigTests/Private/DcTestProperty3.h
UPROPERTY(meta = (DcSkip)) int SkipField1;
However the metadata would be stripped in packaged builds and FProperty::GetMetaData/HasMetaData()
set of methods would be gone. The state is roughly like this:
- In packaged builds there's no metadata C++ methods nor any data.
- Program targets can choose whether to keep metadata methods by setting
bBuildWithEditorOnlyData
inTarget.cs
. This toggles macroWITH_EDITORONLY_DATA
. However the actual metadata would not be auto loaded. - In editor metadata works as intended.
In release 1.4.1 we fixed compilation on !WITH_EDITORONLY_DATA
. This would allow you to include DataConfigCore
as a runtime module and ship it with your project. This is an intended and supported use case.
However with no metadata some default DataConfig behaviors would be different:
- All builtin metas (
DcSkip/DcMsgPackBlob
) are ignored. - Bitflag enums (
UENUM(meta = (Bitflags))
) are ignored. All enums are now treated as scalar.
The good news is that you can workaround this by writing custom serialization handlers. For example for bitflag enums:
- Name your bitflag enums with a prefix like
EFlagBar
and check for it in predicates. - Collect a list of bitflag enum names and test for it in predicates.
- On serializing
PeekRead
for next data entry. If it's aArrayRoot
then treat it as a bitflag enum.
Note that you should be able to implement these without modifying DataConfigCore
code.
Automation
One thing we find that's really important when maintaining DataConfig across multiple UE versions, is that proper automation is a must. On this page we document how to run the bundled automation tests.
Note that all instructions shown here are using Windows with a Cmd shell. You'll need to adapt to your system setup.
DataConfigHeadless
We provide a standalone, command target that runs tests on core, non editor DataConfig features. This requires a source build to work, that means downloaded dists from Epic Launcher can not build this target.
-
Get a copy of Unreal Engine source code. Also checkout DataConfig supported UE versions.
-
Run
./GenerateProjectFiles.bat
under Unreal Engine root. Note that you don't need to build the editor. This step would buildUnrealBuildTool
which is enough for the headless target to work. -
Get a copy of DataConfig repository. At the folder where
DataConfig.uplugin
andDataConfig4.uplugin
resides:# UE5 <PathToUE5SourceBuild>/Engine/Build/BatchFiles/RunUBT -project="%CD%/DataConfig.uplugin" DataConfigHeadless Win64 Debug # UE4 <PathToUE4SourceBuild>/Engine/Binaries/DotNET/UnrealBuildTool.exe -project="%CD%/DataConfig4.uplugin" DataConfigHeadless Win64 Debug
Note how UE4/5 uses
DataConfig4.uplugin/DataConfig.uplugin
respectively. -
Run the headless target binary:
%CD%/Binaries/Win64/DataConfigHeadless-Win64-Debug.exe
A success run looks like this:
LogDataConfigCore: Display: UE Version: 5.1.0, DataConfigCore Version: 1.4.0, 10400 LogDataConfigCore: Display: Filters: DataConfig. - OK | DataConfig.Core.Property.StackScalarRoots - OK | DataConfig.Extra.SerDe.Base64 ....... - OK | DataConfig.Extra.InlineStructUsage - OK | DataConfig.Core.Property.Primitive1 Run: 122, Success: 122, Fail: 0
Running the benchmarks
DataConfigHeadless
accepts command line arguments as filters. The benchmarks runs slower are skipped by default.
You can build and run the benchmarks with the commands below:
# remember to build in Shipping
<PathToUE5SourceBuild>Engine/Build/BatchFiles/RunUBT -project="%CD%/DataConfig.uplugin" DataConfigHeadless Win64 Shipping
%CD%/Binaries/Win64/DataConfigHeadless-Win64-Shipping.exe DataConfigBenchmark
Build and run Linux target with WSL2
UE officially support cross compiling for linux and distribute toolchains on its website. Here we demonstrate how to build the headless target for Linux and run it under WSL2.
-
Setup WSL2 following this guide. You should get the latest LTS Ubuntu. Run
wsl lsb_release -ir
to validate it's working.> wsl lsb_release -ir Distributor ID: Ubuntu Release: 22.04
-
Install UE cross compile toolchain. Grab the installer and install on your machine.
Note that each UE version matches a different toolchain. You can download and install multiple toolchains and select which to use at build time through environment variable.
-
Build the headless target for Linux.
# UE5.1 set LINUX_MULTIARCH_ROOT=<PathToToolchains>/v20_clang-13.0.1-centos7 <PathToUE5SourceBuild>/Engine/Build/BatchFiles/RunUBT -project="%CD%/DataConfig.uplugin" DataConfigHeadless Linux Debug # UE4 set LINUX_MULTIARCH_ROOT=<PathToToolchains>/v19_clang-11.0.1-centos7 <PathToUE4SourceBuild>/Engine/Binaries/DotNET/UnrealBuildTool.exe -project="%CD%/DataConfig4.uplugin" DataConfigHeadless Linux Debug
-
Run the headless target through WSL.
wsl ./Binaries/Linux/DataConfigHeadless-Linux-Debug
A success run looks like this:
Using Mimalloc. LogInit: Build: ++UE5+Release-5.1-CL-0 LogInit: Engine Version: 5.1.0-0+++UE5+Release-5.1 LogInit: Compatible Engine Version: 5.1.0-0+++UE5+Release-5.1 LogInit: OS: Ubuntu 22.04.1 LTS (5.10.102.1-microsoft-standard-WSL2) ....... LogDataConfigCore: Display: UE Version: 5.1.0, DataConfigCore Version: 1.4.0, 10400 LogDataConfigCore: Display: Filters: DataConfig. - OK | DataConfig.Core.MsgPack.TestSuite - OK | DataConfig.Core.Property.Blob2 ....... - OK | DataConfig.Core.Serialize.ObjectRef - OK | DataConfig.Core.MsgPack.Extension Run: 122, Success: 122, Fail: 0 LogCore: Engine exit requested (reason: DataConfigHeadless Main Exit) LogExit: Preparing to exit. LogExit: Object subsystem successfully closed. LogExit: Exiting.
DcCoreTestsCommandlet
DataConfigHeadless
is a Program target that does not depend on Engine
and UnrealEd
module. It get faster compile and iteration time, but losing functionality to touch any gameplay and editor code.
With the editor target we have a DcCoreTestsCommandlet
that can be run through the commandline editor target. On top of that this works with pre-built editor downloaded from Epic Launcher.
-
Integrate DataConfig plugin into your project. You can also find a clean project bundled at
Misc/Project
. -
Build the editor target. You can do it in Visual Studio or with commands below:
# UE5 # build <PathToUE5>/Engine/Build/BatchFiles/RunUBT DcProjectEditor Win64 Development %CD%/DcProject5.uproject -NoHotReload -NoEngineChanges # run commandlet <PathToUE5>/Engine/Binaries/Win64/UnrealEditor-Cmd.exe %CD%/DcProject5.uproject DataConfigEditorExtra.DcCoreTestsCommandlet # UE4 # build <PathToUE4>/Engine/Binaries/DotNET/UnrealBuildTool.exe DcProjectEditor Win64 Development %CD%/DcProject4.uproject -NoHotReload -NoEngineChanges # run commandlet <PathToUE4>/Engine/Binaries/Win64/UE4Editor-Cmd.exe %CD%/DcProject4.uproject DataConfigEditorExtra.DcCoreTestsCommandlet
A success run looks like this:
[2022.10.25-14.33.52:893][ 0]LogTextureFormatETC2: Display: ETC2 Texture loading DLL: TextureConverter.dll [2022.10.25-14.33.52:903][ 0]LogTargetPlatformManager: Display: Loaded TargetPlatform 'Android' [2022.10.25-14.33.52:903][ 0]LogTargetPlatformManager: Display: Loaded TargetPlatform 'Android_ASTC' [2022.10.25-14.33.52:903][ 0]LogTargetPlatformManager: Display: Loaded TargetPlatform 'Android_DXT' ....... [2022.10.25-14.39.30:999][ 0]LogDataConfigCore: Display: Filters: DataConfig. - OK | DataConfig.Core.RoundTrip.MsgPack_Persistent_StringSoftLazy - OK | DataConfig.EditorExtra.GameplayEffect ....... - OK | DataConfig.Core.Property.PropertyBuilder3 - OK | DataConfig.EditorExtra.BPObjectInstance Run: 129, Success: 129, Fail: 0
Running automation in the editor
Lastly you can run and debug automation tests from the editor.
It's the most conventional way to run automation and is well supported in DataConfig.
Unreal Engine Upgrades
DataConfig is committed to support multiple UE versions with no deprecations and warnings. On this page we'll document important upgrade and migration info.
UE5.5
EAutomationTestFlags
is now aenum class
type.TIsTriviallyDestructible
is deprecated overstd::is_trivially_destructible_v
.TFieldPath(OtherPropertyType*)
constructor now checks for actual type safety.FProperty::ElementSize
deprecated overGetElementSize
.StructUtils
plugin is deprecated with all things moved in engine.PER_MODULE_BOILERPLATE
not needed anymore as UBT handles it automatically.FVerseValueProperty
renamed toFVValueProperty
, also compiled only withWITH_VERSE_VM
define.
UE5.4
- New property
Optional
andVValue
are added. We fully supportOptional
starting by addingEDcDataEntry::OptionalRoot/OptionalEnd
and then evantually it works with all DataConfig APIs including JSON/MsgPack serialization and property builder. FObjectPtrProperty/FClassPtrProperty
are deprecated. It's introduced in 5.0 now removed and alias toFObjectProperty/FClassProperty
respectively.- Defaults to MSVC
\W4
flag now which checks for unreachable code. It reports confusing locations and you can setbUseUnity = false
in your*.Build.cs
module rules to locate whereabout. FText
internal pointer changed fromTSharedRef
toTRefCountPtr
.
UE5.3
- Introduces
BuildSettingsVersion.V4
which now defaults to C++ 20. TRemoveConst
is deprecated overstd::remove_const
.FScriptDelegate
etc now has additional checkers based on threading model and debug/release build. Thus we change howFScriptDelegateAccess
works.
UE5.2
TIsSame
is deprecated overstd::is_same
.- In
Build.cs
bEnforceIWYU
is changed to enumIWYUSupport
.
UE5.1
- UE5.1 deprecates
ANY_PACKAGE
in favor of a new methodFindFirstObject
. In DataConfig we providedDcSerdeUtils::FindFirstObject
which callsFindObject(ANY_PACKAGE)
pre 5.1 and callsFindFirstObject()
for 5.1 and onwards.
UE5.0
-
New
TObjectPtr
to replace raw UObject pointers. Turns out this is mostly handled within the engine and is transparent to DataConfig. -
New property types
FObjectPtrProperty
andFClassPtrProperty
are added. They're handled the same asFObjectProperty
andFClassProperty
respectively. -
FVector
now is 3double
s, andReal
data type in Blueprint now is also double. This is also mostly transparent to DataConfig. -
FScriptArrayHelperAccess
size changes with a addeduint32 ElementAlignment
. -
TStringBuilderWithBuffer
API changes. At call sites we now doSb << TCHAR('\n')
instead ofSb.Append(TCHAR('\n'))
.
UE4
- The oldest version DataConfig supports is UE 4.25, in which it introduces a major refactor that changes
UProperty
toFProperty
. We intended to support UE4 in the foreseeable future, especially when we now have separateduplugin
for UE4 and UE5.
Breaking Changes
On this page we'll document breaking changes across versions.
1.7.0
- UE 5.5 deprecated
StructUtils
plugin, thus we'll need to setup multiple uplugins for UE5. See: DataConfigXX.uplugin
1.6.0
-
Added a optional
FName
toFDcSerializer/FDcDeserializer::AddPredicatedHandler
which can be used to identify an entry so one can replace it later like this:Ctx.Deserializer->PredicatedDeserializers.FindByPredicate([](auto& Entry){ return Entry.Name == FName(TEXT("Enum")); })->Handler = FDcDeserializeDelegate::CreateStatic(DcEngineExtra::HandlerBPEnumDeserialize);
-
Removed
DcDeserializeUtils/DcSerializeUtils::PredicateIsUStruct
. UseFDcSerializer/FDcDeserializer::AddStructHandler
instead. It's more accurate and overall gets better performance.For an concrete example, change this:
Ctx.Deserializer->AddPredicatedHandler( FDcDeserializePredicate::CreateStatic(DcDeserializeUtils::PredicateIsUStruct<FMsgPackTestSuite>), FDcDeserializeDelegate::CreateStatic(HandlerMsgPackTestSuite) );
to this:
Ctx.Deserializer->AddStructHandler( TBaseStructure<FMsgPackTestSuite>::Get(), FDcDeserializeDelegate::CreateStatic(HandlerMsgPackTestSuite) );
and that's done. For more details see: Serializer Deserializer Setup.
-
Move many
DataConfigEditorExtra
content to the newDataConfigEngineExtra
runtime module so it can be used at runtime. This is mostly for the blueprint nodes to be available at engine runtime.
1.5.0
- Renamed
EDcDataEntry::Nil
toEDcDataEntry::None
. This is a fix for MacOS build.
Changes
All notable changes to this project will be documented in this file.
1.7.0 - 2024-10-2
- Initial UE 5.5 support.
- BREAKING Separate
DataConfig54.uplugin
andDataConfig.uplugin
.- See: Breaking - 1.7.0
1.6.2 - 2024-5-18
- Misc Shipping configuration build fixes.
1.6.1 - 2024-4-28
- Fix compilation for UE 5.4.
1.6.0 - 2024-3-31
- BREAKING Removed
PredicateIsUStruct
. Use struct handlers instead.- See: Breaking - 1.6.0
- NEW JSON Blueprint Library Nodes
- See: Extra - Blueprint Nodes
- Moved most editor extra samples to engine so it can be used at runtime.
- NEW Struct handlers added to serializer/deserializers.
- NEW JSON Writer improvements.
- Allows override config for inline object/arrays.
See: JSON - Override Config - Add float/int format strings to config.
- Allows override config for inline object/arrays.
- NEW UE core types serializers and deserializers.
- See: UE Core Types
- NEW Property reader/writer heuristic stack overflow detection.
- This avoids a hard crash when serializing running into infinite loop.
- FIX EditorConfig fixes.
- Now properly indented with tab just like UE code base.
1.5.0 - 2024-2-20
- BREAKING
EDcDataEntry::Nil
renamed toEDcDataType::None
.- This is necessary to fix Mac builds.
- UE 5.4 support.
- NEW Optional support.
- See Optional
1.4.3 - 2023-8-27
- NEW Extra samples.
- CHANGE small QoL changes:
- Fix
FDcDiagnostic
enum argument captures. - Better
FDcJsonReader::FinishRead()
behavior. - Diagnostic message fixes.
- Relax JSONReader a bit to allow root object/array.
- Add
PredicateIsRootProperty
util to select root property.
- Fix
1.4.2 - 2023-8-1
- UE 5.3 support.
1.4.1 - 2023-2-15
- Fix compilation on
!WITH_EDITORONLY_DATA
.- See No MetaData.
- UE 5.2 support.
1.4.0 - 2022-11-17
Checkout blog post "DataConfig 1.4 Released with UE 5.1 Support".
- NEW Support for UE 5.1.
- See UE version upgrade.
- NEW Extra samples and docs.
- CHANGE Use separated
uplugin
for UE4 and UE5.- See Integration
- FIX Core changes and fixes:
- Fail when
FPropertyWriter::WriteObjectReference()
takes nullptr. See test:DataConfig.Core.Property.DiagNullObject
- Additional check for class mismatch for default object deserialize handlers.
See test:
DataConfig.Core.Deserialize.DiagObjectClassMismatch
- Fail when
1.3.0 - 2022-6-20
Checkout blog post "DataConfig Core and JSON Asset 1.3 Release".
- CHANGE Property reader/writer improvement and fixes
- Allow reading
Array/Set/Map
and native array likeint arr[5]
as root.
See: Non Class/Struct Root - Expand property with
ArrayDim > 1
as array.
See: Property - Performance improvement by caching
FScript/Array/SetHelper
- Allow reading
- NEW Extra samples:
- FIX Core changes and fixes:
- Fix stale enum property fields serialization crash.
- Fix
TSet/TMap
serialization crashes. - Fix soft object/class reference serialize to nil when it's not loaded.
- Fix pipe property class/object handlers.
- Fix
TObjectPtr<>
serialialzation and deserialization. - Fix
PeekReadDataPtr
on class property. - Removed
FScopedStackedReader/FScopedStackedWriter
usage. - Fix
HeuristicVerifyPointer
diagnostic. - Fix
DC_TRY
shadowing variableRet
. - JSON now support non string keyed
TMap<>
as[{ "$key": <foo>, "$value": <bar> }]
.
See: JSON - Map - Fix
DcPropertyUtils::DcIsSubObjectProperty()
Now it only checks forCPF_InstancedReference
. - Update screenshots to UE5.
Note that DataConfig still supports from 4.25 and onwards. - Update Property docs.
1.2.2 - 2022-4-5
- Support for UE 5.0.0
- Add
DebugGetRealPropertyValue
for double BP fields.
1.2.1 - 2022-2-23
- Support for UE5 Preview 1.
- FIX Compile fixes for examples on UE 4.25.
- FIX UE 4.25/4.26 editor extra BP serde automation test fixes.
- FIX
FDcAnsiJsonWriter
writes non ascii char to?
when string contains escaping characters.FDcJsonWriter
was unaffected. This only happens to the ansi char writer and only when input has escapes like\t
.
1.2.0 - 2022-1-26
Checkout blog post "Introducing DataConfig 1.2".
- NEW Serializer. Previously we only have deserializers.
- Serializer API mirrors deserializers.
- Builtin serialization and deserialization handlers are all roundtrip-able.
DcDiagnosticDeserialize -> DcDiagnosticSerDe
for sharing diagnostics.DcDeserializeUtils -> DcSerDeUtils
for sharing code.
- NEW MsgPack reader and writer.
- Full spec implemented, minus the "Timestamp extension type".
- Integrate and passes kawanet/msgpack-test-suite.
- NEW JSON writer.
- With
WIDECHAR/ANSICHAR
specialization as JSON Reader. - Accept config to output pretty or condensed output.
- With
- NEW Builtin metas.
DcSkip
- skip marked fieldsDcMsgPackBlob
- marked TArray<>/Struct would be read as blob by MsgPack SerDe
- CHANGE Core type changes.
FDcReader/FDcWriter
changes.FDcStruct/ClassStat
renamed toFDcStruct/ClassAccess
.ReadStruct/ClassRoot()
renamed toReadStruct/ClassRootAccess
.- Add
ReadStruct/ClassRoot()
that takes no argument for common use cases. FDcReader::Coercion()
now returns aFDcResult
[Read/Write]Soft[Object/Class]Reference
takesFSoftObjectPtr
.- RTTI with
GetId()
andCastById()
FDcSerializer/FDcDeserializer
changes.DcDiagnosticDeserialize
->DcDiagnosticSerDe
for sharing diagnostics.DcDeserializeUtils
->DcSerDeUtils
for sharing code.- Add handlers to read/write
Soft/Lazy
references as is, without loading the object. - Implicit call
Properties.Push()
before contextPrepare()
. - Removed
FDcScopedProperty
in favor ofDcDeserializeUtils::RecursiveDeserialize()
it's more concise.
FDcPropertyReader/FDcPropertyWriter
changes.- When reading class object keys any one with
$
will be ignored.- previously only allow
$type
,$path
. - note that struct by default don't check for these. It's trivia to add the logic if you want to.
- previously only allow
- Add
FDcPropertyReader::PeekReadDataPtr
matches withPeekProperty
- When reading class object keys any one with
FDcJsonReader
changes.- Remove object key length limit, which was 2048 and it's incorrect.
- Though
FName
is capped at 1024, which is a Unreal Engine limit.
- Though
- Fix
ReadName()
which previously would fail. - Fix quoted string parsing/escaping in
ParseQuotedString
- Remove object key length limit, which was 2048 and it's incorrect.
- Add
EDcDataEntry::Extension
. - Add
FDcPropertyDatum
template constructor to directly construct one from aFSturct*
. FPrettyPrintWriter
now print blobs with hash, previously it's pointer value.DcAutomationUtils::SerializeIntoJson/DeserializeIntoJson
->SerializeInto/DeserializeFrom
as we're supporting other formats.- Add
HeuristicVerifyPointer
to check common magic invalid pointers.
- Misc fixes and QOL improvements.
- Fixed linux build. Now the headless program target cross compiles and runs under wsl.
- Serializer/Deserializer now also report diagnostics.
- Better Json reader diagnostic formatting. Now it clamps long lines properly.
- Add
DataConfigEditorExtra.DcCoreTestsCommandlet
as tests runner. - Add performance benchmark.
- Restructured DataConfig book for topics on serializer and MsgPack.
1.1.1 - 2021-10-6
- Support for UE 4.27.
- Support for UE 5.
1.1.0 - 2021-4-24
- Integrate nst/JSONTestSuite. Now
DcJSONParser
pass most of them. Skipped tests are also documented. FDcAnsiJsonReader
now detect and convert non ASCII UTF8 characters. Previously these characters are dropped.- Headless test runner pass along parameters to tests, for example
DataConfigHeadless-Win64-Debug.exe Parsing -- n_array_just_minus
License
DataConfig is released under MIT License.
MIT License
Copyright (c) 2021-2024 Chen Tao
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
Attribution
If you find DataConfig useful in your project, consider credit us in your project with the full license above or the shorter snippets below.
DataConfig <https//slowburn.dev/dataconfig>
MIT License, Copyright (c) 2021-2024 Chen Tao
There's also DataConfig JSON Asset on UE Marketplace. It's a premium plugin for importing JSON to UE data assets.
You can reach us by email to [email protected] or on twitter @slowburndev.
We'll setup a section showcasing projects using DataConfig in the future.