Dev Blog 2.828427² - Is This Thing Even On?

I greet you with a little dance as you tune into this dev blog. It’s very awkward, and you take note of the fact that I don’t seem to really understand how to use my legs.

Hi! It’s me, Bites. As of late, I’ve been a total dancing FIEND. I’ve been playing Dance Dance Revolution multiple times per week, and I’m becoming so strong and powerful that soon nothing will be able to stop me. I’m even planning on building my own dancepad!

I doubt we could build a rhythm-based party game for Rolled Out! that would work very well, but it’s something fun to think about. If you can think of something that would fit the game well, be sure to throw your ideas into the #suggestions channel in our official Discord server!

Let’s blast right into the update.

Our character artist, Brandon, has a WIP of the Armadillo character to share.

They’re coming along spectacularly! Please look forward to their completed model.

For me, though, I’ve been working on stages, as per usual. JUST KIDDING! I’ve been working on backgrounds!

Our lead artist, Laurence, has been painting over screenshots of our existing backgrounds, to add in extra details and changes to make them feel more full and alive. I’ve been working on bringing those paintovers into reality.

Speaking of Laurence… they’ve finished mailing out ALL physical merch! If you don’t receive yours within a week or so, be sure to let us know so we can resend it.
Now that the merch is all done, we will be able to make a TON more progress in the art side of things. I’m so excited to get down and dirty with new backgrounds!

Backgrounds are cool and all, but stages are of course the cream of the crop. The game wouldn’t exist without them, and our current system for storing the database of all stages isn’t exactly exquisite. Here’s a little writeup from CraftedCart on how we’re planning on fixing that.

Revamping the local game database

As beta testers may know, the first launch of the game can take a fair while before you’re able to launch into a game as the game goes ahead and indexes all stages and courses in the background (Usually about 10 to 15-ish seconds on an SSD, or a few minutes on a HDD). The game keeps a local SQLite database of stage and course metadata such that the game can quickly search for stages at a later point at the cost of some indexing time.

Side note about indexing time: Turns out each stage indexed counts for its own transaction, meaning each stage indexed has to write a little bit to the database, which is slow. I forgot to wrap the whole indexing process into one big transaction as opposed to several hundred little transactions.

If you’ve come across relational databases before, you’re probably familiar with SQL, the language used to fetch stuff from and modify a database. Here’s an example SQL command to add some indexed stage metadata into a table UStageMeta (The question marks are placeholders that are filled in later).

INSERT INTO UStageMeta (Uuid, DirectoryName, DirectoryLastModified, StageName, CreatorName, Difficulty, DifficultyName, Description, StageConfigPath) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)

Looks a lot like plain English huh? Use enough of these commands and you’ll get the following.

Don’t worry about the UUID column - that’s what happens when you try and display raw binary data as text.

To help make working with the database a bit easier, I wrote a Python script a while ago to auto-generate C++ and SQL for me, to convert between data structures in C++ and data structures in the database. It was ugly… but it worked!

// DB_CLASS() //
class RO_API UStageMeta : public UObject
{
    GENERATED_BODY()

protected:
    UPROPERTY()
    // DB(PRIMARY_KEY) //
    FGuid Uuid;

    UPROPERTY()
    // DB() //
    FString DirectoryName;

    UPROPERTY()
    // DB() //
    int64 DirectoryLastModified;

    UPROPERTY()
    // DB() //
    FString StageName;

    // ...

    // DB_GEN_START() //
    // Auto-generated database code - do not edit between DB_GEN_SATRT and DB_GEN_END
public:
    UFUNCTION(BlueprintCallable, Category = "Database", Meta = (DisplayName = "UStageMeta: DB CreateTable"))
    static void DB_CreateTable(UDatabaseManager* DatabaseManager);

    UFUNCTION(BlueprintCallable, Category = "Database", Meta = (DisplayName = "UStageMeta: DB InsertObject"))
    void DB_InsertObject(UDatabaseManager* DatabaseManager);

    UFUNCTION(BlueprintCallable, Category = "Database", Meta = (DisplayName = "UStageMeta: DB GetObjectFromPK"))
    static UStageMeta* DB_GetObjectFromPK(UDatabaseManager* DatabaseManager, const FGuid pk);

    UFUNCTION(BlueprintCallable, Category = "Database", Meta = (DisplayName = "UStageMeta: DB GetObjectsMatching_DirectoryName"))
    static TArray<UStageMeta*> DB_GetObjectsMatching_DirectoryName(UDatabaseManager* DatabaseManager, const FString Pattern);

    // ...

    UFUNCTION(BlueprintCallable, Category = "Database", Meta = (DisplayName = "UStageMeta: DB GetField_StageName_Matching_DirectoryName"))
    static TArray<FString> DB_GetField_StageName_Matching_DirectoryName(UDatabaseManager* DatabaseManager, const FString Pattern);

    // ... repeat for every combination

Seeing as I’m deprecating the old stage and course metadata structures in C++ (as part of the stage loading rewrite), I figured I should also figure out a better way to handle database queries that doesn’t require generating and spamming the code with hundreds of functions.

Starting off, core functionality of the database has been split into three different objects: an engine (this handles the connection to the database file), transactions (this handles batching up multiple statements into a single read/write operation), and statements (this handles actually making SQL commands to the database). Using this layer, I can insert a new stage into the database as such.

IDbTransaction* Trans = Engine.BeginTransaction().Expect(ERROR_MESSAGE);

IDbPreparedStatement* Stmt = Engine.PrepareStatement(TEXT("INSERT INTO stage_metadata(uuid, name, description) VALUES (?, ?, ?)")).Expect(ERROR_MESSAGE);
Stmt->BindUuid(1, StageUuid).Expect(ERROR_MESSAGE);
Stmt->BindString(2, StageName).Expect(ERROR_MESSAGE);
Stmt->BindString(2, StageDescription).Expect(ERROR_MESSAGE);
Stmt->Execute().Expect(ERROR_MESSAGE);
delete Stmt;

Trans->Commit().Expect(ERROR_MESSAGE);
delete Trans;

That’s not much of an improvement over just using the raw SQLite API, although there’s some error handling in there now (The .Expect(ERROR_MESSAGE)). Ideally I wouldn’t have to write SQL - that stuff can get a bit kludgey especially with larger statements. Introducing the query builder, a pure C++ way to create SQL, so I don’t need to write a programming language… inside another programming language. With the query builder, the above code transforms into…

IDbTransaction* Trans = Engine.BeginTransaction().Expect(ERROR_MESSAGE);

TResult<void, FGameErrorRef> InsertRes = Engine.BuildQuery().Insert()->
    IntoTable("stage_metadata")->
    WithUuidField(TEXT("uuid"), StageUuid)->
    WithStringField(TEXT("name"), StageName)->
    WithStringField(TEXT("description"), StageDescription)->
    ExecuteAndDelete().
    Expect(ERROR_MESSAGE);

Trans->Commit().Expect(ERROR_MESSAGE);
delete Trans;

Ok that certainly feels nicer to work with over programming-language-inception - also fewer points where I have to handle potential errors. That’s still not how the old script was like though - I can’t just shove a stage metadata object in C++ into the database easily yet. Enter models.

FStageMetadataDbModel::FStageMetadataDbModel()
{
    SetTableName(TEXT("stage_metadata"));

    RegisterField(TEXT("uuid"), FTypeTag::From<FGuid>(), EDbColumnConstraints::PrimaryKey);
    RegisterField(TEXT("name"), FTypeTag::From<FString>(), EDbColumnConstraints::NotNull);
    RegisterField(TEXT("description"), FTypeTag::From<FString>(), EDbColumnConstraints::NotNull);
}

Here is a model defining the data structure for stage metadata. The intent with models is to be able to map various C++ variables to fields in a database without even having to think about constructing database queries - something like a quick FStageMetadataDbModel::Insert(Engine, StageMetadata).Expect(ERROR_MESSAGE); would be ideal. I say would be because I haven’t quite gotten there yet, that’s still a work-in-progress.

Now, these examples have been simplified to help keep things somewhat simple. In practice, stage metadata has to store more than just a name and description. I can’t even just store simple strings of text for those either - these bits of text can have multiple translations associated with them, and you can’t just store multiple bits of data in one field in a database table. In this case, the translatable strings are stored in a different table, which maps a single stage UUID to multiple translatable strings.

And then in code, the model for that looks looks like…

FStageNameDbModel::FStageNameDbModel()
{
    SetTableName(TEXT("stage_name"));

    RegisterField(TEXT("uuid"), FTypeTag::From<FGuid>(), EDbColumnConstraints::PrimaryKey);
    RegisterRelatedField(TEXT("stage_uuid"), FStageMetadataDbModel::GetInstance(), TEXT("uuid"), EDbColumnConstraints::NotNull);
    RegisterField(TEXT("culture_code"), FTypeTag::From<FString>(), EDbColumnConstraints::NotNull);
    RegisterField(TEXT("value"), FTypeTag::From<FString>(), EDbColumnConstraints::NotNull);
}

So in conclusion, the current way the game database is managed, via hundreds of inflexible auto-generated functions, is rather ugly. I’ve been writing various layers of code, each building off of the previous layer, to help make working with the local game database a fair bit easier, such that we can get back to the ease of use with the original auto-generated code (that is, not even having to think about converting between data in the database and data in C++ data structures), but with a far more flexible interface (such that I don’t need to modify the auto-generator script for some specific functionality, generating an even bigger sea of functions).

---

That’s everything for today. What, did you expect me to acknowledge the fact that yesterday was Halloween? Get over yourself! I subscribe to no holidays!

See you on the 15th.