NuoDB ingestion rate measurement

NuoDB is a memory centric, ACID compliant distributed SQL database.

This test was written using the C++ API accessing the NuoDB Community Edition Release 2.5.6 for Windows which is a single-host version of the NuoDB distributed database.

The ingestion rate was measured on an Intel Core i7-4700MQ 2.4GHz laptop with 16GB RAM, running Windows 10 64 bit, having a pair of SSDs in RAID0 (striped).

Test 1

A single table was created in an otherwise empty database named 'pt' on localhost:48004 using the following command:

create table names (id int primary key, name string)

The following C++ code was used to populate the table with a million records using one thousand transactions, where each record has a 64 bit key and a mapped value of 4kB:

const int NUM_TXN = 1000;
const int NUM_ROWS_PER_TXN = 1000;
const int MAPPED_VALUE_SIZE = 4096;
std::string nameStr(MAPPED_VALUE_SIZE,'x');
const char* name = nameStr.c_str();
PreparedStatement* stmt = connection->prepareStatement("insert into names (id,name) values (?,?)");
try
{
    int id = 0;
    for (int t=0 ; t < NUM_TXN ; ++t)
    {
        for (int r = 0; r < NUM_ROWS_PER_TXN; r++)
        {
            stmt->setInt(1, id++);
            stmt->setString(2, name);
            stmt->addBatch();
        }
        stmt->executeBatch();
        connection->commit();
    }
    stmt->close();
}
catch (SQLException& xcp)
{
    connection->rollback();
    throw;
}

The time taken to run the test was about 13 minutes. The final size of the database files (C:\ProgramData\nuodb\production-archives\pt) was 4.31GB.

Test 2

The previous test was only using 32 bit keys, and using strings for the mapped values. This test instead uses the 'bigint' datatype for the keys which is 64 bit as used in the performance measurements on the other DBMS products that have been tested, and also uses 'varbinary' for the mapped values, in case this is more efficient than 'string'.

create table names (id bigint primary key, name varbinary(8192))

The following C++ code was used to populate the table with a million records using one thousand transactions:

const int NUM_TXN = 1000;
const int NUM_ROWS_PER_TXN = 1000;
const int MAPPED_VALUE_SIZE = 4096;
std::string nameStr(MAPPED_VALUE_SIZE,'x');
const char* name = nameStr.c_str();
PreparedStatement* stmt = connection->prepareStatement("insert into names (id,name) values (?,?)");
try
{
    int64_t id = 0;
    for (int t=0 ; t < NUM_TXN ; ++t)
    {
        for (int r = 0; r < NUM_ROWS_PER_TXN; r++)
        {
            stmt->setLong(1, id++);
            stmt->setBytes(2, MAPPED_VALUE_SIZE, name);
            stmt->addBatch();
        }
        stmt->executeBatch();
        connection->commit();
    }
    stmt->close();
}
catch (SQLException& xcp)
{
    connection->rollback();
    throw;
}

The time taken to run the test was 749 seconds (12.5 minutes).

According to the Windows Task Manager, during this time the following processes were taking up the CPU and memory resources:

Process name	Memory	CPU
MsMpEng.exe	61MB	12%
nuodb.exe	2.4GB	10%

Note that when one of the eight processors is at 100%, the CPU usage is reported as 12.5% in the Windows Task Manager on this machine.

Test 3

This is essentially Test 2 with various mapped value sizes: 4,8,16,32,64,128,...,8192.

For each mapped value size, before the ingestion of a million records the table was dropped but the database was not deleted.

Mapped value size	Time (s)	Effective data rate (MB/s)
4	37.8	0.303
8	37.4	0.408
16	39.2	0.584
32	38.8	0.984
64	40.4	1.70
128	42.0	3.09
256	47.1	5.34
512	51.1	9.70
1024	107	9.24
2048	425	4.61
4096	916	4.27
8192	2070	3.77

Interestingly for 4096 byte mapped values the test took 916 seconds (previously was 749). It appears the database performance degrades over time.

To get an idea of the disk space usage the following information was obtained on the folder C:\ProgramData\nuodb\production-archives\pt:

Size	17.7 GB (19,053,864,558 bytes)
Size on disk	18.7 GB (20,158,586,880 bytes)
Contains	310,182 files 3,129 folders

The average file size is only 60kB.