Code for performance testing with ZeroC Ice

The code is derived from one of the C++ Demos written by ZeroC (ice-demos-master/cpp/ICE/throughput).

The following tests are performed:

Batch one million one-way messages on the sender. This test only measures the time to batch the calls by the sender (i.e. without ensuring the calls have been made on the receiver)
Batch one million small one-way messages and wait for them all to be invoked on the receiver (it is assumed this can be achieved by sending a single two-way message after the million one-way messages)
Synchronously call one thousand small two-way messages. The next message is not sent until the response from the previous message has been received. The following message is used (which simply returns i+1)
int inc(int i);
Send a 500000 byte array a thousand times:
void sendByteSeq(["cpp:array"] ByteSeq seq);
Send an array of structs a thousand times, where the array has 50000 elements and where each struct has two 32bit ints and a double float
void sendFixedSeq(FixedSeq seq);

In each case the test is executed 10 times and the minimum time is used to report the performance. This helps to reduce variability in the results, but it should be kept in mind that there is a tendency to quote performance results that are better than the average.

Files in the project

The following are the files in the project:

C++ lambdas are used to represent the code making calls on the proxy to be timed. The lambda includes the loop which repeats the call on the proxy to amortise away the overhead of the lambda itself.

Even though some of the files have been heavily modified the original copyright notices remain on the files.

Small one-way messages

The following functions are used in the slice interface (see Throughput.ice) to measure the time taken to send batched one-way messages:


void v();

void b1(byte i0);
void b2(byte i0,byte i1);
void b4(byte i0,byte i1,byte i2, byte i3);
void b8(byte i0,byte i1,byte i2, byte i3,byte i4,byte i5,byte i6, byte i7);

void s1(short i0);
void s2(short i0,short i1);
void s4(short i0,short i1,short i2, short i3);
void s8(short i0,short i1,short i2, short i3,short i4,short i5,short i6, short i7);

void i1(int i0);
void i2(int i0,int i1);
void i4(int i0,int i1,int i2, int i3);
void i8(int i0,int i1,int i2, int i3,int i4,int i5,int i6, int i7);

void l1(long i0);
void l2(long i0,long i1);
void l4(long i0,long i1,long i2, long i3);
void l8(long i0,long i1,long i2, long i3,long i4,long i5,long i6, long i7);

void f1(float i0);
void f2(float i0,float i1);
void f4(float i0,float i1,float i2, float i3);
void f8(float i0,float i1,float i2, float i3,float i4,float i5,float i6, float i7);

void d1(double i0);
void d2(double i0,double i1);
void d4(double i0,double i1,double i2, double i3);
void d8(double i0,double i1,double i2, double i3,double i4,double i5,double i6, double i7);

Example of generated output


Batched one way messages
v  count: 1000000  time: 336.5 ms  rate: 3.0 M calls/sec
b1  count: 1000000  time: 346.3 ms  rate: 2.9 M calls/sec
b2  count: 1000000  time: 345.1 ms  rate: 2.9 M calls/sec
b4  count: 1000000  time: 359.1 ms  rate: 2.8 M calls/sec
b8  count: 1000000  time: 373.2 ms  rate: 2.7 M calls/sec
s1  count: 1000000  time: 343.1 ms  rate: 2.9 M calls/sec
s2  count: 1000000  time: 352.8 ms  rate: 2.8 M calls/sec
s4  count: 1000000  time: 364.2 ms  rate: 2.7 M calls/sec
s8  count: 1000000  time: 383.0 ms  rate: 2.6 M calls/sec
i1  count: 1000000  time: 346.0 ms  rate: 2.9 M calls/sec
i2  count: 1000000  time: 358.1 ms  rate: 2.8 M calls/sec
i4  count: 1000000  time: 373.6 ms  rate: 2.7 M calls/sec
i8  count: 1000000  time: 407.0 ms  rate: 2.5 M calls/sec
l1  count: 1000000  time: 348.4 ms  rate: 2.9 M calls/sec
l2  count: 1000000  time: 364.5 ms  rate: 2.7 M calls/sec
l4  count: 1000000  time: 394.0 ms  rate: 2.5 M calls/sec
l8  count: 1000000  time: 433.2 ms  rate: 2.3 M calls/sec
f1  count: 1000000  time: 347.8 ms  rate: 2.9 M calls/sec
f2  count: 1000000  time: 355.4 ms  rate: 2.8 M calls/sec
f4  count: 1000000  time: 367.4 ms  rate: 2.7 M calls/sec
f8  count: 1000000  time: 408.5 ms  rate: 2.4 M calls/sec
d1  count: 1000000  time: 351.5 ms  rate: 2.8 M calls/sec
d2  count: 1000000  time: 364.0 ms  rate: 2.7 M calls/sec
d4  count: 1000000  time: 388.9 ms  rate: 2.6 M calls/sec
d8  count: 1000000  time: 435.0 ms  rate: 2.3 M calls/sec

Batched and delivered one way messages
v  count: 1000000  time: 429.0 ms  rate: 2.3 M calls/sec
b1  count: 1000000  time: 467.1 ms  rate: 2.1 M calls/sec
b2  count: 1000000  time: 469.2 ms  rate: 2.1 M calls/sec
b4  count: 1000000  time: 477.5 ms  rate: 2.1 M calls/sec
b8  count: 1000000  time: 492.9 ms  rate: 2.0 M calls/sec
s1  count: 1000000  time: 470.0 ms  rate: 2.1 M calls/sec
s2  count: 1000000  time: 469.2 ms  rate: 2.1 M calls/sec
s4  count: 1000000  time: 484.6 ms  rate: 2.1 M calls/sec
s8  count: 1000000  time: 519.2 ms  rate: 1.9 M calls/sec
i1  count: 1000000  time: 471.8 ms  rate: 2.1 M calls/sec
i2  count: 1000000  time: 474.5 ms  rate: 2.1 M calls/sec
i4  count: 1000000  time: 497.7 ms  rate: 2.0 M calls/sec
i8  count: 1000000  time: 543.0 ms  rate: 1.8 M calls/sec
l1  count: 1000000  time: 481.7 ms  rate: 2.1 M calls/sec
l2  count: 1000000  time: 496.8 ms  rate: 2.0 M calls/sec
l4  count: 1000000  time: 532.0 ms  rate: 1.9 M calls/sec
l8  count: 1000000  time: 617.7 ms  rate: 1.6 M calls/sec
f1  count: 1000000  time: 467.9 ms  rate: 2.1 M calls/sec
f2  count: 1000000  time: 483.7 ms  rate: 2.1 M calls/sec
f4  count: 1000000  time: 507.1 ms  rate: 2.0 M calls/sec
f8  count: 1000000  time: 542.2 ms  rate: 1.8 M calls/sec
d1  count: 1000000  time: 479.2 ms  rate: 2.1 M calls/sec
d2  count: 1000000  time: 507.4 ms  rate: 2.0 M calls/sec
d4  count: 1000000  time: 539.3 ms  rate: 1.9 M calls/sec
d8  count: 1000000  time: 606.3 ms  rate: 1.6 M calls/sec

Synchronous 2-way messages
inc (2-way)  count: 1000  time: 30.1 ms  rate: 33.2 k calls/sec

500000 byte sequence
sendByteSeq (1-way)  Num bytes: 500MB  time: 1087.4 ms  rate: 3678 Mbit/sec
sendByteSeq (2-way)  Num bytes: 500MB  time: 1157 ms  rate: 3456 Mbit/sec
recvByteSeq (2-way)  Num bytes: 500MB  time: 1178 ms  rate: 3395 Mbit/sec
echoByteSeq (2-way)  Num bytes: 1000MB  time: 1614 ms  rate: 4958 Mbit/sec

50000 structs with two ints and a double
sendFixedSeq (1-way)  Num bytes: 800MB  time: 1967 ms  rate: 3254 Mbit/sec
sendFixedSeq (2-way)  Num bytes: 800MB  time: 2577 ms  rate: 2483 Mbit/sec
recvFixedSeq (2-way)  Num bytes: 800MB  time: 2611 ms  rate: 2451 Mbit/sec
echoFixedSeq (2-way)  Num bytes: 1600MB  time: 5654 ms  rate: 2264 Mbit/sec

50000 strings
sendStringSeq (1-way)  Num bytes: 250MB  time: 779 ms  rate: 2566 Mbit/sec
sendStringSeq (2-way)  Num bytes: 250MB  time: 1173 ms  rate: 1705 Mbit/sec
recvStringSeq (2-way)  Num bytes: 250MB  time: 2737 ms  rate: 731 Mbit/sec
echoStringSeq (2-way)  Num bytes: 500MB  time: 3937 ms  rate: 1016 Mbit/sec

50000 structs with string + double
sendStructSeq (1-way)  Num bytes: 650MB  time: 3038 ms  rate: 1711 Mbit/sec
sendStructSeq (2-way)  Num bytes: 650MB  time: 4191 ms  rate: 1241 Mbit/sec
recvStructSeq (2-way)  Num bytes: 650MB  time: 4832 ms  rate: 1076 Mbit/sec
echoStructSeq (2-way)  Num bytes: 1300MB  time: 9186 ms  rate: 1132 Mbit/sec