cxRmi Performance

Some performance measurements have been made on various machines using a loopback TCP/IP connection within a single process.

Tests

Most of the tests are concerned with measuring the throughput of small batched one-way function calls on a loopback connection.

There are seven tests which have been defined:

Test	Description
TEST1	Batch one million one-way messages on the sender. This test only measures the time to batch the calls by the sender (i.e. without ensuring the calls have been made on the receiver)
TEST2	Batch one million small one-way messages and wait for them all to be invoked on the receiver
TEST3	Batch many small two-way messages (i.e. messages are buffered on the sender but may not have been sent)
TEST4	Batch many small two-way messages then wait for all responses to be received
TEST5	Synchronously call many small two-way messages. The next message is not sent until the response from the previous message has been received.
TEST6	Synchronously call a two-way message which sends 500000 bytes
TEST7	Synchronously call a two-way message which sends a vector of 50000 elements where each element is a struct containing two ints and a double

Implementation of the tests

For details about the implementation of these tests for each middleware system see the following:

• cxRmi
• ZeroC Ice for C++

Machines

These tests have been performed on the following machines:

Machine configuration	Hardware	OS
M1	Intel core duo E8400 3GHz desktop	Windows XP professional SP3 32 bit
M2	Intel Core 2 Quad Q9400 2.66GHz desktop with 4GB RAM	Windows 7 64 bit
M3	Intel Core i7-4700MQ 2.4GHz laptop with 16GB RAM	Windows 10 64 bit
M4	Intel Core i7-4700MQ 2.4GHz laptop with 16GB RAM	Linux Ubuntu 14.04 64-bit on Oracle VM VirtualBox with one processor and 2GB RAM on Windows 10 64bit
M5	Intel Core i7-4790 3.60GHz with 8GB RAM	Windows 7 64 bit professional SP1

Results

TEST1

Batch many small one-way messages on the sender. This test only measures the time to batch the calls by the sender (i.e. without ensuring the calls have been made on the receiver).

Marshall 0 bytes

Machine configuration	Middleware	Network	v msg/s
M3	cxRmi x64	loopback	51 M
M4	cxRmi x64	loopback	54 M
M5	cxRmi x64	loopback	74 M
M5	ZeroC Ice 3.6.3 x64	loopback	2.9 M

Marshall 1 byte

Machine configuration	Middleware	Network	b1 msg/s
M3	cxRmi x64	loopback	42 M
M4	cxRmi x64	loopback	64 M
M5	cxRmi x64	loopback	64 M
M5	ZeroC Ice 3.6.3 x64	loopback	2.9 M

Marshall 2 bytes

Machine configuration	Middleware	Network	b2 msg/s	s1 msg/s
M3	cxRmi x64	loopback	35 M	42 M
M4	cxRmi x64	loopback	59 M	63 M
M5	cxRmi x64	loopback	55 M	62 M
M5	ZeroC Ice 3.6.3 x64	loopback	2.8 M	2.9 M

Marshall 4 bytes

Machine configuration	Middleware	Network	b4 msg/s	s2 msg/s	i1 msg/s	f1 msg/s
M3	cxRmi x64	loopback	27 M	35 M	41 M	43 M
M4	cxRmi x64	loopback	49 M	58 M	61 M	58 M
M5	cxRmi x64	loopback	44 M	54 M	61 M	59 M
M5	ZeroC Ice 3.6.3 x64	loopback	2.8 M	2.8 M	2.9 M	2.9 M

Marshall 8 bytes

Machine configuration	Middleware	Network	b8 msg/s	s4 msg/s	i2 msg/s	l1 msg/s	f2 msg/s	d1 msg/s
M3	cxRmi x64	loopback	14 M	20 M	26 M	29 M	27 M	29 M
M4	cxRmi x64	loopback	16 M	26 M	25 M	30 M	27 M	27 M
M5	cxRmi x64	loopback	23 M	32 M	36 M	39 M	36 M	39 M
M5	ZeroC Ice 3.6.3 x64	loopback	2.7 M	2.7 M	2.8 M	2.8 M	2.8 M	2.8 M

Marshall 16 bytes

Machine configuration	Middleware	Network	s8 msg/s	i4 msg/s	l2 msg/s	f4 msg/s	d2 msg/s
M3	cxRmi x64	loopback	13 M	19 M	24 M	20 M	25 M
M4	cxRmi x64	loopback	14 M	19 M	19 M	20 M	21 M
M5	cxRmi x64	loopback	21 M	28 M	31 M	28 M	32 M
M5	ZeroC Ice 3.6.3 x64	loopback	2.6 M	2.7 M	2.7 M	2.7 M	2.7 M

Marshall 32 bytes

Machine configuration	Middleware	Network	i8 msg/s	l4 msg/s	f8 msg/s	d4 msg/s
M3	cxRmi x64	loopback	14 M	19 M	15 M	20 M
M4	cxRmi x64	loopback	12 M	15 M	13 M	15 M
M5	cxRmi x64	loopback	21 M	29 M	21 M	27 M
M5	ZeroC Ice 3.6.3 x64	loopback	2.5 M	2.5 M	2.5 M	2.5 M

Marshall 64 bytes

Machine configuration	Middleware	Network	l8 msg/s	d8 msg/s
M3	cxRmi x64	loopback	14 M	15 M
M4	cxRmi x64	loopback	10 M	10 M
M5	cxRmi x64	loopback	19 M	19 M
M5	ZeroC Ice 3.6.3 x64	loopback	2.3 M	2.3 M

TEST2

Batch many small one-way messages and wait for them all to be invoked on the receiver.

Marshall 0 bytes

Machine configuration	Middleware	Network	v msg/s
M3	cxRmi x64	loopback	13.4 M
M4	cxRmi x64	loopback	11.7 M
M5	cxRmi x64	loopback	20.1 M
M5	ZeroC Ice 3.6.3 x64	loopback	2.4 M

Marshall 1 byte

Machine configuration	Middleware	Network	b1 msg/s
M3	cxRmi x64	loopback	10.8 M
M4	cxRmi x64	loopback	10.9 M
M5	cxRmi x64	loopback	16.6 M
M5	ZeroC Ice 3.6.3 x64	loopback	2.2 M

Marshall 2 bytes

Machine configuration	Middleware	Network	b2 msg/s	s1 msg/s
M3	cxRmi x64	loopback	10.0 M	10.5 M
M4	cxRmi x64	loopback	10.5 M	10.8 M
M5	cxRmi x64	loopback	15.1 M	15.6 M
M5	ZeroC Ice 3.6.3 x64	loopback	2.2 M	2.2 M

Marshall 4 bytes

Machine configuration	Middleware	Network	b4 msg/s	s2 msg/s	i1 msg/s	f1 msg/s
M3	cxRmi x64	loopback	8.4 M	9.3 M	9.7 M	9.9 M
M4	cxRmi x64	loopback	9.7 M	10.1 M	10.3 M	10.0 M
M5	cxRmi x64	loopback	13.1 M	13.7 M	14.0 M	14.0 M
M5	ZeroC Ice 3.6.3 x64	loopback	2.1 M	2.2 M	2.2 M	2.2 M

Marshall 8 bytes

Machine configuration	Middleware	Network	b8 msg/s	s4 msg/s	i2 msg/s	l1 msg/s	f2 msg/s	d1 msg/s
M3	cxRmi x64	loopback	6.8 M	7.6 M	7.9 M	8.4 M	8.3 M	8.4 M
M4	cxRmi x64	loopback	7.8 M	8.9 M	9.0 M	9.0 M	8.9 M	8.9 M
M5	cxRmi x64	loopback	10.2 M	11.0 M	11.5 M	11.7 M	11.6 M	11.8 M
M5	ZeroC Ice 3.6.3 x64	loopback	2.1 M	2.1 M	2.2 M	2.2 M	2.2 M	2.2 M

Marshall 16 bytes

Machine configuration	Middleware	Network	s8 msg/s	i4 msg/s	l2 msg/s	f4 msg/s	d2 msg/s
M3	cxRmi x64	loopback	5.7 M	5.9 M	6.3 M	6.3 M	6.4 M
M4	cxRmi x64	loopback	7.8 M	7.5 M	7.6 M	7.5 M	7.7 M
M5	cxRmi x64	loopback	8.3 M	8.7 M	8.9 M	8.7 M	8.9 M
M5	ZeroC Ice 3.6.3 x64	loopback	2.1 M	2.2 M	2.2 M	2.2 M	2.2 M

Marshall 32 bytes

Machine configuration	Middleware	Network	i8 msg/s	l4 msg/s	f8 msg/s	d4 msg/s
M3	cxRmi x64	loopback	4.1 M	4.3 M	4.1 M	4.3 M
M4	cxRmi x64	loopback	6.2 M	6.1 M	5.9 M	6.1 M
M5	cxRmi x64	loopback	6.1 M	6.1 M	6.0 M	6.2 M
M5	ZeroC Ice 3.6.3 x64	loopback	2.0 M	2.0 M	2.0 M	2.0 M

Marshall 64 bytes

Machine configuration	Middleware	Network	l8 msg/s	d8 msg/s
M3	cxRmi x64	loopback	2.5 M	2.5 M
M4	cxRmi x64	loopback	4.8 M	4.5 M
M5	cxRmi x64	loopback	3.6 M	3.6 M
M5	ZeroC Ice 3.6.3 x64	loopback	1.8 M	1.8 M

TEST3-7

Machine	TEST1 (call/s)	TEST2 (call/s)	TEST3 (call/s)	TEST4 (call/s)	TEST5 (call/s)	TEST6 (bit/s)	TEST7 (bit/s)
M1 loopback	27M	-	4.5M	-	34k	-	-
M2 loopback	39M	6.8M	5.8M	2.2M	18k	1760M	1520M
M3 loopback	35M	9.5M	6.4M	2.3M	25k	4000M	2900M
M4 loopback	52M	9.5M	9.4M	2.8M	41k	7600M	4400M
M5 loopback	62M	14M	9.8M	3.9M	42k	4300M	3000M
M6 loopback	2.9M	2.2M	-	-	33k	3500M	2500M

Some of the results are blank for the E8400 machine because more tests have been added over time and this machine is no longer available.

TEST3 and TEXT4 are blank for ZeroC Ice configurations because Ice doesn't support batching of two-way function calls.

Other middleware providers

See Who's Who in Middleware (Gartner Analyst report 1Q04)

• ZeroC Ice
• ZeroMQ
• RabbitMQ
• Java RMI
• Microsoft WCF (Windows Communication Foundation)
• nanomsg

Comparison to other benchmarks of middleware performance

Some middleware performance tests which may also be of interest

• Tran, Gosper and Gorton. Evaluating the sustained performance of COTS-based messaging systems. Sep 2003.
• Pang and Maheshwari. Benchmarking Message-Oriented Middleware – TIB/RV vs. SonicMQ. 2005
• Sachs, Kounev, Bacon and Buchmann. Performance Evaluation of Message-oriented Middleware using the SPECjms2007 Benchmark. 2009.
• Henning. Choosing Middleware: Why Performance and Scalability do (and do not) Matter. 2009.
• Khan, Qureshi and Rashid. Performance Comparison of ICE, HORB, CORBA and Dot NET Remoting Middleware Technologies. July 2010.
• Dworak, Ehm, Sliwinski and Sobczak. MIDDLEWARE TRENDS AND MARKET LEADERS 2011. CERN, Geneva, Switzerland.
• Fengguang and Xie. Networked Automatic Test System based on Message-oriented Middleware. 2015.
• Erikson. Comparing message-oriented middleware for financial assets trading. 2016.

WCF means Windows Communication Foundation

By way of comparison in Choosing Middleware: Why Performance and Scalability do (and do not) Matter by Henning 2009 the following results were reported:

Sending of a message with no in/out parameters to measure latency

Middleware	Send message size (bytes)	Receive message size (bytes)	Loopback invocations per second	gigabit network invocations per second
Ice for C++	32	25	10500	2300
Ice for .NET	32	25	7500	2270
Microsoft WCF with binary encoding	65	58	7500	2200
Microsoft WCF with SOAP encoding	620	531	2500	570
Ice for Java	32	25	8000	2300
Java RMI	41	22	10600	2220

Sending of a 500kB buffer to measure throughput

Middleware	Throughput 500kB (loopback) (send) MBit/sec	Throughput 500kB (loopback) (recv) MBit/sec	Throughput 500kB (network) (send) MBit/sec	Throughput 500kB (network) (recv) MBit/sec
Ice for .NET	630	610	520	410
Microsoft WCF with binary encoding	515	550	425	370
Microsoft WCF with SOAP encoding	144	165	140	130
Ice for Java	825	730	660	560
Java RMI	825	800	283	280

Sending of 50000 struct instances each having two int32s and a float64

Middleware	Throughput (loopback) (send) MBit/sec	Throughput (loopback) (recv) MBit/sec	Throughput (network) (send) MBit/sec	Throughput (network) (recv) MBit/sec
Ice for .NET	380	300	300	205
Microsoft WCF with binary encoding	29	28	31	29
Microsoft WCF with SOAP encoding	21	21	23	24
Ice for Java	138	118	252	148
Java RMI	48	52	66	66