58 cxRmi Performance
Some performance measurements have been made on various machines using a loopback TCP/IP connection within a single process.
Tests
Most of the tests are concerned with measuring the throughput of small batched one-way function calls on a loopback connection.
There are seven tests which have been defined:
| Test | Description |
|---|---|
| TEST1 | Batch one million one-way messages on the sender. This test only measures the time to batch the calls by the sender (i.e. without ensuring the calls have been made on the receiver) |
| TEST2 | Batch one million small one-way messages and wait for them all to be invoked on the receiver |
| TEST3 | Batch many small two-way messages (i.e. messages are buffered on the sender but may not have been sent) |
| TEST4 | Batch many small two-way messages then wait for all responses to be received |
| TEST5 | Synchronously call many small two-way messages. The next message is not sent until the response from the previous message has been received. |
| TEST6 | Synchronously call a two-way message which sends 500000 bytes |
| TEST7 | Synchronously call a two-way message which sends a vector of 50000 elements where each element is a struct containing two ints and a double |
Implementation of the tests
For details about the implementation of these tests for each middleware system see the following:
Machines
These tests have been performed on the following machines:
| Machine configuration | Hardware | OS |
|---|---|---|
| M1 | Intel core duo E8400 3GHz desktop | Windows XP professional SP3 32 bit |
| M2 | Intel Core 2 Quad Q9400 2.66GHz desktop with 4GB RAM | Windows 7 64 bit |
| M3 | Intel Core i7-4700MQ 2.4GHz laptop with 16GB RAM | Windows 10 64 bit |
| M4 | Intel Core i7-4700MQ 2.4GHz laptop with 16GB RAM | Linux Ubuntu 14.04 64-bit on Oracle VM VirtualBox with one processor and 2GB RAM on Windows 10 64bit |
| M5 | Intel Core i7-4790 3.60GHz with 8GB RAM | Windows 7 64 bit professional SP1 |
Results
TEST1
Batch many small one-way messages on the sender. This test only measures the time to batch the calls by the sender (i.e. without ensuring the calls have been made on the receiver).
Marshall 0 bytes
| Machine configuration |
Middleware | Network | v msg/s |
|---|---|---|---|
| M3 | cxRmi x64 | loopback | 51 M |
| M4 | cxRmi x64 | loopback | 54 M |
| M5 | cxRmi x64 | loopback | 74 M |
| M5 | ZeroC Ice 3.6.3 x64 | loopback | 2.9 M |
Marshall 1 byte
| Machine configuration |
Middleware | Network | b1 msg/s |
|---|---|---|---|
| M3 | cxRmi x64 | loopback | 42 M |
| M4 | cxRmi x64 | loopback | 64 M |
| M5 | cxRmi x64 | loopback | 64 M |
| M5 | ZeroC Ice 3.6.3 x64 | loopback | 2.9 M |
Marshall 2 bytes
| Machine configuration |
Middleware | Network | b2 msg/s |
s1 msg/s |
|---|---|---|---|---|
| M3 | cxRmi x64 | loopback | 35 M | 42 M |
| M4 | cxRmi x64 | loopback | 59 M | 63 M |
| M5 | cxRmi x64 | loopback | 55 M | 62 M |
| M5 | ZeroC Ice 3.6.3 x64 | loopback | 2.8 M | 2.9 M |
Marshall 4 bytes
| Machine configuration |
Middleware | Network | b4 msg/s |
s2 msg/s |
i1 msg/s |
f1 msg/s |
|---|---|---|---|---|---|---|
| M3 | cxRmi x64 | loopback | 27 M | 35 M | 41 M | 43 M |
| M4 | cxRmi x64 | loopback | 49 M | 58 M | 61 M | 58 M |
| M5 | cxRmi x64 | loopback | 44 M | 54 M | 61 M | 59 M |
| M5 | ZeroC Ice 3.6.3 x64 | loopback | 2.8 M | 2.8 M | 2.9 M | 2.9 M |
Marshall 8 bytes
| Machine configuration |
Middleware | Network | b8 msg/s |
s4 msg/s |
i2 msg/s |
l1 msg/s |
f2 msg/s |
d1 msg/s |
|---|---|---|---|---|---|---|---|---|
| M3 | cxRmi x64 | loopback | 14 M | 20 M | 26 M | 29 M | 27 M | 29 M |
| M4 | cxRmi x64 | loopback | 16 M | 26 M | 25 M | 30 M | 27 M | 27 M |
| M5 | cxRmi x64 | loopback | 23 M | 32 M | 36 M | 39 M | 36 M | 39 M |
| M5 | ZeroC Ice 3.6.3 x64 | loopback | 2.7 M | 2.7 M | 2.8 M | 2.8 M | 2.8 M | 2.8 M |
Marshall 16 bytes
| Machine configuration |
Middleware | Network | s8 msg/s |
i4 msg/s |
l2 msg/s |
f4 msg/s |
d2 msg/s |
|---|---|---|---|---|---|---|---|
| M3 | cxRmi x64 | loopback | 13 M | 19 M | 24 M | 20 M | 25 M |
| M4 | cxRmi x64 | loopback | 14 M | 19 M | 19 M | 20 M | 21 M |
| M5 | cxRmi x64 | loopback | 21 M | 28 M | 31 M | 28 M | 32 M |
| M5 | ZeroC Ice 3.6.3 x64 | loopback | 2.6 M | 2.7 M | 2.7 M | 2.7 M | 2.7 M |
Marshall 32 bytes
| Machine configuration |
Middleware | Network | i8 msg/s |
l4 msg/s |
f8 msg/s |
d4 msg/s |
|---|---|---|---|---|---|---|
| M3 | cxRmi x64 | loopback | 14 M | 19 M | 15 M | 20 M |
| M4 | cxRmi x64 | loopback | 12 M | 15 M | 13 M | 15 M |
| M5 | cxRmi x64 | loopback | 21 M | 29 M | 21 M | 27 M |
| M5 | ZeroC Ice 3.6.3 x64 | loopback | 2.5 M | 2.5 M | 2.5 M | 2.5 M |
Marshall 64 bytes
| Machine configuration |
Middleware | Network | l8 msg/s |
d8 msg/s |
|---|---|---|---|---|
| M3 | cxRmi x64 | loopback | 14 M | 15 M |
| M4 | cxRmi x64 | loopback | 10 M | 10 M |
| M5 | cxRmi x64 | loopback | 19 M | 19 M |
| M5 | ZeroC Ice 3.6.3 x64 | loopback | 2.3 M | 2.3 M |
TEST2
Batch many small one-way messages and wait for them all to be invoked on the receiver.
Marshall 0 bytes
| Machine configuration |
Middleware | Network | v msg/s |
|---|---|---|---|
| M3 | cxRmi x64 | loopback | 13.4 M |
| M4 | cxRmi x64 | loopback | 11.7 M |
| M5 | cxRmi x64 | loopback | 20.1 M |
| M5 | ZeroC Ice 3.6.3 x64 | loopback | 2.4 M |
Marshall 1 byte
| Machine configuration |
Middleware | Network | b1 msg/s |
|---|---|---|---|
| M3 | cxRmi x64 | loopback | 10.8 M |
| M4 | cxRmi x64 | loopback | 10.9 M |
| M5 | cxRmi x64 | loopback | 16.6 M |
| M5 | ZeroC Ice 3.6.3 x64 | loopback | 2.2 M |
Marshall 2 bytes
| Machine configuration |
Middleware | Network | b2 msg/s |
s1 msg/s |
|---|---|---|---|---|
| M3 | cxRmi x64 | loopback | 10.0 M | 10.5 M |
| M4 | cxRmi x64 | loopback | 10.5 M | 10.8 M |
| M5 | cxRmi x64 | loopback | 15.1 M | 15.6 M |
| M5 | ZeroC Ice 3.6.3 x64 | loopback | 2.2 M | 2.2 M |
Marshall 4 bytes
| Machine configuration |
Middleware | Network | b4 msg/s |
s2 msg/s |
i1 msg/s |
f1 msg/s |
|---|---|---|---|---|---|---|
| M3 | cxRmi x64 | loopback | 8.4 M | 9.3 M | 9.7 M | 9.9 M |
| M4 | cxRmi x64 | loopback | 9.7 M | 10.1 M | 10.3 M | 10.0 M |
| M5 | cxRmi x64 | loopback | 13.1 M | 13.7 M | 14.0 M | 14.0 M |
| M5 | ZeroC Ice 3.6.3 x64 | loopback | 2.1 M | 2.2 M | 2.2 M | 2.2 M |
Marshall 8 bytes
| Machine configuration |
Middleware | Network | b8 msg/s |
s4 msg/s |
i2 msg/s |
l1 msg/s |
f2 msg/s |
d1 msg/s |
|---|---|---|---|---|---|---|---|---|
| M3 | cxRmi x64 | loopback | 6.8 M | 7.6 M | 7.9 M | 8.4 M | 8.3 M | 8.4 M |
| M4 | cxRmi x64 | loopback | 7.8 M | 8.9 M | 9.0 M | 9.0 M | 8.9 M | 8.9 M |
| M5 | cxRmi x64 | loopback | 10.2 M | 11.0 M | 11.5 M | 11.7 M | 11.6 M | 11.8 M |
| M5 | ZeroC Ice 3.6.3 x64 | loopback | 2.1 M | 2.1 M | 2.2 M | 2.2 M | 2.2 M | 2.2 M |
Marshall 16 bytes
| Machine configuration |
Middleware | Network | s8 msg/s |
i4 msg/s |
l2 msg/s |
f4 msg/s |
d2 msg/s |
|---|---|---|---|---|---|---|---|
| M3 | cxRmi x64 | loopback | 5.7 M | 5.9 M | 6.3 M | 6.3 M | 6.4 M |
| M4 | cxRmi x64 | loopback | 7.8 M | 7.5 M | 7.6 M | 7.5 M | 7.7 M |
| M5 | cxRmi x64 | loopback | 8.3 M | 8.7 M | 8.9 M | 8.7 M | 8.9 M |
| M5 | ZeroC Ice 3.6.3 x64 | loopback | 2.1 M | 2.2 M | 2.2 M | 2.2 M | 2.2 M |
Marshall 32 bytes
| Machine configuration |
Middleware | Network | i8 msg/s |
l4 msg/s |
f8 msg/s |
d4 msg/s |
|---|---|---|---|---|---|---|
| M3 | cxRmi x64 | loopback | 4.1 M | 4.3 M | 4.1 M | 4.3 M |
| M4 | cxRmi x64 | loopback | 6.2 M | 6.1 M | 5.9 M | 6.1 M |
| M5 | cxRmi x64 | loopback | 6.1 M | 6.1 M | 6.0 M | 6.2 M |
| M5 | ZeroC Ice 3.6.3 x64 | loopback | 2.0 M | 2.0 M | 2.0 M | 2.0 M |
Marshall 64 bytes
| Machine configuration |
Middleware | Network | l8 msg/s |
d8 msg/s |
|---|---|---|---|---|
| M3 | cxRmi x64 | loopback | 2.5 M | 2.5 M |
| M4 | cxRmi x64 | loopback | 4.8 M | 4.5 M |
| M5 | cxRmi x64 | loopback | 3.6 M | 3.6 M |
| M5 | ZeroC Ice 3.6.3 x64 | loopback | 1.8 M | 1.8 M |
TEST3-7
| Machine | TEST1 (call/s) |
TEST2 (call/s) |
TEST3 (call/s) |
TEST4 (call/s) |
TEST5 (call/s) |
TEST6 (bit/s) |
TEST7 (bit/s) |
|---|---|---|---|---|---|---|---|
| M1 loopback | 27M | - | 4.5M | - | 34k | - | - |
| M2 loopback | 39M | 6.8M | 5.8M | 2.2M | 18k | 1760M | 1520M |
| M3 loopback | 35M | 9.5M | 6.4M | 2.3M | 25k | 4000M | 2900M |
| M4 loopback | 52M | 9.5M | 9.4M | 2.8M | 41k | 7600M | 4400M |
| M5 loopback | 62M | 14M | 9.8M | 3.9M | 42k | 4300M | 3000M |
| M6 loopback | 2.9M | 2.2M | - | - | 33k | 3500M | 2500M |
Some of the results are blank for the E8400 machine because more tests have been added over time and this machine is no longer available.
TEST3 and TEXT4 are blank for ZeroC Ice configurations because Ice doesn't support batching of two-way function calls.
Other middleware providers
See Who's Who in Middleware (Gartner Analyst report 1Q04)
Comparison to other benchmarks of middleware performance
Some middleware performance tests which may also be of interest
- Tran, Gosper and Gorton. Evaluating the sustained performance of COTS-based messaging systems. Sep 2003.
- Pang and Maheshwari. Benchmarking Message-Oriented Middleware – TIB/RV vs. SonicMQ. 2005
- Sachs, Kounev, Bacon and Buchmann. Performance Evaluation of Message-oriented Middleware using the SPECjms2007 Benchmark. 2009.
- Henning. Choosing Middleware: Why Performance and Scalability do (and do not) Matter. 2009.
- Khan, Qureshi and Rashid. Performance Comparison of ICE, HORB, CORBA and Dot NET Remoting Middleware Technologies. July 2010.
- Dworak, Ehm, Sliwinski and Sobczak. MIDDLEWARE TRENDS AND MARKET LEADERS 2011. CERN, Geneva, Switzerland.
- Fengguang and Xie. Networked Automatic Test System based on Message-oriented Middleware. 2015.
- Erikson. Comparing message-oriented middleware for financial assets trading. 2016.
WCF means Windows Communication Foundation
By way of comparison in Choosing Middleware: Why Performance and Scalability do (and do not) Matter by Henning 2009 the following results were reported:
Sending of a message with no in/out parameters to measure latency
| Middleware | Send message size (bytes) | Receive message size (bytes) | Loopback invocations per second | gigabit network invocations per second |
|---|---|---|---|---|
| Ice for C++ | 32 | 25 | 10500 | 2300 |
| Ice for .NET | 32 | 25 | 7500 | 2270 |
| Microsoft WCF with binary encoding | 65 | 58 | 7500 | 2200 |
| Microsoft WCF with SOAP encoding | 620 | 531 | 2500 | 570 |
| Ice for Java | 32 | 25 | 8000 | 2300 |
| Java RMI | 41 | 22 | 10600 | 2220 |
Sending of a 500kB buffer to measure throughput
| Middleware | Throughput 500kB (loopback) (send) MBit/sec | Throughput 500kB (loopback) (recv) MBit/sec | Throughput 500kB (network) (send) MBit/sec | Throughput 500kB (network) (recv) MBit/sec |
|---|---|---|---|---|
| Ice for .NET | 630 | 610 | 520 | 410 |
| Microsoft WCF with binary encoding | 515 | 550 | 425 | 370 |
| Microsoft WCF with SOAP encoding | 144 | 165 | 140 | 130 |
| Ice for Java | 825 | 730 | 660 | 560 |
| Java RMI | 825 | 800 | 283 | 280 |
Sending of 50000 struct instances each having two int32s and a float64
| Middleware | Throughput (loopback) (send) MBit/sec | Throughput (loopback) (recv) MBit/sec | Throughput (network) (send) MBit/sec | Throughput (network) (recv) MBit/sec |
|---|---|---|---|---|
| Ice for .NET | 380 | 300 | 300 | 205 |
| Microsoft WCF with binary encoding | 29 | 28 | 31 | 29 |
| Microsoft WCF with SOAP encoding | 21 | 21 | 23 | 24 |
| Ice for Java | 138 | 118 | 252 | 148 |
| Java RMI | 48 | 52 | 66 | 66 |