MultiplexedMsgConnection

A MultiplexedMsgConnection provides a layer on top of TcpMsgConnection in order to support multiplexing of messages.

To allow for the one socket connection to be used simultaneously by different modules in the system we allow for multiplexing of messages. i.e. it is assumed that multiple and independent message protocols may want to share the same communication channel (i.e. octet stream).

Multiplexing is only at the granularity of whole messages. i.e. messages are not divided up into pieces that are interleaved with parts of other messages.

Multiplexing is achieved by having a concept of an integer MessageId on each message, and this provides the basis for passing the message on to the appropriate handler - i.e. "demultiplexing" can be achieved because the handlers are assigned mutually exclusive ranges of MessageIds.

A handler for received messages implements the interface IMessageReader. The MessageId isn't just on the wire, every message begins with the MessageId serialised as a variable length integer. used to identify the handler, it is also used within the handler to distinguish different kinds of messages.

It is assumed each message writer or reader supports a fixed number of messages. The number isn't necessarily determined at compile time, but it must be fixed for the duration of the session. Also for a given session a matching writer/reader pair must agree on the number of messages they support.

It is assumed a MultiplexedMsgConnection is created to manage both reading and writing on a connected socket. There will normally be one MultiplexedMsgConnection at each end of a connection.

A local MultiplexedMsgConnection records an ordered list of message writers. The remote MultiplexedMsgConnection records a corresponding ordered list of message readers. So for each message writer there is a matching message reader (i.e. they are paired between sender and receiver). They share the channel by using identical offsets on the MessageId.

The following functions are used to add all the message readers and writers to a given MultiplexedMsgConnection. This must be done before sending/receiving messages can begin.


    void AddWriter(MultiplexedMsgConnection* c, IMessageWriter* w, ssize_t numMessages);
    void AddReader(MultiplexedMsgConnection* c, IMessageReader* r, ssize_t numMessages);

Note that the number of messages handled by a given IMessageWriter or IMessageReader must be specified. This is fixed for the life of the session.

On that basis they are assigned mutually exclusive ranges of MessageIds, starting from 0 and The IMessageReaders are recorded in an array, in the order they are added with calls to AddReader(). increasing as they are added.

    MessageId   0
                -------------------------------------------------------------------->

                [-----------)[--------------)[--------------------------)
                   handler       handler              handler
                      0             1                    2

Since the values monotone increase the array is sorted and therefore allows for binary search. This makes it very efficient to look up the message handler for an incoming message.

The IMessageWriters are assigned ranges of MessageIds in likewise fashion.

However, as far as a IMessageReader or IMessageWriter is concerned, the MessageIds begin at index position 0. i.e. the offsets to the MessageId is handled purely by the MultiplexedMsgConnection.

We don't allow IMessageWriters to concurrently prepare messages in memory. We assume at most one thread prepares a message in memory at a time. This leads to lower memory footprint and lower latency because messages are only prepared when the socket is ready to accept them. Also it can be possible to avoid mutexes. The IMessageWriters are registered in priority order – so a lower priority writer cannot run until all high priority writers have nothing more to write. A disadvantage is that a high priority IMessageWriters waiting on a mutex ends up blocking lower priority IMessageWriters.

If any of the writers have more to send, then

NotifyMoreMessagesToWrite(MultiplexedMsgConnection*)

should be called. This leads to calls on all the writers in priority order until they have all indicated there are no more messages to write.

It is guaranteed that ReleaseWriter() is the last method called on each IMessageWriter, and ReleaseReader() is the last method called on each IMessageReader.

`typedef ssize_t MessageId`

`MultiplexedMsgConnection* CreateMultiplexedMsgConnection()`

Create a MultiplexedMsgConnection. Must be paired with a call to Close(MultiplexedMsgConnection*). Never returns nullptr.

`void* GetSession(MultiplexedMsgConnection* c)`

Often applications will associate a MultiplexedMsgConnection with some application defined session object (in a one-to-one correspondence). The MultiplexedMsgConnection allows for recording a pointer to its associated session object.

`void SetSession(MultiplexedMsgConnection* c, void* session)`

void AddWriter(
    MultiplexedMsgConnection* c,
    IMessageWriter* w,
    ssize_t numMessages)

Allow the application to specify all the IMessageWriters that will be used to send messages.

Must be called before the call to OpenConnection().

AddWriter() and AddReader() are not threadsafe. The application should call these functions just after a call to CreateMultiplexedMsgConnection() in order to prepare the MultiplexedMsgConnection to be used.

void AddReader(
    MultiplexedMsgConnection* c,
    IMessageReader* r,
    ssize_t numMessages)

Allow the application to specify all the IMessageReaders that will be used to receive messages.

Must be called before the call to OpenConnection().

void OpenConnection(
        MultiplexedMsgConnection* c,
        Iocp* iocp,
        boost::asio::ip::tcp::socket* s,
        const MessageReaderSettings& readerSettings,
        const MessageWriterSettings& writerSettings)

Must be called at most once, and before any call to NotifyMoreMessagesToWrite()

`void StartReading(MultiplexedMsgConnection* c, IMultiplexedMsgHandler* handler)`

Must be called before the first call to NotifyMoreMessagesToWrite()

`bool HaveStartedReading(MultiplexedMsgConnection* c)`

Can be used to test whether it is safe to call NotifyMoreMessagesToWrite()

`bool NotifyMoreMessagesToWrite(MultiplexedMsgConnection* c)`

Threadsafe, can be called before OpenConnection() or StartReading() have been called

`void RequestClose(MultiplexedMsgConnection* c)`

Calls OnRequestClose on the associated IMultiplexedMsgHandler. Must be called after StartReading has been called.