ITcpMsgHandler

Implemented by applications in order to both read and write an ordered sequence of variable length messages on a single connected TCP socket. Typically both ends of a connection will implement a ITcpMsgHandler. The ITcpMsgHandler is only a part of the story. It is intended to be used with a TcpMsgConnection which is implemented by cxMessage.dll

It is possible that the supported message protocol isn't symmetric. i.e. the messages sent by a peer may have a different format to the messages that are received by that same peer. Actually, as far as this layer is concerned, there is no concept of a message protocol or format. A message is only considered to be a contiguous block of memory of a given size, regarded as a sequence of octets. The size can be zero, in which case the address of the block of memory is immaterial.

Both the reading and writing of messages is passive, in the sense that the handler is asked to read or write the next message. This would suggest the sending of messages is based on polling. However that is not really the case. The TcpMsgConnection won't make calls on the handler to write messages until it is prompted to do so. More specifically, when messages become available to be sent, the application should call NotifyMoreMessagesToWrite() on the TcpMsgConnection. The TcpMsgConnection will continue making requests on the handler to write messages until there is a socket error or until the handler indicates there are currently no more messages available to send. That puts the TcpMsgConnection back to sleep as it were, and will only wake up again with another call to NotifyMoreMessagesToWrite().

Processing entire messages in memory

A TcpMsgConnection reads each message in its entirety from a given socket into a contiguous buffer in memory, allowing the messages to be processed very efficiently, e.g. deserialised into data structures in memory using an InputArchive (which is a thin wrapper on a pointer that is stepped though memory as values are read).

Since messages are deserialised from memory independently of the socket there are significant benefits:

• Deserialisation is buffered which leads to better performance by avoiding overheads as simple types like int32 are deserialised (e.g. recv() calls on a socket are very expensive). In fact reads from the buffer become a few inlined machine code instructions, even avoiding the overhead of a function call.
• Buffered reads allow for peaking ahead, executing deserialisation more than once etc
• Buffered reads allow for shared read access and therefore opens up possibilities for concurrency.
• Since the entire message is read into memory there cannot be a buffer underflow while reading the message. Buffer underflow checks can represent a significant deserialisation overhead.
• Per message validation (e.g. using a hash) can optionally be performed after reading the message from the socket and before deserialisation begins. It is also possible to optionally run a full blown message validation (that actually performs deserialisation but doesn't update any objects in memory).
• Deserialisaion is infallible. Reading from the buffer in memory cannot throw exceptions. (quite unlike I/O on a socket which is fallible, and therefore requires error checking and error handling and all the overheads and complexity that this entails). This can avoid the need to implement roll-back (which might otherwise be needed if updating objects while reading from a socket, and for some reason the socket connection is lost)
• Deserialisation may require locks on shared resources (like a database). Separating I/O from deserialisation allows the time for which locks are held to be minimised and more importantly to be unaffected by the network.

It is possible that the TcpMsgConnection is used to read messages that are only sent intermittently. Therefore, it is important that messages are processed as soon as they are received (i.e. without waiting for receive buffers to be filled). This is made possible by having each message begin with the size of the message in octets.

struct ITcpMsgHandler
{
   // Reading
    virtual OsErrorCode ReadMessage(const octet_t* msgBuffer, ssize_t msgSize) = 0;
    virtual void OnReadFailure(TcpMsgConnection* c, SocketErrorCode ec) = 0;

   // Writing
    virtual ssize_t GetNextMessageSize() = 0;
    virtual void WriteMessage(octet_t* msgBuffer, ssize_t msgSize) = 0;
    virtual void OnWriteFailure(TcpMsgConnection* c, SocketErrorCode ec) = 0;

    virtual void OnRelease(TcpMsgConnection* c) = 0;
};

---

OsErrorCode ReadMessage(const octet_t* msgBuffer, ssize_t msgSize)

Called by an IOCP worker thread after the next message has been received (i.e. landed in its entirety in a contiguous buffer in user space memory).

Typically the handler will process the received message in some fashion. It can be assumed that calls to ReadMessage() are made sequentially according to the order in which the messages were sent by the peer.

Note that if msgSize == 0 then msgBuffer might be nullptr.

Return OEC_ok to indicate that the message was read successfully. Otherwise return an error code to indicate a fatal error deserialising the message (e.g. because of an invalid format). This will stop the TcpMsgConnection from reading more messages from the socket, and result in a call to OnReadFailure() with ec.sc = SC_BadMessage and ec.windowsErrorCode = value returned by ReadMessage().

Must not throw an exception.

---

void OnReadFailure(TcpMsgConnection* c, SocketErrorCode ec)

Called exactly once (by an IOCP worker thread) when the last call to ReadMessage() has been made and no further reading is possible - either because of an end-of-stream indicator or an error was encountered

Possible values of ec.sc are:

• SC_WSARecv
• SC_DequeueWSARecv
• SC_ValidEndOfStream
• SC_InvalidEndOfStream
• SC_InvalidMessageSize
• SC_BadMessage

SC_BadMessage results from a call to ReadMessage() that returned false.

End of stream either results in SC_ValidEndOfStream or SC_InvalidEndOfStream depending on whether it was encountered part way through reading a message.

ec.windowsErrorCode is always 0 for the following values of ec.sc:

• SC_ValidEndOfStream
• SC_InvalidEndOfStream
• SC_InvalidMessageSize
• SC_BadMessage

Must not throw an exception.

---

ssize_t GetNextMessageSize()

Called before each call to WriteMessage() to retrieve the size of the next message to be sent. This allows the TcpMsgConnection to ensure appropriate sized buffer is allocated in memory in order to call WriteMessage() with a contiguous buffer.

Returns either:

>= 0 : The size of the next message

-1 : There is currently no message ready to be sent. This will put message writing back to sleep, and will only be restarted when NotifyMoreMessagesToWrite() is called by the application on the TcpMsgConnection.

-2 : There is no message to be sent, and there never will be, so it is appropriate for this to be indicated on the socket. This is part of what's needed for a "graceful shutdown", allowing the peer to read messages from the socket until reading the end-of-stream indicator.

Must not throw an exception.

---

void WriteMessage(octet_t* msgBuffer, ssize_t msgSize)

Called immediately after each successful call to GetNextMessageSize() (i.e. that doesn't return -1 or -2), in order to write the message of size msgSize to the given buffer. 'msgSize' is the value returned from the preceding call to GetNextMessageSize().

Must not throw an exception.

---

void OnWriteFailure(TcpMsgConnection* c, SocketErrorCode ec)

Called exactly once for every call to NotifyMoreMessagesToWrite() that returns true.

Indicates either there was a fatal error associated with writing to the socket, or writing was stopped for the time being because GetNextMessageSize() returned -1, or writing stopped forever because GetNextMessageSize() returned -2.

Possible values of ec.sc are:

• SC_WSASend
• SC_DequeueWSASend
• SC_PostQueuedCompletionStatus
• SC_WriteTemporarilyStopped
• SC_WriteShutdown

SC_WSASend,SC_DequeueWSASend,SC_PostQueuedCompletionStatus indicate a fatal error associated with writing to the socket, and more specific information can be obtained with ec.windowsErrorCode.

SC_WriteTemporarilyStopped indicates that writing was stopped for the time being because GetNextMessageSize() returned -1.

SC_WriteShutdown indicates that writing was shutdown permanently because GetNextMessageSize() returned -2.

This function is only called with SC_WriteTemporarilyStopped or SC_WriteShutdown after the writes have been completed at the interface level of the socket handle. It is therefore possible to subsequently close the socket without losing the sent messages.

---

void OnRelease(TcpMsgConnection* c)

When Close() is called on the TcpMsgConnection, OnRelease() is called on the associated ITcpMsgHandler.

OnRelease() is the last method to be called on this interface. It is safe for OnRelease() to delete this ITcpMsgHandler.