61 Deprecated DGIndepNodeForAsyncPref

Status: This chapter records a deprecated ROT design and is retained for historical reference.

DGIndepNodeForAsyncPref

Instantiated for each OID for which an asychronous I/O is performed.

Represents

    -   an independent node of the DGS
    -   an async job (one of the following)
        -   a queued job to be executed
        -   a job being executed by an I/O worker thread, to load the serial element into memory
        -   a job being executed by a pure CPU thread, to deserialise the serial element

todo: instances of DGIndepNodeForAsyncPref tend to be allocated in bursts, and a pool might be useful to avoid too many heap allocations.

Non-relocability of DGIndepNodeForAsyncPref instances

Note that both DGIndepNode and DGDepNode objects that take part in the DGS are not relocatable, because edges of the dependency graph contain pointers to the nodes. i.e. the pointers m_in, m_out in a DGConnector:

struct DGConnector
{
    const DGBaseNode* m_in;
    DGConnector* m_prevIn;
    DGConnector* m_nextIn;

    const DGDepNode* m_out;
    DGConnector* m_prevOut;
    DGConnector* m_nextOut;
};

In this implementation, DGIndepNodeForAsyncPref nodes are heap allocated and referenced by a std::map keyed by OID in the ROT, so relocatability isn't an issue.

                                             +-------------+
                                             | DGIndepNode |
                                             +-------------+
                                                   /|\
                                                    |
                                                    |
        +------------+             *   +-------------------------+
        |    ROT     |---------->------| DGIndepNodeForAsyncPref |
        +------------+                 +-------------------------+

Lifetime management

Previously DGIndepNodeForAsyncPref::OnNoMoreOutnodes() was called when the indep node became inactive. But we no longer have that concept. DGIndepNodeForAsyncPref is initialised with a ref count of 2 and is deleted when the count falls to 0.

Instead DGIndepNodeForAsyncPref::OnEvict() takes on that roll. We assume the DGS framework guarantees OnEvict() is called on every DGS node

Consider that a DGIndepNodeForAsyncPref has been created for a given OID and recorded in the map, and the async I/O and deserialisation and has been initiated.

If this completes then we want to:

  • Update the ROT, replacing the pointer to the DGIndepNodeForAsyncPref with the ptr<IPersistable>
  • Invalidate the outnodes of the DGIndepNodeForAsyncPref. These will become hard-dirty and therefore they will detach from the DGIndepNodeForAsyncPref
  • delete the DGIndepNodeForAsyncPref. It will automatically remove itself from the DGS eviction queue.

But to do this we actually:

    void OnFinished()
    {
        CSpaceLock lock;

        < update the ROT >

        // This
        //    -   invalidates the outnodes
        //    -   clears DF_IN_EVICTION_QUEUE bit
        //    -   remove node from the eviction queue
        //    -   calls OnEvict()
        TryEvict();
    }

    void OnEvict()
    {
        assert(HaveLock());

        if (!updatedROT)
        {
            // Cancel async operations if possible
            abort_ = true;

            < Update the ROT, erasing the entry for the given OID >

        }
        DecRef();
    }

DGIndepNodeForAsyncPref.h

// DGIndepNodeForAsyncPref.h
//
// Author David Barrett-Lennard
// (C)opyright Cedanet Pty Ltd 2011

@import "IPersistable.h"
@import "IPersistStore.h"
@import "Ceda/cxObject/DGNode.h"
#include "Ceda/cxLss/ILogStructuredStore.h"
#include "Ceda/cxUtils/Magic.h"
#include <map>
#include <atomic>

namespace ceda
{
class ROT;

class DGIndepNodeForAsyncPref final : public DGIndepNode
{
    cxNotCloneable(DGIndepNodeForAsyncPref)
public:
    DGIndepNodeForAsyncPref(ROT& rot, OID oid);
    ~DGIndepNodeForAsyncPref();

    // Implementation of DGIndepNode
    virtual ssize_t ByteSize() const { return 4; }     // what should happen here?
    virtual void VisitContainingObject(IObjectVisitor& v) const {}
    virtual void OnEvict() const;
    virtual xstring Name() const;

    /*
    Read a serial element from the LSS into a contiguous memory block then queue a pure CPU task
    to deserialise the serial element into a dynamically created IPersistable object.
    Called by the AsyncObjectLoader worker thread.
    This is essentially a pure I/O task which returns before completing the deserialise CPU task
    */
    void AsyncLoad();

    void AsyncUpdateRot();

    // Called with a CSpace lock, typically via implementor of IAsyncBindRequestHandler when it calls
    // MakeObjectPersistWithGivenOID().
    void SetObject(ptr<IPersistable> po);

private:
    // Threadsafe.  The ref count is initially 2, so the second call to DecRef() results in
    // 'delete this'
    void DecRef() const;
    friend void DecRef(DGIndepNodeForAsyncPref*);

    bool HandleAsyncLoad(ptr<IPersistable> po);

    void UpdateRot();

    /*
    Perform the async deserialisation of the serial element assumed to have already been read
    from the LSS.
    This is a pure CPU task
    */
    bool AsyncDeserialise();

private:
    friend class ROT;

    MagicTestDestructed magic_;
    ROT& rot_;

    // The oid of the object to be async loaded.  Initialied in the ctor and not changed from
    // that point
    const OID oid_;

    /*
    The DGIndepNodeForAsyncPref is ref counted, so it can be deleted when both
        1.   The queued AsyncLoad()/AsyncDeserialise() functions have been called; and
        2.   OnEvict() has been called because the node is inactive
    Note that these two conditions can be met in any order, so we use a thread safe ref count.

    A node is only created as it becomes active.  It eventually becomes inactive and cannot
    become active again.  Instead a different node is created.
    */
    mutable std::atomic<int> refCount_;

    // Returned by the LSS to support reading a serial element as a contiguous buffer in memory
    IContiguousSerialElement* cse_;

    // The serial element as a contiguous buffer in memory
    ReadOnlyBuffer rob_;

    bool needEndAsyncBindRequest_;      // A pending call to EndAsyncBindRequest() is required

    /*
    Only assigned by a thread with a CSpace lock.  However read by a thread in order to
    quickly abort a redundant calculation.
    */
    mutable volatile bool abort_;

    /////////////////////// State protected by CSpace lock //////////////////////////

    /*
    Iterator to the associated entry in the ROT map, to allow the mapped value to be
    updated efficiently when the async load has completed, or to allow the entry to be erased
    from the ROT.
    */
    std::map<OID,ptr<IPersistable> >::iterator rotIterator_;

    /*
    Set by a thread with a CSpace lock that takes on responsibility for updating the ROT
    using rotIterator_.
    */
    enum EUpdatedRot
    {
        UR_PENDING,                 // Initial state
        UR_OBJECT_FOUND,            // Object found in LSS, have updated ROT map using the iterator
        UR_OBJECT_DOES_NOT_EXIST,   // It has been determined that the object doesn't exist
        UR_NODE_EVICTED,            // OnEvict() has been called
    };
    mutable EUpdatedRot updatedRot_;

    /////////////////////// State protected by OID lock //////////////////////////
    // The following state is only accessed by a thread that has locked ROT::GetMutexForOid(oid_).

    /*
    Set if ROT::FindOrLoadResidentObject() has found that this async load is already in progress,
    and therefore is waiting on the event ROT::blockSyncThreadWaitingOnAsyncLoad_.
    */
    bool waiting_;

    // Set once asyncobj_ has been assigned with its final value (which may be null)
    bool readyToUpdateRot_;

    // Records the result of the AsyncLoad (which may be null if the object was not found in the
    // LSS)
    // While there is a working thread loading the object from the LSS this member is protected
    // by the oid mutex.  However once this has been completed this member is protected by the
    // CSpace mutex.
    mutable ptr<IPersistable> asyncobj_;
};
} // namespace ceda

DGIndepNodeForAsyncPref.cpp

// DGIndepNodeForAsyncPref.cpp
//
// Author David Barrett-Lennard
// (C)opyright Cedanet Pty Ltd 2011

@import "DGIndepNodeForAsyncPref.h"
@import "ROT.h"
@import "PSpace.h"
@import "PersistStore.h"
#include "Ceda/cxLss/LssExceptions.h"
#include "Ceda/cxUtils/Tracer.h"
#include "Ceda/cxUtils/FileException.h"

/*
Testing of async loading
------------------------

It seems appropriate to use class Node in lsPersistStore.  In order for async loading to
actually occur an active agent is required.  No other dependents are needed.

A node can store many children - e.g. 1000 children.  Therefore it can be used in a flat way
where we have lots of leaf nodes which are async loaded at random.

So, when agent is calculated, we pick a random number of children.  For each child we call the
async bind function. This causes lots of edges to form - and we have the effect that many edges
are broken and reformed all the time - so we really hammer the aborting and restarting of
async I/O tasks.

It is appropriate to sometimes wait until a few invalidations have occurred before the agent
is calculated again.
*/

namespace ceda
{
@def bool trace_DGIndepNodeForAsyncPref = false

DGIndepNodeForAsyncPref::DGIndepNodeForAsyncPref(ROT& rot, OID oid) :
    rot_(rot),
    oid_(oid),
    refCount_(2),      // Delete once both AsyncLoad() and OnEvict() have been called.
    cse_(nullptr),
    needEndAsyncBindRequest_(false),
    abort_(false),
    updatedRot_(UR_PENDING),
    waiting_(false),
    readyToUpdateRot_(false)
{
    @if (trace_DGIndepNodeForAsyncPref)
    {
        Tracer() << "DGIndepNodeForAsyncPref() oid= " << oid_ << '\n';
    }

    // DGIndepNodeForAsyncPref should never be created with a null oid
    cxAssert(oid_);

    cxAssert(!asyncobj_);

    ++rot_.blockUntilNoMoreAsyncNodes_;
}

DGIndepNodeForAsyncPref::~DGIndepNodeForAsyncPref()
{
    /*
    Note that we cannot predict whether this thread has locked the CSpace at this point.  This
    is because DecRef() is only sometimes called with a CSpace lock.
    */

    @if (trace_DGIndepNodeForAsyncPref)
    {
        Tracer() << "~DGIndepNodeForAsyncPref() oid= " << oid_ << '\n';
    }

    // asyncobj_ owns the object it points at.  This is convenient to avoid memory leaks in certain
    // cases where an object is loaded from the LSS but for whatever reason it doesn't end up in
    // the ROT.
    if (asyncobj_)
    {
        /*
        This object will have an oid assigned and yet it has not been registered in the CSpace

        DynCreateDeserialise() assigns an oid to the object before it is deserialised to allow an
        object to trigger its own schema evolution when it is deserialised.  In such cases the object is
        marked as dirty and place in the Dirty Object Set.

        We cannot allow this to happen on ROD objects, since ROD objects don't necessarily end up in
        the ROT but instead are deleted here.

        cxAssert(!GetIObjectFlag(asyncobj_,DBP_PO_DIRTY));
        */
        @if (trace_DGIndepNodeForAsyncPref)
        {
            Tracer() << "Destroying asyncobj_ = " << asyncobj_ << '\n';
        }
        asyncobj_->Destroy();
    }

    if (needEndAsyncBindRequest_)
    {
        PSpace& pspace = rot_.pspace_;
        SetThreadPtr<PSpace>(&pspace);
        SetThreadPtr<CSpace>(GetCSpace(&pspace));
        CSpaceLock lock;
        if (pspace.asyncBindRequestHandler_)
        {
            pspace.asyncBindRequestHandler_->EndAsyncBindRequest(oid_);
        }
    }

    if (cse_)
    {
        //--s_numReads;
        //Tracer() << s_numReads << "  END READ " << oid_ << '\n';
        cse_->Close();
    }

    --rot_.blockUntilNoMoreAsyncNodes_;
}

void DGIndepNodeForAsyncPref::DecRef() const
{
    if (--refCount_ == 0)
    {
        delete this;
    }
}

xstring DGIndepNodeForAsyncPref::Name() const
{
    return cxMakeString("DGIndepNodeForAsyncPref:oid=" << oid_);
}

inline void DecRef(DGIndepNodeForAsyncPref* p)
{
    p->DecRef();
}

// Perform the actual async load.  Called by the AsyncObjectLoader worker thread without a CSpace
// lock
void DGIndepNodeForAsyncPref::AsyncLoad()
{
    @if (trace_DGIndepNodeForAsyncPref)
    {
        TraceIndenter fullIndent(20);
        Tracer() << "DGIndepNodeForAsyncPref::AsyncLoad() oid= " << oid_ << '\n';
        TraceIndenter indent;
    }

    PSpace* pspace = &rot_.pspace_;
    SetThreadPtr<PSpace>(pspace);
    SetThreadPtr<CSpace>(GetCSpace(pspace));

    cxAssert(!rot_.HasLock());

    // For every call to AsyncLoad() there needs to be a call to DecRef().
    AutoDecRef<DGIndepNodeForAsyncPref> adr(this);

    if (!abort_)
    {
        /*
        Read a contiguous serial element from the LSS.  Provides read only access to a
        contiguous buffer in memory.
        */
        try
        {
            ILogStructuredStore* lss = GetLss(&pspace->m_ps);
            cxAssert(lss);

            // todo: Ideally the LSS would support aborting of the I/O.  Indeed ideally
            // we wouldn't even have a thread block on I/O in the first place - then there's
            // nothing to abort.
            // There are other reasons to want the LSS to support asynchronous reading of serial
            // elements:  it will allow the LSS to use an elevator seeking algorithm.

            cxAssert(cse_ == nullptr);
            //++s_numReads;
            //Tracer() << s_numReads << "  Read " << oid_ << '\n';
            cse_ = lss->ReadContiguousSerialElement(oid_);

            if (cse_)
            {
                rob_ = cse_->GetBuffer();

                // todo: Queue 100% CPU task which is to deserialise the serial element
                // For now we call the deserialise function directly, but later we will queue this
                // task with a thread pool.
                bool handlingDecRef = AsyncDeserialise();
                if (handlingDecRef)
                {
                    // AsyncDeserialise() took on the responsibility to call DecRef()
                    adr.Release();
                }
            }
            else
            {
                /*
                The object doesn't exist in the LSS.

                If there is a IAsyncBindRequestHandler then we need to call BeginAsyncBindRequest()
                on it.  This must be done with a CSpace lock.  But we can't get a CSpace lock here,
                or else there is a deadlock scenario:

                1.  A thread has locked the CSpace and called ROT::FindOrLoadResidentObject().
                    In this function the thread is waiting for the blockSyncThreadWaitingOnAsyncLoad_
                    event to be signalled.

                2.  This async loader worker thread is blocking trying to get a CSpace lock in
                    order to call BeginAsyncBindRequest()

                It is important to signal the blockSyncThreadWaitingOnAsyncLoad_ event before
                locking the CSpace in order to call BeginAsyncBindRequest().  We achieve this by
                calling BeginAsyncBindRequest() inside DGIndepNodeForAsyncPref::UpdateRot() instead
                of trying to call it here.
                */
                bool handlingDecRef = HandleAsyncLoad(null);
                if (handlingDecRef)
                {
                    // HandleAsyncLoad() took on the responsibility to call DecRef()
                    adr.Release();
                }
            }
        }
        catch(CorruptLSSException&)
        {
            // todo: If there is a thread with a CSpace lock blocking on ROT::blockSyncThreadWaitingOnAsyncLoad_
            // then that thread must be made to throw this exception!
            cxAssert(0);
        }
        catch(FileException&)
        {
            // todo: If there is a thread with a CSpace lock blocking on ROT::blockSyncThreadWaitingOnAsyncLoad_
            // then that thread must be made to throw this exception!
            cxAssert(0);
        }
    }
}

bool DGIndepNodeForAsyncPref::AsyncDeserialise()
{
    @if (trace_DGIndepNodeForAsyncPref)
    {
        Tracer() << "DGIndepNodeForAsyncPref::AsyncDeserialise() oid= " << oid_ << '\n';
        TraceIndenter indent;
    }

    if (abort_)
    {
        return false;    // Caller needs to DecRef()
    }
    else
    {
        // A non-empty serial element of the LSS should already have been read into a
        // contiguous buffer in memory.
        cxAssert(rob_.buffer);
        cxAssert(rob_.size > 0);

        /*
        todo: actually DynCreateDeserialise() can throw an exception after allocating the
        object from the heap, so really we should avoid a local variable which won't delete
        the object as the stack unwinds.  Otherwise we have a memory leak in the presence of
        exceptions.
        */
        ptr<IPersistable> po;

        // DynCreateDeserialise() heap allocates the IPersistable object, but doesn't register
        // it in a CSpace.  It then deserialises the state from the archive.
        // todo: may throw exceptions.
        rot_.pspace_.m_ps.DynCreateDeserialise(rob_.buffer, rob_.buffer + rob_.size, po, oid_,false);

        cxAssert(cse_);
        //--s_numReads;
        //Tracer() << s_numReads << "  End read " << oid_ << '\n';
        cse_->Close();
        cse_ = nullptr;

        @if (trace_DGIndepNodeForAsyncPref)
        {
            Tracer() << "ROT: Async Loaded object " << po << " with OID = " << oid_ << " from disk\n";
        }

        return HandleAsyncLoad(po);
    }
}

bool DGIndepNodeForAsyncPref::HandleAsyncLoad(ptr<IPersistable> po)
{
    @if (trace_DGIndepNodeForAsyncPref)
    {
        Tracer() << "DGIndepNodeForAsyncPref::HandleAsyncLoad() oid= " << oid_ << " po = " << po << '\n';
        TraceIndenter indent;
    }

    // Note that po can be null, indicating that the attempted async load failed because the object
    // doesn't exist in the LSS.

    @if (CEDA_VALIDATE_MARK_AS_DIRTY)
    {
        if (po)
        {
            SetCrc32(po, CalculateCrc(po));
            @if (trace_CRCValidationChecks)
            {
                Tracer() << "After load object CRC = " << GetCrc32(po) << '\n';
            }
        }
    }

    /*
    We can't immediately get a CSpace lock or else we can dead-lock - because another
    thread that has already acquired the lock may be blocked on the event
    blockSyncThreadWaitingOnAsyncLoad_, waiting for it to be signalled when the
    asynchonous I/O has completed.
    */
    bool waiting;
    {
        std::lock_guard<std::mutex> lock(rot_.GetMutexForOid(oid_));  // Lock mutex for given oid

        // Transfer ownership from po to asyncobj_
        asyncobj_ = po;
        readyToUpdateRot_ = true;

        waiting = waiting_;
    }
    if (waiting)
    {
        /*
        A thread which has locked the CSpace is currently waiting on event
        blockSyncThreadWaitingOnAsyncLoad_.
        We choose to handball the responsibility to make the required updates to the
        ROT and DGS to the thread that is blocked and already has a CSpace lock.
        */
        @if (trace_DGIndepNodeForAsyncPref)
        {
            Tracer() << "Another thread was waiting so signaling event\n";
        }
        rot_.blockSyncThreadWaitingOnAsyncLoad_.Signal();
    }
    else
    {
        /*
        It is not possible to call AsyncUpdateRot() directly because it leads to the following
        dead-lock scenario:

            1.  There are async bind requests for oid1, oid2.  So two calls to AsyncLoad()
                are queued with the AsyncLoadThread.

            2.  A thread locks a CSpace and makes a synchronous request for oid2.  This cannot
                proceed until AsyncLoadThread processes oid2.

            3.  AsyncLoad() is called for oid1, the object is loaded from the LSS, deserialised
                and the thread blocks on getting a CSpace lock in order to register the oid1
                object in the CSpace.

        The requirement to call DecRef() is effectively passed on to the queued task to update
        the ROT.
        */
        rot_.pspace_.m_ps.asyncUpdateRotThread_.Push(this);
        return true;    // Caller doesn't need to DecRef()
    }
    return false;       // Caller needs to DecRef()
}

/*
Example scenarios:

A)
    1.  DGIndepNodeForAsyncPref is created for active indep node, asyncload job is posted

    2.  AsyncLoad() called, loads serial element from LSS

    3.  AsyncDeserialise() called, DynCreateDeserialise successfuly creates po and intialises state
        from serial element.

    4.  HandleAsyncLoad(po) called

    5.  In OID mutex
            asyncobj_ = po;
            readyToUpdateRot_ = true;
        waiting false so push AsyncUpdateRot() call

    6.  AsyncUpdateRot() called, gets ceda lock then calls UpdateRot()

    7.  UpdateRot() called with updatedRot_ == UR_PENDING.

B)  Like A) except that waiting true so event is signalled and the thread blocking on the event calls
    UpdateRot() instead.

C)  Like A) except that node becomes inactive after having assigned asyncobj_ but before the call to
    UpdateRot().
*/

void DGIndepNodeForAsyncPref::UpdateRot()
{
    cxAssert(rot_.HasLock());
    cxAssert(readyToUpdateRot_);

    if (updatedRot_ == UR_PENDING)
    {
        @if (trace_DGIndepNodeForAsyncPref)
        {
            Tracer() << "Updating ROT after asynchronous I/O\n";
            TraceIndenter indent;
        }

        /*
        UpdateRot() is only called with a CSpace lock after having assigned asyncobj_.  It is
        assumed it is no longer necessary to lock the oid mutex when accessing asyncobj_.  Instead
        asyncobj_ is now protected by the CSpace mutex.
        */
        // No need to lock oid mutex now!
        //std::lock_guard<std::mutex> lock(rot_.GetMutexForOid(oid_));  // Lock mutex for given oid

        if (!asyncobj_)
        {
            // There is no serial element with the given oid in the LSS.
            if (rot_.pspace_.asyncBindRequestHandler_)
            {
                // Allow a hook to take on responsibility for binding to the object.
                MAsyncBind ab = rot_.pspace_.asyncBindRequestHandler_->BeginAsyncBindRequest(oid_);
                if (ab.result == EAsyncBindResult::Bound)
                {
                    cxAssert(ab.po);
                    asyncobj_ = ab.po;
                }
                else if (ab.result == EAsyncBindResult::Pending)
                {
                    cxAssert(updatedRot_ == UR_PENDING);

                    // Ensure each call to BeginAsyncBindRequest() that returns true is paired
                    // with a call to EndAsyncBindRequest()
                    needEndAsyncBindRequest_ = true;
                }
                else
                {
                    cxAssert (ab.result == EAsyncBindResult::DoesNotExist ||
                              ab.result == EAsyncBindResult::NotHandled);
                    updatedRot_ = UR_OBJECT_DOES_NOT_EXIST;
                    OnChange();
                }
            }
            else
            {
                // There is no hook for handling missing objects in the LSS so therefore we assume
                // the object doesn't exist.
                updatedRot_ = UR_OBJECT_DOES_NOT_EXIST;
                OnChange();
            }
        }

        if (asyncobj_)
        {
            /*
            updatedRot_ must be set before the call to TryEvict()
            since the latter results in a call to OnEvict() which checks updatedRot_.
            */
            updatedRot_ = UR_OBJECT_FOUND;

            RegisterGcObject(asyncobj_);

            // Replace the mapped value with the ptr<IPersistable> of the object that has been loaded.
            rotIterator_->second = asyncobj_;

            asyncobj_ = null;

            // Invalidate the outnodes and detach them
            // This synchronously results in a call to OnEvict() and therefore may cause the node to
            // be deleted
            cxVerify( TryEvict(false) );
        }
    }
    else
    {
        @if (trace_DGIndepNodeForAsyncPref)
        {
            Tracer() << "ROT already updated\n";
        }
    }
}

void DGIndepNodeForAsyncPref::AsyncUpdateRot()
{
    @if (trace_DGIndepNodeForAsyncPref)
    {
        TraceIndenter fullIndent(40);
        Tracer() << "DGIndepNodeForAsyncPref::AsyncUpdateRot() oid = " << oid_ << '\n';
    }

    PSpace* pspace = &rot_.pspace_;
    CSpace* cspace = GetCSpace(pspace);
    cxAssert(cspace);

    SetThreadPtr<PSpace>(pspace);
    SetThreadPtr<CSpace>(cspace);

    CSpaceLock lock(cspace);
    UpdateRot();

    @if (trace_DGIndepNodeForAsyncPref)
    {
        Tracer() << "DecRef on DGIndepNodeForAsyncPref at end of AsyncUpdateRot()\n";
    }
    DecRef();
}

void DGIndepNodeForAsyncPref::SetObject(ptr<IPersistable> po)
{
    @if (trace_DGIndepNodeForAsyncPref)
    {
        TraceIndenter fullIndent(40);
        Tracer() << "DGIndepNodeForAsyncPref::SetObject() oid = " << oid_ << " po = " << po << '\n';
    }

    cxAssert(po);
    cxAssert(rot_.HasLock());

    CSpace* cspace = rot_.pspace_.m_cspace;
    cxAssert(cspace);

    // RegisterGcObject() has already been called on the object
    cxAssert(GetAssociatedCSpace(po) == cspace);

    /*
    Since we have a lock there can't be some other thread blocking on I/O in a call to
    ROT::FindOrLoadResidentObject.
    */

    abort_ = true;

    // Note that SetObject() doesn't update readyToUpdateRot_ or asyncobj_.  So no need for it to get an oid
    // mutex.

    // OnEvict() can't have been called yet
    cxAssert(updatedRot_ != UR_NODE_EVICTED);

    // Ensure any subsequent calls to AsyncUpdateRot() and OnEvict() do nothing more than
    // a DecRef()
    updatedRot_ = UR_OBJECT_FOUND;

    SetThreadPtr<CSpace>(cspace);

    // Invalidate the outnodes and detach them
    // This synchronously results in a call to OnEvict() and therefore may cause the node to
    // be deleted
    cxVerify( TryEvict(false) );
}

/*
Called when
    -   this independent node becomes unreachable from an active agent - e.g. because an agent
        was deactivated.

    -   the async load has completed, and ROT::AsyncDeserialise() has called
        TryEvict()
*/
void DGIndepNodeForAsyncPref::OnEvict() const
{
    @if (trace_DGIndepNodeForAsyncPref)
    {
        Tracer() << "DGIndepNodeForAsyncPref::OnEvict() oid= " << oid_ << '\n';
        TraceIndenter indent;
    }
    cxAssert(rot_.HasLock());

    // This node has transitioned to inactive so the job no longer needs to be performed.
    abort_ = true;

    cxAssert(updatedRot_ != UR_NODE_EVICTED);

    if (updatedRot_ == UR_PENDING)
    {
        std::lock_guard<std::mutex> lock(rot_.GetMutexForOid(oid_));  // Lock mutex for given oid

        if (asyncobj_)
        {
            // Update the rot if the object has been loaded from disk - even though the node has
            // become inactive.
            // todo : is this worth doing?
            @if (trace_DGIndepNodeForAsyncPref)
            {
                Tracer() << "Applying object on inactive node\n";
            }
            updatedRot_ = UR_OBJECT_FOUND;
            RegisterGcObject(asyncobj_);
            cxAssert(rotIterator_->second.m_self == this);
            rotIterator_->second = asyncobj_;
            asyncobj_ = null;
        }
    }

    if (updatedRot_ == UR_PENDING || updatedRot_ == UR_OBJECT_DOES_NOT_EXIST)
    {
        @if (trace_DGIndepNodeForAsyncPref)
        {
            Tracer() << "Erasing ROT entry\n";
        }
        cxAssert(rotIterator_->second.m_self == this);

        // Async task was never completed, so erase the ROT map entry.
        rot_.map_.erase(rotIterator_);
    }

    // Ensures any subsequent call to UpdateRot() does nothing.  It would be bad for example
    // to leave updatedRot_ == UR_PENDING after having erased the map entry pointed at by rotIterator_
    updatedRot_ = UR_NODE_EVICTED;

    @if (trace_DGIndepNodeForAsyncPref)
    {
        Tracer() << "DecRef on DGIndepNodeForAsyncPref at end of OnEvict()\n";
    }

    // This DGIndepNodeForAsyncPref represents a queued job, so can't just delete it.
    // Instead we decrement a thread safe ref count
    DecRef();       // Potentially performs 'delete this'
}
} // namespace ceda