59 Deprecated ROT implementation
Status: This chapter records a deprecated ROT design and is retained for historical reference.
See ROT design.txt
Links
Access to the associated PSpace
The ROT is passed a reference to the associated PSpace in its constructor and this is saved in the member variable pspace_.
class ROT
{
ROT(PSpace& pspace);
PSpace& pspace_;
};
ROT::ROT(PSpace& pspace) :
pspace_(pspace)
{
}Use of std::map
The ROT is essentially a map from OID to ptr<IPersistable>, used to index all the
persistent objects in the associcated PSpace that are currently resident in memory.
class ROT
{
typedef std::map<OID,ptr<IPersistable> > MAP;
MAP map_;
};This std::map is also used to index DGIndepNodeForAsyncPref nodes that are in the process of performing asynchronous I/O to fetch objects from the LSS.
A ptr<IPersistable> contains a pair of pointers:
$struct AnyInterface
{
void* m_self;
const void* m_table;
};
When m_table == nullptr, we assume m_self is a pointer to a DGIndepNodeForAsyncPref!
The DGIndepNodeForAsyncPref in turn records a member asyncobj_ of type ptr<IPersistable>
to hold the result of an asynchronous I/O.
This map is only accessed by a thread which has exclusive locked the CSpace.
FindResidentObject()
FindResidentObject() finds the object with given OID, or returns null if not currently resident in memory. This function never blocks on I/O (either directly or indirectly).
FindResidentObject must only be called by a thread that has a locked the PSpace. That means we can safely access the map.
Since DGIndepNodeForAsyncPref instances are recorded in the map, we have to also check m_table to ensure we actually have an object resident in memory.
ptr<IPersistable> ROT::FindResidentObject(OID oid) const
{
cxAssert(HasLock());
MAP::const_iterator i = map_.find(oid);
if (i != map_.end() && i->second.m_table)
{
return i->second;
}
else
{
return null;
}
}Eviction of the IPersistable objects from the ROT
The ROT is not itself an IObject that takes part in the CSpace GC trace, and there is no concept of visiting the IPersistable objects referenced by the ROT. In that sense they only represent weak references - meaning that they don't protect the objects from the CSpace GC.
The CSpace GC locks the CSpace associated with the PSpace in order to
- Run a trace from a set of GC roots
- Find the objects that are unreachable
- Of the unreachable objects, determine which ones haven't been accessed recently and will be deleted in order to recovery memory as needed
- Call OnGarbageCollect() on the objects to be deleted
Note that OnGarbageCollect() is called by the GC thread while it has locked the CSpace, and this is during the "mark" phase, not the subsequent "sweep" phase when the objects are actually deleted.
The trace hasn't included the "weak references" in the ROT. Therefore any IPersistable objects referenced by the ROT can easily be garbage collected.
Each IPersistable implements OnGarbageCollect() so as to remove the corresponding entry from the ROT.
// IPersistable.h
// If po has an associated PSpace then this function requires it to have been locked.
// It is not necessary to set either the CSpace or PSpace in thread local storage.
$function+ void OnGarbageCollectPersistable(ptr<IPersistable> po);
$mixin PersistableMixin
{
void OnGarbageCollect()
{
OnGarbageCollectPersistable($this);
BaseClass::OnGarbageCollect();
}
...
};
// PersistStore.cpp
$function+ void OnGarbageCollectPersistable(ptr<IPersistable> po)
{
cxAssert(po);
if (PSpace* pspace = GetAssociatedPSpace(po))
{
pspace->OnGarbageCollectPersistable(po);
}
}
void PSpace::OnGarbageCollectPersistable(ptr<IPersistable> po)
{
cxAssert(po);
// The GC thread should have an exclusive lock on the CSpace
cxAssert( HaveWriteAccess() );
// Must not evict dirty objects. Note that dirty objects should be strongly reachable via the DOS.
cxAssert(!IsPersistableDirty(po));
// Note that the GC thread calls OnGarbageCollectPersistable() on transient IPersistable
// objects that become unreachable. Therefore it is possible that the oid is null.
if (OID oid = GetOid(po))
{
rot_.RemoveResidentObject(oid);
}
}When a CSpace is destroyed, OnGarbageCollect() is not called on the remaining IObjects in the CSpace. This avoids the problem that would occur if the CSpace destructs after the ROT so that calls to RemoveResidentObject() are made after the ROT has destructed. However it means that we cannot expect the ROT to already be empty when it destructs.
RemoveResidentObject()
The CSpace GC is used to evict IPersistable objects cached in the ROT. The DGS eviction queue is not used for that purpose.
RemoveResidentObject() is called from PSpace::OnGarbageCollectPersistable(ptr<IPersistable> po)
i.e. by the CSpace garbage collector (GC) when it has locked the PSpace and has decided to evict the object with the given OID.
Since it is only called for an object resident in memory, there shouldn't be any chance that the m_table entry being erased is not nullptr
void ROT::RemoveResidentObject(OID oid)
{
// Must have locked the CSpace
cxAssert(HasLock());
cxAssert(oid);
#ifdef CEDA_CHECK_ASSERTIONS
MAP::iterator i = map_.find(oid);
cxAssert(i != map_.end());
cxAssert(i->second.m_table != nullptr);
#endif
[[maybe_unused]] ssize_t numRemoved = map_.erase(oid);
cxAssert(numRemoved == 1);
}Permanently deleting objects
When objects are permanently deleted it is necessary to ensure they are no longer referenced from the following:
- The DOS
- The ROT
- All other objects in the PSpace/CSpace
The actual C++ object will be deleted by the garbage collector. This will call OnGarbageCollect() and delete the entry from the ROT.
Implementation of the ROT
It is proposed that we comment out the existing ROT, and write an alternative that doesn't use the CacheMap, and doesn't support shared read access. Furthermore the map is protected by the CSpace mutex. Synchronous loading is trivial - simply load the object and update the ROT - there is no risk of two threads trying to load the same object at the same time.
There is a queue for async I/O requests, protected with its own mutex, so that I/O jobs can be
popped without needing a CSpace lock. As I/O jobs are completed, pure CPU deserialise jobs are
queued. When these are completed, a CSpace lock is acquired in order to update the ROT. This
involves deletion of the dependent node (and corresponding removal of edges for its outnodes),
marking outnodes as hard-dirty, and updating the mapped value of the OID to the ptr
Lifetime management
Queued I/O requests only need to record a queue of OIDs. Queued deserialise requests only need
to queue (OID, xvector
Async loading of objects from the LSS
Currently each opened instance of the LSS creates 4 worker threads for the following:
- Lazy check pointer
- Lazy flusher
- Lazy writer
- Lazy cleaner
It is proposed that the LSS only creates a single worker thread that performs all these tasks, and in addition it supports a facility for asynchronous loading of objects.
Loading objects from the LSS involves calling the method:
/*
Provides an alternative to ReadSerialElement() for reading a serial element as a contiguous
block of memory. Obviously this function shouldn't be called for very large serial elements
that don't fit in physical memory and therefore would result in page faulting.
The given Seid must not be null
The returned IContiguousSerialElement must be closed after it is used (including when
exceptions are thrown by the LSS).
Returns nullptr if no serial element exists with the given Seid
It is an error to call this function on a serial element that is currently opened for
writing (within a transaction), or being deleted using a call to DeleteSerialElement().
Shared reading of serial elements is supported. I.e. any number of threads can independently
(and concurrently) read the same serial element
*/
virtual IContiguousSerialElement* ReadContiguousSerialElement(Seid seid) const = 0;
Synchronous request for an object that is being loaded asynchonously
Consider that an object is being loaded asynchronously, when a synchronous request for that same object is made. We seem to need to make the synchronous request block on the completion of the asynchronous load. Unfortunately a complicating factor is that the blocked thread holds a CSpace lock, and this stops all async I/O from being applied to the ROT - including the very I/O that we are waiting on.
Idea 1: handball completion of task to the blocked thread
This was implemented but it is rather complicated.
It is proposed that there is state that is set during the CSpace lock to indicate that a thread
is blocking on the load of a certain OID. When each async I/O request is about to get the CSpace
lock it first checks this state to see whether it can handball the completion work to the blocked
thread that already has a CSpace lock. So in such a case it will assign a ptr
PROBLEM: How do we avoid this dead-lock: Consider that async load of the object has completed and is waiting on a CSpace lock. The thread with the CSpace lock then calls SyncGet(). If it waits on the event then dead-lock is inevitable.
IDEA: Let a hash on the OID map to a fixed array of mutexes. So we achieve concurrency with a constrained number of mutexes. This provides a basis for the ROT to manage threads for a given oid.
Idea 2: don't even support it
Another option to simplify the ROT: What if we simply disallow mixing of synchronous and asynchronous reads of a given object? i.e. for a given OID you have to pick one or the other. Putting it another way, if the synchronous mode finds that the object is already being loaded asychronously it balks.
Idea 3: use condition variables with the CSpace mutex
Consider that we introduce a condition variable to allow the thread calling FindOrLoadResidentObject() to block on the condition that the object has been loaded.
There can be any number of threads blocked in calls to FindOrLoadResidentObject(), they can be for the same OID or for different OIDs. So we need a distinct condition variable for each OID. The best way to do this would seem to be to store the condition variable in the DGIndepNodeForAsyncPref object instance for that OID. This shouldn't be an issue for memory consumption because the number of DGIndepNodeForAsyncPref instances in memory at a given time should be fairly small - they represent the currently executing async loads.
It is assumed DGIndepNodeForAsyncPref has at least the following state:
class DGIndepNodeForAsyncPref
{
std::condition_variable_any cv_;
ptr<IPersistable> obj_;
int refCount_;
};cv_ is a condition variable associated with the condition that the object has been loaded from disk.
obj_ is initially null and is set when the object is loaded from disk.
It is assumed all access to obj_ and refCount_ is by threads that have locked the CSpace.
ptr<IPersistable> ROT::FindResidentObject(OID oid) const
{
look up map_ with oid
if (DGIndepNodeForAsyncPref* i = find a DGIndepNodeForAsyncPref in the map)
{
++i->refCount_;
// Hmmm need a std::unique_lock<std::mutex> on the CSpace mutex;
i->cv_.wait(lk, []{return obj_ != null;});
if (--i->refCount_ == 0)
{
delete i;
}
}
};FindOrLoadResidentObject: If FindOrLoadResidentObject(OID oid) finds there is a DGIndepNodeForAsyncPref for the given OID in the std::map then it increments refCount_ and waits on cv_, with a predicate on obj_. After waiting for the object to be loaded, we know implicitly that it has a lock on the CSpace once again. It decrements the refCount_ and if refCount_ has fallen to zero it deletes the DGIndepNodeForAsyncPref instance.
Worker thread: The worker thread given the task to load the object with the given OID is assumed to already own a ref count on the DGIndepNodeForAsyncPref. It does the following:
- load the object from the LSS
- With a lock on the CSpace:
- update the std::map so it points at the loaded object instead of the DGIndepNodeForAsyncPref
- if (--refCount_ == 0) delete the DGIndepNodeForAsyncPref else { set obj_ to the loaded object; cv_.notify_all(); }
Idea 4: treat as an unsual corner case and don't try to reuse effort
Consider that FindOrLoadResidentObject() looks in the std::map and finds an DGIndepNodeForAsyncPref instance, suggesting that the object is being loaded asynchronously at that time.
In that case let FindOrLoadResidentObject() proceed as though no entry was found in the ROT for that OID - so it synchronously loads the object from the LSS and updates the std::map while holding the CSpace lock.
We need to check that the LSS is ok with two threads that concurrently load the same object. That kicks the can down the road - i.e it pushes the same kind of problem into the LSS, but maybe a similar solution works there. In any case we want the LSS to be robust to concurrent load requests for the same object.
Later when the async task completes it will find that the ROT has already been updated (it doesn't find itself in the m_self member of the ptr). In that case it deletes the redundant IPersistable object it downloaded and carries on as though it finished instead with the IPersistable object already in the ROT. So basically, it will notify dependents of an update to the ROT.
This may seem inefficient, but it's a very unusual corner case, that probably almost never happens. It's very unusual for an application to both synchronously and asynchronously load the same object.
ROT
ROT.h
// ROT.h
//
// Author David Barrett-Lennard
// (C)opyright Cedanet Pty Ltd 2011
@import "IPersistable.h"
@import "IPersistStore.h"
#include "Ceda/cxThread/AutoResetEvent.h"
#include "Ceda/cxThread/ThreadBlockerWhileUsed.h"
#include <map>
#include <mutex>
namespace ceda
{
$adt PSpace;
class ROT
{
cxNotCloneable(ROT)
public:
////////////// Public methods called without a lock on the PSpace
ROT(PSpace& pspace);
~ROT();
/*
Called when the PSpace is being closed.
Blocks until all DGIndepNodeForAsyncPref which were created by the ROT have deleted themselves,
implying all asynchronous tasks have completed.
*/
void Close();
/*
todo: this currently does a cxAssert(0). Should it be removed?
MakeMemoryResident() is normally called without a lock on the PSpace. The idea is to allow
a worker thread to be tied up loading an object from disk without tieing up more important
threads (such as the main GUI thread). The general rule is: don't do I/O whilst holding a
lock on the PSpace!
This function consults the ROT and does nothing if the object is already resident. Otherwise
it either blocks on an event in the CacheMap or else takes on responsibility for loading the
object into memory. In the latter case it will have created one or more objects in a
temporary CSpace. After loading from disk it declares a shared read lock on the PSpace and
transfers the created objects from the temporary CSpace to the PSpace. It returns after
having made sure the object was memory resident. However assuming this has been called
without a lock on the PSpace, clients cannot assume that the object is still memory resident
at the time the function returns because the GC may have evicted it!
Currently called:
- from PSpace::MakeMemoryResident(OID oid)
*/
void MakeMemoryResident(OID oid);
////////////// Public methods called with a lock on the PSpace
@if (CEDA_VALIDATE_MARK_AS_DIRTY)
{
void ValidateCRCs();
}
/*
Find the object with given OID, or returns null if not currently resident in memory
This function never blocks on I/O (either directly or indirectly).
Must only be called by a thread that has a locked the PSpace.
Currently called :
- in debug builds from PSpace::MakeObjectPersist() with a CSpace lock in order to
ensure that a freshly allocated oid doesn't already exist in the ROT.
- from PSpace::BindObjectIfMemoryResident(OID oid) which must be called with a
CSpace lock. prefbase::GetInMemory() calls BindObjectIfMemoryResident().
*/
ptr<IPersistable> FindResidentObject(OID oid) const;
/*
Bind to the object with given oid - either to an object that is already resident in memory,
or else load the object from disk. Blocks on I/O as required. Returns null if no object
with the given oid was found in the LSS. May throw exceptions.
Must only be called by a thread that has locked the PSpace.
Currently called :
- From PSpace::BindObject(OID oid) which asserts there is a CSpace lock
- From PSpace::TryBindObject(OID oid) which asserts there is a CSpace lock
*/
ptr<IPersistable> FindOrLoadResidentObject(OID oid);
/*
Called by PSpace::MakeObjectPersistWithGivenOID() to directly add new entries in the ROT as
objects becomes persistent reachable for the first time.
Must only be called by a thread that has a locked the PSpace.
This function can be called when an async load for the given oid is in progress. In that case
the async load is aborted.
If there is already an object with the given OID referenced by the ROT, then no change to the
ROT is made and this function returns false.
*/
bool AddResidentObject(OID oid, ptr<IPersistable> po);
/*
Called by the GC when it has an exclusive lock on the PSpace and has decided to
evict the object with the given OID.
It is not permissible to call this function for an object that is not already resident
in memory.
*/
void RemoveResidentObject(OID oid);
// Transfer all entries from src into this ROT
void TransferResidentObjects(ROT& src);
MAsyncBind AsyncBind2(OID oid);
ptr<IPersistable> AsyncBind(OID oid)
{
return AsyncBind2(oid).po;
}
private:
#ifdef CEDA_CHECK_ASSERTIONS
bool HasLock() const;
#endif
ptr<IPersistable> SyncLoadObject(OID oid);
// Note that it is ok to lock the CSpace THEN lock the oid mutex, but not vice versa.
std::mutex& GetMutexForOid(OID oid) const
{
return mutex_[oid.low_ & 0x0F];
}
private:
friend class DGIndepNodeForAsyncPref;
PSpace& pspace_;
/*
Incremented/decremented in the constructor/destructor of DGIndepNodeForAsyncPref. This allows
ROT::Close() to block until all DGIndepNodeForAsyncPref instances have deleted themselves.
*/
ThreadBlockerWhileUsed blockUntilNoMoreAsyncNodes_;
// A thread blocks on this event during a call to FindOrLoadResidentObject() if it needs
// to synchronously load an object for which an async I/O has already begun.
AutoResetEvent blockSyncThreadWaitingOnAsyncLoad_;
/*
Unfortunately a CSpace lock isn't sufficient for controlling access. The problem is a
dead-lock scenario that occurs between two threads:
Client thread : has a CSpace lock, is calling FindOrLoadResidentObject() and is waiting
for the async load to complete - i.e. for the
blockSyncThreadWaitingOnAsyncLoad_ event to be signaled.
Worker thread : has finished loading and deserialisating a persistent object and needs a
CSpace lock in order to be able to update the ROT, invalidate DGS nodes,
and signal the blockSyncThreadWaitingOnAsyncLoad_ event.
To avoid this issue, we instead introduce another mutex
to serialise access to the following member of DGIndepNodeForAsyncPref
ptr<IPersistable> asyncobj_;
Different OIDs can optionally share the mutex. At one extreme
DGIndepNodeForAsyncPref has a mutex member, so each OID has its own mutex.
At the other extreme the ROT declares a mutex to be shared for all OIDs.
We choose a solution between these two extremes by using the lowest 4 bits of the OID
to index into an array of 16 mutexes. This gives us some measure of
concurrency.
*/
mutable std::mutex mutex_[16];
/////////////////////// State protected by CSpace lock //////////////////////////
/*
The ROT is essentially a map from OID to ptr<IPersistable>, used to index all the
persistent objects in the associcated PSpace that are currently resident in memory.
The ROT map is also used to index DGIndepNodeForAsyncPref nodes that are in the process
of performing asynchronous I/O.
A ptr<IPersistable> contains a pair of pointers.
$struct AnyInterface
{
void* m_self;
const void* m_table;
};
When m_table == nullptr, we assume m_self is a pointer to a DGIndepNodeForAsyncPref!
The DGIndepNodeForAsyncPref in turn records a member asyncobj_ of type ptr<IPersistable>
to hold the result of an asynchronous I/O.
This map is only accessed by a thread which has exclusive locked the CSpace.
*/
typedef std::map<OID,ptr<IPersistable> > MAP;
MAP map_;
};
} // namespace cedaROT.cpp
// ROT.cpp
//
// Author David Barrett-Lennard
// (C)opyright Cedanet Pty Ltd 2011
@import "ROT.h"
@import "PSpaceDos.h"
@import "pref.h"
@import "PSpace.h"
@import "PersistStore.h"
@import "Ceda/cxObject/IObjectVisitor.h"
#include "Ceda/cxLss/LssExceptions.h"
#include "Ceda/cxUtils/Tracer.h"
#include "Ceda/cxUtils/Crc32.h"
#include "Ceda/cxUtils/Hex.h"
#include "Ceda/cxUtils/FileException.h"
namespace ceda
{
@def bool trace_ROT = false
@if (CEDA_VALIDATE_MARK_AS_DIRTY)
{
@println(*** Warning: This build is validating calls to MarkAsDirty())
}
ROT::ROT(PSpace& pspace) :
pspace_(pspace),
blockUntilNoMoreAsyncNodes_(0)
{
@if (trace_ROT)
{
Tracer() << "ROT()\n";
}
}
ROT::~ROT()
{
@if (trace_ROT)
{
Tracer() << "~ROT()\n";
}
}
void ROT::Close()
{
@if (trace_ROT)
{
Tracer() << "ROT::Close()\n";
TraceIndenter indent;
}
blockUntilNoMoreAsyncNodes_.Wait();
@if (trace_ROT)
{
Tracer() << "Waiting on blockUntilNoMoreAsyncNodes_ ended\n";
}
}
#ifdef CEDA_CHECK_ASSERTIONS
bool ROT::HasLock() const
{
return pspace_.HaveWriteAccess();
}
#endif
@if (CEDA_VALIDATE_MARK_AS_DIRTY)
{
const bool trace_ShowRedundantDirtyObjects = true;
void ROT::ValidateCRCs()
{
cxAssert(HasLock());
for (MAP::iterator i = map_.begin() ; i != map_.end() ; ++i)
{
ptr<IPersistable> po = i->second;
cxAssert(po.m_self);
if (po.m_table) // process ROT entries not associated with async I/O
{
// If DBP_PO_MARKED_DIRTY_CRC is set then DBP_PO_DIRTY should be set as well
cxAssert(implies(GetIObjectFlag(po,DBP_PO_MARKED_DIRTY_CRC), GetIObjectFlag(po,DBP_PO_DIRTY)));
bool isDirty = GetIObjectFlag(po,DBP_PO_MARKED_DIRTY_CRC);
uint32 newCrc = CalculateCrc(po);
uint32 prevCrc = GetCrc32(po);
bool valid = (isDirty || newCrc == prevCrc);
if (!valid ||
trace_CRCValidationChecks)
{
Tracer() << "po= " << po << " isDirty= " << isDirty << " newCrc = " << AsHex(newCrc) << " prevCrc = " << AsHex(prevCrc) << '\n';
//cxAssert(0);
}
// This assertion indicates there was a missing call to MarkAsDirty() on po.
cxAlwaysAssert(valid);
if (trace_ShowRedundantDirtyObjects && isDirty && newCrc == prevCrc)
{
Tracer() << "Redundant MarkAsDirty() on object " << po << '\n';
}
// Update the CRC on the object and mark the object as no longer having a dirty CRC
SetCrc32(po,newCrc);
SetIObjectFlag(po, DBP_PO_MARKED_DIRTY_CRC, false);
}
}
}
}
// Note that this may throw various exceptions, but not a DerefDanglingOidException!
ptr<IPersistable> ROT::SyncLoadObject(OID oid)
{
@if (trace_ROT)
{
Tracer() << "ROT::SyncLoadObject() oid= " << oid << '\n';
TraceIndenter indent;
}
ptr<IPersistable> po = pspace_.TryLoadPOGivenOid(oid);
@if (CEDA_VALIDATE_MARK_AS_DIRTY)
{
if (po)
{
SetCrc32(po, CalculateCrc(po));
@if (trace_CRCValidationChecks)
{
Tracer() << "After load object CRC = " << GetCrc32(po) << '\n';
}
}
}
return po;
}
// Called by application thread with a CSpace lock in order to synchronously bind oid to
// IPersistable in memory. Blocks on I/O as required.
ptr<IPersistable> ROT::FindOrLoadResidentObject(OID oid)
{
@if (trace_ROT)
{
Tracer() << "ROT::FindOrLoadResidentObject() oid= " << oid << '\n';
TraceIndenter indent;
}
cxAssert(HasLock());
MAP::iterator i = map_.find(oid);
if (i == map_.end())
{
// Note that we don't need to worry about other threads wanting to load the same
// object either synchronously or asynchonously - it is not possible, because only
// one thread can lock the CSpace at a time.
// Note: if SyncLoadObject() throws an exception, we avoid updating the map, as required
ptr<IPersistable> p = SyncLoadObject(oid);
if (p)
{
cxAssert(p.m_self);
cxAssert(p.m_table);
map_[oid] = p;
}
return p;
}
else
{
cxAssert(i->second.m_self);
if (i->second.m_table)
{
@if (trace_ROT)
{
Tracer() << "Found po in map " << i->second << '\n';
}
return i->second;
}
else
{
@if (trace_ROT)
{
Tracer() << "Found DGIndepNodeForAsyncPref in map\n";
TraceIndenter indent;
}
DGIndepNodeForAsyncPref* node = reinterpret_cast<DGIndepNodeForAsyncPref*>(i->second.m_self);
cxAssert(node);
bool readyToUpdateRot;
{
// Lock mutex for given oid
std::lock_guard<std::mutex> lock(GetMutexForOid(oid));
readyToUpdateRot = node->readyToUpdateRot_;
if (!readyToUpdateRot)
{
cxAssert(!node->waiting_);
/*
Reset the event we are going to wait on. Note that there is only one event
object in the ROT because only this thread has a CSpace lock, so no other
thread has a need to wait on an async load.
*/
blockSyncThreadWaitingOnAsyncLoad_.Reset();
/*
It is necessary to flag which node this thread with the CSpace lock is waiting
on. It is very important that a worker thread calling AsyncDeserialise() only
signals blockSyncThreadWaitingOnAsyncLoad_ for this node and not some other
node.
*/
node->waiting_ = true;
}
}
if (readyToUpdateRot)
{
@if (trace_ROT)
{
Tracer() << "No need to wait on blockSyncThreadWaitingOnAsyncLoad_\n";
}
}
else
{
@if (trace_ROT)
{
Tracer() << "Waiting on blockSyncThreadWaitingOnAsyncLoad_\n";
}
// async load in progress. Wait until it has completed
// todo : If worker thread has exception, then we need corresponding exception
// to be thrown by this thread.
blockSyncThreadWaitingOnAsyncLoad_.Wait();
}
/*
Regardless of whether this thread needed to wait for the async load to complete, this
thread may take on responsibility to perform all the "update ROT" processing that's
required by a thread that has locked the CSpace.
The AsyncUpdateRotThread will later get a CSpace lock and try to perform this processing
as well, but will find that updatedRot_ has already been set to UR_OBJECT_FOUND or
UR_OBJECT_DOES_NOT_EXIST.
*/
ptr<IPersistable> p = node->asyncobj_;
node->UpdateRot();
return p;
}
}
}
MAsyncBind ROT::AsyncBind2(OID oid)
{
@if (trace_ROT)
{
Tracer() << "ROT::AsyncBind2() oid= " << oid << '\n';
TraceIndenter indent;
}
cxAssert(oid);
cxAssert(HasLock());
MAP::iterator i = map_.find(oid);
if (i == map_.end())
{
// No entry for the given oid currently exists in the map.
// Queue async load of the object
DGIndepNodeForAsyncPref* node = new DGIndepNodeForAsyncPref(*this,oid);
// Insert node into the ROT
ptr<IPersistable> p;
p.m_self = node;
cxAssert(p.m_table == nullptr);
std::pair<MAP::iterator,bool> r = map_.insert(MAP::value_type(oid,p));
cxAssert(r.second);
node->rotIterator_ = r.first;
// Form the edge to the node
node->ReadBarrier();
// The PersistStore has a thread represented by asyncLoadThread_ which calls node->AsyncLoad()
// for each node that is pushed onto its job queue.
pspace_.m_ps.asyncLoadThread_.Push(node);
return MAsyncBind(EAsyncBindResult::Pending,null);
}
else
{
ptr<IPersistable> p = i->second;
cxAssert(p.m_self);
if (p.m_table)
{
@if (trace_ROT)
{
Tracer() << "Found po in map " << p << '\n';
}
if (GetIObjectFlag(p,DBP_PO_SYNC_DELETED))
{
// Object has been synchronously deleted
return MAsyncBind(EAsyncBindResult::DoesNotExist,null);
}
else
{
return MAsyncBind(EAsyncBindResult::Bound,p);
}
}
else
{
@if (trace_ROT)
{
Tracer() << "Found DGIndepNodeForAsyncPref in map\n";
TraceIndenter indent;
}
DGIndepNodeForAsyncPref* node = reinterpret_cast<DGIndepNodeForAsyncPref*>(p.m_self);
cxAssert(node);
node->magic_.Check();
/*
We must call the read barrier in order to form the edge even though the async task
to load the object has already been started.
This ensures the dependent will be invalidated if and when the async bind completes.
*/
node->ReadBarrier();
/*
The abort_ flag is only assigned true by DGIndepNodeForAsyncPref::OnEvict(), by
a thread with a CSpace lock. If this has already been done then the ROT must have been
updated - either by updating the mapped value to the asynchronously loaded IPersistable
object, or else by removing the entry from the ROT. Either way, it would not be possible
to have bound to a DGIndepNodeForAsyncPref using the ROT.
Note that we don't need special code to deal with restarting aborted tasks. This is
because restarting a task always involves allocating a new DGIndepNodeForAsyncPref and
queuing it as a new async I/O task.
*/
cxAssert(!node->abort_);
// If the object had been found and applied to the ROT then we shouldn't have been able
// to bind to a node.
cxAssert(node->updatedRot_ != DGIndepNodeForAsyncPref::UR_OBJECT_FOUND);
// OnEvict() should not have been called on the node yet
cxAssert(node->updatedRot_ != DGIndepNodeForAsyncPref::UR_NODE_EVICTED);
cxAssert(node->updatedRot_ == DGIndepNodeForAsyncPref::UR_PENDING ||
node->updatedRot_ == DGIndepNodeForAsyncPref::UR_OBJECT_DOES_NOT_EXIST);
// todo fails. Does that make sense?
//cxAssert(node->readyToUpdateRot_);
// todo fails. Does that make sense?
//cxAssert(!node->asyncobj_);
if (node->updatedRot_ == DGIndepNodeForAsyncPref::UR_PENDING)
{
return MAsyncBind(EAsyncBindResult::Pending,null);
}
else
{
cxAssert(node->updatedRot_ == DGIndepNodeForAsyncPref::UR_OBJECT_DOES_NOT_EXIST);
return MAsyncBind(EAsyncBindResult::DoesNotExist,null);
}
}
}
}
void ROT::TransferResidentObjects(ROT& src)
{
@if (trace_ROT)
{
Tracer() << "ROT::TransferResidentObjects()\n";
TraceIndenter indent;
}
//todo: this implementation needs to deal properly with async downloads.
//cxAssert(0);
// Must have locked both CSpaces
cxAssert(HasLock());
cxAssert(src.HasLock());
for (MAP::iterator i = src.map_.begin() ; i != src.map_.end() ; ++i)
{
cxAssert(map_.find(i->first) == map_.end());
map_[i->first] = i->second;
}
src.map_.clear();
}
void ROT::RemoveResidentObject(OID oid)
{
@if (trace_ROT)
{
Tracer() << "ROT::RemoveResidentObject() oid = " << oid << '\n';
TraceIndenter indent;
}
/*
Called from PSpace::OnGarbageCollectPersistable(ptr<const IPersistable> po)
i.e. by the GC when it has an exclusive lock on the PSpace and has decided to evict the
object with the given OID.
Since it is only called for an object resident in memory, there shouldn't be any chance
that the m_table entry being erased is not nullptr
*/
// Must have locked the CSpace
cxAssert(HasLock());
cxAssert(oid);
#ifdef CEDA_CHECK_ASSERTIONS
MAP::iterator i = map_.find(oid);
cxAssert(i != map_.end());
cxAssert(i->second.m_table != nullptr);
#endif
[[maybe_unused]] ssize_t numRemoved = map_.erase(oid);
cxAssert(numRemoved == 1);
}
ptr<IPersistable> ROT::FindResidentObject(OID oid) const
{
@if (trace_ROT)
{
Tracer() << "ROT::FindResidentObject() oid = " << oid << '\n';
TraceIndenter indent;
}
cxAssert(HasLock());
/*
Since DGIndepNodeForAsyncPref are recorded in the ROT map, we have to also check m_table
to ensure we actually have an object resident in memory.
*/
MAP::const_iterator i = map_.find(oid);
if (i != map_.end() && i->second.m_table)
{
return i->second;
}
else
{
return null;
}
}
void ROT::MakeMemoryResident(OID oid)
{
@if (trace_ROT)
{
Tracer() << "ROT::MakeMemoryResident() oid = " << oid << '\n';
TraceIndenter indent;
}
// This function is currently called from the MultiResImage, but can no longer work this way
// because FindOrLoadResidentObject() asserts there is a CSpace lock.
cxAssert(0);
/*
Typically called by a thread without a lock on the PSpace
We want to call FindOrLoadResidentObject(oid). However the problem is that if the object needs to be
loaded, dynamic creation will occur and that will fail because this thread isn't associated
with a PSpace.
Our solution is to declare a local CSpace and then transfer all objects into the PSpace
*/
CSpaceCreator src(CEDA_GC_THREAD_DONT_START);
CSpaceLock lock(src);
// Calling FindOrLoadResidentObject() could appear rather dangerous since we don't have any lock on
// the CSpace, so therefore for example we could bind to an object in the ROT on which
// the GC thread is calling Destroy() and is about to call ROT::Remove(). I.e. we
// return an object that has been deleted.
// However we completely ignore the return value, so there is no problem!
// The purpose of this function is to take action because there is no entry in the ROT
// and we want this thread to take on the responsibility to load the object from disk.
FindOrLoadResidentObject(oid);
// Transfer objects from the cspace (if any)
if (GetNumObjects(src) > 0)
{
CSpace* dst = pspace_.GetCSpace();
// todo: It would be desirable to only get a shared read lock on the PSpace
CSpaceLock lock(dst);
/*
// Test - make sure that object in the ROT has the expected OID.
if (ptr<IPersistable> p = TryFind(oid))
{
cxAssert(GetOid(p) == oid);
}
*/
TransferCSpace(src,dst); // Must only be called with exclusive write lock on both CSpaces
}
}
bool ROT::AddResidentObject(OID oid, ptr<IPersistable> po)
{
@if (trace_ROT)
{
Tracer() << "ROT::AddResidentObject() oid = " << oid << " po = " << po << '\n';
TraceIndenter indent;
}
cxAssert(oid);
cxAssert(po.m_self);
cxAssert(po.m_table);
cxAssert(HasLock());
ptr<IPersistable>& v = map_[oid];
if (v.m_self)
{
if (v.m_table)
{
@if (trace_ROT)
{
Tracer() << "Trying to replace existing object in ROT : oid=" << oid << " po= " << po << '\n';
}
return false;
}
else
{
/*
There is an indep DGS node for this oid. We can't just delete it because it may have
posted a task to perform an async load from the LSS, or an async task to update the ROT.
Our approach is to call SetObject() which notifies the dependents that the binding has
changed, and makes appropriate changes to the DGIndepNodeForAsyncPref so it will be as
inocuous as possible (e.g. abort tasks and avoid changing the ROT).
*/
DGIndepNodeForAsyncPref* node = reinterpret_cast<DGIndepNodeForAsyncPref*>(v.m_self);
cxAssert(node);
node->SetObject(po);
}
}
v = po;
return true;
}
} // namespace cedaRemoved functions
A few years ago there was a function named MakeMemoryResident() on the ROT but that has been removed.