Major Objects: Difference between revisions

From Bitpost wiki
No edit summary
Line 20: Line 20:


=== Delayed delete pattern ===
=== Delayed delete pattern ===
             1) all deletion work should be done in one place:
             1) to dynamically delete an object:  
                bool SqliteLocalModel::deleteBrokerAccount(BrokerAccount& ba)
                  a) delete from db
                  b) call inherited::() and use container to delete from all container indices
                  c) delete memory allocation
            2) deletion check should happen in delayed write check:
                if (pau->bDirtyOrDeleted())
                    bNeeded = true;
                ----
                if (pa->bDeleted())
                    deleteBrokerAccount(*pa);
                else...
            3) to dynamically delete an object:  
                 ba.setDeleted();
                 ba.setDeleted();
             4) include deleted status in active check:
             2) include deleted status in active check:
                 bool bActive() const { return b_active_ && !bDeleted(); }
                 bool bActive() const { return b_active_ && !bDeleted(); }
            3) all deletion work is done in MemoryModel::saveDirtyObjectsAsNeeded(), see that code
                a) deletion check should happen in delayed write check:
                    if (pau->bDirtyOrDeleted())
                        bNeeded = true;
                b) if bNeeded, always do deletions first, starting with greatest grandparent container, to minimize work
                c) use the erase-remove pattern to remove all deleted items in one loop
                    eg: removeDeletedUsers();
                      a) iterate and remove item from all secondary indices
                      b) iterate primary index, and use the lambda of the erase-remove operation to delete memory allocation and remove db record

Revision as of 15:22, 26 March 2017

Overview

  • Major Objects
    • Use Major Objects for fast in-memory handling of large amount of data that is thread-safe but must be persisted
    • We must support complex objects with simple keys, crud, and fast lookup by multiple keys.
    • Our most useful containers are vector, set (key in object) and map (<key,value> pair). Set can give us almost every positive feature, when used to store the PersistentIDObject class.
    • Use an unordered_set of const pointers to objects derived from PersistentIDObject
    • The default container should index by db_id primary key
    • Always use the db_id for foreign keys
    • Other containers can be created with alternate keys using object members; just define new hash functions.
  • PersistentIDObject
    • Add a dirty flag to all objects, set to true on any change that must be persisted
    • Use an internal in-memory counter to generate the next db_id for a newly created object
    • This means that when creating new objects, there is NO NEED to access db, VERY IMPORTANT!
    • Use delayed-write tactics to write all dirty objects on idle time
  • Memory Model
    • Use a Datastore manager (aka "MemoryModel") to hold sets
    • It can look up objects by any key, and strip away const to return a mutable object. NOTE that the user must not damage the key values!
    • Derive a class from the memory model for persistence; it can use any persistence method (local, remote, sql, nosql, etc.).
    • Make sure that the base MemoryModel class is concrete not abstract, thread-safe and self-contained; this makes parallel calculations trivial, helps scalability, etc.

Delayed delete pattern

           1) to dynamically delete an object: 
               ba.setDeleted();
           2) include deleted status in active check:
               bool bActive() const { return b_active_ && !bDeleted(); }
           3) all deletion work is done in MemoryModel::saveDirtyObjectsAsNeeded(), see that code
               a) deletion check should happen in delayed write check:
                   if (pau->bDirtyOrDeleted())
                       bNeeded = true;
               b) if bNeeded, always do deletions first, starting with greatest grandparent container, to minimize work
               c) use the erase-remove pattern to remove all deleted items in one loop
                   eg: removeDeletedUsers();
                     a) iterate and remove item from all secondary indices
                     b) iterate primary index, and use the lambda of the erase-remove operation to delete memory allocation and remove db record