A better Trader

PATTERNS

• Load from SQL tables into Major Objects
• Use a tight single-threaded message loop with async timers that set booleans when it is time to perform tasks
• Offload heavy automated analysis to after-hours external analyzers, with the goal of applying yesterday's best fit to today

DOXYGEN

Doxygen charts and graphs are the best way to get the big picture. Here are some central ones:

• PersistentObject class hierarchy shows all persistent classes
• AtHttpServer has call graphs for functions like GetAccount
• BrokerAccount ProcessQuote call hierarchy

PRIORITIZED OPERATIONAL USE CASES

   ALL WE ASK OF USERS:
   
       -----------------------------------------------------
       BE  INTUITIVE   ABOUT   A   STOCK'S   NEXT  DIRECTION
       -----------------------------------------------------
   
   IF A PICKED IS LOOKING GOOD:    MOVE TO TOP!  or even buy
   IF A PICKED IS LOOKING BAD:     MOVE TO BOTTOM or even deactivate!
   IF AN OWNED IS LOOKING GOOD:    MOVE TO TOP - or even HOLD
   IF AN OWNED IS LOOKING BAD:     SELL!  or raise sell trigger
   
   THAT'S THE DAILY USAGE PATTERN
   we must do everything else!  
   all that boring analysis should be done for them, unless they really want to obsess

MODEL

AtController
  ui_
  memory_model_
  timers

MemoryModel: delayed-write datastore manager; use dirty flags + timed, transactioned saveDirtyObjects() call
  prefs
  sq_
  brokers
  aps_
  users_

TradingModel
  AppUser
    BrokerAccount: runs_ (sorted by id) runsByRank_ (sorted by rank)
      broker (not worthy of its own layer)
      StockRun: rank, active, owned
        StockPick+AutotradedStock: quote processing
        SPASBracketEvent: stores one bracket-change event; includes triggering quote, bActive, buy/sell {quantity, commission, value}
      StockSnapshot: run, symbol, quote, quantity (for account snapshot history)

Order lifespan
  Sim and analysis buy: place order, wait for next stock quote, buyWasExecuted()
  Live buy: place order, poll for execution, buyWasExecuted()

Stock model

       Stock
         StockQuote& quote_;
           typedef std::pair<const string,StockQuoteDetails*> StockQuote;
             StockQuoteDetails
               double d_quote_;
               time_t timestamp_;
               (+spike logic)
         int64_t n_quantity_;
         StockOrder so_;
           int64_t order_id_;
           ORDER_STATUS status_;
           int64_t quote_db_id_;

DEBUG INTO API HANDLERS

break on server_http.hpp ln~370: if(REGEX_NS::regex_match(request->path, sm_res, regex_method.first)) {
watch regex_method.first.str
watch request->path

CI

MASTER SCRIPT: atci

We will have a live site, a constantly running CI site, and multiple dev environments.

RUN LIVE at bitpost.com:

m@bitpost rs at
m@bitpost # if that doesn't work, start a session: screen -S at 
m@bitpost cd ~/development/thedigitalage/AbetterTrader/server-prod
m@bitpost atlive
 ========================================================
    *** LIVE MODE ***
 ========================================================
CTRL-A CTRL-D

RUN CI at bitpost.com:

# Keep this running to ensure that changes are dynamically built as they are committed
# It should run at a predictable publicly available url that can be checked regularly
# It runs in TEST but should run in a test mode that has an account assigned to it so it is very much like LIVE
# It runs release build in test mode
CTRL-A CTRL-D

RUN DEV anywhere but bitpost:

# Dev has complete control; most common tasks:
#   Code fast with a local CI loop - as soon as a file is changed, CI should restart server in test mode, displaying server log and refreshing server page
#       kill server, build, run, refresh browser
#   Turn off CI loop to debug via IDE
#   Stop prod, pull down production database, run LIVE mode in debugger to diagnose production problems

THREAD LOCKING MODEL

OLD model was to do async operations, sending them through the APIRequestCache. The problem with that was that the website could not give immediate feedback. FUCKING WORTHLESS. New model uses the same exact locking, just does it as needed, where needed. We just need to chose that wisely.

• Lock at USER LEVEL, as low-level as possible, but as infrequently as possible - not necessarily easy
• Lock container reads with reads lock, which allows multiple reads but no writing

 // Lock user for reading
 boost::shared_lock<boost::shared_mutex> lock(p_user->rw_mutex_);

• Lock container writes with exclusive write lock

 // Lock user for writing
 boost::lock_guard<boost::shared_mutex> uniqueLock(p_user->rw_mutex_);

There is also a global mutex, for locking during AppUser container operations, etc.

• reading

 boost::shared_lock<boost::shared_mutex> lock(g_p_local->rw_mutex_);          

• writing

 boost::lock_guard<boost::shared_mutex> uniqueLock(g_p_local->rw_mutex_);

DAILY MAINTENANCE

• Data is segmented into files, one per day
• to determine end-of-day, timestamps are checked as they come in with quotes (we had no better way to tell)
• At end of day, perform maintenance
  • perform maintenance only once, checking for existing filename with date of "today"
  • purge nearly all quotes and bracket events, while ensuring the new database remains self-contained
    • preserve last-available quote for all stocks
    • create a fresh new starting snapshot of all accounts using the preserved quotes
  • postpone next quote retrieval until market is ready to open again

Pseudo:

 EtradeInterface::processQuotes()
   if (patc_->bTimestampIsOutsideMarketHours(pt))
     patc_->checkCurrentTimeForOutsideMarketHours()
     ---
     checkCurrentTimeForOutsideMarketHours
       // Do not rely on quote timestamps.
       ptime ptnow = second_clock::local_time();
 
       if (bMarketOpenOrAboutTo(ptnow))
         return false;
 
       // If we are just now switching to outside-hours, immediately take action.
         set_quotes_timer_to_next_market_open();     // This will cause the quotes timer to reset to a loooong pause.
         if (bTimestampIsAfterHours(ptnow))          // must be AFTER not before
           if (g_p_local->performAfterHoursMaintenance(ptnow))
               // Always start each new day with a pre-market account snapshot, 
                           pa->addSnapshotToMemory(snaptime);
           runAnalysis();

QAB charts

CHART TIMEFRAME DESIGN

 use cases:
   user wants to see performance across a variety of time frames <- PERFORMANCE PAGE only!
   user wants to see historical brackets for older days <- PERFORMANCE PAGE only!  lower priority!
   user wants to perform immediate actions on realtime chart
   user wants to do autoanalysis across a range and then manually tweak it
 
 requirements
   round 1: we can satisfy everything with TODAY ONLY (show today archive if after hours)
   round 2: add a separate per-day performance chart
   round 3: add a date picker to the chart to let the user select an older day to show
 
 node reduction
   data DISPLAY only needs to show action points and highs/lows
       aggressively node-reduce to fit the requested screen size!
       given: number of pixels of width
       provide: all bracket quotes plus the lowest+highest quotes in each 2-pixel range (minimum - adjustable to more aggressive clipping if desired)
   internal data ANALYSIS should use all points

CHART DATA RETRIEVAL

  function addStock(cycle) {
    restAndApply('GET','runs/'+cycle.run+'/live.json?pixel_limit='+$(window).width()*2...
    ---
    void AtHttpsServer::GetRunLive(API_call& call)
      g_p_local->readRunLive(p_user->db_id_, account_id, run_id, symbol, pixel_limit, atc_.bAfterHours(), rh);
      atc_.thread_buildRunJSON(rh,*sr.paps_,run_id);
 
  var analysisChange = function(event) {
    $.getJSON('runs/'+run+'/analysis.json?pixel_limit='+$(window).width()*2+'&aggressiveness='+event.value, function( data ) {
    ---
    AtHttpsServer::GetRunAnalysis(API_call& call)
      atc_.thread_handleAnalysisRequest(
      ---
      AtController::thread_handleAnalysisRequest(BrokerAccount& ba,int64_t run_id,bool b_autoanalyze,double d_aggressiveness,int32_t pixel_limit,string& http_reply)
        g_p_local->readRunHistory()
        thread_analyzeHistory()
        thread_buildRunJSON(rh,apsA,apsA.run_id_);
 
  -- NOT CURRENTLY CALLED --
  function displayHistory(run)
    $.getJSON('runs/'+run+'/history.json?pixel_limit='+$(window).width()*2+'&days=3', function( data ) {
    ---
    AtHttpsServer::GetRunHistory(API_call& call)
      g_p_local->readRunHistory(p_user->db_id_,account_id,run_id,symbol,days,sr.paps_->n_analysis_quotes_per_day_reqd_,pixel_limit,rh);
      atc_.thread_buildRunJSON(rh,*sr.paps_,run_id);

ANALYZE PSEUDO (REFACTOR FOUR)

• we run autoanalysis for every known symbol, with minimal interaction from user - they just decide to use it or not
• we do a standard deviation on the data, and a monte-carlo-like loop through ranges APS values based on sd multipliers
• we are working toward a microservice approach where we can run many analysis engines distributed across LAN

Function hierarchy:

     void AtController::runAnalysis()      runs at end of day to analyze all targeted picks
                                           can also be kicked off in test via command line param
                                           current status: does nothing, needs to use refactor 3 functions
                                           
     thread_analyzeOneAPS()                not called anywhere!
     
     GetRunAutoAnalysis()                  auto = true, aggressiveness = -1.0
     GetRunAnalysis()                      auto = false, aggressiveness = a specific point (0-100)
       both call:
       thread_handleAnalysisRequest()      if -1.0
         thread_analyzeHistory()
           if (bAutoanalyze)
               analyzeAcrossFullAggressivenessRange(symbol,*pba,rh);
               analyzeAcrossMonteCarloRange(symbol,*pba,rh);
               (and run ba.analyze() a bunch)
           else
               generateAPSFromAggressiveness(apsAnalysis);
               pba->analyze(symbol,rh);
               (just run it once)

DEBUG LIVE

NOTE that you WILL lose stock quote data during the debugging time, until we set up a second PROD environment.

• WRITE INITIAL DEBUG CODE in any DEV environment
• UPDATE DEBUGGER to run with [live] parameter instead of debug ones
• COPY DATABASE directly from PROD environment to DEV environment: atimport prod
• STOP PROD environment at_server
• DEBUG. quickly.  ;-)
• PUSH any code fix (without debug code) back to PROD env
• RESTART PROD and see if the fix worked
• REVERT DEV environment: clean any debug code, redo an atimport and reset the debugger parameters

HTML SHARED HEADER/FOOTER

    -------------------------------------------------------------------------------
    THREE PARTS THAT MUST BE IN EVERY HTML FILE:
    -------------------------------------------------------------------------------
    
      1) all code above <container>, including these replaceables:
          a) logout:      <button type='button' id='logout' class='btn btn-margined btn-xs btn-primary pull-right'>Log out</button>
          b) breadcrumbs: <!--bread--><li><a href="/1">1</a></li><li class="active">2</li><!--crumbs-->
      2) logout button handler
        $( document ).ready(function() {
      3) footer and [Bootstrap core Javascript]
      
    what a maintenance nightmare - but it seems best to do all 10-12 files manually
    -------------------------------------------------------------------------------

HAPROXY and LOAD BALANCING

For the first 1000 paid users, we will NOT do load balancing.

• Use haproxy Layer 7 (http) load balancing to redirect abettertrader.com requests to a bitpost.com custom port.

For load balancing, there are two database design choices:

• Each server gets its own quotes and saves all its own data
  • Need to read user id from each request and send each user to a predetermined server
  • Need multiple Etrade accounts, one for each server, unless we get a deal with Etrade
• Switch to a distributed database with master-master replication
  • A lot of work
  • Might kill sub-second performance? Might not. We already have delayed-write.

API PSEUDO (no longer of much use)

   APIGetRunLive::handle_call()
       g_p_local->getRunLiveJSON()
           readRunQAB(s_str_db_name...)
           (SAME as readRunLive!!)

   APIGetRunHistory::handle_call()
       g_p_local->getRunHistoryJSON()
           readRunHistory
               readRunQAB

Qt Creator settings (we are now mostly using CLion)

• Make sure you have already run [atbuild] and [atbuild debug].
• Open CMakeLists.txt as a Qt Creator project.
• It will force you to do CMake - pick cmake-release folder and let it go.
• Rename the build config to debug.
• Clone it to release and change folder to release.
• Delete make step and replace it with custom build:

./build.sh
(no args)
%{buildDir}

• Create run setups:

you have to use hardcoded path to BASE working dir (or leave it blank maybe?): 

   /home/m/development/thedigitalage/AbetterTrader/server

[x] run in terminal
I recommend using TEST args for both debug and release: localhost 8000 test reanalyze (matches attest)
LIVE args may occasionally be needed for use with [atimport prod]: localhost 8080 live (matches atlive)