A better Trader
Use Case
ALL WE ASK OF USERS IS TO WATCH THE NEWS AND...
-----------------------------------------------------
BE INTUITIVE ABOUT A STOCK'S NEXT DIRECTION
-----------------------------------------------------
IF A PICKED IS LOOKING GOOD: MOVE TO TOP! or even buy. or reset the bracket...
IF A PICKED IS LOOKING BAD: Deactivate! there are more fish in the sea, get OUT
IF AN OWNED IS LOOKING GOOD: MOVE TO TOP - or even HOLD
IF AN OWNED IS LOOKING BAD: SELL! or raise sell trigger
THAT'S THE DAILY USAGE PATTERN
we must do everything else!
all that boring analysis should be done for them, unless they really want to obsess
Patterns
Basics
- Load from SQL/nosql tables into Major Objects
- Use a tight single-threaded message loop with async timers that set booleans when it is time to perform tasks
- Offload heavy automated analysis to after-hours external analyzers, with the goal of applying yesterday's best fit to today
- Server should provide minimal concise data to client, and client Javascript should do all UI rendering work.
- Traditionally, we would inject const variables into html
- Best practice but more of a refactor: make three separate calls to server to get html, javascript, and data. The html and javascript become cacheable static files.
Object model
JSON schema is used to generate base data classes and database read/write code, to provide the most agile schema refactoring. Follow these patterns to keep it consistent:
- use a constructor with defaults for all parameters:
// This constructor serves several purposes:
// 1) standard full-param constructor, efficient for both deserializing and initializing
// 2) no-param constructor for reflection via quicktype
// 3) id constructor for loading via id + quicktype fields
// 4) id constructor for use as key for unsorted_set::find()
BrokerAccount(
int64_t db_id = PersistentIDObject::DBID_DO_NOT_SAVE,
int64_t aar_add_arca_enabled = -1,
...
:
// Call base class
inherited(ba_max_db_id_,db_id),
// internal members
...
{
// persistent members
...
}
- There are three use cases for new objects:
- objects about to be loaded - use constructor params, and load in a value for db_id_
- new objects that should be made persistent - track a max_db_id in the parent to provide the "next" db_id constructor parameter
- temporary objects - use the default constructor
- use addXXXToMemory() to init parent-child relationships
StockRun& BrokerAccount::addStockRunToMemory(StockRun* psr, bool bInsertNewRank)
psr->setParent(*this);
runsByRank_.push_unsorted(psr);
runs_.insert(psr);
BrokerAccount &AppUser::addBrokerAccountToMemory(BrokerAccount *pba)
// We need a valid db id.
assert(pba->db_id_ != PersistentIDObject::DBID_UNSAVED);
// The caller is responsible for ensuring the account doesn't exist.
assert(findBrokerAccount(pba->db_id_) == accounts_.end());
pba->setParent(*this);
accounts_.insert(pba);
accountsByStringId_.insert(pba);
return *pba;
- use pointers for parent objects
set always-and-only once, on load
use functions that return a reference, to use those objects
example:
// EXTERNAL REFERENCES
// NOTE we are not responsible for these allocations.
// Access pointers to parents as references.
// Accces nullable pointers directly.
void setParent(AppUser& au) { pau_ = &au; }
AppUser& au() { assert(pau_ != 0); return *pau_; }
AppUser& au() const { assert(pau_ != 0); return *pau_; }
- squash 1:1 contained members into parent
StockRun (Cycle):
flatten these:
Stock stock_;
StockPick sp_;
AutotradedStock as_;
- store contained containers/vectors in separate tables, and fill in secondary pass
StockRun (Cycle):
BracketEvents sbe_;
Web UI
- bootstrap header and footer
- possible subsection navbar (eg Accounts, Admin) that sticks to top with bootstrap header
- forms: see at.js
- tables: we define tables entirely in JSON and pop them up with bootstrap-table, see AnalysisData getJSON, getJSONColumnNames
- ajax: see at.js
- patch can provide partial JSON to do partial updates (don't touch fields that are not provided)
- dates: moment.js - need to convert d3 date functions to moment
- money: accounting.js
Documentation
Performance Tracking
We measure three types of performance:
- SR: As a stock cycles, it tracks the %gain on each buy-sell; Each cycle may have a diff # of stocks but aggregating %change-per-cycle should have value
- BA: This is where we can say "at time T1 we had value X; at time T2 we had value Y" and get precise gains.
- AD: This is used across many cycles and accounts and needs aggregation similar to SR.
- SR and AD performance should be avg-percentage based as there is no base "value" like with BA
- use an "average gain per buy-sell cycle" => d_avg_pct_gain_[GTT_COUNT] + sells_count_[GTT_COUNT]
- cycle stopsells may need closer inspection
- track stopsells; perhaps red-flag at 1 stopsell, then bail on 2?
- cycle stopsell should (eventually?) flag the aps as in critical need of an update via reanalysis of recent history; rerun analysis, then reset the stopsell to zero
- track performance of ad
- nothing to do with need to rerank or reanalyze
- but just so we learn over time what the best metaranges are
- we want to keep working in this area, expanding as it makes sense
- eg separate stocks' volatility by price, volume, market...etc.
Doxygen and SchemaSpy diagrams
SchemaSpy was used on Sqlite relationships to generate a nice foreign key map.
Doxygen shows class diagrams. Here are some central relationships:
- PersistentObject class hierarchy shows all persistent classes
- AtHttpServer has call graphs for functions like GetAccount
- BrokerAccount ProcessQuote call hierarchy
MODEL
AtController
ui_
memory_model_
timers
MemoryModel: delayed-write datastore manager; use dirty flags + timed, transactioned saveDirtyObjects() call
prefs
sq_
brokers
aps_
users_
TradingModel
AppUser
BrokerAccount:
runs_ (sorted by id)
runsByRank_
contains all cycles sorted by rank
ONLY MANUAL has valid rank values
but additional sort criteria is used to handle all cycles
see StockRunsByRank_lessthan()
broker (not worthy of its own layer)
StockRun: rank, active, owned
StockPick+AutotradedStock: quote processing
SPASBracketEvent: stores one bracket-change event; includes triggering quote, bActive, buy/sell {quantity, commission, value}
StockSnapshot: run, symbol, quote, quantity (for account snapshot history)
Analysis data
StockQuote
db only has time+symbol+price
memory also has p_aad
AutoAnalysisData
symbol
aps_id
stopsells
profitable_sells
StockRun
aps
AutotradeParameterSet
bracket params
many analysis vars - move these out to aad
BracketEvent
contains all details about any possible event
for report and analysis we have a tighter version:
typedev vector<RunQAB> RunHistory
RunQAB is just quote+time + optional Bracket ptr
Order lifespan
Sim and analysis buy: place order, wait for next stock quote, buyWasExecuted()
Live buy: place order, poll for execution, buyWasExecuted()
Stock model
Stock
StockQuote& quote_;
typedef std::pair<const string,StockQuoteDetails*> StockQuote;
StockQuoteDetails
double d_quote_;
time_t timestamp_;
(+spike logic)
int64_t n_quantity_;
StockOrder so_;
int64_t order_id_;
ORDER_STATUS status_;
int64_t quote_db_id_;
DEBUG INTO REST API HANDLERS
break on server_http.hpp ln~370: if(REGEX_NS::regex_match(request->path, sm_res, regex_method.first)) {
watch regex_method.first.str
watch request->path
CI
MASTER SCRIPT: atci
We will have a live site, a constantly running CI site, and multiple dev environments.
RUN LIVE at bitpost.com:
m@bitpost rs at
m@bitpost # if that doesn't work, start a session: screen -S at
m@bitpost cd ~/development/thedigitalage/AbetterTrader/server-prod
m@bitpost atlive
========================================================
*** LIVE MODE ***
========================================================
CTRL-A CTRL-D
RUN CI at bitpost.com:
# Keep this running to ensure that changes are dynamically built as they are committed # It should run at a predictable publicly available url that can be checked regularly # It runs in TEST but should run in a test mode that has an account assigned to it so it is very much like LIVE # It runs release build in test mode CTRL-A CTRL-D
RUN DEV anywhere but bitpost:
# Dev has complete control; most common tasks: # Code fast with a local CI loop - as soon as a file is changed, CI should restart server in test mode, displaying server log and refreshing server page # kill server, build, run, refresh browser # Turn off CI loop to debug via IDE # Stop prod, pull down production database, run LIVE mode in debugger to diagnose production problems
Thread locking model
OLD model was to do async operations, sending them through the APIRequestCache. The problem with that was that the website could not give immediate feedback. FUCKING WORTHLESS. New model uses the same exact locking, just does it as needed, where needed. We just need to chose that wisely.
- Lock at USER LEVEL, as low-level as possible, but as infrequently as possible - not necessarily easy
- Lock container reads with reads lock, which allows multiple reads but no writing
// Lock user for reading boost::shared_lock<boost::shared_mutex> lock(p_user->rw_mutex_);
- Lock container writes with exclusive write lock
// Lock user for writing boost::lock_guard<boost::shared_mutex> uniqueLock(p_user->rw_mutex_);
There is also a global mutex, for locking during AppUser container operations, etc.
- reading
boost::shared_lock<boost::shared_mutex> lock(g_p_local->rw_mutex_);
- writing
boost::lock_guard<boost::shared_mutex> uniqueLock(g_p_local->rw_mutex_);
DAILY MAINTENANCE
- Data is segmented into files, one per day
- to determine end-of-day, timestamps are checked as they come in with quotes (we had no better way to tell)
- At end of day, perform maintenance
- perform maintenance only once, checking for existing filename with date of "today"
- purge nearly all quotes and bracket events, while ensuring the new database remains self-contained
- preserve last-available quote for all stocks
- create a fresh new starting snapshot of all accounts using the preserved quotes
- postpone next quote retrieval until market is ready to open again
Pseudo:
EtradeInterface::processQuotes()
if (patc_->bTimestampIsOutsideMarketHours(pt))
patc_->checkCurrentTimeForOutsideMarketHours()
---
checkCurrentTimeForOutsideMarketHours
// Do not rely on quote timestamps.
ptime ptnow = second_clock::local_time();
if (bMarketOpenOrAboutTo(ptnow))
return false;
// If we are just now switching to outside-hours, immediately take action.
set_quotes_timer_to_next_market_open(); // This will cause the quotes timer to reset to a loooong pause.
if (bTimestampIsAfterHours(ptnow)) // must be AFTER not before
if (g_p_local->performAfterHoursMaintenance(ptnow))
// Always start each new day with a pre-market account snapshot,
pa->addSnapshotToMemory(snaptime);
runAnalysis();
PRODUCTION ASSETS
Due to potentially large sizes, I moved all bitpost production live assets to the software raid. Extra log backups are in logs folder. Extra db backups are in db_archive folder.
at_server_live.log -> /spiceflow/softraid/development/thedigitalage/AbetterTrader/ProductionAssets/logs/at_server_live.log db_analysis -> /spiceflow/softraid/development/thedigitalage/AbetterTrader/ProductionAssets/db_analysis db_archive -> /spiceflow/softraid/development/thedigitalage/AbetterTrader/ProductionAssets/db_archive
ANALYZE PSEUDO (REFACTOR FOUR) in process...
- we run autoanalysis for every known symbol, with minimal interaction from user - they just decide to use it or not
- we do a standard deviation on the data, and a monte-carlo-like loop through ranges APS values based on sd multipliers
- we are working toward a distributed microservice approach with many analysis engines across LAN
Function overview:
bool AtController::load_startup_data()
if (b_analyze_on_startup_)
getAnalyzerController().requeue_analyses();
bool AtController::checkCurrentTimeForOutsideMarketHours()
if (getModel().bMarketOpenOrAboutTo(ptnow))
return false;
if (!b_after_hours_)
b_after_hours_= true;
// Attempt maintenance and analysis, but only if we are AFTER hours (not before).
if (getModel().bTimestampIsAfterHours(ptnow))
getModel().performAfterHoursMaintenance(ptnow);
void AnalyzerController::requeue_analyses()
stop_and_clear_jobs();
fill_all_job_slots();
REPORTING JSON
PATTERN
-------
bool handler()
mm().readXxxJson(json) (done in derived model)
for r
buildAccountCyclesPerformanceJSONRow( (done in MM)
row["snapshot_action"].as<int64_t>()
)
archive().readXxxJson(json) (done in derived model)
for row
buildAccountCyclesPerformanceJSONRow( (done in MM)
query.getColumn(0).getInt64(),
FOLLOW OUR PATTERN WITH all handlers:
* PostAccountPerformanceCycles
AtHttpServer::PostAccountPerformanceCycles()
readAccountCyclesPerformanceJSON (derived models)
buildAccountCyclesPerformanceJSONRow (mm)
* GetAccountPerformance
* PostAccountPerformance
NOTE that these ones actually completely harvest the data first,
due to reporting requirements (JSON can't be built directly from db rows)
- PostAccountActivity
- PostAccountActivityTable
QAB charts
CHART TIMEFRAME DESIGN
use cases:
user wants to see performance across a variety of time frames <- PERFORMANCE PAGE only!
user wants to see historical brackets for older days <- PERFORMANCE PAGE only! lower priority!
user wants to perform immediate actions on realtime chart
user wants to do autoanalysis across a range and then manually tweak it
requirements
round 1: we can satisfy everything with TODAY ONLY (show today archive if after hours)
round 2: add a separate per-day performance chart
round 3: add a date picker to the chart to let the user select an older day to show
node reduction
data DISPLAY only needs to show action points and highs/lows
aggressively node-reduce to fit the requested screen size!
given: number of pixels of width
provide: all bracket quotes plus the lowest+highest quotes in each 2-pixel range (minimum - adjustable to more aggressive clipping if desired)
internal data ANALYSIS should use all points
CHART DATA RETRIEVAL
function addStock(cycle) {
restAndApply('GET','runs/'+cycle.run+'/live.json?pixel_limit='+$(window).width()*2...
---
void AtHttpsServer::GetRunLive(API_call& call)
g_p_local->readRunLive(p_user->db_id_, account_id, run_id, symbol, pixel_limit, atc_.bAfterHours(), rh);
atc_.thread_buildRunJSON(rh,*sr.paps_,run_id);
var analysisChange = function(event) {
$.getJSON('runs/'+run+'/analysis.json?pixel_limit='+$(window).width()*2+'&aggressiveness='+event.value, function( data ) {
---
AtHttpsServer::GetRunAnalysis(API_call& call)
atc_.thread_handleAnalysisRequest(
---
AtController::thread_handleAnalysisRequest(BrokerAccount& ba,int64_t run_id,bool b_autoanalyze,double d_aggressiveness,int32_t pixel_limit,string& http_reply)
g_p_local->readRunHistory()
thread_analyzeHistory()
thread_buildRunJSON(rh,apsA,apsA.run_id_);
-- NOT CURRENTLY CALLED --
function displayHistory(run)
$.getJSON('runs/'+run+'/history.json?pixel_limit='+$(window).width()*2+'&days=3', function( data ) {
---
AtHttpsServer::GetRunHistory(API_call& call)
g_p_local->readRunHistory(p_user->db_id_,account_id,run_id,symbol,days,sr.paps_->n_analysis_quotes_per_day_reqd_,pixel_limit,rh);
atc_.thread_buildRunJSON(rh,*sr.paps_,run_id);
DEBUG LIVE
NOTE that you WILL lose stock quote data during the debugging time, until we set up a second PROD environment.
- WRITE INITIAL DEBUG CODE in any DEV environment
- UPDATE DEBUGGER to run with [live] parameter instead of debug ones
- COPY DATABASE directly from PROD environment to DEV environment: atimport prod
- STOP PROD environment at_server
- DEBUG. quickly. ;-)
- PUSH any code fix (without debug code) back to PROD env
- RESTART PROD and see if the fix worked
- REVERT DEV environment: clean any debug code, redo an atimport and reset the debugger parameters
HTML SHARED HEADER/FOOTER
-------------------------------------------------------------------------------
THREE PARTS THAT MUST BE IN EVERY HTML FILE:
-------------------------------------------------------------------------------
1) all code above <container>, including these replaceables:
a) logout: <button type='button' id='logout' class='btn btn-margined btn-xs btn-primary pull-right'>Log out</button>
b) breadcrumbs: <!--bread--><li><a href="/1">1</a></li><li class="active">2</li><!--crumbs-->
2) logout button handler
$( document ).ready(function() {
3) footer and [Bootstrap core Javascript]
what a maintenance nightmare - but it seems best to do all 10-12 files manually
-------------------------------------------------------------------------------
HAPROXY and LOAD BALANCING
For the first 1000 paid users, we will NOT do load balancing.
- Use haproxy Layer 7 (http) load balancing to redirect abettertrader.com requests to a bitpost.com custom port.
For load balancing, there are two database design choices:
- Each server gets its own quotes and saves all its own data
- Need to read user id from each request and send each user to a predetermined server
- Need multiple Etrade accounts, one for each server, unless we get a deal with Etrade
- Switch to a distributed database with master-master replication
- A lot of work
- Might kill sub-second performance? Might not. We already have delayed-write.
TIMESTAMP STANDARDIZATION
Standardize internal times as int64_t milliseconds since 1970 in UTC. That's not ideal as it doesn't deal with leap seconds. But makes our time handling code much faster, so worth the tradeoff.
Display times in local time.
| Chart URL | The time should default to 9:30am EST which should display in URL as something like: 2019-09-13T13:30:00.000Z |
| Performance page | ? |
| Activity page | ? |
OLDER NOTES
WALKING DATABASE FILES
(we are moving to postgres for archiving!)
There are two types of historical requests:
- a specific date range, usually requested by user; use this:
getDatabaseNames(startdate,enddate)
- a specific number of days, usually requested by analysis; loop with this:
getPreviousDatabaseName(dt,db_name)
- there is also this, which skips non-market days, but we want them for now when walking db files:
get_previous_market_day(pt)
Move to mongo soon! :-)
API PSEUDO (no longer of much use)
APIGetRunLive::handle_call()
g_p_local->getRunLiveJSON()
readRunQAB(s_str_db_name...)
(SAME as readRunLive!!)
APIGetRunHistory::handle_call()
g_p_local->getRunHistoryJSON()
readRunHistory
readRunQAB
Qt Creator settings (moved on > CLion > vs code)
- Make sure you have already run [atbuild] and [atbuild debug].
- Open CMakeLists.txt as a Qt Creator project.
- It will force you to do CMake - pick cmake-release folder and let it go.
- Rename the build config to debug.
- Clone it to release and change folder to release.
- Delete make step and replace it with custom build:
./build.sh
(no args)
%{buildDir}
- Create run setups:
you have to use hardcoded path to BASE working dir (or leave it blank maybe?): /home/m/development/thedigitalage/AbetterTrader/server [x] run in terminal I recommend using TEST args for both debug and release: localhost 8000 test reanalyze (matches attest) LIVE args may occasionally be needed for use with [atimport prod]: localhost 8080 live (matches atlive)
MONTHLY MANUAL MAINTENANCE
(This is now available via the admin Summary page.)
Automate as much as possible, but this is not that bad and safer to do manually when we know the time is right: monthly db maintenance
just monthly:
update PrefStr set value = "JANUARY 2018 LEADERBOARD" where name="LeaderboardTitle";
update PrefStr set value = "Leader at 1/31 closing bell wins
December winner: cfjaques Go Cara!" where name="LeaderboardDescription";
update Accounts set leaderboard_initial_value = total_managed_value;
update AnalysisData set avg_pct_gain_mtd = 0, sells_count_mtd = 0;
update AnalysisData set avg_pct_gain_mtd = 0, sells_count_mtd = 0;
update StockRuns set avg_pct_gain_mtd = 0, sells_count_mtd = 0;
update StockRuns set avg_pct_gain_mtd = 0, sells_count_mtd = 0;
AND ANNUAL! HAPPY NEW YEAR 2018!
update PrefStr set value = "JANUARY 2018 LEADERBOARD" where name="LeaderboardTitle";
update PrefStr set value = "Leader at 1/31 closing bell wins
December winner: cfjaques Go Cara!" where name="LeaderboardDescription";
update Accounts set leaderboard_initial_value = total_managed_value, year_initial_value = total_managed_value;
update AnalysisData set avg_pct_gain_ytd = 0, sells_count_ytd = 0, avg_pct_gain_mtd = 0, sells_count_mtd = 0;
update AnalysisData set avg_pct_gain_ytd = 0, sells_count_ytd = 0, avg_pct_gain_mtd = 0, sells_count_mtd = 0;
update StockRuns set avg_pct_gain_ytd = 0, sells_count_ytd = 0, avg_pct_gain_mtd = 0, sells_count_mtd = 0;
update StockRuns set avg_pct_gain_ytd = 0, sells_count_ytd = 0, avg_pct_gain_mtd = 0, sells_count_mtd = 0;
if you need to hard-reset all the simulation accounts, do this and restart to fix all CASH:
update Accounts set initial_managed_value = 100000, total_managed_value = 100000, net_account_value = 100000 where broker_id=2;
Developer environment setup
The setup script is [as setup [nopostres]]. The installation script can be used for initial setup, and rerun to upgrade components like boost SWS SWSS and postgres.
First time install
- First, set up YET ANOTHER GODDAMN BOX:
setup_linux.sh [desktop | nodesktop]
- Clone the good stuff
mh c Get all the goodness
Choose local postgres server or remote
You can choose to install a full postgres server (usually desired on a laptop):
at setup
Or just install postgres client, and point your dev installation at another postgres server installation (typically use positronic if you're on the LAN):
at se nopostgres
To upgrade boost
- update the boost version in .bashrc
- [cdl] and remove existing boost install
- rerun [at se] or [at se nopostres] as appropriate