I spent this morning exploring available tech to address this goal:

Add a bigdata database to my application, to archive older data out of the realtime local model

This is for my stock app, which deals with realtime in-memory data during market hours, with a delayed-write to local storage.  At the end of the day, it can then archive most of the data collected during market hours.

Because I have not achieved “success” in life yet, at least enough to allow me to pursue my larger goals uninhibited, I have to be very careful about how I apply my limited resources.  To be more precise:

  • Follow patterns that are as simple as possible (but no less), and sustainable into the next decade.
  • For my projects, limit languages, libraries and tools to those that are
    • well maintained
    • solve difficult problems more elegantly than I could solve with a medium level of effort

The result of today’s philosophically-informed research:

  • The primary languages of my software projects should be Javascript and C++
  • All data should be defined by JSON schema that is used to generate code, via quicktype
  • Long-term libraries and tools include boost, jquery, bootstrap, accounting.js, moment.js, nlohmann::json, sqlite, postgres

Note that using quicktype with nlohmann::json is an elegant way to effectively get C++ reflection.  Once you serialize an object to JSON you can walk all its fields.  Then you can do things like automatically build SQL queries for your classes based on the JSON schema.  Beautiful.

PS. I avoided spotify-json, StaticJson/autojsoncxx, Google Prototype Buffers, Code Synthesis’s ODB, the sqlite JSON1 Extension, C++ reflection libraries like RTTR, lots of code from Stiffstream and Chilkat, etc. because while they are all brilliant and compelling, they bring extra weight.  The world keeps churning though, so keep searching.  Also, there are cases where my choices do not fit, most obviously being cross-platform mobile apps, which will have to be saved for another post… 🙂

After seeing this little gem on the xunitpatterns site:

(We are running all the tests before every check-in, aren’t we?)

I was glad that I baked my tests into the startup of A better Trader. As I build and run while coding, I’m constantly running my regression tests. Steady on.

Simple-Web-Server has turned out to be a perfect fit for A better Trader… more to come…

Update: add these awesome ones!

https://pocoproject.org/
https://macchina.io/docs/
http://vinniefalco.github.io/beast/beast/intro/example.html

Goal: find the fastest possible C++ https library that can score an A on the SSLLabs test. Ties broken by ease of use, then by support for websockets.

Initial results (see wiki for latest…):

LIBRARY ab

MB/sec

ab

pages/sec

SSL LABS
SCORE
EASE WS? COMMENTS
apache A 3 N need to write a module, apache remains in charge of message loop (unacceptable)
libwebsockets 7 Y
proxygen 7 only easy to build on ubuntu so hasn’t been stood up on my gentoo server yet; huge kitchen sink of helpers
Simple-Web-Server  B 10 Y websockets in a seperate compatible project; may be able to leverage asio to improve score (capped by RC4); no forward secrecy
websocketpp  uses asio, specifies a “modern” mode that only allows TSL1.2
mongoose 3 messy ton of hand-crafted portability C code scared me off
onion looks like a strong C lib
libmicrohttpd GNU c lib, not sure it supports modern algos, check score…

I use my rad-scripts node module to quickly set up management and automation of my active projects.  I now have a couple private node modules:

  • mah-haus is my node module that mass-manages all my repos – it’s great; with one command I can completely sync my entire dev station when I start/finish a session
  • atci is for my AbetterTrader project and gives me all the dev tools I need to build, publish, and run continuous integration

Thanks node.js, I love my new automated life.  Should I publish these modules even though they have no real public use?  Nah, why pollute the npm-isphere?  But I will put this post here so I can use it for the module home pages.  😉  Carry on!