I spent this morning exploring available tech to address this goal:

Add a bigdata database to my application, to archive older data out of the realtime local model

This is for my stock app, which deals with realtime in-memory data during market hours, with a delayed-write to local storage.  At the end of the day, it can then archive most of the data collected during market hours.

Because I have not achieved “success” in life yet, at least enough to allow me to pursue my larger goals uninhibited, I have to be very careful about how I apply my limited resources.  To be more precise:

• Follow patterns that are as simple as possible (but no less), and sustainable into the next decade.
• For my projects, limit languages, libraries and tools to those that are
  • well maintained
  • solve difficult problems more elegantly than I could solve with a medium level of effort

The result of today’s philosophically-informed research:

• The primary languages of my software projects should be Javascript and C++
• All data should be defined by JSON schema that is used to generate code, via quicktype
• Long-term libraries and tools include boost, jquery, bootstrap, accounting.js, moment.js, nlohmann::json, sqlite, postgres

Note that using quicktype with nlohmann::json is an elegant way to effectively get C++ reflection.  Once you serialize an object to JSON you can walk all its fields.  Then you can do things like automatically build SQL queries for your classes based on the JSON schema.  Beautiful.

PS. I avoided spotify-json, StaticJson/autojsoncxx, Google Prototype Buffers, Code Synthesis’s ODB, the sqlite JSON1 Extension, C++ reflection libraries like RTTR, lots of code from Stiffstream and Chilkat, etc. because while they are all brilliant and compelling, they bring extra weight.  The world keeps churning though, so keep searching.  Also, there are cases where my choices do not fit, most obviously being cross-platform mobile apps, which will have to be saved for another post… 🙂

I’m writing trading software called A better Trader.  For the web UI, I started using some basic JQuery and JQuery UI, with a mind to kick the tires on Bootstrap.  I dug in and after looking at a  bazillion frameworks and libraries (some fading, some up and coming, NONE really owning it)… K.I.S.S. is ruling the day.  I’m finding myself happy with the following:

• HTML5 with support for multimedia, flexboxes, dragon droppings, etc.
• JQuery && (JQuery UI || bootstrap)
• D3 for all things visual… so good…
• iframes!  Yep!  API calls typically return SVG-based graphic components in iframes, and API parent pages combine them into application-like views.

On the back end, I’m doing RESTful services with boost ASIO.  It’s a light clean set of tools.  I will probably stand up bootstrap on the front end, and node.js on the back end, for comparison.  We’ll see how it goes.

The stupid Windoze gaming sirens had me out of linux for a bit on my desktop. Back with a vengeance, and a huge emerge world. Hours of fun! The nice thing is I’ve gotten into gentoo enough that I no longer feel the need to document every massive emerge. For the most part, just keep using these strategies until it works itself out – while keeping your brain engaged of course:

• emerge -DavuN @world
• emerge -DavuN –with-bdeps=y @world
• emerge -DavuN @world –autounmask-write
• emerge –resume –skipfirst
• emerge –unmerge (i unmerge the SHIT out of all conflicting non-world packages! keep it clean! :O)
• emerge -1v (anything that’s missing!)
• gcc-config (as early as possible)
• eselect (boost|java-vm|php|python|etc….)
• use eix and equery to answer any package questions
• get the latest gentoo-sources and configure that kernel!
• don’t forget to emerge nvidia-drivers or any other kernel-specific package
• google for help – gentoo support information is AMAZING
• dispatch-config
• emerge -av –depclean
• revdep-rebuild
• lafilefixer –justfixit
• python-updater
• perl-cleaner –all
• targeted rebuild, for those nasty upgrades: revdep-rebuild –library libpng14.so.14
• find broken autotools stuff: find /usr/ -name ‘*.la’ -exec grep png14 -c {} +|grep \:1
• rinse and repeat!

I had a few new wrinkles…

• considered emerging gnome-base/gnome (new) to get GNOME 3 but it is soft-masked and who knows how much trouble would follow
• stupid libpng upgrades always cause everyone all kinds of headaches – see this awesome post

revdep-rebuild --library '/usr/lib64/libpng14.so.14' -- --keep-going
emerge -1av --keep-going $(find /usr \( -name "*.la" -o -name "*.pc" -o -name "*-config" -o -name "*.pm" \) -exec grep -H png14 {} \; | cut -d : -f 1 | xargs qfile -CSq | sort | uniq)

• removed nsplugin USE flag from picasa and acroread since it thought that firefox (not firefox-bin) was a requirement, stooopid things
• Qt needed a bump, which should happen all at once, which is impossible 🙂
• of course i had to add a FEW more bells and whistles!
• emerge -DavuN xfce4-meta xfce4-verve-plugin xfce4-mixer xfce4-taskmanager xfwm4-themes thunar thunar-volman tumbler thunar-archive-plugin google-chrome
• etc

I’m using the cross-platform Qt C++ library to write an app for linux, OS X, and Windows. Qt source can be compiled to a wide range of systems and devices, with two obvious omissions that are not likely to be filled any time soon: iPhone and Android. I know I am going to need client code on these devices, and I know that the client code is going to be extremely similar to the Qt client code. How do you design for this?

Applying the concept of separation of concerns, you design your classes into layers. The base layer consists of high level concept classes that perform all fundamental actions using generic data types. Then, the derived layer is used to get the job done, using the most useful and powerful lower level tools at your disposal. Push everything you can into the base layer, until you hit the limit of non-portable information and data types. It’s a beautiful way to code.

To be more specific, in my case, the base class layer will be using C++, including the STL, which gives me tons of power. I will pull in boost if even more horsepower is needed. The derived class layer for the first round of clients will be Qt-powered. The Qt library has a massive amount of real-world usefulness, supporting everything from http to video playback. I have not gotten to the iPhone and Android clients yet so the whole concept may change, but here’s my current plan. The iPhone code will be Objective C with the C++ base class layer linked in. I will attempt to incorporate the C++ base class layer into the Android code using the NDK.

Here’s a quick example of the layer approach, in this case used to quickly set up profiling using Qt’s QTime class at the lower level: (continued…)

I’m updating my boost installations to latest CVS, here are a couple quick notes. In addition, instead of having blog chatter all over the place, I’ll consolidate my boost installation notes onto my wiki, see here. (continued…)