|
|
(29 intermediate revisions by the same user not shown) |
Line 16: |
Line 16: |
| {| class="wikitable" | | {| class="wikitable" |
| ! [http://www.aosabook.org/en/index.html Architecture of major open source apps] | | ! [http://www.aosabook.org/en/index.html Architecture of major open source apps] |
| | |} |
| | {| class="wikitable" |
| | ! [[Security]] |
| |} | | |} |
| {| class="wikitable" | | {| class="wikitable" |
Line 21: |
Line 24: |
| |} | | |} |
| {| class="mw-collapsible mw-collapsed wikitable" | | {| class="mw-collapsible mw-collapsed wikitable" |
| ! String escape formatting across different languages and systems | | ! nosql enhances, not replaces, SQL |
| |- | | |- |
| | | | | Not all data should be denormalized, and not all data should be normalized. The optimal mix considers the extent of the data. |
| * c++ to JSON: always use nlohmann::json j.dump() to encode, to ensure strings are properly escaped | | |
| * JSON to c++: always use nlohmann::json j.parse() " | | * Precise schemas are good when not overdone |
| * c++ to Javascript: use raw_to_Javascript() to properly escape | | * When a container has an array with a large number of elements, it should be normalized |
| * c++ to sqlite: use SqliteLocalModel::safestr(), which uses double_doublequotes(str)
| | * Sparse data and heterogeneous data are the best candidates for denormalization |
| | |
| | Postgres with JSON allows an elegant combination of nosql and SQL. |
| |} | | |} |
| {| class="mw-collapsible mw-collapsed wikitable" | | {| class="mw-collapsible mw-collapsed wikitable" |
Line 59: |
Line 64: |
| |- | | |- |
| | Among competing hypotheses, the one with the fewest assumptions should be selected. | | | Among competing hypotheses, the one with the fewest assumptions should be selected. |
| |}
| |
| {| class="mw-collapsible mw-collapsed wikitable"
| |
| ! Security
| |
| |-
| |
| | Notes to review before interviews, etc:
| |
|
| |
| ---------------------------
| |
| fedramp cloud certification
| |
| ---------------------------
| |
| strong encryption, eg AES 256-bit
| |
| data should be encrypted before it leaves end-user "organization"
| |
| encrypt: data-at-rest, data-in-transit, data-in-use
| |
| encryption keys MUST be kept within end-user org
| |
| CSP requirements: implement security, third-party assess, maintain authorization, comply with continous monitoring
| |
|
| |
| basics review
| |
|
| |
| public-private key
| |
| ------------------
| |
| plain -> encrypted -> plain
| |
| ^ ^
| |
| publickey privatekey
| |
|
| |
|
| |
| SSL HANDSHAKE
| |
| -------------
| |
| client server
| |
| -> hello ->
| |
| < cert
| |
| (check)
| |
| send secret encryped with server public key > decrypt secret)
| |
| (opt) send client cert > (check - but not otherwise used?)
| |
| ack>
| |
| <ack
| |
| exchange with shared secret key
| |
|
| |
| best encryption to date
| |
| ssl labs:
| |
| Key RSA 2048 bits (e 65537)
| |
| signature SHA256withRSA
| |
| certchain includes Let's Encrypt Authority X3, RSA 2048 bits, Signature: SHA256withRSA
| |
| TLS 1.2 (not allowed: TLS 1.1, 1.0; SSL 3, SSL 2
| |
| TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (0xc02f) ECDH secp256r1 (eq. 3072 bits RSA) FS 128
| |
| TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256 (0xc027) ECDH secp256r1 (eq. 3072 bits RSA) FS 128
| |
| TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA (0xc013) ECDH secp256r1 (eq. 3072 bits RSA) FS 128
| |
| TLS_DHE_RSA_WITH_AES_128_GCM_SHA256 (0x9e) DH 2048 bits FS 128
| |
| TLS_DHE_RSA_WITH_AES_128_CBC_SHA256 (0x67) DH 2048 bits FS 128
| |
| TLS_DHE_RSA_WITH_AES_128_CBC_SHA (0x33) DH 2048 bits FS
| |
|
| |
| review
| |
|
| |
| AES 256-bit for encryption
| |
| aka Rijndael, which won in the original NIST AES selection process (in 2001!)
| |
| symmetric key algorithm (same key to encrypt and decrypt)
| |
| block size 128
| |
| key length 256 (can be 128 and 192)
| |
|
| |
| SHA-256 for signatures
| |
| a bit-specific SHA-2
| |
| deprecates SHA-1
| |
| NOT deprecated by SHA-3 (an alternative developed through NIST competition)
| |
| 256-bit "hash values" aka digests aka signatures)
| |
| server + client ssl certificate process
| |
|
| |
| HMAC is a signature with a key - what I'm doing with JWT (HMAC-SHA256)
| |
|
| |
| broken: RC2 RC4 DES IDEA ...
| |
|
| |
| next-gen: https://security.stackexchange.com/questions/135054/whats-after-aes
| |
| TLS 1.3, which is still a draft, is going with two authenticated ciphers as its required choices:
| |
| AES-GCM, with either 128- or 256-bit keys;
| |
| ChaCha20/Poly1305
| |
| The reasons we have two are the following:
| |
| As a backup—if one of them is broken, everybody can switch to the other;
| |
| They have different strengths:
| |
| AES-GCM has excellent hardware support in many platforms;
| |
| ChaCha20/Poly1305 has faster all-software implementations than AES-GCM does.
| |
|
| |
| good for https with SSL certs, according to SSL labs:
| |
| TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (0xc02f) ECDH secp256r1 (eq. 3072 bits RSA) FS 128
| |
| TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256 (0xc027) ECDH secp256r1 (eq. 3072 bits RSA) FS 128
| |
| TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA (0xc013) ECDH secp256r1 (eq. 3072 bits RSA) FS 128
| |
| TLS_DHE_RSA_WITH_AES_128_GCM_SHA256 (0x9e) DH 2048 bits FS 128
| |
| TLS_DHE_RSA_WITH_AES_128_CBC_SHA256 (0x67) DH 2048 bits FS 128
| |
| TLS_DHE_RSA_WITH_AES_128_CBC_SHA (0x33) DH 2048 bits FS
| |
|
| |
|
| |
| NIST (National Institute of Standards and Technology)
| |
|
| |
| SHA-3 competition 2007-2012 (SHA = Secure Hash Algorithm)
| |
| 64 entries
| |
| 5 finalists: BLAKE, Grøstl, JH, Keccak and Skein
| |
| winner: Keccak
| |
| schneier's feedback:
| |
| Yes, I would have rather my own Skein had won, but it was a good choice.
| |
|
| |
| NSA has Suite A (classified algorithms that will not be released) and Suite B cryptography algorithms
| |
| Suite B's components are:
| |
|
| |
| Advanced Encryption Standard (AES) with key sizes of 128 and 256 bits.
| |
| For traffic flow, AES should be used with either
| |
| the Counter Mode (CTR) for low bandwidth traffic or
| |
| the Galois/Counter Mode (GCM) mode of operation for high bandwidth traffic (THIS CAN BE PARALLELIZED)
| |
| Elliptic Curve Digital Signature Algorithm (ECDSA) – digital signatures
| |
| Elliptic Curve Diffie–Hellman (ECDH) – key agreement
| |
| Secure Hash Algorithm 2 (SHA-256 and SHA-384) – message digest
| |
|
| |
| talk about my projects
| |
|
| |
| nopc
| |
| energy market
| |
| need to measure power behind the consumers meter
| |
| i wrote a client that lives on embedded devices
| |
| i cross-compiled openssl, libmodbus, libcurl, libwebsockets, zlib
| |
|
| |
| ssl / websockets work:
| |
| libwebsockets uses openssl, but i have played a bit with optional ssl: wolfssl
| |
| has streaming media support, including the HC-128 and RABBIT stream ciphers
| |
| Standard ciphers are supported, including EDH on both the client and server side.
| |
| wolfSSL recently added support for BLAKE2b, one of the SHA-3 finalists as well.
| |
| wolfSSL’s underlying cryptography library is called wolfCrypt.
| |
| i wouldn't want to stray from openssl unless there was a more compelling reason, since it gets so many eyeballs on it
| |
| people don't like its depth of support for other algorithms, just stick with best-in-class
| |
| Key RSA 2048 bits (e 65537)
| |
| signature SHA256withRSA
| |
| pki: rsa
| |
|
| |
| given a guid token, it uses that to generate ssl certs on the device
| |
| then it runs 24/7 on the embedded device
| |
| it has a tight loop with callbacks, C is gross but you can do pretty much anything
| |
| in factthats the problem, you have to do everything!
| |
| again, i heavily rely on reusable helpers
| |
| so it has a single-threaded loop with callbacks
| |
| poll:
| |
| if !config configure client
| |
| else poll and store meter data, buffered to memory then volatile then nonvolatile storage
| |
| if ws not connected initiate connection
| |
| service the connection callbacks (sending bi-directionally as needed)
| |
|
| |
| just added jwt to abettertrader
| |
| uses HMAC-SHA256 (uses a secret during signature generation)
| |
| client sends u/p
| |
| server builds header+payload+HMAC-SHA256 signature with its own secret
| |
| server sends back base64-encoded token to client
| |
| client uses token on every API call
| |
| when token times out, user must log in again
| |
| server can almost instantly determine user's priviledges without a db hit
| |
| that's why they call it stateless
| |
|
| |
| abettertrader c++ based webserver, gets A+ rating at ssllabs
| |
| via haproxy!
| |
| I serve up several domains from my home
| |
| most of them use SNI with apache
| |
| with SNI the hostname is sent outside of the encrypted traffic
| |
| so you can serve up different apache sites based on the requested hostname
| |
| it's fantastic - for years i had to serve up my different sites on different ports
| |
| and it was hard to do https that way
| |
| but! ... i'm running a C++ https server as well
| |
| and i had to solve the problem of getting incoming https traffic to that server running on that port
| |
| haproxy is THE BOMB for these kinds of things
| |
| i configured haproxy to read the domain name and redirect traffic to the c++ http server and port
| |
| but i am so excited about this, i got these bonuses with zero effort:
| |
| ALL ssl handshaking is now done by haproxy! i just give it ALL my certs, and it does the negotiations
| |
| i was able to limit availabe ciphers to those listed as secure at ssllabs
| |
| ssl-default-bind-options no-sslv3 no-tls-tickets force-tlsv12
| |
| ssl-default-bind-ciphers AES128+EECDH:AES128+EDH
| |
| ALSO i was able to turn on HSTS - this forces all http requests into https requests
| |
| and that got me an A+ rating on ssllabs - for ALL My sites - in one fell swoop!
| |
| i felt like that giant that killed 7 flies or whatever
| |
|
| |
| abettertrader uses a map of lambas and regexs
| |
| when a url comes in, it plays it against all the registered regexs
| |
| if it finds a match, it calls the lambda
| |
| it's really fast and really fun
| |
|
| |
| i set up node.js scripting for my continuous integration
| |
| i have an open source package called radscripts - it does an automatic semver bump on every commit
| |
| i'm totally addicted to that
| |
| you can turn it off and still get all the benefits -
| |
| i turned off auto-tagging at causam because they wanted to control the specific numbers of releases
| |
| but semver dictates: ...
| |
|
| |
| i am always tinkering, i keep track of my projects in phabricator, an agile ticket tool that sprang out of facebook
| |
| do you want to see it?
| |
| willcodeforcoffee.org moodboom/G
| |
|
| |
| write a c++ app that sorts an array then encrypts it then decrypts it
| |
| int main() {
| |
| return 0;
| |
| }
| |
|
| |
| review c++11, c++14, interview questions
| |
| RAII resource acquisition is initializaztion- constructor acquires, destructor releases
| |
|
| |
| c++
| |
| c++11 features I love:
| |
| automatic type detection - this is great esp for iterators so you don't have to type as much
| |
| for loops
| |
| if you don't need to walk forwards or backwards as you loop, these greatly simplify code profile
| |
| but if you need the iterator as you loop, you can stick with the old way
| |
| lambda expressions - really fast to write inline functions
| |
| move semantics - you don't have to copy out results when you're done in a function, you can move them - and it's largely automatic, really nice
| |
| initialization syntax - i love this for creating test data, you can easily initialize big arrays, whatever, right in code
| |
| delegating constructors - this si great, so you don't have to rewrite all the base class constructors to be able to add a new one
| |
| threading - a lot of this is available with boost - but it's nice to have the standard incorporate all the best boost work
| |
| c++14 seems much more incremental than 11 was
| |
| you can use auto for function return types - that was already how lambdas worked in c++11, i thought...
| |
| lamdba parameters can be auto - kind of like templating, seems really crazy and cool - haven't played with this much yet
| |
| and closures, how cool is that!!! "lambda captures" it carries along the scope from where it was called.
| |
| javascript of course makes us a huge fan of this.
| |
| but i wonder about the performance penalty involved. - again, haven't played with it much
| |
| deprecated keyword, binary literals
| |
| and then c++17 is on the way - c++ has always made me happy, i find it incredibly elegant compared to lower C or higher java/C# arenas
| |
| i think javascript actually hits a sweeter spot than those
| |
| so i'm enjoying C++ and javascript more than anything these days
| |
| i wrote some utilities in node so i can use node for scripting, which has been really productive
| |
| but i have to say, python would be a welcome addition
| |
|
| |
| c++ containers
| |
|
| |
| i use hashmaps of pointers via unordered_map
| |
| unordered set lets you contain pointers, and specify the hash and equals values for the object pointed to
| |
| typedef std::unordered_set<AutotradeParameterSet*,PersistentIDObject_hash,PersistentIDObjects_equal > AutotradeParameterSets;
| |
| then you can set up a second "index" into the object store
| |
| you have to maintain all indexes as you add and remove, of course
| |
|
| |
| unordered_map uses the hash to find the right bucket O(1)
| |
| map uses a binary tree and a comparison operator O(log(n))
| |
| std::find() on vector uses quicksort O(N*log(n)) or insertsort which uses heapsort for worst case
| |
| mergesort is good to preserve order of equal items (in-place)
| |
|
| |
| python has TimSort, pretty cool - looks for presorted sections, then merges those, COOL
| |
| in use in python since about 2002
| |
|
| |
| i also use a sorted vector class i derived from the standard vector class
| |
| it too can use pointers instead of objects, allowing for multiple indexes on a set of objects
| |
| it has push_unsorted(), bSorted() and sort(sort_function) functions
| |
| usually with vectors you use lower_bound to find things
| |
| sorted vector has find helpers, and they always sort if unsorted
| |
| it's really useful when you have a huge amount of objects and only sort on occasion
| |
|
| |
| - nagging question: how do you add functionality to a product that's new?
| |
| this was a very valuable question that I didn't answer well
| |
| can i ammend my answer?
| |
| i mentioned that I would diagram function flows - that would definitely be a key strategy
| |
| and to read any and all available documentation on the software
| |
| what should have also been part of my answer:
| |
| generate my own documentation as needed
| |
| a fantastic tool for that is doxygen
| |
| i ran it yesterday against a recent project, and it did a nice job creating class hierarchies etc.
| |
|
| |
| clang-tidy is suggested to be the best on reddit
| |
| even has a -fix flag to fix in place, ha
| |
| cmake can call it for you!!
| |
| cppcheck
| |
| coverity for Paid solutions
| |
|
| |
| jsonlint
| |
| |} | | |} |
| |} | | |} |
Line 829: |
Line 576: |
| |- | | |- |
| | | | | |
| | {| class="mw-collapsible mw-collapsed wikitable" |
| | ! String escape formatting across different languages and systems |
| | |- |
| | | |
| | * c++ to JSON: always use nlohmann::json j.dump() to encode, to ensure strings are properly escaped |
| | * JSON to c++: always use nlohmann::json j.parse() " |
| | * c++ to Javascript: use raw_to_Javascript() to properly escape |
| | * c++ to sqlite: use SqliteLocalModel::safestr(), which uses double_doublequotes(str) |
| | |} |
| | [[Postgres]] |
| | |
| [[Simple-Web-Server]] | | [[Simple-Web-Server]] |
| | |
| | [[Robot Operating System]] |
|
| |
|
| [[C++ https libraries]] | | [[C++ https libraries]] |
Line 889: |
Line 649: |
| CC=gcc-4.9 | | CC=gcc-4.9 |
| |} | | |} |
| | |} |
| | <!-- |
| | |
| | |
| | =========================================================================================================================================================================================================================================================================================== |
| | |
| | |
| | --> |
| | {| class="wikitable" |
| | ! [[git]] |
| |} | | |} |
| <!-- | | <!-- |
Line 898: |
Line 668: |
| --> | | --> |
| {| class="mw-collapsible mw-collapsed wikitable" | | {| class="mw-collapsible mw-collapsed wikitable" |
| ! C/C++ debugging | | ! Debugging |
| |- | | |- |
| | | | | |
| | {| class="mw-collapsible mw-collapsed wikitable" |
| | ! Chrome capture large JSON variable |
| | |- |
| | | This is just pointlessly bizarre: |
| | * hit a breakpoint in the chrome debugger |
| | * right-click a variable and say "copy to global variable" (console will show name, typically "temp1") |
| | * push the variable to the clipboard by typing this in the console: |
| | copy(temp1) |
| | |} |
| | {| class="mw-collapsible mw-collapsed wikitable" |
| | ! Visual Studio Code capture large string variable |
| | |- |
| | | While debugging, you can use the Debug Console to print memory, including the content of strings that are clipped by default in the variables and watch windows. |
| | View > Open View > Debug Console |
| | From there, send gdb a command to print memory – 300 characters of a string in this example: |
| | -exec x/300sb Query.c_str() |
| | |} |
| {| class="mw-collapsible mw-collapsed wikitable" | | {| class="mw-collapsible mw-collapsed wikitable" |
| ! Qt Creator conditional breakpoint | | ! Qt Creator conditional breakpoint |
Line 1,026: |
Line 813: |
|
| |
|
| --> | | --> |
| {| class="mw-collapsible mw-collapsed wikitable" | | {| class="wikitable" |
| ! RESTFul http | | ! [[Node.js]] |
| |- | | |}<!-- |
| |
| | |
| {| class="mw-collapsible mw-collapsed wikitable" | | |
| ! Verbs | | =========================================================================================================================================================================================================================================================================================== |
| |- | | |
| |
| | |
| * get - to make requests that are simple -response may be complex
| | --> |
| * post - to push a form or JSON to the server
| | {| class="wikitable" |
| | ! [[React]] |
| | |}<!-- |
| | |
| | |
| | =========================================================================================================================================================================================================================================================================================== |
| | |
| | |
| | --> |
| | {| class="wikitable" |
| | ! [[Vite]] |
| |} | | |} |
| | <!-- |
| | |
| | |
| | =========================================================================================================================================================================================================================================================================================== |
| | |
| | |
| | --> |
| | {| class="wikitable" |
| | ! [[JSON]] |
| |} | | |} |
| <!-- | | <!-- |
Line 1,045: |
Line 851: |
|
| |
|
| --> | | --> |
| {| class="mw-collapsible mw-collapsed wikitable" | | {| class="wikitable" |
| ! css | | ! [[HTML]] |
| |- | | |} |
| |
| | <!-- |
| /* class="first second" */
| | |
| .first.second {}
| | |
| /* class="first" OR class="second" */
| | =========================================================================================================================================================================================================================================================================================== |
| .first, .second {}
| | |
| /* class="first second", or class="second", or class="third second", or class="second third" */
| |
| .second {}
| |
| /* apply to any .child at any depth under .parent */
| |
| .parent .child {}
| |
| /* apply to .child if it is DIRECTLY under .parent */
| |
| .parent > .child {}
| |
|
| |
|
| | --> |
| | {| class="wikitable" |
| | ! [[CSS]] |
| |} | | |} |
| <!-- | | <!-- |
Line 1,073: |
Line 876: |
| | | | | |
| {| class="mw-collapsible mw-collapsed wikitable" | | {| class="mw-collapsible mw-collapsed wikitable" |
| ! Sqlite table size analysis | | ! Count records within a range |
| |- | | |- |
| | | | | This groups records into ranges, sorts by them, and gives a count, sweet: |
| running out of space on bp
| | select count(*), id/1000000 as groupid from AccountHistory group by groupid; |
| which is due to sqlite db bloat
| |
| which is due to
| |
| found with sqlite3_analyzer, here:
| |
| http://www.renedohmen.nl/blog/2014/05/determine-sqlite-table-size-on-disk/
| |
| http://www.sqlite.org/download.html
| |
| (put it under viper ~/apps)
| |
| ran it like this:
| |
| cd ~/development/thedigitalage/AbetterTrader/server/db_archive
| |
| ~/apps/sqlite-tools/sqlite-tools-linux-x86-3220000/sqlite3_analyzer at_server_test.sqlite__2018-03-14_EOD__.sqlite
| |
| |} | | |} |
| {| class="mw-collapsible mw-collapsed wikitable"
| | [[postgres]] - [[sqlite]] - [[mysql]] - [[SQL Server]] - [[Robo 3T]] - [[DBeaver]] - [[pgadmin4]] |
| ! Sqlite timestamp-to-readable-date query
| |
| |-
| |
| |
| |
| select quote, timestamp, strftime('%Y-%m-%d %H:%M:%S', datetime(timestamp, 'unixepoch')) from StockQuotes as s where s.symbol="TSLA" order by timestamp;
| |
| |} | | |} |
| | <!-- |
| | |
| | |
| | =========================================================================================================================================================================================================================================================================================== |
| | |
| | |
| | --> |
| | {| class="wikitable" |
| | ! [[Meteor]] |
| |} | | |} |
| <!-- | | <!-- |
Line 1,103: |
Line 902: |
| {| class="wikitable" | | {| class="wikitable" |
| ! [[Android]] | | ! [[Android]] |
| | |} |
| | <!-- |
| | |
| | |
| | =========================================================================================================================================================================================================================================================================================== |
| | |
| | |
| | --> |
| | {| class="wikitable" |
| | ! [[Arduino]] |
| | |} |
| | <!-- |
| | |
| | |
| | =========================================================================================================================================================================================================================================================================================== |
| | |
| | --> |
| | {| class="wikitable" |
| | ! [[Raspberry Pi]] |
| | |} |
| | <!-- |
| | |
| | |
| | =========================================================================================================================================================================================================================================================================================== |
| | |
| | --> |
| | {| class="wikitable" |
| | ! [[iOS]] |
| | |} |
| | <!-- |
| | |
| | |
| | =========================================================================================================================================================================================================================================================================================== |
| | |
| | |
| | --> |
| | {| class="wikitable" |
| | ! [[.NET Core]] |
| |} | | |} |
| <!-- | | <!-- |
Line 1,122: |
Line 959: |
| --> | | --> |
| {| class="wikitable" | | {| class="wikitable" |
| ! [[Python]] | | ! [[Kotlin]] |
| | |} |
| | <!-- |
| | |
| | |
| | =========================================================================================================================================================================================================================================================================================== |
| | |
| | |
| | --> |
| | {| class="wikitable" |
| | ! [[Maven]] |
| |} | | |} |
| <!-- | | <!-- |
Line 1,133: |
Line 980: |
| {| class="wikitable" | | {| class="wikitable" |
| ! [[Scala]] | | ! [[Scala]] |
| | |} |
| | <!-- |
| | |
| | |
| | =========================================================================================================================================================================================================================================================================================== |
| | |
| | |
| | --> |
| | {| class="wikitable" |
| | ! [[Python]] |
| |} | | |} |
| <!-- | | <!-- |
Line 1,167: |
Line 1,024: |
| /etc/init.d/apache restart | | /etc/init.d/apache restart |
| |} | | |} |
| |}
| |
| <!--
| |
|
| |
|
| |
| ===========================================================================================================================================================================================================================================================================================
| |
|
| |
|
| |
| -->
| |
| {| class="wikitable"
| |
| ! [[git]]
| |
| |} | | |} |
| <!-- | | <!-- |