Causam: Difference between revisions

From Bitpost wiki
No edit summary
(Replaced content with "=== DEPLOY A NEW RELEASE === === DEV SETUP === === INSTALL CRAWLER ON A PI === === COMMONS WORKFLOW (thanks greg and brandon!) === === nop-client-c === === MAINTE...")
 
(8 intermediate revisions by the same user not shown)
Line 1: Line 1:
=== DEPLOY A NEW RELEASE ===
=== DEPLOY A NEW RELEASE ===
TODO improve to use git release branches (git branches are cheeeeap and allow backporting of fixes)
TODO improve to simplify and automate
* update [nu set-scala-version] to use command line params, and output a diff
* switch from [nu force-tag] to [nu create-branch]
* add support to clone all of jenkins, rename it to release name, and update to use the new release branches instead of master
* make sure the stripping of the -SNAPSHOT (eg 4.0.0-SNAPSHOT => 4.0.0) happens in the release branch
* then when you are ready to bump master, it is simply a matter of bumping scala versions from (eg) 4.0.0-SNAPSHOT to 4.1.0-SNAPSHOT (it STAYS a development branch)


nu force-tag 4.0.0-2016Oct22-AmazingNewFeature # force a git tag of the stable old version on all repos
=== DEV SETUP ===
nu set-scala-version # stamp the new scala version into all projects
nu diff # completely verify that everything looks good - there's almost always something that needs manual correction!
nu sync --major Stamping components with new 4.1.0 version
* now use Jenkins to kick it all off - dependencies are all set, you just need to build [http://jenkins.energynet.link:8080/job/OLYV4/job/PXRS/ pa-xi-monitor-requests-scala]


 
=== INSTALL CRAWLER ON A PI ===
 
== ORIGINAL PROCESS used for 4.0.0 ==
* we will mainly use nop-util to set up the new version
* first edit util.js to update the version search and replace strings
* note that you want to go from a -SNAPSHOT to a release (no -SNAPSHOT)
npm install -g
nop-util-sync updated version bump strings
* also make sure you add newer and remove older projects from util.js
* then it's all scriptable...
nu force-tag 4.0.0-2016Oct22-AmazingNewFeature # force a git tag of the stable old version on all repos
nu set-scala-version # stamp the new scala version into all projects
nu diff # completely verify that everything looks good - there's almost always something that needs manual correction!
nu sync --major Stamping components with new 4.1.0 version
* now use Jenkins to kick it all off - dependencies are all set, you just need to build [http://jenkins.energynet.link:8080/job/OLYV4/job/PXRS/ pa-xi-monitor-requests-scala]


=== COMMONS WORKFLOW (thanks greg and brandon!) ===
=== COMMONS WORKFLOW (thanks greg and brandon!) ===


code away
=== [[nop-client-c]] ===
 
sbt publishLocal (or via gpy - NOTE that only libraries have a gpy)
 
puts it in ~/.ivy2/local
 
(sbt will use that then)
 
 
(code is happy and pushed)
 
(once you push it to stash, it will be in artifactory)
 
 
ungpy (removes ivy/local AND cache - good to do ofen to avoid sbt confusion) (esp before coding)
 
(then it will reresolve next time)
 
"sbt uses ivy instaed of maven, doesn't know snapshots"
 
 
then either publishLocal again, or just carry on and it will use artifactory
 
 
(these just do a self-contained local build)
 
activator dist
 
activator compile
 
 
=== nop-client message flow ===
 
    EDGE                                CLIENT
   
                                        initial certificate provisioning
                                       
                                        olympicmachines < hello
                                        simplepointframe < heartbeat keepalives (as needed)
    EDGE becomes device-aware
    build config
    send to machine                    olympicmachines > polling config
                                        olympicmachines < polling config ACK
                                       
    start polling                      simplepointframe < data
                                       
                                        (dead) < (buffer until server becomes responsive)


=== MAINTENANCE JOBS ===
=== MAINTENANCE JOBS ===


    Rich explained again about where he puts maintenance job type tasks
=== Set up a Windows client to use Stash ssh key ===
 
    kick off AggregationHelper
        /home/m/development/causam/git/np/nop-commons-parent-scala/mongo/src/main/scala/com/pa/mongo/aggregations/AggregationHelper.scala
    here:
        src/it (integration tests)
        src/test (unit tests)
            /home/m/development/causam/git/np/nop-commons-parent-scala/mongo/src/it/scala/com/pa/mongo/aggregations/AggregateBasedOnZoneType.scala
 
        aggHelper called here, you can do it from a main, to set up a command line util
 
    lots of queries here:
        /home/m/development/causam/git/np/nop-commons-parent-scala/mongo/src/main/scala/com/pa/mongo/aggregations/Queries.scala
 
    no cron jobs for production yet
    rich does things manually
    schedule worker is a new project, like queue worker but on a schedule


=== GEM provisioning ===


{| class="mw-collapsible mw-collapsed wikitable"
=== INSTALL AND COMPILE north-poller ON UBUNTU ===
! Set up a Windows client to use Stash ssh key
|-
| Use git-for-windows exclusively, it doesn't need putty.
* Install Git for Windows, set it up to use its own ssh (not plink).
* Set up this folder with .ssh configuration
C:\Users\##user##\.ssh\config
* Copy the .ssh folder from [https://www.dropbox.com/home/Power%20Analytics dropbox], the config looks like this:
Host git.poweranalytics.io
    IdentityFile ~/.ssh/git.poweranalytics.io/id_rsa
* Now you should be able to pull:
cd C:\Michael's Data\development\power_analytics\gec
git clone ssh://git@git.poweranalytics.io:7999/gec/pa-pdp-xiagent-csharp.git
|}
{| class="mw-collapsible mw-collapsed wikitable"
! GEM provisioning
|-
|
    # note that ruby will let you do this as any user
    # so you have a choice to avoid root if you have access to /etc
    # if you cannot write to /etc, you'll need to run as root
    su -
    gem sources
        # if no energynet, do this
        gem sources -a http://gem.energynet.io/
    gem list --remote nop-client
        *** REMOTE GEMS ***
        gem nop-client (1.4.0)
    gem install nop-client
        (done, didn't need [gem install activesupport] prolly bc i have already done it)
        (or if errors: gem install activesupport, and continue)
    go to:
        https://certman-prod.energynet.io/tokens
        taylor: your certman u/p (box and http auth)
        ryan "grab any customer=2(belllabs) or 3(test) with redeemed=false"
    nop-redeem-token 756f50e4-2672-faf3-a157-b4e624411525
        DONE
        installed it all to:
            /etc/nop-client/ssl/CA-belllabs.energynet.io/


    TO DISABLE
=== INSTALL AND RUN north-poller BINARY ON PI ===
        cp /etc/nop-client/ssl/(provider) /etc/nop-client/ssl-disabled/
|}
{| class="mw-collapsible mw-collapsed wikitable"
! INSTALL AND COMPILE north-poller ON UBUNTU
|-
|
    install sbt (see wiki)
    add the repos used by the team's projects:
        mkdir ~/.sbt
        subl ~/.sbt/repositories
            [repositories]
            local
            artifactory-releases: http://artifactory.energynet.link:8081/artifactory/libs-release/
            artifactory-snapshots: http://artifactory.energynet.link:8081/artifactory/libs-snapshot/
            ivy-releases: http://artifactory.energynet.link:8081/artifactory/ivy-releases, [organization]/[module]/(scala_[scalaVersion]/)(sbt_[sbtVersion]/)[revision]/[type]s/[artifact](-[classifier]).[ext]
            anormcypher: http://artifactory.energynet.link:8081/artifactory/anormcypher
    install "commons parent" (includes all common shared libraries):
        cd ~/development/causam/git
        git clone ssh://git@git.energynet.io:7999/np/nop-commons-parent-scala.git
        cd nop-commons-parent-scala
        # make sure you are connected to causam vpn, sbt will pull from energynet.link
        sbt compile
        # OR gpy.sh <-- this will run tests, etc.  (not needed)
        (you may have to run it twice to get everything to resolve, esp if you forgot to run vpn)
    install north-poller
        cd ~/development/causam/git
        git clone ssh://git@git.energynet.io:7999/en/causam-north-poller-scala.git
        cd causam-north-poller-scala
        sbt compile
    run it!
        # put the providers you want to talk to in /etc/nop-client/ssl
        # push the ones you want to ignore to /etc/nop-client/ssl-disabled
        sbt run
    there she goes


|}
=== JAVASCRIPT REVERSE ROUTING TO CALL SCALA FUNCTIONS ===
{| class="mw-collapsible mw-collapsed wikitable"
! INSTALL AND RUN north-poller BINARY ON PI
|-
|
    # add apt repo: apt.energynet.io
    # don't try to use add-apt-repository from package software-properties-common
    # it checks up and fails bc there is no "Raspbian/jessie template" in the repo
    # first, add support for https-over-apt
    su -
    apt-get install apt-transport-https
    # then just use Brandon's list!
    emacs /etc/apt/sources.list
        deb http://mirrordirector.raspbian.org/raspbian/ jessie main contrib non-free rpi
        # Uncomment line below then 'apt-get update' to enable 'apt-get source'
        #deb-src http://archive.raspbian.org/raspbian/ jessie main contrib non-free rpi
        deb https://apt.energynet.io/ gecv3 main
        deb https://apt.energynet.io/ nopv3 main
    # add GPG key
    apt-key adv --keyserver keyserver.ubuntu.com --recv-keys D38F4191
    apt-get update
    # add the user under which north poller runs
    adduser app (p*)
    apt-get install causam-north-poller-scala
    service causam-north-poller-scala start
    tail -f /var/log/syslog
|}

Latest revision as of 19:08, 7 January 2017

DEPLOY A NEW RELEASE

DEV SETUP

INSTALL CRAWLER ON A PI

COMMONS WORKFLOW (thanks greg and brandon!)

nop-client-c

MAINTENANCE JOBS

Set up a Windows client to use Stash ssh key

GEM provisioning

INSTALL AND COMPILE north-poller ON UBUNTU

INSTALL AND RUN north-poller BINARY ON PI

JAVASCRIPT REVERSE ROUTING TO CALL SCALA FUNCTIONS