Causam
DEPLOY A NEW RELEASE
- see [causam/docs/Release Process.MD]
- use Jenkins to kick it all off - dependencies are all set, in theory at least, you just need to build pa-xi-monitor-requests-scala
TODO improve to simplify and automate
TODO update [nu set-scala-version] to use command line params, and output a diff
ORIGINAL PROCESS used for 4.0.0
- we will mainly use nop-util to set up the new version
- first edit util.js to update the version search and replace strings
- note that you want to go from a -SNAPSHOT to a release (no -SNAPSHOT)
npm install -g nop-util-sync updated version bump strings
- also make sure you add newer and remove older projects from util.js
- then it's all scriptable...
nu force-tag 4.0.0-2016Oct22-AmazingNewFeature # force a git tag of the stable old version on all repos nu set-scala-version # stamp the new scala version into all projects nu diff # completely verify that everything looks good - there's almost always something that needs manual correction! nu sync --major Stamping components with new 4.1.0 version
- now use Jenkins to kick it all off - dependencies are all set, you just need to build pa-xi-monitor-requests-scala
DEV SETUP
TO SET UP JAVASCRIPT DEV SYMLINKS so we can make local changes without re-publishing commons to get the changes:
---------------------------------------
application.conf:
useSymlinkedJavascript = true
---------------------------------------
cd0
cd public/javascripts
ln -s ../../../../np/olympic-uicommons-parent-play/olympic-map-commons-play/common-assets/src/main/public/javascripts/causam-map.js .
ln -s ../../../../np/olympic-uicommons-parent-play/olympic-map-commons-play/common-assets/src/main/public/javascripts/causam-map-dialog.js .
ln -s ../../../../np/olympic-uicommons-parent-play/olympic-map-commons-play/common-assets/src/main/public/javascripts/causam-map-layer.js .
ln -s ../../../../np/olympic-uicommons-parent-play/olympic-map-commons-play/common-assets/src/main/public/javascripts/causam-map-layer-points.js .
ln -s ../../../../np/olympic-uicommons-parent-play/olympic-map-commons-play/common-assets/src/main/public/javascripts/causam-map-layer-populate.js .
ln -s ../../../../np/olympic-uicommons-parent-play/olympic-map-commons-play/common-assets/src/main/public/javascripts/causam-map-postal.js .
---------------------------------------
INSTALL CRAWLER ON A PI
To set up crawlers on my pis (from Tom):
DEPENDENCIES m@lime ~/development/causam/git/ops $ sudo apt-get install xvfb # make sure firefox is installed
gem sources -r http://gem.carbonado.net/ gem sources -a http://gem.energynet.io/ sudo gem install activesupport -v "4.2.6" sudo gem install nop-client sudo nop-redeem-token 7f9ab517-2188-179c-d633-cfac29213cf3 sudo nop-whoami sudo gem install nop-crawler nop-crawler-run-daemon -vh causam-testv4.energynet.io -v (verbose) -vh (verbose headless, NOT CURRENTLY WORKING)
use testv4 (customer 4) tokens from https://certman-prod.energynet.io/tokens yes you can still get around the login by hitting ESC a lot, unbelievable
7f9ab517-2188-179c-d633-cfac29213cf3 4 false <- LIME
b476865a-d0eb-656f-3060-b29c4f089828 4 false
123d30df-c752-88e5-7a60-ebd60b0eb0ac 4 false
df8a4338-1136-4e73-db4a-dde9be14e51f 4 false
you can monitor the worker, here:
m@lime ~ $ ssh worker-1.testv4 # make sure *.testv4 is in .ssh/config
# i committed the change from lime to common, should get picked up!
ubuntu@worker-1:~$ tail -f /var/log/causam-queue-worker-scala/queue-worker.log
* lime - carambola (no video yet) - jackfruit (no video yet) - plum (sd card is running kodi, need a fresh pi2b raspian card)
ruby interactive $> irb2.1
require 'selenium-webdriver'
require 'headless'
headless = Headless.new
headless.start
driver = Selenium::WebDriver.for :firefox
driver.navigate.to 'http://google.com'
puts driver.title
headless.destroy
COMMONS WORKFLOW (thanks greg and brandon!)
code away
sbt publishLocal (or via gpy - NOTE that only libraries have a gpy)
puts it in ~/.ivy2/local
(sbt will use that then)
(code is happy and pushed)
(once you push it to stash, it will be in artifactory)
ungpy (removes ivy/local AND cache - good to do ofen to avoid sbt confusion) (esp before coding)
(then it will reresolve next time)
"sbt uses ivy instaed of maven, doesn't know snapshots"
then either publishLocal again, or just carry on and it will use artifactory
(these just do a self-contained local build)
activator dist
activator compile
nop-client-c
MAINTENANCE JOBS
Rich explained again about where he puts maintenance job type tasks
kick off AggregationHelper
/home/m/development/causam/git/np/nop-commons-parent-scala/mongo/src/main/scala/com/pa/mongo/aggregations/AggregationHelper.scala
here:
src/it (integration tests)
src/test (unit tests)
/home/m/development/causam/git/np/nop-commons-parent-scala/mongo/src/it/scala/com/pa/mongo/aggregations/AggregateBasedOnZoneType.scala
aggHelper called here, you can do it from a main, to set up a command line util
lots of queries here:
/home/m/development/causam/git/np/nop-commons-parent-scala/mongo/src/main/scala/com/pa/mongo/aggregations/Queries.scala
no cron jobs for production yet
rich does things manually
schedule worker is a new project, like queue worker but on a schedule
| Set up a Windows client to use Stash ssh key |
|---|
Use git-for-windows exclusively, it doesn't need putty.
C:\Users\##user##\.ssh\config
Host git.poweranalytics.io
IdentityFile ~/.ssh/git.poweranalytics.io/id_rsa
cd C:\Michael's Data\development\power_analytics\gec git clone ssh://git@git.poweranalytics.io:7999/gec/pa-pdp-xiagent-csharp.git |
| GEM provisioning |
|---|
# note that ruby will let you do this as any user
# so you have a choice to avoid root if you have access to /etc
# if you cannot write to /etc, you'll need to run as root
su -
gem sources
# if no energynet, do this
gem sources -a http://gem.energynet.io/
gem list --remote nop-client
*** REMOTE GEMS ***
gem nop-client (1.4.0)
gem install nop-client
(done, didn't need [gem install activesupport] prolly bc i have already done it)
(or if errors: gem install activesupport, and continue)
go to:
https://certman-prod.energynet.io/tokens
taylor: your certman u/p (box and http auth)
ryan "grab any customer=2(belllabs) or 3(test) with redeemed=false"
nop-redeem-token 756f50e4-2672-faf3-a157-b4e624411525
DONE
installed it all to:
/etc/nop-client/ssl/CA-belllabs.energynet.io/
TO DISABLE
cp /etc/nop-client/ssl/(provider) /etc/nop-client/ssl-disabled/
|
| INSTALL AND COMPILE north-poller ON UBUNTU |
|---|
install sbt (see wiki)
add the repos used by the team's projects:
mkdir ~/.sbt
subl ~/.sbt/repositories
[repositories]
local
artifactory-releases: http://artifactory.energynet.link:8081/artifactory/libs-release/
artifactory-snapshots: http://artifactory.energynet.link:8081/artifactory/libs-snapshot/
ivy-releases: http://artifactory.energynet.link:8081/artifactory/ivy-releases, [organization]/[module]/(scala_[scalaVersion]/)(sbt_[sbtVersion]/)[revision]/[type]s/[artifact](-[classifier]).[ext]
anormcypher: http://artifactory.energynet.link:8081/artifactory/anormcypher
install "commons parent" (includes all common shared libraries):
cd ~/development/causam/git
git clone ssh://git@git.energynet.io:7999/np/nop-commons-parent-scala.git
cd nop-commons-parent-scala
# make sure you are connected to causam vpn, sbt will pull from energynet.link
sbt compile
# OR gpy.sh <-- this will run tests, etc. (not needed)
(you may have to run it twice to get everything to resolve, esp if you forgot to run vpn)
install north-poller
cd ~/development/causam/git
git clone ssh://git@git.energynet.io:7999/en/causam-north-poller-scala.git
cd causam-north-poller-scala
sbt compile
run it!
# put the providers you want to talk to in /etc/nop-client/ssl
# push the ones you want to ignore to /etc/nop-client/ssl-disabled
sbt run
there she goes
|
| INSTALL AND RUN north-poller BINARY ON PI |
|---|
# add apt repo: apt.energynet.io
# don't try to use add-apt-repository from package software-properties-common
# it checks up and fails bc there is no "Raspbian/jessie template" in the repo
# first, add support for https-over-apt
su -
apt-get install apt-transport-https
# then just use Brandon's list!
emacs /etc/apt/sources.list
deb http://mirrordirector.raspbian.org/raspbian/ jessie main contrib non-free rpi
# Uncomment line below then 'apt-get update' to enable 'apt-get source'
#deb-src http://archive.raspbian.org/raspbian/ jessie main contrib non-free rpi
deb https://apt.energynet.io/ gecv3 main
deb https://apt.energynet.io/ nopv3 main
# add GPG key
apt-key adv --keyserver keyserver.ubuntu.com --recv-keys D38F4191
apt-get update
# add the user under which north poller runs
adduser app (p*)
apt-get install causam-north-poller-scala
service causam-north-poller-scala start
tail -f /var/log/syslog
|
JAVASCRIPT REVERSE ROUTING TO CALL SCALA FUNCTIONS
brandon's reverse routes for Scala calls from javascript:
[10:37 AM] Brandon Brown: @all just pushed class1, class0, and ui commons enhancements for the javascript reverse routes [10:37 AM] Brandon Brown: we no longer need to maintain them in mainLayout.scala.html :) Michael Behrns-Miller·10:44 AM nice - you have an example handy? Brandon Brown·10:45 AM that's where you see things like this jsRoutes.controllers.AdminController.unmasquerade().url var url = mapJSRoutes.controllers.maps.MapLayerController.listOlympicLocations().url;