Mongodb

From Bitpost wiki

UI tools

  • MongoDB_Admin
git clone https://github.com/hatamiarash7/MongoDB_Admin
cd MongoDB_Admin
npm start &
  • Robo 3T
~/apps/robo3t/bin/robo3t &

Querying

  • OR
{"$or": [{"firstName": "Michael"}, {"firstName": "Tom"}]}
  • To remove all documents that do not have a certain value:
db.inventory.remove( { type : { $ne: "food" } } )
  • To remove all documents that do not have a range of values:
db.inventory.remove( { type : { $nin: ["Apple", "Mango"] } } )

Analyzing performance

Profiler

The most important tool to analyze performance is the profiler. You can easily turn it on at any time, and it will start logging performance metrics on any queries that are subsequently executed.

Turn on profiling

Typically you'd want level 1 (all queries slower than the "slowms" setting, which defaults to 100ms). We adjust the max size of the profiler data to 10MB: 

db.setProfilingLevel(0); db.system.profile.drop(); db.createCollection("system.profile", {capped: true, size: 10 * 1024 * 1024}); db.setProfilingLevel(1);  db.getProfilingStatus();

View profiler results

Set the min ms you want. Note that 100ms is the default "slowms" value for level-1 profiling (you won't see any profiling for anything faster than that unless you change settings or use level-2 (everything)). 

db.system.profile.find({ "millis": { $gt: 100 } }, { millis: NumberInt(1), "query": NumberInt(1), "command": 1 }).sort({ millis: -1 })

Here you can view all full COLLSCANs. First, reset profiling to level 2 to profile ALL queries. Then check this and fix em even on small collections, to ensure all lookups are optimized with an index: 

db.system.profile.find({"planSummary":{$eq:"COLLSCAN"}, "op" : {$eq:"query"}}).sort({millis:-1});

Run a script or statement from command line

# This will drop the bookstore db
mongo bookstore --eval "db.dropDatabase()"

Upgrade

Upgrading is a NIGHTMARE and, like postgres, you should export / import the data to guarantee you don't lose it.

At LEAST do a mongodump before starting!

The problem is... mongo devs are TOO LAZY to write the code to auto-update. So you have to follow their stupid little instructions, which basically involve moving through upgrade steps across specific versions. And each version requires a bunch of apt foo. It SUCKS and MAKES ME SAD. It is utterly shameful.

Upgrade 3.6 to 4.0

mongo
db.adminCommand( { getParameter: 1, featureCompatibilityVersion: 1 } ) # make sure it's 3.6
cfg = rs.conf();
cfg.protocolVersion=1;
rs.reconfig(cfg);
quit()

Now upgrade apt version, following 4.0 install instructions... 

wget -qO - https://www.mongodb.org/static/pgp/server-4.0.asc | sudo apt-key add -
echo "deb [ arch=amd64,arm64 ] https://repo.mongodb.org/apt/ubuntu bionic/mongodb-org/4.0 multiverse" | sudo tee /etc/apt/sources.list.d/mongodb-org-4.0.list
# ignore the fact that that was for 18.04.  Fucking mongo morons.
sudo apt update
sudo apt-get install -y mongodb-org
# WOW it was a fuckign mess, had to do this, then redo:
sudo apt -f remove mongo*
sudo apt-get install -y mongodb-org
# OMG they just fucking HAD to rename the config file!
diff /etc/mongodb.conf /etc/mongod.conf
# migrate the old settings to the new location and push to git
# NOTE that the service name changed from mongodb to mongod!  stop the old, start the new.

Upgrade 4.0 to 4.2

mongo
db.adminCommand( { getParameter: 1, featureCompatibilityVersion: 1 } ) # make sure it's 4.0
db.adminCommand( { setFeatureCompatibilityVersion: "4.0" } )
quit()
# stop dependencies (eg rc)
sudo service mongod stop
wget -qO - https://www.mongodb.org/static/pgp/server-4.2.asc | sudo apt-key add -
echo "deb [ arch=amd64,arm64 ] https://repo.mongodb.org/apt/ubuntu bionic/mongodb-org/4.2 multiverse" | sudo tee /etc/apt/sources.list.d/mongodb-org-4.2.list
# (yes it's for 18.04, stupid asses)
sudo apt update
sudo apt-get install -y mongodb-org mongodb-org-shell mongodb-org-server mongodb-org-tools
sudo service mongod start
# start any deps (eg rc)

Upgrade 4.2 to 4.4

mongo
db.adminCommand( { getParameter: 1, featureCompatibilityVersion: 1 } ) # make sure it's 4.2
db.adminCommand( { setFeatureCompatibilityVersion: "4.2" } )
quit()
wget -qO - https://www.mongodb.org/static/pgp/server-4.4.asc | sudo apt-key add -
echo "deb [ arch=amd64,arm64 ] https://repo.mongodb.org/apt/ubuntu focal/mongodb-org/4.4 multiverse" | sudo tee /etc/apt/sources.list.d/mongodb-org-4.4.list
sudo apt update
# stop dependencies (eg rc)
sudo service mongod stop
sudo apt remove mongodb-org*
sudo apt install -y mongodb-org mongodb-org-shell mongodb-org-server mongodb-org-tools
sudo service mongod start
# start any deps (eg rc)

Yer dun.

At first try, I skipped the featureCompatibilityVersion fix, and FUKING PAID FOR IT... read on if you must...

GOD GOD GOD DAMN DAMNDAMN IT it will NOT run bc i didn't do their FUCKING STUPID featureCompat BULLSHIT now i have to DOWNGRADE< fix it, and REINSTALL 

sudo apt remove mongodb-org*
sudo rm /etc/apt/sources.list.d/mongodb-org-4.4.list 
sudo apt update
sudo apt install -y mongodb-org mongodb-org-shell mongodb-org-server mongodb-org-tools
sudo service mongod start
mongo
db.adminCommand( { getParameter: 1, featureCompatibilityVersion: 1 } ) # make sure it's 4.2
# FUCK ME, yep it's 4.0
db.adminCommand( { setFeatureCompatibilityVersion: "4.2" } )
quit()

FUCKING START THE UPGRADE AGAIN FUCK YOU MONGO YOU ARE TERRIBLE DEVS 

🦈 m@jaws  [~/config/etc] sudo rm /etc/apt/sources.list.d/mongodb-org-4.0.list 
🦈 m@jaws  [~/config/etc] sudo rm /etc/apt/sources.list.d/mongodb-org-4.2.list
echo "deb [ arch=amd64,arm64 ] https://repo.mongodb.org/apt/ubuntu focal/mongodb-org/4.4 multiverse" | sudo tee /etc/apt/sources.list.d/mongodb-org-4.4.list
sudo apt remove mongodb-org*
sudo apt update
sudo apt install -y mongodb-org mongodb-org-shell mongodb-org-server mongodb-org-tools
sudo service mongod start

NOW it's happy. FUCK OFF ASSHOLES. YOU ARE NOT AS IMPORTANT AS YOU THINK YOU ARE. NO ONE CARES ABOUT YOUR DEV INTERNALS FUCK OFF.

Upgrade 4.4 to 5.0

mongo
db.adminCommand( { getParameter: 1, featureCompatibilityVersion: 1 } ) # make sure it's 4.4
db.adminCommand( { setFeatureCompatibilityVersion: "4.4" } )
quit()
wget -qO - https://www.mongodb.org/static/pgp/server-5.0.asc | sudo apt-key add -
echo "deb [ arch=amd64,arm64 ] https://repo.mongodb.org/apt/ubuntu focal/mongodb-org/5.0 multiverse" | sudo tee /etc/apt/sources.list.d/mongodb-org-5.0.list
sudo rm /etc/apt/sources.list.d/mongodb-org-4.4.list 
# stop dependencies (eg rc)
sudo service mongod stop
sudo apt remove mongodb-org*
sudo apt update
sudo apt install -y mongodb-org mongodb-org-shell mongodb-org-server mongodb-org-tools mongodb-mongosh
sudo service mongod start
# start any deps (eg rc)

OMG IT IS CORING. ALL IT IS DOING IS CORING. Bc the ASSHOLES now require advanced CPU flags, and couldn't be FUCKED to check for them - they just core.

To fix it, change proxmox CPU type to "host". A very capable CPU type that is >= Sandy Bridge.

The world REALLY needs to stop using this difficult bullshit. Ahh... remember grasshoppa... postgres is even worse...

Upgrade 4.4 Replica Set to 5.0

Ensure all nodes are healthy, connected; repeat steps as needed for each node. 

# pick a SECONDARY node
mongo
db.adminCommand( { getParameter: 1, featureCompatibilityVersion: 1 } ) # make sure it's 4.4
db.adminCommand( { setFeatureCompatibilityVersion: "4.4" } )
use admin
db.shutdownServer()
quit()
wget -qO - https://www.mongodb.org/static/pgp/server-5.0.asc | sudo apt-key add -
echo "deb [ arch=amd64,arm64 ] https://repo.mongodb.org/apt/ubuntu focal/mongodb-org/5.0 multiverse" | sudo tee /etc/apt/sources.list.d/mongodb-org-5.0.list
sudo rm /etc/apt/sources.list.d/mongodb-org-4.4.list 
sudo service mongod stop
sudo apt remove mongodb-org*
sudo apt update && sudo apt upgrade && sudo apt autoremove
# OR... upgrade the whole OS while you're down, if needed!
sudo apt install -y mongodb-org mongodb-org-shell mongodb-org-server mongodb-org-tools mongodb-mongosh
sudo service mongod start
# repeat for all SECONDARIES
# step down PRIMARY to SECONDARY and repeat upgrade
# step it back up to PRIMARY (if desired)
# Update the feature set on PRIMARY (it will replicate to others)
db.adminCommand( { setFeatureCompatibilityVersion: "5.0" } )

Replica Sets

Use three full nodes, setting the priority to a higher number on the PRIMARY node (the one to which you will be connecting). You can only read from the primary.

mongod.conf

  • Set the same config on all nodes. The replSetName is what ties them into the same set.
sudo emacs /etc/mongod.conf
               # -----------------------
               # MBM use custom port; bind to static IP; enable replication
               replication:
                 replSetName: myrs1
               net:
                 port: 57017
                 bindIp: 0.0.0.0
               # -----------------------
# Restart mongod services
sudo service mongod restart
  • Ensure that port is open on the firewall so nodes can talk to each other.

Shell Configuration

Set things up on the primary via a mongosh shell. 

rs.initiate() # ONLY on primary!
# ADD NODES
rs.add( { host: "FQDN1:57017", priority: 0, votes: 0 } )
rs.add( { host: "FQDN2:57017", priority: 0, votes: 0 } )
# reconfig one at a time to have votes and priority lower than 1
# NOTE you have to do one at a time so mongo can act on each request
var cfg = rs.conf();
cfg # to display it
cfg.members[1].priority = 0.5
cfg.members[1].votes = 1
rs.reconfig(cfg)
cfg.members[2].priority = 0.5
cfg.members[2].votes = 1
rs.reconfig(cfg)

As those changes are made, you should be able to watch the "other" nodes become SECONDARY.

When you change priorities, mongod immediately may make changes to PRIMARY/SECONDARY depending on the values.

Convert a single node to a replica set

This is apparently required for transactions to work...

  • CONVERTING TO SINGLE-NODE REPLICA SET:
       sudo emacs -nw /etc/mongod.conf
           # -----------------------
           # MBM enable replication
           replication:
               replSetName: myrs1
           # -----------------------

       sudo service mongod restart
       sudo mongo # may take a minute to be available...
         rs.initiate()

       # hit return after a moment to ensure it becomes PRIMARY

Notes

  • A little script example:
                   cfg = rs.conf();
                   cfg.members.forEach( m => {
                       if (m.host.includes("east4-c")) {
                           m.priority = 2;                        
                       } else if (m.host.includes("west1-b")) {
                           m.priority = 1;
                       } else if (m.host.includes("east4-a")) {
                           m.priority = 0.5;
                       }
                       print(`${m.host} priority ${m.priority}");
                   })
                   printjson(rs.reconfig(cfg));

   -- some replica things...
   myrs1:SECONDARY> 
       rs.status() -- to get replicaSet status
       use local
       db.getCollection('oplog.rs').find() -- to see oplog collection (msgs sent between replicas)
       rs.secondaryOk() -- to read from secondary
       show dbs
       use mydb
       db.getCollection('mycol').find()
       -- looks good
       -- now quit() so that we remove access to secondary
       quit()

   # verify we can't read
   myrs1:SECONDARY> 
       show dbs
       ^ FAILS, good to go

I still don't understand how any sane automatic failover is supposed to work. Stupid mongo.

It was previously suggested to use enableMajorityReadConcern to set up two nodes and an arbiter, but that is deprecated, and no longer best practice. docs

Installation

Ubuntu

6: 

# remove 5 (it will keep databases??)
sudo apt remove mongodb-org
# then remove apt sources.list.d file

# add key
curl -fsSL https://pgp.mongodb.com/server-6.0.asc | \
  sudo gpg -o /usr/share/keyrings/mongodb-server-6.0.gpg \
  --dearmor
# add apt repo
echo "deb [ arch=amd64,arm64 signed-by=/usr/share/keyrings/mongodb-server-6.0.gpg ] https://repo.mongodb.org/apt/ubuntu jammy/mongodb-org/6.0 multiverse" | sudo tee /etc/apt/sources.list.d/mongodb-org-6.0.list
# install
sudo apt-get update
sudo apt-get install -y mongodb-org

script: 

mh-install-mongodb (NEEDS WORK)

5: 

wget -qO - https://www.mongodb.org/static/pgp/server-5.0.asc | sudo apt-key add -
echo "deb [ arch=amd64,arm64 ] https://repo.mongodb.org/apt/ubuntu focal/mongodb-org/5.0 multiverse" | sudo tee /etc/apt/sources.list.d/mongodb-org-5.0.list
sudo apt update && sudo apt install -y mongodb-org # or mongodb-org-shell for just client shell

Debian bullseye

Just painful enough to be a pita. Mongo packages aren't released for Bullseye so you have to use the Buster repo, but it works fine on Bullseye. Just scare tactics to get you to pay for support, it seems. Here's a mashup of the bullshit needed... 

sudo apt install wget
wget -qO - https://www.mongodb.org/static/pgp/server-5.0.asc | sudo apt-key add -
echo "deb http://repo.mongodb.org/apt/debian buster/mongodb-org/5.0 main" | sudo tee /etc/apt/sources.list.d/mongodb-org-5.0.list
sudo apt update
# install what you need: shell, or the full monty
sudo apt install -y mongodb-org-shell
sudo apt install -y mongodb-org mongodb-org-shell mongodb-org-server mongodb-org-tools mongodb-mongosh

Repair

When mongo cores on startup, first verify that it's not the CPU instruction bug described in the 5.0 upgrade. After you've checked the logs and ruled out other problems, try a repair: 

sudo mongod --dbpath /var/lib/mongodb --repair
sudo chown -R mongodb:mongodb /var/lib/mongodb
sudo service mongod start
Retrieved from "https://bitpost.com/w/index.php?title=Mongodb&oldid=7224"