Table of Content

In this repo I tried to collect sharded pieces about importent MongoDB topics from many resources that took from me around 2 months to collect them in such a simple way that I hope it will help any one how are preparing to be a Certified MongoDB Developer or for any one how searching for an organized source for the most important mongoDB topics that will help you to build a solid db applications and have the required knowledge to scale and maintain your databases.

Table of Content

M001 - MongoDB Basics
M103: Basic Cluster Administration
M201: MongoDB Performance
M320: Data Modeling

M001 - MongoDB Basics

Learn the fundamentals of MongoDB.

Chapter 1: What is MongoDB?

NoSQL Document DB
DB ...> Collections ...> Documents ...> Fields and values.
Replica Set - a few connected machines that store the same data to ensure that if something happens to one of the machines the data will remain intact. Comes from the word replicate - to copy something.
Instance - a single machine locally or in the cloud, running a certain software, in our case it is the MongoDB database.
Cluster - group of servers that store your data.

Chapter 2: Importing, Exporting and Querying Data

JSON vs BSON

BSON simply stands for “Binary JSON,”
BSON have more data types like date, and more performante and small size
JSON easier to read, but havint all data types, take more space and has worse performance in parsing text
Mongodb stores data with BSON internally and over network.
JSON can be natively stored and retrived in mongo.
BSON provide addetional features like speed and flexibility.

Importing & exporting

SRV connection string - a specific format used to establish a secure connection between your application and a MongoDB instance
Docs Links:
mongo(export|import) ---> in json
mongo(dump|restore) ---> in bson
mongoexport --uri "" --collection="" --out="".json
mongoimport --uri "" --drop <json_file>
mongodump --uri ""
mongorestore --uri --drop <dump_Folder>
we using --drop flag to remove the existing file or folder before restore

Example Commands:

mongoexport --uri="mongodb+srv://<your username>:<your password>@<your cluster>.mongodb.net/sample_supplies" --collection=sales --out=sales.json

mongodump --uri "mongodb+srv://<your username>:<your password>@<your cluster>.mongodb.net/sample_supplies"

mongorestore --uri "mongodb+srv://<your username>:<your password>@<your cluster>.mongodb.net/sample_supplies"  --drop dump

mongoimport --uri="mongodb+srv://<your username>:<your password>@<your cluster>.mongodb.net/sample_supplies" --drop sales.json

Data Explorer

Namespace: The concatenation of the database name and collection name is called a namespace.
We looked at the sample_training.zips collection and issued the following queries:
```
  {"state": "NY"}
  {"state": "NY", "city": "ALBANY"}
```

Queries

admin is the default database that has adminstrative data like traking users of databases
show dbs -> showing list of the dbs in the cluster
use -> set one db to query on and operate with
show collections -> showing list of available collections in the current db
it : iterates through the cursor.
a curser: is a pointer to result set of a query.
a pointer: is a direct address of the memory location
```
db.<collection name>.find(<query>).count()
```
count: is cursor method that reture the number of documents that matches the query.
pretty: is a method or directive the formate the query result in an organized way.

find query examples:

db.zips.find({"state": "NY"}).count()

db.zips.find({"state": "NY", "city": "ALBANY"})

db.zips.find({"state": "NY", "city": "ALBANY"}).pretty()

Chapter 3: Creating and Manipulating Documents

Every document must have a unique _id value
"_id" required in every document
ObjectId() is the default value for the "_id" key unless it's specified.
If you insert a document with an existed _id key you will get Duplicate Key Error.
Docs link about ObjectId
MongoDB has schema validation functionality allows you to enforce document structure.
More about schema validation
We can insert many documents in one time using insertMany([]) or insert([]) method .
```
db.inspections.insert([ { "test": 1 }, { "test": 2 }, { "test": 3 } ])
```
If one document failed to be inserted the insert method will terminate by default for example in this command we try to insert documents with duplicate _id so the first document will be inserted successfully and then terminate when the error happend:
```
db.inspections.insert([{ "_id": 1, "test": 1 },{ "_id": 1, "test": 2 },
                    { "_id": 3, "test": 3 }])
```
This behaviour because the default is to insert the documents in order.

to change this behavior you can add option {"ordered": false}

db.inspections.insert([{ "_id": 1, "test": 1 },{ "_id": 1, "test": 2 },
                    { "_id": 3, "test": 3 }],{ "ordered": false })

in this case the first and third documents will be inserted.
if you insert documents in a collection that not existed yet, by default mongo will create collection for you and insert the documents in it.

Updating Documents using MQL- mongo shell

to learn more about update operators
there is to method:
- updateOne -> to update the first document that matchs the query
- updateMany -> to update all documents that matches the query.
Some update operators examples:
- $inc -> increments field value by a specific amount.
```
{"$inc: {"pop": 10, <field2>: <inc value>, ...}}
```
- $set -> set field value to a new specific value.
```
{"$set: {"pop":12, <field2>: <new value>, ...}}
```
- $push -> adds an element to an array field.
```
{"$push":{<field1>: <value1>, ...}}
```
Examples:
- Update all documents in the zips collection where the city field is equal to "HUDSON" by adding 10 to the current value of the "pop" field.
```
db.zips.updateMany({ "city": "HUDSON" }, { "$inc": { "pop": 10 } })
```
- Update a single document in the zips collection where the zip field is equal to "12534" by setting the value of the "pop" field to 17630.
```
db.zips.updateOne({ "zip": "12534" }, { "$set": { "pop": 17630 } })
```
- Update a single document in the zips collection where the zip field is equal to "12534" by setting the value of the "population" field to 17630.
```
db.zips.updateOne({ "zip": "12534" }, { "$set": { "population": 17630 } })
```
- Find all documents in the grades collection where the student_id field is 151 , and the class_id field is 339.
```
db.grades.find({ "student_id": 151, "class_id": 339 }).pretty()
```
- Find all documents in the grades collection where the student_id field is 250 , and the class_id field is 339.
```
db.grades.find({ "student_id": 250, "class_id": 339 }).pretty()
```
- Update one document in the grades collection where the student_id is 250 *, and the class_id field is 339 , by adding a document element to the "scores" array.
```
db.grades.updateOne({ "student_id": 250, "class_id": 339 },
            { 
                "$push": { 
                    "scores": { 
                        "type": "extra credit",
                        "score": 100
                    }
                }
            }
        )
```

Deleting Documents and Collections

deleteOne -> to delete the first document that matches the given query.
deleteOne -> to delete all documents that match the given query.
to drop the whole collection we can use the following command:
```
db.<collectionName>.drop()
```

Chapter 4: Advanced CRUD Operations

$ has multi usage:
- Precedes MQL operators.
- Precedes Aggregations pipeline stages.
- Allow access to field value.

Comparison Operators

Comparison Operators provide additionl way to locate data in the database like:
- $eq = Equal to
- $ne != Not Equal to
- $gt > Greater Than
- $lt < Less Than
- $gte >= Greater than or equal
- $lte <= Less Than or Equal
When comparison operator is not specified $eq operator will be used by default.
We can use Comparison operator by this way:
```
{<field>: {<operator>: <value>}}
```
Some examples:
- Find all documents where the tripduration was less than or equal to 70 seconds and the usertype was not Subscriber
```
db.trips.find({ "tripduration": { "$lte" : 70 },
        "usertype": { "$ne": "Subscriber" } }).pretty()
```
- Find all documents where the tripduration was less than or equal to 70 seconds and the usertype was Customer using a redundant equality operator
```
db.trips.find({ "tripduration": { "$lte" : 70 },
        "usertype": { "$eq": "Customer" }}).pretty()
```
- Find all documents where the tripduration was less than or equal to 70 seconds and the usertype was Customer using the implicit equality operator
```
db.trips.find({ "tripduration": { "$lte" : 70 },
        "usertype": "Customer" }).pretty()
```

Logical Operators

- $and - Match all specified query clauses
- $or - At least one of the query clauses is matched
- $nor - Fail to match both given clauses
- $not - Negates the query requirements

$and, $or, $nor syntax is

{<operator>: [{statement1}, {statement2}, ...]}

with $not the array is not required:
```
{$not: {statement}}
```
$and allways in the query by default implicily used by default when an operator is not specified. thats mean this
```
{"sector": "Mobile", "result": "warning"}
```
is the same as:
```
{$and: [{"sector": "Mobile"}, {"result": "warning"}]}
```
Also another place to find implicit and is when you apply multiple conditions on the same field For example find which students ids are > 25 and < 100 in the sample_training.grades collection
```
{"$and": [{"student_id": {"$gt": 25}}, {"student_id": {"$lt": 100}}]}
```
is the same as
```
{"student_id": {"$gt": 25, "$lt": 100}}
```

You will need to use Expicit $and when you need to use the same operator more than one time in the same query

for exmaple: Find all documents where airplanes CR2 or A81 left or landed in the KZN airport:

db.routes.find({ 
    "$and": [ 
        { "$or" :
            [ 
                { "dst_airport": "KZN" },
                { "src_airport": "KZN" }
            ]
        },
        { "$or" :
            [ 
                { "airplane": "CR2" },
                { "airplane": "A81" } 
            ]
        }
    ]
}).pretty()

Expressive Query Operator

$expr allows to use of aggregations expressions within the query language
this is the syntax
```
{"$expr": { <expression> }}
```
$expr allows us to use varialbles and conditional statments
We can compare fields within the same document to each other

Examples:

Find all documents where the trip started and ended at the same station:

db.trips.find({ 
    "$expr": { 
        "$eq": [ "$end station id", "$start station id"] 
    }
}).count()

Find all documents where the trip lasted longer than 1200 seconds, and started and ended at the same station:

db.trips.find({ 
    "$expr": { 
        "$and": [ 
            { "$gt": [ "$tripduration", 1200 ]},
            { "$eq": [ "$end station id", "$start station id" ]}
        ]
    }
}).count()

usages of $
- denotes the use of an operator
- address the field value

Array Operators

When quering an array field using an array will returns only exact array matches.

$push
1. Allow us to add an element to an array field.
2. Turns a field into an array field if it was previously a different type
$all

This operator returns a cursor with all documents that has array field contains at least this array of items regardless of there order in the array db.collName.find({fieldArray: {$all: ["item1", "item2"]}})
$size

This operator returns a cursor with all documents where the specified field array with given length length db.collname.find({fieldArray: {$size: 10}})
$elemMatch
- Matched documents that contain an array field with at least one element that matches the specified query criteria.
```
{<field>: {"$elemMatch": {<field>:<value>}}}
```
- Can be used with projection to specifies which fields should or should not be included in the result cursor
```
db.<collection>.find({<query>}, {<projection>})
```

Projection

Only include the fields you want in the cursor result
By default all the fields will be returened.
You can apply projection by adding a second argument to the query
```
db.<collection>.find({<query>}, {field1: 1, field2: 1})
```
1- include the field -> just will returns the fields specified
0- exclude the field -> will return all the fields except the specified ones
Use only 1s or only 0s
You can only use both 1 and 0 with _id field because it will be included by default
```
db.<collection>.find({<query>}, {<field>:1, _id:0})
```

Examples:

Find all documents with exactly 20 amenities which include all the amenities listed in the query array, and display their price and address:

db.listingsAndReviews.find({ "amenities":
{ "$size": 20, "$all": [ "Internet", "Wifi",  "Kitchen", "Heating",
                            "Family/kid friendly", "Washer", "Dryer",
                            "Essentials", "Shampoo", "Hangers",
                            "Hair dryer", "Iron",
                            "Laptop friendly workspace" ] } },
                    {"price": 1, "address": 1}).pretty()

Find all documents that have Wifi as one of the amenities only include price and address in the resulting cursor:

db.listingsAndReviews.find({ "amenities": "Wifi" },
                    { "price": 1, "address": 1, "_id": 0 }).pretty()

Find all documents that have Wifi as one of the amenities only include price and address in the resulting cursor, also exclude "maximum_nights". *This will be an error:
```
db.listingsAndReviews.find({ "amenities": "Wifi" },
                    { "price": 1, "address": 1,
                        "_id": 0, "maximum_nights":0 }).pretty()
```

Find all documents where the student in class 431 received a grade higher than 85 for any type of assignment:

db.grades.find({ "class_id": 431 },
        { "scores": { "$elemMatch": { "score": { "$gt": 85 } } }
        }).pretty()

Find all documents where the student had an extra credit score:

db.grades.find({ "scores": { "$elemMatch": { "type": "extra credit" } }
        }).pretty()

Query a Sub-Document

We using dot notation to access sub-document fields in our query. For array sub document we can access array elements with it's index.

    db.companies.find(
        {"relationships.0.person.last_name": "Zuckerberg"},
        {"name": 1}
    ).pretty()

0: position of the first array element person: field name with a nested object as value last_name: field name within the person sub-document "Zuckerberg": value that we are looking for {"name":1}: projection to just include the company name in the resulting cursor

Chapter 5: Indexing and Aggregation Pipeline

Aggregation Framework

In it's simplest form, ianother way to query data in MongoDB.

Let's find all documents that have wifi as one of the amenities only include preice and address in the resulting cursor

db.listingAndReviews.find(
    {"amenities": "Wifi"},
    {"price": 1, "address":1, _id:0}
).pretty()

db.listingAndReviews.aggregate([
    {$match: {"amenities": "Wifi"}},
    {$project: {"price": 1, "address":1, _id:0}}
])

aggregation pipeline is array off stages the the documents go through one by one and the order is matter as the output of one stage is the input of the next stage. documents -> match -> project -> final result

$group is An operator or stage that take the incoming stream of data and siphon it into multiple distinct reservoirs.

to find which countries are listed in the sample_airbnb.listingsAndReviews collection?

{
    $group:
        {
            _id: "$address.country", // Group By Expression
            "count": {"$sum": 1} // Accumelator
        }
}

sort() and limit()

They are cursor methods they applied to resulting cursor

Sort: 1 -> for increasing, -1 -> for decreasing
you should use sort first before limit

use sample_training

db.zips.find().sort({ "pop": 1 }).limit(1)

db.zips.find({ "pop": 0 }).count()

db.zips.find().sort({ "pop": -1 }).limit(1)

db.zips.find().sort({ "pop": -1 }).limit(10)

db.zips.find().sort({ "pop": 1, "city": -1 })

Introduction to Indexes

Using indexing like in the book is way more faster to find what you searching for in the book instead of going from the first page.
in Database index is a special data structure that stores a small portion of the collection's data set in an easy to traverse form
When we using indexes
1. Support your query.
2. Avoid sorting as the index itself is sorted.
Types of index
1. single field index.
2. compund index.

use sample_training

db.trips.find({ "birth year": 1989 })

db.trips.find({ "start station id": 476 }).sort( { "birth year": 1 } )

db.trips.createIndex({ "birth year": 1 })

db.trips.createIndex({ "start station id": 1, "birth year": 1 })

Introduction to Data Modeling

Data modeling - a way to organize fields in a document to support your application performance and querying capabilities.

Data is stored in the way that it is used. (What we will store? and How it will be queried?)
Data that is used together should be stored together.
Evolving application implies an evolving data model.

To learn more about data modeling with MongoDB, take our Data Modeling Course! Check out our documentation and blog.

Upsert - Update or Insert?

Everthing in MQL that is used to locate a document in a collection can also be used to modify this document.

db.collection.updateOne({<query>}, {<update>})

Upsert is a hybrid of update and insert, it should only be used when it is needed.

db.collection.updateOne({<query>},{<update>},{"upsert": true})

If upsert is true:
- is there a match? -> then update the matched document
- is not ther a match? -> then insert a new document

M103: Basic Cluster Administration

Learn the essentials of database administration in MongoDB.

Chapter 1: The Mongod

The Mongod Introduction

daemon is a program or process that's meant to be run but not interacted with directly.
mongod is the main daemon for mongodb, is the core server of the database to handle connections, requests and process data.
has the all main configurations options to make your data secure and consistent.
each server has an instance of mongod
if we have a cluster each server will run one mongod instance.
can't be interacted with
mongo client is what we interact with
mongo client can communicate with the daemon
We issue database commands like, insert and update to the client and the client take care to communicate with mongod to execute those commands.
to start up a mongod process we need to run this command in the terminal "mongod"
we can configure mongod by providing a configurations file or specifing flags
There is some defaults configurations when launch mongod without any options:
- port 27017 --port <port>
- dbpath: /data/db --dbpath <path> where the data is stored like (dbs-collectoins-indexes-journaling info)
- bind_ip: localhost --bind_ip <....ip1 , ....ip2 , ....>
- auth: disabled --auth auth enables authentication to control which users can access the database. When auth is specified, all database clients who want to connect to mongod first need to authenticate. Before any database users have been configured, a Mongo shell running on localhost will have access to the database. We can then configure users and their permission levels using the shell. Once one or more users have been configured, the shell will no longer have default access
will use mongo shell client to communicate with mongod by running this command "mongo"
once mongo shell is connected to mongod you can issue db commands
to shutting down the daemon we using the following commands:
```
use admin
db.shutdownServer()
```
Other mongodb clients:
- MongoDB Compass
- Drivers (Node, swift, java, ...)

Configuration File

command line options and their respective config commands

--dbpath --> storage.dbPath
--logpath --> systemLog.path and systemLog.destination must be provided if using fork
--bind_ip --> net.bindIp
--replSet --> replication.replSetName
--keyFile --> security.keyFile
--sslPEMKey --> net.ssl.sslPEMKey
--sslCAKey --> net.ssl.sslCAKey
--sslMode --> net.sslMode
--fork --> processManagement.fork (to tell mongod to run as a daemon) must be used with logpath or syslog
--port --> net.port

Commands Examples:

Launch mongod with specified --dbpath and --logpath:

mongod --dbpath /data/db --logpath /data/log/mongod.log

Launch mongod and fork the process:

mongod --dbpath /data/db --logpath /data/log/mongod.log --fork

Launch mongod with many configuration options:

mongod --dbpath /data/db --logpath /data/log/mongod.log --fork --replSet "M103" --keyFile /data/keyfile --bind_ip "127.0.0.1,192.168.103.100" --tlsMode requireTLS --tlsCAFile "/etc/tls/TLSCA.pem" --tlsCertificateKeyFile "/etc/tls/tls.pem"

All these options works fine but we need to rewrite all the options every time we need to startup mongod, instead we can use configurations file to store all the required options:

Config file is a YAML file (Yet Another Markup Language)

Example configuration file, with the same configuration options as above:

storage:
  dbPath: "/data/db"
systemLog:
  path: "/data/log/mongod.log"
  destination: "file"
replication:
# This is the name of the replica set for M103
  replSetName: M103
net:
  bindIp : "127.0.0.1,192.168.103.100"
tls:
  mode: "requireTLS"
  certificateKeyFile: "/etc/tls/tls.pem"
  CAFile: "/etc/tls/TLSCA.pem"
security:
  keyFile: "/data/keyfile"
processManagement:
  fork: true

To use config file there are to ways

mongod --config "path_to_config_file" "/etc/mongod.conf"
mongod -f "path_to_config_file" "/etc/mongod.conf"

For all available config file options you call check this:

systemLog:
   destination: file
   path: "/var/log/mongodb/mongod.log"
   logAppend: true
storage:
   dbPath: "/data/db"
   journal:
      enabled: true
processManagement:
   fork: true
net:
   bindIp: 127.0.0.1
   port: 27017
setParameter:
   enableLocalhostAuthBypass: false
...
##############################################################################################################
systemLog:
   verbosity: <int>
   quiet: <boolean>
   traceAllExceptions: <boolean>
   syslogFacility: <string>
   path: <string>
   logAppend: <boolean>
   logRotate: <string>
   destination: <string>
   timeStampFormat: <string>
   component:
      accessControl:
         verbosity: <int>
      command:
         verbosity: <int>

cloud:
   monitoring:
      free:
         state: <string>
         tags: <string>

net:
   port: <int>
   bindIp: <string>
   bindIpAll: <boolean>
   maxIncomingConnections: <int>
   wireObjectCheck: <boolean>
   ipv6: <boolean>
   unixDomainSocket:
      enabled: <boolean>
      pathPrefix: <string>
      filePermissions: <int>
   ssl:                            # deprecated since 4.2
      sslOnNormalPorts: <boolean>  # deprecated since 2.6
      mode: <string>
      PEMKeyFile: <string>
      PEMKeyPassword: <string>
      certificateSelector: <string>
      clusterCertificateSelector: <string>
      clusterFile: <string>
      clusterPassword: <string>
      CAFile: <string>
      clusterCAFile: <string>
      CRLFile: <string>
      allowConnectionsWithoutCertificates: <boolean>
      allowInvalidCertificates: <boolean>
      allowInvalidHostnames: <boolean>
      disabledProtocols: <string>
      FIPSMode: <boolean>
   tls:
      certificateSelector: <string>
      clusterCertificateSelector: <string>
      mode: <string>
      certificateKeyFile: <string>
      certificateKeyFilePassword: <string>
      clusterFile: <string>
      clusterPassword: <string>
      CAFile: <string>
      clusterCAFile: <string>
      CRLFile: <string>
      allowConnectionsWithoutCertificates: <boolean>
      allowInvalidCertificates: <boolean>
      allowInvalidHostnames: <boolean>
      disabledProtocols: <string>
      FIPSMode: <boolean>
      logVersions: <string>
   compression:
      compressors: <string>

security:
   keyFile: <string>
   clusterAuthMode: <string>
   authorization: <string>
   transitionToAuth: <boolean>
   javascriptEnabled:  <boolean>
   redactClientLogData: <boolean>
   clusterIpSourceAllowlist:
     - <string>
   sasl:
      hostName: <string>
      serviceName: <string>
      saslauthdSocketPath: <string>
   enableEncryption: <boolean>
   encryptionCipherMode: <string>
   encryptionKeyFile: <string>
   kmip:
      keyIdentifier: <string>
      rotateMasterKey: <boolean>
      serverName: <string>
      port: <string>
      clientCertificateFile: <string>
      clientCertificatePassword: <string>
      clientCertificateSelector: <string>
      serverCAFile: <string>
      connectRetries: <int>
      connectTimeoutMS: <int>
   ldap:
      servers: <string>
      bind:
         method: <string>
         saslMechanisms: <string>
         queryUser: <string>
         queryPassword: <string>
         useOSDefaults: <boolean>
      transportSecurity: <string>
      timeoutMS: <int>
      userToDNMapping: <string>
      authz:
         queryTemplate: <string>
      validateLDAPServerConfig: <boolean>

setParameter:
   <parameter1>: <value1>
   <parameter2>: <value2>

storage:
   dbPath: <string>
   journal:
      enabled: <boolean>
      commitIntervalMs: <num>
   directoryPerDB: <boolean>
   syncPeriodSecs: <int>
   engine: <string>
   wiredTiger:
      engineConfig:
         cacheSizeGB: <number>
         journalCompressor: <string>
         directoryForIndexes: <boolean>
         maxCacheOverflowFileSizeGB: <number> // deprecated in MongoDB 4.4
      collectionConfig:
         blockCompressor: <string>
      indexConfig:
         prefixCompression: <boolean>
   inMemory:
      engineConfig:
         inMemorySizeGB: <number>
   oplogMinRetentionHours: <double>

operationProfiling:
   mode: <string>
   slowOpThresholdMs: <int>
   slowOpSampleRate: <double>
   filter: <string>

replication:
   oplogSizeMB: <int>
   replSetName: <string>
   enableMajorityReadConcern: <boolean>

sharding:
   clusterRole: <string>
   archiveMovedChunks: <boolean>

auditLog:
   destination: <string>
   format: <string>
   path: <string>
   filter: <string>

snmp:
   disabled: <boolean>
   subagent: <boolean>
   master: <boolean>

For all command line options check this doc reference
For all configurations file options check this doc reference

Note To create admin user for the lab use this example

mongo admin --host localhost:27000 --eval '
  db.createUser({
    user: "m103-admin",
    pwd: "m103-pass",
    roles: [
      {role: "root", db: "admin"}
    ]
  })
'

File Structure (WiredTiger)

You typically don't need to interact with this data folder to be modified may be to be read only. These files isn't designed for user modefications and if you modefied them you may face crashes or data lose.

To List --dbpath directory:

ls -l /data/db

WiredTiger      
WiredTige.wt  
WiredTiger.lock 
WiredTiger.turtle  
WiredTigerLAS.wt   
_mdb_catalog.wt
mongod.lock
sizeStorer.lock
collection-n.wt
index-n.wt
diagnostic.data
journal
storage.bson

wiredTiger files: this group of files is related to how wiredTiger storage engine keep track of info like cluster metadata and wiredTiger specific configurations options

wiredTiger.lock: act as a safty that prevent another mongodb process to point the same data folder

Collections and Index files .wt:

The next group is the files end with .wt these files is the collection and index data itself.
each collection and index has it's own file.
These files are designed to interact with through the Mongodb server process rathera third party tool.
Modeifying these can lead to data lose and crashes

diagnostic.data folder:

to collect the dignostic data which is captured by Full Time Data Capture, or FTDC module.
These data is only use for diagnostic purpose by mongodb support engineers

It collects data from the following commands:

serverStatus: db.serverStatus({tcmalloc: true})
replSetGetStatus: rs.status()
collStats for local.oplog.rs: db.getSiblingDB('local').oplog.rs.status()
getCmdLineOpts: db.adminCommand({getCmdLineOpts: true})
buildInfo: db.adminCommand({buildInfo: true})
hostInfo: db.adminCommand(hostInfo: true)

journal files:

write operations are buffered in memory and flushed every 60 seconds or when the journal files reach 2 Gbytes
write ahead logging system to on-disk journal file. It's first buffered in memory and synced to disk every 50ms
journal file max 100 mega bytes in size
wiredTiger can use journal files to recover data in case of failure

mongod.lock is like wiredTiger.lock file

mongod.log file saves log files!

mongodb-27017.sock file in /tmp folder: socket file use to create socket connection to the specified port

Basic Commands

Shell helpers ---> wraps db commands

db.<method>() --> database commands
rs.<method>() --> replica set commands
sh.<method>() --> sharding commands

db.<'Collection'>.<'method'>
user management:
```
- db.createUser()
- db.dropUser()
```

collection management:

- db.renameCollection()
- db.collection.createIndex()
- db.collection.drop()

Database management:

- db.dropDatabase()
- db.createCollection()

Database status:
```
- db.serverStatus()
```

Database commands: under the hood

- db.runCommand({ <'Command'> })
- db.commandHelp("command")

Creating index with Database Command:

db.runCommand({
   "createIndexes":"<collection_name>",
      "indexes":[
         {
            "key":{ "product": 1 },
            "name": "name_index"
         }
      ]
   }
)

Creating index with Shell Helper:

db.<collection>.createIndex(
   { "product": 1 },
   { "name": "name_index" }
)

Logging Basics

MongoDB provide to logging facilities to tracking activites on your database:

Process Log: collectes the activites into one of the following components:
- ACCESS - messages related to access control, such as authentication
- COMMAND - messages related to database commands
- CONTROL - messages related to control activities such as initialization
- FTDC - messages related to the diagnostic data collection mechanism
- GEO - messages related to parsing geo-spatial shapes
- INDEX - messages related to indexing operations
- NETWORK - messages related to network activities such as accepting connections
- QUERY - messages related to query planner and other query activities
- REPL - messages related to replica set, such as initial sync or hearbeats
- REPL_HB - messages related to replica set heartbeats nested under replication
- ROLLBACK - messages related to replica set ROLLBACK nested under replication
- SHARDING - messages related to sharding operations
- STORAGE - messages related to storage activities
- JOURNAL - messages related to journaling activities
- WRITE - messages related to write operations, such as update commands

to retrive a log commponents from db I can use this command: db.getLogComponents()

Log verbosity Levels

verbosity: parent verbosity on the object level ranges (0 -> 5) the higher the more verbose
component.verbosity: for each individual component if set to -1 it will inherit the parent verbosity
-1 : inherit from parent
0: Default verposity to include informational message
1-5: increase the verbosity level to include debug messages

To look to logs we can use one of the following:

db.adminCommand({"getLog": "global"})
tail -f 100 /path/to/log/file

To change the verbosity level of one component use the following command: db.setLogLevel(<level>: "<component>")

Severity levels:

F - fatal E - error W - warning I - info (Verbosity Level 0) D - debug (Verbosity Level 1-5)

Log Message sections:

<timestamp> 
<severity> 
<component> 
[<connection>] 
action <action>  ex command admin.$cmd  $ indicates db command
appName: <clientThatTriggeredOperation> ex mongoShell 
command: <the command itself> 
<metadata>
<operationTime>   10ms

Profiling the Database

Profiling stores more detailed info than logging
not all actions are captured on profiler
Profiler is enabled on database level for each db separatly
if enabled, stores all operations on db in a new collection called system.profile

Events Captured by the Profiler:

CRUD
Adminstrative operations
Configuration operations events are captured by profiler

Profiler Settings:

0 -> Profiler is off and does not collect any data, this is the default.
1 -> Profiler collects data for the operations that take longer than the value of ms
2 -> Profiler collect all data for all operations

slow ops: by default any operation that takes longer than 100ms and can be adjusted by setting the slowms variable

To get the Profiling Level: db.getProfilingLevel()
To set the Profiling Level: db.setProfilingLevel(<level>, {slowms: <Number>})

Basic Security

Authentication Mechanisms:

SCRAM(Salted Challeng Response Authentication Mechanism): is the default and the most basic form of client authentication (Community Edition)
X.509: this form using X.509 certificate for authentication. this is more secure and complex(Community Edition)
LDAP(Lightweight Directory Access Protocol)(Enterprise Only)
KERBEROS:(Enterprise Only)

Authorization: Role Based Access Control

Each user has one or more roles.
Each role has one or more privileges.
A privilege represents a group of actions and the resources to those actions apply to.
Roles support a high level of responsibility isolation for operational tasks.

Note

when the mongodb is run for the first time, no users exist in the db to connect you must connect from the local machine on which the server is run after creating the first user, the local host execution is closed

Localhost Exceptions:

Allows you to access a MongoDB server that enforces authentiaction but doesn't yet have a configured user for you to authenticate with.
Must run the mongo shell from the same host
The local exception closes after you created the first user
Always create a user with adminstrative privileges first

Create new user with the root role (also, named root):

use admin
db.createUser({
  user: "root",
  pwd: "root123",
  roles : [ "root" ]
})

Connect to mongod and authenticate as root:

mongo --username root --password root123 --authenticationDatabase admin

Roles

Database users are granted roles
Custom Roles : tailored roles to attend specific needs of sets of users
Built-In Roles: Pre-packed MongoDB Roles
Role structure:
- set of privileges each privilege defines a set of actions over a resource
- resources:
  - specific db and specific collection {db: "products", collection: "inventory"}
  - all dbs and all collections {db: "", collection: ""}
  - all dbs and specific collection {db:"", collection:"accounts"}
  - specific db and any collection {db:"products", collection: ""}
  - cluster --> replicasets or shards {cluster: true}
- Actions allowed over a resource { resource: { cluster: true }, actions: ["shutdown"] }
- role can inherit from other roles
- network auth previliges (ip whitelist)

Built-in Roles Sets:

Per Database level:
- Database User
  - read
  - readWrite
- Database Administration
  - dbAdmin
  - userAdmin
  - dbOwner
- Cluster Administration
  - clusterAdmin
  - clusterManager
  - clusterMonitor
  - hostManager
- Backup/Restore
  - backup
  - restore
- Super user
  - root
All Databases level:
- readAnyDatabase
- readWriteAnyDatabase
- dbAdminAnyDatabase
- userAdminAnyDatabase
- root

Main Common Roles:

userAdmin role:
- have all actions on user management (create user, change password, grant role, view role .....)
  - changeCustomData
  - changePassword
  - createRole
  - createUser
  - dropRole
  - dropUser
  - grantRole
  - revokeRole
  - setAuthenticationRestriction
  - viewRole
  - viewUser
- doesn't have any access on data cannot list, read or write any data other than the users
```
use admin
db.createUser(
   {
      user: "m103-application-user",
      pwd: "m103-application-pass",
      roles: [
         {
               db: "applicationData",
               role: "readWrite"
         }
      ]
   }
);
```
dbAdmin role:
- has access to DDL operations only.
- Provides the ability to perform administrative tasks such as schema-related tasks, indexing, and gathering statistics.
- This role does not grant privileges for user and role management.
```
db.creatUser(
  {
     user: "user_name",
     pwd: "p@ssw0rd",
     roles: [
        {
              db: "m103",
              role: "dbAdmin"
        }
     ]
  }
);
```
- system.profile collection:
  - changeStream
  - collStats
  - convertToCapped
  - createCollection
  - dbHash
  - dbStats
  - dropCollection
  - find
  - killCursors
  - listCollections
  - listIndexes
  - planCacheRead
- All non-system collections (i.e. database resource):
  - bypassDocumentValidation
  - collMod
  - collStats
  - compact
  - convertToCapped
  - createCollection
  - createIndex
  - dbStats
  - dropCollection
  - dropDatabase
  - dropIndex
  - enableProfiler
  - listCollections
  - listIndexes
  - planCacheIndexFilter
  - planCacheRead
  - planCacheWrite
  - reIndex
  - renameCollectionSameDB
  - storageDetails
  - validate
dbOwner role:
- The database owner can perform any adminstrative action on the database
- This role combines the privileges granted by the readWrite, dbAdmin and dbUser roles.

Note

in most cases you create user using admin database.

we can add roles to user using the foloowing

db.grantRolesToUser( "dba",  [ { db: "playground", role: "dbOwner"  } ] )

to Show role privileges:

db.runCommand( { rolesInfo: { role: "dbOwner", db: "playground" }, showPrivileges: true} )

Server Tools Overview

List mongodb binaries: this will list all tools installed with mongodb

find /usr/bin/ -name "mongo*"

Use mongostat to get stats on a running mongod process:
```
mongostat --help
mongostat --port 30000
```

Use mongodump to get a BSON dump of a MongoDB collection:

mongodump --help
mongodump --port 30000 --db applicationData --collection products
ls dump/applicationData/
cat dump/applicationData/products.metadata.json

Use mongorestore to restore a MongoDB collection from a BSON dump:
```
mongorestore --drop --port 30000 dump/
```

Use mongoexport to export a MongoDB collection to JSON or CSV (or stdout!):

mongoexport --help
mongoexport --port 30000 --db applicationData --collection products
mongoexport --port 30000 --db applicationData --collection products -o products.json

Tail the exported JSON file:
```
tail products.json
```
Use mongoimport to create a MongoDB collection from a JSON or CSV file:
```
mongoimport --port 30000 products.json
```

Notes

Mongodump can create a data file and a metadata file, but mongoexport just create a data file only.
By default, mongoexport send the output to standard output, but mongodump write to a file
mongoexport is slower because its convert every file to json before export

Chapter 2: Replication

What is Replication?

Replication is the concept of manitaining multiple copies of your data. This because you never assume that all your servers will be all over available. To make sure at anytime any server is down you can still access your data Availability.

MongoDB Replica Set

Replicaset is a group of mongod nodes that work on the same data
consists of one primary node that handles the data and secondary nodes that sync up with the primary
if the primary fails, a secondary node takes it's place in a process called failover where nodes vote for which node will become the primary in a process called election
Default Replication protocol ----> pv1 which is based on raft protocol
Read more about the Simple Raft Protocol and the Raft Consensus Algorithm.
Operation log oplog is statement based log for each node in the Replicaset that keeps track for all write operations.
every time a write is successfully applied to the primary node will get recorded in the oplog
Arbiter member in a Replicaset doesn't hold data, cann't be primary and is used only as a tie-breaker in leader-primary node election
odd number of nodes is prefered for election
if you used even number make sure the magority are available as you will need to have at least 3 nodes available
any failer or vote happen is add as s topology change in the replica set configuration which is defined in one node and shared between all the nodes
the election will not happen if the majority isn't available for example the majority of 4 is 3 so if 2 nodes down the election won't happen
Avoid using Arbiter
We can defin a hidden node to provide a read only node or have a copy off the data hidden of the application.
We can set a node as delayed for a specific time to work as a backup
Replicaset can have up to 50 member
only max of 7 nodes can be voting to minimize election time

Setting Up a Replica Set

We will independtly launching 3 mongod process and try to connect them to replicate data for us.

adding keyFile to security section in config file so all members can authentiacte each other using this keyFile
This is adition to client auth
Create keyFile using openssl openssl rand -base64 741 > /var/mongodb/pki/m103-keyfile
change permission of file to read permission chmod 600 /var/mongodb/pki/103-keyfile

add replication.replSetName to config file

storage:
   dbPath: /var/mongodb/db/node1
net:
   bindIp: 192.168.103.100,localhost
   port: 27011
security:
   authorization: enabled
   keyFile: /var/mongodb/pki/m103-keyfile
systemLog:
   destination: file
   path: /var/mongodb/db/node1/mongod.log
   logAppend: true
processManagement:
   fork: true
replication:
   replSetName: m103-example

Create the dbPath folder mkdir -p /var/mongodb/db/node1
Start mongod using this config file mongod -f node1.conf

create other 2 nodes with the same replSetName and keyfile just edit the dbPath, logPath and port by coping the configurations file

cp  node1.conf node2.conf
cp  node1.conf node3.conf

storage:
   dbPath: /var/mongodb/db/node2
net:
   bindIp: 192.168.103.100,localhost
   port: 27012
security:
   authorization: enabled
   keyFile: /var/mongodb/pki/m103-keyfile
systemLog:
   destination: file
   path: /var/mongodb/db/node2/mongod.log
   logAppend: true
processManagement:
   fork: true
replication:
   replSetName: m103-example

storage:
   dbPath: /var/mongodb/db/node3
net:
   bindIp: 192.168.103.100,localhost
   port: 27013
security:
   authorization: enabled
   keyFile: /var/mongodb/pki/m103-keyfile
systemLog:
   destination: file
   path: /var/mongodb/db/node3/mongod.log
   logAppend: true
processManagement:
   fork: true
replication:
   replSetName: m103-example

Create the dbPath folder for node2 mkdir -p /var/mongodb/db/node2
Start mongod using this config file of node2 mongod -f node2.conf
Create the dbPath folder for node3 mkdir -p /var/mongodb/db/node3
Start mongod using this config file of node3 mongod -f node3.conf
Connecting to node1: mongo --port 27011
Initiating the replica set on one like node1 rs.initiate()

Creating a user: to use to connect to the rest node using it

use admin
db.createUser({
   user: "m103-admin",
   pwd: "m103-pass",
   roles: [
      {role: "root", db: "admin"}
   ]
})

Exiting out of the Mongo shell and connecting to the entire replica set:

exit
mongo --host "m103-example/192.168.103.100:27011" -u "m103-admin"
-p "m103-pass" --authenticationDatabase "admin"

Getting replica set status: rs.status()

Adding other members to replica set:

rs.add("m103:27012")
rs.add("m103:27013")

Getting an overview of the replica set topology: rs.isMaster()
Stepping down the current primary: rs.stepDown()
Checking replica set overview after election: rs.isMaster()

Replication Configuration Document

BSON doc that holds the configuration of the replicaset and is shared across all nodes
JSON Object that define the configuration options of our replica set.
Can be configured manually from the shell
There are set of mongo shell replication helper methods that make it easier to manage :
```
rs.add
rs.initiate
res.remove
rs.reconfig
rs.config || rs.conf
```

{
    _id: string,  // the name of the replicaset
    version: int,  // gets incremented every time the configuration changes
    members: [
        {
            _id: string, // can't be changed once set
            host: string,
            arbiterOnly: bool,
            hidden: bool,  // Not visible to app handles specific operations
            priority: number,  // range (0 - 1000) higher priority members tend to be elected more often, 
                               // changing the priority triggers an election
                               // priority 0 is excluded from being a primary  (arbiteronly and hidden should be 0)
            slaveDelay: int  // the riplication delay of the node in seconds
                             // setting this setting implies that the node will be hidden and the priority is 0
        },
        ...
    ]
}

Replication Commands

rs.status():

reports health of the nodes
uses data from heartbeats so it can be seconds out of date
optime: the last time the node did an operation from the oplog

rs.isMaster():

describes the role of the node

db.serverStatus()['repl']:

is a section of the server status full output
similar to isMaster
rbid field doesn't appear on isMaster and it shows the number of times a rollback occured on the node

rs.printReplicationInfo():

returns the oplog data relative to the current node
contains timestamp to the first and last oplog event in the node

Local DB

Display all databases (by default, only admin and local):
```
mongo
show dbs
```
Display collections from the local database (this displays more collections from a replica set than from a standalone node):
```
use local
show collections
```
has startup_log collection only in standalone node
has several collections on replicset:

me oplog.rs replset.electoin replset.minvali startup_log system.replset system.rollback.id

oplog.rs:
- is the center point of the replication mechanism
- It keeps track of all statements that is being replicated
- is a capped collection (has max size)
- if you run var stats = db.oplog.rs.stats() then:
- stats.capped --> true if a capped collection
- stats.size --> Get current size of the oplog
- stats.maxSize --> Get size limit of the oplog
- by default it has 5% of the free disk space but can be cofigured through oplogSizeMB option under replication in the config file
- is created after creating replset
- once the limit is reached, the early operations are overriden
- the replication window:
  - the time it takes to fill the oplog completely and start overriding old statements
  - important to determine the time a node can afforded to be down without requireing human intervention to help it recover
  - it's inversly proportional to the system load
- one operation can reslut in many entries in the oplog such as updateMany
- data written in local dbs won't be replicated

Reconfiguring a Running Replica Set

Let's assume we have a replica set of 3 nodes and we need to add 2 more 1 as Arbiter and 1 as secodary node.

node4.conf:

storage:
   dbPath: /var/mongodb/db/node4
net:
   bindIp: 192.168.103.100,localhost
   port: 27014
systemLog:
   destination: file
   path: /var/mongodb/db/node4/mongod.log
   logAppend: true
processManagement:
   fork: true
replication:
   replSetName: m103-example

arbiter.conf:

storage:
   dbPath: /var/mongodb/db/arbiter
net:
   bindIp: 192.168.103.100,localhost
   port: 28000
systemLog:
   destination: file
   path: /var/mongodb/db/arbiter/mongod.log
   logAppend: true
processManagement:
   fork: true
replication:
   replSetName: m103-example

Starting up mongod processes for our fourth node and arbiter:
```
mongod -f node4.conf
mongod -f arbiter.conf
```
From the Mongo shell of the replica set, adding the new secondary and the new arbiter:
```
rs.add("m103:27014")
rs.addArb("m103:28000")
```
Checking replica set makeup after adding two new nodes:
```
rs.isMaster()
```
Removing the arbiter from our replica set:
```
rs.remove("m103:28000")
```
Assigning the current configuration to a shell variable we can edit, in order to reconfigure the replica set:
```
cfg = rs.conf()
```
Editing our new variable cfg to change topology - specifically, by modifying cfg.members:
```
cfg.members[3].votes = 0
cfg.members[3].hidden = true
cfg.members[3].priority = 0
```
Updating our replica set to use the new configuration cfg:
```
rs.reconfig(cfg)
```

Reads and Writes on a Replica Set

By default read and write operations aren't allowed on secondary nodes
to enable reading on a secondary node: rs.slaveOk()
writing is forbidden to replica sets
if no nodes are secondary the primary will be secondary and we will not be able to write to the replica set

Failover and Elections

suppose the following scenario:

replicaset with 3 nodes 1p and 2s
to do a rolling upgrade we do:
1. stop one of the secondary and bring it up with the new version
2. do the same to the other secondary node
3. perform an election by running rs.stepDown() -> this will lead to that the primary node becomes a secondary
4. do the step 1 to the secondary that was a primary

Election: happens when

there is a change in topology
reconfiguring a replicaset
the primary node becomes unavailable (must elect a new node to be primary)
using rs.stepDown() (must elect a new node to be primary)

Election candidates:

if all nodes has the same priority then the one with the latest data will vote for itself and ask other nodes for support
if two nodes run for election simaltaneously in case of odd number of nodes ---> the odd node will decide the winner
in case of even number of nodes ---> a tie can happen then election will be repeated
a node with priority of 1 or higher can be elected
a node with priority of 0 can't run election and can't be primary, but can vote
a higher priority has a higher chance of being a primary
Note when running rs.isMaster() nodes that can't be elected show up as passives
if a primary can't reach any voting secondary, it wil automatically step down and be a secondary and if no primary in the replset then it won't be reachable

Write Concerns

write concerns are acknowledgement mechanism to increase durability
for a write to be durable, majority of the nodes must acknowledge the success of the write
Levels:
1. 0 no wait for acknowledgement means the write might successed or failed
2. 1 (Default) wait for acknowledgment from primary only
3. greater than 1 waint for primary and one or more secondary members
4. majority wait for the majority of the replicaset
Options :
1. wtimeout: (int) the time to wait for write concern before marking the operation as failed
2. j: (bool) node acknowledge the write and commit it in the journal files before returning an acknowledge. if j is false then the node will report success when the write stored in memory and before waiting for journaling
Write Concern Commands:
- insert
- update
- delete
- findandmodify
write concern is 1 by default

db.collection.insert(
   { }, 
   { writeConcern: { w: "majority", wtimeout: 60 } }
)

Read Concerns

returns the data if it has been saved to a number of nodes Levels:
- local --> (default) most recent data to the cluster on primary only and doesn't guarantee that it is durable
- available --> same as local and default against secondary differs in sharded clusters
- majority --> returns if in a majority of the nodes not the latest note supported in MMApv1 storage engine
- linearizable --> like majority and read your only write functionality

Notes:

local & available ->> fast and latest but not safe
majority ->> fast and safe not latest
linearizable ->> safe and latest - not fast - single doc reads only
for secondary reads ->> local & available is fast only but not always latest

READ preference

Route read operations to secondary nodes
is a driver-side setting modes:
- primary (default) only to primary
- primaryPreferred (primary and if not available the secondary)
- secondary routes to only secondary
- secondaryPreferred if no secondary then primary
- nearest the least network latency to the host

Chapter 3: Sharding

What is Sharding?

higher cost of vertical scaling
scaling horizontaly by divid the data into multiple instances
impact on operational tasks like (backup) ---> backing up several 2 TB hard disks in parallel is faster than 20 TB single hard disk
single threaded operations benefit from distributed environment ex: aggregation framework
geographically distributed datasets

Sharding Architecture

we setting up a router process that accept queries from clients this router process called Mongos
We can have any number of Mongos processes
Mongos using Metadata stored in config servers that has info about where each piece is stored.
We need to make sure that the Metadata is highly availabe using replication
We deploy a Config Server Replica Set.
Primary Shard
- every database has a primary shard
- that holds non-sharded collections
- merge aggregation data from different shards

Setting Up a Sharded Cluster

The minumum requirements to have a sharded cluster is to have (mongos process, one shard, CSRS)

deploy config server replicaset with mongod config file csrs_1.conf, csrs_2.conf, csrs_3.conf:

sharding:
   clusterRole: configsvr
replication:
   replSetName: m103-csrs
security:
   keyFile: /var/mongodb/pki/m103-keyfile
net:
   bindIp: localhost,192.168.103.100
   port: 26001
systemLog:
   destination: file
   path: /var/mongodb/db/csrs1.log
   logAppend: true
processManagement:
   fork: true
storage:
   dbPath: /var/mongodb/db/csrs1

sharding:
   clusterRole: configsvr
replication:
   replSetName: m103-csrs
security:
   keyFile: /var/mongodb/pki/m103-keyfile
net:
   bindIp: localhost,192.168.103.100
   port: 26002
systemLog:
   destination: file
   path: /var/mongodb/db/csrs2.log
   logAppend: true
processManagement:
   fork: true
storage:
   dbPath: /var/mongodb/db/csrs2

sharding:
   clusterRole: configsvr
replication:
   replSetName: m103-csrs
security:
   keyFile: /var/mongodb/pki/m103-keyfile
net:
   bindIp: localhost,192.168.103.100
   port: 26003
systemLog:
   destination: file
   path: /var/mongodb/db/csrs3.log
   logAppend: true
processManagement:
   fork: true
storage:
   dbPath: /var/mongodb/db/csrs3

Starting the three config servers:

mongod -f csrs_1.conf
mongod -f csrs_2.conf
mongod -f csrs_3.conf

Connect to one of the config servers:mongo --port 26001
Initiating the CSRS: rs.initiate()

Creating super user on CSRS:

use admin
db.createUser({
   user: "m103-admin",
   pwd: "m103-pass",
   roles: [
      {role: "root", db: "admin"}
   ]
})

Authenticating as the super user: db.auth("m103-admin", "m103-pass")

Add the second and third node to the CSRS:

rs.add("192.168.103.100:26002")
rs.add("192.168.103.100:26003")

prepare mongos config file:

sharding:
   configDB: m103-csrs/192.168.103.100:26001,192.168.103.100:26002,192.168.103.100:26003
security:
   keyFile: /var/mongodb/pki/m103-keyfile
net:
   bindIp: localhost,192.168.103.100
   port: 26000
systemLog:
   destination: file
   path: /var/mongodb/db/mongos.log
   logAppend: true
processManagement:
   fork: true

Start the mongos server: mongos -f mongos.conf
Connect to mongos:

mongo --port 26000 --username m103-admin --password m103-pass --authenticationDatabase admin

Check sharding status: sh.status()
Updated configuration for node1.conf:

sharding:
   clusterRole: shardsvr
storage:
   dbPath: /var/mongodb/db/node1
   wiredTiger:
      engineConfig:
         cacheSizeGB: .1
net:
   bindIp: 192.168.103.100,localhost
   port: 27011
security:
   keyFile: /var/mongodb/pki/m103-keyfile
systemLog:
   destination: file
   path: /var/mongodb/db/node1/mongod.log
   logAppend: true
processManagement:
   fork: true
replication:
   replSetName: m103-repl

Updated configuration for node2.conf:

sharding:
   clusterRole: shardsvr
storage:
   dbPath: /var/mongodb/db/node2
   wiredTiger:
      engineConfig:
         cacheSizeGB: .1
net:
   bindIp: 192.168.103.100,localhost
   port: 27012
security:
   keyFile: /var/mongodb/pki/m103-keyfile
systemLog:
   destination: file
   path: /var/mongodb/db/node2/mongod.log
   logAppend: true
processManagement:
   fork: true
replication:
   replSetName: m103-repl

Updated configuration for node3.conf:

sharding:
   clusterRole: shardsvr
storage:
   dbPath: /var/mongodb/db/node3
   wiredTiger:
      engineConfig:
         cacheSizeGB: .1
net:
   bindIp: 192.168.103.100,localhost
   port: 27013
security:
   keyFile: /var/mongodb/pki/m103-keyfile
systemLog:
   destination: file
   path: /var/mongodb/db/node3/mongod.log
   logAppend: true
processManagement:
   fork: true
replication:
   replSetName: m103-repl

Connecting directly to secondary node (note that if an election has taken place in your replica set, the specified node may have become primary):

mongo --port 27012 -u "m103-admin" -p "m103-pass" --authenticationDatabase "admin"

Shutting down node:

use admin
db.shutdownServer()

Restarting node with new configuration:

mongod -f node2.conf

Stepping down current primary: rs.stepDown()
Adding new shard to cluster from mongos: sh.addShard("m103-repl/192.168.103.100:27012")

Config DB

Config DB maintained and used internally by mongodb, so generally you should never write any data to it. However it's got some useful information so we are going to read from it.

If you'd like to explore the collections on the config database, you can find the instructions here:

Switch to config DB:
```
use config
```

Show config DB collections

show collections
---------------------------
actionlog
changelog
chunk
collections
databases
lockpings
locks
migrations
mongos
shards
tags
transactions
version

Query config.databases: will return each database in our cluster as one document

db.databases.find().pretty()
-------------------------
{"_id": "m103", "primary": "m103-repl", "partitioned": true}

Query config.collections: gives us info on collections that have been sharded and the shard key used

db.collections.find().pretty()
-----------------------------
{
   "_id": "config.system.sessions",
   "lastmodEpoch": ObjectId("sdfdgfghh556546fgh6fdgh"),
   "lastmod": ISODate("------------------------"),
   "dropped": false,
   "key": {
      "_id": 1
   },
   "unique": false,
   "uuid": UUID("f5f4ff-gfggfdgdg45-4455-d4f544f5d5df4")
},
{
   "_id": "m103.products",
   "lastmodEpoch": ObjectId("sdfdgfghh556546fgh6fdgh"),
   "lastmod": ISODate("------------------------"),
   "dropped": false,
   "key": {
      "salePrice": 1
   },
   "unique": false,
   "uuid": UUID("f5f4ff-gfggfdgdg45-4455-d4f544f5d5df4")
}

Query config.shards: This tell us about the shards in our cluster

db.shards.find().pretty()
-------------------------
{
   "_id": "m103-repl",
   "host": "m103-repl/192.168.103.100:27011,192.168.103.100:27012,192.168.103.100:27013",
   "state": 1
},
{
   "_id": "m103-shard-2",
   "host": "m103-shard-2/192.168.103.100:27014,192.168.103.100:27015,192.168.103.100:27016",
   "state": 1
}

Query config.chunks: Each chunk for every collection in this database is returned as one document. The enclusive and exclusive maximum define the chunk range of the shard key value. That means that any document in the associated collection who's shard key value falss into this chunks range will end up in this chunk, and this chunk only

db.chunks.find().pretty()
-------------------------
{
   "_id": "m103.products-salesPrice_MinKey",
   "lastmodEpoch": ObjectId("sdfdgfghh556546fgh6fdgh"),
   "lastmod": Timestamp(2,0),
   "ns": "m103.productions",
   "min": {
      "salePrice": {"$minKey": 1}
   },
   "max": {
      "salePrice": 14.99
   },
   "shard": "m103-shard-2"
},
{
   "_id": "m103.products-salesPrice_14.99",
   "lastmodEpoch": ObjectId("sdfdgfghh556546fgh6fdgh"),
   "lastmod": Timestamp(2,1),
   "ns": "m103.productions",
   "min": {
      "salePrice": 14.99
   },
   "max": {
      "salePrice": 33.99
   },
   "shard": "m103-shard-2"
},
{
   "_id": "m103.products-salesPrice_33.99",
   "lastmodEpoch": ObjectId("sdfdgfghh556546fgh6fdgh"),
   "lastmod": Timestamp(2,1),
   "ns": "m103.productions",
   "min": {
      "salePrice": 33.99
   },
   "max": {
      "salePrice": {"$maxKey": 1}
   },
   "shard": "m103-shard-2"
}

Query config.mongos: holds data about the mongos processes connected to the cluster

db.mongos.find().pretty()
--------------------------
{
   "_id": "m103:26000",
   "mongoVersion": "3.6.2-rc0",
   "ping": ISODate("------------------------"),
   "up": NumberLong(3892),
   "waiting": true
}

Shard Keys

This is the indexed field(s) used to partition collection on shards in our cluster.

How the shard key is used to distribute your data?

Consider a collection with some number of documents in them. MongoDB uses the shard key to divide up these documents into logical groupings that MongoDB then distributes across our sharded cluster.

MongoDB he refers to these groupings as chunks. The value of the field or fields we choose as our shard key help to define the inclusive lower bound, and the exclusive upper bound of each chunk.

Because the shard key is used to define chunk boundaries, it also defines which chunk a given document is going to belong to.

Every time you write a new document to the collection, the MongoS router checks which shard contains the appropriate chunk for that documents key value, and routes the document to that shard only.

The shard key must be present on every document in the collection, and every new inserted document.
The shard key Fields must be indexed and indexes must exists first before you can select the indexed fields for your shard key.
Shard Keys are immutable:
- cannot be changed after sharding
- cannot specify another key or update a value for that key in any document
Shard Keys are permanent:
- cannot unshard a collection unless dropped and restored again.

How To Shard:

Use sh.enableSharding("<database>") to enable sharding for the specified database.
Use db.collection.createIndex() to create Index on shard key fields
Use ```sh.shardCollection("database.collection", {shard key: 1}) to shard the collection

Picking a Good Shard Key:

What makes a good Shard Key? The goal is a shard key whose values provides good write distribution.

Cardinality:
- Higher Cardinality = many possible unique shard key values
- The higher the better
Frequency:
- High Frequency = low repetition of a given unique shard key value.
- If we have 90% of document have the same the shard key value that means that they well be distributed to the same shard which is bad.
- The lower the better.
Type of Change:
- Avoid shard keys that change monotonically
- Like a counter, timestamp, objectId ...
- This will lead to all documents will be in the shard with max range
- Should be avoided

Read Isolation:

Which shard has the data that meets our query parameter?
MongoDb directs tatgeted queries to a single shard
Without the shard key, MongoDB has to ask every shard
The key should be used frequently in query operations to direct the router to it (faster reading)

Hashed shard key:

That is a shard key where the underlying index is hashed. Like hash tables, the key is hashed and the hash is used to distribute the data When to Use:

monotonic changed keys

Drawbacks:

Queries on ranges of shard key values are more likely to be scatter-gathered
Cannot support geographically isolated read operations using zone sharding
Hashed index must be on a single non-array field
Hashed index don't support fast sorting

Sharding using a Hashed Shard Key:

Use sh.enableSharding("<database>") to rnable sharding for the specified database
Use db.collection.createIndex({"field": "hashed"}) to create the index for your shard key fields
Use sh.shardCollection("<db>.<collection>", {field: "hashed"}) to shard the collection

Chunks

Default size 64MB Min size 1MB Max size 1024MB Can be changed at runtime using:

using config db
inserting into settings collection: db.settings.save({_id: "chunksize", value: <inMB> (ex: 2)})
changes will be applied when importing or saving new data

Shard key value frequency affects the number of chunks: Jumbo Chunks:

Larger than the defined chunksize
Can't be moved or split
Once marked as jumbo the balancer skips these cunks and avoid trying to move them
In some cases these will not be able to split
this results from poor choice of key

Balancing

MongoDB balancer identifies the shard with too many chunks and distribute them to other shards.
Balancer runs on primary member of config servers starting 3.2 before it ran in mongos.
When it detects a migration threshold, it starts a balancer round
The balancer can migrate chunks in parallel but a shard can only participate in a single migration at a time
Number of chunks to migrate in a round = floor(n / 2) n = #shards
Then a nother round takes place until the shards are balaced
Balancer affects performance

Balancer Management Methods: Start / Stop the Balancer

sh.startBalacer(timeout, interval) timeout-> how mins to wait to start or stop the balancer
sh.stopBalancer(timeout, interval) interval-> how mins the client wait to check the status of the balancer again
sh.setBalacerState(bool) true, flase on/off

You can read more about scheduling the balancer on the MongoDB Sharding docs.

Queries in a Sharded Cluster

The mongos is responsible for routing queries
If the shard key is in the query predicate, then it will target a specific shards (very efficient) otherwise it performs a scatter gather on list of shards (all), opens a cursor in each one, performs the query then merges the result from each shard.

Sort, Limit and Skip in sharded clusters:

sort( )
- The mongos pushes the sort to each shard and merge-sorts the results
limit( )
- The mongos passes the limit to each targeted shard, then re-applies the limitto the merged set of results
skip( )
- The mongos performs the skip against the merged set of results

You can read more about routing Aggregation queries in a sharded cluster on the MongoDB sharding docs.

Targeted Queries vs Scatter Gather

Each mongos keeps local cashed map of the shard chunk relationships that exists on the config server

Shard	Data
1	minKey -> 10000000
2	10000000 -> 20000000
3	30000000 -> maxKey

So when the mongos receives a query whose predicate includes the shard key, the mongos can look at the table and know exactly which shards to direct that query to.

The mongos opens a cursor against only those shards that can satisfy the query predicate. Because the mongos is targeting the query to a subset of shards in the cluster, these targeted queries are generally faster than having to check in with every shard in the cluster.

If, for example, the mongos can satisfy the entire query by targeting a single shard, then the mongos can even bypass the merge stage and just return the results. This is particularly fast.

When the query predicate does not include the shard key, then the mongos cannot derive exactly which shards satisfythe query. These scatter gather queries must necessarily ping and wait for the reply of every shard in the cluster, regardless if they have something to contribute towards the execution of the query or not.

Depending on the number of shards in your cluster, the amount of network latency between shards and mongos and a number of other factors, these queries can be slow.

Compound indexes can be used as shard keys in this case, using any index prefix in the pridicate will result in a target query for example:

shard key: {a: 1, b: 1, c: 1}

Targeted Queries:
db.col.find({a: })
db.col.find({a: , b: })
db.col.find({a: , b: , c: })

Scatter-gather Queries:
db.col.find({b: })
db.col.find({c: })

Note Using db.col.find().explain() shows us detailed info about how we got the results

M201: MongoDB Performance

Learn how to optimize the performance of your MongoDB deployment.

Chapter 1: Introduction

Hardware Considerations & Configurations

MongoDB is a High Performance Database and to support your requirements it will require good hardware.

Let's discuss how mongodb uses hardware resources:

1. Memory

For execution of the operations
very important
Database is designed arround the usage of memory
mongodb engines are either very dependant on ram or has fully in-memory execution for its data management operations.
A signficant number of operations are rely heavily in RAM like:
- Aggregations
- Index Traversing
- Write Operations (are first performed on ram allocated pages)
- Query Engine
- connections (1MB per connection)
We can say the more RAM you have, the more performant you get from your mongodb

2. CPU

Used by all aplications for computational processing.
MongoDB using CPU with 2 main factors:
- Storage engine
- Concurrency Model (wired tiger has non locking concurrency control mechanism)
Mongodb try to use all available CPU cores
There is others operations that will require availability of CPU cores:
- page compression
- data calculation
- aggergation framework operations
- map reduce
Don't forget that not all write or read operations are non-locking operations, for example:
- Writing or updating to the same document will require each write to block other writes on that same document to comply.
- In situations like this multible CPUs do not help performance because the Threads cann't do their work in parallel

3. I/O

For presistance and communications between servers or withen the host services
Data presistence on disk
The higher IOPS (Input/Output Operations per Second)your disk provide, the faster the db operations
The type of disks will realy affect the overall performance of your MongoDB
We can used RAID architectures for redundancy of read and write operations
RAID 10 is the best for mongodb
RAID 0, 5 or 6 musn't be used
using several disks benifits mongodb by distributing the IO loads
the faster the network, the better performance with client apps
distributed architecture in mongodb means it needs low latency connection between different components to enhance performance
The type of newtork switches, Load balancers and firewall will affect the latency which need to be taken into consideration when analizing the network architecture of your application.

Chapter 2: MongoDB Indexes

Introduction to Indexes

What is the problem Indexes try to solve?
- Slow queries as instead of looking to all documents one by one [Collection Scan] O(n) we got the documentID from the index directly like the concept of dictionry index.
Collection Scan: is going through all docs in the collection 1 by 1 O(n) linear
The data structure used to store indexes B-tree
The index is associate with one or more fields.
The _id field is automaticlly indexed on all collections
Index Overhead:
- slow writes ---> the tree will be updated
- slow updates and deletes ---> the tree will be adjusted if the indexed field changes
- We don't have too many unneeded indexes which will affect the performance of insert, delete and update operations.

You can learn more about indexes by visiting the Indexes Section of the MongoDB Manual.

How Data is Stored on Disk

The MongoDB use to store data will differ between different storage engines that mongodb support(MMAPv1 - Wired Tiger - Other).
Each Collection has a file.wt
Each Index has a file.wt
Database Catalog: _mdb_catalog.wt
- Database Catalog contains info about collections and indexes that this mongod has.
Running mongod with these flags:
- --directoryperdb ---> each db has its own directory
- --wiredTigerDirectoryForIndexes ---> directory for indexes and directory for collections inside the db folder
- benefits of the previous structure:
  - increasing performance: if multiple disks are available, then distributing these files across different disks (using symbolic links) will increase IO parrallization.
  - We can make a disk for collections and disk for indexes.
mongodb can store data in compressed format which will increase the performance of data presistence but will require more cpu cycles
writing data from memory to disk will be triggered by two methods:
- user side: by specifying write concern that syncs operation with other instances like { writeConcern: { w: 3 } } which means at least one primary and 2 secondaries
- periodical internal process that regulates how data will be flushed and synced (sync periods)
Journaling:
- journal flushes are performed using group commits in compressed format.
- all writes are atomic
- { writeConcern: { j: true } } wont acknowledge a write unless it has been written in the journal
- will have impact on performance as it will wait until data is written on disk

You can learn more about how data is stored on disk in MongoDB by visiting the MongoDB Storage FAQ in the MongoDB Manual.

Single Field Indexes

the simplest index that can be created
db.<collection>.createIndex({ <field>: <direction> })
Key Features:
- Key from only one field
- Can find a single value for the indexed field
- Can find a range of values
- Can use dot notation to index fileds in subdocuments
- Can be used to finding several distict values in a single query

db.people.find({"ssn": "720-38-5636"}).explain("executionStats") ---> to view extra info about the query

queryPlanner: {
    namespace: 'myFirstDatabase.people',
    indexFilterSet: false,
    parsedQuery: { ssn: { '$eq': '720-38-5636' } },
    maxIndexedOrSolutionsReached: false,
    maxIndexedAndSolutionsReached: false,
    maxScansToExplodeReached: false,
    winningPlan: {
        stage: 'COLLSCAN',<--------
        filter: { ssn: { '$eq': '720-38-5636' } },
        direction: 'forward'
    },
    rejectedPlans: []
},
executionStats: {
    executionSuccess: true,
    nReturned: 1, <----------------|
    executionTimeMillis: 81,       |
    totalKeysExamined: 0,          |
    totalDocsExamined: 50474,<-----|
    executionStages: {
        stage: 'COLLSCAN', <-------
        filter: { ssn: { '$eq': '720-38-5636' } },
        nReturned: 1,
        executionTimeMillisEstimate: 27,
        works: 50476,
        advanced: 1,
        needTime: 50474,
        needYield: 0,
        saveState: 50,
        restoreState: 50,
        isEOF: 1,
        direction: 'forward',
        docsExamined: 50474 <----------
    }
}

We will notice that the winning plan is COLLSCAN that means in order to returned 1 document it made full collection scan for 50474 documents which is not efficient
Let's create an index on ssn
db.people.createIndex({ ssn: 1 }) to create index on ssn field
1 means that the index is ordered in ASC
explainable object:
- let exp = db.people.explain("executionStats")
- then we can run our queries on the exp object
- exp.find({"ssn": "720-38-5636"})

queryPlanner: {
    namespace: 'myFirstDatabase.people',
    indexFilterSet: false,
    parsedQuery: { ssn: { '$eq': '720-38-5636' } },
    maxIndexedOrSolutionsReached: false,
    maxIndexedAndSolutionsReached: false,
    maxScansToExplodeReached: false,
    winningPlan: {
        stage: 'FETCH',<-----
        inputStage: {
            stage: 'IXSCAN',<----
            keyPattern: { ssn: 1 },
            indexName: 'ssn_1',
            isMultiKey: false,
            multiKeyPaths: { ssn: [] },
            isUnique: false,
            isSparse: false,
            isPartial: false,
            indexVersion: 2,
            direction: 'forward',
            indexBounds: { ssn: [ '["720-38-5636", "720-38-5636"]' ] }
        }
    },
    rejectedPlans: []
},
executionStats: {
    executionSuccess: true,
    nReturned: 1,<-------------------|
    executionTimeMillis: 1,          |
    totalKeysExamined: 1,<-----------|--- the number of used indexed keys
    totalDocsExamined: 1,<-----------|
    executionStages: {
        stage: 'FETCH',
        nReturned: 1,
        executionTimeMillisEstimate: 0,
        works: 2,
        advanced: 1,
        needTime: 0,
        needYield: 0,
        saveState: 0,
        restoreState: 0,
        isEOF: 1,
        docsExamined: 1,
        alreadyHasObj: 0,
        inputStage: {
            stage: 'IXSCAN',
            nReturned: 1,
            executionTimeMillisEstimate: 0,
            works: 2,
            advanced: 1,
            needTime: 0,
            needYield: 0,
            saveState: 0,
            restoreState: 0,
            isEOF: 1,
            keyPattern: { ssn: 1 },
            indexName: 'ssn_1',
            isMultiKey: false,
            multiKeyPaths: { ssn: [] },
            isUnique: false,
            isSparse: false,
            isPartial: false,
            indexVersion: 2,
            direction: 'forward',
            indexBounds: { ssn: [ '["720-38-5636", "720-38-5636"]' ] },
            keysExamined: 1,
            seeks: 1,
            dupsTested: 0,
            dupsDropped: 0
        }
    }
},

Note:

If more than one fields are queried and one of them is indexed the indexed results will be returned then filtered by the other fields

db.examples.insertOne({_id: 0, subdoc: {indexedField: "value", otherField: "value"}})
db.examples.insertOne({_id: 1, subdoc: {indexedField: "wrongValue", otherField: "value"}})
db.examples.createIndex({"subdoc.indexedField": 1})
db.examples.explain("executionStats").find({"subdoc.indexedField": "value"})

  queryPlanner: {
      namespace: 'myFirstDatabase.examples',
      indexFilterSet: false,
      parsedQuery: { 'subdoc.indexedField': { '$eq': 'value' } },
      maxIndexedOrSolutionsReached: false,
      maxIndexedAndSolutionsReached: false,
      maxScansToExplodeReached: false,
      winningPlan: {
          stage: 'FETCH',
          inputStage: {
              stage: 'IXSCAN',<-------
              keyPattern: { 'subdoc.indexedField': 1 },
              indexName: 'subdoc.indexedField_1',
              isMultiKey: false,
              multiKeyPaths: { 'subdoc.indexedField': [] },
              isUnique: false,
              isSparse: false,
              isPartial: false,
              indexVersion: 2,
              direction: 'forward',
              indexBounds: { 'subdoc.indexedField': [ '["value", "value"]' ] }
          }
      },
      rejectedPlans: []
  },
  executionStats: {
      executionSuccess: true,
      nReturned: 1,
      executionTimeMillis: 0,
      totalKeysExamined: 1,
      totalDocsExamined: 1,
      executionStages: {
          stage: 'FETCH',
          nReturned: 1,
          executionTimeMillisEstimate: 0,
          works: 2,
          advanced: 1,
          needTime: 0,
          needYield: 0,
          saveState: 0,
          restoreState: 0,
          isEOF: 1,
          docsExamined: 1,
          alreadyHasObj: 0,
          inputStage: {
              stage: 'IXSCAN',
              nReturned: 1,
              executionTimeMillisEstimate: 0,
              works: 2,
              advanced: 1,
              needTime: 0,
              needYield: 0,
              saveState: 0,
              restoreState: 0,
              isEOF: 1,
              keyPattern: { 'subdoc.indexedField': 1 },
              indexName: 'subdoc.indexedField_1',
              isMultiKey: false,
              multiKeyPaths: { 'subdoc.indexedField': [] },
              isUnique: false,
              isSparse: false,
              isPartial: false,
              indexVersion: 2,
              direction: 'forward',
              indexBounds: { 'subdoc.indexedField': [ '["value", "value"]' ] },
              keysExamined: 1,
              seeks: 1,
              dupsTested: 0,
              dupsDropped: 0
          }
      }
  }```

We can use Single Field Index to query on range of values

exp.find({ssn: {$gte: "555-00-0000", $lt: "556-00-0000"}})

queryPlanner: {
    namespace: 'myFirstDatabase.people',
    indexFilterSet: false,
    parsedQuery: {
    '$and': [
        { ssn: { '$lt': '556-00-0000' } },
        { ssn: { '$gte': '555-00-0000' } }
    ]
    },
    maxIndexedOrSolutionsReached: false,
    maxIndexedAndSolutionsReached: false,
    maxScansToExplodeReached: false,
    winningPlan: {
    stage: 'FETCH',
    inputStage: {
        stage: 'IXSCAN',
        keyPattern: { ssn: 1 },
        indexName: 'ssn_1',
        isMultiKey: false,
        multiKeyPaths: { ssn: [] },
        isUnique: false,
        isSparse: false,
        isPartial: false,
        indexVersion: 2,
        direction: 'forward',
        indexBounds: { ssn: [ '["555-00-0000", "556-00-0000")' ] }
    }
    },
    rejectedPlans: []
},
executionStats: {
    executionSuccess: true,
    nReturned: 49,<-----------
    executionTimeMillis: 8,
    totalKeysExamined: 49,
    totalDocsExamined: 49,<------------
    executionStages: {
        stage: 'FETCH',
        nReturned: 49,
        executionTimeMillisEstimate: 4,
        works: 50,
        advanced: 49,
        needTime: 0,
        needYield: 0,
        saveState: 0,
        restoreState: 0,
        isEOF: 1,
        docsExamined: 49,
        alreadyHasObj: 0,
        inputStage: {
            stage: 'IXSCAN',
            nReturned: 49,
            executionTimeMillisEstimate: 0,
            works: 50,
            advanced: 49,
            needTime: 0,
            needYield: 0,
            saveState: 0,
            restoreState: 0,
            isEOF: 1,
            keyPattern: { ssn: 1 },
            indexName: 'ssn_1',
            isMultiKey: false,
            multiKeyPaths: { ssn: [] },
            isUnique: false,
            isSparse: false,
            isPartial: false,
            indexVersion: 2,
            direction: 'forward',
            indexBounds: { ssn: [ '["555-00-0000", "556-00-0000")' ] },
            keysExamined: 49,
            seeks: 1,
            dupsTested: 0,
            dupsDropped: 0
        }
    }
},

You can learn more about single field indexes by visiting the Single Field Indexes Section of the MongoDB Manual.

We can use a single field index to query in a set of single values

exp.find({ssn: {$in: ['001-29-9184', '177-45-0950', '265-67-9973']}})

  queryPlanner: {
      namespace: 'myFirstDatabase.people',
      indexFilterSet: false,
      parsedQuery: { ssn: { '$in': [ '001-29-9184', '177-45-0950', '265-67-9973' ] } },
      maxIndexedOrSolutionsReached: false,
      maxIndexedAndSolutionsReached: false,
      maxScansToExplodeReached: false,
      winningPlan: {
      stage: 'FETCH',
      inputStage: {
          stage: 'IXSCAN',
          keyPattern: { ssn: 1 },
          indexName: 'ssn_1',
          isMultiKey: false,
          multiKeyPaths: { ssn: [] },
          isUnique: false,
          isSparse: false,
          isPartial: false,
          indexVersion: 2,
          direction: 'forward',
          indexBounds: {
          ssn: [
              '["001-29-9184", "001-29-9184"]',
              '["177-45-0950", "177-45-0950"]',
              '["265-67-9973", "265-67-9973"]'
          ]
          }
      }
      },
      rejectedPlans: []
  },
  executionStats: {
      executionSuccess: true,
      nReturned: 3,
      executionTimeMillis: 0,
      totalKeysExamined: 6,
      totalDocsExamined: 3,
      executionStages: {
          stage: 'FETCH',
          nReturned: 3,
          executionTimeMillisEstimate: 0,
          works: 6,
          advanced: 3,
          needTime: 2,
          needYield: 0,
          saveState: 0,
          restoreState: 0,
          isEOF: 1,
          docsExamined: 3,
          alreadyHasObj: 0,
          inputStage: {
              stage: 'IXSCAN',
              nReturned: 3,
              executionTimeMillisEstimate: 0,
              works: 6,
              advanced: 3,
              needTime: 2,
              needYield: 0,
              saveState: 0,
              restoreState: 0,
              isEOF: 1,
              keyPattern: { ssn: 1 },
              indexName: 'ssn_1',
              isMultiKey: false,
              multiKeyPaths: { ssn: [] },
              isUnique: false,
              isSparse: false,
              isPartial: false,
              indexVersion: 2,
              direction: 'forward',
              indexBounds: {
              ssn: [
                  '["001-29-9184", "001-29-9184"]',
                  '["177-45-0950", "177-45-0950"]',
                  '["265-67-9973", "265-67-9973"]'
              ]
              },
              keysExamined: 6,
              seeks: 3,
              dupsTested: 0,
              dupsDropped: 0
          }
      }
  },

Understanding Explain

used to analyse the query
info provided by explain:
- indexes used
- indexes used to provide sort
- indexes used to provide projections
- how selective is the index
- which part of the plan is the most expensive
to create explainable object:
- exp = db.people.explain()
- exp.find()
- parameters to pass to explain func:
  - "queryPlanner" --default (won't run the query) ***
  - "executionStats" (runs and returns specific stats about the execution)
  - "allPlansExecution" (runs then gives all alternative plans)
in case of sharded cluster:
- each shard will use a winning plan
- then the plans are merged on the mongos
Note

the memory limit for execute in memory sort is 33554432 byte around 32MB and if your query exceed this limit then the sort will canceled

Sorting with Indexes

There are two ways to execute sorting:
1. In memory
2. Using index
In memory:
- all documents in collections are stored on disk in unknown order
- the server will return the documents in the order it find them
- all data will be stored in RAM
- then sorting algorithm is run
- max memory usage in sorting is 32MB if exceeded then the server will abort the operation
Using indexes:
- docs are ordered while saved in index using the index field
- the docs returned will be sorted by the index then no need for additional sorting
- direction will be forward if the sorting order is the same as the index and backward if not (asc - desc)

You can learn more about sorting with indexes by visiting the Use Indexes to Sort Query Results section of the MongoDB Manual.

Querying on Compound Indexes

Compound index is an index on two or more fields
Even Index is conmpound of more than one field it still on dimentional.
We can use compound index to query a range of values too.
The main key is the first key in the index then the subsequent keys are indexing within that main index and so on
- index {name: 1, age: 1}
- {name: "a", age: "1"}, {name: "a", age: "2"}
- {name: "b", age: "3"}, {name: "b", age: "8"}
Index Prefixes:
- Index prefixes are the beginning subsets of indexed fields.
- For example consider the following compound index {"item": 1, "location": 1, "stock": 1}
- The index has the followning index prefixes:
  - {item: 1}
  - {item: 1, location: 1}
- prefexis will use the index although it doesn't utilize the whole index the remaining will be ignored

You can learn more about compound indexes by visiting the Compound Indexes and Create Indexes to Support Your Queries sections of the MongoDB Manual.

When you can sort with compound Indexes ?

Sorting can utilize index if the filter fields combined with sort fields gives index prefix
- find({a: "", b: ""}).sort({c: 1}) will work for index {a: 1, b: 1, c: 1}
Using single prefix ---> both directions will utilize the index
else, won't use index unless all is in the same direction as the index or all are reversed

You can learn more about when you can sort with indexes by visiting the Use Indexes to Sort Query Results section of the MongoDB Manual.

Multikey Indexes

Indexing on a field that is an array is a Multikey Index
for each entry in the array, the server will create a separate index key.
it can also work in emedded docs inside the array
- arr: [{a: "", b: ""}, ...] ---> creatIndex({arr.a: 1})
For each indexed document, we can have at most one index field whose value is an array
only one array index field is allowed per compound index
Multikey indexes don't support covered queries

You can learn more about multikey indexes by visiting the Multikey Indexes section of the MongoDB Manual.

Partial Indexes

Index only a subset of the collection

  db.collection.creatIndex(
      {a: 1, b: 1},
      {partialFilterExpression: {c: {$gte: 5}}}
  )

the above will index docs with c >= 5

Can be used with multi-key indexes

Sparse index: Is a spectial case of partial index (only index if the fields exist)

  db.collection.creatIndex(
      {a: 1},
      {sparse: true}
  )

the above is equal to

db.collection.creatIndex(
    {a: 1},
    {partialFilterExpression: {a: {$exists: true}}}
)

In order to use a partial index, the filter query must be guarateed to match a subset of the documents, specified by the partialFilterExpression
Partial Index Restrections:
- You can't specify both the partialFilterExpression and the sparse options
- _id indexes can't be partial indexed
- Shard key index can't be partial index

You can learn more about partial indexes by visiting the Partial Indexes section of the MongoDB Manual.

Text Indexes

Querying docs based on words in a certain text field
createIndex({a: "text"})
db.collection.find({ $text: { $search: "..." } })
similar to multi-key indexes will split the text and create index key for each unique word in the string
Text indexes are case insensitive
Will be effected if the text is long
- More keys to examine
- Increased index size
- Increased time to build index
- Decreased write performance
One solution for long text index is to use compound index
text querys uses or between the delimited words
- "a b" will match a or b
textScore:
- is the relevance of the result to the text query
- find({}, {score: {$meta: "textScore"}})

Collations

Collations allows users to specify language specific rules for string comparison

{
    locale: string,   the ICU supported locale
    caseLevel: boolean,
    caseFirst: string,
    strength: int,
    numericOrder: boolean,
    alternate: string,
    maxVariable: string,
    backwards: boolean
}

Where to specify collation:
- In collection level db.createCollection("name", {collation: {...}})
- For a specific requests like queries and aggregations:
  - db.find().collation({...}) as a iterator method
  - db.aggregate([{}, {collation: {...}}]) as a stage
- For Indexes db.createIndex({}, {collation: {...}})
  - To using this index for quering the collation specified with the index must match the one specified with this key on collection level
Benifits of using Collations:
- correctness by matching certain locale
- it has a marginal performance impact
- allows the use of case-insensitive indexes by using strength: 1 which will ignore case and diacritics

You can learn more about collations by visiting the Collations section of the MongoDB Manual.

Wildcard Index Type

New in 4.2
Allows a dynamically create indexes on all fields or subset of fields
how it works:
- mongodb analysis the document and indexes all fields in the document
- for arrays, indexes for each value like multi-key
- for subdocs, indexes for each field in the sub document
db.<collection>.createIndex({'$**': 1})
it creates one virtual single field index at execution time
if filtering by multiple fields, create multiple plans using multiple single indexes then the query planner uses the score to determine which one to use.
To creating wild card index on sub docs only:
- db.<collection>.createIndex({"field.$**": 1}) only subdoc and fields in the subdoc will be indexed
Wild card projection option:
- specifies set of fields to include or execlude from the index
- db.<collection>.createIndex({'$**'}, {wildcardProjection: {a: 1}}) index only a and all its subpaths if it's a subdoc
- db.<collection>.createIndex({'$**'}, {wildcardProjection: {a: 0}}) index all but a
A Covered Query is a query that can be satisfied entirely using an index and does not have to examine any documents.
An index covers a query when all of the following apply:
- all the fields in the query are part of an index, and
- all the fields returned in the results are in the same index.
- no fields in the query are equal to null (i.e. {"field" : null} or {"field" : {$eq : null}} ).
wild card index can only cover a query if the query is on a single field
Used Cases Examples:
- unpredictable work loads
- queries with many fields
- arbitrary query patterns
- implementing the attribute pattern

Chapter 3: Index Operations

Building Indexes

Hyprid index build new in Mongodb 4.2
Types of index builds:
1. Foreground index build:
  - very performant
  - locks the entire db until the build is done
  - createIndex({a: 1})
  - prior to 4.2
2. Background index build:
  - doesn't lock the db
  - less performant
  - it uses incremental approach to create the index
  - the index resulting is less efficient than the foreground index
  - createIndex({a: 1}, {background: true})
  - prior to 4.2
3. Hyprid:
  - This is the only option available since 4.2
  - Using the best of both worlds:
    - No locks on the db like the Background
    - Performant like Forground

You can learn more about building indexes by visiting the Index Build Operations section of the MongoDB Manual.

Query Plans

Are a series of stages that feen into one another to run a query
For a single query, many plans can be proposed based on the indexes
Only one winning plan is used
How it works:
1. Fresh query comes to the db for the first time
2. The service looks for all indexes on the collection
3. Choose indexes viable for the query (candidate indexes)
4. The query optimizer uses candidate indexes to generate candidate plans
5. The quey planner is emperical query planner --> means all candidate plans will be tried over a short period of time then choose which plan performs best
6. Then mongodb caches the winning plan for that query shape
7. The plans will be removed from cache if:
  - The server is restarted
  - Index is rebuilt
  - Index is created or dropped
  - If threshold is reached, that is amount of work done by the first portion of the query exceeds the work done by the winning plan by a factor of 10

You can learn more about query plans by visiting the Query Plans section of the MongoDB Manual.

Forcing Indexes with Hint

Overriding default mongodb index selection with hint()
Sometimes query optimizer uses another index that we want it to use
db.collection.find().hint({ a: 1, b: 1 }) index shape
db.collection.find().hint("a_b_1") index name
Use it with caution

Resource Allocation for Indexes

Indexes help in:
- Optimizing queries
- Decreasing response time
db.stats() shows the index sizes
db.col.stats() shows the index sizes per collection
Disk
- Size isn't a big issue
- If no space is available, the index won't be created
- If we use a separate disk to store the indexes, insure it has enough disk space
Memory:
- we should have enough space to accomodate the indexes
- if not, then disk access is required to traverse index file and will slow down queries
- db.col.stats({indexDetails: true}) shows the index sizes per collection
Edge Cases:
- Occasional reports
  - Indexes used in operational workloads should be in Memory
  - Indexes used in BI tools do not have to be always in memory
  - Usuallly BI tools should only talk to a secondary not the primary node then their related indexes should also be created only on those nodes
- Right-end-side index incerements
  - Indexes on fields that grow monotonically ex: counts, dates, incremental ids should not always be in Memory
  - The B-Tree would be unbalanced and tend to grow to the right hand side
  - Only the right hand side is needed to be in memory (new added data)

Basic Benchmarking

public test suite
private testing environment
Types of Perfromance Benchmarking:
1. Low Level Benchmarking:
  - File I/O Performance
  - Scheduler Performance
  - Memory allocation and transfer speed
  - Database server performance
  - Thread performance
  - ... Tools used:
    - sysbench
    - iibench
2. Database Server Benchmarking:
  - Data set load
  - Writes per second
  - Reads per second
  - Balanced workloads
  - Read / Write ratio Tools:
    - YCSB
    - TPC
3. Distributed Systems Benchmarking:
  - Linearization of reads and writes
  - Serialization of requests
  - Fault tolerance if node fails tools:
    - highbench
    - jepsen
Benchmarking Conditions:
- POCDriver tool to test mongodb workloads
- benchmarking anti-patterns:
  - Database swap replace (turning tables in sql dbs to collections)
  - Using mongo shell for write and reads requests
  - Using mongoimport to test writes response
  - Local laptop to run tests
  - Using default mongodb parameters

Chapter 4: CRUD Optimization

Optimizing your CRUD Operations

Index selectivity ---> minimizing index keys scanned
- Range queries aren't very selective
- Equality is very selective
- createIndex({the more selective, the less selective, ...})
- To use index for both sorting and filtering, the query predicate should be all equalities and no range
- find({a: "", b: {$gt: 1}}).sort(c: 1)
  - an index with shape {a: 1, b: 1, c: 1} won't be use in sorting that's because the prefix used to reach the sort will be [a, b, c] and b is range, but index with shape {a: 1, c: 1, b: 1} will be used in sorting because the prefix used to reach sort is [a, c] which has equality only
Equality, Sort, Range rule: best way to define an index would be: {equality_condition, sort_conditions, range_conditions}

You can learn more about optimizing your CRUD operations by visiting the Create Indexes to Support Your Queries, Use Indexes to Sort Query Results, and Create Queries that Ensure Selectivity sections of the MongoDB Manual.

Covered Queries

What are covered queries?
- Very performant way to service the queries to our database
- Satisfied entirely by index
- 0 docs needs to be examined
A covered query has:
- all index fields are the only fields in the predicate
- projection to return only fields from index fields and the fields must be stated explicitly
- no fields in the query are equal to null (i.e. {"field" : null} or {"field" : {$eq : null}} ).
- No fetch stage is present in the query plan
You can't cover a query if:
- Any of the indexed fields are arrays
- Any of the indexed fields are embedded docs
- When run against a mongos if the index does not contain the shard key

You can learn more about covered queries by visiting the Query Optimization section of the MongoDB Manual.

Regex Performance

Creating index on fields that we want to run regex on increases performance but we still have to run the regex against all the keys in the index
to address this issue try to use /^regex/ to only examine a subset of the keys
if we use it like this /^.regex/, then it has no effect

Aggregation Performance

Realtime Processing:
- Providing data to applications
- Performance is important
Batch Processing:
- Providing data for analytics
- Performance is less important
Index usage:
- if in a certain stage the index can't be used, then it won't be used in the following stages
- pass { explain: true } as option to view the steps of execution
- operators that use index should be at the start of the pipline
- match, sort, limit should be in front
Memory constraints:
- Results are subject to 16MB doc size limit applies
  - Use $limit and $projection to reduce the results size
- 100MB Ram per stage
  - Use indexes
  - We can use disk db.orders.aggregate([...], {allowDiskUse: true})
  - allowDiskUse less performant
  - graphLookup doesn't support allowDiskUse as it doesn't support spilling to disk

Chapter 5: Performance on Clusters

Performance Considerations in Distributed Systems

Distributed Systems are Replicaset Cluster and Shard Cluster.
Consider latency
Data is spread across different nodes
Read implications
Write implications
In Replication:
- Offloading eventual consistency data to secondaries
- specific work load to target indexes on secondaries like BI
In Sharding:
- Shard nodes must be themselves replicasets
- Sharding is for horizontal scaling
- You should reach vertical scaling limit before sharding
- You need to understand how data grows and how your data is accessed to determine a good shard key
- Sharding works by defining key based ranges - our shard key
- It's important to get a good shard key
- Latency between different cluster elements
- Putting mongos on the same server as the application server should reduce latency
- In MongoDB there are two types of read we can perform in a shard cluster:
  - Scatter Gather: Where we ping all nodes of our shard cluster for the information corresponding to a given query
  - Routed Queries: Where we ask one signle shard node or a small amount of shard nodes for the data that your application is requesting
  - If we don't use shard key will use Scatter Gather
  - Routed queries are more performant than scatter gathered queries
- Sorting limit and skip is performed locally on each shard then merged on the primary shard
Note:

In MongoDB 4.2, we can use any of the shards (or mongos) to do final sort, limit and skip steps. For more details, you can refer to How mongos Handles Query Modifiers section in the documentation.

You can learn more about distributed system performance considerations by visiting the Distributed Queries section of the MongoDB Manual.

Increasing Write Performance with Sharding

Picking a good shard key
With a shard key our data are divided up into bite size piece called chunks
Each chunk has an inclusive lower bound and exclusive upper bound
By default a Chunk max size is 64MB
We need to insure that the chunks are distributed evenly across shards
Shard Key Factors:
- Cardinality:
  - The number of distinct values for a given shard key
  - High Cardinality is good
  - Cardinality determines the max number of chunks that can exist in our cluster
  - Using Compound shard key insure increase the cardinality
- Frequency:
  - The number of occurance of the same values in our cluster
  - High frequency will limit the equal distribution of chuncks which is bad
  - When the frequency is high then the throughput of our application would be constrained by the shard contains this repeated values and we defined this as a Hot Shard
  - Typically, when a chunck is close to its max size, Mongo will split it into two chunks
  - Jumbo Chunk is a chunk with the same lower and upper bound and it will be no longer eligible for spliting and this will reduce the effectiveness of horizontal scaling because we won't be able to move these chuncks between shards
  - We can mitigate the issue of uneven frequency if we create a good compound shard key
- Rate of Change:
  - How our values change over time.
  - Avoid monotonically increasing or decreasing values in our shard key.
  - The examble of this is ObjectID
  - When using monotonically increasing shard key, all of our writes are going to the same shard, This is the shard that contains the upper bound of max key, which is often refered to as the last shard.
  - When using monotonically decreasing shard key, all of our writes are going to the same shard, This is the shard that contains the lower bound of min key, which is often refered to as the first shard.
  - Monotonically changing shard keys should be avoided unless they are used in compound keys and shouldn't be the first field using them this way increases the cardinality
High cardinality, low frequency, low rate of change for shard key
Compound keys have high cardinality and low frequency
Bulk Write:
- ```
  db.collection.bulkWrite(
      [<operation1>, <operation2>, ...],
      {ordered: <boolean>}
  )
```
- Ordered bulkwrites are less performant on sharded cluster because the server will execute thesd operations one after another waiting for the last response to succeed, if an operation fails we immediately stop the bulk insertion and report back to the client
- Unordered bulkwrites can utilize parallization in sharded cluster, as the server will execute all these operations in parallel

You can learn more about increasing write performance with sharding by visiting the Distributed Write Operations and Bulk Write Operations sections of the MongoDB Manual.

Note: A monotonically increasing or decreasing values in our shard key is not desired for a heavy write workload, but may be fine for a heavy read workload.

Reading from Secondaries

Read preference by default is set to primary node
db.find().readPref("primary")
There are several other read preferences available:
- Primary db.find().readPref("primary") all reads will routed to primary node
- Primary Prefereddb.find().readPref("primaryPrefered")
- Secondary db.find().readPref("secondary") all reads will be routed to one of the secondaries, write however can only be routed to primary node
- Secondary Prefereddb.find().readPref("secondaryPrefered") reads will always be routed to a secondary unless there aren't any available in which case will be routed to primary
- Nearest db.find().readPref("nearest") will read from node with the lowest network latency
Note that when we read data from secondary node there is a possibility to read stale data
When reading from a Secondary node is a Good idea:
- Analytics queries
  - Are generally resource intensive and long running
- Local read
  - in geo distributed clusters for low latency

You can learn more about reading from secondaries by visiting the Read Preference section of the MongoDB Manual.

Note As of MongoDB 3.6 you can read from secondaries on sharded clusters safely.

Replica Sets with Differing Indexes

Use secondary with differing indexes for:
- Analytics
- Reporting delayed consistency data
- Text search
Secondary Node Considerations: it should be
- Prevent such a secondary from becoming primary
  - Priority = 0
  - Hidden node
  - Delayed secondary
- They should not be allowed to be primary as they may not have the indexes to handle application queries
Steps to create index on secondary only:
1. shut down the server
2. bring up the secondary in stand alone mode
3. create the index
4. bring up the replicaset again

Aggregation Pipeline on a Sharded Cluster

If match stage uses shard key
- all the pipeline will be routed to that shard
- the results will be returned to mongos
No match
- some stages will be splitted on each shard
- then the results will be merged together on a single shard
- it normally happens on random shard unless ($out, $facet, $lookup, $graphLookup) are used
- in those cased the primary shard will do the merging

Aggregation Optimizations

if a sort is followed by match, the query optimizer will move the match above to limit the docs sorted
if skip is followed by limit, the query optimizer will move the limit up and change the values of each stage to result in correct data ----> {skip: 10, limit: 5} will be coalesce { limit: 5, skip: 10 }
some stages can be combined together ---> {limit: 5}, {limt: 10} ---> {limit: 15} or skip or match if possible

You can learn more about aggregation in a sharded cluster by visiting the Aggregation Pipeline and Sharded Collections section of the MongoDB Manual.

M320: Data Modeling

Learn everything you need to know about data modeling for MongoDB.

Chapter 1: Introduction to Data Modeling

Data Modeling in MongoDB

One of the most misconception about mongodb is that modeling is Schemaless means that it doesn't really matter which field documents have or how different the documents can be from one another, or how many collection you might have per database.
Even that mongodb give you this flexibility, this is not practical in reality.
MongoDB has very flexible data model.
But most importantly all data as some sort of structure and there for a Schema.
MongoDB just happens to make it easier for you to deal with that later rather than sooner.
Before building ERD or UML it tends to be preferable to start building your application and finding out from that particular experience what the data structure should look like.
However if you do know: - Usage pattern. - How your data is accessed. - Which queries are critical to your application, - Ratios between reads and writes you will be able to extract a good model, even before rewriting the full application to make it scale with mongodb.
Being flexible means that your application changes. and it's not unreasonable to think that that will not be the case.
With mongo you will be able to accomodate those changes without experiencing a painful migration process like in traditional relational databases.
When have a good idea about the structure of your documents you will be able to enforce those rules in MongoDB by using Document Validation
Another misconception is that all information regardless of how data should be manipilated can be stored in one single document, but the reality is that this is not the way applications in general use data.
Keep the amount of the iformation stored per individual document to the data that your application uses and having different models to deal with historical data or other types of data that are not always accessed
We can berform data joining process using $lookup

The Document Model in MongoDB

Database -> Collection -> Document
Document stored in BSON
Instead of but your related data in multible table you can put the related data nestead in the same document and put it all down in a single query
You can have multiple version of your document schema and they can be coexists in the same collection

You can read more about Document Structure and BSON Data Types

Constraints in Data Modeling

Hardware:
- RAM
- SSD, HDD
Data:
- Size
- Security, sovereignty
Application:
- Network latency
DB Server:
- MongoDB has limitaion on doc size 16MB

Working Set: Is the data that the Application uses in normal operations
Tips:
- Keep the frequently used documents in RAM
- Keep the indexes in RAM
- Prefer SSD to HDD
- Infrequently accessed data in hdd

To know more about Transactions with MongoDB, please consult the MongoDB Documentation on Transactions and some videos explaining their implementation .

The Data Modeling Methodology

Phases:

Describing workloads:
- Requirement document
- Business domain analysis
- Production logs and stats if migrating existing db
Indentifying relationships between entities:
- pieces of info that can be grouped together (entity)
- Indentifying the relationships between entities
- decide either keeping as embedded doc or in a new collection
Applying design patterns to address performance requirements

Model for Simplicity or Performance

The main tradoff you will face is Simplicity vs Performance or try to find the balance between them

Modeling for Simplicity: - limitied expectations - low resources requiremens cpu ram disk - fewer collections and embedding documents - less disk access
Modeling for Performance: - need more resources - Sharding - fast read, writes - larger teams needed - must adhere to the phases of methodology
Modeling for a Mix of Simplicity and Performance
Summary of Modeling Approaches:

Chapter 2: Relationships

Introduction to Relationships

Even MongoDB is classified as non-relationa database but the pieces of data inside will have relations.
Mainly these relations is done by embeding or referencing.
The relationships represent all the entities are related to each other.
For example:
- customer and customer_id is One-to-One relationship
- Customer and invoices is One-to-Many relationship
- Invoices and Products is Many-to-Many relationship

Relationship Types and Cardinality

Cardinalities:
- one-to-one (1-1) grouped in the same doc
- one-to-many (1-N)
- many-to-many (N-N)
one-to-many or many-to-many:
- it depends on how much likely the many will be
- in father - children situation, it's likely to be max of 5 or 10
- in follower - following situation, it may be 10 or 10,000
one-to-zilions is useful in the Big Data world
Numerical Notation: [min, likely, max]
- minimum
- most likely
- maximum

One-to-Many Relationship

For exmaple: Person and credit cards
Solutions:
1. Embedded doc:
  - all in one collection
  - usually, embedding in the entity the most queried
  - embed the many in the one side document
    - most common
    - will create a multi-key index
  - embed the one in the many side documents (like shipping_address in orders)
    - less often used
    - useful if many is queried more often than one
    - embedded object is duplicated
2. Reference:
  - a collection for each
  - usually, referencing in the many side
  - refer to many in the one side document like (stores in zips)
    - Array of references
    - Allows large number of documents
    - List of references available when retrieving the main object
    - Cascade deletes are not handeled by mongo db and must be done by application logic
  - refer to one in many side documents
    - preferred in references
    - no need to manage references in the one side

Many-to-Many Relationship

Many documents in the first side associated with many document in the second side and vice versa like (stores and products)
In tradetional relational databases you will had to create new table to manage the relation
MongoDB Implementations:
1. Embedding
  - Array of sub docs in the many side
  - Array of sub docs in the other many side
  - Usually the most queried is the main consideration
    - Embed the documents from the least queried side in the most queried
    - Results in duplication
    - Keep the source of the embedded documents in another collection
    - Indexing is done on the Array
2. Referencing
  - Array of references in each many side
  - References readily available upon first query on the main collection

One-to-One Relationship

Like user and his email, name, phone

Embedded in the same document
- fields in the doc at the same level
- grouped in sub document
Referencing
- For example store and store_details
- use identifier in either documents
- add complexity
- Possible performance improvements with:
  - smaller disk access
  - smaller amount of RAM needed

One-to-Zillions Relationship

Means one to something huge like 100 millions
It is a special case of one-to-many relationship
can't use Embedding as it won't be performant
can't reference the many in the one side as it won't be performant
Reference the one in the many side

Chapter 3: Patterns

Introduction to Patterns

Patterns are not the full solution of the problem.
Patterns are smaller sections of those solutions, they are reusable units of knowledge.
Patterns are like software design patterns but for Data Modeling and Schema Design.

Handling Duplication, Staleness and Integrity

Applying Pattern may lead to...

Duplication of data across documents
Data staleness in some pieces of data
Data integrity issues:
- may have to write application side logic to ensure referential integrity

Note If these three concerns are more important than the potential simpicity of performance gains provided by the pattern, you should not use the pattern.

Duplications:
- Why?
  - Results of embedding info in a given document for faster access for example: embedding the customer address in the shipment info document so it can't be changed after the order is shipped if user changed his address.
- Concern??
  - Represents challenges to insure correctness and consistency.
- Situations:
  - Duplication is the solution: like embedding the shipping_address in the document so if the user change the address that didn't affect the order.
  - Duplication has minimal effect: like embedding actors in the movie document as they won't change once the movie released
  - Duplication should be handled: duplication of a piece of info that may change with time for example the revenuse for the given movie
Staleness:
- Why?
  - New events come along at such a rate that updating some data constantly that updating can cause performance issues.
- Concern??
  - Challenges in insuring data quality and reliability
- Situations:
  - Batch update
  - Change streams
- To know more about change streams, please consult the MongoDB documentation on change streams.
Referential integrity:
- Why?
  - Linking info between docs
  - Not supporting cascading deletes or foreign keys
- Concern??
  - challenges in data quality
- Situations:
  - Using Change Streams
  - Embedding data in single document
  - Using Multi-Documents transactions

Attribute Pattern

Having different types with different needed information represented in the same collection for example a collection of products as every product has specific data and specification that may not exists in other products like the size property is measured in ml in drinks and in cm in charger. and another field that don't exists in another products
Orthogonal Pattern to Polymorphism

Steps:

The charactristics that are almost always present (common schema parts), represented as fields in the document
may need a lot of indexes

for the special attributes :

for each field, create a key value pair inside a new array field ex:

{ 
  "manufacturer":"China",
  "brand": "MongoDB",
  "sub_brand": "University",
  "price":0.0,
  ...
  "color":"black",
  "size": "100x70x10mm",
  ...
  "input": "5v/1300 mA",
  "output": "5v/1A",
  "capacity": "4200 mAh"
}

will be like this:

{ 
  "manufacturer":"China",
  "brand": "MongoDB",
  "sub_brand": "University",
  "price":0.0,
  ...
  "color":"black",
  "size": "100x70x10mm",
  ...
  "add_specs": [
  	{"k": "input", "v": "5v/1300 mA"},
  	{"k": "output", "v": "5v/1A"},
  	{"k": "capacity", "v": 4200, "u": "mAh"},
  ]
}

Use cases Examples:

Fields that share common characteristics in the same document. for example:

 {
 	"title": "Dunkrik",
 	...
 	"release_USA": "2017/07/23",
 	"release_Mexico": "2017/08/01",
 	"release_France": "2017/0/01",
 	"release_Festival_San_Jose": "2017/07/22",
 }

will be

 {
 	"title": "Dunkrik",
 	...
 	"releases":[
 		{"k": "release_USA", "v": "2017/07/23"},
 		{"k": "release_Mexico", "v": "2017/08/01"},
 		{"k": "release_France", "v": "2017/08/01"},
 		{"k": "release_Festival_San_Jose", "v": "2017/07/22"},
 	]
 }

so now we can run this query easily: db.movies.find({"releases.v": {$gte: "2017/07", $lt: "2017/08"}})

Want to search across many fields once
Fields that present only in a small subset of docs
You can create index on the key and value pairs to optimize searching

Benefits:
- Easier to index
- Allow non-deterministic field names
- Ability to qualify the relationship of the original field and value

Note With the release of the Wildcard Index functionality in MongoDB 4.2, some use cases of the Attribute Pattern can be replaced by this new index type.

Extended Reference Pattern

Problem: avoiding joining data at query time

Solution: Embed fields on the lookup side in the docs in the from collection

Use cases:

Catalog
Mobile apps
Real-Time Analytics

Pros:

faster reads
reduce the number of joins and lookups

Cons:

Duplication:
1. Minimize it: a. Select fields that don't change often b. Bring only the fields you need to avoid joins
2. After a source is updated: a. What are the extended references to changed b. When should the extended references be updated
3. Duplication may be better than a unique reference

Subset Pattern

Problem:

Working set is too big, bigger than ram
solution:
- add ram
- scale with sharding
- reduce the size of the working set (the pattern)

Solution:

divide the document into two collections
- one that is frequently accessed data
- one with the remaining data

Use cases:

list of reviews of a product
list of comments on an article
list of actors in a movie

Pros:

smaller working set
shorter disk accesse

Cons:

duplication
more round trips to server

Computed Pattern

Problem:

Costly computation or manipulation of data like:
- Mathematical Operations
- Fan Out Operations
- Roll-up Operations
Executed frequently on the same data produces the same result

Solution:

Perform operations and store the result in the appropriate doc and collection
if needed to redo the operations, keep the source

Use cases:

IOT
Event Sourcing
Time Series Data
Frequent Aggregatoin Framework queries

Pros:

Overuse of resources (CPU)
Reduce latency for read operations

Cons:

Avoid applying or overusing it unless needed
May be difficult to identify the need

Bucket Pattern

Problem:

Avoiding too many docs
Avoiding too big docs
A 1-to-Many relationship that can't be embedded

Solution:

Define the optimal amount of data to group together
Create arrays to store the information in the main object
It is basically an embedded 1-to-Many relationship, where you get N documents each having an average of many/N sub documents

Use cases:

IOT
Data Warehousing
Lots of info associated with one object

Pros:

Good balance in the number of acess and size of data
Makes data more manageable
Easy to prune data

Cons:

Can lead to poor query results if not designed correctly
Less friendly to bi tools
Random insertions or deletios in buckets
Difficult to sort across buckets
Ad hoc queries may be more complex, again across buckets
Works best when the "complexity" is hidden through the application code

Schema Versioning Pattern

Updating a Relational Database Schema:

Need time to update the Data
Usually done by stopping the Application
Hard to revert if something goes wrong

Application Lifecycle:

Modify Application
- Can read/process all versions of documents
  - Have different handler per version
  - Reshape the document before processing it
Update all Application servers
- Install updated application
- Remove old processes
Once migration completed
- remove the old code to process old versions

Document Lifecycle:

New Documents:
- Application write them in latest version
Existing Documents
- A Use updates to documents
  - to transform to latest version
  - keep forever documents that never need an update
- B ... or transform all documents in batch
  - no worry even if process takes days

Problem:

Avoiding downtime while schema upgrades
Upgrading all documents can take hours, days or even weeks when dealing with big data
Don't want to update all documents

Solution:

Each document get a "schema_version" field
Modify the application to handle all versions
Choose your strategy to migrate the documents

Use cases:

Every appication that use a database deployed in production env and heavily used
System with a lot of legacy data

Pros:

No downtime needed
Feel in control of the migration
Less future technical debt

Cons:

May need 2 indexes for the same field during migration period

Tree Patterns

Problem:

How to model heirarical structures information

Use Cases:

Company organization charts
Subject areas structures in a given domain like books
Categories of products for a given e-commerce site or shop.

Hierarchical nodes relationship's common operations:

Who are the ancestors of node X?
Who reports to Y?
Find all nodes that are under Z?
Change all categories under N to under P

Patterns to Model Tree Structures:

Parent References:
- The document holds a reference to the parent node.
- Parents references are prefer to perform operations like:
  - Who reports to Y?
  - Change all categories under N to under P
- We can collect all ancestors by running an aggregation pipeline with a $graphlookup stage to retrieve all subsequent parents of the immediate parents traversing the full tree
```
// Who are the ancestors of node X?
// all ancestors
db.categories.aggregate([
    {$graphLookup: {
        from: 'categories',
        startWith: '$name',
        connectFromField: 'parent',
        connectToField: 'name',
        as: 'ancestors'
    }}
])
```
- To find all reports of a given parent, we can run a find command matching for the parent and then retrieving all children nodes.
```
// Who reports to Y?
// immediate ancestor
db.categories.find({parent: 'Y'})
```
- In order to change all nodes that report to or are children of one parent in other words change all categories under N to be under P we can use an update operation
```
// Change all categories under N to under P
// all ancestors
db.categories.updateMany(
    {parent: N},
    {$set: {parent: P}}
)
```
Child References
- The parent contains a single array off all the immediate childs
```
{
    name: "office",
    children: ["Books", "Electronics", "Stickers"]
    ...
}
```
- To perform the operation finding all nodes that are under Z we use a single request to retrieves all of that information
- However, other questions like, who are the ancestors of X become a bit more complicated.
- Finding all nodes that reports to Y or even changing all nodes under N to under P are not ideal for this pattern
Array of Ancestors:
- This model uses an ordered array to store a list of all of a node's ancestores on that node
```
{
    name: "Books",
    ancestores: ["Swag", "Office"]
}
```
- This model is very efficient for finding :
  - Who are the ancestors of node X?
  - Who reports to Y?
  - Find all nodes that are under Z?
Materialized Paths:
- In this model we use a string value to describe the node's ancestors with value separtor
```
{
    name: "Books",
    ancestores: ".Swag.Office"
}
```
- We can use a single regular expression over a single index field value for all queries prepended on the root tree node
```
// immedate ancestor of Y
db.categories.find({ancestors: /\.Y$/})

// if descends from X and Z
db.categories.find({ancestors: /^\.X.*Y/i})
```
- This mode is effeciant to answer this question:
  - Who are the ancestors of node X?

Note We can use any compination of different patterns in our mode for example:
- here is the categories collection where we using both Array of Ancestors and Parent Reference models
```
{
    _id: 8,
    "name": "Umbrellas",
    "parent": "Fashion",
    "ancestors": ["Swag","Fashion"]
}
```

Use Cases:

Org charts
Product Categories

Pros:

Child Reference: easy to navigate to a child node or tree descending access patterns
Parent Reference: Immediate parent node discovery and tree updates
Array of Ancestors: Navigate upwards on the ancestors path
Materialized Path: Makes use of regular expression to find nodes in the tree

Note You can find more information by going to documentation page of Model Tree Structures

Polymorphic Pattern

We should put things together if we need to query them together
Car, Truck and Boat are polymorphic object as they have some similarity with some differences.

The usual implementation that represents polymorphic objects as a field that describes the name of this shape

// Car
{
    "vehicle_type": "car",
    ...
    "owner": "x",
    "taxes": "100",
    "wheels": 4
}
// Truck
{
    "vehicle_type": "truck",
    ...
    "owner": "y",
    "taxes": "800",
    "wheels": 10,
    "axles": 3
}
// Boat
{
    "vehicle_type": "boat",
    ...
    "owner": "z",
    "taxes": "2000",
}

We can apply polymorphic pattern on Subdocuments
We usualy using Polymorphic pattern to allow single view solution
Polymorphism in the Schema Versioning Pattern

Problem:

Objects more similar than different
Want to keep objects in the same collection

Solution:

Field tracks the type of document or sub-document
Applicaion has different code paths per type, or has subclasses

Use Cases:

Single View
Product Catalog
Content Management

Pros:

Easier to implement
Allow query across a single collection

Other Patterns

Approximation Pattern:

Used to reduce resources needed to perform some write operations.
This Pattern using approximation function to produce the result
Problem:
- Data is expensive to calculate
- it does not matter if the number is not precise
Solution
- Fewer writes with higher payload
Use Cases:
- Web page counters
- Any counters with impercision tolerance
- Metric statistics
Pros:
- Less writes
- Less contention on docs
- Statistically valid numbers
Cons:
- Not exact numbers
- Must be implemented in the application

Outlier pattern:

Problem:
- Few documents would drive the solution
- Impact would be negative on the majority of queries
Solution
- implementation that works with the majority
- Add field that indetify the Outliers
- Outliers are handeled differently on the application side
Use Cases:
- Social Networks
- Popularity
Pros:
- Optimized for most use cases
Cons:
- Differnces handled application side
- Difficult for aggregation or ad hoc queries

Name		Name	Last commit message	Last commit date
Assets		Assets
.DS_Store		.DS_Store
1-Basics-M001-Notes.md		1-Basics-M001-Notes.md
2-Basic-cluster-administrations-M103-Notes.md		2-Basic-cluster-administrations-M103-Notes.md
3- The MongoDB Aggregation Framework: M121.md		3- The MongoDB Aggregation Framework: M121.md
4- MongoDB Performance- M201.md		4- MongoDB Performance- M201.md
5- Data Modeling - M320.md		5- Data Modeling - M320.md
README.md		README.md

Latest commit

History

Folders and files

Repository files navigation

Table of Content

M001 - MongoDB Basics

Chapter 1: What is MongoDB?

Chapter 2: Importing, Exporting and Querying Data

JSON vs BSON

Importing & exporting

Data Explorer

Queries

Chapter 3: Creating and Manipulating Documents

Updating Documents using MQL- mongo shell

Deleting Documents and Collections

Chapter 4: Advanced CRUD Operations

Comparison Operators

Logical Operators

Expressive Query Operator

Array Operators

Projection

Examples:

Query a Sub-Document

Chapter 5: Indexing and Aggregation Pipeline

Aggregation Framework

sort() and limit()

Introduction to Indexes

Introduction to Data Modeling

Upsert - Update or Insert?

M103: Basic Cluster Administration

Chapter 1: The Mongod

The Mongod Introduction

Configuration File

command line options and their respective config commands

File Structure (WiredTiger)

Basic Commands

Logging Basics

Log verbosity Levels

Severity levels:

Log Message sections:

Profiling the Database

Basic Security

Authentication Mechanisms:

Authorization: Role Based Access Control

Roles

Server Tools Overview

Chapter 2: Replication

What is Replication?

MongoDB Replica Set

Setting Up a Replica Set

Replication Configuration Document

Replication Commands

Local DB

Reconfiguring a Running Replica Set

Reads and Writes on a Replica Set

Failover and Elections

Write Concerns

Read Concerns

Chapter 3: Sharding

What is Sharding?

Sharding Architecture

Setting Up a Sharded Cluster

Config DB

Shard Keys

Picking a Good Shard Key:

Hashed shard key:

Chunks

Balancing

Queries in a Sharded Cluster

Targeted Queries vs Scatter Gather

M201: MongoDB Performance

Chapter 1: Introduction

Hardware Considerations & Configurations

Let's discuss how mongodb uses hardware resources:

1. Memory

2. CPU

3. I/O

Chapter 2: MongoDB Indexes

Introduction to Indexes

How Data is Stored on Disk