Working With MongoDB & Python Using PyMongo and Pandas

Working With MongoDB & Python Using PyMongo and Pandas

In this blog, we will connect MongoDB with Python with help of PyMongo. Then we will create a database, collection and documents. After that we will documents in Pandas's dataframe.

Prerequisites

  1. Install Python and pip
  2. Install Jupyter notebook
  3. Install MongoDB community server

What is MongoDB?

MongoDB is a source-available cross-platform document-oriented database program. Classified as a NoSQL database program, MongoDB uses JSON-like documents with optional schemas. MongoDB is developed by MongoDB Inc. and licensed under the Server Side Public License (SSPL).

What is PyMongo?

PyMongo is a Python distribution containing tools for working with MongoDB, and is the recommended way to work with MongoDB from Python. The pymongo package is a native Python driver for MongoDB.

Install PyMongo

PyMongo can be installed with pip:

pip install pymongo

conda install

conda install -c anaconda pymongo

Connect to MongoDB

In [16]:
import pymongo
In [17]:
from pymongo import MongoClient

class pymongo.mongo_client.MongoClient(host='localhost', port=27017, document_class=dict, tz_aware=False, connect=True, **kwargs)

Client for a MongoDB instance, a replica set, or a set of mongoses. Means tools for connecting to MongoDB.
In [18]:
client = MongoClient()
client
Out[18]:
MongoClient(host=['localhost:27017'], document_class=dict, tz_aware=False, connect=True)

MongoClient with optional parameters.

In [19]:
client = MongoClient("mongodb://localhost:27017")
client
Out[19]:
MongoClient(host=['localhost:27017'], document_class=dict, tz_aware=False, connect=True)

Show databases

In [20]:
print(client.list_database_names())

['admin', 'config', 'local']

Create a new database

I am creating a new database called "sampledb". You can give database name according to your choice.

In [21]:
db = client.sampledb
db
Out[21]:
Database(MongoClient(host=['localhost:27017'], document_class=dict, tz_aware=False, connect=True), 'sampledb')

View list of database again after creating a new database

In [22]:
print(client.list_database_names())

['admin', 'config', 'local']

If the database doesn’t exist, then MongoDB creates it for you, but only when we perform the first operation on the database.

Create a collection

In [23]:
student_collection = db["students"]
student_collection
Out[23]:
Collection(Database(MongoClient(host=['localhost:27017'], document_class=dict, tz_aware=False, connect=True), 'sampledb'), 'students')

Insert document in the collection

MongoDB generates the ObjectId dynamically, so no need to add id.

In [24]:
student1 = { "name": "John", "age": 10, "class": "VI", "section": "A" }
In [25]:
result = student_collection.insert_one(student1)
result
Out[25]:
<pymongo.results.InsertOneResult at 0x2223a301700>
In [26]:
print(f"Inserted new record id: {result.inserted_id}")

Inserted new record id: 623df5c9a2f230bd4ff5fe28

Get the inserted record/document using find()

find(filter=None, projection=None, skip=0, limit=0, no_cursor_timeout=False, cursor_type=CursorType.NON_TAILABLE, sort=None, allow_partial_results=False, oplog_replay=False, batch_size=0, collation=None, hint=None, max_scan=None, max_time_ms=None, max=None, min=None, return_key=False, show_record_id=False, snapshot=False, comment=None, session=None, allow_disk_use=None)

Query the database.

The filter argument is a prototype document that all results must match.
In [27]:
for student in student_collection.find():
    print(student) 

{'_id': ObjectId('623df5c9a2f230bd4ff5fe28'), 'name': 'John', 'age': 10, 'class': 'VI', 'section': 'A'}

Insert multiple document

insert_many(documents, ordered=True, bypass_document_validation=False, session=None):

Insert an iterable of documents.

Create the list of students.

In [28]:
students = [
    { "name": "Maria", "age": 9, "class": "VI", "section": "B"},
    { "name": "Michel", "age": 11, "class": "VII", "section": "A"},
    { "name": "Priyanka", "age": 8, "class": "IV", "section": "B"},
    { "name": "Jeena", "age": 12, "class": "X", "section": "A" }
]
In [29]:
result = student_collection.insert_many(students)
result
Out[29]:
<pymongo.results.InsertManyResult at 0x2223a301b80>

View the all inserted records

In [30]:
for student in student_collection.find():
    print(student) 

{'_id': ObjectId('623df5c9a2f230bd4ff5fe28'), 'name': 'John', 'age': 10, 'class': 'VI', 'section': 'A'}
{'_id': ObjectId('623df5cca2f230bd4ff5fe29'), 'name': 'Maria', 'age': 9, 'class': 'VI', 'section': 'B'}
{'_id': ObjectId('623df5cca2f230bd4ff5fe2a'), 'name': 'Michel', 'age': 11, 'class': 'VII', 'section': 'A'}
{'_id': ObjectId('623df5cca2f230bd4ff5fe2b'), 'name': 'Priyanka', 'age': 8, 'class': 'IV', 'section': 'B'}
{'_id': ObjectId('623df5cca2f230bd4ff5fe2c'), 'name': 'Jeena', 'age': 12, 'class': 'X', 'section': 'A'}

Find the first document in the student collection:

find_one(filter=None, *args, **kwargs)

Get a single document from the database.
In [31]:
student = student_collection.find_one()
student
Out[31]:
{'_id': ObjectId('623df5c9a2f230bd4ff5fe28'),
 'name': 'John',
 'age': 10,
 'class': 'VI',
 'section': 'A'}

Convert collection into Pandas dataframe

In [32]:
import pandas as pd
In [33]:
students = student_collection.find()
students
Out[33]:
<pymongo.cursor.Cursor at 0x2223a2fd040>

Convert students Cursor to list

In [34]:
list_students = list(students)
list_students
Out[34]:
[{'_id': ObjectId('623df5c9a2f230bd4ff5fe28'),
  'name': 'John',
  'age': 10,
  'class': 'VI',
  'section': 'A'},
 {'_id': ObjectId('623df5cca2f230bd4ff5fe29'),
  'name': 'Maria',
  'age': 9,
  'class': 'VI',
  'section': 'B'},
 {'_id': ObjectId('623df5cca2f230bd4ff5fe2a'),
  'name': 'Michel',
  'age': 11,
  'class': 'VII',
  'section': 'A'},
 {'_id': ObjectId('623df5cca2f230bd4ff5fe2b'),
  'name': 'Priyanka',
  'age': 8,
  'class': 'IV',
  'section': 'B'},
 {'_id': ObjectId('623df5cca2f230bd4ff5fe2c'),
  'name': 'Jeena',
  'age': 12,
  'class': 'X',
  'section': 'A'}]

Convert list to the Pandas Dataframe

In [35]:
df = pd.DataFrame(list_students)
df
Out[35]:
_id name age class section
0 623df5c9a2f230bd4ff5fe28 John 10 VI A
1 623df5cca2f230bd4ff5fe29 Maria 9 VI B
2 623df5cca2f230bd4ff5fe2a Michel 11 VII A
3 623df5cca2f230bd4ff5fe2b Priyanka 8 IV B
4 623df5cca2f230bd4ff5fe2c Jeena 12 X A

Convert collection with query into Pandas dataframe

In [36]:
students1 = student_collection.find({ "class": "VI" })
students1
Out[36]:
<pymongo.cursor.Cursor at 0x2223c6625e0>
In [37]:
list_students1 = list(students1)
list_students1
Out[37]:
[{'_id': ObjectId('623df5c9a2f230bd4ff5fe28'),
  'name': 'John',
  'age': 10,
  'class': 'VI',
  'section': 'A'},
 {'_id': ObjectId('623df5cca2f230bd4ff5fe29'),
  'name': 'Maria',
  'age': 9,
  'class': 'VI',
  'section': 'B'}]
In [39]:
df1 = pd.DataFrame(list_students1)
df1
Out[39]:
_id name age class section
0 623df5c9a2f230bd4ff5fe28 John 10 VI A
1 623df5cca2f230bd4ff5fe29 Maria 9 VI B
In [ ]:

kindergarten

  1. Arithmetic
  2. Identify Pictures
  3. Learn Words

Python for kids

  1. How We Teach Exponents and Powers to Kids
  2. Plotting points
  3. Learning shapes, size and color
  4. Learning Coordinate Geometry
  5. Draw Shapes With Points Using Matplotlib Module
  6. Learn Coordinate Geometry With Intuitive Way - Part 1
  7. Teaching Geometry to Grade 3 Kids
  8. Plotting Simple Line, Bar And Pie Chart
  9. Draw Dream House Using Python Matplotlib Module
  10. Draw Circle - Diameter, Radius, Arc and Segment Using Python Matplotlib Module
  11. Draw Badminton Court Using Python Matplotlib Module
  12. Let Us Play With Carrom Board in Python
  13. Draw Some Interesting Pattern With Numpy & Matplotlib Module
  14. Step by Step Draw Boy Using Matplotlib Module
  15. Make Shapes With SystemOfBobies
  16. Draw Cartoon
  17. Draw Two Cartoons Inside House
  18. 3Dimension Plotting in Python
  19. Solar System
  20. Plot World, Continent and Cities

Fourier series

  1. Plotting fourier series

Linear Equations

  1. Linear Equations

Geometry

  1. Cone geometry
  2. Cube geometry
  3. Cylinder geometry
  4. Disk geometry
  5. Open top box geometry
  6. Rectangle geometry
  7. Ring geometry
  8. Saturn geometry
  9. Sphere geometry

Laplace

  1. Laplace transfomation plotting in 3D

Vectors

  1. Reference frames 3d
  2. Vector implimentation 2 vectors
  3. Vector with numpy and sympy

Differential equations

  1. First order differential equations

Functions

  1. Function plotting in 3D
  2. Function plotting in 2D

Jacobian

  1. Jacobian

Lagrangian

  1. Lagrange multipliers
  2. Lagrange example

Waves

  1. Plot electromagnectic waves

Electromagnetism

  1. Field plot
  2. Electric fields
  3. Fields
  4. Charges on Equilateral Triangle

Optics

  1. Lens optics geometry
  2. Mirror optics geometry

Quantum mechanics concepts

  1. Dirac notation and photon polarisation
  2. Photon polarisation
  3. Electron spin
  4. Position momentum and heisenberg
  5. The EPR paradox explanation
  6. Energy operators the hamiltonian and schrodinger
  7. The harmonic oscillator
  8. Schrodinger equation a simple derivation

Theory of relativity

  1. Derivation of time dilation
  2. Gamma factor
  3. Spacetime reference frames equations

Kinematics

  1. Circular motion

Thermodynamics

  1. Entropy and the second law of thermodynamics
  2. Introductory chemical engineering thermodynamics

Formulae

  1. Formulae

A level physics

  1. Classical mechanics

Chemistry

  1. ChemCalcManager
  2. ChemistryFormulaManager
  3. ElementGroupManager
  4. ElementsManager
  5. EnvironmentManager
  6. MolarManager
  7. MoleculesManager
  8. PeriodicTableManager
  9. ReactivityManager
  10. SmilesManager
  11. VisualizationManager

English

  1. Book Manager
  2. Word Manager
  3. Grammar Manager
  4. Tense Manager
  5. Vocabulary Manager
  6. Learn Language Manager
  7. I sat with cat - Poem by Ishwarya
  8. There was a bug - Poem by Ishwarya
  9. The dog was doing a jog - Poem by Ishwarya

Geography

  1. Solar System
  2. Plot World, Continent and Cities

Animation

  1. Animate Circle With SVG
  2. Animate Stars in Different Directions Kids
  3. Move Box in HTML

Plotting

  1. Matplotlib Introduction
  2. Single Line Plots
  3. Bar Plots
  4. Histograms Plots
  5. Types of Plotting
  6. Seaborn Introduction
  7. Seaborn Scatterplot
  8. Seaborn Barplot
  9. Seaborn Boxplot
  10. Seaborn Pairplot
  11. Seaborn Heatmap
  12. Seaborn Seaborn Pandas
  13. Seaborn Plot a Simple Linear Relationship Between Two Variables
  14. Seaborn Displot
  15. Seaborn Palette
  16. Three-Dimensional Plotting in Matplotlib
  17. Interactive Visualizations With Bokeh Part 1
  18. Data Visualization With Seaborn - Part 1

SVG

  1. SVG Shapes
  2. SVG Shapes Advanced
  3. Animate Circle With SVG
  4. SVG Animation
  5. SVG Text
  6. Can We Write Alphabets With Help of SVG?

Python

  1. Searching Text in Multiple Files in Python
  2. Working With MongoDB & Python Using PyMongo and Pandas
  3. Capture Date, Phone and Email From Text With Regular Expression in Python
  4. Set Operations in Python
  5. Extract Table Data From Wikipedia Using Web Scraping With Python
  6. Word Cloud for a Website
  7. Sqlite Relational Database Management With Python
  8. How to Import Data From a MySql Database Into Pandas Data Frame
  9. Python Object Serialization and Deserialization With Pickle
  10. Regular Expression Operations in Python
  11. Creating and Writing to Different Type of Files in Python
  12. Convert Word file to PDF, HTML and PDF to JPG, PNG in Python

Machine Learning

  1. Deal Banking Marketing Campaign Dataset With Machine Learning

TensorFlow

  1. Difference Between Scalar, Vector, Matrix and Tensor
  2. TensorFlow Deep Learning Model With IRIS Dataset
  3. Sequence to Sequence Learning With Neural Networks To Perform Number Addition
  4. Image Classification Model MobileNet V2 from TensorFlow Hub
  5. Step by Step Intent Recognition With BERT
  6. Sentiment Analysis for Hotel Reviews With NLTK and Keras
  7. Simple Sequence Prediction With LSTM
  8. Image Classification With ResNet50 Model
  9. Predict Amazon Inc Stock Price with Machine Learning
  10. Predict Diabetes With Machine Learning Algorithms
  11. TensorFlow Build Custom Convolutional Neural Network With MNIST Dataset
  12. Deal Banking Marketing Campaign Dataset With Machine Learning

PySpark

  1. How to Parallelize and Distribute Collection in PySpark
  2. Role of StringIndexer and Pipelines in PySpark ML Feature - Part 1
  3. Role of OneHotEncoder and Pipelines in PySpark ML Feature - Part 2
  4. Feature Transformer VectorAssembler in PySpark ML Feature - Part 3
  5. Logistic Regression in PySpark (ML Feature) with Breast Cancer Data Set

PyTorch

  1. Build the Neural Network with PyTorch
  2. Image Classification with PyTorch
  3. Twitter Sentiment Classification In PyTorch
  4. Training an Image Classifier in Pytorch

Natural Language Processing

  1. Spelling Correction Of The Text Data In Natural Language Processing
  2. Handling Text For Machine Learning
  3. Extracting Text From PDF File in Python Using PyPDF2
  4. How to Collect Data Using Twitter API V2 For Natural Language Processing
  5. Converting Text to Features in Natural Language Processing
  6. Extract A Noun Phrase For A Sentence In Natural Language Processing

Others

  1. test-nutan
  2. QuestionManager.ipynb
  3. ActivityManager.ipynb
  4. Why us?
  5. How to Setup Wordpress Website Step by Step in Ubuntu VPS
  6. Setup Plesk Panel and Deploy Wordpress Website in Ubuntu VPS - Part 1
  7. Deploy a New Server With Plesk Control Panel - Part 2
  8. Setup Plesk Control Panel For Wordpress Website - Part 3
  9. Deploy Wordpress Website In Plesk Control Panel - Part 4
  10. HTML Basic Example
  11. Install MongoDB Community Server in Windows10
  12. How to Setup SAP BTP Trial Account and Create SAP HANA Cloud Database
  13. Connect SAP Hana Cloud Database Through Python
  14. How To Create a Self-Signed SSL Certificate for Apache in Ubuntu Virtual Cloud Server
  15. Consume Rest API in Django Web Application
  16. Setup GIT Repository in Ubuntu VPS
▲