Role of OneHotEncoder and Pipelines in PySpark ML Feature - Part 2

Role of OneHotEncoder and Pipelines in PySpark ML Feature - Part 2

Part 1 - What is StringIndexer?

We have already discussed regarding StringIndexer (link)

What is OneHotEncoder?

class pyspark.ml.feature.OneHotEncoder(inputCols=None, outputCols=None, handleInvalid='error', dropLast=True, inputCol=None, outputCol=None) - One Hot Encoding is a technique for converting categorical attributes into a binary vector.

A one-hot encoder that maps a column of category indices to a column of binary vectors, with at most a single one-value per row that indicates the input category index.

For example with 5 categories, an input value of 2.0 would map to an output vector of [0.0, 0.0, 1.0, 0.0].

The last category is not included by default (configurable via dropLast), because it makes the vector entries sum up to one, and hence linearly dependent. So an input value of 4.0 maps to [0.0, 0.0, 0.0, 0.0].

Let us see an example

Create SparkSession

In [2]:
#import SparkSession
from pyspark.sql import SparkSession

SparkSession is an entry point to Spark to work with RDD, DataFrame, and Dataset. To create SparkSession in Python, we need to use the builder() method and calling getOrCreate() method.

If SparkSession already exists it returns otherwise create a new SparkSession.

In [3]:
spark = SparkSession.builder.appName('xvspark').getOrCreate()

Create dataframe by declaring the schema

In [4]:
from pyspark.sql.types import *

StructType class to define the structure of the DataFrame.

In [5]:
#create the structure of schema
schema = StructType().add("id","integer").add("name","string").add("qualification","string").add("age", "integer").add("gender", "string")
In [6]:
#create data
data = [
    (1,'John',"B.A.", 20, "Male"),
    (2,'Martha',"B.Com.", 20, "Female"),
    (3,'Mona',"B.Com.", 21, "Female"),
    (4,'Harish',"B.Sc.", 22, "Male"),
    (5,'Jonny',"B.A.", 22, "Male"),
    (6,'Maria',"B.A.", 23, "Female"),
    (7,'Monalisa',"B.A.", 21, "Female")
]
In [7]:
#create dataframe
df = spark.createDataFrame(data, schema=schema)
In [8]:
#columns of dataframe
df.columns
Out[8]:
['id', 'name', 'qualification', 'age', 'gender']
In [9]:
df.show()

+---+--------+-------------+---+------+
| id|    name|qualification|age|gender|
+---+--------+-------------+---+------+
|  1|    John|         B.A.| 20|  Male|
|  2|  Martha|       B.Com.| 20|Female|
|  3|    Mona|       B.Com.| 21|Female|
|  4|  Harish|        B.Sc.| 22|  Male|
|  5|   Jonny|         B.A.| 22|  Male|
|  6|   Maria|         B.A.| 23|Female|
|  7|Monalisa|         B.A.| 21|Female|
+---+--------+-------------+---+------+

Apply OneHotEncoder to qualification and gender column

We can not apply OneHotEncoder to string columns directly. We need to first convert string columns to numeric value. For that we will use StringIndexer. After that we can apply OneHotEncoder.

Apply StringIndexer to qualification column

In [10]:
#import required libraries
from pyspark.ml.feature import StringIndexer
In [11]:
qualification_indexer = StringIndexer(inputCol="qualification", outputCol="qualificationIndex")

#Fits a model to the input dataset with optional parameters.
df1 = qualification_indexer.fit(df).transform(df)
df1.show()

+---+--------+-------------+---+------+------------------+
| id|    name|qualification|age|gender|qualificationIndex|
+---+--------+-------------+---+------+------------------+
|  1|    John|         B.A.| 20|  Male|               0.0|
|  2|  Martha|       B.Com.| 20|Female|               1.0|
|  3|    Mona|       B.Com.| 21|Female|               1.0|
|  4|  Harish|        B.Sc.| 22|  Male|               2.0|
|  5|   Jonny|         B.A.| 22|  Male|               0.0|
|  6|   Maria|         B.A.| 23|Female|               0.0|
|  7|Monalisa|         B.A.| 21|Female|               0.0|
+---+--------+-------------+---+------+------------------+

"B.A." gets index 0 because it is the most frequent, then "B.Com" gets index 1 and "B.Sc." gets index 2.

Apply StringIndexer to gender column

In [12]:
gender_indexer = StringIndexer(inputCol="gender", outputCol="genderIndex")

#Fits a model to the input dataset with optional parameters.
df2 = gender_indexer.fit(df).transform(df)
df2.show()

+---+--------+-------------+---+------+-----------+
| id|    name|qualification|age|gender|genderIndex|
+---+--------+-------------+---+------+-----------+
|  1|    John|         B.A.| 20|  Male|        1.0|
|  2|  Martha|       B.Com.| 20|Female|        0.0|
|  3|    Mona|       B.Com.| 21|Female|        0.0|
|  4|  Harish|        B.Sc.| 22|  Male|        1.0|
|  5|   Jonny|         B.A.| 22|  Male|        1.0|
|  6|   Maria|         B.A.| 23|Female|        0.0|
|  7|Monalisa|         B.A.| 21|Female|        0.0|
+---+--------+-------------+---+------+-----------+

Apply OneHotEncoder to qualificationIndex column

In [13]:
from pyspark.ml.feature import OneHotEncoder
In [14]:
#onehotencoder to qualificationIndex
onehotencoder_qualification_vector = OneHotEncoder(inputCol="qualificationIndex", outputCol="qualification_vec")
df11 = onehotencoder_qualification_vector.fit(df1).transform(df1)
In [15]:
df11.show()

+---+--------+-------------+---+------+------------------+-----------------+
| id|    name|qualification|age|gender|qualificationIndex|qualification_vec|
+---+--------+-------------+---+------+------------------+-----------------+
|  1|    John|         B.A.| 20|  Male|               0.0|    (2,[0],[1.0])|
|  2|  Martha|       B.Com.| 20|Female|               1.0|    (2,[1],[1.0])|
|  3|    Mona|       B.Com.| 21|Female|               1.0|    (2,[1],[1.0])|
|  4|  Harish|        B.Sc.| 22|  Male|               2.0|        (2,[],[])|
|  5|   Jonny|         B.A.| 22|  Male|               0.0|    (2,[0],[1.0])|
|  6|   Maria|         B.A.| 23|Female|               0.0|    (2,[0],[1.0])|
|  7|Monalisa|         B.A.| 21|Female|               0.0|    (2,[0],[1.0])|
+---+--------+-------------+---+------+------------------+-----------------+
In [16]:
#onehotencoder to genderIndex
onehotencoder_gender_vector = OneHotEncoder(inputCol="genderIndex", outputCol="gender_vec")
df12 = onehotencoder_gender_vector.fit(df2).transform(df2)
In [17]:
df12.show()

+---+--------+-------------+---+------+-----------+-------------+
| id|    name|qualification|age|gender|genderIndex|   gender_vec|
+---+--------+-------------+---+------+-----------+-------------+
|  1|    John|         B.A.| 20|  Male|        1.0|    (1,[],[])|
|  2|  Martha|       B.Com.| 20|Female|        0.0|(1,[0],[1.0])|
|  3|    Mona|       B.Com.| 21|Female|        0.0|(1,[0],[1.0])|
|  4|  Harish|        B.Sc.| 22|  Male|        1.0|    (1,[],[])|
|  5|   Jonny|         B.A.| 22|  Male|        1.0|    (1,[],[])|
|  6|   Maria|         B.A.| 23|Female|        0.0|(1,[0],[1.0])|
|  7|Monalisa|         B.A.| 21|Female|        0.0|(1,[0],[1.0])|
+---+--------+-------------+---+------+-----------+-------------+

One hot encoding of a numeric column

Direct one-hot-encoding without indexing

In [18]:
onehotencoder_age_vector = OneHotEncoder(inputCol="age", outputCol="age_vec")
df13 = onehotencoder_age_vector.fit(df2).transform(df2)
In [19]:
df13.show()

+---+--------+-------------+---+------+-----------+---------------+
| id|    name|qualification|age|gender|genderIndex|        age_vec|
+---+--------+-------------+---+------+-----------+---------------+
|  1|    John|         B.A.| 20|  Male|        1.0|(23,[20],[1.0])|
|  2|  Martha|       B.Com.| 20|Female|        0.0|(23,[20],[1.0])|
|  3|    Mona|       B.Com.| 21|Female|        0.0|(23,[21],[1.0])|
|  4|  Harish|        B.Sc.| 22|  Male|        1.0|(23,[22],[1.0])|
|  5|   Jonny|         B.A.| 22|  Male|        1.0|(23,[22],[1.0])|
|  6|   Maria|         B.A.| 23|Female|        0.0|     (23,[],[])|
|  7|Monalisa|         B.A.| 21|Female|        0.0|(23,[21],[1.0])|
+---+--------+-------------+---+------+-----------+---------------+

Direct one-hot-encoding after indexing

In [22]:
age_indexer = StringIndexer(inputCol="age", outputCol="ageIndex")
df13 = age_indexer.fit(df).transform(df)


onehotencoder_age_vector = OneHotEncoder(inputCol="ageIndex", outputCol="age_vec")
df14 = onehotencoder_age_vector.fit(df13).transform(df13)

df14.show()

+---+--------+-------------+---+------+--------+-------------+
| id|    name|qualification|age|gender|ageIndex|      age_vec|
+---+--------+-------------+---+------+--------+-------------+
|  1|    John|         B.A.| 20|  Male|     0.0|(3,[0],[1.0])|
|  2|  Martha|       B.Com.| 20|Female|     0.0|(3,[0],[1.0])|
|  3|    Mona|       B.Com.| 21|Female|     1.0|(3,[1],[1.0])|
|  4|  Harish|        B.Sc.| 22|  Male|     2.0|(3,[2],[1.0])|
|  5|   Jonny|         B.A.| 22|  Male|     2.0|(3,[2],[1.0])|
|  6|   Maria|         B.A.| 23|Female|     3.0|    (3,[],[])|
|  7|Monalisa|         B.A.| 21|Female|     1.0|(3,[1],[1.0])|
+---+--------+-------------+---+------+--------+-------------+

Using Pipeline

In [23]:
#import module
from pyspark.ml import Pipeline

Reload Data

In [24]:
schema = StructType().add("id","integer").add("name","string").add("qualification","string").add("age", "integer").add("gender", "string")
df = spark.createDataFrame(data, schema=schema)

Create Pipeline and pass all stages

In [25]:
qualification_indexer = StringIndexer(inputCol="qualification", outputCol="qualificationIndex")
gender_indexer = StringIndexer(inputCol="gender", outputCol="genderIndex")
age_indexer = StringIndexer(inputCol="age", outputCol="ageIndex")

onehotencoder_qualification_vector = OneHotEncoder(inputCol="qualificationIndex", outputCol="qualification_vec")
onehotencoder_gender_vector = OneHotEncoder(inputCol="genderIndex", outputCol="gender_vec")
onehotencoder_age_vector = OneHotEncoder(inputCol="ageIndex", outputCol="age_vec")


pipeline = Pipeline(stages=[qualification_indexer, 
                            gender_indexer, 
                            age_indexer,
                            onehotencoder_qualification_vector, 
                            onehotencoder_gender_vector,
                            onehotencoder_age_vector
                           ])
In [27]:
df_transformed = pipeline.fit(df).transform(df)
df_transformed.show()

+---+--------+-------------+---+------+------------------+-----------+--------+-----------------+-------------+-------------+
| id|    name|qualification|age|gender|qualificationIndex|genderIndex|ageIndex|qualification_vec|   gender_vec|      age_vec|
+---+--------+-------------+---+------+------------------+-----------+--------+-----------------+-------------+-------------+
|  1|    John|         B.A.| 20|  Male|               0.0|        1.0|     0.0|    (2,[0],[1.0])|    (1,[],[])|(3,[0],[1.0])|
|  2|  Martha|       B.Com.| 20|Female|               1.0|        0.0|     0.0|    (2,[1],[1.0])|(1,[0],[1.0])|(3,[0],[1.0])|
|  3|    Mona|       B.Com.| 21|Female|               1.0|        0.0|     1.0|    (2,[1],[1.0])|(1,[0],[1.0])|(3,[1],[1.0])|
|  4|  Harish|        B.Sc.| 22|  Male|               2.0|        1.0|     2.0|        (2,[],[])|    (1,[],[])|(3,[2],[1.0])|
|  5|   Jonny|         B.A.| 22|  Male|               0.0|        1.0|     2.0|    (2,[0],[1.0])|    (1,[],[])|(3,[2],[1.0])|
|  6|   Maria|         B.A.| 23|Female|               0.0|        0.0|     3.0|    (2,[0],[1.0])|(1,[0],[1.0])|    (3,[],[])|
|  7|Monalisa|         B.A.| 21|Female|               0.0|        0.0|     1.0|    (2,[0],[1.0])|(1,[0],[1.0])|(3,[1],[1.0])|
+---+--------+-------------+---+------+------------------+-----------+--------+-----------------+-------------+-------------+

You can convert it to Pandas DataFrame

In [29]:
df_transformed.toPandas()
Out[29]:
id name qualification age gender qualificationIndex genderIndex ageIndex qualification_vec gender_vec age_vec
0 1 John B.A. 20 Male 0.0 1.0 0.0 (1.0, 0.0) (0.0) (1.0, 0.0, 0.0)
1 2 Martha B.Com. 20 Female 1.0 0.0 0.0 (0.0, 1.0) (1.0) (1.0, 0.0, 0.0)
2 3 Mona B.Com. 21 Female 1.0 0.0 1.0 (0.0, 1.0) (1.0) (0.0, 1.0, 0.0)
3 4 Harish B.Sc. 22 Male 2.0 1.0 2.0 (0.0, 0.0) (0.0) (0.0, 0.0, 1.0)
4 5 Jonny B.A. 22 Male 0.0 1.0 2.0 (1.0, 0.0) (0.0) (0.0, 0.0, 1.0)
5 6 Maria B.A. 23 Female 0.0 0.0 3.0 (1.0, 0.0) (1.0) (0.0, 0.0, 0.0)
6 7 Monalisa B.A. 21 Female 0.0 0.0 1.0 (1.0, 0.0) (1.0) (0.0, 1.0, 0.0)
In [ ]:

kindergarten

  1. Arithmetic
  2. Identify Pictures
  3. Learn Words

Python for kids

  1. How We Teach Exponents and Powers to Kids
  2. Plotting points
  3. Learning shapes, size and color
  4. Learning Coordinate Geometry
  5. Draw Shapes With Points Using Matplotlib Module
  6. Learn Coordinate Geometry With Intuitive Way - Part 1
  7. Teaching Geometry to Grade 3 Kids
  8. Plotting Simple Line, Bar And Pie Chart
  9. Draw Dream House Using Python Matplotlib Module
  10. Draw Circle - Diameter, Radius, Arc and Segment Using Python Matplotlib Module
  11. Draw Badminton Court Using Python Matplotlib Module
  12. Let Us Play With Carrom Board in Python
  13. Draw Some Interesting Pattern With Numpy & Matplotlib Module
  14. Step by Step Draw Boy Using Matplotlib Module
  15. Make Shapes With SystemOfBobies
  16. Draw Cartoon
  17. Draw Two Cartoons Inside House
  18. 3Dimension Plotting in Python
  19. Solar System
  20. Plot World, Continent and Cities

Fourier series

  1. Plotting fourier series

Linear Equations

  1. Linear Equations

Geometry

  1. Cone geometry
  2. Cube geometry
  3. Cylinder geometry
  4. Disk geometry
  5. Open top box geometry
  6. Rectangle geometry
  7. Ring geometry
  8. Saturn geometry
  9. Sphere geometry

Laplace

  1. Laplace transfomation plotting in 3D

Vectors

  1. Reference frames 3d
  2. Vector implimentation 2 vectors
  3. Vector with numpy and sympy

Differential equations

  1. First order differential equations

Functions

  1. Function plotting in 3D
  2. Function plotting in 2D

Jacobian

  1. Jacobian

Lagrangian

  1. Lagrange multipliers
  2. Lagrange example

Waves

  1. Plot electromagnectic waves

Electromagnetism

  1. Field plot
  2. Electric fields
  3. Fields
  4. Charges on Equilateral Triangle

Optics

  1. Lens optics geometry
  2. Mirror optics geometry

Quantum mechanics concepts

  1. Dirac notation and photon polarisation
  2. Photon polarisation
  3. Electron spin
  4. Position momentum and heisenberg
  5. The EPR paradox explanation
  6. Energy operators the hamiltonian and schrodinger
  7. The harmonic oscillator
  8. Schrodinger equation a simple derivation

Theory of relativity

  1. Derivation of time dilation
  2. Gamma factor
  3. Spacetime reference frames equations

Kinematics

  1. Circular motion

Thermodynamics

  1. Entropy and the second law of thermodynamics
  2. Introductory chemical engineering thermodynamics

Formulae

  1. Formulae

A level physics

  1. Classical mechanics

Chemistry

  1. ChemCalcManager
  2. ChemistryFormulaManager
  3. ElementGroupManager
  4. ElementsManager
  5. EnvironmentManager
  6. MolarManager
  7. MoleculesManager
  8. PeriodicTableManager
  9. ReactivityManager
  10. SmilesManager
  11. VisualizationManager

English

  1. Book Manager
  2. Word Manager
  3. Grammar Manager
  4. Tense Manager
  5. Vocabulary Manager
  6. Learn Language Manager
  7. I sat with cat - Poem by Ishwarya
  8. There was a bug - Poem by Ishwarya
  9. The dog was doing a jog - Poem by Ishwarya

Geography

  1. Solar System
  2. Plot World, Continent and Cities

Animation

  1. Animate Circle With SVG
  2. Animate Stars in Different Directions Kids
  3. Move Box in HTML

Plotting

  1. Matplotlib Introduction
  2. Single Line Plots
  3. Bar Plots
  4. Histograms Plots
  5. Types of Plotting
  6. Seaborn Introduction
  7. Seaborn Scatterplot
  8. Seaborn Barplot
  9. Seaborn Boxplot
  10. Seaborn Pairplot
  11. Seaborn Heatmap
  12. Seaborn Seaborn Pandas
  13. Seaborn Plot a Simple Linear Relationship Between Two Variables
  14. Seaborn Displot
  15. Seaborn Palette
  16. Three-Dimensional Plotting in Matplotlib
  17. Interactive Visualizations With Bokeh Part 1
  18. Data Visualization With Seaborn - Part 1

SVG

  1. SVG Shapes
  2. SVG Shapes Advanced
  3. Animate Circle With SVG
  4. SVG Animation
  5. SVG Text
  6. Can We Write Alphabets With Help of SVG?

Python

  1. Searching Text in Multiple Files in Python
  2. Working With MongoDB & Python Using PyMongo and Pandas
  3. Capture Date, Phone and Email From Text With Regular Expression in Python
  4. Set Operations in Python
  5. Extract Table Data From Wikipedia Using Web Scraping With Python
  6. Word Cloud for a Website
  7. Sqlite Relational Database Management With Python
  8. How to Import Data From a MySql Database Into Pandas Data Frame
  9. Python Object Serialization and Deserialization With Pickle
  10. Regular Expression Operations in Python
  11. Creating and Writing to Different Type of Files in Python
  12. Convert Word file to PDF, HTML and PDF to JPG, PNG in Python

Machine Learning

  1. Deal Banking Marketing Campaign Dataset With Machine Learning

TensorFlow

  1. Difference Between Scalar, Vector, Matrix and Tensor
  2. TensorFlow Deep Learning Model With IRIS Dataset
  3. Sequence to Sequence Learning With Neural Networks To Perform Number Addition
  4. Image Classification Model MobileNet V2 from TensorFlow Hub
  5. Step by Step Intent Recognition With BERT
  6. Sentiment Analysis for Hotel Reviews With NLTK and Keras
  7. Simple Sequence Prediction With LSTM
  8. Image Classification With ResNet50 Model
  9. Predict Amazon Inc Stock Price with Machine Learning
  10. Predict Diabetes With Machine Learning Algorithms
  11. TensorFlow Build Custom Convolutional Neural Network With MNIST Dataset
  12. Deal Banking Marketing Campaign Dataset With Machine Learning

PySpark

  1. How to Parallelize and Distribute Collection in PySpark
  2. Role of StringIndexer and Pipelines in PySpark ML Feature - Part 1
  3. Role of OneHotEncoder and Pipelines in PySpark ML Feature - Part 2
  4. Feature Transformer VectorAssembler in PySpark ML Feature - Part 3
  5. Logistic Regression in PySpark (ML Feature) with Breast Cancer Data Set

PyTorch

  1. Build the Neural Network with PyTorch
  2. Image Classification with PyTorch
  3. Twitter Sentiment Classification In PyTorch
  4. Training an Image Classifier in Pytorch

Natural Language Processing

  1. Spelling Correction Of The Text Data In Natural Language Processing
  2. Handling Text For Machine Learning
  3. Extracting Text From PDF File in Python Using PyPDF2
  4. How to Collect Data Using Twitter API V2 For Natural Language Processing
  5. Converting Text to Features in Natural Language Processing
  6. Extract A Noun Phrase For A Sentence In Natural Language Processing

Others

  1. test-nutan
  2. QuestionManager.ipynb
  3. ActivityManager.ipynb
  4. Why us?
  5. How to Setup Wordpress Website Step by Step in Ubuntu VPS
  6. Setup Plesk Panel and Deploy Wordpress Website in Ubuntu VPS - Part 1
  7. Deploy a New Server With Plesk Control Panel - Part 2
  8. Setup Plesk Control Panel For Wordpress Website - Part 3
  9. Deploy Wordpress Website In Plesk Control Panel - Part 4
  10. HTML Basic Example
  11. Install MongoDB Community Server in Windows10
  12. How to Setup SAP BTP Trial Account and Create SAP HANA Cloud Database
  13. Connect SAP Hana Cloud Database Through Python
  14. How To Create a Self-Signed SSL Certificate for Apache in Ubuntu Virtual Cloud Server
  15. Consume Rest API in Django Web Application
  16. Setup GIT Repository in Ubuntu VPS
▲