資料預處理 (Label encoding/One hot encoding/get_dummies) - Python

Posted by: bart30508 | in Data Science, Python | 1 month, 1 week ago |

Label encoding : 把每個類別 mapping 到某個整數,不會增加新欄位

from sklearn.preprocessing import LabelEncoder
labelencoder = LabelEncoder()
data_le=pd.DataFrame(dic)
data_le['Country'] = labelencoder.fit_transform(data_le['Country'])
data_le
--- 兩種寫法,上面fit_transform一起,下面fit與transaform分開 ---
le = LabelEncoder()
le.fit(train_x[c].fillna('NA'))

# 訓練資料、測試資料的轉換
train_x[c] = le.transform(train_x[c].fillna('NA'))
test_x[c] = le.transform(test_x[c].fillna('NA'))

One hot encoding : 為每個類別新增一個欄位,用 0/1 表示是否

from sklearn.preprocessing import OneHotEncoder
onehotencoder = OneHotEncoder(categorical_features = [0])
data_str_ohe=onehotencoder.fit_transform(data_le).toarray()
pd.DataFrame(data_str_ohe)

categorical_features = [0]: 表示欲在data上執行One hot encoding的index為0

data_le: 為經過Label encoding編碼的資料(註:OneHotEncoder的輸入要為2-D array,而Label encoding為1-D array)

OneHotEncoder會轉出scipy.csr_matrix資料結構用.toarray()轉array

以下為Kaggle Titanic用On hot encoding使Sex/Embarked/Pclass進行轉換

# -----------------------------------
# 建立邏輯斯迴歸模型所需特徵
# -----------------------------------
from sklearn.preprocessing import OneHotEncoder
import pandas as pd

# 讀取訓練資料、測試資料
train = pd.read_csv('../input/ch01-titanic/train.csv')
test = pd.read_csv('../input/ch01-titanic/test.csv')

# 訓練與測試資料
train_x2 = train.drop(['Survived'], axis=1)
test_x2 = test.copy()

# 去除訓練、測試資料中的 PassengerId 欄位
train_x2 = train_x2.drop(['PassengerId'], axis=1)
test_x2 = test_x2.drop(['PassengerId'], axis=1)

# 去除訓練、測試資料中的 Name、Ticket、Cabin 欄位
train_x2 = train_x2.drop(['Name', 'Ticket', 'Cabin'], axis=1)
test_x2 = test_x2.drop(['Name', 'Ticket', 'Cabin'], axis=1)

# 建立 one-hot 編碼的物件與進行相關設置 (告訴它編碼的方式)
cat_cols = ['Sex', 'Embarked', 'Pclass']
ohe = OneHotEncoder(categories='auto', sparse=False)
ohe.fit(train_x2[cat_cols].fillna('NA'))

# 對指定的欄位的數據進行 one-hot 編碼
ohe_columns = []
for i, c in enumerate(cat_cols):
    ohe_columns += [f'{c}_{v}' for v in ohe.categories_[i]]

print(ohe_columns)

Output: ['Sex_female', 'Sex_male', 'Embarked_C', 'Embarked_NA', 'Embarked_Q', 'Embarked_S', 'Pclass_1', 'Pclass_2', 'Pclass_3']

get_dummies : 在pandas中,我們可以透過get_dummies()直接將類別型資料轉換成二元的數值型資料,不需再透過LabelEncoder先將類別型資料轉換成數值型,再將其轉換成二元數值型。

df_dum=pd.get_dummies(df)
# 指定欄位寫法
df_dum=pd.get_dummies(df[['Area','weather']])
df_new=pd.concat([df_dum,df['temperature']],axis=1)

Reference: https://medium.com/@PatHuang/%E5%88%9D%E5%AD%B8python%E6%89%8B%E8%A8%98-3-%E8%B3%87%E6%96%99%E5%89%8D%E8%99%95%E7%90%86-label-encoding-one-hot-encoding-85c983d63f87

https://pyecontech.com/2020/08/27/python_onehot_encoder/

Currently unrated
 or 

Subscribe

* indicates required

Recent Posts

Archive

2022
2021

Categories

Apache 1

Data Science 2

Dbfit 1

Design Pattern 1

Devops 3

DigitalOcean 1

Django 1

English 3

Excel 5

Flask 3

Git 1

HackMD 1

Heroku 1

Html/Css 1

Linux 4

Machine Learning 2

Manufacture 1

Mezzanine 18

Oracle 1

Postgresql 7

PowerBI 4

Powershell 4

Python 21

SEO 2

SQL Server 51

SQLite 1

Windows 1

database 8

work-experience 1

其他 1

自我成長 1

資料工程 1

Tags

SEO(1) Github(2) Title Tag(2) ML(1) 李宏毅(1) SQL Server(18) Tempdb(1) SSMS(1) Windows(1) 自我成長(2) Excel(1) python Flask(1) python(5) Flask(2)

Authors

bart30508 (146)

Feeds

RSS / Atom

資料預處理 (Label encoding/One hot encoding/get_dummies) - Python

© COPYRIGHT 2011-2022. Max的文藝復興. ALL RIGHT RESERVED.