信用風險預測
2021-10-12
目的：利用邏輯斯迴歸與隨機森林模型建立分類問題模型
資料：German Credit
說明：利用每個人的信用風險(好、壞)及20個解釋變數建立分類模型，並進行模型評估
參考： https://mlr3gallery.mlr-org.com/posts/2020-03-11-basics-german-credit/
1 資料
1.1 匯入相關套件
首先匯入R語言機器學習套件mlr3verse與mlr3learners，以及data.table、ggplot。
library("mlr3verse")## Loading required package: mlr3library("mlr3learners")
library("data.table")
library("ggplot2")1.2 匯入資料
由rchallenge套件取得範例資料german。
library("rchallenge")
data("german", package = "rchallenge")1.3 探索資料
利用dim()與str()觀察資料筆數與型態，資料集內共有1000筆資料，21個變數(欄位)。 其中，反應變數(被預測的變數)為二元變數credit_risk (只有good或bad兩類)， 每個人會依其信用風險被分為兩類。此外，也可利用skimr套件查看各變數統計資訊。
dim(german)## [1] 1000 21str(german)## 'data.frame': 1000 obs. of 21 variables:
## $ status : Factor w/ 4 levels "no checking account",..: 1 1 2 1 1 1 1 1 4 2 ...
## $ duration : int 18 9 12 12 12 10 8 6 18 24 ...
## $ credit_history : Factor w/ 5 levels "delay in paying off in the past",..: 5 5 3 5 5 5 5 5 5 3 ...
## $ purpose : Factor w/ 11 levels "others","car (new)",..: 3 1 10 1 1 1 1 1 4 4 ...
## $ amount : int 1049 2799 841 2122 2171 2241 3398 1361 1098 3758 ...
## $ savings : Factor w/ 5 levels "unknown/no savings account",..: 1 1 2 1 1 1 1 1 1 3 ...
## $ employment_duration : Factor w/ 5 levels "unemployed","< 1 yr",..: 2 3 4 3 3 2 4 2 1 1 ...
## $ installment_rate : Ord.factor w/ 4 levels ">= 35"<"25 <= ... < 35"<..: 4 2 2 3 4 1 1 2 4 1 ...
## $ personal_status_sex : Factor w/ 4 levels "male : divorced/separated",..: 2 3 2 3 3 3 3 3 2 2 ...
## $ other_debtors : Factor w/ 3 levels "none","co-applicant",..: 1 1 1 1 1 1 1 1 1 1 ...
## $ present_residence : Ord.factor w/ 4 levels "< 1 yr"<"1 <= ... < 4 yrs"<..: 4 2 4 2 4 3 4 4 4 4 ...
## $ property : Factor w/ 4 levels "unknown / no property",..: 2 1 1 1 2 1 1 1 3 4 ...
## $ age : int 21 36 23 39 38 48 39 40 65 23 ...
## $ other_installment_plans: Factor w/ 3 levels "bank","stores",..: 3 3 3 3 1 3 3 3 3 3 ...
## $ housing : Factor w/ 3 levels "for free","rent",..: 1 1 1 1 2 1 2 2 2 1 ...
## $ number_credits : Ord.factor w/ 4 levels "1"<"2-3"<"4-5"<..: 1 2 1 2 2 2 2 1 2 1 ...
## $ job : Factor w/ 4 levels "unemployed/unskilled - non-resident",..: 3 3 2 2 2 2 2 2 1 1 ...
## $ people_liable : Factor w/ 2 levels "3 or more","0 to 2": 2 1 2 1 2 1 2 1 2 2 ...
## $ telephone : Factor w/ 2 levels "no","yes (under customer name)": 1 1 1 1 1 1 1 1 1 1 ...
## $ foreign_worker : Factor w/ 2 levels "yes","no": 2 2 2 1 1 1 1 1 2 2 ...
## $ credit_risk : Factor w/ 2 levels "bad","good": 2 2 2 2 2 2 2 2 2 2 ...在21個變數中，有18個變數資料型態為factor，3個為numeric。 沒有遺失值，3個數值型態的變數為右偏分配。
skimr::skim(german)| Name | german |
| Number of rows | 1000 |
| Number of columns | 21 |
| _______________________ | |
| Column type frequency: | |
| factor | 18 |
| numeric | 3 |
| ________________________ | |
| Group variables | None |
Variable type: factor
| skim_variable | n_missing | complete_rate | ordered | n_unique | top_counts |
|---|---|---|---|---|---|
| status | 0 | 1 | FALSE | 4 | …: 394, no : 274, …: 269, 0<=: 63 |
| credit_history | 0 | 1 | FALSE | 5 | no : 530, all: 293, exi: 88, cri: 49 |
| purpose | 0 | 1 | FALSE | 10 | fur: 280, oth: 234, car: 181, car: 103 |
| savings | 0 | 1 | FALSE | 5 | unk: 603, …: 183, …: 103, 100: 63 |
| employment_duration | 0 | 1 | FALSE | 5 | 1 <: 339, >= : 253, 4 <: 174, < 1: 172 |
| installment_rate | 0 | 1 | TRUE | 4 | < 2: 476, 25 : 231, 20 : 157, >= : 136 |
| personal_status_sex | 0 | 1 | FALSE | 4 | mal: 548, fem: 310, fem: 92, mal: 50 |
| other_debtors | 0 | 1 | FALSE | 3 | non: 907, gua: 52, co-: 41 |
| present_residence | 0 | 1 | TRUE | 4 | >= : 413, 1 <: 308, 4 <: 149, < 1: 130 |
| property | 0 | 1 | FALSE | 4 | bui: 332, unk: 282, car: 232, rea: 154 |
| other_installment_plans | 0 | 1 | FALSE | 3 | non: 814, ban: 139, sto: 47 |
| housing | 0 | 1 | FALSE | 3 | ren: 714, for: 179, own: 107 |
| number_credits | 0 | 1 | TRUE | 4 | 1: 633, 2-3: 333, 4-5: 28, >= : 6 |
| job | 0 | 1 | FALSE | 4 | ski: 630, uns: 200, man: 148, une: 22 |
| people_liable | 0 | 1 | FALSE | 2 | 0 t: 845, 3 o: 155 |
| telephone | 0 | 1 | FALSE | 2 | no: 596, yes: 404 |
| foreign_worker | 0 | 1 | FALSE | 2 | no: 963, yes: 37 |
| credit_risk | 0 | 1 | FALSE | 2 | goo: 700, bad: 300 |
Variable type: numeric
| skim_variable | n_missing | complete_rate | mean | sd | p0 | p25 | p50 | p75 | p100 | hist |
|---|---|---|---|---|---|---|---|---|---|---|
| duration | 0 | 1 | 20.90 | 12.06 | 4 | 12.0 | 18.0 | 24.00 | 72 | ▇▇▂▁▁ |
| amount | 0 | 1 | 3271.25 | 2822.75 | 250 | 1365.5 | 2319.5 | 3972.25 | 18424 | ▇▂▁▁▁ |
| age | 0 | 1 | 35.54 | 11.35 | 19 | 27.0 | 33.0 | 42.00 | 75 | ▇▆▃▁▁ |
2 建模
一般而言，在建置機器學習模型時我們會問以下幾個問題：
1. 我們要解決什麼問題？
2. 那一種演算法比較合適？
3. 如何評估模型？
本範例利用mlr3套件建模，會利用以下5個元件建模：
1. Task definition
2. Learner Definition
3. Training
4. Prediction
5. Performance Evaluation
2.1 Task definition
首先，定義問題型態，是迴歸問題還是分類問題或其他問題(非監督式學習)。 本範例為分類問題，因此使用TaskClassif這個任務類別。可以利用new()或 as_task_classif()這兩個方法建立任務物件。
task = as_task_classif(german, id = "GermanCredit", target = "credit_risk")利用as_task_classif()這個方法定義任務。其中，german為資料集， 其名稱為GermanCredit，要預測的目標(二元)變數為credit_risk。
2.2 Learner Definition
在設定目標變數後，接著就是要如何建模，用那一種演算法？ mlr3learners套件中提供多種不同的演算法(learner)。
匯入mlr3learners套件，以lrn()方法建立邏輯斯迴歸模型物件。
library("mlr3learners")
learner_logreg = lrn("classif.log_reg")
print(learner_logreg)## <LearnerClassifLogReg:classif.log_reg>
## * Model: -
## * Parameters: list()
## * Packages: stats
## * Predict Type: response
## * Feature types: logical, integer, numeric, character, factor, ordered
## * Properties: loglik, twoclass, weights2.3 Training
將資料以隨機抽樣方法分割成訓練樣本(80%)與測試樣本(20%)兩個資料集， 首先，建立邏輯斯迴歸模型，利用learner_logreg物件的train()方法先對訓練樣本建模。
train_set = sample(task$row_ids, 0.8 * task$nrow)
test_set = setdiff(task$row_ids, train_set)
train()中包含兩個引數，第一個為任務物件，第二個為訓練樣本的id。
learner_logreg$train(task, row_ids = train_set)取得模型並查看參數估計等模型資訊，可根據統計顯著性找出判斷個人信用風險高低的有效解釋變數，再進行模型精簡。
learner_logreg$model##
## Call: stats::glm(formula = task$formula(), family = "binomial", data = data,
## model = FALSE)
##
## Coefficients:
## (Intercept)
## 0.5611286
## age
## -0.0238784
## amount
## 0.0001123
## credit_historycritical account/other credits elsewhere
## 0.2138134
## credit_historyno credits taken/all credits paid back duly
## -0.8774705
## credit_historyexisting credits paid back duly till now
## -1.3076398
## credit_historyall credits at this bank paid back duly
## -1.7200082
## duration
## 0.0285633
## employment_duration< 1 yr
## -0.1547321
## employment_duration1 <= ... < 4 yrs
## -0.6748958
## employment_duration4 <= ... < 7 yrs
## -1.1947945
## employment_duration>= 7 yrs
## -0.5470958
## foreign_workerno
## 1.7302093
## housingrent
## -0.4749418
## housingown
## -0.3315796
## installment_rate.L
## 0.6429470
## installment_rate.Q
## -0.0387670
## installment_rate.C
## 0.0068883
## jobunskilled - resident
## 1.0410954
## jobskilled employee/official
## 0.9968647
## jobmanager/self-empl./highly qualif. employee
## 1.1103330
## number_credits.L
## 0.2709746
## number_credits.Q
## -0.2503767
## number_credits.C
## 0.0791030
## other_debtorsco-applicant
## 1.2334315
## other_debtorsguarantor
## -0.6258988
## other_installment_plansstores
## 0.4314923
## other_installment_plansnone
## -0.2644009
## people_liable0 to 2
## -0.2152718
## personal_status_sexfemale : non-single or male : single
## -0.4632567
## personal_status_sexmale : married/widowed
## -0.9369170
## personal_status_sexfemale : single
## -0.2847131
## present_residence.L
## 0.3189674
## present_residence.Q
## -0.5087215
## present_residence.C
## 0.3792300
## propertycar or other
## 0.2953297
## propertybuilding soc. savings agr./life insurance
## 0.4808495
## propertyreal estate
## 0.8989240
## purposecar (new)
## -1.8151663
## purposecar (used)
## -0.7484877
## purposefurniture/equipment
## -0.8546651
## purposeradio/television
## -0.5386350
## purposedomestic appliances
## -0.0501370
## purposerepairs
## 0.1121867
## purposevacation
## -1.5563986
## purposeretraining
## -0.7033181
## purposebusiness
## -1.4907164
## savings... < 100 DM
## -0.4901587
## savings100 <= ... < 500 DM
## -0.7474129
## savings500 <= ... < 1000 DM
## -1.3447793
## savings... >= 1000 DM
## -0.7753959
## status... < 0 DM
## -0.3486837
## status0<= ... < 200 DM
## -1.0402232
## status... >= 200 DM / salary for at least 1 year
## -1.6004925
## telephoneyes (under customer name)
## -0.4283926
##
## Degrees of Freedom: 799 Total (i.e. Null); 745 Residual
## Null Deviance: 979.1
## Residual Deviance: 699.2 AIC: 809.2class(learner_logreg$model)## [1] "glm" "lm"summary(learner_logreg$model)##
## Call:
## stats::glm(formula = task$formula(), family = "binomial", data = data,
## model = FALSE)
##
## Deviance Residuals:
## Min 1Q Median 3Q Max
## -1.9791 -0.6701 -0.3592 0.6563 2.6787
##
## Coefficients:
## Estimate Std. Error
## (Intercept) 5.611e-01 1.412e+00
## age -2.388e-02 1.089e-02
## amount 1.123e-04 5.227e-05
## credit_historycritical account/other credits elsewhere 2.138e-01 6.667e-01
## credit_historyno credits taken/all credits paid back duly -8.775e-01 5.193e-01
## credit_historyexisting credits paid back duly till now -1.308e+00 5.564e-01
## credit_historyall credits at this bank paid back duly -1.720e+00 5.216e-01
## duration 2.856e-02 1.058e-02
## employment_duration< 1 yr -1.547e-01 5.112e-01
## employment_duration1 <= ... < 4 yrs -6.749e-01 4.864e-01
## employment_duration4 <= ... < 7 yrs -1.195e+00 5.285e-01
## employment_duration>= 7 yrs -5.471e-01 4.864e-01
## foreign_workerno 1.730e+00 8.049e-01
## housingrent -4.749e-01 2.671e-01
## housingown -3.316e-01 5.250e-01
## installment_rate.L 6.429e-01 2.464e-01
## installment_rate.Q -3.877e-02 2.285e-01
## installment_rate.C 6.888e-03 2.306e-01
## jobunskilled - resident 1.041e+00 7.744e-01
## jobskilled employee/official 9.969e-01 7.499e-01
## jobmanager/self-empl./highly qualif. employee 1.110e+00 7.766e-01
## number_credits.L 2.710e-01 8.085e-01
## number_credits.Q -2.504e-01 7.047e-01
## number_credits.C 7.910e-02 5.735e-01
## other_debtorsco-applicant 1.233e+00 4.564e-01
## other_debtorsguarantor -6.259e-01 4.758e-01
## other_installment_plansstores 4.315e-01 4.982e-01
## other_installment_plansnone -2.644e-01 2.878e-01
## people_liable0 to 2 -2.153e-01 2.807e-01
## personal_status_sexfemale : non-single or male : single -4.633e-01 4.752e-01
## personal_status_sexmale : married/widowed -9.369e-01 4.630e-01
## personal_status_sexfemale : single -2.847e-01 5.423e-01
## present_residence.L 3.190e-01 2.490e-01
## present_residence.Q -5.087e-01 2.298e-01
## present_residence.C 3.792e-01 2.276e-01
## propertycar or other 2.953e-01 2.909e-01
## propertybuilding soc. savings agr./life insurance 4.808e-01 2.739e-01
## propertyreal estate 8.989e-01 4.647e-01
## purposecar (new) -1.815e+00 4.326e-01
## purposecar (used) -7.485e-01 3.001e-01
## purposefurniture/equipment -8.547e-01 2.839e-01
## purposeradio/television -5.386e-01 1.084e+00
## purposedomestic appliances -5.014e-02 5.877e-01
## purposerepairs 1.122e-01 4.316e-01
## purposevacation -1.556e+00 1.186e+00
## purposeretraining -7.033e-01 3.713e-01
## purposebusiness -1.491e+00 1.026e+00
## savings... < 100 DM -4.902e-01 3.219e-01
## savings100 <= ... < 500 DM -7.474e-01 5.195e-01
## savings500 <= ... < 1000 DM -1.345e+00 6.057e-01
## savings... >= 1000 DM -7.754e-01 2.940e-01
## status... < 0 DM -3.487e-01 2.542e-01
## status0<= ... < 200 DM -1.040e+00 4.324e-01
## status... >= 200 DM / salary for at least 1 year -1.600e+00 2.645e-01
## telephoneyes (under customer name) -4.284e-01 2.336e-01
## z value Pr(>|z|)
## (Intercept) 0.397 0.691086
## age -2.193 0.028312 *
## amount 2.149 0.031632 *
## credit_historycritical account/other credits elsewhere 0.321 0.748448
## credit_historyno credits taken/all credits paid back duly -1.690 0.091112 .
## credit_historyexisting credits paid back duly till now -2.350 0.018771 *
## credit_historyall credits at this bank paid back duly -3.298 0.000974 ***
## duration 2.700 0.006941 **
## employment_duration< 1 yr -0.303 0.762121
## employment_duration1 <= ... < 4 yrs -1.388 0.165257
## employment_duration4 <= ... < 7 yrs -2.261 0.023784 *
## employment_duration>= 7 yrs -1.125 0.260711
## foreign_workerno 2.150 0.031580 *
## housingrent -1.778 0.075383 .
## housingown -0.632 0.527662
## installment_rate.L 2.610 0.009066 **
## installment_rate.Q -0.170 0.865277
## installment_rate.C 0.030 0.976171
## jobunskilled - resident 1.344 0.178819
## jobskilled employee/official 1.329 0.183714
## jobmanager/self-empl./highly qualif. employee 1.430 0.152781
## number_credits.L 0.335 0.737518
## number_credits.Q -0.355 0.722350
## number_credits.C 0.138 0.890294
## other_debtorsco-applicant 2.702 0.006887 **
## other_debtorsguarantor -1.315 0.188356
## other_installment_plansstores 0.866 0.386441
## other_installment_plansnone -0.919 0.358214
## people_liable0 to 2 -0.767 0.443185
## personal_status_sexfemale : non-single or male : single -0.975 0.329590
## personal_status_sexmale : married/widowed -2.023 0.043022 *
## personal_status_sexfemale : single -0.525 0.599597
## present_residence.L 1.281 0.200211
## present_residence.Q -2.213 0.026870 *
## present_residence.C 1.666 0.095625 .
## propertycar or other 1.015 0.310035
## propertybuilding soc. savings agr./life insurance 1.755 0.079176 .
## propertyreal estate 1.935 0.053051 .
## purposecar (new) -4.195 2.72e-05 ***
## purposecar (used) -2.494 0.012629 *
## purposefurniture/equipment -3.011 0.002606 **
## purposeradio/television -0.497 0.619296
## purposedomestic appliances -0.085 0.932015
## purposerepairs 0.260 0.794895
## purposevacation -1.312 0.189427
## purposeretraining -1.894 0.058172 .
## purposebusiness -1.452 0.146437
## savings... < 100 DM -1.523 0.127845
## savings100 <= ... < 500 DM -1.439 0.150209
## savings500 <= ... < 1000 DM -2.220 0.026398 *
## savings... >= 1000 DM -2.638 0.008351 **
## status... < 0 DM -1.372 0.170146
## status0<= ... < 200 DM -2.405 0.016151 *
## status... >= 200 DM / salary for at least 1 year -6.050 1.44e-09 ***
## telephoneyes (under customer name) -1.834 0.066633 .
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## (Dispersion parameter for binomial family taken to be 1)
##
## Null deviance: 979.07 on 799 degrees of freedom
## Residual deviance: 699.21 on 745 degrees of freedom
## AIC: 809.21
##
## Number of Fisher Scoring iterations: 5其次，建立隨機森林模型。匯入ranger套件，並以lrn()方法建立隨機森林模型物件。 另依據permutation feature importance找出重要的屬性(feature)。
library("ranger")
learner_rf = lrn("classif.ranger", importance = "permutation")
learner_rf$train(task, row_ids = train_set)取得各變數的重要性數值。可與邏輯斯迴歸模型有顯著係數的變數比較，這些變數的重要性是否也較高？
learner_rf$importance()## status duration amount
## 0.0297877482 0.0186788980 0.0151902438
## credit_history age property
## 0.0125123267 0.0084049324 0.0070544413
## savings employment_duration purpose
## 0.0056303670 0.0055769788 0.0030500192
## installment_rate housing other_debtors
## 0.0027879024 0.0027855027 0.0027315671
## number_credits present_residence other_installment_plans
## 0.0024010749 0.0019706516 0.0012121633
## job telephone people_liable
## 0.0011598856 0.0009207099 0.0007531060
## personal_status_sex foreign_worker
## 0.0006056635 0.0003388003以ggplot2繪製重要性數值的長條圖。
importance = as.data.table(learner_rf$importance(), keep.rownames = TRUE)
colnames(importance) = c("Feature", "Importance")
ggplot(importance, aes(x = reorder(Feature, Importance), y = Importance)) +
geom_col() + coord_flip() + xlab("")
2.4 Prediction
當完成邏輯斯迴歸與隨機森林的訓練樣本模型後，利用predict()方法預測測試資料的信用風險
prediction_logreg = learner_logreg$predict(task, row_ids = test_set)
prediction_rf = learner_rf$predict(task, row_ids = test_set)
prediction_logreg## <PredictionClassif> for 200 observations:
## row_ids truth response
## 8 good good
## 18 good good
## 20 good good
## ---
## 980 bad good
## 984 bad bad
## 994 bad goodprediction_rf## <PredictionClassif> for 200 observations:
## row_ids truth response
## 8 good good
## 18 good good
## 20 good good
## ---
## 980 bad good
## 984 bad good
## 994 bad good利用模型物件的confusion屬性計算confusion matrix
prediction_logreg$confusion## truth
## response bad good
## bad 29 21
## good 30 120prediction_rf$confusion## truth
## response bad good
## bad 22 13
## good 37 128除了預測二元變數的類別(good or bad，設定predict_type = “response”), 邏輯斯迴歸模型也可以進行“不確定性”的預測。 設定模型的predict_type屬性為prob
learner_logreg$predict_type = "prob"
learner_logreg$predict(task, row_ids = test_set)## <PredictionClassif> for 200 observations:
## row_ids truth response prob.bad prob.good
## 8 good good 0.04835367 0.9516463
## 18 good good 0.29180662 0.7081934
## 20 good good 0.08671436 0.9132856
## ---
## 980 bad good 0.18793240 0.8120676
## 984 bad bad 0.80021069 0.1997893
## 994 bad good 0.45282738 0.54717262.5 Performance Evaluation
通常我們會用一組新的資料驗證模型的成效。 常見的做法是將資料分割為訓練樣本及測試樣本。 若只做一次樣本分割，利用訓練樣本建置一個模型，再利用測試樣本評估模型的有效性。
resampling = rsmp("holdout", ratio = 4/5)
print(resampling)## <ResamplingHoldout> with 1 iterations
## * Instantiated: FALSE
## * Parameters: ratio=0.8利用rsmp()方法分割樣本，“holdhout”表示僅做一次分割，ratio為訓練樣本數佔總樣本數的比重。 計算分類錯誤率(classification error, ce)，數值愈小愈好。
res = resample(task, learner = learner_logreg, resampling = resampling)## INFO [13:24:37.043] [mlr3] Applying learner 'classif.log_reg' on task 'GermanCredit' (iter 1/1)res## <ResampleResult> of 1 iterations
## * Task: GermanCredit
## * Learner: classif.log_reg
## * Warnings: 0 in 0 iterations
## * Errors: 0 in 0 iterationsres$aggregate()## classif.ce
## 0.235然而一般而言，不會只利用一組樣本建置一個模型， 而是會分割多組樣本，建立並測試數個模型，再計算模型的平均分類錯誤率。 例如：進行10次重複抽樣，計算10個測試樣本模型的平均分類錯誤率。
resampling = rsmp("subsampling", repeats = 10, ratio = 4/5)
rr = resample(task, learner = learner_logreg, resampling = resampling)## INFO [13:24:37.257] [mlr3] Applying learner 'classif.log_reg' on task 'GermanCredit' (iter 1/10)
## INFO [13:24:37.299] [mlr3] Applying learner 'classif.log_reg' on task 'GermanCredit' (iter 5/10)
## INFO [13:24:37.341] [mlr3] Applying learner 'classif.log_reg' on task 'GermanCredit' (iter 9/10)
## INFO [13:24:37.373] [mlr3] Applying learner 'classif.log_reg' on task 'GermanCredit' (iter 3/10)
## INFO [13:24:37.476] [mlr3] Applying learner 'classif.log_reg' on task 'GermanCredit' (iter 6/10)
## INFO [13:24:37.514] [mlr3] Applying learner 'classif.log_reg' on task 'GermanCredit' (iter 2/10)
## INFO [13:24:37.581] [mlr3] Applying learner 'classif.log_reg' on task 'GermanCredit' (iter 7/10)
## INFO [13:24:37.622] [mlr3] Applying learner 'classif.log_reg' on task 'GermanCredit' (iter 8/10)
## INFO [13:24:37.666] [mlr3] Applying learner 'classif.log_reg' on task 'GermanCredit' (iter 10/10)
## INFO [13:24:37.714] [mlr3] Applying learner 'classif.log_reg' on task 'GermanCredit' (iter 4/10)rr$aggregate()## classif.ce
## 0.252或以交叉驗證cross-validation的方式。
resampling = rsmp("cv", folds = 5)
rr = resample(task, learner = learner_logreg, resampling = resampling)## INFO [13:24:37.882] [mlr3] Applying learner 'classif.log_reg' on task 'GermanCredit' (iter 2/5)
## INFO [13:24:37.925] [mlr3] Applying learner 'classif.log_reg' on task 'GermanCredit' (iter 5/5)
## INFO [13:24:37.959] [mlr3] Applying learner 'classif.log_reg' on task 'GermanCredit' (iter 4/5)
## INFO [13:24:38.003] [mlr3] Applying learner 'classif.log_reg' on task 'GermanCredit' (iter 1/5)
## INFO [13:24:38.038] [mlr3] Applying learner 'classif.log_reg' on task 'GermanCredit' (iter 3/5)rr$aggregate()## classif.ce
## 0.25另外也可利用其他衡量指標評估模型，例如false positive rate、false negative rate。
measures = msrs(c("classif.fpr", "classif.fnr"))
rr$aggregate(measures)## classif.fpr classif.fnr
## 0.1370517 0.5127113mlr3套件提供多種重複抽樣方法以及模型衡量指標。
1. 重複抽樣方法
mlr_resamplings## <DictionaryResampling> with 9 stored values
## Keys: bootstrap, custom, custom_cv, cv, holdout, insample, loo,
## repeated_cv, subsampling- 模型衡量指標
mlr_measures## <DictionaryMeasure> with 85 stored values
## Keys: aic, bic, classif.acc, classif.auc, classif.bacc, classif.bbrier,
## classif.ce, classif.costs, classif.dor, classif.fbeta, classif.fdr,
## classif.fn, classif.fnr, classif.fomr, classif.fp, classif.fpr,
## classif.logloss, classif.mbrier, classif.mcc, classif.npv,
## classif.ppv, classif.prauc, classif.precision, classif.recall,
## classif.sensitivity, classif.specificity, classif.tn, classif.tnr,
## classif.tp, classif.tpr, clust.ch, clust.db, clust.dunn,
## clust.silhouette, clust.wss, debug, dens.logloss, oob_error,
## regr.bias, regr.ktau, regr.mae, regr.mape, regr.maxae, regr.medae,
## regr.medse, regr.mse, regr.msle, regr.pbias, regr.rae, regr.rmse,
## regr.rmsle, regr.rrse, regr.rse, regr.rsq, regr.sae, regr.smape,
## regr.srho, regr.sse, selected_features, surv.brier, surv.calib_alpha,
## surv.calib_beta, surv.chambless_auc, surv.cindex, surv.dcalib,
## surv.graf, surv.hung_auc, surv.intlogloss, surv.logloss, surv.mae,
## surv.mse, surv.nagelk_r2, surv.oquigley_r2, surv.rmse, surv.schmid,
## surv.song_auc, surv.song_tnr, surv.song_tpr, surv.uno_auc,
## surv.uno_tnr, surv.uno_tpr, surv.xu_r2, time_both, time_predict,
## time_train例如：計算AUC、分類錯誤率(CE)、false positive rate (fpr)、false negative rate (fnr)。
measures = msrs(c("classif.auc","classif.ce","classif.fpr", "classif.fnr"))
rr$aggregate(measures)## classif.auc classif.ce classif.fpr classif.fnr
## 0.7756007 0.2500000 0.1370517 0.51271132.6 模型比較
可以利用以下程式自動執行邏輯斯迴歸與隨機森林的模型比較
learners = lrns(c("classif.log_reg", "classif.ranger"), predict_type = "prob")
grid = benchmark_grid(
tasks = task,
learners = learners,
resamplings = rsmp("cv", folds = 10)
)
bmr = benchmark(grid)## INFO [13:24:38.563] [mlr3] Running benchmark with 20 resampling iterations
## INFO [13:24:38.569] [mlr3] Applying learner 'classif.log_reg' on task 'GermanCredit' (iter 1/10)
## INFO [13:24:38.620] [mlr3] Applying learner 'classif.log_reg' on task 'GermanCredit' (iter 4/10)
## INFO [13:24:38.884] [mlr3] Applying learner 'classif.log_reg' on task 'GermanCredit' (iter 3/10)
## INFO [13:24:38.926] [mlr3] Applying learner 'classif.ranger' on task 'GermanCredit' (iter 7/10)
## INFO [13:24:39.217] [mlr3] Applying learner 'classif.log_reg' on task 'GermanCredit' (iter 8/10)
## INFO [13:24:39.253] [mlr3] Applying learner 'classif.ranger' on task 'GermanCredit' (iter 8/10)
## INFO [13:24:39.528] [mlr3] Applying learner 'classif.log_reg' on task 'GermanCredit' (iter 7/10)
## INFO [13:24:39.572] [mlr3] Applying learner 'classif.ranger' on task 'GermanCredit' (iter 6/10)
## INFO [13:24:39.845] [mlr3] Applying learner 'classif.ranger' on task 'GermanCredit' (iter 5/10)
## INFO [13:24:40.113] [mlr3] Applying learner 'classif.ranger' on task 'GermanCredit' (iter 3/10)
## INFO [13:24:40.383] [mlr3] Applying learner 'classif.ranger' on task 'GermanCredit' (iter 2/10)
## INFO [13:24:40.662] [mlr3] Applying learner 'classif.log_reg' on task 'GermanCredit' (iter 5/10)
## INFO [13:24:40.702] [mlr3] Applying learner 'classif.ranger' on task 'GermanCredit' (iter 10/10)
## INFO [13:24:40.985] [mlr3] Applying learner 'classif.ranger' on task 'GermanCredit' (iter 9/10)
## INFO [13:24:41.254] [mlr3] Applying learner 'classif.log_reg' on task 'GermanCredit' (iter 6/10)
## INFO [13:24:41.296] [mlr3] Applying learner 'classif.ranger' on task 'GermanCredit' (iter 1/10)
## INFO [13:24:41.559] [mlr3] Applying learner 'classif.ranger' on task 'GermanCredit' (iter 4/10)
## INFO [13:24:41.831] [mlr3] Applying learner 'classif.log_reg' on task 'GermanCredit' (iter 10/10)
## INFO [13:24:41.903] [mlr3] Applying learner 'classif.log_reg' on task 'GermanCredit' (iter 9/10)
## INFO [13:24:41.936] [mlr3] Applying learner 'classif.log_reg' on task 'GermanCredit' (iter 2/10)
## INFO [13:24:41.980] [mlr3] Finished benchmark利用分類錯誤率與AUC兩個指標進行模型比較，兩個模型的效力接近。
measures = msrs(c("classif.ce", "classif.auc"))
performances = bmr$aggregate(measures)
performances[, c("learner_id", "classif.ce", "classif.auc")]## learner_id classif.ce classif.auc
## 1: classif.log_reg 0.248 0.7772869
## 2: classif.ranger 0.232 0.7978613