class: center, middle, title-slide # Bootstrapping ## AU STAT-427/627 ### Emil Hvitfeldt ### 2021-2-23 --- <div style = "position:fixed; visibility: hidden"> `$$\require{color}\definecolor{orange}{rgb}{1, 0.603921568627451, 0.301960784313725}$$` `$$\require{color}\definecolor{blue}{rgb}{0.301960784313725, 0.580392156862745, 1}$$` `$$\require{color}\definecolor{pink}{rgb}{0.976470588235294, 0.301960784313725, 1}$$` </div> <script type="text/x-mathjax-config"> MathJax.Hub.Config({ TeX: { Macros: { orange: ["{\\color{orange}{#1}}", 1], blue: ["{\\color{blue}{#1}}", 1], pink: ["{\\color{pink}{#1}}", 1] }, loader: {load: ['[tex]/color']}, tex: {packages: {'[+]': ['color']}} } }); </script> <style> .orange {color: #FF9A4D;} .blue {color: #4D94FF;} .pink {color: #F94DFF;} </style> # Bootstrapping Last week we looked at a couple of different Cross-Validation methods - Leave-One-Out Cross-Validation (LOOCV) - K-fold Cross-Validation --- # Bootstrapping This week we will look at .orange[Bootstrapping] This is a technique that uses resampling with replacement to estimate the uncertainty with a given estimator or statistical learning method It is a powerful and general statistical tool, and can be used with most estimators/methods --- # Bootstrapping VS Cross-Validation - .blue[Cross-Validation]: provide estimates of the test error. - .orange[Bootstrap]: provides the standard error of the estimates. --- # Motivation .pull-left[ Suppose We have an estimate we want to find out how variable it is. We could collect data `\(n\)` times and calculate the estimates. We then have a distribution of and can see the how well it is doing ] .pull-right[ 1000 realizations .pink[pink] line is the mean .orange[orange] lines 95% percent quantiles <img src="index_files/figure-html/unnamed-chunk-11-1.png" width="700px" style="display: block; margin: auto;" /> ] --- # Motivation ## The Problem We are not always able to conduct multiple data collections at will Sometimes for resource issues or time-sensitive data We need the different samples to come from the same underlying distribution --- # Motivation ## The Solution We take our one data set and resample the rows with replacement. This allows us to get new data sets that approximate the original data set If the original data set is close to the underlying true distribution then the resampled data sets are also approximations of the true underlying distribution --- # Example From "An Introduction to Statistical Learning" <img src="index_files/figure-html/unnamed-chunk-13-1.png" width="700px" style="display: block; margin: auto;" /> --- # Example Visualizing multiple bootstrappings <img src="index_files/figure-html/unnamed-chunk-14-1.png" width="700px" style="display: block; margin: auto;" /> --- # Example We want to minimize `$$\alpha = \dfrac{\sigma^2_Y - \sigma_{XY}}{\sigma^2_X + \sigma^2_Y - 2\sigma_{XY}}$$` Where `\(\sigma^2_X = \text{Var}(X)\)`, `\(\sigma^2_Y = \text{Var}(Y)\)`, and `\(\sigma_{XY} = \text{Cov}(X, Y)\)` --- # Bootstrapping results ``` ## # A tibble: 1,000 x 5 ## id var_x var_y cov_xy estimate ## * <chr> <dbl> <dbl> <dbl> <dbl> ## 1 Bootstrap0001 1.04 1.33 0.583 0.618 ## 2 Bootstrap0002 0.958 1.21 0.416 0.596 ## 3 Bootstrap0003 0.950 1.44 0.479 0.671 ## 4 Bootstrap0004 0.909 1.27 0.326 0.617 ## 5 Bootstrap0005 1.05 1.24 0.413 0.563 ## 6 Bootstrap0006 0.747 1.52 0.386 0.759 ## 7 Bootstrap0007 0.899 1.33 0.488 0.673 ## 8 Bootstrap0008 0.897 1.43 0.515 0.705 ## 9 Bootstrap0009 1.21 1.29 0.531 0.527 ## 10 Bootstrap0010 0.879 1.06 0.381 0.576 ## # … with 990 more rows ``` --- # Bootstrapping results With `\(n = 100\)` in original data set <img src="index_files/figure-html/unnamed-chunk-16-1.png" width="700px" style="display: block; margin: auto;" /> --- # Bootstrapping results With `\(n = 1000\)` in original data set <img src="index_files/figure-html/unnamed-chunk-17-1.png" width="700px" style="display: block; margin: auto;" /> --- # Bootstrapping results With `\(n = 10000\)` in original data set <img src="index_files/figure-html/unnamed-chunk-18-1.png" width="700px" style="display: block; margin: auto;" /> --- # What size of bootstrappings are we looking for? We are using bootstrapping sizes to be the same size of to get a comparatively estimate of the variation --- # Rsample We are back with `rsample` and the `mtcars` data set ```r library(rsample) mtcars ``` ``` ## mpg cyl disp hp drat wt qsec vs am gear carb ## Mazda RX4 21.0 6 160.0 110 3.90 2.620 16.46 0 1 4 4 ## Mazda RX4 Wag 21.0 6 160.0 110 3.90 2.875 17.02 0 1 4 4 ## Datsun 710 22.8 4 108.0 93 3.85 2.320 18.61 1 1 4 1 ## Hornet 4 Drive 21.4 6 258.0 110 3.08 3.215 19.44 1 0 3 1 ## Hornet Sportabout 18.7 8 360.0 175 3.15 3.440 17.02 0 0 3 2 ## Valiant 18.1 6 225.0 105 2.76 3.460 20.22 1 0 3 1 ## Duster 360 14.3 8 360.0 245 3.21 3.570 15.84 0 0 3 4 ## Merc 240D 24.4 4 146.7 62 3.69 3.190 20.00 1 0 4 2 ## Merc 230 22.8 4 140.8 95 3.92 3.150 22.90 1 0 4 2 ## Merc 280 19.2 6 167.6 123 3.92 3.440 18.30 1 0 4 4 ## Merc 280C 17.8 6 167.6 123 3.92 3.440 18.90 1 0 4 4 ## Merc 450SE 16.4 8 275.8 180 3.07 4.070 17.40 0 0 3 3 ## Merc 450SL 17.3 8 275.8 180 3.07 3.730 17.60 0 0 3 3 ## Merc 450SLC 15.2 8 275.8 180 3.07 3.780 18.00 0 0 3 3 ## Cadillac Fleetwood 10.4 8 472.0 205 2.93 5.250 17.98 0 0 3 4 ## Lincoln Continental 10.4 8 460.0 215 3.00 5.424 17.82 0 0 3 4 ## Chrysler Imperial 14.7 8 440.0 230 3.23 5.345 17.42 0 0 3 4 ## Fiat 128 32.4 4 78.7 66 4.08 2.200 19.47 1 1 4 1 ## Honda Civic 30.4 4 75.7 52 4.93 1.615 18.52 1 1 4 2 ## Toyota Corolla 33.9 4 71.1 65 4.22 1.835 19.90 1 1 4 1 ## Toyota Corona 21.5 4 120.1 97 3.70 2.465 20.01 1 0 3 1 ## Dodge Challenger 15.5 8 318.0 150 2.76 3.520 16.87 0 0 3 2 ## AMC Javelin 15.2 8 304.0 150 3.15 3.435 17.30 0 0 3 2 ## Camaro Z28 13.3 8 350.0 245 3.73 3.840 15.41 0 0 3 4 ## Pontiac Firebird 19.2 8 400.0 175 3.08 3.845 17.05 0 0 3 2 ## Fiat X1-9 27.3 4 79.0 66 4.08 1.935 18.90 1 1 4 1 ## Porsche 914-2 26.0 4 120.3 91 4.43 2.140 16.70 0 1 5 2 ## Lotus Europa 30.4 4 95.1 113 3.77 1.513 16.90 1 1 5 2 ## Ford Pantera L 15.8 8 351.0 264 4.22 3.170 14.50 0 1 5 4 ## Ferrari Dino 19.7 6 145.0 175 3.62 2.770 15.50 0 1 5 6 ## Maserati Bora 15.0 8 301.0 335 3.54 3.570 14.60 0 1 5 8 ## Volvo 142E 21.4 4 121.0 109 4.11 2.780 18.60 1 1 4 2 ``` --- # Rsample .pull-left[ We can use the `bootstraps()` function on a data.frame to create a `bootstraps` object ```r mtcars_boots <- bootstraps(mtcars, times = 100) mtcars_boots ``` ] .pull-right[ ``` ## # Bootstrap sampling ## # A tibble: 100 x 2 ## splits id ## <list> <chr> ## 1 <split [32/12]> Bootstrap001 ## 2 <split [32/11]> Bootstrap002 ## 3 <split [32/12]> Bootstrap003 ## 4 <split [32/9]> Bootstrap004 ## 5 <split [32/10]> Bootstrap005 ## 6 <split [32/11]> Bootstrap006 ## 7 <split [32/12]> Bootstrap007 ## 8 <split [32/11]> Bootstrap008 ## 9 <split [32/11]> Bootstrap009 ## 10 <split [32/11]> Bootstrap010 ## # … with 90 more rows ``` ] --- # Rsample An under the hood, we have 100 analysis/assesment splits similar to `initial_split()` and `vfold_cv()` .pull-left[ ```r mtcars_boots <- bootstraps(mtcars, times = 100) mtcars_boots$splits ``` ] .pull-right[ ``` ## [[1]] ## <Analysis/Assess/Total> ## <32/12/32> ## ## [[2]] ## <Analysis/Assess/Total> ## <32/12/32> ## ## [[3]] ## <Analysis/Assess/Total> ## <32/9/32> ## ## [[4]] ## <Analysis/Assess/Total> ## <32/14/32> ## ## [[5]] ## <Analysis/Assess/Total> ## <32/16/32> ## ## [[6]] ## <Analysis/Assess/Total> ## <32/13/32> ## ## [[7]] ## <Analysis/Assess/Total> ## <32/15/32> ## ## [[8]] ## <Analysis/Assess/Total> ## <32/12/32> ## ## [[9]] ## <Analysis/Assess/Total> ## <32/14/32> ## ## [[10]] ## <Analysis/Assess/Total> ## <32/11/32> ## ## [[11]] ## <Analysis/Assess/Total> ## <32/14/32> ## ## [[12]] ## <Analysis/Assess/Total> ## <32/10/32> ## ## [[13]] ## <Analysis/Assess/Total> ## <32/9/32> ## ## [[14]] ## <Analysis/Assess/Total> ## <32/14/32> ## ## [[15]] ## <Analysis/Assess/Total> ## <32/10/32> ## ## [[16]] ## <Analysis/Assess/Total> ## <32/10/32> ## ## [[17]] ## <Analysis/Assess/Total> ## <32/8/32> ## ## [[18]] ## <Analysis/Assess/Total> ## <32/14/32> ## ## [[19]] ## <Analysis/Assess/Total> ## <32/10/32> ## ## [[20]] ## <Analysis/Assess/Total> ## <32/14/32> ## ## [[21]] ## <Analysis/Assess/Total> ## <32/12/32> ## ## [[22]] ## <Analysis/Assess/Total> ## <32/14/32> ## ## [[23]] ## <Analysis/Assess/Total> ## <32/11/32> ## ## [[24]] ## <Analysis/Assess/Total> ## <32/11/32> ## ## [[25]] ## <Analysis/Assess/Total> ## <32/12/32> ## ## [[26]] ## <Analysis/Assess/Total> ## <32/16/32> ## ## [[27]] ## <Analysis/Assess/Total> ## <32/10/32> ## ## [[28]] ## <Analysis/Assess/Total> ## <32/12/32> ## ## [[29]] ## <Analysis/Assess/Total> ## <32/12/32> ## ## [[30]] ## <Analysis/Assess/Total> ## <32/13/32> ## ## [[31]] ## <Analysis/Assess/Total> ## <32/8/32> ## ## [[32]] ## <Analysis/Assess/Total> ## <32/10/32> ## ## [[33]] ## <Analysis/Assess/Total> ## <32/12/32> ## ## [[34]] ## <Analysis/Assess/Total> ## <32/9/32> ## ## [[35]] ## <Analysis/Assess/Total> ## <32/9/32> ## ## [[36]] ## <Analysis/Assess/Total> ## <32/12/32> ## ## [[37]] ## <Analysis/Assess/Total> ## <32/10/32> ## ## [[38]] ## <Analysis/Assess/Total> ## <32/13/32> ## ## [[39]] ## <Analysis/Assess/Total> ## <32/11/32> ## ## [[40]] ## <Analysis/Assess/Total> ## <32/13/32> ## ## [[41]] ## <Analysis/Assess/Total> ## <32/13/32> ## ## [[42]] ## <Analysis/Assess/Total> ## <32/11/32> ## ## [[43]] ## <Analysis/Assess/Total> ## <32/14/32> ## ## [[44]] ## <Analysis/Assess/Total> ## <32/11/32> ## ## [[45]] ## <Analysis/Assess/Total> ## <32/13/32> ## ## [[46]] ## <Analysis/Assess/Total> ## <32/12/32> ## ## [[47]] ## <Analysis/Assess/Total> ## <32/8/32> ## ## [[48]] ## <Analysis/Assess/Total> ## <32/13/32> ## ## [[49]] ## <Analysis/Assess/Total> ## <32/13/32> ## ## [[50]] ## <Analysis/Assess/Total> ## <32/9/32> ## ## [[51]] ## <Analysis/Assess/Total> ## <32/13/32> ## ## [[52]] ## <Analysis/Assess/Total> ## <32/12/32> ## ## [[53]] ## <Analysis/Assess/Total> ## <32/11/32> ## ## [[54]] ## <Analysis/Assess/Total> ## <32/13/32> ## ## [[55]] ## <Analysis/Assess/Total> ## <32/13/32> ## ## [[56]] ## <Analysis/Assess/Total> ## <32/12/32> ## ## [[57]] ## <Analysis/Assess/Total> ## <32/11/32> ## ## [[58]] ## <Analysis/Assess/Total> ## <32/11/32> ## ## [[59]] ## <Analysis/Assess/Total> ## <32/8/32> ## ## [[60]] ## <Analysis/Assess/Total> ## <32/10/32> ## ## [[61]] ## <Analysis/Assess/Total> ## <32/11/32> ## ## [[62]] ## <Analysis/Assess/Total> ## <32/11/32> ## ## [[63]] ## <Analysis/Assess/Total> ## <32/10/32> ## ## [[64]] ## <Analysis/Assess/Total> ## <32/11/32> ## ## [[65]] ## <Analysis/Assess/Total> ## <32/11/32> ## ## [[66]] ## <Analysis/Assess/Total> ## <32/13/32> ## ## [[67]] ## <Analysis/Assess/Total> ## <32/10/32> ## ## [[68]] ## <Analysis/Assess/Total> ## <32/9/32> ## ## [[69]] ## <Analysis/Assess/Total> ## <32/10/32> ## ## [[70]] ## <Analysis/Assess/Total> ## <32/12/32> ## ## [[71]] ## <Analysis/Assess/Total> ## <32/14/32> ## ## [[72]] ## <Analysis/Assess/Total> ## <32/8/32> ## ## [[73]] ## <Analysis/Assess/Total> ## <32/12/32> ## ## [[74]] ## <Analysis/Assess/Total> ## <32/13/32> ## ## [[75]] ## <Analysis/Assess/Total> ## <32/14/32> ## ## [[76]] ## <Analysis/Assess/Total> ## <32/9/32> ## ## [[77]] ## <Analysis/Assess/Total> ## <32/12/32> ## ## [[78]] ## <Analysis/Assess/Total> ## <32/13/32> ## ## [[79]] ## <Analysis/Assess/Total> ## <32/11/32> ## ## [[80]] ## <Analysis/Assess/Total> ## <32/11/32> ## ## [[81]] ## <Analysis/Assess/Total> ## <32/12/32> ## ## [[82]] ## <Analysis/Assess/Total> ## <32/12/32> ## ## [[83]] ## <Analysis/Assess/Total> ## <32/9/32> ## ## [[84]] ## <Analysis/Assess/Total> ## <32/11/32> ## ## [[85]] ## <Analysis/Assess/Total> ## <32/10/32> ## ## [[86]] ## <Analysis/Assess/Total> ## <32/11/32> ## ## [[87]] ## <Analysis/Assess/Total> ## <32/10/32> ## ## [[88]] ## <Analysis/Assess/Total> ## <32/11/32> ## ## [[89]] ## <Analysis/Assess/Total> ## <32/15/32> ## ## [[90]] ## <Analysis/Assess/Total> ## <32/13/32> ## ## [[91]] ## <Analysis/Assess/Total> ## <32/10/32> ## ## [[92]] ## <Analysis/Assess/Total> ## <32/13/32> ## ## [[93]] ## <Analysis/Assess/Total> ## <32/13/32> ## ## [[94]] ## <Analysis/Assess/Total> ## <32/13/32> ## ## [[95]] ## <Analysis/Assess/Total> ## <32/10/32> ## ## [[96]] ## <Analysis/Assess/Total> ## <32/14/32> ## ## [[97]] ## <Analysis/Assess/Total> ## <32/10/32> ## ## [[98]] ## <Analysis/Assess/Total> ## <32/13/32> ## ## [[99]] ## <Analysis/Assess/Total> ## <32/12/32> ## ## [[100]] ## <Analysis/Assess/Total> ## <32/11/32> ``` ] --- # Using resamples in action We start by creating a linear regression specification and create a `workflow` object with `workflows()` ```r library(parsnip) linear_spec <- linear_reg() %>% set_mode("regression") %>% set_engine("lm") library(workflows) linear_wf <- workflow() %>% add_model(linear_spec) %>% add_formula(mpg ~ disp + hp + wt) ``` --- # Tune We can use `fit_resamples()` to fit the workflow we created within each bootstrap ```r library(tune) linear_fold_fits <- fit_resamples( linear_wf, resamples = mtcars_boots ) ``` --- # Tune The results of this resampling comes as a data.frame ```r linear_fold_fits ``` ``` ## # Resampling results ## # Bootstrap sampling ## # A tibble: 100 x 4 ## splits id .metrics .notes ## <list> <chr> <list> <list> ## 1 <split [32/12]> Bootstrap001 <tibble [2 × 4]> <tibble [0 × 1]> ## 2 <split [32/12]> Bootstrap002 <tibble [2 × 4]> <tibble [0 × 1]> ## 3 <split [32/9]> Bootstrap003 <tibble [2 × 4]> <tibble [0 × 1]> ## 4 <split [32/14]> Bootstrap004 <tibble [2 × 4]> <tibble [0 × 1]> ## 5 <split [32/16]> Bootstrap005 <tibble [2 × 4]> <tibble [0 × 1]> ## 6 <split [32/13]> Bootstrap006 <tibble [2 × 4]> <tibble [0 × 1]> ## 7 <split [32/15]> Bootstrap007 <tibble [2 × 4]> <tibble [0 × 1]> ## 8 <split [32/12]> Bootstrap008 <tibble [2 × 4]> <tibble [0 × 1]> ## 9 <split [32/14]> Bootstrap009 <tibble [2 × 4]> <tibble [0 × 1]> ## 10 <split [32/11]> Bootstrap010 <tibble [2 × 4]> <tibble [0 × 1]> ## # … with 90 more rows ``` --- # Tune `collect_metrics()` can be used to extract the CV estimate ```r library(tune) collect_metrics(linear_fold_fits) ``` ``` ## # A tibble: 2 x 6 ## .metric .estimator mean n std_err .config ## <chr> <chr> <dbl> <int> <dbl> <chr> ## 1 rmse standard 2.95 100 0.0633 Preprocessor1_Model1 ## 2 rsq standard 0.828 100 0.00670 Preprocessor1_Model1 ``` --- # Tune Setting `summarize = FALSE` in `collect_metrics()` Allows us the see the individual performance metrics for each fold ```r collect_metrics(linear_fold_fits, summarize = FALSE) ``` ``` ## # A tibble: 200 x 5 ## id .metric .estimator .estimate .config ## <chr> <chr> <chr> <dbl> <chr> ## 1 Bootstrap001 rmse standard 2.78 Preprocessor1_Model1 ## 2 Bootstrap001 rsq standard 0.938 Preprocessor1_Model1 ## 3 Bootstrap002 rmse standard 3.53 Preprocessor1_Model1 ## 4 Bootstrap002 rsq standard 0.752 Preprocessor1_Model1 ## 5 Bootstrap003 rmse standard 2.49 Preprocessor1_Model1 ## 6 Bootstrap003 rsq standard 0.802 Preprocessor1_Model1 ## 7 Bootstrap004 rmse standard 2.52 Preprocessor1_Model1 ## 8 Bootstrap004 rsq standard 0.811 Preprocessor1_Model1 ## 9 Bootstrap005 rmse standard 2.98 Preprocessor1_Model1 ## 10 Bootstrap005 rsq standard 0.826 Preprocessor1_Model1 ## # … with 190 more rows ```