The logitr R Package: Obtaining Willingness to Pay Estimates from Preference Space and Willingness to Pay Space Utility Models

<div class="footer-small">
<span>
https://github.com/jhelvy/2021-sawtooth-conf
</span>
</div>

---

# The `logitr` <svg style="height:0.8em;top:.04em;position:relative;fill:white;" viewBox="0 0 581 512"><path d="M581 226.6C581 119.1 450.9 32 290.5 32S0 119.1 0 226.6C0 322.4 103.3 402 239.4 418.1V480h99.1v-61.5c24.3-2.7 47.6-7.4 69.4-13.9L448 480h112l-67.4-113.7c54.5-35.4 88.4-84.9 88.4-139.7zm-466.8 14.5c0-73.5 98.9-133 220.8-133s211.9 40.7 211.9 133c0 50.1-26.5 85-70.3 106.4-2.4-1.6-4.7-2.9-6.4-3.7-10.2-5.2-27.8-10.5-27.8-10.5s86.6-6.4 86.6-92.7-90.6-87.9-90.6-87.9h-199V361c-74.1-21.5-125.2-67.1-125.2-119.9zm225.1 38.3v-55.6c57.8 0 87.8-6.8 87.8 27.3 0 36.5-38.2 28.3-87.8 28.3zm-.9 72.5H365c10.8 0 18.9 11.7 24 19.2-16.1 1.9-33 2.8-50.6 2.9v-22.1z"/></svg> Package: <a href='https://jhelvy.github.io/logitr/'><img src='images/logitr-hex.png' align="right" height="250"/></a>

## Obtaining Willingness to Pay Estimates from Preference Space and Willingness to Pay<br>Space Utility Models

### by John Paul Helveston

Sawtooth Software Conference: Turbo Choice Modeling Seminar

April 20, 2021

---

# Which would you choose?

---

# .center[Estimate marginal utilities]

</br>

```
#>               Estimate StdError    tStat pVal signif
#> price        -0.388627 0.024269 -16.0131    0    ***
#> brandhiland  -3.116712 0.144968 -21.4993    0    ***
#> brandyoplait  1.446362 0.088698  16.3066    0    ***
#> branddannon   0.644086 0.054360  11.8486    0    ***
```
]

---

# .center[Convert marginal _utilities_ to marginal _WTPs_]

</br>

```
#>               Estimate StdError    tStat pVal signif
#> brandhiland  -8.019803 0.460815 -17.4035    0    ***
#> brandyoplait  3.721723 0.440084   8.4568    0    ***
#> branddannon   1.657337 0.183712   9.0214    0    ***
```
]

---

## Alternative approach: **Estimate a WTP-Space Model**

<center>
<img src="images/wtpComputed.png" width=200></br>
<img src="images/lambda.png" width=160>
</center>
]

## "WTP Space"

<center>
<img src="images/utilityWtp2.png" width=400>
</center>
]

---

# What's the difference?

.leftcol[
## Preference Space
<center>
<img src="images/utilityPreference2Inverse.png" width=500>
</center>
.font200[&darr;]
<center>
<img src="images/wtpHatComputedInverse.png" width=200></br>
</center>
]

.rightcol[
## WTP Space
</br></br>
<center>
<img src="images/utilityWtp2Inverse.png" width=520>
</center>
]

---

## .center[Oops...WTP has undefined mean and variance]

</br>

<center>
<img src="images/utilityPreference2.png" width=500>
</center>
.center[.font200[&darr;]]
<center>
<img src="images/wtpHatComputed.png" width=200></br>
</center>
]

.rightcol60[
</br>
`$\hat{\alpha}$` & `$\hat{\beta}$` are assumed to have asymptotically normal distributions .gray[(Bockstael and Strand, 1987)]

Implies error around `$\hat{\omega}$` follows a</br>[Cauchy distribution](https://en.wikipedia.org/wiki/Cauchy_distribution)
]

---

## **Mixed logit**:
## Unreasonably large WTP variance across population

</br>

.leftcol40[
<center>
<img src="images/utilityPreference2.png" width=500>
</center>
.center[.font200[&darr;]]
<center>
<img src="images/wtpHatComputed.png" width=200></br>
</center>
]

.rightcol60[
<center>
<img src="images/betaNormal.png" width=380>
</br>
<img src="images/alphaNormal.png" width=380>
</center>
]

---

background-color: #fff
class: center

### Preference space model produces unreasonably large variance in WTP

---

# .center[I'm not the first to identify these issues]

.font70[
- **Carson and Czajkowski (2019)**. "A new baseline model for estimating willingness to pay from discrete choice models." _Journal of Environmental Economics and Management_, 95:57–61.
]]

.font70[
- **Train and Weeks (2005)**. "Discrete Choice Models in Preference and Willingness-to-Pay Space". In _Appl. Simul. Methods Environ. Resour. Econ._, Chapter 1, pages 1–16.
- **Sonnier, Ainslie, & Otter (2007)**. "Heterogeneity distributions of willingness-to-pay in choice models." _Quant. Mark. Econ._, 5(3):313–331.
]]

---

# .fancy[Practical Considerations]

---

## .fancy[Practical Considerations]

> ### WTP space models produce immediately interpretable results</br>(with correct standard errors)

```
#>               Estimate StdError    tStat pVal signif
#> lambda        0.388627 0.024341  15.9658    0    ***
#> brandhiland  -8.019803 0.460815 -17.4035    0    ***
#> brandyoplait  3.721723 0.440084   8.4568    0    ***
#> branddannon   1.657337 0.183712   9.0214    0    ***
```
]]

---

## .fancy[Practical Considerations]

> ### WTPs can be directly compared across different models</br>(even estimates from different data sets)

<center>
<img src="images/utilityPreferenceScaled2.png" width=420>
<img src="images/utilityPreference2.png" width=300>
</center>
]

<center>
<img src="images/utilityWtpScaled2.png" width=450>
<img src="images/utilityWtp2.png" width=350>
</center>
]

---

## .fancy[Practical Considerations]

> ### No theoretical basis for believing that marginal _utilities_ versus marginal _WTPs_ should follow standard distributions

---

## .center[.fancy[Practical Considerations]]

> ### .center[Neither space systematically predicts choice better]

.leftcol70[
-  **Train and Weeks (2005)** and **Sonnier et al. (2007)** found preference space model fit data better.

- **Das et al. (2009)** found nearly identical model fit on out-of-sample predictions with each model specification.
]

---

# ...but most software is built for

# not

---

# `logitr` to the rescue!

---

# The logitr <svg style="height:0.8em;top:.04em;position:relative;fill:#165CAA;" viewBox="0 0 581 512"><path d="M581 226.6C581 119.1 450.9 32 290.5 32S0 119.1 0 226.6C0 322.4 103.3 402 239.4 418.1V480h99.1v-61.5c24.3-2.7 47.6-7.4 69.4-13.9L448 480h112l-67.4-113.7c54.5-35.4 88.4-84.9 88.4-139.7zm-466.8 14.5c0-73.5 98.9-133 220.8-133s211.9 40.7 211.9 133c0 50.1-26.5 85-70.3 106.4-2.4-1.6-4.7-2.9-6.4-3.7-10.2-5.2-27.8-10.5-27.8-10.5s86.6-6.4 86.6-92.7-90.6-87.9-90.6-87.9h-199V361c-74.1-21.5-125.2-67.1-125.2-119.9zm225.1 38.3v-55.6c57.8 0 87.8-6.8 87.8 27.3 0 36.5-38.2 28.3-87.8 28.3zm-.9 72.5H365c10.8 0 18.9 11.7 24 19.2-16.1 1.9-33 2.8-50.6 2.9v-22.1z"/></svg> Package <a href='https://jhelvy.github.io/logitr/'><img src='images/logitr-hex.png' align="right" height="240"/></a>

### Estimation of multinomial and mixed logit models in with "Preference" space or "Willingness-to-pay" (WTP) space utility parameterizations.

- Homogeneous multinomial logit models.
- Heterogeneous mixed logit models (normal and log-normal parameter distributions).
- Preference & WTP space utility parameterizations.
- Optional multistart optimization loop.
- Computing and comparing WTP from preference space and WTP space models.
- Simulating expected shares.

Source code: https://github.com/jhelvy/logitr

---

# .center[Installation]

```r
install.packages("logitr")
```

The development version can be installed from GitHub:

```r
# install.packages("remotes")
remotes::install_github("jhelvy/logitr")
```
]

---

# .center[Data format]

Data must be arranged in a "long" format:

- Each row is an alternative from a choice observation. 
- Choice observations do _not_ have to be symmetric.

Required variables:

- `choiceName`: A dummy variable for the chosen alternative (`1` or `0`).
- `obsIDName`: A sequence of repeated numbers identifying each unique choice observation, e.g. `1, 1, 2, 2, 3, 3`.
- `parNames`: Any other variables to use as model covariates.

---

# .center[Data format]

```r
head(yogurt, 10)
```

```
#>    choice obsID alt price   brand
#> 1       0     1   1   8.1  dannon
#> 2       0     1   2   6.1  hiland
#> 3       1     1   3   7.9  weight
#> 4       0     1   4  10.8 yoplait
#> 5       1     2   1   9.8  dannon
#> 6       0     2   2   6.4  hiland
#> 7       0     2   3   7.5  weight
#> 8       0     2   4  10.8 yoplait
#> 9       1     3   1   9.8  dannon
#> 10      0     3   2   6.1  hiland
```
]

---

# .center[Multinomial logit in **Preference Space**]

```r
library(logitr)

mnl_pref <- logitr(
  data       = yogurt,
  choiceName = "choice",
  obsIDName  = "obsID",
  parNames   = c("price", "brand"))
)

*summary(mnl_pref)
```

<center>
<img src="images/utilityPreference2.png" width=300>
</center>
]

```
#> =================================================
#> MODEL SUMMARY: 
#>                           
#> Model Space:    Preference
#> Model Run:          1 of 1
#> Iterations:             20
#> Elapsed Time:  0h:0m:0.12s
#> Exit Status:             3
#> Weights Used?:       FALSE
#> 
#> Model Coefficients: 
#>               Estimate StdError    tStat pVal signif
#> price        -0.388627 0.024269 -16.0131    0    ***
#> brandhiland  -3.116712 0.144968 -21.4993    0    ***
#> brandyoplait  1.446362 0.088698  16.3066    0    ***
#> branddannon   0.644086 0.054360  11.8486    0    ***
#> ---
#> Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
#> 
#> Model Fit Values: 
#>                                      
#> Log.Likelihood.         -2665.1101915
#> Null.Log.Likelihood.    -3343.7419990
#> AIC.                     5338.2204000
#> BIC.                     5361.3732000
#> McFadden.R2.                0.2029558
#> Adj..McFadden.R2            0.2017595
#> Number.of.Observations.  2412.0000000
```
]]

---

# .center[Multinomial logit in **WTP Space**]

```r
library(logitr)

mnl_wtp <- logitr(
  data       = yogurt,
  choiceName = "choice",
  obsIDName  = "obsID",
* parNames   = "brand",
* priceName  = "price",
* modelSpace = "wtp")
)

summary(mnl_wtp)
```

<center>
<img src="images/utilityWtp2.png" width=300>
</center>
]

```
#> =================================================
#> MODEL SUMMARY: 
#>                                  
#> Model Space:   Willingness-to-Pay
#> Model Run:                6 of 10
#> Iterations:                    36
#> Elapsed Time:         0h:0m:0.23s
#> Exit Status:                    3
#> Weights Used?:              FALSE
#> 
#> Model Coefficients: 
#>               Estimate StdError    tStat pVal signif
#> lambda        0.388631 0.024269  16.0132    0    ***
#> brandhiland  -8.019752 0.455555 -17.6044    0    ***
#> brandyoplait  3.721705 0.157656  23.6065    0    ***
#> branddannon   1.657322 0.165715  10.0010    0    ***
#> ---
#> Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
#> 
#> Model Fit Values: 
#>                                      
#> Log.Likelihood.         -2665.1101915
#> Null.Log.Likelihood.    -3343.7419990
#> AIC.                     5338.2204000
#> BIC.                     5361.3732000
#> McFadden.R2.                0.2029558
#> Adj..McFadden.R2            0.2017595
#> Number.of.Observations.  2412.0000000
```
]]

---

## **.red[Caution]**</br>

## Log-likelihood function for WTP space models is</br>**non-convex**  😔

---

# .center[Use a Multistart]

```r
library(logitr)

mnl_wtp <- logitr(
  data       = yogurt,
  choiceName = "choice",
  obsIDName  = "obsID",
  parNames   = "brand", 
  priceName  = "price", 
  modelSpace = "wtp", 
* options = list(numMultiStarts = 10)
)

summary(mnl_wtp)
```

<center>
<img src="images/utilityWtp2.png" width=300>
</center>
]

```
#> =================================================
#> SUMMARY OF ALL MULTISTART RUNS:
#> 
#>    run    logLik iterations status
#> 1    1 -2833.932         80      4
#> 2    2 -2665.110         34      3
#> 3    3 -2665.110         34      3
#> 4    4 -2665.110         32      3
#> 5    5 -2834.285         76      3
#> 6    6 -2665.110         36      3
#> 7    7 -2665.110         30      3
#> 8    8 -2834.989         69      3
#> 9    9 -2834.567         70      3
#> 10  10 -2834.183         75      4
#> ---
#> Use statusCodes() to view the meaning of the status codes
#> 
#> Below is the summary of run 6 of 10 multistart runs
#> (the run with the largest log-likelihood value)
#> =================================================
#> MODEL SUMMARY: 
#>                                  
#> Model Space:   Willingness-to-Pay
#> Model Run:                6 of 10
#> Iterations:                    36
#> Elapsed Time:         0h:0m:0.23s
#> Exit Status:                    3
#> Weights Used?:              FALSE
#> 
#> Model Coefficients: 
#>               Estimate StdError    tStat pVal signif
#> lambda        0.388631 0.024269  16.0132    0    ***
#> brandhiland  -8.019752 0.455555 -17.6044    0    ***
#> brandyoplait  3.721705 0.157656  23.6065    0    ***
#> branddannon   1.657322 0.165715  10.0010    0    ***
#> ---
#> Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
#> 
#> Model Fit Values: 
#>                                      
#> Log.Likelihood.         -2665.1101915
#> Null.Log.Likelihood.    -3343.7419990
#> AIC.                     5338.2204000
#> BIC.                     5361.3732000
#> McFadden.R2.                0.2029558
#> Adj..McFadden.R2            0.2017595
#> Number.of.Observations.  2412.0000000
```
]]

---

# .center[Mixed logit in **Preference Space**]

```r
library(logitr)

mxl_pref <- logitr(
  data       = yogurt,
  choiceName = "choice",
  obsIDName  = "obsID",
  parNames   = c("price", "brand"), 
* randPars   = c(brand = "n"),
  options = list(numMultiStarts = 10)
)

summary(mxl_pref)
```

<center>
<img src="images/utilityPreference2.png" width=300>
<img src="images/betaNormal.png" width=200>
</center>
]

```
#> =================================================
#> MODEL SUMMARY: 
#>                          
#> Model Space:   Preference
#> Model Run:        5 of 10
#> Iterations:            43
#> Elapsed Time:   0h:0m:17s
#> Exit Status:            3
#> Weights Used?:      FALSE
#> 
#> Model Coefficients: 
#>                     Estimate StdError    tStat   pVal signif
#> price              -0.389810 0.044362  -8.7870 0.0000    ***
#> brandhiland_mu     -3.124798 0.243803 -12.8169 0.0000    ***
#> brandyoplait_mu     1.449109 0.140574  10.3085 0.0000    ***
#> branddannon_mu      0.655452 0.138290   4.7397 0.0000    ***
#> brandhiland_sigma   0.056181 1.177854   0.0477 0.9620       
#> brandyoplait_sigma  0.011127 0.388879   0.0286 0.9772       
#> branddannon_sigma   0.172808 2.458086   0.0703 0.9440       
#> ---
#> Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
#> 
#> Model Fit Values: 
#>                                      
#> Log.Likelihood.         -2665.1077011
#> Null.Log.Likelihood.    -3343.7419990
#> AIC.                     5344.2154000
#> BIC.                     5384.7329000
#> McFadden.R2.                0.2029565
#> Adj..McFadden.R2            0.2008631
#> Number.of.Observations.  2412.0000000
#> 
#> Summary of 10k Draws for Random Coefficients: 
#>                               Min.    1st Qu.     Median       Mean    3rd Qu.      Max.
#> brandhiland (normal)  -3.343157864 -3.1627301 -3.1248413 -3.1248770 -3.0869654 -2.921920
#> brandyoplait (normal)  1.406487337  1.4415994  1.4491051  1.4490984  1.4566085  1.488621
#> branddannon (normal)  -0.009543227  0.5388061  0.6553104  0.6551855  0.7718474  1.259228
```
]]

---

# .center[Mixed logit in **WTP Space**]

```r
library(logitr)

mxl_wtp <- logitr(
  data       = yogurt,
  choiceName = "choice",
  obsIDName  = "obsID",
  parNames   = "brand",
  priceName  = "price",
* randPars   = c(brand = "n"),
* modelSpace = "wtp",
  options = list(numMultiStarts = 10)
)

summary(mxl_wtp)
```

<center>
<img src="images/utilityWtp2.png" width=300>
<img src="images/omegaNormal.png" width=200>
</center>
]

```
#> =================================================
#> MODEL SUMMARY: 
#>                                  
#> Model Space:   Willingness-to-Pay
#> Model Run:                6 of 10
#> Iterations:                    51
#> Elapsed Time:           0h:0m:27s
#> Exit Status:                    3
#> Weights Used?:              FALSE
#> 
#> Model Coefficients: 
#>                     Estimate StdError    tStat   pVal signif
#> lambda              0.391114 0.037629  10.3938 0.0000    ***
#> brandhiland_mu     -8.001475 0.579751 -13.8016 0.0000    ***
#> brandyoplait_mu     3.715322 0.181690  20.4487 0.0000    ***
#> branddannon_mu      1.682783 0.169619   9.9210 0.0000    ***
#> brandhiland_sigma   0.182098 2.623297   0.0694 0.9447       
#> brandyoplait_sigma  0.034227 0.975524   0.0351 0.9720       
#> branddannon_sigma   0.598862 3.586766   0.1670 0.8674       
#> ---
#> Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
#> 
#> Model Fit Values: 
#>                                      
#> Log.Likelihood.         -2665.1066726
#> Null.Log.Likelihood.    -3343.7419990
#> AIC.                     5344.2133000
#> BIC.                     5384.7308000
#> McFadden.R2.                0.2029568
#> Adj..McFadden.R2            0.2008634
#> Number.of.Observations.  2412.0000000
#> 
#> Summary of 10k Draws for Random Coefficients: 
#>                             Min.   1st Qu.    Median      Mean   3rd Qu.      Max.
#> brandhiland (normal)  -8.7092358 -8.124422 -8.001614 -8.001730 -7.878848 -7.343892
#> brandyoplait (normal)  3.5842248  3.692225  3.715311  3.715291  3.738391  3.836856
#> branddannon (normal)  -0.6217451  1.278549  1.682292  1.681859  2.086149  3.775156
```
]]

---

# .fancy[Convenient helper functions]

---

## .center[`wtp()`: Compute WTP from Preference Space model]

</br>

```r
wtp(mnl_pref, priceName = "price")
```

```
#>               Estimate StdError    tStat pVal signif
#> lambda        0.388627 0.024204  16.0565    0    ***
#> brandhiland  -8.019803 0.459432 -17.4559    0    ***
#> brandyoplait  3.721723 0.437562   8.5056    0    ***
#> branddannon   1.657337 0.183184   9.0474    0    ***
```
]

---

### .center[`wtpCompare()`: Compare WTP from Preference & WTP space models]

</br>

```r
wtpCompare(mnl_pref, mnl_wtp, priceName = "price")
```

```
#>                      pref           wtp difference
#> lambda           0.388627     0.3886309  3.940e-06
#> brandhiland     -8.019803    -8.0197522  5.083e-05
#> brandyoplait     3.721723     3.7217045 -1.848e-05
#> branddannon      1.657337     1.6573215 -1.547e-05
#> logLik       -2665.110192 -2665.1101915  2.000e-08
```
]

---

### .center[`simulateShares()`: Expected shares for a set of alternatives]

</br>

```r
alts <- subset(
  yogurt, obsID == 42, 
  select = c('price', 'brand'))

alts
```

```
#>   price   brand
#> 1   6.3  dannon
#> 2   6.1  hiland
#> 3   7.9  weight
#> 4  11.5 yoplait
```
]

```r
simulateShares(
  model = mnl_pref, 
  alts = alts, 
  alpha = 0.025
)
```

```
#>        share_mean  share_low share_high
#> Alt: 1 0.62391502 0.56509154 0.67356217
#> Alt: 2 0.01568871 0.01205067 0.02013178
#> Alt: 3 0.17593678 0.16175071 0.19017909
#> Alt: 4 0.18445949 0.13613464 0.24552081
```
]

---

<br>

# .center[.font150[Thanks!]]

### `logitr` documentation: https://jhelvy.github.io/logitr/

### Slides: https://jhelvy.github.io/2021-sawtooth-conf

.footer-large[
.right[
@johnhelveston <svg style="height:0.8em;top:.04em;position:relative;fill:white;" viewBox="0 0 512 512"><path d="M459.37 151.716c.325 4.548.325 9.097.325 13.645 0 138.72-105.583 298.558-298.558 298.558-59.452 0-114.68-17.219-161.137-47.106 8.447.974 16.568 1.299 25.34 1.299 49.055 0 94.213-16.568 130.274-44.832-46.132-.975-84.792-31.188-98.112-72.772 6.498.974 12.995 1.624 19.818 1.624 9.421 0 18.843-1.3 27.614-3.573-48.081-9.747-84.143-51.98-84.143-102.985v-1.299c13.969 7.797 30.214 12.67 47.431 13.319-28.264-18.843-46.781-51.005-46.781-87.391 0-19.492 5.197-37.36 14.294-52.954 51.655 63.675 129.3 105.258 216.365 109.807-1.624-7.797-2.599-15.918-2.599-24.04 0-57.828 46.782-104.934 104.934-104.934 30.213 0 57.502 12.67 76.67 33.137 23.715-4.548 46.456-13.32 66.599-25.34-7.798 24.366-24.366 44.833-46.132 57.827 21.117-2.273 41.584-8.122 60.426-16.243-14.292 20.791-32.161 39.308-52.628 54.253z"/></svg><br>
@jhelvy <svg style="height:0.8em;top:.04em;position:relative;fill:white;" viewBox="0 0 496 512"><path d="M165.9 397.4c0 2-2.3 3.6-5.2 3.6-3.3.3-5.6-1.3-5.6-3.6 0-2 2.3-3.6 5.2-3.6 3-.3 5.6 1.3 5.6 3.6zm-31.1-4.5c-.7 2 1.3 4.3 4.3 4.9 2.6 1 5.6 0 6.2-2s-1.3-4.3-4.3-5.2c-2.6-.7-5.5.3-6.2 2.3zm44.2-1.7c-2.9.7-4.9 2.6-4.6 4.9.3 2 2.9 3.3 5.9 2.6 2.9-.7 4.9-2.6 4.6-4.6-.3-1.9-3-3.2-5.9-2.9zM244.8 8C106.1 8 0 113.3 0 252c0 110.9 69.8 205.8 169.5 239.2 12.8 2.3 17.3-5.6 17.3-12.1 0-6.2-.3-40.4-.3-61.4 0 0-70 15-84.7-29.8 0 0-11.4-29.1-27.8-36.6 0 0-22.9-15.7 1.6-15.4 0 0 24.9 2 38.6 25.8 21.9 38.6 58.6 27.5 72.9 20.9 2.3-16 8.8-27.1 16-33.7-55.9-6.2-112.3-14.3-112.3-110.5 0-27.5 7.6-41.3 23.6-58.9-2.6-6.5-11.1-33.3 2.6-67.9 20.9-6.5 69 27 69 27 20-5.6 41.5-8.5 62.8-8.5s42.8 2.9 62.8 8.5c0 0 48.1-33.6 69-27 13.7 34.7 5.2 61.4 2.6 67.9 16 17.7 25.8 31.5 25.8 58.9 0 96.5-58.9 104.2-114.8 110.5 9.2 7.9 17 22.9 17 46.4 0 33.7-.3 75.4-.3 83.6 0 6.5 4.6 14.4 17.3 12.1C428.2 457.8 496 362.9 496 252 496 113.3 383.5 8 244.8 8zM97.2 352.9c-1.3 1-1 3.3.7 5.2 1.6 1.6 3.9 2.3 5.2 1 1.3-1 1-3.3-.7-5.2-1.6-1.6-3.9-2.3-5.2-1zm-10.8-8.1c-.7 1.3.3 2.9 2.3 3.9 1.6 1 3.6.7 4.3-.7.7-1.3-.3-2.9-2.3-3.9-2-.6-3.6-.3-4.3.7zm32.4 35.6c-1.6 1.3-1 4.3 1.3 6.2 2.3 2.3 5.2 2.6 6.5 1 1.3-1.3.7-4.3-1.3-6.2-2.2-2.3-5.2-2.6-6.5-1zm-11.4-14.7c-1.6 1-1.6 3.6 0 5.9 1.6 2.3 4.3 3.3 5.6 2.3 1.6-1.3 1.6-3.9 0-6.2-1.4-2.3-4-3.3-5.6-2z"/></svg><br>
@jhelvy <svg style="height:0.8em;top:.04em;position:relative;fill:white;" viewBox="0 0 576 512"><path d="M385.2 167.6c6.4 0 12.6.3 18.8 1.1C387.4 90.3 303.3 32 207.7 32 100.5 32 13 104.8 13 197.4c0 53.4 29.3 97.5 77.9 131.6l-19.3 58.6 68-34.1c24.4 4.8 43.8 9.7 68.2 9.7 6.2 0 12.1-.3 18.3-.8-4-12.9-6.2-26.6-6.2-40.8-.1-84.9 72.9-154 165.3-154zm-104.5-52.9c14.5 0 24.2 9.7 24.2 24.4 0 14.5-9.7 24.2-24.2 24.2-14.8 0-29.3-9.7-29.3-24.2.1-14.7 14.6-24.4 29.3-24.4zm-136.4 48.6c-14.5 0-29.3-9.7-29.3-24.2 0-14.8 14.8-24.4 29.3-24.4 14.8 0 24.4 9.7 24.4 24.4 0 14.6-9.6 24.2-24.4 24.2zM563 319.4c0-77.9-77.9-141.3-165.4-141.3-92.7 0-165.4 63.4-165.4 141.3S305 460.7 397.6 460.7c19.3 0 38.9-5.1 58.6-9.9l53.4 29.3-14.8-48.6C534 402.1 563 363.2 563 319.4zm-219.1-24.5c-9.7 0-19.3-9.7-19.3-19.6 0-9.7 9.7-19.3 19.3-19.3 14.8 0 24.4 9.7 24.4 19.3 0 10-9.7 19.6-24.4 19.6zm107.1 0c-9.7 0-19.3-9.7-19.3-19.6 0-9.7 9.7-19.3 19.3-19.3 14.5 0 24.4 9.7 24.4 19.3.1 10-9.9 19.6-24.4 19.6z"/></svg><br>
jhelvy.com <svg style="height:0.8em;top:.04em;position:relative;fill:white;" viewBox="0 0 512 512"><path d="M326.612 185.391c59.747 59.809 58.927 155.698.36 214.59-.11.12-.24.25-.36.37l-67.2 67.2c-59.27 59.27-155.699 59.262-214.96 0-59.27-59.26-59.27-155.7 0-214.96l37.106-37.106c9.84-9.84 26.786-3.3 27.294 10.606.648 17.722 3.826 35.527 9.69 52.721 1.986 5.822.567 12.262-3.783 16.612l-13.087 13.087c-28.026 28.026-28.905 73.66-1.155 101.96 28.024 28.579 74.086 28.749 102.325.51l67.2-67.19c28.191-28.191 28.073-73.757 0-101.83-3.701-3.694-7.429-6.564-10.341-8.569a16.037 16.037 0 0 1-6.947-12.606c-.396-10.567 3.348-21.456 11.698-29.806l21.054-21.055c5.521-5.521 14.182-6.199 20.584-1.731a152.482 152.482 0 0 1 20.522 17.197zM467.547 44.449c-59.261-59.262-155.69-59.27-214.96 0l-67.2 67.2c-.12.12-.25.25-.36.37-58.566 58.892-59.387 154.781.36 214.59a152.454 152.454 0 0 0 20.521 17.196c6.402 4.468 15.064 3.789 20.584-1.731l21.054-21.055c8.35-8.35 12.094-19.239 11.698-29.806a16.037 16.037 0 0 0-6.947-12.606c-2.912-2.005-6.64-4.875-10.341-8.569-28.073-28.073-28.191-73.639 0-101.83l67.2-67.19c28.239-28.239 74.3-28.069 102.325.51 27.75 28.3 26.872 73.934-1.155 101.96l-13.087 13.087c-4.35 4.35-5.769 10.79-3.783 16.612 5.864 17.194 9.042 34.999 9.69 52.721.509 13.906 17.454 20.446 27.294 10.606l37.106-37.106c59.271-59.259 59.271-155.699.001-214.959z"/></svg><br>
jph@gwu.edu <svg style="height:0.8em;top:.04em;position:relative;fill:white;" viewBox="0 0 512 512"><path d="M476 3.2L12.5 270.6c-18.1 10.4-15.8 35.6 2.2 43.2L121 358.4l287.3-253.2c5.5-4.9 13.3 2.6 8.6 8.3L176 407v80.5c0 23.6 28.5 32.9 42.5 15.8L282 426l124.6 52.2c14.2 6 30.4-2.9 33-18.2l72-432C515 7.8 493.3-6.8 476 3.2z"/></svg>
]]

---

# Extra slides

---

**_A note about the error scale_**

**Preference Space**

**WTP Space**