IOPS Winter 2021

class: left, bottom, inverted, title-slide

# IOPS Winter 2021
## Different perspectives on the Bayes factor
### Jorge N. Tendeiro
### Hiroshima University
### <div class="line-block">09-Dec-2021 
Email: <a href="mailto:tendeiro@hiroshima-u.ac.jp" class="email">tendeiro@hiroshima-u.ac.jp</a> 
Webpage: <a href="https://www.jorgetendeiro.com/">www.jorgetendeiro.com</a> 
 
 
Photo by <a href="https://www.pexels.com/@pixabay">Pixabay</a> from <a href="https://www.pexels.com/photo/macro-outdoors-perspective-rocky-268953/">Pexels</a>.</div>

---

background-image: url(data:image/png;base64,#Figures/pexels-terje-sollie-313700.png)
background-size:  cover

## Today

- What is the Bayes factor?

- What did I think of it? 
 Tendeiro and Kiers (2019)

- vRW trying to come to the rescue. 
 van
Ravenzwaaij and Wagenmakers (2021)

- Nope, I don't think so. 
 Tendeiro and Kiers (2021)

.footnote[
Photo by [Terje Sollie](https://www.pexels.com/@solliefoto) from [Pexels](https://www.pexels.com/photo/close-up-of-menu-313700/).
]

---
class: inverse, middle
#  What is the Bayes factor?

.confused::before { 
 content: "";
 background-image: url(Figures/confused2.gif);
 background-size: cover;
 position: absolute;
 top: 0px;
 right: 0px;
 bottom: 0px;
 left: 0px;
 opacity: 0.06;
 z-index: -1;
}
</style>

---
background-image: url(data:image/png;base64,#Figures/pexels-olya-kobruseva-5428833.png)
background-size:  cover
##  What is the Bayes factor?

.footnote[
Photo by [Olya Kobruseva](https://www.pexels.com/@olyakobruseva) from [Pexels](https://www.pexels.com/photo/question-marks-on-paper-crafts-5428833/).
]

To put it lightly:
> *The Bayes factor (BF) is the Bayesian way of testing hypotheses.*

To put it accurately:
> *The BF compares the predictive ability of two competing models. *

---
class: confused
##  What is the Bayes factor?

.footnote[
GIF by [Achievement Hunter](https://giphy.com/achievementhunter) from [Giphy](https://giphy.com/gifs/achievementhunter-rooster-teeth-achievement-hunter-off-topic-ah-4JVTF9zR9BicshFAb7).
]

From Bayes rule we can derive that

`$$\boxed{
\underset{\text{prior odds}}{\underbrace{\frac{p(\mathcal{M}_0)}{p(\mathcal{M}_1)}}}\times
\underset{\text{Bayes factor, }BF_{01}}{\underbrace{\frac{p(D|\mathcal{M}_0)}{p(D|\mathcal{M}_1)}}} = 
\underset{\text{posterior odds}}{\underbrace{\frac{p(\mathcal{M}_0|D)}{p(\mathcal{M}_1|D)}}}
}$$`

Two possible interpretations for `$BF_{01}$`:

1. It conveys the relative probability of the observed data under either model. 
In other words, it indicates what the relative predictive ability of the two models is.

2. It indicates how an *a priori* relative model belief should be rationally updated, in light of the observed data.

---
class: inverse, middle
# What did I think of it? Tendeiro and Kiers (2019)

---
background-image: url(data:image/png;base64,#Figures/pexels-photo-1181343.png)
background-size: cover
## What did I think of it? Tendeiro and Kiers (2019)
.footnote[

Photo by [Christina Morillo](https://www.pexels.com/@divinetechygirl) from [Pexels](https://www.pexels.com/photo/man-wearing-blue-dress-shirt-facing-whiteboard-1181343/).
]

### Why not scrutinize the BF?
> *Any methodological approach has its advantages as well as its drawbacks and NHBT is no different.* (TK2019, p. 775)

### Talk about humility in science:
> *One year before this writing, the authors felt they did not know enough about NHBT to properly understand how it works, what its merits are, and what its potential limitations are. Therefore, we carried out an extensive literature study on NHBT and related topics. This article is the result of putting together a range of discussions on NHBT.* (TK2019, p. 775)

.footnote[

Photo by [Christina Morillo](https://www.pexels.com/@divinetechygirl) from [Pexels](https://www.pexels.com/photo/man-wearing-blue-dress-shirt-facing-whiteboard-1181343/).
]

### The semantics of what's an ''issue'':
> *In this article we offer a wide overview of issues about NHBT which is currently missing in the literature.* (TK2019, Abstract)

> *An ''issue'' can either be a limitation (according to us) or a feature that may (according to us) increase the risk of misuse or misinterpretation of a Bayes factor.* (TK2019, p. 775)

1. Bayes factors can be hard to compute.
2. Bayes factors are sensitive to within-model priors.
3. Use of ''default'' Bayes factors.
4. Bayes factors are not posterior model probabilities.
5. Bayes factors do not imply a model is probably correct.
6. Qualitative interpretation of Bayes factors.
7. Bayes factors test model classes.
8. Mismatch between Bayes factors and parameter estimation.
9. Bayes factors favor the point null model.
10. Bayes factors favor the alternative.
11. Bayes factors often agree with `$p$` values.

---
class: inverse, middle
# vRW trying to come to the rescue van Ravenzwaaij and Wagenmakers (2021)

---
background-image: url(data:image/png;base64,#Figures/pexels-klaus-nielsen-6303535.png)
background-size: cover
## vRW trying to come to the rescue van Ravenzwaaij and Wagenmakers (2021)
.footnote[

Photo by [Klaus Nielsen](https://www.pexels.com/@klaus-nielsen) from [Pexels](https://www.pexels.com/photo/expressive-doctor-in-superhero-costume-6303535/).
]

### From the Abstract:
> *But although we agree with many of their thoughtful recommendations, we believe that Tendeiro and Kiers are overly pessimistic, and that several of their 'issues' with NHBT may in fact be conceived as pronounced advantages.*

<div style="margin-bottom:30px;"></div>
> *We (...) end with a critical discussion of one of the recommendations by Tendeiro and Kiers, which is that ''estimation of the full posterior distribution offers a more complete picture'' than a Bayes factor hypothesis test.*

### On the 11 issues:
> *In our opinion, many of the 'issues' listed by TK are a blessing rather than a curse.* (vRW2021, pp. 2-3)

---
background-image: url(data:image/png;base64,#Figures/pexels-cottonbro-4631066.png)
background-size: cover
## vRW trying to come to the rescue van Ravenzwaaij and Wagenmakers (2021)
### Their take on our 11 issues:
<center>
<img src="data:image/png;base64,#Figures/vRandW2021.png" title="van Ravenzwaaij and Wagenmakers (2021)." alt="van Ravenzwaaij and Wagenmakers (2021)." width="70%" />
</center>

.footnote[

Photo by [cottonbro](https://www.pexels.com/@cottonbro) from [Pexels](https://www.pexels.com/photo/photo-of-person-s-hand-with-words-4631066/).
]

.stressed::before { 
 content: "";
 background-image: url(Figures/stressed.gif);
 background-size: cover;
 position: absolute;
 top: 0px;
 right: 0px;
 bottom: 0px;
 left: 0px;
 opacity: 0.1;
 z-index: -1;
}
</style>

---
class: inverse, middle
# Nope, I don't think so Tendeiro and Kiers (2021)

---
class: stressed
## So what did we make of it?
.footnote[
GIF by [1091](https://giphy.com/1091/) from [Giphy](https://giphy.com/gifs/orchfilms-f-the-prom-3ohhwphLpCpe7hF2Bq).
]

Me and Henk thought long and hard about the reply by vRW to our paper.

> *Could it be that they are right?*

But, really, in our very *biased* opinion:
- We offered completely unassailable arguments that were not refuted by vRW.

- vRW have points of view with which we cannot concur *in the least*.

---
background-image: url(data:image/png;base64,#Figures/kostiantyn-li-Fi_nhg5itCw-unsplash.png)
background-size:  cover
## Bits of what vRW had to say

E.g., concerning the sensitivity of the BF to the priors, this is both ideal:
> *The reason that Bayes factors are sensitive to within-model priors is that Bayes factors evaluate models by the predictions they make, and predictions are determined partly by the prior.* (vRW2021, p. 6),

something to be careful about:
> *(...) sensitivity analysis, that is, a comprehensive investigation of the extent to which the Bayes factor differs across alternative specification of the prior distribution.* (vRW2021, p. 13),

something that may be avoided by means of relying on defaults:
> *(...) default Bayes factors allow efficient communication of evidence that many researchers may consider less sensitive to human bias than more subjective or 'informed' alternatives.* (vRW2021, p. 13),

and it's even not that bad to close your eyes to it:
> *In our opinion, a default Bayes factor, however blindly applied, is usually preferable over no Bayes factor at all.* (vRW2021, p. 13)

---
background-image: url(data:image/png;base64,#Figures/kostiantyn-li-Fi_nhg5itCw-unsplash.png)
background-size:  cover
## Bits of what vRW had to say
### Advising researchers to use BFs, even if only based on a superficial working knowledge:

> *We believe that NHBT, even if executed as a thoughtless ritual, still markedly improves on the status quo.* (vRW2021, p. 33)

### About the truth of a point null hypothesis:
It most likely should be false:
> *(...) it should be stressed that the Bayes factor is based on a comparison of predictive performance that is independent of the notion of absolute or relative model truth (Wagenmakers, Grünwald, & Steyvers, 2006). The idea that we should not use models that we know to be false can easily result in an inferential impasse, because all statistical models are ultimately false.* (vRW2021, p. 26)

but at the same time it is, purportedly, a common research goal:
> *[Estimation] assumes the falsity of the null hypothesis, which is often the very target of inference.* (vRW2021, p. 34)

.yeah::before { 
 content: "";
 background-image: url(Figures/yeah.gif);
 background-size: cover;
 position: absolute;
 top: 0px;
 right: 0px;
 bottom: 0px;
 left: 0px;
 opacity: 0.12;
 z-index: -1;
}
</style>

---
class: yeah
## This is us (Tendeiro and Kiers, 2021)

So we still have something to add to the discussion:

> *Tendeiro, J. N. and Kiers, H. A. L. (2021). On the white, the black, and the many shades of gray in between: Our Reply to van Ravenzwaaij and Wagenmakers (2021). Preprint: 10.31234/osf.io/tjxvz.*

.footnote[
GIF from [Giphy](https://giphy.com/gifs/excited-yes-nicolas-cage-RrVzUOXldFe8M).
]

---
background-image: url(data:image/png;base64,#Figures/pexels-caio-65547.png)
background-size: cover
## Some give-aways (I) (Tendeiro and Kiers, 2021)

.footnote[
Photo by [Caio](https://www.pexels.com/@caio) from [Pexels](https://www.pexels.com/photo/m-m-s-chocolates-in-bowl-65547/).
]

- Bayes factors aren't easy to understand.

- Priors, as part of the model being compared, warrant careful attention.

- Disentangle BFs from posterior model odds.

- Relative model assessment `$\not=$` truthfulness of either model.

- Conclusions derived from a Bayes factor are *always* relative, thus avoid absolute conclusions. So, no such thing as absolute evidence for the null!

---
background-image: url(data:image/png;base64,#Figures/pexels-caio-65547.png)
background-size:  cover

.footnote[
Photo by [Caio](https://www.pexels.com/@caio) from [Pexels](https://www.pexels.com/photo/m-m-s-chocolates-in-bowl-65547/).
]

## Some give-aways (II) (Tendeiro and Kiers, 2021)
- There's no such thing as the `$\not=0$` alternative hypothesis. 
There's a prior that goes with it!

- Be aware of relying on qualitative labels for BFs.

- Bayes factors can easily point at the wrong model (if you're into that kind of thing), especially for small `$n$` and ES.

- Bayes factors, in its default shape, favor the point null model. 
See our worked out example.

---
background-image: url(data:image/png;base64,#Figures/pexels-caio-65547.png)
background-size:  cover

.footnote[
Photo by [Caio](https://www.pexels.com/@caio) from [Pexels](https://www.pexels.com/photo/m-m-s-chocolates-in-bowl-65547/).
]

## Some give-aways (III) (Tendeiro and Kiers, 2021)
- No blind endorsing of Bayes factors. Please.

- Point null hypothesis, really? At least think about it for a bit.

- Bayes factors *cannot* be used to **establish** the existence of an effect!

- We clearly reject the "first test, then who knows estimate" pseudo-inferential algorithm.

- Only reporting Bayes factor is *really* poor practice. Quantify effects sizes as much as you can! Estimate!

--
And remember...

--
<center>

The Bayes factor is nothing but a number, guys!

</center>

---
background-image: url(data:image/png;base64,#Figures/pexels-pixabay-268953.png)
background-size:  cover

## References

.footnote[
Photo by [Pixabay](https://www.pexels.com/@pixabay) from [Pexels](https://www.pexels.com/photo/macro-outdoors-perspective-rocky-268953/).
]

Ravenzwaaij, D. van and E. Wagenmakers (2021). _Advantages Masquerading
as `Issues' in Bayesian Hypothesis Testing: A Commentary on Tendeiro
and Kiers (2019)_. Preprint. PsyArXiv. DOI:
[10.31234/osf.io/nf7rp](https://doi.org/10.31234%2Fosf.io%2Fnf7rp).

Tendeiro, J. N. and H. A. L. Kiers (2019). "A Review of Issues about
Null Hypothesis Bayesian Testing." In: _Psychological Methods_ 24.6,
pp. 774-795. DOI:
[10.1037/met0000221](https://doi.org/10.1037%2Fmet0000221).

Tendeiro, J. N. and H. A. L. Kiers (2021). _On the White, the Black,
and the Many Shades of Gray in between: Our Reply to van Ravenzwaaij
and Wagenmakers (2021)_. DOI:
[10.31234/osf.io/tjxvz](https://doi.org/10.31234%2Fosf.io%2Ftjxvz).