Results from claims assignment

Today’s agenda

Review results of claims assignment
Discuss vignette guidelines
Remaining time to organize in groups

Preface

My goal here is to use the results of your work on the last assignment to learn as a group what worked well and what didn’t.

As a disclaimer, many groups did not successfully generate predictions according to instructions. This is okay.

My goal is not to judge anyone based on results or having completed the assignment.

Accessing claims predictions

Results from the last assignment are now available online.

library(tidyverse)

github_url <- "https://github.com/pstat197/pstat197a/raw/main/materials/slides/data/f22-claims-evals.RData"

load(url(github_url))

evals %>% head()

# A tibble: 6 × 2
  group eval            
  <int> <list>          
1     1 <smplErrr>      
2     2 <tibble [6 × 5]>
3     3 <tibble [6 × 5]>
4     4 <rlng_rrr>      
5     5 <tibble [6 × 5]>
6     6 <rlng_rrr>

Checking errors: example

evals %>%
  filter(group == 4) %>%
  pull(eval)

[[1]]
<error/rlang_error>
Error in `metric_set()`:
! Failed to compute `sensitivity()`.
Caused by error in `dplyr::summarise()`:
! Problem while computing `.estimate = metric_fn(...)`.
---
Backtrace:
  1. tibble(group = 1:15) %>% ...
 15. yardstick (local) panel(., truth = mclass, estimate = mclass.pred, estimator = "macro")
 16. base::mapply(...)
 17. yardstick (local) `<fn>`(dots[[1L]][[1L]], dots[[2L]][[1L]])
Caused by error in `multiclass_checks()`:
! `truth` and `estimate` levels must be equivalent.
`truth`: N/A: No relevant content., Physical Activity, Possible Fatality, Potentially unlawful activity, Other claim content
`estimate`: N/A: No relevant content., Other claim content, Physical Activity, Possible Fatality, Potentially unlawful activity
---
Backtrace:
  1. tibble(group = 1:15) %>% ...
 34. yardstick (local) metric_fn(...)
 35. yardstick::metric_vec_template(...)
 36. yardstick:::validate_truth_estimate_checks(...)
 38. yardstick:::validate_truth_estimate_types.factor(...)
 40. yardstick:::multiclass_checks.factor(truth, estimate)

Accuracies

accuracies <- evals %>%
  rowwise() %>%
  filter(is_tibble(eval)) %>%
  ungroup() %>%
  unnest(eval)

accuracies %>% head(8) %>% knitr::kable()

group	.metric	.estimator	.estimate	class	n
2	sensitivity	macro	0.8081203	multiclass	99
2	specificity	macro	0.9370310	multiclass	99
2	accuracy	multiclass	0.8181818	multiclass	99
2	sensitivity	binary	0.8431373	binary	99
2	specificity	binary	0.8750000	binary	99
2	accuracy	binary	0.8585859	binary	99
3	sensitivity	macro	0.2058975	multiclass	828
3	specificity	macro	0.8035735	multiclass	828

Comparisons

Vignette guidelines

Let’s review as a group.