Assess Stickleback predictions

Stickleback supports a temporal tolerance when assessing prediction accuracy, which complicates standard assessments like confusion matrices. Here, a true positive is defined as "a predicted event closest in time to a true event within the temporal tolerance". As an example, consider a Stickleback model with a 5 s tolerance. If there's a true event at 12:00:00 and predicted events at 12:00:01, 12:00:04, and 12:00:10, then 12:00:01 is a true positive (closest and within tolerance), but 12:00:04 (within tolerance, but not closest) and 12:00:10 are false positives (outside tolerance).

sb_assess(sb, predicted, events)

Arguments

sb

[Stickleback] A Stickleback model (see Stickleback)

predicted

[Predictions] Stickleback model predictions (see sb_predict)

events

[Events] Labeled behavioral events (see Events)

Value

[Outcomes] The outcomes (true positive, false positive, or false negative) of the predictions predicted compared to known events eventsaccording to the temporal tolerance specified in the Stickleback model sb. Printing an Outcomes object displays the outcome count by deployment (see Examples).

Examples

# Load sample data and split test/train
c(lunge_sensors, lunge_events) %<-% load_lunges()
test_deployids <- deployments(lunge_sensors)[1:3]
c(sensors_test, sensors_train) %<-% divide(lunge_sensors, test_deployids)
c(events_test, events_train) %<-% divide(lunge_events, test_deployids)

# Define a time series classifier
tsc <- compose_tsc(module = "interval_based",
                   algorithm = "SupervisedTimeSeriesForest",
                   params = list(n_estimators = 2L, random_state = 4321L),
                   columns = columns(lunge_sensors))

# Define a Stickleback model
sb <- Stickleback(tsc,
                  win_size = 50,
                  tol = 5,
                  nth = 10,
                  n_folds = 4,
                  seed = 1234)

# Fit the model to the training data and make predictions on the test data
sb_fit(sb, sensors_train, events_train)
predictions <- sb_predict(sb, sensors_test)

# Assess prediction accuracy
outcomes <- sb_assess(sb, predictions, events_test)
outcomes
#> Outcomes 
#> # A tibble: 3 × 4
#>   deployid       TP    FP    FN
#>   <chr>       <int> <int> <int>
#> 1 bw180905-42    45     3     1
#> 2 bw180905-49    41     0     3
#> 3 bw180905-53    26     1     0