Correct pred_measure values after post-hoc LOO methods

[florence-bockting]

The problem

This issue refers to loo v3.0.0. It comes from my work on the new pred_measure family.

A post-hoc method corrects PSIS-LOO for the observations with a high Pareto k. Examples are moment matching and reloo. loo_pred_measure() takes ypred, mupred, and ylp from the original draws. It takes the weights from psis_object.

• Moment matching: if the loo object comes from a post-hoc method, ypred still refers to the original draws. It does not refer to draws from the updated parameters.
• reloo: the updated weights are not written into psis_object. Thus psis_object does not match the updated loo object.

Only elpd, mlpd, and ic are correct. They come directly from loo$pointwise. For all other measures, users get mismatched values. They cannot correct them.

What I currently do instead

If moment matching or reloo is used, the current code (branch integrate-loo_compare) warns the user when a measure other than elpd, mlpd, or ic is used. The warning offers no way to get the correct values.

Proposed solution

The design doc in the attached .md file provides a solution for this problem.

It proposes:

• Each post-hoc method stores its weights and the matching ylp, ypred, and mupred columns in the loo object.
• loo_pred_measure() then replaces the matching columns of the user's matrices with the stored columns.

The design suggests changes in loo and brms.

Feedback wanted:

• What do you think about the proposed solution?
• Do you see risks?
• Do you have ideas for an alternative design?

Timing: we could do this after we merge integrate-loo_compare and pred_measure into loo-v3.0.0.

Tagging: @jgabry, @avehtari, @VisruthSK, @paul-buerkner

[codecov-commenter]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 92.50%. Comparing base (7991145) to head (ed2da5b).

Additional details and impacted files

@@           Coverage Diff           @@
##           master     #399   +/-   ##
=======================================
  Coverage   92.50%   92.50%           
=======================================
  Files          31       31           
  Lines        3107     3107           
=======================================
  Hits         2874     2874           
  Misses        233      233           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[github-actions]

This is how benchmark results would change (along with a 95% confidence interval in relative change) if ed2da5b is merged into master:

• ✔️loo_function: 1.89s -> 1.9s [-0.28%, +0.42%]
• ✔️loo_matrix: 1.64s -> 1.65s [-0.19%, +0.79%]
  Further explanation regarding interpretation and methodology can be found in the documentation.