Dynamic noise estimation: A generalized method for modeling noise fluctuations in decision-making

Computational cognitive modeling is an important tool for understanding the processes supporting human and animal decision-making. Choice data in decision-making tasks are inherently noisy, and separating noise from signal can improve the quality of computational modeling. Common approaches to model decision noise often assume constant levels of noise or exploration throughout learning (e.g., the ϵ-softmax policy). However, this assumption is not guaranteed to hold – for example, a subject might disengage and lapse into an inattentive phase for a series of trials in the middle of otherwise low-noise performance. Here, we introduce a new, computationally inexpensive method to dynamically infer the levels of noise in choice behavior, under a model assumption that agents can transition between two discrete latent states (e.g., fully engaged and random). Using simulations, we show that modeling noise levels dynamically instead of statically can substantially improve model fit and parameter estimation, especially in the presence of long periods of noisy behavior, such as prolonged attentional lapses. We further demonstrate the empirical benefits of dynamic noise estimation at the individual and group levels by validating it on four published datasets featuring diverse populations, tasks, and models. Based on the theoretical and empirical evaluation of the method reported in the current work, we expect that dynamic noise estimation will improve modeling in many decision-making paradigms over the static noise estimation method currently used in the modeling literature, while keeping additional model complexity and assumptions minimal.

We note that simulations of the dynamic noise model were often mis-classified as being generated by the static noise model in RLWM and 2-step datasets.This is because most subjects in these datasets did not benefit substantially from dynamic noise estimation, and the parameters inferred made the dynamic noise model very similar to the static noise model.Thus, simulated behavior was in a range where both models were indistinguishable (since the static noise model is nested in the dynamic one).In these cases, the trivial improvements on likelihoods would be insu cient to o↵set the penalty incurred by the extra parameter in the dynamic model.p(Engaged) recovered well across datasets, with most recovered values between 0.9 and 1. T R E recovery was robust overall, while T E R recovered inadequately.This is because the lack of data in the random state led to insu cient potential transitions from the random to engaged state, which under-powered T E R recovery.[4,5] datasets.On Dynamic Foraging, the learning curves around switches appear random-like during putative lapses.On the IGT dataset, choice frequencies of decks A and D regressed to the random level (one-tailed Wilcoxon signed-rank test p = 9.35⇥10 20 for A, p = 0.48 for B, p = 0.11 for C, and p = 2.83 ⇥ 10 5 for D).For 2-step, the accuracy decreased for all trial types (one-tailed Wilcoxon signed-rank test p = 1.73 ⇥ 10 5 for common and rewarded previous trials, p = 0.019 for rare and rewarded previous trials, p = 5.33 ⇥ 10 4 for common and unrewarded previous trials, and p = 0.002 for rare and unrewarded previous trials).On the RLWM dataset, the win-stay probability decreased more than the lose-shift probability overall (set size of 2: p = 0.056 for win-stay and p = 0.38 for lose-shift; set size of 3: p = 0.07 for win-stay and p = 0.092 for lose-shift; set size of 4: p = 2.9 ⇥ 10 4 for win-stay and p = 0.34 for lose-shift; set size of 5: p = 0.006 for win-stay and p = 0.28 for lose-shift).Note that the estimated engaged probability does not always follow the same trend as accuracy: towards the end of the block, while the di↵erence in accuracy between set sizes of 3 and 6 shrinks, the di↵erence in p(Engaged) does not.
Note that the model-based action values do not need to be updated and can be computed directly: where T is the transition probability from the first-stage choice a 1 t to the second-stage state s 2 i , which the agent is assumed to know.40

Figure A. 7 :
Figure A.7: Both models with static and dynamic noise estimation can fully capture behavior and recover generative parameter values when the true model has static noise.A: Evaluation of model fit with AIC on the data of 1,000 participants simulated using the static noise model.Each dot shows the di↵erence in AIC for an individual between the static and dynamic models.A positive value (orange) indicates that the static model is favored and a negative value (green) means that the dynamic model is preferred by the criterion.The inset shows the mean di↵erence in AIC between the models at the group level.B: Learning curves of both models and data.C: Parameter recovery using the static model.D: Parameter recovery using the dynamic model.For the dynamic equivalent of the static model, T R E = ✏ and T E R = 1 ✏.

Figure A. 9 :
Figure A.9: Model validation results on the empirical datasets.Dynamic noise estimation did not alter the qualitative behavioral predictions made by the models.

Figure A. 11 :
Figure A.11: Improved fit by dynamic noise estimation is correlated to decreased estimation of the transition probability from the the random to engaged state.
Figure A.14: Putative lapses identified by dynamic noise estimation on the IGT [30] and 2-step [33] datasets, both with fixed numbers of trials across participants.The lapses were identified as trials with p(Engaged) < 0.5, sorted by the start trial, and shown across participants.
Figure A.16: Di↵erent ways to initialize p(Engaged) lead to di↵erent latent state occupancy estimations in the first few trials, but similar trajectories afterwards.Note that the estimated engaged probability does not always follow the same trend as accuracy: towards the end of the block, while the di↵erence in accuracy between set sizes of 3 and 6 shrinks, the di↵erence in p(Engaged) does not.