Noradrenergic but not dopaminergic neurons signal task state changes and predict re-engagement after a failure

The two catecholamines, noradrenaline and dopamine, have been shown to play comparable roles in behaviour. Both noradrenergic and dopaminergic neurons respond to salient cues predicting reward availability and to stimulus novelty, and shape action selection strategies. However, their roles in motivation have seldom been directly compared. We therefore examined the activity of noradrenergic neurons in the locus coeruleus and putative midbrain dopaminergic neurons in monkeys cued to perform effortful actions for rewards. The activity in both regions correlated with the likelihood of engaging with a presented option. By contrast, only noradrenaline neurons were also (i) predictive of engagement in a subsequent trial following a failure to engage and (ii) sensitive to the task state change, the discovery of the new task condition in unrepeated trials. This indicates that while dopamine is primarily important for the promotion of actions directed towards currently available rewards, noradrenergic neurons play a crucial complementary role in mobilizing resources to promote future engagement.


Introduction 31
Catecholaminergic neuromodulation is thought to be critical for numerous aspects of 32 behaviour, including motivation, learning, decision-making and behavioural flexibility 33 Since the tripartite relationship among LC activity, processing of expected vs 77 unexpected stimuli, and motivation remain unexplored, we re-analysed a data set of 78 noradrenergic neurons in the LC recorded in monkeys presented with cues signalling 79 how much effort they would need to expend to gain rewards of various sizes (Varazzani 80 et al. 2015). The task was designed such that rejecting an offer caused it to be re-81 presented on the subsequent trial, and the analyses reported by Varazzani et al. (2015) 82 deliberately excluded such repeated trials. Here, by including those trials, we could 83 investigate separately (i) the sensitivity to task state changes in unrepeated vs. neurons, we could examine separately the relation between dopaminergic neurons 95 and sensitivity to task state changes and willingness to perform the presented option. 96 We found that that although the magnitude of the neuronal response at the cue 97 predicted the engagement in effortful actions similarly in the two catecholaminergic 98 systems, only noradrenaline neurons were sensitive to changes in task state, i.e. to 99 the difference between repeated (and therefore perfectly expected) and unrepeated 100 factor (effort and reward) and recording type (NA or DA) onto -β(effort) and β(reward): 130 main effect of task factor F(1,348)=3.35, p=0.07) but no main effect of recording 131 session type (F(1,348)=2.14, p=0.15) and no interaction (F(2,348)=0.23, p=0.63), 132 meaning that engagement was affected in the same way by the two task factors in both 133 types of recordings.  We have seen previously that the task factors (i.e. effort level, reward level and trial 158 number) influenced the probability of monkeys to engage with the effortful action. 159 Therefore, we first measured the influence of these task factors on neurons' activity at there was a significant difference between the weights of effort and reward in the firing 170 rates of both noradrenergic and dopaminergic neurons (2-way ANOVA measuring the 171 effect of task factor (effort and reward) and recording type (NA or DA) onto -β(effort) 172 and β(reward): main effect of task factor F(1,348)=9.71, p=0.02) but no main effect of 173 recording session type (F(1,348)=0.61, p=0.4) and no interaction (F(2,348)=0.04, 174 p=0.8). This means that the relative sensitivity of noradrenergic and dopaminergic 175 neurons to the task factors was similar, with a greater sensitivity for reward than effort 176 (post-hoc T-test on the distribution of -β(effort) and β(reward): t(350)=-3.13, p=0.002). 177 After having considered the relation between neuronal activity and task factors, we 178 looked at the relationship between neuronal activity and the engagement in the effortful 179 action. First, we did it across the nine task conditions (defined by a combination of 180 effort and reward levels) by using an aggregate measure of the engagement for each 181 condition (the probability to engage given the task condition). This tested whether 182 neuronal activity directly reflected the probability for the monkeys to engage in a 183 particular task condition. For each recording, we regressed this z-scored probability of  In order to understand if catecholaminergic neurons also encode changes in task 223 states (i.e. when their responses to cues differed between repeated and non-repeated 224 trials) and to determine the relationship between this factor and motivation 225 (engagement), we compared the encoding of these two variables at the time of cue. 226 To examine the effect of changes in task states, we compared cue-evoked activity in 227 repeated ('non-informative cue') versus non-repeated ('informative cue') trials. Since 228 erroneous trials were repeated, and monkeys knew the structure of the task, they could 229 predict following an error that the same condition (with the same visual cue) would be 230 presented again, such that the visual cue provided no information about the task state. 231 By contrast, after a correct trial, any of the nine task conditions could be pseudo-232 randomly presented to the monkey, such that visual cues now provided information 233 about the upcoming reward and effort levels (task state). Erroneous trials were mainly 234 of two types: (i) monkeys broke the fixation (no engagement) and (ii) monkeys engaged 235 (tried to squeeze the clamp) but did not execute the action correctly. Therefore, as not 236 all trials in which monkeys engaged were successful, we were able to look conjointly 237 at the effect of engagement and the information being presented on neuronal activity. given condition (β=0.08±0.04, t(83)=2.05, p=0.04), they were not sensitive to the task 260 state change (p=0.56). There was also no significant interaction between the two 261 effects (p=0.36), and the main effects were similar when we removed the interaction. 262 A direct comparison of these effects between noradrenergic and dopaminergic 263 neurons confirmed that, while there was no difference in the strength of their encoding 264 of engagement in the task (p=0.59) noradrenergic neurons encoded significantly more 265 task state change than dopaminergic neurons (p<0.001). 266 Here again, this effect was specific of the onset of the cue as when we examined the 267 500ms pre-cue period, there was neither an effect of engagement (NA: p=0.17, DA: 268 p=0.77) nor an effect of task state change (NA: p=0.96, DA: p=0.07). There was also 269 no effect of engagement in the pre-cue period if we only examined repeated trials 270 where monkeys already knew the task condition (NA: p=0.31, DA: p=0.47). In short, 271 when comparing the encoding of engagement and task state change (unrepeated vs.

301
Only noradrenergic neurons were activated after a failure to engage and are sensitive 302 to the task condition 303 We next examined the activity of dopaminergic and noradrenergic neurons time-locked 304 to fixation break, which resulted in trial abortion. We focused our analysis on three 305 epochs: a baseline epoch from -600 to -300ms prior to fixation; a pre-fixation break 306 epoch corresponding to the 300ms prior to fixation break, and post fixation break epoch 307 corresponding to the 300ms following fixation break. There was neither a significant 308 activation of dopaminergic neurons before fixation break (p=0.62) nor after the fixation 309 break (p=0.49). By contrast, noradrenergic neurons were significantly activated after 310 (mean difference=0.30±0.09 spikes/s, t(83)=3.31, p=0.001), but not before (p=0.81) 311 the fixation break had occurred. This activation corresponds to an average change of 312 16.5%±0.04 of activity between before (average firing rate = 2.83 spikes/s) and after 313 (average firing rate = 3.12 spikes/s) the fixation break ( fig 4A). At the single neuron 314 level, 18.1% noradrenergic neurons were activated at the fixation break (one-tailed T-315 test: firing rate(pre fixation break) < firing rate(post fixation break), p<0.05 were 316 considered as significant). Note that all results hold true if we removed fixation break 317 events that occurred less than 500ms after the cue onset. 318 We then looked at the modulation of fixation-break related activity across task 319 conditions. The firing of dopaminergic neurons did not show any significant modulation 320 across task conditions (probability to engage with the task condition: p=0.97) or 321 behavioural responses (engagement in the next trial: p=0.45) and it will not be 322

350
Noradrenergic neurons activity predicted the engagement on the next trial 351 Finally, we examined the relationship between fixation-break evoked activity and the 352 probability, across sessions, that the monkeys engaged on the next trial. Here again, 353 we only looked at fixation break events that occurred after cue onset, meaning that the 354 monkeys always knew the task condition at the time of the fixation break. Given this strong relation between LC activity and probability of engagement in the 371 current trial when monkeys erroneously break fixation, we were interested to examine 372 whether this activity could also predict monkeys' likelihood of engagement in the 373 following trial. After a fixation break, two things could happen on the next trial (and 374 therefore in the same task condition): monkeys could now choose to engage with the 375 same task condition or could again reject the offer (fig 5A). We therefore examined if 376 LC activity at the time of fixation break could provide information about engagement in 377 the next trial, over and above task condition. Finally, we looked whether the effect of the engagement in the next trial could be found 400 before the cue of the next trials. In other words, we looked if we could predict the 401 engagement before the cue (-500 -0ms) for trials where a fixation break occurred. We 402 found that it was not the case (p=0.25) and could therefore only conclude that 403 noradrenergic neurons predict the engagement on a trial-by-trial basis. 404 In summary, we found that noradrenergic but not dopaminergic neurons' activity at 405 fixation break reflected the probability to engage both in the current and in the 406 subsequent trial, over and above cost-benefit task conditions.

Discussion 427
In this task, monkeys were presented with informative (non-repeated) and non-428 informative (repeated) cues instructing them to produce actions of different intensities 429 to gain rewards of different magnitudes. The probability that monkeys would try to 430 produce the action (engagement) depended on the task condition (effort and reward 431 levels) but failing to engage would only lead to the repetition of the same task condition. 432 Repeated trials constituted series of actions towards the same goal: the reward. This 433 goal directed behaviour ended when the goal was reached. From that perspective, 434 there is a clear transition in behaviour after a correct trial, as animals get started on 435 another trial, another goal directed behaviour (Bouret & Richmond 2009). Hence, given 436 the structure of the task, unrepeated trials are more likely to constitute a task state 437 changes than repeated ones from a goal-directed behaviour perspective. We used this 438 task structure to reveal the precise roles of noradrenergic and dopaminergic neurons 439 in encoding motivation to engage in the task and in signalling task state changes. We 440 used the engagement in a task condition on a specific trial as a measure of motivation 441 and found that both noradrenergic and dopaminergic neurons' activities were 442 predictive of the engagement. Their activities were not only correlated with the session-443 average probability to engage in a particular task condition, but also with the trial-by-444 trial engagement. Furthermore, their activities were correlated with engagement over 445 and above the specific task condition. This strengthens the role of both 446 catecholaminergic systems in motivating effortful, reward directed actions. probabilistic outcome to a deterministic outcome, and that this preference was 514 controlled by the dopaminergic system (Naudé et al. 2016). These two studies show 515 that dopaminergic neurons are sensitive to information as a variable that can influence 516 choices through preferences, since it acted as a reward (Charpentier et al. 2018). In 517 our task, as the cost-benefit cues were all well known, information (as provided by the 518 cues in non-repeated, but not in repeated trials) would neither cause sensory surprise 519 (as cues themselves were not novel) nor be relevant for modulating future choices. 520 Therefore, although we cannot rule out that some individual dopamine neurons do 521 code for this factor, it seems that dopamine neurons as a population do not encode 522 the information about task state changes when this is not relevant to guide the 523 behaviour. 524

Noradrenergic neurons' activity reflects the role of noradrenaline in information 525
processing and engagement after a failure 526 The crucial difference between dopaminergic and noradrenergic neurons was that 527 noradrenergic neurons were sensitive to the repetition of a trial at cue. Because task 528 state changes only occur after a successful trial, lower activation of LC neurons at cue 529 on repeated trials could reflect the fact that an error just occurred. However, we found 530 no significant effect of error on the previous trial in baseline activity before the cue. 531 Therefore, it is unlikely that there is a carry-over effect of error on the next trial. This 532 lower activation in repeated trials could also be simply due to the repetition of a visual 533 cue. However, there was no significant difference in the sensitivity to the task factors 534 (effort and reward levels) in repeated and non-repeated trials. Hence, there is no 535 evidence in our data for a simple stimulus repetition suppression effect. Moreover, from 536 a goal directed behavior perspective, there is much more likely to be a state transition 537 after a sequence ended with a reward, which would argue against a simple cue 538 repetition response. Therefore, we attributed this lower activation to the fact that the 539 monkeys already knew the task condition in repeated trials. Noradrenergic neurons 540 would be sensitive to the information about task state changes, which corresponds to 541 the discovery of a new state of the world either at the time of cue (i.e., which task 542 condition has been selected for the current trial) but also at fixation break (an error 543 means that the trial is terminated and that the same task condition is coming next).This Crucially, only noradrenergic neurons were activated following a break in fixation, 553 which represents a failure to engage in the effortful action. Similar patterns of activity 554 at the break of fixation have also been observed in mid-cingulate cortex (MCC), here 555 modulated by how close to reward delivery the error occurred or how much effort was 556 already invested in the task (Amiez et al. 2005). Given the connections between LC 557 and MCC, this suggests that MCC and LC might well interact when required to signal 558 salient events. A break of fixation was an important event not only as it signalled the 559 end of the trial, but also the re-occurrence of same task condition in the next one. This 560 post-fixation break activity was tightly linked to firing rates at the time of cue, which in 561 turn reflected the probability of engagement in the effortful action. A potential scenario 562 is that if the activity at the cue was too small to enable maintenance of the fixation and 563 the engagement in the trial, then activity at the fixation break reflects a prospective 564 update to enable performance of the action on the subsequent trial. Indeed, we found 565 that when we controlled for task condition, noradrenaline neurons were more active 566 after fixation break when monkeys then engaged in the subsequent trial. Finally, as we 567 were never able to predict the engagement in the trial from the baseline activity at the 568 cue, even for repeated trials and even for trials following a fixation break, we only 569 conclude that noradrenergic neurons predict the engagement on a trial-by-trial basis 570 but have no evidence that they do so through a slow fluctuation of activity that lasts 571 beyond the range of a trial. The behavioural paradigm has previously been described in detail in Varazzani et al. 596 (2015). In brief, each monkey sat in a primate chair positioned in front of a monitor on 597 which visual stimuli were displayed. A pneumatic grip (M2E Unimecanique, Paris, 598 France) was mounted on the chair at the level of the monkey's hands. Water rewards 599 were delivered from a tube positioned between the monkey's lips. Behavioural 600 paradigm was controlled using the REX system (NIH, MD, USA) and Presentation 601 software (Neurobehavioral systems, Inc, CA, USA). 602 The task consisted of squeezing the grip to a minimum imposed force threshold to 603 obtain rewards, delivered at the end of each successful squeeze (fig 1A and B). At the 604 beginning of each trial, subject had to fixate a red dot at the centre of the screen before 605 a cue appeared. The cue indicated the minimum amount of force to produce to obtain 606 the reward (3 force levels) and the amount of reward at stake (3 reward levels: 1, 2 607 and 4 drops of water). After a variable delay (1500±500ms from cue display), the dot 608 at the centre of the cue turned green (Go signal) and subject had 1000ms to initiate 609 the action, meaning squeezing the clamp very little (threshold set to detect any attempt 610 to perform the action). If the monkey reached the minimum force threshold indicated 611 by the cue, the dot tuned blue and remained blue if the effort was sustained for 612 500±100ms. At the end of this period, if at least the minimum required effort had been 613 maintained, the water reward was delivered. 614 Fixation of the central dot had to be maintained through the different phases of the 615 task. A trial was incorrect if: (i) the monkey broke fixation before the reward delivery, 616 (ii) he squeezed the clamp before the go signal, (iii) he failed to squeeze the clamp at 617 all or (iv) at the minimum force threshold or (v) didn't maintain the effort long enough. 618 After an error the same trial was repeated until it was successfully completed. Within 619 a session, the nine combinations of effort and reward conditions were selected with 620 equal probability and presented in a random order. As erroneous trials were repeated, 621 the policy with the highest reward rate was to always engage until satiety. 622 Monkeys were trained for several months on this task. They first learned to distinguish 623 and perform two different force levels and the difficulty of the task was progressively 624 increased until they were could do so with the nine experimental conditions. Finally, 625 they learned that they had to fixate the central dot to go through a trial. 626

Electrophysiological recordings 627
Single unit recording using vertically movable single electrodes was carried out using 628 conventional techniques. The electrophysiological signals were acquired, amplified 629 (x10,000), digitized, and band-pass filtered (100 Hz to 2 kHz) using the OmniPlex 630 system (Plexon). Precise description of the recording procedures can be found in the 631 article where LC and SNc/VTA data used here were originally reported (Varazzani et   In all our analyses we only considered trials (correct and incorrect) in which monkeys 641 did not break the fixation before the onset of the cue (NA: 324 trials on average for 642 monkey A and 281 for monkey D, DA: 314 trials on average for monkey D and 274 for 643 monkey E). We took all those trials and computed the probability that for a given effort 644 and reward level (or a given task condition), subjects would engage with the trial. We 645 considered that monkeys engaged if they maintained fixation throughout the trial and 646 initiated the action even if it occurred before the Go signal, (5% of trials in both 647 noradrenergic (NA) and dopaminergic (DA) neurons recording sessions), not strongly 648 (0% and 0.1% of trials in NA and DA sessions respectively) or long enough (8% and 649 10% of trials in NA and DA sessions respectively). Although it was possible to fail to 650 engage with a trial by maintaining fixation but not squeezing the clamp, this type of 651 mistake was rare (2% and 1% of trials in NA and DA sessions respectively) and 652 monkeys mostly rejected a trial by breaking fixation (20% of all trials in both NA and 653 DA sessions). Erroneous trials were therefore mainly of two types: i) monkeys broke 654 the fixation and failed to engage with the trial (no engagement and no new information 655 as the same trial type is presented again: 20% of all trials in both NA and DA sessions) 656 and ii) monkeys engaged (tried to squeeze the clamp) but did not complete the correct 657 action (engagement but no new information: 17% and 20% of engaged trials, which 658 corresponds to 13% and 15% of all trials in NA and DA sessions respectively). 659 We examined the effects of effort, reward and trial number on the engagement in the 660 action using a multi-level logistic regression for each session. The three variables were 661 z-scored so that we could compare their weights across sessions. We then went on to 662 examine task conditions influenced neuronal activity. To assess the effect of task 663 conditions on neurons' activity at the time of cue onset, we used a window from 0 to 664 500ms from cue onset. When we looked at these effects in the pre-cue period, we 665 used a window from -500 to 0ms from cue onset. Neurons' activity was measured in 666 firing rates (spikes per second) and were z-scored scored for each session to compare 667 the activity across neurons. First, the effects of the task factors: effort, reward and trial 668 number in a session on neurons' activity were estimated using a multi-level linear 669 regression for each neuron. Second, we assessed the relationship between neurons' 670 firing rates and engagement in a given trial by running a logistic regression of neurons' 671 firing rates on engagement. Finally, we looked at the linear encoding of the z-scored 672 probability to engage given the task condition on neurons' firing rates using a linear 673

regression. 674
When we looked at the effect of the novelty of the trial state (here referred to as "task 675 state change") on neuronal activity, we first looked at whether the fact that a cue was 676 informative (I=1) or not (I=0) changed the sensitivity of neurons for the task factor (E, 677 R, N) at the time cue by regressing the task factors and the interaction between the 678 task factors and the informativity (I=0 or 1) onto the trial-by-trial neurons' activity. A 679 significant interaction would mean that an informative cue (signalling the new task 680 state) would increase of decrease the sensitivity for the task factor. We then wanted to 681 assess the conjoint effect of engagement and task state change on neurons' firing 682 rates above and beyond the effect of effort and reward levels. To do so, we ran a multi-683 level linear regression taking into account the task condition variability. In other words, 684 we removed from neurons' firing rates the effect of the task condition using a mixed where β0 a constant, β0(task condition) a constant fitted for each combination of effort 688 and reward level (9 possibilities), xi the experimental factors and βi their weights in the 689 linear regression (e.g. engagement, task state change, interaction). When looking at 690 the effect of engagement and task state change at cue, we tested the following 691 experimental factors: engagement, task state change and interaction between effect. 692 We then added to the regression the following confounds: trial number, interaction 693 between trial number and engagement and interaction between trial number and task 694 state change. All results hold when adding the confounds. 695 We then moved on to assess whether noradrenergic and dopaminergic neurons were 696 activated before the fixation break. We only considered fixation breaks that occurred 697 after the display of the cue. We compared firing rates from 600ms before the fixation 698 break to 300ms after (in 300ms windows). For all analyses at fixation break, we only 699