Hippocampus, delay discounting, and vicarious trial‐and‐error

In decision‐making, an immediate reward is usually preferred to a delayed reward, even if the latter is larger. We tested whether the hippocampus is necessary for this form of temporal discounting, and for vicarious trial‐and‐error at the decision point. Rats were trained on a recently developed, adjustable delay‐discounting task (Papale et al. (2012) Cogn Affect Behav Neurosci 12:513–526), which featured a choice between a small, nearly immediate reward, and a larger, delayed reward. Rats then received either hippocampus or sham lesions. Animals with hippocampus lesions adjusted the delay for the larger reward to a level similar to that of sham‐lesioned animals, suggesting a similar valuation capacity. However, the hippocampus lesion group spent significantly longer investigating the small and large rewards in the first part of the sessions, and were less sensitive to changes in the amount of reward in the large reward maze arm. Both sham‐ and hippocampus‐lesioned rats showed a greater amount of vicarious trial‐and‐error on trials in which the delay was adjusted. In a nonadjusting version of the delay discounting task, animals with hippocampus lesions showed more variability in their preference for a larger reward that was delayed by 10 s compared with sham‐lesioned animals. To verify the lesion behaviorally, rat were subsequently trained on a water maze task, and rats with hippocampus lesions were significantly impaired compared with sham‐lesioned animals. The findings on the delay discounting tasks suggest that damage to the hippocampus may impair the detection of reward magnitude. © 2014 Wiley Periodicals, Inc.

Delay discounting refers to the tendency for an individual to prefer an immediate reward over one that is available only after a delay. Thus, humans and other animals tend to discount -place a smaller value upon -a delayed reward relative to an immediate reward.
Researchers have begun to identify the neural correlates of this temporal discounting in humans (Ballard & Knutson, 2009), and our interest was in whether the hippocampus is necessary for this type of decision making.
To explore this question experimentally we employed a delay discounting task recently developed by Papale et al. (2012). In their task, a T maze with return arms was used, and the rat received a larger reward (e.g., 3 food pellets) for choosing one arm of the T, and a smaller reward (1 pellet) for choosing the opposite arm of the T. After consuming the reward, the rat would continue down a return arm to the base of the T, and then enter the central stem to begin another "lap" in the task. A key feature of this task was that the smaller reward was always available after a 1 s delay, whereas the larger reward was available after a > 1 s delay. To determine the delay at which the larger reward had equal value to the immediate, small reward (the indifference point), an adjusting delay procedure was used (Mazur, 1997;2001). Selection of the large reward maze arm increased the delay for that reward by 1 s, whereas selection of the smaller reward arm decreased the delay on the large reward arm by 1 s. In this way, the rat could titrate the delay on the large reward side until the point where the value of the large delayed reward matched that of the small immediate reward.
Papale and colleagues found that the rats' behavior on this task was as one would predict from a hyperbolic delay discounting function. That is, rats tolerated longer delays for larger rewards. In a typical session, rats first alternated between the two maze arms for several trials, and this presumably allowed them to establish which arm contained the large reward, and how delayed the reward was. This was termed the investigation phase of a session. Next, in the titration phase of the session, rats typically chose the immediate, small reward arms for several trials, with the result that the delay on the large reward arm was reduced. Finally, rats began to alternate between the small-and large reward arms, which stabilized the large reward delay. Papale et al. termed this the exploitation phase of the session. The authors also found that rats made more vicarious trial-and-errors (VTEs) -back-and-forth head movements at a choice point -during laps where the delay was changed as opposed to those where the delay was stable. These VTEs have been hypothesised to be a behavioral manifestation of deliberative decision making (Johnson and Redish, 2007;Dudchenko et al., 2013).
Evidence suggests that the hippocampus may play a role in deciding between immediate and delayed rewards. Rawlins et al. (1985) tested rats with aspiration lesions of the hippocampus on a task where they had a choice between a maze arm that was reinforced on a continuous reinforcement schedule and one that was reinforced on a partial reinforcement schedule. Both the lesion and sham-lesioned animals preferred the arm that was continuously reinforced. However, when a 10 s delay of reward was introduced in this arm, the hippocampus lesioned animals changed their preference to the arm with partial, but still immediate, reward. A comparable switch in preference was also seen in animals with medial septal lesions. These findings, and comparable results by Cheung and Cardinal (2005), suggest that septo-hippocampal system is necessary for tolerating delayed reward.
A subsequent study by McHugh et al. (2008) demonstrated that the preference for immediate reward following hippocampus lesions was not simply due to impaired memory for the location of the high reward after a delay. McHugh et al. tested animals with ibotenic acid lesions of the hippocampus and sham-lesioned animals on a T maze task where one maze arm contained 10 food pellets (high reward), while the other maze arm contained 2 food pellets (low reward). In the high reward arm, the pellets were provided after a delay of 10 s; in the low reward arm, the pellets were presented immediately. Compared to control animals, animals with partial or complete hippocampus lesions showed less preference for the delayed, high reward arm relative to the immediate, low reward arm. When a 10s delay was introduced to both arms, both control and lesioned animals preferred the high reward arm. Thus, animals with hippocampus damage could remember which arm held the larger reward over the 10 s delay period, but they discounted the value of this delayed reward relative to an immediate reward more than control animals. Together, these data are consistent with a role of the hippocampus in deliberative decision making, as animals with hippocampal lesions appear to be more impulsive (and less deliberative) than control animals.
Other work has suggested a dissociation between the hippocampus and the orbitofrontal cortex in delay discounting task. In a non-spatial maze task, Mariano et al. (2009) found that hippocampus lesioned animals had less preference than control animals for a larger reward that was delayed by 10 s compared to a smaller, immediate reward. Rats with orbitofrontal lesions, like controls, preferred the larger, delayed reward. More recently, Abela and Chudasama (2013) have observed a double-dissociation between the ventral hippocampus and orbitofrontal cortex in delay-and probability-discounting tasks. As in previous studies, they found that animals with ventral hippocampus lesions had more preference than sham lesioned animals for a smaller, immediate reward relative to a larger, delayed reward. In a parallel task, animals with orbitofrontal lesions preferred a small, high probability reward to a larger, less probable reward, when compared to sham-lesioned animals. No effect was seen in the delayed reward task for the orbitofrontal-lesioned animals, or in the variable probability task for the hippocampus-lesioned animals. Thus, orbitofrontal lesions appear to have a smaller effect on delay discounting than hippocampus lesions.
The current study tested whether the hippocampus is necessary for the temporal discounting of reward. Previous studies (described above) have suggested that rats with hippocampus damage tend to prefer an immediate, small reward over a delayed, larger reward.
However, these studies used fixed delay and rewards. The ability to adjust (titrate) the delay preceding a large reward allows a continuous assessment of the animal's valuation of a reward.
Variable delays and rewards have been used in a study of a patient with hippocampus damage, and in this study the patient did not differ from control subjects (Kwan et al., 2012). In the current study, we were able to assess the point at which a delayed reward was valued as being equivalent to an immediate reward in a group of animals with hippocampus damage using the variable reward method of Papale et al. (2012). Our results show that rats hippocampus damage adjust delays to a point that is similar to sham-lesioned animals, but they spend significantly more trials in the investigatory phase of each session compared to controls, and are often less sensitive to changes in reward magnitude. This suggests that hippocampus may be essential for valuation of a reward.

Materials and Methods
Subjects 14 male Lister Hooded rats (Charles Rivers, U.K.), weighing 200-300 g at the start of the experiment and housed in cages of four in a 12 hr light/dark cycle, were used in this project.
Experiments were carried out during the light phase of the cycle, and food intake was controlled such that individual weights were no less than 85% of normal free feeding body weight, whilst water intake was not restricted. All procedures complied with the UK (Animals

Experiment 1 Apparatus
Rats were trained on a delay discounting task on a modified version of the T-maze (Figure 1).
The maze had return arms stemming from the end of each T arm to the base of the T, and was painted white. The T was 67 cm long and 90 cm wide, with 10 cm wide alleyways and 8 cm walls. At the end of each T alleyway, an automated food pellet dispenser was available to provide food pellets. The maze was placed on top of a metal frame raising it 120 cm from the floor. Two small speakers, one near each food dispenser, were situated on a nearby platform at a similar height to the maze itself. Rats were tracked using a ceiling mounted camera and video tracking system from Axona, Ltd. (St. Albans, U.K.). A custom-written program read the video coordinates of the rat, and controlled the pellet dispensers.

Task
Rats ran continuous laps on the maze, from the central stem, turning left or right, and then along the return arm to the base of the central stem. Rats were rewarded with 45 mg food pellets (Bioserve, NJ, USA) on each lap at the feeder site at the end of the T arm. When a rat entered one of the feeding sites, a sequence of descending frequency 100 ms pure tones was initiated through both speakers. Each sequence consisted of tones separated by 1 s and each tone was 175 Hz lower than the last until the final tone, which was always 1 kHz. Typically, one feeder delivered a high reward (3 pellets) and the other a low reward (1 pellet). Pellet delivery at the high reward feeder was delayed from 1-30 s, whereas pellet delivery at the low reward feeder always occurred 1 s after entry to the feeding site. The length of delay that a rat was required to wait at the high reward feeder could change depending on the behavioural choice made on the previous trial. If the rat visited the high reward feeder on two consecutive laps, the length of the delay increased by 1 s, and by another second for each consecutive visit.
However, if the rat visited the low reward side, the delay at the high reward side was decreased by 1 s. If the rat alternated its choices, the delay at the high reward stayed constant over laps.
Data obtained For each session, the indifference point was defined as the average delay for the high reward arm over the last 10 trials in which the high reward arm was entered. Vicarious trial-and-errors were back-and-forth head movements at the choice point of the T, and these were scored manually by the experimenter during each session. When a rat at the choice point turned its head into one alley and then the other, this was scored as one VTE. If the rat then turned his head to face the first alley, two VTEs would be scored on this lap, three for the next head turn, and so forth, for subsequent head turns prior to the rat entering one alley.

Procedure
Shaping Prior to surgery, rats were trained with one side of the maze blocked such they were forced to travel in one direction to complete a circuit from the start area, up to the choice point and then to one of the arms and back via a return alley. One food pellet was delivered on each lap after a delay of 1 s from the time the rat entered the feeder site, accompanied by a twotone countdown. The blocked side of the maze alternated daily until each rat looped reliably around the maze to collect rewards.
Delay discounting task Following shaping, rats were trained on the main task (as described above) with both arms of the maze open. On different days, the initial delays on the high reward arm were 1, 2, 5, 10, 20 or 30 s. This was pseudo randomised between left and right feeders over days. Each daily session continued until the animals completed approximately 80 laps or failed to continue running on the maze. Across days, the training sessions continued until each rat was able to adjust the initial delay up or down. At this point, rats were then performance matched by indifference points and VTE levels, and assigned to either the lesion group or the sham surgery group.
Post-surgery testing After recovery from surgery, rats were tested on the delay discounting task for 24 sessions, four at each of the six different initial delays, according to a counterbalanced schedule. Following this, rats were trained on a series of sessions where the ratio of reward pellets between each side of the maze was changed between sessions. Fifteen sessions were conducted, with ratios of 1:1 (HR:LR), 2:1, 3:1, 4:1 and 5:1 and delays of 1, 10 and 30 s.
Surgery Rats were anaesthetized with isoflurane (Abbott, UK) and placed in a stereotaxic frame (Kopf Instruments, USA). Anaesthesia was maintained via an inhalation nose cone affixed to the mouth bar on the frame. Under sterile conditions, a midline incision was made and the skull exposed. For the sham animals (n = 6), holes were drilled into the skull bilaterally over the hippocampus. The dura was then pierced several times on each hemisphere and a small piece of absorbable gelatin sponge (Johnson & Johnson, UK) was added to the hole before the animal's skin was sutured. For the lesion rats (n = 8), holes were drilled over the hippocampus and ibotenic acid (Tocris, UK; 10 mg/ml, pH 7.4) was injected at 13 sites in each hemisphere (see Table 1 for coordinates and volumes used). During surgery, rats received a subcutaneous injection of small animal Rimadyl (Pfizer Ltd, UK; 0.08 ml/kg body weight). All rats were given at least 14 days to recover, with free access to food and water, followed by 3 days of food restriction before returning to the experiment. Papale et al. (2012) identified different phases of each discounting session:

Task phase analysis
investigation of the delay for the high reward, titration of this delay up or down, and exploitation of the delay at a stable level. We applied the same analysis to the current data by dividing each session into 5 trial segments in a sliding window. Each segment was classified as an investigation segment, a titration segment or an exploitation segment depending on the number of adjustment (non-alternation) laps. If a segment had 2 or more adjustment laps it was classed as a titration segment, as these non-alternation laps altered the delay at the high reward arm. If a segment had less than 2 adjustment laps, it was classified as either investigation or exploitation. Alternation laps were classed as investigatory if they occurred within the first 30 laps of a session or preceding a titration segment (whichever came first).
After this, alternation laps were classed as exploitation.

Experiment 2 Task
Following completion of Experiment 1, the same rats were tested on a non-titrating version of the delay discounting task. The motivation here was to see whether animals with hippocampus lesions would be less likely to select larger, but delayed reward, compared to control animals.
Testing was done on a traditional T-maze with a painted wood base and transparent Plexiglass walls (16 cm height; base length: 81 cm; arm length: 69 cm; alleyway width: 12 cm). Testing occurred in the same room as Experiment 1, with the maze in approximately the same position (centred in the room).
Entry to one arm of the T-maze resulted in the delivery of 1 food pellet from an automated food pellet dispenser at the arm end. Entry into the other arm yielded 3 food pellets. As in Experiment 1, a tone countdown preceded pellet delivery. To ensure that the rats experienced each of these arms, each daily session began with two trials, one with only the 1-pellet arm open, and the other with only the 3-pellet arm open. The rats then had 10 additional trials where both arms were open, and the same contingencies were in effect.

Procedure
In the initial phase of training, reward was given in either arm after a 1 s delay. Rats were given daily sessions until they chose the 3-pellet arm on 9/10 trials. Following this, rats were tested on sessions in which the delay on the 3-pellet arm was extended. First, a 5 s delay was tested, and then rats were tested on six trials with a 10 s delay on the high reward arm. They were then retested on the 1 s delay for both arms for three session, and then given 1 session with a 15 s delay for the 3-pellet reward, and then three sessions with a 20 s delay for the same reward.

Experiment 3 Task
Following completion of Experiment 2, the same rats were tested on a Morris Water maze task.
The motivation for this experiment was to provide a behavioural validation of the hippocampus lesions in a task that requires the hippocampus. The maze was 2 m in diameter, and filled with water (25 + 1 degrees C) made opaque with 200 ml of latex solution. A circular platform 11.8 cm in diameter was placed in a fixed location within the pool. The maze was centred in a room with salient cues affixed to the wall, though these were obscured when the maze was curtained off from the remainder of the room in the initial visible platform version of the task. Rat positions within the maze were tracked with an overhead video camera and WaterMaze software (Actimetrix).

Procedure
Rats were first tested for three sessions in a visible platform version of the task. Here the maze was curtained off from the remainder of the room, and the platform location was cued by a tennis ball placed on the platform. The rats' task was to swim to the platform to escape from the water. On each trial, the platform was moved to a new, pseudorandom location. Following training on the visible platform task, the curtains surrounding the maze were opened, and the rats were tested for seven days on a hidden platform version of the water maze. After these sessions, a probe session without the platform was run to assess the strength of the rats' preference for the quadrant of the maze in which the platform was previously located.

Experiment 1
Pre-surgery performance Rats were trained on the spatial delay discounting task with a different initial delay used each day until they reliably adjusted both a high or low initial delay for the high reward arm to a stable indifference point. Examples of this can be seen in Figure 2.
On the right (Figure 2A), the initial delay for the high reward arm is 1 s, and the rat alternates between the high-and low-reward arms for the initial 20 trials of the session, before increasing the delay on the high reward arm to about 12 s. In the left plot ( Figure 2B), the initial delay for the high-reward arm is 30 s, and the rat decreases this to about 10 s. Rats were given, on average, 38.9 training sessions prior to surgery (range 31 -59). A one-way ANOVA performed on the indifference points from the last two sessions of each of the six different delays showed that animals titrated to a similar indifference point regardless of the initial delay (F(5,70) = 1.29, p = 0.28; Figure 3).

Vicarious trial-and-error (VTE)
VTEs at the choice point of the T-maze were scored for each trial during each session. Mean VTEs per trial were calculated for each rat. For the last two sessions at each initial delay (12 sessions), an ANOVA showed that the number of VTEs differed depending on the initial delay for the high reward arm (F(5,70) = 5.28, p < 0.001). Specifically, post hoc comparisons (Fisher's LSD test) revealed that initial delays of either 20 or 30 s resulted in higher mean VTE per trial than with initial delays of 1, 2, 5, or 10 s (all p values < 0.05; Figure   4).

Histology
After initial training, rats were given either bilateral hippocampus lesions or a sham surgery.
Lesions were quantified using Image J to count the length of all hippocampal cell layers for both lesion and control groups. For each lesion rat, hippocampus sparing was calculated as a percentage of the mean sham group total. For the 8 lesion rats, the mean hippocampal damage was 75 % with a range of between 61 and 88%. Mean damage to dorsal hippocampus was 89%, whereas damage to ventral hippocampus was 59% ( Figure 5).

Rats with hippocampal lesions had similar indifference points to control rats
Following recovery from surgery, rats were tested for 24 sessions on the delay discounting task, with four sessions at each of the six initial delays (1, 2, 5, 10, 20 and 30 s). The indifference point was calculated for each session as the mean delay on the last 10 high-reward trials. The animals with hippocampus lesions did not differ significantly from the control animals overall (F(1,12) = 2.95, p = 0.11), and there was no significant effect of initial delay on indifference point (F(5,60) = 1.29, p = 0.28). Lesion and control animals appeared to have different indifference points at the 1, 2, and 5 s delays (Figure 6), however the interaction between initial delay and group did not reach significance (F(5,60) =1.35, p = 0.26).

Rats with hippocampal lesions spend more trials in the investigatory phase of sessions
As shown in Figure 8A, rats with hippocampal lesions spent a greater percentage of their session in the investigatory phase of the task compared to controls (F(1,12) = 12.99, p = 0.004).
By trial 30, lesion rats were still investigating around 32% of the time, whereas the sham rats had a mean investigation of around 8%. Overall, there was a decrease in investigatory trials in the first 30 trials of the session (F(1.4, 48) = 56.9, p < .001), and this decrease did not differ between the groups (F(1.4, 48) = 2.37, p = .066).
For the final phase within each session, the exploitation phase, no overall differences were observed between the hippocampus lesion group and the sham animals (F(1,12) = .10, p = .75).

Experiment 2
On the traditional T-maze apparatus with a 1 s delay for both the low-and high-rewards, one animal from the hippocampus lesion group and one from the sham-lesion group failed to show a stable preference for the high reward arm after 18 training sessions, and did not continue in this experiment. For the remaining animals with hippocampus lesions (n = 7), a preference for the high reward (90% high reward choices in one or more sessions) was reached after 5-8 sessions. For the sham-lesioned animals (n = 5), the criterion was likewise reached in 5-8 sessions (see Figure 11) When delivery of reward on the 3-pellet arm was delayed by 5s for a single session, both hippocampus and sham-lesioned animals still preferred this arm and chose it on 9 or more trials out of 10, and the two groups did not differ (F(1, 10) = .18, p = .68). When the reward for the 3pellet arm was delayed by 10s for six sessions, the hippocampus lesioned animals tended to choose the 3-pellet arm somewhat less than the sham-lesioned animals (see Figure 11), but this difference did not reach statistical significance (F(1,10) = 1.23, p = .29).
As is evident in Figure 11, however, the hippocampus-lesioned group showed more variability in their choice of the 3-pellet arm compared to the sham group. This was confirmed statistically. On days 1, 2, 4, and 6, the variability in the hippocampus-lesion group was greater than that of the sham group (Levene's test for equality of variances: all p's < .05). On days 3 and 5, the hippocampus-lesion groups again showed more variability (standard deviations 3.36 and 1.8, respectively) than the sham group (standard deviations 1.52 and 1.52, respectively), but these differences did not reach significance (Levene's test: p's > .05).
Upon return to a 1s delay on both arms, the two groups did not differ (F(1, 10) = .01, p = .93).
When a 15s delay was introduced before the reward on the 3-pellet arms, both groups chose this arm somewhat less than the preceding 1s session, but the groups again did not differ from one another (F(1,10) = 0.0, p = .98). Similarly, no differences between the hippocampus-and sham-lesioned groups was found with a 20s delay for reward on the 3-pellet arm (F(1,10) = .20; p = .66; Figure 11), though both groups decreased their choice of the 3-pellet arms across sessions with this delay (linear trend (sessions): F(1,10) = 6.58, p < .03).

Experiment 3
In the initial training on a visible platform version of the water maze, rats with hippocampus lesions took longer to find the platform than control animals ( Figure 12A; F(1,12) = 18.27, p < .001). The lesioned animals appeared to reach the same level of performance as the sham group by day 3 of testing, though this Group x Day interaction did not reach statistical significance (F(2, 24) = 3.3, p = .054). This pattern of results was also observed in the pathlengh data, and pathlengths and latencies were highly correlated (r = .96, p < .0001). This suggests that the transient difference between groups on the visible platform was not due to differences in swim speeds.
In the hidden platform version of the maze, rats with hippocampus lesions took significantly longer to find the hidden platform compared to control animals ( Figure 12B; F(1,12) = 12.78, p < .005). Overall latency to find the platform decreased across the seven sessions of training (F(6,72) = 2.37, p < .04), and the difference in latencies between groups did not change across sessions (session x group interactions: F(6, 72) = .85; p = .54). In the platform removal probe session following training, the sham animals spend more time in the quadrant of the pool formerly containing the hidden platform than the animals with hippocampus lesions ( Figure   12C), though this difference did not reach statistical significance (t(12) = 1.67, p = .12). Both groups spend more time in the correct quadrant than expected by chance (Sham animals: t(5) = 4.5, p < .006; Hippocampus lesion animals: t(7) = 2.57, p < .037). This suggests that the animals with hippocampus damage, though impaired in learning the platform location compared to control animals, did acquire some association between the platform location and the cues in the environment.

Discussion
In the current study, we used the continuous spatial delay discounting task developed by Papale and colleagues (2012) to test whether the hippocampus is essential for delay discounting. Rats were trained on the task prior to surgery, and we replicated the results of Papale et al. (2012) in terms of the behavioral stages of delay adjustment within sessions (investigation, titration, and exploitation), the linear increase in indifference point with increasing reward magnitude, and the association of VTEs with delay adjustment. We extended these results by testing the effects of hippocampus lesions in this task. Animals with hippocampus damage did not differ from sham lesioned animals in the indifference points with a 3:1 (high reward: low reward) pellet ratio. However, when the pellet ratios were varied between sessions, the lesioned animals showed significantly lower indifference points for the 2:1, 3:1, and 4:1 ratios, but not for the 1:1 and 5:1 ratios. Hippocampus-lesioned rats also spent more time in the investigatory phase of each session. The same animals showed a trend toward a preference for an immediate reward over a larger, delayed reward on a T-maze task, and were impaired in learning a spatial location in a water maze. Below, we consider these findings in turn.

Behavior on the continuous delay discounting task
Prior to surgery, rats were tested with initial delays on the high reward arm of 1-30 s. As observed by Papale et al., within sessions where the initial delay for the high reward was longer (e.g., 30 s), rats tended to "adjust" this delay to be shorter. For sessions with short initial delays for the high reward (e.g., 1 or 2 s), rats increased the delay. Thus, we observed an average delay near the end of each session (the indifference point) between 5-7 s regardless of the session's starting delay. This is comparable to the values found by Papale et al., where the majority of delays at the end of the sessions were between 3 -9 s (from a starting point of 1 -30 s).
Longer initial delays were associated with more vicarious trial-and-error. As longer delays likely require more adjustment trials to reach the same indifference point, these VTEs may reflect a larger proportion of adjustment trials compared to alternation trials (see below) when compared with shorter delays.

Hippocampus lesions alter the behavior phases of the continuous delay discounting task
Following surgery, both hippocampus-and sham-lesioned rats were tested for 24 sessions in the delay discounting task (four sessions at each of six delays for the high reward arm). We divided each session into three phases: investigation, titration, and exploitation. We operationalised the investigation phase as the trials in the beginning of a session where the animal alternated choices of maze arms, and thereby experienced the delay associated with the high reward arm for that session. We took as potential investigatory trials alternation laps within the first 30 laps of each session. One of the main findings of this study is that rats with hippocampus lesions spend more time during these first 30 laps within each session in the investigation phase compared to sham-lesioned animals.
Such a result is unexpected. If, as previous studies suggest, rats with hippocampus lesions prefer immediate, small rewards to delayed, larger rewards, we would have expected that lesioned animals to quickly switch to a choice of the small reward arm on our maze (and thus move to the titration phase). That animals with hippocampus lesions spent a larger portion of the first laps alternating between the small-and high-reward arms suggests that they either had difficulty learning the day's reward contingencies, or that they placed more value on the larger, delayed reward, than did the sham-lesioned animals.
The data from the final indifference point for each initial high reward delay argue against the latter interpretation. If the hippocampus-lesioned animals valued the high reward more than sham-lesioned animals, we would expect them to choose this reward at a higher indifference point. However, this wasn't observed. Rather, although the indifference points of the hippocampus lesioned rats did not differ significantly from those of the sham-lesioned animals, there was a non-significant trend towards lower indifference points in the lesioned animals at 1, 2, and 5s (see Figure 6). This suggests comparable or less valuation of the high reward in the lesioned animals compared to the controls. Thus, animals with hippocampus lesions may spend more time in the investigatory phase of the task because they are impaired at learning the day's reward contingencies.
Such a learning impairment is consistent with previous work. For example, hippocampus lesions impair a version of the Morris water maze in which the animals must learn a new platform location each day (Steele & Morris, 1995). In additional, pharmacological blockage of plasticity or neural activity in the hippocampus blocks one-trial learning of a specific reward location (Day et al., 2003).

Animals with hippocampus lesions are less responsive to increased reward
A feature of delay discounting is its sensitivity to reward magnitude: put simply, the bigger the reward, the longer the animal is willing to wait. This effect was found both in the Papale et al. study, and in the current study with the sham-lesioned animals. The latter showed an increase in the indifference point as the number of pellets in the high reward arm was increased.
A different pattern of results, however, was observed in the animals with hippocampus lesions. They had significantly lower indifference points for the 2:1, 3:1, and 4:1 pellet ratios (high reward: low reward). This is consistent with previous findings that hippocampus lesions decrease the value of the delayed reward (McHugh et al., 2008;Mariano et al., 2009). This effect was not absolute, however. With the highest reward ratio (5 pellets: 1 pellet), the hippocampus-lesioned animals showed a similar indifference point (around 7 s) to the shamlesioned animals. This suggests that hippocampus lesioned animals retain some capacity to tolerate a delay for a larger reward, provided the reward is of sufficient magnitude.
The lower sensitivity to reward seen in the animals with hippocampus lesion may also be interpreted as a learning impairment. The spatial firing of neurons in the hippocampus is sensitive to changes in reward contingencies (Wikenheiser & Redish, 2011), and thus lesions to the hippocampus may diminish the capacity to learn new reward ratios.

Hippocampus lesions did not affect vicarious trial-and-error behavior
One of the motivations for the current experiment was the observation by Johnson and Redish (2007) that place cells in the hippocampus show forward sweeps as animals looks back and forth at a choice point of a maze. If such sweeps during VTEs contribute to decision making, and if VTEs are a behavioural marker of deliberative decision making, then rats with hippocampal lesions might be expected to show fewer VTEs. In the current study, most VTEs were observed during titration (adjustment) laps. There was a trend towards lower incidence of VTEs per lap on adjustment laps for lesioned animals, but this was not significant (p = 0.07).
We also observed that rats with hippocampus lesions showed somewhat fewer VTEs than sham-lesioned animals at 1 s, 2 s, and 10 s initial delays, but did not differ at other delays.
This effect is consistent with the modest effects of hippocampus lesions on VTEs in other tasks (Bett et al., 2012). An earlier study by Hu and Amsel (1995) did show that hippocampus lesions yielded a significant effect on VTEs, but even in that study the effects were transient. Together, these studies suggest that while the hippocampus may be involved in VTEs, this behavior can be maintained by additional structures.
VTEs were more evident in titration laps than in the investigation or exploitation laps Papale et al. (2012) found that VTEs were associated with adjustment laps (laps where the delay for the high reward arm was being increased or decreased) compared to the alternation laps (where the delay was relatively stable). Our results are consistent with this.
More VTE was observed during the titration laps than in either the investigation or the exploitation laps. No differences were observed between hippocampus-and sham-lesioned animals in this regard. Thus, VTEs in this task appear to be associated with the adjustment of the delay to the high reward, and, it may be speculated, more deliberative responding.
However, this does not appear to require the hippocampus.

The effect of hippocampus lesions on a simpler delay discounting maze
After the continuous delay discounting maze, we tested the same animals on a nonadjusting delayed reward task where the rat's choice was between a 1 pellet reward given after a 1 s delay, or a 3-pellet reward, given after various delays. In the initial trials on this task, where the small and large reward were both given after 1 s, both the sham-and hippocampuslesioned animals showed a preference for the large reward arm. When a 10 s delay was added for the high reward arm, animals with hippocampus lesions showed more variability in their preference for this arm compared to sham-lesioned animals. When the delay was increased to 20 s for the high reward arm, both the sham and hippocampus groups shifted their preferences to the 1 s delay -low reward arm.
These results differ from those of McHugh et al. (2008) in a comparable T-maze task.
They found a somewhat stronger preference for the low reward arm in the hippocampuslesioned animals, compared to that observed in the current study. However, there are several procedural differences in the studies that may account for these differences. First, animals in this study had considerable experience (2-3 months) on the continuous discounting maze before experiencing the discrete trial T-maze task. In the McHugh et al. study, the pre-training of the animals prior to surgery appeared to be comprised of about 19 training sessions. It may be that in the current study, the extensive training of rats with hippocampus lesion enabled them to bridge the delay to the larger reward. Second, related to this, the tones presented upon entry to the choice arms may have acted as a secondary reinforcer. A tone-filled delay, as opposed to one without tones, may have allowed animals with hippocampus lesions to bridge the delay in reward delivery. Third, in McHugh et al. study the low reward was larger (2 pellets) as opposed to 1 pellet in the current study. In McHugh et al. the low reward was in delivered immediately, as opposed to after a 1 s delay in the current study. Thus, hippocampus-lesioned animals in this previous study may have shown a stronger preference for the low reward because it was larger and more immediate than the reward in the current study. Finally, in the current study, the animals with hippocampus lesions were significantly more variable in their preference for the high reward, compared to the control animals. This suggests that the even in our highly-trained animals, the preference for the delayed reward was varied when the hippocampus is damaged.

Impairments in the water-maze indicate that the hippocampus lesions were effective
It may be argued that in the current study the hippocampus lesions were large (74.7% of the cell layers), but not sufficient to cause more marked changes in delay discounting or VTE behavior. To assess whether our lesions were sufficient to cause an impairment in a known hippocampus-dependent task, we tested the same rats on a Morris water maze task. Animals platform, though by the third testing session the two groups performed nearly identically.
However, across seven sessions with a hidden platform, the hippocampus-lesioned animals were significantly worse than the sham-lesioned animals in learning the location of the platform. Thus, the hippocampus lesions were sufficient to produce a significant impairment in spatial learning.

Summary
The current study makes three contributions. First, we confirmed the behavioral stages and sensitivity to reward magnitude seen in the novel delay discounting task of Papale et al. (2012). Second, we show that animals with hippocampus damage show prolonged investigatory phases, suggesting that the hippocampus is necessary for assessing initial reward contingencies. This is also supported by the lower sensitivity of hippocampus lesioned animals to changes in reward ratios. Finally, we show that vicarious trial-and-error behavior is observed more frequently when rats adjust the delay that precedes the high reward.  and a turn to the left yielded a low reward (1 food pellet) after a 1 second delay. A turn to the right yielded a high reward (typically 3 pellets), but only after a delay of 1-30 seconds. After the choice of the high or low reward, the rat returned to the central stem via the connecting alleyway for the next lap. The delay for the high reward could be decreased by repeatedly choosing the low reward arm. Conversely, repeated choices of the high reward arm led to an increase in the delay for the high reward. A stable delay at the high reward arm could be achieved by alternating choices between the high and low reward arms.

Figure 2
Examples of changes in the delay for the high reward arm within daily sessions. Light grey triangles are the delays to the low reward arm, which was fixed at 1 s. Black diamonds are the delays for the high reward arm. A) In this example, the initial delay for the high reward arm started at 1 s, and increased to ~ 13 s. The indifference point is the average delay for the last 10 high reward trials in a session. B) In this example, the delay for the high reward arm starts at 30 s, and decreases to ~ 11 s.

Figure 6
Indifference points for sham-and hippocampus-lesioned animals across different initial delays. The indifference points of the two groups did not differ significantly.

Figure 7
Vicarious trial-and-error (VTE) across different initial delays for the sham-and hippocampus-lesioned groups. Overall, VTEs increased with increasing initial delays, and hippocampus-lesioned animals showed significantly fewer VTEs and 2 s and 10 s.

Figure 8
Phase analysis for sham and hippocampus lesion groups. A) Animals with hippocampus lesions had significantly more investigatory (alternating trial) segments than sham-lesioned animals. B) The hippocampus lesion group had significantly fewer titration (repeated trials) than the sham lesion group. C) Hippocampus lesioned animals tended to have fewer exploitation segments (alternation trials after the investigation and titration phases) than sham lesioned animals within early trials in the sessions, but showed a similar percentage of exploitation trials towards the end of the sessions.

Figure 9
Vicarious trial-and-error (VTE) during repetition (investigation or exploitation) and alternation (titration) trials. Both the sham-and hippocampus-lesioned groups showed more VTEs on alternation laps compared to repetition laps.

Figure 10
Indifference points for the sham lesion and hippocampus lesion groups across different reward ratios. Sham-lesioned animals exhibited a higher indifference point (indicating a higher sensitivity to reward magnitude) than the hippocampus-lesioned animals for the 2:1, 3:1, and 4:1 (high reward: low reward) pellet ratios.

Figure 11
Delay discounting on a simple T-maze. Both hippocampus-and sham-lesioned rats acquired a preference for the high reward arm when a delay of 1 s preceded reward on both the high-and low-reward arms, and maintained this when the delay for the high reward arm was increased to 5 s. When the delay for the high-reward arm increased to 10 s, the hippocampus lesioned animals showed more variability than the sham-lesioned animals in their preference for the high-reward arm. Both groups subsequently showed a preference for the high reward arm with a 1 s delay, but this preference decreased across sessions with a 15 s delay of the high reward.

Figure 12
Performance on the Morris water maze. A) Animals with hippocampus lesions took longer to find the visible platform in the initial three days of testing. B) Across seven days of testing with a hidden platform, animals with hippocampus lesions took significantly longer to find the platform than did sham-lesioned animals. C) Both groups showed evidence of a preference for the former platform quadrant in a platform-removal probe session at the end of training.