Chapter 9 Extinction of Conditioned Behavior

partial reinforcement promotes persistence in two different ways:

1. Frustration theory 2. sequential theory

Effects of Extinction Procedures

Two basic behavioral effects of extinction 1. target response decreases with the response no longer results in reinforcement - primary behavioral effect of extinction - investigations of extinction have been concerned with how rapidly responding decreases and how long the response suppression lasts 2. Extinction increases response variability - at first -Study done by Neuringer, Kornell & Olufs: rats were required to perform variable response sequences for reinforcement (variable) or received reinforcement regardless of their response sequence (yoke). Findings: decline in responding occurred in the face of an increase in variability of the response sequences the subjects performed. Extinction produced a decline in the rate of responding in both groups. Extinction decreased the rate of responding and increased response variability, but did not overall alter the basic structure of the instrumental behavior

Extinction procedures also often produce strong __________ effects.

emotional -if an organism has become accustomed to receiving reinforcement for a particular response, it may become upset when reinforcers are no longer delivered.

Retention of Knowledge of the Reinforcer

extinction does not erase what was originally learned because conditioned behavior can be restored through spontaneous recovery, renewal, and reinstatement

The phenomenon of reinstatement suggests that the _____ and ______ may return full blown if the client experiences an abusive encounter later in life. Because of reinstatement, responses that are successfully _______ during the course of therapeutic intervention can _______ if the individual is exposed to the _______ stimulus again

fear; anxiety; extinguished; reoccur; unconditioned

The loss of conditioned behavior that occurs as a result of extinction is not the same as the loss of responding that be occur because of ________

forgetting

The behavior change that occurs in extinction is the _______ of what was observed in acquisition. Because of this, extinction appears to be the ______ of acquisition

reverse; opposite

A powerful technique for determining whether conditioned behavior reflects knowledge about the reinforcer is to....

test the effects of reinforcer devaluation -if conditioned behavior reflects an S-O or R-O association, devaluation of the reinforcer should produce a decrement in responding -we can determine whether extinction eliminates S-O and R-O associations by seeing if reinforcer devaluation also suppresses conditioned responding after extinction

What evidence do we have that indicates that knowledge of the reinforcer is not lost during the course of extinction?

tests of the specificity of reinstatement -ex of recent study: Presentation of a reinforcer after extinction processed a selective recovery of lever pressing. Much more responding occurred on the lever whose associated reinforcer had been used for the reinstatement procedure. The food pellet selectively increased responding on the lever that previously produced food and the sugar water selectively increased responding on the lever that previously produced a dew drops of sugar water. These results indicate that the extinction procedure did not erase knowledge of which reinforcer had been used with which response during original training (reinstatement of lever pressing depending on whether the reinstatement reinforcer was the same or different from the reinforcer originally used to train the response)

Extinction is also one of the hot areas for _________ research that seeks to improve clinical practice based on laboratory findings

translational

Give an example from the text book of frustrative aggression induced by extinction (pg 303 bottom)

two animals placed in the same skinner box -one of them is initially reinforced for pecking a response key, while the other animal is restrained in a corner of the experimental chamber -the key pecking bird ignored the other one as long as pecking in reinforced with food -when extinction is introduced and reinforcement ceases, the previously rewarded animal is likely to attack restrained bird -aggression also occurs with stuffed model

Renewal

refers to a recovery of acquisition performance when the contextual cues that were present during extinction are changed -the change can be a return to the context of original acquisition or a shift to a neutral context -translational research: suggests that clinical improvements that are achieved in the context of a therapist's office may not persist when the client returns home

In classical conditioning, extinction involves...

repeated presentations of the CS by itself. -conditioned responding declines

Rescorla and spontaneous recovery

-characterized it as one of the basic phenomena of pavlovian conditioning -experiment: original acquisition was conducted with two different unconditioned stimuli (sugar and solid food pellet) delivered into cups recessed in one wall of the experimental chamber. Infrared detectors identified each time the rat poked its head into the food cups. Experimental chamber dark. Unconditioned stimuli was signaled by a noise CS and the other was signaled by light CS. As conditioning progressed, each CS quickly came to elicit the goal tracking conditioned response, with the two CS's eliciting similar levels of responding. S1: conditioned stimuli and 8 day resting period, S2: test trials started immediately after extinction training. The recovery of responding observed to S1 represents spontaneous recovery, but the recovery was not complete

Why is it that original acquisition is less disrupted (if at all) by a change in context when extinction performance is highly context specific? (Hint: Bouton)

-contextual cues serve to disambiguate the significance of a conditioned stimulus -this function is similar to the function of semantic context in disambiguating the meaning of a word -e.g.: cut, what it means depends on the context of the sentence -A CS that has undergone excitatory conditioning and then extinction also has an ambiguous meaning in that the CS could signify that shock is about to occur (acquisition) or that shock won't occur (extinction). This ambiguity allows the CS to come under contextual control more easily -after acquisition training, the CS is not ambiguous because it only signifies one thing (shock will occur) -therefore, such a CS is not as susceptible to contextual control

What is learned in extinction?

-extinction does not involve unlearning and leaves response-outcome (R-O) and stimulus outcome (S-O) associations intact -if these associations remain intact, what produces the response decrement? 1. Inhibitory S-R associations 2. Paradoxical Reward Effects **Partial-Reinforcement Extinction effect

Renewal of Original Excitatory Conditioning

-extinction does not result in permanent loss of conditioned behavior -> this is because of renewal

Extinction and Original learning

-extinction produces important behavioral and emotional effects, it DOES NOT reverse the effects of acquisition -evidence: studies of spontaneous recovery, renewal, reinstatement, and reinforcer devaluation

What type of therapy is basically an extinction procedure in which participants are exposed to cues that elicit fear in the absence of the aversive stimuli?

Exposure therapy -exposure to the actual fearful stimuli is the best way to conduct exposure therapy, but that is not often practical -also employed in treating drug addiction, with the aim of extinguishing cues associated with drug taking behavior

Renewal Effect

Phenomenon that occurs if a response is extinguished in a different environment than it was acquired; the extinguished response will reappear if the animal is returned to the original environment where acquisition took place. -evidence indicates that the renewal effect occurs because the memory of extinction is specific to the cues that were present during the extinction phase -a shift away from the context of extinction disrupts retrieval of the memory of extinction, with the result that extinction performance is lost

How is compounding extinction stimuli (CES) to further extinguish a response like the Rescorla Wagner model (RWM)?

RWM: associative values are adjusted if the outcome of a trial is contrary to what is expected. Original acquisition creates an expectation that the US will occur. This expectation is violated when the US is omitted in extinction, and that ERROR is correction by reduced responding on subsequent trials CES: compounding two conditioned stimuli increases the resulting error when the trial ends without a reinforcer. This induces a larger correction and greater reduction of responding

Reinstatement of Conditioned Excitation

Reinstatement: refers to the recovery of conditioned behavior produced by exposures to the unconditioned stimuli -learning an aversion to fish because you got sick after eating fish on a trip -> aversion extinguished by nibbling on fish without getting sick on a number of occasions -> Suggests that if you were to become sick again for some reason, your aversion to fish would return even if your illness had nothing to do with eating fish

extinction is an _____ process produced by the unexpected absence of the ___ or the ______. Forgetting is a decline in responding that may occur simply because of the passage of ____ and does not require ___-_____ encounters with the CS or the instrumental response.

active; US; reinforcer; time; non-reinforced

Learning mechanisms are useful because the new responses that are acquired promote ________ to a changing environment

adjustments

Allyn, & Bacon

demonstrated reinstatement of conditioned suppression -four reinstatement shocks were delivered wither in the training and test context (shock same) or in a different context (shock different) after just excitatory conditioning (conditioned only CS) or after conditioning and extinction (conditioned and extinguished CS) -subjects conditioned by weak shock and did not receive extinction: it did not make any difference whether the reinstatement shocks occurred in the test context (shock same) or elsewhere (shock different) -> shows that contextual conditioning did not summate with the suppression elicited by the target CS -subjects that received extinction reinstatement shocks given in the same context as testing produced significantly more response suppression than shocks given in a different context -> shows that context conditioning facilitates the reinstatement effect -Results encouraged Bouton to think about reinstatement as a form of renewal -> conditioned contextual cues provide some of the contextual cues for excitatory responding under ordinary circumstances -> contextual cues become extinguished when the CS is presented by itself during extinction -> reinstatement US presentations in the test context serve to restore the excitatory properties of the contextual cues and thereby enable those cues to be more effective in reactivating the memory of the original acquisition training.

The emotional reaction induced by withdrawal of an expected reinforcer is called __________.

frustration -frustrative non-reward energizes behavior -frustration may be intense enough to induce aggression

Reducing Spontaneous Recovery: Rescorla and Extinction of appetitive conditioning

increasing the interval between training and extinction reduced the degree of spontaneous recovery that occurred -differences between Rescorla and Myers Experiments that tested the same thing: 1. different motivational systems (fear conditioning vs. appetitive) 2. different experimental designs (Rescorla: within vs. Myers: between)

Extinction

involves omitting the US, or reinforcer. -conditioned responding declines

Spontaneous recovery from extinction is similar in that it...

is also produced by the introduction of a period of rest

overtraining extinction effect

less persistence of instrumental behavior in extinction following extensive training with reinforcement than following only moderate levels of reinforcement training. the effect is most prominent with continuous reinforcement -more rapid extinction following training that est. greater expectations of reward -the more training that is provided with reinforcement, the stronger will be the expectancy of reward, and therefore the stronger will be the frustration that occurs when extinction is introduced. That is turn should produce more rapid extinction -paradoxical because it represents fewer responses in extinction after more extensive reinforcement training (you would think that more extensive training should create a stronger response that would be more resistant to extinction, but you would be WRONG, especially when training is continuous rather than intermittent)

Magnitude reinforcement extinction effect

less persistence of instrumental behavior in extinction following training with a large reinforcer than following training with a small or moderate reinforcer; effect is most prominent with continuous reinforcement -more frustrating if the individual has come to expect a large reward than if the individual expects a small reward

In instrumental conditioning, extinction involves...

no longer presenting the reinforcer as a consequence of the instrumental response

Experiments have shown that S-O associations are ____ _____ during pavlovian extinction. Thus, an extinguished CS continues to ________ a representation of the US. Information about the reinforcer is also not lost during the course of extinction of an instrumental response.

not lost; activate -rescorla: R-O associations, once trained, are relatively impervious to modification

Inhibitory S-R associations

-extinction seems to leave S-O and R-O associations alone, so what else can they look at but S-R associations to explain extinction -conclusion that nonreinforcement produces an inhibitory S-R association -nonreinforcement of a response in the presence of a specific stimulus produces an inhibitory S-R association that serves to suppress that response whenever S is present -> predicts that the effects of extinction will be highly specific to the context in which the response was extinguished -Extinction: involves non reinforcement after a history of conditioning with repeated presentations of the reinforcer -> it would just be habituation without this history -ex: if you never received an allowance, you would not be disappointed when you didn't get one. It is only the omission of an expected reward that creates disappointment/frustration -emotions play huge part in behavioral decline that occurs in extinction -extinction involves both behavioral & emotional effects -emotions: frustration triggered by an expected reward that doesn't happen -> triggers an unconditioned aversive reaction -> discourages responding through the establishment of an inhibitory S-R association -**specific to a particular stimulus and response**

Spontaneous Recovery

-extinction typically produces a decline in conditioned behavior, but this effect dissipates with time -if rest period is introduced after extinction training, responding is observed to recover -because nothing specific is done during the rest period to produce the recovery, the effect is called spontaneous recovery -originally identified by pavlov -prominent phenomenon following extinction of instrumental behavior: critical factor is introducing a period of rest between the end of extinction training and assessments of responding

Why should a change in context restore behavior characteristic of original acquisition?

-have to make assumption that original acquisition performance generalizes from one context to another more easily than extinction performance does

Partial Reinforcement Extinction Effect (PREE)

-most extensively investigated -key factor that determines the magnitude of behavioral and emotional effects of extinction procedure: Schedule of reinforcement that is in effect before extinction procedure -most important variable: whether the instrumental response was reinforced every time it occurred (continuous reinforcement) or only some of the time (intermittent or partial reinforcement) -partial reinforcement: extinction slower and fewer frustrations -> this is PREE -emergence of PREE is related to the rapid maturation of the hippocampus (in rat pups) -habitual gamblers at the mercy of partial reinforcement -> occasionally winning keeps them going -also demonstrated in pavlovian conditioning -evidence of PREE found in studies that compared the effects of continuous and partial reinforcement training in different gourds of subjects -eventually found that can occur in the SAME subjects IF they experience CONTINUOUS reinforcement in the presence of one set of cues and INTERMITTENT reinforcement in the presence of other stimuli

Mechanisms of the partial-reinforcement extinction effect

-most obvious explanation of the PREE: introduction of extinction is easier to detect after continuous reinforcement than after partial reinforcement -discrimination hypothesis -frustration theory -sequential theory

Enhancing Extinction

-mounting evidence that extinction does not erase much of what was originally learned is bad news for exposure therapy. Can impact of extinction be increased so as to make such procedures more effective? 1. Number and Spacing of extinction trials 2. Reducing Spontaneous Recovery 3. Reducing Renewal 4. Compounding Extinction Stimuli

Studies on Human Fear Conditioning and Reinstatement

-skin conductance increased during the course of fear conditioning and decreased during extinction -subsequent US presentations in the same room resulted in recovery of the extinguished skin conductance response -US presentations in a different room did not produce this recovery -Therefore, reinstatement effect was context specific -the results have indicated that context conditioning is important, but not because it permits summation of excitation. Rather, as was the case with renewal, the rule of context is to disambiguate the significance of a stimulus that has mixed history of conditioning and extinction -conclusions are supported by study by Bouton

What important implications does the renewal effect have for behavior therapy?

-suggests that even if therapeutic procedure is effective in extinguishing a pathological fear or phobia in relative safety of therapists office, the conditioned fear may may easily return when the client encounters the fear CS in a different context -effects of excitatory conditioning readily generalize from one context to another -if you overcome your fear in one context, may not generalize to other contexts -fear is widespread context, overcoming fear specific context How do we overcome this? 1. conduct extinction in a variety of different contexts -extinction performance is less context specific if extinction training (or exposure therapy) is carried out in different contexts 2. conditioned inhibition training, 3. differential conditioning, 4. and presenting the CS explicitly unpaired with the US

What would happen if an extinction cue is compounded with a conditioned inhibitor during extinction training?

-there SHOULD be an interference rather than a facilitation of the extinction process -in the fear system a conditioned inhibitor is a safety signal indicating that the aversive US will not occur -if such a safety signal is compounded with a fear stimulus during extinction, the absence of the US will be fully predicted by the safety signal -Result: there won't be any error to encourage learning that the fear stimulus no longer ends in shock. Thus, **the safety signal will block extinction of the fear stimulus**

Reducing Spontaneous Recovery

1. Introduce a period of rest after extinction and then test for recovery -> repeat periods of periods of rest and testing -less and less recovery occurs with successive cycles of rest and testing 2. Interval between initial training and extinction -inconsistent results unfortunately -Myers, Ressler and Davis: reported fear extinction conducted 24-72 hours after fear acquisition showed usual spontaneous recovery, renewal, and reinstatement effects -if extinction was conducted 10-60 minutes after fear acquisition, recovery effects not observed -fear conditioning were more permanent if extinction was conducted right after acquisition 3. introduce cues associated with extinction -ex: introducing stimuli that were present during extinction can reactivate extinction performance (this is analogous to renewal by returning to context) -taste aversion learning as well as in appetitive conditioning preparations

Compounding Extinction Stimuli

1. Present two stimuli at the same time that are both undergoing extinction -can even deepen extinction of those cues Experiment: rats were first conditioned to press a response lever during each of three different discriminative stimuli, a light (L) and a noise and a tone stimulus (X and Y). During initial acquisition training lever pressing during these stimuli was reinforced on a VI 30-second schedule with food. Lever pressing was not reinforced when these stimuli were absent. Following acquisition, the light, tone and noise stimuli were each presented repeatedly by themselves with lever presses no longer reinforced. Responding during each of these cues declined to close to zero. Compound extinction trials were introduced to evaluate the possibility that sub-threshold tendencies may have remained. During second extinction pass, the light was presented simultaneously with one of the auditory cues (X). The other auditory cue (Y), continued to be presented alone without reinforcement, as a control. The effects of compound extinction were evaluated at the end of the experiment by testing responding during X and Y, each presented by itself Results: -L presented with X: substantial elevation of responding -> means that there was a summation of sub threshold responding that remained to the L and X stimuli despite their individual extinction treatments -compounding two extinction cues deepens the extinction of the individual stimuli -> suggests that extinction operates at least as an error-correction process like the RESCORLA WAGNER MODEL

Paradoxical Reward Effects

1. overtraining extinction effect 2. Magnitude reinforcement extinction effect 3. partial reinforcement extinction effect (PREE)

Number and Spacing of Extinction Trials

1. simplest way to increase impact of extinction: increase extinction trials -large numbers of extinction trials = profound decrease in conditioned response -taste-aversion, eyeblink, context conditioning 2. conduct extinction trials spaced close together in time (massed) rather than spread out -Cain, Blouin & Barad found greater loss of fear with massed extinction trials than with spaced trials - persisted to the next day -not sure if these results apply to appetitive behavior -produce a more rapid decrement in responding within a session -sometimes just temporary performance effect

discrimination hypothesis

A hypothesis explaining why Partial Reinforcement Extinction Effect occurs. In order for the subject's behaviour to change once extinction begins, the subject must be able to discriminate the changes in the reinforcement contingencies. In Continuous Reinforcement Schedules, every response is reinforced, so when extinction occurs, the subject can discriminate the change very easily, but in intermittent reinforcement schedules, it is harder to discriminate the change. -plausible explanation for PREE -Jenkins and Theios: Found that subjects that initially received partial reinforcement training responded more in extinction. Results indicate that the response persistence produced by partial reinforcement does not come from greater difficulty in detecting the start of extinction. Rather, subjects learn something long lasting from partial reinforcement that is carried over even if they eventually receive continuous reinforcement -> teaches subjects to not give up in the face of failure

Bouton and King

(from their abstract): Using barpress conditioned suppression, we studied the renewal of conditioned fear in rats, an animal model for the relapse of human fears and phobias. We demonstrated ABA renewal when the only differences between Contexts A and B included (1) their odor, (2) their location (i.e., side of room), and (3) unintended differences between copies of the same box at the two sites. Removing either the odor or location cues abolished the renewal effect. We then directly compared the effects of ABA and AAB procedures under two levels of context similarity. Although AAB renewal occurred, ABA renewal was stronger. Adding multiple context dis- tinctions to the three listed above did not significantly enhance either form of renewal. Finally, we directly compared the strengths of AAB, ABC, and ABA renewal. AAB renewal, though again significant, was weaker than ABA and ABC renewal, which did not differ significantly. Fear renewal (relapse) can thus be reduced by extinguishing the fear in the acquisition context, regardless of the nature of the test context. -Group NE: did not receive extinction: showed strongest degree of suppression -Group A: least suppression - Group B: showed substantial levels of suppression when first retuned to context A -> conditioned fear was RENEWED when Group B was removed from the extinction context (B) and returned to the context of original training (A) -the difference between GroupA and B is significant because these two groups showed similar losses of conditioned fear during the extinction phase -the fast that conditioned fear was renewed in Group B indicates that the loss of suppression evident during the extinction phase for this group did not reflect the unlearning of the conditioned fear response

Reducing Renewal

**Increasing the impact of extinction training is to reduce the renewal effect** 1. conducting extinction in several different contexts -helps to increase stimulus generalization of extinction performance, so as to reduce renewal when subjects are shifted out of the extinction context -doesn't always happen 2. present reminder cues of extinction in the renewal context -reactivating extinction performance in the renewal context -appetitive conditioning -exposure therapy with people afraid of spiders -ex: Participants who were instructed to mentally recall the treatment context showed less fear of spiders in a novel situation that participants who did not engage in the rimier exercise -> can be applied more broadly to increase generalization of treatment outcomes by encouraging clients to carry a card, repeat a short phrase, or call a help line whenever they are concerned about relapsing, to remind them of the therapeutic context

Frustration theory

-Abram Amsel A theory of the partial reinforcement extinction effect, according to which extinction is retarded after partial reinforcement because the instrumental response becomes conditioned to the anticipation of frustrative non reward. -continue to respond even when you expect non reinforcement -occurs in stages -intermittent reinforcement involves both rewarded and non rewarded trials -rewarded trials lead individuals to expect reinforcement and non rewarded trials lead them to expect the absence of reward -two competing expectations lead to conflicting behaviors: expectation of reward encourages responses & anticipation of non reinforcement which discourages responding -conflict resolved by responding -resolution occurs because reinforcement is not predictable in the typical partial reinforcement schedule -key to persistent responding in extinction: with sufficient training intermittent reinforcement results in learning to make the instrumental response when the subject expects non reward -once response becomes conditioned to expectation of non reward, responding persists when extinction is introduced -> there is nothing about the experience of continuous reinforcement that encourages subjects to respond when they expect non reward -continuous reinforcement does not produce persistence in extinction

Resistance to Change and Behavioral Momentum

-another way to think about response persistence in extinction: represents resistance to the change in reinforcement contingencies that occurs when the extinction procedure is introduced -behavioral momentum: characterize the susceptibility of behavior to disruptions -> based on an analogy to physical momentum in newtonian physics -> behavioral momentum states that behavior that has a great deal of momentum will also be hard to "stop" or disrupt by various manipulations research: -multiple schedules of reinforcement -popular because they enable investigators to compare the susceptibility of behavior to disruption under two different conditions in the same session and the same subject -different sources of disruption: extra food before the experimental session, extra food during intervals between components of multiple schedule, and terminating reinforcement two major conclusions: 1. behavioral momentum is directly related to the rate of reinforcement -higher rate of reinforcement produces behavior that has greater momentum and is less susceptible to disruption 2. behavioral momentum is unrelated to response rate -not as universal as #1 -two behaviors that occur at similar rates do not necessarily have similar degrees of influence on behavioral momentum -reinforcement rate primary determinant of behavioral momentum -Nevin and Grace

Nevin and Grace

-behavioral momentum should be increased by adding reinforcers to a component of a multiple schedule even if those reinforcers are not contingent on responding -there are two separable aspects of the discriminated operant that independently govern the rate at which a response occurs and the persistence of that response in the face of operant disruption such as punishment, extinction, differential reinforcement of alternative behaviors, etc. -relative resistance to change and preference both have been conceptualized as expressions of an underlying construct termed response strength, conditioned reinforcement value, or more generally, behavioral mass of discriminated operant behavior -major finding was the responding was less disrupted in the presence of the reinforced stimuli that was associated with the higher reinforcement rate

Sequential Theory

-capaldi The explanation for the PRE that assumes that nonreinforced responses create memories of trials on which the response was not reinforced. These memories can serve as a cue for persistent responding during extinction and can weaken responding by promoting generalization and decrement -major alternative to frustration theory -memory concepts -assumes subjects remember whether or not they were reinforced for performing instrumental response in the recent past -assumes during intermittent reinforcement raining, the memory of non reward becomes a cue for performing instrumental response -how this happens depends on sequence of rewarded (R) and non rewarded (N) trials that were administered -with enough experiences of this type -> subject learns to respond whenever it remembers not having been reinforced on the preceding trials -> learning creates persistence of the instrumental response in extinction -frustration theory and sequential theory as competing explanations of PREE -unlikely that one theory is correct and the other wrong -better way to think about it: theories point out different ways PREE can promote responding during extinction