Chapter 7 Instrumental Conditioning: Motivational Mechanisms

Kenneth Spence

-Colleague of Hull, stated that the environmental cues present during reward become associated with the reward and subsequently produce a conditioned or anticipatory goal response (1956).

Problems with Behavioral Regulation Approaches

-Problem: how the bliss point or preferred combination of activities is determined -typically bliss point determined during a free operant baseline period -> choices observed during this baseline period are used to predict performance after an instrumental conditioning procedure -> for such predictions to work, one has to assume that responses performed in absence of experimenter imposed constraints are basically the same as the responses that occur when an instrumental contingency is in effect -responses that occur during free-operant baseline period do not always have the same value as responses that occur as part of an arranged instrumental contingency -doing something when there are no externally imposed requirements (jogging for fun) is different from doing the same thing when it is required by an instrumental contingency (jogging in gym class) -Problem: behavioral regulation and economic approaches to instrumental behavior do not say much about how organisms manage to defend a preferred combination of goods or activities -does not tell us the processes that are involved in making trade-offs

Hierarchical S(R-O) Relations

-R-O associations cannot act alone to produce instrumental behavior -instrumental response activates an expectancy of the reinforcer is not sufficient to tell us what caused the espouse in the first place -an additional factor is required to activate the R-O association -> one possibility: R-O association is activated by the stimuli (S) that are present when the response is reinforced -> according to this view, S does not activate R directly but rather activates the R-O association -the subject comes to think of the R-O association when it encounters S and that motivates it to make the instrumental response

Behavioral Regulation and the Behavioral Bliss Point

-a shift away from the physiologically optimal or homeostatic level triggers changes that serve to return them system to the homeostatic level -Behavioral regulation theories assume that analogous mechanisms exist within respect to behavior --focus on the extent to which an instrumental response-reinforcer contingency disrupts behavioral stability and forces the individual away from its preferred or optimal distribution of activities --if preferred or optimal balance of activities is upset, behavior is assumed to change so as to correct the deviation from the homeostatic level

Response interactions in pavlovian instrumental transfer

-an elicited emotional state is not necessary to understand an outcome -an elicited emotional sate is not necessary if the classically conditioned stimulus elicited overt responses that were similar to the instrumental behavior -investigators concerned with possibility that results of Pavlovian instrumental transfer experiments due to fact that Pavlovian CS's elicit overt responses that either interfere with or summate with the behavior required for instrumental reinforcement -# of experimental strategies have been designed to rule out such response interactions -these strategies generally successful in showing many instances of Pavlovian instrumental transfer not produced by interactions between overt responses

Charlop, Kurtz, and Casey

-assessed the efficacy of using autistic children's aberrant behaviors as reinforcers to increase their correct task responding - the conditions were delayed echolalia, food, and varied (food or delayed echolalia) -Results indicated that, in general, task performance was highest when brief opportunities to engage in aberrant behaviors were provided as reinforcers. Edibles were associated with the lowest performance. Furthermore, no negative side effects (e.g., an increase in aberrant behaviors) occurred. -high probability responses can serve to reinforce lower probability responses, even if the reinforcer responses are not characteristic of normal behavior

Habits

-automatic -same way each time -don't think about it -constitute 45% of human behavior

How could we test the idea that an S-O association can motivate instrumental behavior?

-basic experimental design -Pavlovian instrumental transfer test

The S-R association and the law of effect

-basic structure of instrumental conditioning procedure permits development of several different types of associations

Imposing and Instrumental Contingency

-behavioral-regulation theory states that organisms will defend against challenges to the behavioral bliss point, just as physiological regulation involves defense against challenges to a physiological set point -free baseline behavioral bliss point usually cannot be reestablished after an instrumental contingency has been introduced -that is often the dilemma posed by an instrumental contingency -> it does not permit getting back to the bliss point -Although instrumental contingency makes it impossible to return to behavioral bliss point, does not mean that the bliss point becomes irrelevant -> when the goal cannot be reached, the redistribution of responses between the instrumental and contingent behaviors becomes a matter of compromise. The rate of one response is brought as close as possible to its preferred level without moving the other response too far away from its preferred level

Evidence of R-O associations: how do you determine if instrumental conditioning leads to the learning of R-O association?

-devalue the reinforcer after conditioning to see if it decreases with instrumental response -US devaluation is used to determine whether the conditioned response is mediated by a CS-US association -if US devaluation after conditioning disrupts the CR can conclude that the CS was mediated by the CS-US association -reinforcer devaluation has been used to determine if an instrumental response is mediated by as association between the response and its reinforcer outcome

Applications of the Premack principle

-enduring impact in the design reinforcement procedures used to help various clinical populations -Mitchell and Stoffelmayr -Charlop, Kurtz, and Casey

Behavioral Regulation

-focuses on how instrumental conditioning procedures put limitations on an organism's activities and cause redistributions of those activities

Economic Concepts and Response Allocation: Consumer Demand

-fundamental to application of economic concepts to problem of reinforcement is the relation between price of commodity and how much of it is purchased -> called the DEMAND CURVE -the degree to which price influences consumption is called elasticity of demand -e.g. of elasticity: demand for candy highly elastic -> more candy costs the less people buy, however demand for gas is less elastic -> regardless of how expensive people still generally buy -concept of consumer demand used to analyze variety of major behavior problems including eating and drug abuse -e.g.: children increased purchases of healthy foods as the price of unhealthy alternatives was increased -a decrease in price was more effective in encouraging the selection of healthy foods than messages encouraging patrons to eat healthy

Income Level

-higher your income, the less deterred you will be by increases in price -more responses or time animals have available, the less their behavior is influenced by increases in the cost of the reinforcer -income level also influences the choice of substitutes -at the low income level, increases in the price of unhealthy food led to increased choice of the healthy alternatives. In contrast, at the high income level, the children continued to purchase unhealthy, but preferred, foods as the price of these foods went up -at the high income level, increases in the price of the unhealthy foods reduced the choice of substitutes

Economic Concepts and Response Allocation: How do the constraints of an instrumental conditioning procedure produce changes in behavior?

-in economic arena, restrictions on behavior are imposed by our income and the price of the goods that we want to purchase -in instrumental conditioning situations, the restrictions are provided by number of responses an organism is able to make (it's "income") and the number of responses required to obtain each reinforcer (the "price" of the reinforcer) -similarities between economic restrictions in the marketplace and schedule constraints in instrumental conditioning

Price Range

-increase in price has less effect at low prices that at high prices -at low prices there is little change in the number of reinforcers obtained as the price increases a bit. With a small increase in price at the low end of the price range, participants adjust by increasing the number of responses they perform to obtain the reinforcer. However dramatic declines occur in the number of reinforcers obtained in the high range of prices

Viewing Reinforcement contingencies in a broader behavioral context

-instrumental contingencies occur in context of variety of reposes and reinforcers the participant has available -broader behavioral context can significantly influence how the person adjusts to a schedule constraint -e.g.: student likes iPod and TV equally, restriction from TV may not increase homework time, may just switch over to listening to iPod -focusing on just instrumental response and its antecedent and consequent stimuli is not enough -effect of particular instrumental conditioning procedure may depend on what alternative sources of reinforcement are available to the organism, how those reinforcers are related to the particular reinforcer involved in the instrumental contingency, and the costs of obtaining those alternative reinforcers

Antecedents of behavioral regulation

-reinforcers were special stimuli that strengthened instrumental behavior

Availability of substitutes

-most important factor -whether increase in the price of one item cause a decline in consumption depends on the availability (and price) of other goods that can be used in place of original item -increases sensitivity of the original item to higher prices -contemporary analyses of drug abuse take into consideration substitute reinforcers -Murphy, Correla, & Barnett: interested in how to reduce college students decisions about drinking are related to relative availability and price of substance free alternative activities; findings: behavioral economic theory predicts that college students' decisions about drinking are related to the relative availability and price of alcohol, the relative availability and price of substance free alternatives, and the extent to which reinforcement from delayed substance-free outcomes is devalued relative to immediate reinforcement associated with drinking -Foltin: Interested in whether food intake would decrease as the price of food was increase, and whether the availability of alternative reinforcers would influence this function. the largest effect was obtained with the intermediate cocaine concentration. With this concentration availability of cocaine on the alternative lever dramatically increased the elasticity of demand for food. Study shows powerful example of substitutability on the elasticity of demand (baboons obtained fewer food pellets as the behavioral price (lever presses) of food was increase. Also the availability of cocaine on the alternate response lever increased the elasticity of demand for food)

R-O and S(R-O) Relations in Instrumental Conditioning

-neither the S-R nor the S-O association involves a direct link between the response (R) and the reinforcer or outcome (O) -counterintuitive: if you asked someone why he/she was performing an instrumental response, the reply would be that he or she expected the response (R) to result in the reinforcer (O) -intuition suggests that instrumental behavior involves R-O associations -weird thing about associative structure of instrumental conditioning assumed by two-process theories is that S is assumed to become associated directly with O on the assumption that the pairing of S with O is sufficient for the occurrence of classical conditioning -> CS-US pairings are not sufficient for the development of pavlovian associations. The CS must provide info about the US or in some way be related to the US -in instrumental conditioning situation, the reinforcer (O) cannot be predicted from S alone. O occurs if the individual makes response (R) in the presence of S -> instrumental conditioning involves a conditional relation in which S is followed by O only if R occurs -> this is ignored in the Two Process Theory

Pavlovian suppression of responding instrumental transfer

-pavlovian CS was paired with shock -presentation of the CS+ for shock was tested when subjects were lever press ion for food -result: pavlovian CS suppressed the instrumental lever-press behavior

Wood and Neal

-proposed new comprehensive model of human habits -idea that habits arise when people repeatedly use a particular behavioral means in particular contexts to pursue their goals -once goal is acquired habits are performed without mediation of a goal -the habitual response is an automatic reaction to the stimulus context in which the goal was previously obtained -similar to Thorndikes S-R Association

Explanation of reinforcement effects

-reinforcement effect identified by an increase in the occurrence of an instrumental response above the level of that behavior in the absence of the response-reinforcer contingency -access to the reinforcer is restricted; to gain more opportunity to engage in the reinforcer response, the individual has to perform more of the instrumental response -increased performance of the instrumental response (a reinforcement effect) results from behavioral-regulatory mechanisms that function to minimize deviations from the behavioral bliss point

Evidence of R-O associations

-the subject has to experience the new value of the reinforcer; the subject has to taste how bad the food became after it was paired with illness or how unpalatable the food is once the subject is no longer hunger -> called incentive learning -only if the subject has had a chance to learn what the new incentive value of the reinforcer is will its instrumental behavior be reduced -the hypothesis if it was an S-O association: if the devaluation of one of the reinforcers had altered the properties of S, that should have changed the two responses equally -what actually happened: devaluation of a reinforcer selectively depressed the particular response that had been trained with that reinforcer -> indicates that each response was associated separately with it's own reinforcer -> learn separate R-O associations

Conditioned Emotional States or Reward-Specific Expectancies?

-two process theory assumes that classical conditioning mediates instrumental behavior through conditioning of positive or negative emotions depending on emotional valence of reinforcer -animals also acquire specific reward expectancies instead of just categorical positive or negative emotions during instrumental and classical conditioning -expectancies for specific rewards rather than a general positive emotional state determined the results in the transfer test -under some circumstances, individuals acquire reinforcer-specific expectancies rather than the more general emotions during instrumental and classical conditioning -reinforcer-specific expectancy learning is a challenging alternative to the two-process theory (this alternative is also based on the assumption that instrumental conditioning involves the learning of an S-O association)

Contributions of Behavioral Regulation

1. Behavioral regulation and the Premack principle moved us away from thinking about reinforcers as special kinds of stimuli or as special kinds of responses - reinforcement effects are regarded as the consequences of schedule constraints on an organism's ongoing activities 2. Instrumental conditioning procedures are no longer considered to stamp in or to strengthen instrumental; rather instrumental conditioning is seen as creating a new distribution, or allocation, of responses 3. there is no fundamental distinction between instrumental and reinforcer responses - rather, instrumental and reinforcer responses are distinguished only by the roles assigned to them by an instrumental conditioning procedure 4. Behavioral regulation and behavioral economics embrace the assumption that organisms respond so as to maximize benefits - the bliss point approach suggests that the optimal distribution of activities is determined not only by physiological needs, but also by the organism's ecological niche and natural or phylogentically (what?) determined response tendencies - it is not always clear what is being maximized 5. behavioral regulation and behavioral economics have provided new and precise ways of describing constraints that various instrumental conditioning procedures impose on an organism's behavioral repertoire - all of the organism's response options at a given time must be considered as a system - changes in one part of the system influence changes in other parts

Name the three events to consider in an analysis of instrumental learning

1. Stimulus context (S) 2. instrumental response (R) 3. response outcome (O) (or reinforcer)

Two distinctively different approaches to why instrumental behavior occurs

1. Thorndike and Pavlov 2. Skinnerian

in what ways did the premack principle advance our thinking about reinforcement?

1. encouraged thinking about reinforcers as responses rather than as stimuli 2. greatly expanded the range of activities investigators started to use as reinforcers 3. any behavior could serve as a reinforcer provided that it was more likely than the instrumental response

List the three events to consider in an analysis of instrumental or operant conditioning that Skinner subscribed to (aka three term contingency)

1. the instrumental (R) response occurs in the presence of a 2. distinctive stimuli (S) and 3. results in delivery of the reinforcer outcome (O)

Clark Hull

1884-1952; Field: motivation; Contributions: maintains that the goal of all motivated behavior is the reduction or alleviation of a drive state, mechanism through which reinforcement operates

response deprivation hypothesis

A model built upon Premack Principle for predicting whether contingent access to one behavior will function as reinforcement for engaging in another behavior based on whether access to the contingent behavior represents a restriction of the activity compared to baseline -a hypothesis that states when access to one activity is restricted to below baseline levels, the person will engage in the targeted activity at a level exceeding baseline rates in order to gain access to the deprived activity; restricting access to below baseline levels, then, serves as an establishing operation -new procedure for creating reinforcers -aka disequilibrium model -eg: tracing and filing both low probability; restrict tracing -> becomes reinforcer for filing -contrary to premack principle -> shows that response deprivation is more basic to reinforcement effects than differential response probability

differential probability principle

A principle that assumes that reinforcement depends on how much more likely the organism is to perform the reinforcer response than the instrumental response before an instrumental conditioning procedure is introduced. The greater the differential probability of the reinforcer and instrumental responses during baseline conditions, the greater is the reinforcement effect of providing opportunity to engage in the reinforcer response after performance of the instrumental response. AKA the Premack principle

Premack Principle

A principle that states that making the opportunity to engage in a high-probability behavior contingent on the occurrence of a low-frequency behavior will function as reinforcement for the low-frequency behavior. -AKA differential probability principle -eating will reinforce bar pressing because eating is typically more likely than bar pressing

Consummatory Response Theory

A theory that assumes that species-typical consummatory responses (eating, drinking, and the like) are the critical features of reinforcers. -attributes renforcement to these species typical behaviors -asserts that species-typical consummatory responses are themselves the critical feature of reinforcers -showed the saccharin can serve as an effective reinforcer even though not nutritious (no biological need) -radical innovation because it moved the search for reinforcers from special kinds of stimuli to species types of responses -theory assumed that consummatory responses (e.g., chewing, swallowing) are fundamentally different from various potential instrumental responses, such as running, jumping or pressing a lever

Premack Experiment

Conducted with young children. Premack gave the children two response alternatives, which were either eating candy or playing a pinball machine. These responses were measured based on which response was more probable for each individual child. Some of the children preferred one activity over the other. In the second phase of the experiment, the children were tested with one of two procedures. In one procedure, eating was the reinforcing response, and playing pinball served as the instrumental response. This means that children had to play pinball in order to eat candy. The results were consistent with the Premack principle, only the children who preferred eating candy over playing pinball showed a reinforcement effect. -more probable behaviors will reinforce less probable behaviors -a person will perform a less (low) desirable activity to get to the more desirable (high) activity

Determinants of the Elasticity of Demand

Economic concepts have helped to identify three major factors that influence how schedule constraints shape the reallocation of behavior 1. Availability of substitutes 2. Price Range 3. Income Level

The concept of consumer demand has been used to analyze _________ behavior by considering the number of _______ performed to be analogous to money and the _______ obtained to be analogous to the commodity that is purchased. The price of a reinforcer then is the time or number of _________ required to obtain the reinforcer. Thus, the price of the reinforcer is determined by the ____________ ____ ________________.

Instrumental; responses; reinforcer; responses; schedule of reinforcement

How would the introduction of an instrumental contingency between studying and watching TV disrupt the students behavioral bliss?

It depends on the nature of contingency

Mitchell and Stoffelmayr

Mitchell & Stoffelmayr (1973) use Premack's principle in schizophrenics Reinforcement items like candy, cigarettes, etc. usually not effective in schizophrenic patients Sitting is a highly probable behavior in negative-symptom schizophrenics (catatonia, social withdrawal) therapists made sitting contingent on doing small amount of work or activity; improved their negative symptoms In unruly nursery-school children, high probability behaviors (running around, screaming) made contingent on low probability behaviors (sitting quietly, paying attention)

Why are the results of Colwill & Rescorla not explained by S-R association?

S-R associations do not include the reinforcer. Therefore, devaluation of the reinforcer cannot alter behavior mediated by an S-R association -instrumental behavior becomes habitual and insensitive to reinforcer devaluation if a single instrumental response is followed by the same outcome over an extended period of training -this effect of extended training is not observed if several instrumental responses are trained, each with its own reinforcer

Behavioral Bliss Point

The preferred distribution of an organism's activities before an instrumental conditioning procedure is introduced that sets constraints and limitations on response allocation. -e.g.: watching tv and homework -> no constraints: watches 60 minutes of T.V. for every 15 minutes of homework is the bliss point

Who developed the idea of the S-R association

Thorndike

S-R association

Thorndike; associative approach; S-R association is the key to instrumental learning and central to his Law of Effect, with the outcome only mediating the relationship; motivation for the behavior is the activation of the S-R association by exposing the subject to contextual stimuli (S) in the presence of which the response was previously reinforced

Law of Effect

Thorndike; used in an experiment with cats and a puzzle box; principle of reinforcement; behavior consistently rewarded will be 'stamped in' as learned behavior, and behavior that brings about discomfort will be 'stamped out'; satisfying effect (reinforcement) is likely to be performed again, whereas behavior that brings about negative effect (punishment) is likely to be suppressed

Hierarchical S(R-O) Relations: Skinner suggested that S, R, O in instrumental conditioning are connected through what kind of relation?

a conditional S(R-O) relation

minimum deviation model

a model of instrumental behavior, according to which participants respond to a response-reinforcer contingency in a manner that gets them as close as possible to their behavioral bliss point. -next best thing to behavioral bliss point

Two-Process Theory

assumes that there are two distinct types of learning: (1) pavlovian and (2) instrumental conditioning -theory further assumes that these two learning processes are related in a special way -during course of instrumental conditioning, the stimuli (S) in the presence of which the instrumental response is reinforced, become associated with the response outcome (O) through pavlovian conditioning, and this results in an S-O association

The second strategy for analyzing motivational processes in instrumental learning is ______ ______

behavioral regulation

The S-R association stipulates an outsourcing of _______ control to ______ cues that were, in the past, contiguous with performance.

behavioral; contextual

Evidence of R-O associations: colwill and rescorla

classical pairing of either food+LiCl or sugar water+LiCl, rats would only press lever to the "good" side not to the devalued response; concluded that a direct R-O association formed during instrumental conditioning -important finding: was that the rats were less likely to make the response whose reinforcer has been made aversive by pairings with lithium chloride -ex: if sucrose was used to reinforce responses to the left and an aversion was the conditioned to sucrose, the rats were less likely to push the rod to the left than to the right

behavioral regulation

concerned with how an instrumental conditioning procedure limits an organisms free flow of activities and the behavioral consequences of such constraints -considers the motivation of instrumental behavior from a more molar perspective -considers long-term goals and how organisms manage to achieve those goals within the context of all of their behavioral options

The instrumental response occurs in the _______ of specific environmental stimuli

context

According to two-process theory, conditioned suppression occurs because the CS+ for shock elicits an emotional state of _____ that is contrary to the _____ emotion or _____ (expectancy) that is established in instrumental conditioning with food

fear; positive; hope

behavioral regulation theory views instrumental behavior from a more ______ perspective. Because it takes a _____ approach, behavioral regulation does not provide as convenient a framework for studying neural mechanisms of _______ ______.

functional; molar; instrumental learning

Resurgence of interest in S-R mechanisms occurred in order to characterize ______ ______ in people.

habitual behavior

Thorndikes S-R association is also being seriously entertained to explain what?

habitual nature of drug addiction

Pavlovian Instrumental Transfer: As predicted, the presentation of a Pavlovian CS for food _____ the rate of instrumental responding for food. This presumably occurs because the positive emotion elicited by the _____ for food summates with the _______ motivation that is involved in lever pressing for food.

increases; CS+; appetitive

Johnson and Bickel

investigated the elasticity of demand for cigarettes and money in smokers who were not trying to quit -the greatest elasticity of demand was evident for the 5 cent monetary reinforcer. Here, the number of reinforcers obtained started decreasing as soon as more than three responses were required to obtain the 5 cents and dropped quickly when 100 or more responses were required. With the 25 cent reinforcer, the demand curve did not start to decline until the response requirement exceeded FR 300. As might be expected, the participants were most resistant to increases in the price of puffs at a cigarette. When cigarette puffs served as the reinforcer, the number of reinforcers obtained did not start to decline until the response requirement was raised above an FR 600. These results show that the participants were willing to make many more responses for puffs at a cigarette than they were willing to perform for the monetary rewards.

An important implication of the Law of Effect is that instrumental conditioning does not involve _______ ____ ___ _______ or the _______ _______ ______ and the ________ _______.

learning about the reinforcer (O); relation between the response; reinforcing outcome (the R-O association)

staddon

minimum-deviation model

the associative approach takes a _____ perspective. It focuses on ______ responses and their specific stimulus ______ and ______.

molecular; individual; antecedents; outcomes

Because associations can be substantiated in the nervous system, the associative approach also provides a convenient framework for studying the ______ mechanisms of ______ conditioning

neural; instrumental

classically conditioned stimuli elicit not only emotional states, but also _____ _____

overt responses

pavlovian instrumental transfer test

phase 1. subjects receive standard instrumental conditioning phase 2. subjects receive a pure pavlovian conditioning procedure phase 3. critical transfer where the subjects are again permitted to perform the instrumental response, but now the pavlovian CS is presented periodically -if a pavlovian S-O association motivates instrumental behavior, then the rate of instrumental behavior should increase when the CS is presented -determines how an independently established Pavlovian CS transfers to influence or motivate instrumental responding

Various appetitive reinforcers are assumed to lead to a common ______ emotional state and various aversive stimuli are assumed to lead to a common _______ emotion.

positive; negative

With _______ use, taking the drug becomes _____ in the sense that it becomes an automatic reaction to ______ cues that elicit drug seeking behavior, without regard to its consequences

repetitive; habitual; contextual

the S-O association activates a _____-_____ or _______ state

reward expectancy; emotional

Pavlovian conditioning is the basic process of ______ learning

signal

Behavioral regulation was developed within what tradition?

skinnerian tradition

Specification of an instrumental response ensures that the participant will always experience certain distinctive _______ in connection with making a response

stimuli (S) -stimuli may involve place where response is to be performed -the texture of the object participants in to manipulate, or distinctive olfactory or visual cues -reinforcement of instrumental response will result in pairing (S) with the reinforcer or response outcome (O)

The law of effect assumes

that the only role of the reinforcer is to strengthen the S-R association -the reinforcer itself is not a party of participant in this association

Thorndike perspective involves:

the associative structure of instrumental conditioning -relies heavily on the concept of associations and hence is compatible with the theoretical tradition of pavlovian conditioning