Tuesday, April 12, 2011

Why Cues Can Be Reinforcing

Last week, while summarizing Kathy Sdao's session at Clicker Expo on cues, I wrote that cues can be used as reinforcers. I was pleased that several of you commented with examples of how you do this, and was intrigued that most of your cues-as-reinforcers were for behaviors the dog really enjoys, like retrieving or sniffing. Of course, this begs the question: if the behavior is inherently pleasurable, is it really the cue that's reinforcing?

 The cue "go get it" is exciting, sure, but not because I conditioned it that way.

Well, yes. The cue allows the behavior to happen. Ask any newly-licensed sixteen-year-old: permission to borrow the car is incredibly reinforcing. I use cues this way all the time. When Maisy and I are out, if she walks nicely instead of dragging me towards a smelly fire hydrant, I'll give her the “go sniff” cue as a reward. Of course, that behavior- the sniffing- is not something I taught her to enjoy, and the cue is simply letting her do something she wants anyway... but it's still reinforcing.

Kathy was talking about so much more, though. She said that any cue, for any behavior can be reinforcing. This is a powerful concept because if we can make the cue for an obedience exercise reinforcing, we will be able to reward our dogs in the ring. Of course, that means the cue for a rather innocuous behavior, like sit or down, must be reinforcing, which is a bit harder to think about. I mean, sure, dogs like to sniff, but what's so exciting about sitting?

The answer is that it's exciting because the trainer has taught the dog to find it exciting. This is done by repeatedly pairing the sitting action with a great outcome, like treats. As a result, if the trainer does her job well, the dog will become quite excited to sit. Karen Pryor talks about this in her book Reaching the Animal Mind, and calls the cue “a promise of happy outcomes.” She says the cue tells the dog “if you understand what I'm saying, and you carry it out correctly, you will definitely win” (page 35). In this way, she says, the cue becomes another kind of conditioned reinforcer.

Kathy called it a “tertiary reinforcer,” because it's the third one out. The cue precedes the click which precedes the treat. Of course, the cue isn't as strong as a click. While a click always results in a treat (or at least, it should), a behavior may not always result in a click. We don't click good tries, after all; we only click behaviors which meet our criteria. Therefore, the click is contingent upon the dog performing the behavior correctly, making it an opportunity... and opportunities can be reinforcing, too.

Let me explain what I mean by expanding upon Kathy's metaphor of traffic lights. A green traffic light is a cue for someone to drive. It tells the driver that pressing the gas pedal is allowed now. For people who like to drive, this will be reinforcing in and of itself because the green light is a cue that gives permission to do something they find pleasurable.

For some of us, though, it's not so simple. Although I don't hate driving, I don't find the action all that thrilling, either. Still, it gets me places I want to go, so despite my ambivalence towards the act itself, I like to drive because of the results. This works with our dogs, too. They may not find sitting all that exciting, but it gets them a tasty treat or a toss of the tennis ball. If this happens enough times, it can transform a relatively mundane action into something associated with a good outcome.

The cue also becomes associated with that good outcome. I have had a long history of driving after seeing a green light, and then ending up at my destination. This has happened often enough that now I want to see the green light, not because it will let me drive, but because it predicts that I will end up at my desired location. Similarly, our dogs get excited when they hear our cue, not because they want to sit, but because they want the treat that they think will come when they do. Thus, my green light- and my dog's cue- becomes reinforcing because of the opportunity it signals.

Of course, things can go wrong. I might get in an accident, and the dog might respond to his cue incorrectly. In both cases, neither of us will get what we want. If this happened too often, neither of us would find our respective cues very exciting anymore. Thankfully, these occurrences are rare- or at least they should be. Just as I should be able to safely drive my car before I get in it, the dog should be able to perform the cue before the trainer gives it. This is why Kathy had three entire sessions at Clicker Expo on how and when to attach cues to behaviors.

As fascinating as all this is, I really think it's only half the story. If we want to use cues to their fullest potential by using them as reinforcers, I think it's important to understand why cues sometimes fail to be reinforcing. Because it's true; while cues can be reinforcing, that doesn't mean they will be. Kathy identified one reason (the use of punishment in training) at Clicker Expo, but I think it's more complicated than that. In my next post, I'll share some of my ideas with you.

In the meantime, though, I'd love to hear what you guys think. Does my explanation make sense? Do you have any reinforcing cues that were conditioned instead of taking advantage of your dog's innate loves? How and when do you use them? Let me know!


Laura, Lance, and Vito said...

Did she talk at all about getting rid of the primary reinforcers or putting them on a thin schedule? It would seem that if you really wanted a cue for a non self rewarding behavior to act as a reinforcement it would not only have to have a strong reinforcement history but a history that was constantly maintained.

I think Lance really loves heeling, although I don't know that I would say the cue to heel is reinforcing, but I also maintain it quite a bit. Heeling is easier to fade rewards because it's a duration behavior but it usually still gets rewarded at the end.

I can see his obedience go out getting to the level of a reinforcer but it still goes back to the behavior of running as enjoyable. But even then I don't know how it would maintain without keeping the rate of reinforcement high. In theory I guess it could be reinforced by the cue to do the jumps. But then we're lucky in that jumping is a behavior that's rewarding by itself. Could a cue to finish really keep as a reward to front if you really lessen the treats?

Joanna said...

I really like your explanations -- I think they're always clear and show good critical thinking skills on your part. :)

I keep up with the clickcompobed Yahoo list, and one of the frequent discussions is attaching high value to obedience cues/behaviors. Because of that, I've been trying hard to make training plain old sit and down fun as well as accurate. I've also sometimes been cuing Dragon to target my hand before he does a Nosework search, to try to build different kinds of reinforcement history and therefore more excitement and more value into it. That's a really good trick to use in the ring, if the dog likes it enough.

Ci Da said...

Garrett had an interesting blog post about criteria which I think relates to your post, at least abstractly. Address here: http://susangarrettdogagility.com/2011/04/criteria-is/

I mentioned that I was working to increase the value in my dog jumping into my arms as a reward for attention while outside. Arguably jumping into someone's arms can be intrinsically rewarding for a dog, but for my dog she was nervous about it when we first started and the behaviour wasn't naturally rewarding for her. There's such a thin line between the cue and the action that it's hard to distinguish where the rewards come from, and I think it's supposed to be like that.

Another example of a reinforcing cue I've been working on is a nose touch without verbal cue (like the commenter above). The cue is my hand outstretched, and it is a reward for my dog paying attention to me. When I stretch my hand out my dog's little nubbin tail starts wagging and she dives for it -- she really seems to enjoy it. Now, I'm sure most dogs aren't huge fans of scrunching their noses up against something -- I would imagine it might be uncomfortable. But because of the amount of value I've put into the game (it's heavily rewarded) my dog gets excited when she's offered the opportunity to play.

To relate this back to the Garrett blog entry I mentioned, I think it's the trainers responsibility to make training a joyful experience for the dog. It's absolutely possible to take what was previously a chore for a dog and turn it into a high-value behaviour. It just takes some skill and enthusiasm.

Sophie said...

I do think that for a cue to become reinforcing, it must be given a reward *every* time. If you put it on a variable reinforcement schedule, it may fail to be reinforcing because it isn't an opportunity to self-reward (through responding appropriately to the cue, and earning whatever it is that they want): it's a -possible- opportunity to do so.

Then again, this is based mostly on my own observations of my dogs.

Jess learned some behaviours that were quickly put on a variable reinforcement schedule - and she isn't inherently excited to perform them. But if I ask her to weave between my legs, or go in a circle on a stool, or spin, her tail is wagging immediately; whilst if she's asked to beg, she's much slower to respond, and doesn't seem genuinely excited to perform the behaviour.

I've been trying, with Lola, to always (or near-always) give some sort of reward for a behaviour. If she sits, I'll open the door. If she downs, I throw the ball. If she targets Jess with her paws ('find your sister!'), I might present her favourite tug toy to her (saved only for training sessions), etc. It could be a result of very minimal use of aversives, or a result of constantly getting some sort of varied reward, but she consistently has happy, relaxed body language and a genuine look of excitement when she's asked to do something she knows.

I'm so jealous that you got to go to Clicker Expo though! They need to do a UK one, now. :)

Crystal said...

I think Sophie has it- that for a cue to be reinforcing, the behavior needs to have a relatively high rate of reinforcement. I plan to write about that in my next entry, by the way. :)

Denise said...

I cannot see the value of this "cue as reinforcer" if you have to heavily reinforce it on an ongoing basis. Might as well just say "yes" or click with your mouth in the ring. Unfortunately all of these are basically lying to your dog, and he'll figure that out soon enough. To me, it makes more sense to build in value for yourself (also a tertiary reinforcer, by the way) in the ring. Teach dog to expect to work a long time to earn a reward, even in training. Make the work interesting whenever possible, simply because its fun to learn and do stuff with your body, not just for a cookie. I'm really struggling to see the value of this "cue as reinforcer" because the dog will soon learn that it doesn't have value in the ring. Then you have a negative situation...the ring is a bad place to be. I'd avoid that at all costs. Why not use a very thin variable reinforcement schedule, adding any naturally reinforcing activities whenever possible?

Crystal said...

Denise, I'm not an expert on this- it's a relatively new concept to me- but I think the point is that you can use cues as reinforcers to create behavior chains. Dog does behavior A, is rewarded by the cue for behavior B, which is rewarded by the cue for behavior C, (going on for behaviors D through however many it takes to create your chain) which is then rewarded with a treat (play, whatever). Done this way, I think it would just be another way teach your dog to work for prolonged periods of time.

This should get you through all the elements of an exercise. Then, between exercises, you could reward the dog with a behavior the dog truly enjoys, praise, etc. Those rewards would happen in the ring, which should prevent the ring from becoming a bad place.

In other words, a cue can reinforce the proper response to another cue. Ultimately there will be some primary reinforcer, but I don't think it has to be as heavily as you're thinking. It certainly doesn't need to be continuous reinforcement. You just need to find the right balance.

That said, I agree that what you're describing is also good training: making the job fun and interesting, being a person my dog wants to work with, and creating a variable reinforcement schedule are all things I try to do. And I appreciate your concern about maintaining the ring as a good place to be- "ring wise" dogs are no fun at all!

Denise said...

Hmm. How would you differentiate that from backchaining?

Crystal Thompson said...

Denise, I'm afraid I don't have an answer for you. I'm not terribly well-versed in chaining in general. My very unsophisticated understanding is that the difference between chaining and back-chaining is simply the order in which you teach each behavior in the sequence. So, chaining would involve teaching behavior A then B then C, while back-chaining would involve teaching behavior C then B then A.

Denise said...

ok. Backchaining automatically does what you are saying since a dog works into a comfort zone (a cue that is known and hopefully sufficiently reinforced that the dog is looking forward to it). But backchaining is a teaching concept, not a trialing one.

I can see how this cue as reinforcer could be used in training but cannot see how it helps in competition. I've often heard people talk about using cues to reinforce prior cues in competition, but I've noticed those people either don't compete, or do competitions where the reinforcers are not removed. That doesn't' mean there isn't something to the theory, but since my area of interest is getting into competition where there is no reinforcer, I'm actively looking for new ideas that will help with that end goal.

Thanks for writing about the topic.

Crystal Thompson said...

Denise, I guess I can't speak to how it would hold up in competition. In theory, it sounds good, but we aren't trialing right now, and when we return, it will be to venues that allow food in the ring. Thanks for talking it through with me, though. It was a very interesting conversation!