Thursday, April 14, 2011

Musings on Why Cues May Not Be Reinforcing

In my last post, I tried to explain why cues can be reinforcers, mostly because I think it's mind-blowingly awesome: how great is it that I can take a legal reward in the ring? Or that I'll always have something for those times that I run out of treats? The concept is awesome, however it's also confusing. One of the cardinal rules of reinforcement is that the receiver gets to decide if something is reinforcing, so just because a cue can be reinforcing doesn't mean it will be.

Of course, this begs the question: why not? What happens to prevent a cue from becoming reinforcing? At Clicker Expo, Kathy Sdao identified one reason: the use of punishment in training. This absolutely makes sense to me. If the reason a cue becomes reinforcing is because the dog sees the cue as an opportunity to earn reinforcement, then it makes perfect sense that punishment would interfere with that. But I don't train with punishment, and not all of my cues are reinforcing. What gives? Am I doing something wrong, or is there something else at work here?

In search of answers, I went right to the source: Kathy Sdao herself. Last week, I had a twenty minute phone conversation with her, and the ultimate answer was: we don't really know. But we came up with some pretty good ideas. Now mind you, there really isn't a lot of science on it yet, so what follows should not be taken as certain truth. These are theories, so if (when) I write something illogical or silly, don't blame it on Kathy! She was so incredibly nice and generous with her time that I'd hate for someone to attribute one of my dumb ideas to her.

Before we go any further, I need to pick on myself first. The reason some of my cues aren't reinforcing is probably because Maisy just doesn't understand them. I'm really not very good at teaching cues (I find it terribly boring), so Maisy seems to guess which behavior I'm asking for most of the time. She's a pretty good guesser, mind you, but it is clear to me that she has trouble discriminating between even simple things like “sit” and “down.” All that uncertainty no doubt colors her perception of the cues.

But what about other dogs? You know- the ones who are lucky enough to have a good handler and have been trained to understand what cues mean and respond reliably. Why is it a cue may not reinforcing for a dog who's been trained with positive methods? This is the question I explored with Kathy, and the one I want to write about today.

First, it is entirely possible that the trainer only thinks she's training positively. Now, I'm not talking about trainers who are suffering from some misconception about what positive training entails. I'm talking about trainers who understand learning theory and know how to use clickers and treats, yet unintentionally do something that the dog finds aversive. Maybe she intimidates the dog by leaning over him unconsciously, or maybe her voice has just an edge of frustration to it. There are many ways our body language can effect the way our dogs respond, and I doubt most of us recognize that we're even doing it!

And then there is luring. Could luring interfere with the end result? Don't get me wrong- I use luring when it makes sense- but as Kathy put it, there is a “continuum of coercion” at play when we lure behaviors. She didn't use the word coercion to imply that it's evil or wrong; instead she defines it as the dog having no choice in how to respond. Isn't it possible that the dog is so into the lure that he isn't really thinking about what he's doing? And by extension, if he's not thinking, can he really be making a choice about what to do? Instinctively following the lure may mean that the dog is being coerced, no matter how nicely. If so, it's possible that the cue for the lured behavior won't be reinforcing. Of course, since it's a continuum, it's equally possible that the dog is thinking, that dog does have a choice, and that the cue could end up being reinforcing as a result. But it's interesting to consider that even methods that are widely considered "positive" could have a downside.

Speaking of how things are taught, Kathy and I talked about the idea that something could happen in the acquisition stage of the behavior that would effect the eventual cue. This could be something as simple (and as common!) as the trainer's criteria being too high or the rate of reinforcement too low- to the point that the dog felt frustrated with the behavior, or was left with some lingering confusion. Or maybe the trainer's timing is so poor that the dog can't quite figure out what he's supposed to be doing? Couldn't this affect a dog's perception of the behavior, and ultimately, how he feels about the cue? It might... it might not... it would depend on the dog, of course, but Kathy said it seemed plausible.

Speaking of early stages of learning, maybe something went so wrong that it forever tainted the dog's view of that behavior. I'm specifically thinking of single-event learning, wherein a dog forms a long-lasting and negative association with something in just one trial. If a dog inadvertently received punishment from the environment while learning a new behavior- for example, a socially shy dog learning something in an overwhelming group class- it seems possible that the dog could associate those initial bad feelings with the behavior. It seems to me that this might ultimately affect the cue as well.

Although this seems more likely during the critical early stages of learning, I imagine that single-event learning could also happen later on, too. Perhaps a loud clap of thunder happened during a training session with a storm-phobic dog, or there was a lot of static electricity on a given day, resulting in accidental little shocks each time the trainer fed the dog a treat. Depending on how aversive the dog found this, couldn't this affect the behavior (and the cue) too?

But let's put aside blame for a minute and pretend the trainer did everything perfectly. This hypothetical trainer is a brilliant shaper who is great at adjusting criteria so that the dog receives a high rate of reinforcement with minimal frustration, does a great job of taking environmental factors into consideration, and is very conscious about her body language so that it doesn't interfere with her training. Is it possible her dog wouldn't find cues reinforcing?

I think so, if the behavior was “self-punishing.” This would be the opposite of self-reinforcing behaviors where the dog gets some kind of internal relief or inherent joy from engaging in a behavior (typically one the trainer doesn't want). A self-punishing behavior would be something that makes doing a particular behavior painful or unpleasant. It might happen when a dog has some kind of physical condition like arthritis or hip dysplasia that makes it physically painful to do something. Or it might happen when the behavior itself is scary. For example, Maisy doesn't like to step on things that move, like wobble boards, and yet she will do it over and over again, simply because I am asking her to (and if I'm honest, because there are treats involved). I have no idea why she persists- it's clear that it scares her each and every time- but she continues to do it anyway. For her, the cue “step up” will probably never be reinforcing.

Finally, I would be remiss if I didn't mention a plain old lack of reinforcement. Cues become reinforcers because of conditioning- they've been frequently paired with treats or other awesome things. If there aren't enough treats following the behavior to make it exciting, then the effect certainly won't carry over to the cue. This doesn't mean that the behavior needs to be followed by a treat every time- Kathy pointed out that intermittent reinforcement is stronger than continuous reinforcement, and she thinks this would apply to cues as well- but the odds do need to work in the dog's favor.

Anyway, those are just some of my musings about why cues may not act as reinforcers. Are they right? Maybe. Maybe not. I don't know- like I said, they're just ideas we kicked around. And now it's your turn! Poke holes in my theories. Point out the stuff I'm not taking into account. Let me know how your experiences stack up against my speculation. If you've seen it, share some research on the topic with me (I know some exists about “poisoned cues”). Or share your own brilliant ideas. Maybe you've got a theory of your own! I'd love to hear any and all of it!


Mary said...

Ken Ramirez mentioned some things in his sessions that could also affect the value of the cue. Let's say you train on an oriental rug (I've done this - bad trainer!) The dog does the behavior, I click and hand the treat to my dog, my dog drops the treat and now has to search the floor for it. Or I toss it on the floor and they have to search for it. Now I've added frustration - it will affect how the dog learns the behavior and is likely to effect how valuable the associated cue is as well.

Kind of brainstorm, but another idea that also stems out of Ken's session - what if you have more than one dog and you give the cue and inadvertently reward one dog more than the other? The cue could lose value to the dog who feels like they are not getting a fair amount of reinforcement.

And something that I did - not on purpose - but the results are the same anyway. I would cue the behavior, then ask for a tweak before I would click and reward. So, I cue "front", but dog comes in and isn't quite straight. So before I click/treat, I ask her to scootch in, then she gets the C/T. Do this too many times and I can attest that the cues will not just lose value, but to a sensitive dog, they can become downright punishing. The cue is an announcement to the dog - "hey, here is your chance to be not-quite-good-enough". That was a hard lesson learned.

Crystal said...

Ooh, thanks for commenting, Mary! I think my husband went to that session with Ken Ramirez. My husband mentioned something about how the trainer's relationship with the trainee can effect the the value of the reinforcement. I wish I had gone to that session...

In your toss-and-search example, it seems possible that for a dog who ENJOYS searching (maybe a tracking dog or nosework dog), that could actually increase the reinforcement value. Of course, that's really going to depend on the dog- I can absolutely see it being frustrating to some dogs.

I love your example of the cue coming to mean "here's your chance to be not-good-enough." I'm guilty of that too! (And, oddly enough, also with fronts!) I keep reminding myself not to name things that aren't correct, but it's SO HARD.

Thanks for sharing your ideas! Lots to think about here!

Laura, Lance, and Vito said...

I don't know if the luring argument holds much weight. I agree that when I lure the dog isn't thinking and so I suppose you could say the dog is coerced into doing it. But usually the cue isn't attached at the luring stage so it wouldn't have any association with the lure. And even if the cue was attached I can't see how blindly following a treat could ruin the ability for that cue to be a reinforcer. The only way I can picture that is if the dog would be nervous about doing the behavior unless lured.

I think the level of frustration while learning a behavior is important in the dog's perception of it. Lance has a love/hate relationship with backing up since I'm not good at teaching this trick; I always get a certain number of steps-stop-bark-1 more step-bark-etc. So Lance is continually frustrated and starts barking on that first step backwards as he doesn't understand how far he has to go in order to get rewarded.

OF course all shaped tricks have had a certain amount of frustration during it as shaping almost depends on it. We want the dog to be a tiny bit frustrated in order to get a little variation. Tiny is key though :)

Really it leaves me wondering if any cue for a behavior that the dog just doesn't naturally love can be used as a reinforcer. It seems like there are so many conditions that need to met in the learning process and such a high rate of reinforcement kept up after learning, plus no "corrections."

Crystal said...

Laura, it's a MUSING, not an argument! :) I'm not sure that luring could wreck a cue as a reinforcer, either, but it's an interesting idea.

But let's assume for a moment that I'm right: there are tons of conditions that must be met in order for a cue to be reinforcing. Let's also assume that cues can be reinforcing, and that it happens frequently enough that trainers have noticed the phenomenon and begun lecturing and writing about it. (That last point is a pretty safe assumption, yes?)

That tells me that:
1. There are a LOT of REALLY good/lucky trainers out there, and/or
2. Dogs are more resilient than we realize, and/or
3. Classical conditioning can be stronger than occasional mistakes or frustrations.

All I know is that Maisy really enjoys heeling. Even though I've got it on a variable reinforcement schedule (and one where she has to do a fair amount of work before she gets the treat), and even though I've made mistakes with it (that static electricity example didn't come from nowhere, after all), she gets INCREDIBLY excited when I cue her to heel. Is the cue reinforcing? I don't know... I think so, but I've never tested it to be sure.

Crystal said...

Oh, and so far as the high rate of reinforcement goes: Kathy did say at Clicker Expo that a cue can reinforce another cue. That's how you build behavior chains. So, maintaining a high rate of reinforcement doesn't need to be with treats (or even play). It can be with other cues. You might give 5 or 10 or whatever cues, reinforcing each one with another cue, and only giving one treat. I think you'd need to work up to that, but in theory, every cue WOULD still be getting a reinforcer, just not a primary reinforcer.

Dawn said...

Its all just too complex for my little brain. Too scientific! Great job on the series though.

Ninso said...

And while we're on the topic of weird training phenomema, how about this one (maybe somewhat related) . . . ever noticed that a cue (or maybe the marker that follows) can build value for the reward that follows? I see this once in awhile with Elo. Not particularly interested in whatever it is I have for him. Give him a cue and mark him for doing it, suddenly he is all over it.

Ci Da said...

My theory based on working with my person-oriented Aussie is that a cue is reinforcing because it signals the opportunity to work with me. Lately I've put a lot of time and effort into making myself more reinforcing to my dog. And that doesn't necessarily mean I supply more treats. I supply a lot of treats, but I ensure that I'm not giving them away for free -- the dog needs to supply me with average-or-better behaviour and I'm always steadily raising my criteria. I play loads of games with my dog that revolve around her working with and interacting with me. The end result is a dog who is jazzed just to work with me.

So when I offer a cue I can see her ears perk up and her attention shifts to me and away from her environment. Sometimes she starts trembling in excitement and anticipation.

Because I get that reaction a great deal of the time, I also notice when that's NOT her reaction -- when she finds her environment more reinforcing than me. For example, at agility class she can be finicky at the start line because the agility course is more reinforcing than me asking her to sit and wait.

Lately I've become very aware of the push and pull of various reinforcers, and as a result have become more adept at managing them. The result is a dog who looks to me more for reinforcement.

So I'm not sure why a cue isn't reinforcing to some dogs (I'm sure a number of the theories you mentioned could be factors) but I think ultimately it's a juggling act that is dependent on where the dog's current reinforcers are coming from.

Laura, Lance, and Vito said...

With Vito I totally see the cue as increasing the value of the reward! He's not the most food motivated dog but is more likely to take the food when asked to do a behavior. Of course there are still times where he'll work for the click but still won't take the food.

Sophie said...

I’m not sure about the luring suggestion, but there’s definitely a lot to think about here! I would argue that although luring does involve *some* element of coercion, it isn’t anywhere near as coercive as, say, physically manipulating the dog’s body into the position you want: my dog has the choice to follow the lure or not. (Incidentally, I’ve done so little work with Lola on luring—only sit and down and the newest trick we’re working on, spinning in a circle—that I actually had to teach her to follow a lure again when we started on our new trick.)

Most of Jess’ tricks were taught through luring, and she seems genuinely thrilled to be cued to give them. One of the videos on my blog, of Jess moving around on the stool, shows one of her favourite things to do – and that was taught through luring (getting her up onto the stool and helping to lean her body in the correct position and direction), mild positive punishment (for I suppose that is what it is – gently leaning/moving in toward her with my legs, so that she shuffled to the side) and shaping (clicking for more and more steps away from me as I moved). And yet she loves to do the behaviour; I would call that a reinforcing game for her.

I like to think I have a good handle on rate of reinforcement, though I do sometimes push a little too hard with Lola – she’s bright, and I sometimes expect more of her than she can give in shaping, and then have to drop back a few levels to get her back in the groove. (Though this could also be due to her having a young terrier brain.) I’m not sure how my timing is: I should record it sometime and see. I also know that I fail to give proper motivation sometimes (just clicking/treating rather than excitedly praising, etc), and that has an adverse effect on Lola’s performance; she seems to be just going through the motions after a while, rather than enjoying what she’s doing. I’d say that (i.e. how often she was being robot-like) would have an effect on how reinforcing the cue was for her.

I do think that for a cue to be reinforcing, the rate of reinforcement must still remain pretty high, even if it is put on a variable schedule – like over 70% of the time, the dog will be rewarded for the behaviour (with a treat, tug, praise, etc). I do try and reward my puppy every time (with a quick excited murmur of praise, or a rousing game of tug or a throw of the ball, or just a food reward) but I’ve also tested to see if she’ll perform a few behaviours for one treat, and she will.

I’m curious to think about what cues are reinforcing for her, though. What would you say is reinforcing for Maisy? Lola enjoys being held and being close to your face – having recently taught her to jump up into my arms, I’ve been using that sporadically as a reinforcer (alongside a more regular reward of some kind), and she does seem excited when she leaps up at me; so I’d say that was pretty rewarding (especially because she then gets a chance to lick my face: so is it the cue that is reinforcing, or the opportunities that the cue leads to?).

Crystal Thompson said...

WARNING: Marathon comment ahead!! :)

DAWN- I LOOOOVE the science behind dog training, even if I don't understand it all yet. I've actually considered a master's in behavior analysis... I'd love to do research some day.

NINSO AND LAURA- I have a stomach on legs, so I've never noticed the phenomenon of which you speak. It's FASCINATING though. I wonder why that is? (See, Dawn, this is the kind of stuff I want to know!!)

CI DA- Your comment about your dog being people oriented caught my attention. Maisy is pretty biddable, and she will do things for me because I ask, and not necessarily because of the cookie that follows. I often wonder how much of that is genetic and how much of that is conditioned/due to a low history of me being fun and having cookies.

SOPHIE- It sounds like between you and Laura, the luring theory isn't panning out. Here's another area where I'd love to do some research. Luring might be coercive in that it takes away some choices, but it seems unlikely that it's coercive enough to affect most dogs. I lured Maisy, Laura's done luring, you have... and we all have great dogs. :) Then again, we are all great trainers, and know how to fade the lure. Perhaps when the lures aren't faded well/at all it becomes more of an issue?

I'd also love to know more about levels of reinforcement. I'm sure there's science on how often a reinforcer needs to follow a behavior in order to maintain it. (And of course it likely depends on the animal as well.)

Things (behaviors which could be cued) Maisy finds reinforcing: praise, running (I use "go!" to reward recalls a lot), chasing, being in heel position. Probably shake/wave. Probably prairie dog/beg. Sometimes going in her crate (as long as the door doesn't shut).

Things she doesn't: stays, fronts, stand for exam (the touching part), rolling over, stepping onto stools or things.

andrea said...

Interesting thinking ... I've been working a bit with the concept of "ready" ..which is becoming a cue I suppose. Sally finds "ready" THRILLING and it's sort of a pre-cue I guess ;)

Luring is coercion though I'd never thought of it quite that way but so is all training really no matter how it's done. My gang are opportunists. They always have an eye open for training opportunities - they coerce me into play(whoops training) sessions and I do the same with them. :)

Sophie said...

Okay, so I was going to reply again (further in-depth) here, but thought it might end up being a tad too long. I've put up a blog post here - - that I'd be interested in hearing your thoughts on, Crystal!

Crystal Thompson said...

Sophie, I think you've written the post better than I did! That's a really great entry- well thought out, well organized, and well written. It's what I was trying to write here. I didn't quite make it. I think you did. :)

Anonymous said...

It is such a relief to hear that there is even a discussion about this concept anywhere!

I have a very independent, beautiful 4 year old hound and know painfully well (being in a dog trainer program that is quite firm about cues being naturally reinforcing if trained correctly) that cues don't work as well as reinforcers for my dog as clicks and treats do and that no amount of click and food/toy/etc. treat can compete wit her much stronger reinforcer of being free to sniff, roam and hunt.

Being relatively new to clicker training, I naturally believed for a while that my training mistakes were the only cause for her clear lack of enthusiasm to work for cues. But the more I weeded out the poisoned cues and focused on using only the ones trained as positively as I am capable of, the more I started wondering why my hound even should find cues themselves reinforcing.

Sure, she loves our clicker sessions... as long as she gets paid well and as long as they are entertaining and fun for her – but she clearly has many ideas of what could be way more rewarding than performing rather unexiting behaviors (since they rarely involve roaming and hunting). It is very interesting to observe how very slowly she responds to a cue as a reinforcer – clearly wondering why I forgot the click and treat.

So, now I wonder if it is possible for the more independent dogs (not just hounds) that the cued behavior itself needs to be fun/rewarding for the concept of “cues as reinforcers” to work well. So far, I have only found one behavior that might fit the bill (“pawing” anything vigorously) and I will keep searching for more to see if we can eventually put together a 10-part chain of behaviors that is reinforced not so much by cues or treats - but by the behaviors themselves.


Crystal Thompson said...

Beachcomber- I do believe that, in theory, positively trained cues SHOULD be reinforcing. Still... the dog gets to decide what's reinforcing, so I don't think that we as trainers should get bent out of shape if something that we think should work doesn't. Let the dog decide what he wants and go with it. :)

However, I do think it's worth the effort to cultivate new reinforcers, and I'd love to hear how your behavior chain works out.