Friday, August 26, 2011

Ken Ramirez Seminar: Non-Food Reinforcers

One of the big objections people have to clicker training is “all that food.” They always want to know when they can stop using it, an attitude that used to baffle me. I mean, I get that people who are active in dog sports need their dog to perform many behaviors for a single treat, but when there are no rules, what's the problem? It's not hard to stick a handful of kibble in your pocket, after all.

Well. Leave it to Ken to not only be entertaining, but to also convince me that non-food reinforcers are both valuable and necessary (mostly because it is much easier to perform husbandry behaviors on a sick animal who is refusing to eat when you have a non-food reinforcer available). He also presented a very thorough method for creating non-food reinforcers, and gave us some tips on how we should and shouldn't use them once they've been established.

Let's start at the beginning: what are non-food reinforcers? Well, obviously, they're not food, but Ken was a bit more scientific than that. When Ken talked about reinforcers, he broke them down into two categories: primary reinforcers and secondary reinforcers. A primary reinforcer is something that is inherently reinforcing; the animal doesn't need to have any experience with it to understand that it is a good thing. Typically, these reinforcers satisfy biological needs, and food is the ultimate primary reinforcer (that's why it is so useful in training). By contrast, secondary reinforcers are something the animal needs to learn is desirable.

Despite the fact that secondary reinforcers are learned, Ken made the point that secondary reinforcers can be very, very powerful. In fact, they can sometimes be more powerful than primaries because of what they represent. For example, money is a secondary reinforcer- the paper itself has no inherent value. However, society teaches us that money is a desirable thing because of what it can buy, and this association is so strong that humans will do some very boring or unpleasant tasks in order to obtain it. In fact, we are more likely to take a job that pays money than one that provides food and shelter.

(As a side note, play can be looked at as both a primary and a secondary reinforcer. Often the act of playing- running or chasing, for example- is innate, making it a primary reinforcer. However, the objects used in play, like balls or tug toys, are secondaries because the dog needs to learn what they are used for. A ball that is not thrown is neither interesting nor reinforcing to most dogs.)

Another way to think of secondary reinforcers is as a “reinforcement substitute,” which emphasizes the fact that secondary reinforcers only become powerful through conditioning. Ken is very, very systematic in the way that he creates secondary reinforcers. His approach is so thorough and slow, in fact, that I suspect some readers will be turned off by it. This is partly because he's found that the more time you spend conditioning them, the more powerful they will be, but also because he believes that if you use secondary reinforcers improperly, it can lead to a lot of frustration. Since frustration is sometimes inherent in training, he tries to minimize it whenever possible, something which is both kind to the animal and practical when working with wild animals who are less tolerant of human mistakes than the domesticated dog.

Ken creates secondary reinforcers in almost exactly the same way he trains a behavior. He starts by choosing a stimulus to act as a reinforcer. This stimulus should be one that is useful- that is, it is easily accessible, and not overly cumbersome to implement. He also thinks it works well to use something that is novel to the animal; choosing something the animal has habituated to and now ignores is going to make things much more difficult. One of his favorite secondary reinforcers is clapping.

Then he does straight-up classical conditioning: he presents the chosen stimulus, and then immediately follows it with a primary reinforcer. So, he claps, and hands over a bit of food. Clap, food, clap, food, until the animal seems to understand that the clapping predicts the food. This shouldn't take long at all unless the animal finds the stimulus aversive (if the animal is sound-sensitive, for example), or if your primary isn't that exciting.

Then he asks the animal for an easy, well-established behavior. This is something the animal already knows well, and has a very strong reinforcement history for. In dogs, the behavior of sit is often a good choice. When the animal does the behavior, the trainer will present the new stimulus, and then give a primary. For example, the dog sits, the trainer claps, and then he gives a treat. Ken will do this daily for several weeks, although the length of time will vary based on the animal, his relationship to the trainer, and his past reinforcement history.

The next step is to cue the same easy, well-established behavior, and then reward with only the new stimulus. Here, it is truly acting like a reinforcement substitute, as the dog will sit and receive only clapping as his reward. Ken will do this a maximum of three times during a training session, and he'll spread it out so that the animal is also getting primary reinforcers for other correct responses in between. Again, he'll stay at this step for several weeks.

This cycle repeats, except now Ken will cue a harder behavior, though it should still be well-established. When the animal responds, he'll give the new stimulus, and follow it by a primary. So, he'll cue, for example, a roll over or a stay, clap, and then give a treat. He stays at this level for several weeks before cuing the harder behavior and using the stimulus as a reinforcement substitute. Again, he continues doing this for several weeks.

Once this process has been completed, you're ready to use your new reinforcement substitute in training... but Ken has a few rules before you do. The most important is the 80/20 rule, which is actually more of a guideline, but basically, he says that you should use primaries approximately 80% of the time, and secondaries approximately 20% of the time. He never uses the same secondary reinforcer twice in row, although he might use two separate secondaries in a row. He always treats the new secondary as a behavior and occasionally “recharges” it so that it retains its strength. Finally, he recommended that novice trainers use secondaries only to maintain existing behaviors, and not to create new ones.

Yes, this is a very regimented way of moving away from food reinforcers, but Ken has a very convincing story to support the importance of being so systematic. It involves a new trainer trying to use tongue scratches to reward a killer whale, with a very poor result. I won't spoil the story for you (and Ken tells it so much better than me anyway), but trust me when I say that I completely understand why Ken is so thorough. His advice is that we never take any reinforcer for granted, and to work to build up as many reinforcers, both primary and secondary, as possible.

If you've found any of this even remotely interesting, you totally need to see Ken speak about it. Imagine this information, only peppered with incredibly funny and informative stories about dolphins, seals, penguins, and yes, even some dogs. He's also got illustrative videos, and I always enjoy seeing familiar concepts used with exotic creatures.

What about you guys? Has anyone used such an in-depth process to create secondaries? Do you see value in doing it with your dogs? What types of non-food reinforcers do you use... or would you like to start using? I'd love to hear your thoughts!


Shenna Lemche AKA Project Leader said...

You're right, I was a bit 'ugh...this would take forever, just to condition a clap?' but I thought about it and I did go through that same process coincidentally with the word "yes." And I would say its 100% worth it to have that handy reinforcer for those times you need to mark a behavior but find yourself without a treat/toy.

What a great recap, by the way, looking forward to more.

Kirsten (peacefuldog) said...

What great ideas! And thank you for all the great explanations.

I agree with you about using food myself, feeling all those wet mouths on my hand and trying new treats. But I know a lot of people can't be bothered! And there are those dogs who are not food motivated.

The clapping is a great idea! Dogs do seem to love it. I use a clicker always in my classes, and sometimes with my own dogs, and I have found that dogs are way more motivated to learn when using a clicker as a marker than with just a food reward--as you explain so well, they seem to really enjoy that striking, novel sound and it makes the whole process more exciting for them than working for food alone.

Great post!

Crystal (Thompson) Barrera said...

Thanks, guys!

Shenna, I agree- it's a cumbersome undertaking, doing it the way Ken suggested. I can see how it would really create a strong reinforcer, though.

Kirsten- I do wonder if people who can't be bothered to use food would be willing to do this process, though...

Kristen said...

I've heard multiple variations of this talk from Ken. It's always worth hearing again!

18 months ago a friend had Griffin in the ClickerExpo lab that corresponded with this talk... she taught him about

8+ months later, someone clapped in excitement during a training session...and he ran over, completely expecting a treat. I was really puzzled until much later when I realized what was going on.

With a strong reinforcer...these can be created quite quickly, though maintaining is definitely another challenge.

Crystal (Thompson) Barrera said...

HA! I love that story, Kristen!!

Jen said...

Good topic! I've read Patricia McConnell a little on primary and secondary reinforcers, and it's always a good thing to think about. Motivators, as well, because while in a perfect world our dogs will do what we say 'cause we want them to, they are mercenary creatures and need to know what's in it for them!

I'm in fact constantly perplexed by peoples' objection to food rewards; I know I work pretty well for food rewards! I've also occasionally seen people in the street use punitive methods on their obviously food-motivated dogs (an overweight pug wearing a harness, is my most recent example).

Ninso said...

I'm not sure I understand the point. What is the difference between the clap, as he's using it, and a click/marker? And why would you need more than one secondary reinforcer?

What is the point of conditioning the secondary reinforcer if all you're doing is creating the expectation of the primary reinforcer? If you can only use it once in awhile un-paired with a primary reinforcer and if it is only to be used for an established behavior, why offer any reinforcement at all? I know I don't reinforce my dogs for "sit" every time I ask them to do it.

andrea said...

Ninso - I think the idea is the clap can stand alone as a reinforcer - the click is usually always paired with the primary reinforcer ...

My husband uses the word "nice" as a reinforcer and the dogs turn inside out for him ...I use toys and tuggng with Sally and sometimes a bridge phrase .. which to be honest usually winds up with a primary reinforcer afterwards "EEEXXXXXCEEELLLLENTTT" draws out well ...
I haven't put the time into the process but I think I may now with the word Good -which is ago to expression around here anyhow:)

Crystal (Thompson) Barrera said...

Ninso- I've tried to write this comment three times, and I'm clearly failing at finding a good way to explain what he said.

Basically, like Andrea said, the clap does eventually become a stand-alone reinforcer... he actually called it a "reinforcement substitute" because it takes the place of a primary reinforcer sometimes.

The 80/20 rule(suggestion) is to make sure that the clapping remains reinforcing/exciting, and to prevent people from overusing it, especially in the earlier stages. (It's absolutely not based on research- he was clear about that.) Ken wants and needs to be able to use non-food at times, especially when an animal is sick and refusing to eat. I could see using it in the performance ring, too- the 80 part in training and the 20 part in the ring.

As for the "established behaviors only," Ken said that guideline is for NOVICE trainers, not experienced ones.

You're right that it's a lot like a click (I mean, this is pretty much how we "charge" the clicker after all), but the clapping is never used as a marker (ie, repeat that behavior), just as a reinforcer. So Ken would click and clap as the reward.

Ken is also huge on wanting to have as many reinforcers as possible. I do wonder if this isn't because he's working with undomesticated animals- I mean, dogs have been selectively bred for thousands of years to work with humans, so I think that for some of them, interactions with people can be reinforcing. Far more than you might see with a penguin or something, I bet.

Ninso said...

I don't know. I think the line between marker and secondary reinforcer is kind of blurry. While a click usually IS paired with a treat, it doesn't have to be, and after awhile, for many dogs, the click itself is reinforcing. I know it is for my dogs. And Laura's Vito who often works for the click and spits out treats is an extreme example of that. I also use a verbal marker of "yes" which I pair with treats as well as other types of rewards, but I know the yes itself is reinforcing. I doubt it would continue to be if I stopped rewarding after it altogether, but neither would these conditioned secondary reinforcers.

And Crystal I was kind of thinking along the lines of the end of your comment, that this might be more useful for exotic animal trainers. I think most dog trainers have tons of conditioned reinforcers already without even trying. My dogs respond to praise, smiles, clapping, etc and I know these responses are because the stimuli are paired with all sorts of good things in their life.

Ashley said...

I went over to Ken's website to see where he's having seminars next, and found this!

Why haven't I heard of this yet!

Crystal (Thompson) Barrera said...

Ninso, I agree on both counts. It IS a blurry line between secondary and marker- I think it really depends on how you use them and what you've taught your animal about them.

And, I agree that the systematic method here is probably far more important for exotic animal trainers. Heaven knows I've accidentally trained tons of odd secondaries, and Maisy really does love praise for itself... I can even use praise to interrupt/prevent a reactive response, or to reward a LAT.

I still think it's a good idea to have plenty of non-food reinforcers AND to build up as many as possible. It just seems smart to have as many reinforcers available as you can.

Crystal (Thompson) Barrera said...

Ashley, you should go! My husband and I are going to the day Patricia is speaking (Ken's talk is basically the same as we saw in Chicago, so we won't do both days).

lorac said...

I agree with Ninso... the way you describe how Ken pairs the primary and secondary is essentially the same as a c/t. I would think that if you want to develop a secondary reinforcer, then you don't necessarily always want to give the 2o before the 1o.

I pair secondary reinforcers all the time, but not the way that Ken does it. My dog loves a back scratch, so when rewarding with food, I scratch her back. Agree with Ken that it's classical conditioning, but my secondary doesn't PREDICT the primary, which is how he's doing it. Also, I think it helps to have secondaries that the dog (or bird or dolphin) already likes, not make up a secondary, such as clapping.

Mary said...

Replying way late with this, but, what the heck.

For those who are confused about the difference between the click and the clap - there isn't one. Both are secondary reinforcers. Both started with no value and gained value through their association with the primary reinforcer.

The difference between the two is how they are used. You could swap the two and you'd be okay (with some caveats).

What I mean to say is that in the way that Ken uses the clap, you could also use the clicker (and likewise, a conditioned clap could be used as a marker).

The clicker, properly conditioned, has reinforcing value to our dogs. And it would remain reinforcing for some time without the pairing of a primary. But, over time, if the pairing of the primary with the clicker was not maintained, the clicker would lose value and become less effective.

As trainers, we have learned, and as a result, choose, to always pair the clicker with a primary reinforcer because this makes our clicker a special and very useful tool for teaching behaviors (but that doesn't change the fact that all it really is is a secondary reinforcer).

We could use the clap in the same way as the clicker. Pair it every time and use it as a marker. And pair it with a primary all the time so the high value never drops off.

What Ken is showing with the clap, though, is how a secondary can also be used as a substitute for a primary. (Another way to say this which might make more sense is to think of this as another way to use your clicker. The key to help understand this may be to recognize that you would not use a conditioned reinforcer as both a marker and a substitute reinforcer at the same time. So use your clicker as a marker or a substitute, but it won't work as both. Likewise with the clap.)