Tuesday, January 17, 2012

The Pleasure of Anticipation

Last spring, I wrote about how cues can be reinforcing for dogs. If the cue predicts a good outcome (a click and treat, for example), then the dog will find the cue exciting. More talented trainers than I have taken advantage of that by reinforcing a dog’s response with another cue.

Some readers met this with skepticism. Maybe my explanations made sense, maybe they didn’t, but let’s be honest: logic and anecdotes alone are not always convincing. That’s fine; I don’t expect everyone to agree with me, and in fact, I would find that rather boring. But when one of those skeptics found this hour-long lecture, she remembered my post and emailed me.

The lecture, given by neurobiologist Robert Sapolsky, explored what makes humans unique. His entire talk is fabulous, and I urge you to watch the entire thing. Personally, I really enjoyed his discussion of how language affects our perceptions of others because of the insular cortex, but what’s relevant today is what he shares about dopamine (starts about 30 minutes in).

Throw it... throooow iiiiiitttttttt.....
Dopamine is a neurotransmitter that helps control the brain’s pleasure and reward centers. For many years, it was believed that when someone (human or animal, it doesn’t matter- dopamine is present in all mammalian brains) received a reward, their brain would release dopamine. In turn, this would result in a pleasurable feeling.

However, when scientists actually studied what was going on, they found something very different. Sapolsky described an experiment in which chimps could receive a food reward if they press a lever when a light turns on. The dopamine levels in the chimps’ brains increased not when they completed the task, but rather when the light went on.

In other words, what the chimps found pleasurable was the opportunity to receive a reward, not the reward itself. After they pressed the level, their brain quit releasing dopamine, even before they received the reward. Anticipating the reward was better than the reward itself.

That light signified an opportunity to receive a reward; press the lever now, it said, and you will be reinforced. This is exactly what we do in dog training. I say “sit,” and if my dog does, she’ll get a treat. So the light was acting as a cue. The study Sapolsky cited says that it was the cue that made dopamine levels rise, which means that my dog will feel good when I say “sit,” not when I give her the treat. The cue is reinforcing.

I suspect that clickers work the same way, although Sapolsky didn’t address that directly. He did, however, say that dopamine is about the anticipation of the reward, not the reward itself. If cues can cause that anticipation, it seems that a sound could, too. Can a click cause dopamine levels to increase because the dog is now expecting to receive his reward? I don’t see why not.

What scientists found even more remarkable, however, was that when the food was given in response to the correct behavior only half the time, the chimps’ dopamine levels went through the roof. This wasn’t exactly surprising to me; dog trainers often talk about how a variable schedule of reinforcement creates stronger, more durable behaviors than when the dog gets a treat for every correct behavior. B.F. Skinner and his students proved that over and over again in the lab, although of course they couldn’t know that it was the result of dopamine. As Sapolsky put it, “maybe is addictive like nothing else.”

Finally, the scientists also found that if they blocked dopamine production in the chimps’ brains, when the light came on, the chimps didn’t care. Instead of eagerly pressing the lever, they sort of shrugged it off. The chimps knew they’d get a reward if they did, but they just didn’t seem to care. Could this be a possible explanation for why a dog doesn’t respond to a cue? Maybe. But I'd point out that there are many, many other reasons dogs don’t perform a behavior, and most of them are probably more logical. Still, it is fun to think about.

I found all of this really interesting. Not only did it support the concept of cues being reinforcing- something I find pretty fascinating in and of itself- but it also suggests that there is more at play in clicker training than just the food. In fact, it would seem that anticipation is what's truly powerful, an idea I find amusing since trainers often get upset when their dogs anticipate what's coming next.

To be fair, having the dog act before we ask them to can be a problem. Still, is that indicative of a corresponding spike in dopamine? And if so... how can we use this to our advantage? What can we do to harness our dog's natural brain chemistry to create a more favorable training outcome? I'll admit, I don't have an answer here, so I turn it over to you: have you ever used the power of anticipation to your advantage? And if so, how?


Laura and The Corgi, Toller, & Duck said...

Well the first thing that came to my mind was the "are you ready" routine that I sometimes do with the dogs before releasing them for a go out, articles, etc. It clearly builds the anticipation of the dog and if I really play on it I can get Vito screaming over the opportunity to do anything! In a smaller way the way I raise my eyebrows when waiting for the judge to let me send the corgi for articles or gloves really excites Lance.

Ninso said...

So excited you wrote about this! I think it is so fascinating!

-A skeptic ;-)

Tegan said...

This is so so so interesting. Thanks so much for posting about it.

I remember hearing a story about someone who had used 'yes' as a reward marker. This individual got himself into trouble on a farm, getting trapped under machinery, and with his mobile phone out of reach. He used 'yes' to shape the dog to the phone, pick it up, and bring it to him. That is, the dog had such a strong reinforcement history with 'yes', that it chose to continue this shaping game without actually having any reward.

Possibly, the marker itself reinforced the behaviour.

Different, but still interesting. :)

Crystal (Thompson) Barrera said...

Laura- I've been doing the "are you ready" thing too. It seems to get Maisy excited for whatever comes next. I love that you use your EYEBROWS.

Ninso- I hope you don't mind me calling you a skeptic. :) I love the way you question things, and the way you find science to support/deny claims.

Tegan- That is the coolest story EVER.

Sarah said...

Sapolsky is one of my favorite scientists. I saw him speak when I was in college and was blown away by his ability to relay complex ideas in light, humorous ways. Definitely check out his books if you enjoyed that video. I learned a lot about human behavior from them but I never thought about applying them to dog things.

Crystal (Thompson) Barrera said...

So many books, so little time. I actually have Sapolsky's book on stress (Why Zebras Don't Get Ulcers) but haven't read it yet. I think it's getting bumped up.

Ninso said...

Not at all. I am a self-admitted skeptic.

Crystal (Thompson) Barrera said...

I like that about you, Ninso. I'm also a skeptic about many things, although I'll admit that I'll believe pretty much anything Kathy Sdao tells me. I love her. :)

Kerry M. said...

Great link. Thanks for sharing. Also, this has been the most easily understood explanation of how a cue can be a reinforcer. Usually I get buried in the language of primary, secondary, tertiary, cue, and antecedents. While I have grasped it, it's definitely been more of a grasp than a secure hold. Just the common sense language of the anticipation is a reward, and here's the dopamine levels to prove it, is great. Makes sense and I can finally explain it to others.

Nadja Zlender said...

Hi Crystal,
I've found your post about "The pleasure of anticipation" while I was surfing the internet to find information about clicker training - in specific the removal of the reward after the click.
Yesterday I was listening to a brilliant lecture by Simon Gadbois as a part of SPARCS conference (http://caninescience.info/) and he was, like your article, talking about dopamine levels and anticipation as a very important part of learning process in dogs. He also mentioned that trainers/dog handlers should, after a time, stop rewarding the dog after a click. Now this is something I haven't heard before and blows my mind because everyone everywhere says that there should always be a reward after the click.
Have you perhaps found more information regarding this topic? I am very interested in researching this in more details

Crystal (Thompson) Barrera said...

Hi Nadja,

I didn't see any of SPARCS- but sounds like it was interesting! I haven't heard about Gadbois' assertion not to reward after the click... did he explain why?

Unknown said...

I stumbled upon this blog by coincidence looking for a quote of somebody quoting me. Regardless, the idea of not always following the secondary reinforcer ("clicker") by the primary is a the root of the anticipatory system in the brain. As demonstrated by Olds & Milner, Berridge (his wanting system), Depue, Panksepp (his SEEKING system), etc., the whole point of the efficacy of a secondary reinforcer in training is to activate that anticipation system. As we have known in neuroscience for a long time, it is not the reward that matters, but the anticipation of the reward.
See Gadbois & Reeve, 2014 [chapter 1 in Horowitz 2014] for a summary).
Simon Gadbois, Dalhousie University