Last fall’s Voice of the Scout results have been out for some time now. I have looked at it a few times but quickly grew frustrated with trying to figure out what it was saying or what it all meant. The documents I have read are cast in pretty tortured jargon and I don’t have a lot of patience with trying to interpret something that could be written simply.
I’ve always questioned how well this sort of marketing science applies to something like Scouting. Does it address the real issues? Is it an accurate gauge of anything? I actually think there’s a real danger of shifting the focus of our efforts into the wrong things but after learning something about how this works and reading the fall report I think that this may actually help identify and effect things that will improve the program.
After getting some help from Walter Underwood I think I can better understand the message and the methods behind the whole endeavor, so I’ll take a stab at interpreting it.
How it Works
A survey is conducted every six months. A notice is emailed to members divided into seven different audiences over a ten week period (Cub Scouts, Boy Scouts/Ventures, Cub Scout Parents, Boy Scout /Venture Parents, Youth Facing Volunteers, District/Council Volunteers, Chartered Organizations.) I do not know how these audiences are chosen.
Survey results focus on something called the Net Promoter Score (NPS). It’s worth understanding exactly what NPS is and how it is calculated.
NPS a score based on the likelihood that members would recommend Scouting to their friends and family. Respondents are asked to rate their likelihood of recommending Scouting to others on a scale of 0 to ten. Zero being not at all likely and ten being very likely.
“Promoters” are those responding with a 9 or 10. “Passively Satisfied” answer 7 or 8, “Detractors” answer 0-6.
The number of Detractors is subtracted from the number of Promoters resulting in the Net Promoter Score (NPS).
So what does the NPS score mean? Why does it matter? It turns out that NPS scoring is a common technique companies use to gauge customer loyalty and satisfaction, it’s actually a pretty big deal in the marketing world. But the numerical score is just a part of the story.
Closing the Loop
After people are asked to score their likelihood of recommending Scouting they are asked another open-ended question; “why?”. The comments ‘close the loop’ they give direct, actionable, feedback that can then be analyzed to identify key issues that effect the way people are perceiving the organization.
This simple question is actually one we ask all the time; “How are we doing?”. If we listen to the replies carefully we may learn something and what we learn can lead us to improve what we are doing.
What Did We learn?
In the fall of 2012 the Voice of the Scout collected around 68K replies and the executive summary of the results (here’s the full document as a PDF) reported two key findings (called Hot Topics in the summary):
Hot Topic #1: frustration with the delivery of the program at the unit level.
- Cub Scout Parents described meetings as too boring, not organized and not fun.
- All audiences cited leadership and leadership support was in need of improvement
- Boy Scout Parents commented that they thought the opportunity to be a leader was not what they think it ought to be and questioned that meetings are a good use of their son’s time.
Hot Topic #2: Specific reference to the membership standards policy increased in all audiences except the Cub Scouts (youth members, not parents).
- Conservatively for every comment in support of the policy there were 15 comments that did not support it.
- A significant number of comments are suggesting that this issue has negatively impacted their perception and/or relationship with Scouting.
The executive summary analyzes these two ‘hot topics’ further, and it’s worth reading through.
What Happens Next?
At some point the Voice of the Scout will be available at a unit level (so far as I can tell), so we’ll be able to look at a much more localized analysis and hopefully this will help us improve what we are doing.
On the national, council and regional level I am sure that folks are looking at all of this and figuring out what to do next. It’s early days yet, so we’ll see what we see.
Will we learn anything new? Perhaps and perhaps not. Most of the difficulties I hear about are nothing new, just reiterations of ones that have been around for decades. Hot Topic #1 are two oft-repeated complaints – boring meetings and inadequate application of youth leadership. We do know how to fix these things and hopefully we’ll get at it!
For What it’s Worth
Two things prevent the Voice of the Scout from being readily useful to us ‘youth facing volunteers’. First is the results are presented in tortured, jargon-laden language that assumes the reader knows the NPS system. I was able to briefly explain the method in a couple of hundred words – how did I do? While I understand that this is a complex process the tendency to over-explain things in the documents I have read is not making them clearer.
The second problem is accessing the information. It is a real challenge to find things at Scouting.org, and I have all but given up on myscouting.org – it is not what I would call an intuitively designed interface.
I applaud the effort being made to communicate with us all online, and there are valuable resources available but trying to get answers, or trying to find nearly anything is a real drag. Maintaining this blog has shown me that making information and resources available is not as easy as it looks, so I do understand the challenge. It would be nice, though, if things were easier to find and easier to understand once you found them.
About the Author
On one hand, it’s great to see the BSA reaching out to the rank & file to find out the perception of the program at the grassroots level. However, as you point out, there is so much missing from the methodology and reporting that these early results cannot be meaningfully compared with later results after the needed corrections are made to the feedback program. I recall the conversation you had with Hab Butler a while back about Journey to Excellence and have been looking forward to seeing the results of Voice of the Scout, but now I realize it’s a work in progress.
It’s interesting to see the summary comment “Boy Scout Parents commented that they thought the opportunity to be a leader was not what they think it ought to be and questioned that meetings are a good use of their son’s time.” I’d like to see some deeper digging into just exactly what these parents think the leadership experience should be and what they think should be happening at troop meetings. This would help answer whether the reason for their comments is (1) their perception of youth leadership doesn’t agree with the BSA’s program, or (2) most units are indeed not truly youth led, by youth leaders trained to do so (by trained adults, I might add). I’m sure it’s a combination of both, which suggests some starting points for us as unit leaders. Parents may also need to be re-educated to expect chaos and disorganization as elements of boys being in charge.
The 15:1 ratio on the membership policy also indicates that nearly all of us take a “don’t ask, don’t tell” approach. It looks very likely that the tail will wag the dog on this one.
I too was impressed with the 15:1 ratio of people who were dissatisfied with the current discriminatory membership policy. This gives me hope that the leadership will change this terrible policy. In my troop, the majority of adults do not support this policy. They do not support such discriminatory policies in their businesses or in their personal lives, so they see no logic in belonging to an organization that supports such a policy. I have worked hard to try and keep people in the troop by using the optimistic hope that this policy will soon change. Even more telling is that the Scouts do not support this policy at all. My SPL wanted to adopt a PLC policy for the troop that our troop would not discriminate against anyone based on sexual orientation as a troop policy. I told him it was his right to have such a discussion and it was the right of the scouts to make such a statement of non-discrimination to the troop committee. They are still deciding what they will do. It is really encouraging and inspiring that the scouts think this way. What behavior are we modeling with such a policy? The answer is that by having such a policy in place, BSA is teaching/modeling discrimination to scouts, and that is not right. To have BSA still stuck in this bad position is the greatest threat to the future of the organization in my view. This policy of discrimination against people based on who they are is the biggest complaint I have about Scouting leadership. This is a problem with the current scout leadership, not with scouting. Let’s not destroy this wonderful organization because of the outdated beliefs of the 1400 people who set such policies for BSA national.
A number of religious, and therefore ethical, points of view in have always been present in Scouting. While you may not agree with some of them I don’t think it is fair to characterize the 1400 people who will be voting on this as having ‘outdated beliefs’.
Our challenge, no matter which side of this question we are on, is finding things that we have in common and starting from there.
I would caution against reading too much into the findings in this report;
1) The methodology used introduces considerable bias – the survey was only conducted by email which weights results more heavily to the opinions of women and college graduates among others. In addition, the quality of the email list varies considerably by council and a multitude of other factors.
2) The companies given as benchmarks are not appropriate – the Fall 2011 Executive Briefing makes it clear that there are no benchmarks for this tool in the non-profit sector and that BSA is “on the cutting edge with the adoption of” this approach. Comparing BSA numbers with those of Costco is ridiculous at best given the differences in the emotional and intellectual complexities of the relationships.
3) The report is designed primarily to sell the services of the survey company and their packaged methodology – the more complex the relationship, the harder it is to map the factors that influence whether or not the organization is meeting its goals. In this case, the research company wants to persuade the reader that its trademarked methodology is a simple way to measure success and as a result, many of the weaknesses in the methodology are minimized.
4) I would expect a much higher ratio of negative to positive comments on any controversial and emotional topic, especially when the research is conducted shortly after a decision is made that creates a winner and a loser in the argument. There is much less emotional incentive for positive comments while those who feel they have lost get some satisfaction from venting.
While I would not say the report is without all value, I do believe it has caused more problems than it has solved. One of the biggest worries I have is that the top management of BSA is using this research to drive their strategic decision making process.
A note on my background: Professionally, I research and advise organizations on demographic and societal dynamics with an emphasis on adolescent development, interests and career choice. I hold an MBA in marketing and a MA in Secondary Education – Teaching and Learning with 25 years of experience in these fields. I am a third generation Scoutmaster with one Eagle son and one working towards that rank.
I appreciate your thoughts on the subject – the means does define the sample. I haven’t any real idea of how or if the results are driving decisions. I did note that a more comprehensive review of the membership policy change is underway (http://blog.scoutingmagazine.org/2013/02/26/the-boy-scouts-of-americas-family-discussion-on-our-membership-policy/).
For the reasons you state above I am not, nor have I ever been a big fan of marketing driven analysis with Scouting or with any of the other non-profits I have been involved with. I think the best thing you can do with survey tools like this is make them a piece of the puzzle – they are only indicators.
This year we are suppose to get a survey for each hat we wear that is being surveyed. For example: “Parent,” “Direct contact Leader”, “District level Scouter”
They don’t survey all positions. Surveys are going out to: Scouts over 14 (they will only receive program related questions, like how do they like their meetings, etc, not policy questions,), Parents of Scouts, Youth-facing Volunteers, District/Council Volunteers and Charter Organizations.
NOTE: If your Internet Service Provider blocks the BSA email you will not receive the survey. Currently the survey cannot be sent to AOL users because of a restriction on the AOL network.
I was one of the people surveyed this time and I found the survey confusing. I am registered as a District volunteer, Cubmaster, and an Assistant Scoutmaster. When I took the survey, I couldn’t tell if I was being surveyed as a ‘youth facing’ volunteer. If there is going to be a distinction in the results, I think they should have let people know in the beginning which position they are being surveyed about.
The other scary score is the NPS of 28% for Cub Scout parents. 28% is not good — that is similar to a second-tier cell phone carrier. Yuk. Cub Scouts are half of membership, so that is huge. Most of our Boy Scouts come from Cub Scouts, so that means that we’re losing new recruits at the beginning of the pipeline and they they are more likely to not move on to Boy Scouts.
An NPS of 32% for youth-facing volunteers is also very bad. These are people who spend their “one hour per week” on Scouting and a lot of them won’t recommend it to others. Uh-oh.
We are on a pretty sharp curve downward nationally, I know that locally our recruiting get’s thinner and thinner each year and I know that we are drawing more and more from ‘legacy’ families, which is not something we create proportionally. The VOS results indicate a pretty solid hit last year with the policy reaffirmation, what we don’t see in that report is the effect of the abuse scandals on our non-member customer base. I think that the impact of these two things are still reverberating and will only get worse.
I’d like to see a VOS type survey of non-members; if the majority of our membership is only passively satisfied or actually detractors the opinion outside of that group is predictably even lower.
I think we need something like a complete rebuild, or at least hit the reset button on the priorities for action; training that has actual affect on program, dislodging the ‘old guard’ that holds innovation and change back, and moving ahead with all speed. I don’t think anything like this can be done on a national scale, I think it has to happen at the local level.
The Cub Scout data was interesting to me since I am a Cubmaster. I made a decision at the first Tiger meeting my son and I attended three years ago. Either I get involved and make this fun and interesting or quit.
I went to Woodbadge, read, attended roundtables and other pack meetings. Without all of these things (which requires a lot of time, motivation and dedication), I really do not think our pack would be around.
My of the volunteer leaders in the pack work long hours, cannot participate in all the training I have taken. Without the Pack And Den Meeting Resource Book, not too much would be accomplished.
I know things are better now, but I would love to see the results for my pack.
Another explanation I find helpful is “NPS scores tell how likely customers are to give you a positive referral. Will folks take the risk of recommending you to friends?” To see if numbers = reality, visit some of the recent NPS scores for businesses such as http://ctsmithiii.wordpress.com/2011/09/30/top-10-u-s-net-promoter-scores/
Here is the statistics that struck me:
1. Conservative estimates assign a 15:1 ratio of negative to positive comments
about the existing membership standards policy
2. Boy Scout/Venture parents had a “NPS drop of -16 percent” – some of which can be attributed to questions about BSA values
Disregarding the double negative, I think we’ve found our source of National revisiting their membership policies this year.
I tend to agree with you – it’s a pretty dramatic set of data.
Completely shocking to me. 15:1? Wow
Those were the two things that struck me as well. 15:1 blows my mind.
I participated in this survey – I can’t recall now if I responded as a scout parent or as a Pack Committee member – and included a comment regarding my position on the membership policy (I am in favor of a policy change to allow for inclusion of scouts and adults who are openly homosexual). This is the first time I have seen the results, and I am really impressed by that 15:1 ratio. This gives me some hope for BSA, which I sorely needed with one son crossing over from cubs to boy scouts and another entering the program this year. Until now, I assumed that the reason for BSA re-evaluating the membership policy this year was purely due to the loss of financial support but this suggests that the National Council may also be considering its current membership’s interests.
Great site, CG – I will surely return.
And that 15:1 is among people already in the program. It doesn’t include people who have already rejected Scouting because of this issue.
Now, I agree with Ron Morris above that this survey is probably not as watertight as it should be, but 15:1 among your current “customers” opposed to something is going to be outside the margin of error for anything except an intentionally skewed survey.