Assessment Instruments

[PWylie]

Wanted to know about people's experiences using different employee engagement surveys. I've worked at an organization that used an inhouse survey, which ended up being quite lengthy and lacked an external benchmark capability. We are currently using Gallup's Q12 engagement survey, which does provide a benchmark, but I question it's applicability at our organization.
Is it better to design your own assessment survey or use an external companies?
How important is benchmarking your organization versus others?
Does anyone have any thoughts?

[Debbie Payne]

Well, I have a few opinions about employee surveys. One is how "engaging" is the actual process. Are people enthused, excited by doing the survey or are they confused or think it is more thing they have to squeeze into their already long list of tasks? Ive used Hewitt survey and have colleagues that use Q12 and others. Participating in a standard survey provides external recognition opportunities. If the organization and its employees really value this external recognition and it engages them to participate then this kind of survey works well. If on the other hand the organization feels somewhat "unique", is not intersted in benchmarking but in engagement, then creating their own works better. Then it is about the definition of engagement in the organization. Senior management should define what engagement is in thecontext of their own culture and organization. Then, find a way to ask their own questions- be it a survey, focus groups, work teams etc. and find ways to increase engagement both through the process and the followup recommendations. I always try to remember that the survey itself is an intervention and this guides the decision making.

Debbie Payne

[Rosa Say]

Aloha PWylie,

I'll be very transparent about this from the get-go: I hate employee surveys that are anonymous, and I strongly advise all my clients to work on why they think they need one to begin with. If your survey is anonymous your people don't feel comfortable telling you what they think in person, and you best work on that first before your survey adds fuel to the fire (at which most managers feel they get burned at the stake).

So my feeling on employee surveys extends somewhat to how we use assessment tools:

I'm a fan of Gallup's Q12 engagement survey because of the way it connects directly to strengths engagement, but whenever we use it we ask people to put their names on it, letting them know that we will then sit down with them and their manager together for a coaching session thereafter, in which we'll refer both to the individual ratings they have made along with the aggregate of the department. (Thus having the internal benchmark, and the ones that Gallup provides).

What is good about Gallup is the language they provide you with, not to mention all the polling and statistical back-up that they are historically founded in. Their language serves to be a great starter tool for managers trying to figure out how they can actually pull off the "soft stuff" of managing people by optimizing their strengths (and thus their engagement) and minimizing their weaknesses or making them irrelevant (quite the downer to one's spirit and a huge step backwards with engagement.)

I'm curious, why don't you feel the Q12 will be applicable for your organization?
Rosa

[PWylie]

Hi Rosa,
Thanks for your comments- I especially like the idea of making it non anonymous. I could see how that could have some dangers, it could also be great if the environment was willing.
What I worry about with the q12 is that the questions are not necessarily actionable- if a person doesn't have a best friend at work, and you have a social committee and opportunities for this to happen- what should you do about it? Yes, I understand how the question is related to engagement- in much the same way as having the sun shine on me each day would probably make me feel more engaged, but in some ways it is just as far out of my control.
I also think the connections that they survey makes between each question and the area of focus (i.e. questions 3-6 are related to management support) are tenuous- if you truly want to find out about management support, then why not ask a direct question about management support? Or at least one that clearly identifies management rather then your supervisor or someone at work.
Sometimes I also think that what makes it good- a survey that works across many different industries for all different sizes of organizations and benchmarks them all, also makes it bad- doesn't engagement change with some of those variables? Aren't we missing important things by limiting ourselves to the 12 questions that work for everyone?
I suppose that's enough of my rant :>) , thank you for reading!
Phil

[Rosa Say]

Aloha Phil, thanks so much for explaining and sharing your thoughts, and as a workplace aloha coach, I've gotta say I could hug you for my own learning in this, so consider this a virtual aloha hug coming your way!

I think you are hitting on a big problem with most assessments and employee surveys: Rarely is their intention adequately shared in organizations, and those who actually take them are left wondering on their own - I wonder why we're doing this, and I wonder what they are looking for, and I wonder what the boss intends to do with this stuff, and I wonder why I even bother with this, and I wonder why no one remembers what happened the last time we did this... then the survey or the assessment gets rendered pretty useless because those who take them just game 'em to play it safe and dispense with the whole affair.

I can tell you from my own experience with it (and I continue to use it) that the Gallup Q-12 in particular is just part of an entire strengths engagement process for them, that IS the goal, and it works well when done right. However what you are bringing up does make it appear that they may not have done a great job this time in helping everyone understand the process first --- I hope you have someone there you can share your questions with, for getting them answered in your company would itself be a valuable process for all of you - particularly your question on what will be considered within or out of your control. Another for instance... I can tell you that the Gallup Q12 is designed to be very actionable (but you are right, when you just read those questions they can seem airy-fairy)... If you take the online assessment they offer to determine your individual strengths for example, the report you get back has your top 5 strengths, with a list of action steps for each you can take to further strengthen them, and thus your engagement.

Somewhat related to this, it gets me to think again about that fine balance with the "free" information consultancies give away in their books (which Gallup does a lot of) that is really a lead-in to buying their services... it can do damage in all those businesses which will never have the budget to hire them.

[Joe Hungler]

We took a survey from Buckingham/Coffman's First Break All the Rules. We made a small change, although it was about 5 years ago now, so I don't remember how we changed it. I let someone borrow the book and can't check it out. We've use SurveyMonkey to implement it and have found it helpful. We track from year to year and by area. It is actually built into our supervisor's reviews. It is short and sweet which helps in getting them back. The book explains the research behind it. By using the scores from different questions, it meaures Basic Needs, Confidence, Belonging, Growth and Retention. I'm too lazy to rewrite it, so here is a link to it (scroll to the bottom). http://unclejoesleadershipblog.com/2008/02/07/employee-engagement-a... . I'd be interested in hearing what you think of it.

[Jean Douglas, CMC]

I too have some very strong opinions about employee surveys. I speak from the perspective of a survey expert (both in my education and professional experience). There are many things identified in the posts above that I agree with:
- the process must engage the people
- results must be relevant and actionable
- the results must inform the business planning and management activities
- employees must see the links between the results and actions

At the heart of this is treating an employee survey as any other business decision, considering things such as:
- objectives of the survey and its relationship to the business
- budgeting (dollars and other resources)
- timing
- communication planning (audiences, messages, media, short-term and long-term, managing expectations, etc.)
- survey development and deployment (standard or custom, paper and/or electronic, etc.)
- dealing with the results (analysis, interpretation and reporting process)
- linking results to action (relevance to the people and the business)

The parts that I have issues with include:
Conducting a non-anonymous survey. Closely tied to this, but not mentioned in the comments is "voluntary" participation. Questions, I have for those who conduct non-anonymous surveys are:
- what happens to those who do not participate? Or are they mandatory?
- who sits down with each employee and manager to review the results? are manager's trained?
I propose that you are in fact not conducting a survey (not in the true sense) but are using survey questions as a pre-meeting checklist for guiding the discussion. In this case, tabulating the results for the division/department make no sense as the results are statistically invalid and unreliable due to the bias introduced to the survey results by the methodology used. I would also caution you about using the results as a form of performance measure - check with your own legal department to see if you are possibly violating any employment laws/codes.

While it would be nice to have a trusting workplace, as long is there is any form of hierarchy, there are positions of power and giving your boss a negative review isn't usually a smart career move. There isn't any way you can prove that the survey results are not biased and therefore the findings are questionable.

From a profession and ethical standpoint the ethical and professional codes of conduct for employee surveys require:
1. The individual survey results are confidential and/or anonymous (note: these two terms are different. I mention this because I often see them used interchangeably). Survey results are presented using grouped data (e.g., results are not shown if there are fewer than 10 respondents). Survey results are not presented in such a way that an individual could be indentified.
2. Participation in an employee survey is voluntary. Any employee who chooses not to participate shall not be subject to negative consequences. In fact, the identity should not be known/revealed if the survey has been anonymous or held confidential.
Custom or Standardize surveys
The benefit of having a standardized survey is that it has been designed and tested and has a known level of reliability and validity (provided the source is reputable). Custom survey items can add that extra bit to a survey that needs to address some very specific topic within the organization. However, writing good survey items isn’t as easy as it may seem.

Some of the participants in this discussion thread mention the Gallup Q12. It wasn’t clear if you are working with Gallup or simply using their survey; if the latter, then hopefully with their permission. If not, then you should check to ensure you are not infringing on their copyright (which can carry very heavy fines). I should also point out that for those who use the Q12 and require employees sign it, the comparison to the Gallup benchmark information is meaningless – since the methodology you use (signed) is different than Gallup’s (unsigned) and in doing so it biases the results.

I will address the use of public survey tools (e.g., survey monkey, zoomerang, etc. in another discussion thread as my commentary in this one is already quite lengthy. The same goes for survey analysis and interpretation – I will post this in a different topic later.

[Warren Young]

Hi Phil, I notice this discussion is a few months old which menas your thinking may well have developed further since Feb. For the past couple of years I have been looking at various ways of measuring engagement. There seem to be two camps; one devotes attention to identifying engagement drivers in your organisation, and the other like the Q12, measures against a global benchmark assuming common drivers.

I have learned from working with over a dozen of organisations using the Q12 for the first time, there are naturally many questions about the methodology, science behind it, relevance of questions, cultural differences and so on, especially when the tool is newly introduced. These questions need to be raised and worked through. But they are not the central issue. They need to be worked through, or they can be distracting if left unanswered.

As we move through a process of gathering understanding, organisations and teams develop their own shared frame of reference of what/how we do engagement. In particular the emphasis rightly goes onto the building part, or what we do in reponse to a survey result. The survey tool itself becomes somewhat less significant, when compared to the process of building engagagement.

I have also experienced a number of examples where introducing a particular tool has brought things to life, sometimes after years of using other staff surveys with benign effect - even if the scores were previosly "higher".

The really important question is does the approach induce a response for improving engagement levels, and is that making an impact on improving organisational performance indicators?

[Judy Nelson]

Can you describe the Q12?
Thanks.

Judy Nelson

[Warren Young]

Hi Judy
Sorry I should have expanded on that. The Gallup Organizations framework for employee engagment is based on behavioural economics called the Q12 - it is actually 13 questions; 12 of which have a statistically proven causal relationship with high engagement levels and an overall satisfaction question. Not dissimilar to Maslows hierarchy of needs it is based on four principles of meeting basic needs, mangerial support, team work and learning and growth. As a company they have enormous depth in data analysis, so one of their real strengths is benchmarking and linking engagement data to organisational metrics.
Their website http://www.gallup.com
Disclaimer - I don't work for them! ;-)

Warren