How to measure employee engagement

[David Neilly]

I have been reviewing the literature on employee engagement and have not found any information about how the researchers...Towers Perrin, Gallup, Blessing White...actually measure and analyze engagement.

For example, Towers Perrin uses their 9 questions to measure engagement and then they put respondents into three "buckets"- Highly engaged, moderately engaged and disengaged. Does anyone know how the "buckets" are defined? For example, do they define highly engaged as scoring "Agree" or "Strongly Agree" on all 9 questions, or an average of 4 or better on a 5 point scale for the 9 questions?

I would appreciate any information that you can provide.

Many thanks,

Dave

[Andrea Vienet]

HI Dave
We use Towers Perrin for our EOS and within the engagement measure - it is broken down into 4 key areas that make that up - that is Think, Feel, Act - Extra Effort, Act - Stay. So the questions that fall into those categories make up the engagement score. Towers Perrin don't go a lot into exactly how those measures are calculated as that would be sharing their intellectual capital. However I"m sure they would with any clients (eg if we asked for more detail around exactly how its done they would)
My view is that the engagement score is made up of Agree and Strongly Agree scores from the items that make up this measure.
We also then get people put into 4 buckets - Actively engaged, Engaged, Engaged but Leaving, Disengaged. I know I haven't answered your question but I think the answer is - if you need more detail on exactly how it is worked out you may need to speak to the experts within Towers Perrin or Hewitt etc. As their clients we tend to leave it to them to do the analysis and provide us with the summaries.

[David Neilly]

Hi Andrea,

Thanks for your comments. I have come to the conclusion that I will not find out how engagement is actually measured because Towers Perrin etc. see this as proprietary. I have to say that I don't really see what is proprietary about these calculations....it is somewhat arbitrary for anyone to define what is "highly engaged", "somewhat engaged", etc. I would also think that revealing the calculation methodology would stimulate some review and criticism, which would in turn improve our collective understanding of this field.

So thanks again for taking the time to comment.

Dave

[Jean Douglas, CMC]

I agree with you, David. In academic research, it is standard practice to define the measures used so that studies can be replicated to determine how sound the resulting conclusions are.

One of our members - Faye Schmidt - wrote a wonderful paper for the BC government that gives a very thorough review of the different versions of engagement and measures. Here is a link to the document:

http://www.psagency-agencefp.gc.ca/arc/hr-rh/wlbps-eeoppfps/documen...

The way in which the "buckets" are used (or rather calculated) is where the various degrees of engagement seem to differ between consulting firms. Some set higher cut-offs for engagment than others and some use averages while others use more complicated measures to fill their buckets.

This alone is a good reason to understand how engagement is measured. Otherwise, it appears that the percentage of engaged employees in the country differ according to the firm one is looking at - when it may simply be a matter of what yardstick and how high is it set.

[David Neilly]

Hi Jean,

Thanks for sending the link, and for your thoughts.

[John Kmiec]

I'd be very interested in seeing more attention given to reporting how the reliability and validity of these instruments are addressed in these studies. It becomes a credibility issue, especially for academic research. More items to consider:

1. Definitions of engagement - Depending on who you ask, and there are many, the definitions vary. So, what exactly are we measuring? How do we know engagement when we see it?

2. Determinants - What causes or contributes to engagement? How do we know what to improve in order to increase employee engagement (another measurement question)?

3. Measures - Depending on the previous two items, what are we using to measure what engagement is and the factors that contribute to it? For instance, the Gallup Workplace Audit or Q12 appears in the literature as a measure of engagement that seems to focus on determinants:

Harter, J. K., Schmidt, F. L., & Hayes, T. L. (2002, April). Business-unit-level relationship between employee satisfaction, employee engagement, and business outcomes: A meta-analysis. Journal of Applied Psychology, 87(2), 268-279.

Another instrument I've seen repeated in the literature is the Utrecht Work Engagement Scale:

Storm, K., Rothmann, S. (2003). A psychometric analysis of the Utrecht work engagement scale in the South African Police Service. SA Journal of Industrial Psychology, 29(4), 62-70.

4. Results: What are the fruits of engagement? Retention? Productivity? Customer Satisfaction? Innovation? Market Share? How do we know for sure? What measures (scientific/academic studies) are available that tie engagement to results, so they can be replicated and further studied?

5. Constructs: What theoretical constructs are we dealing with? Do they deal with Self-efficacy? Organizational Development? Economic Development? How does the research support and build on these?

[David Neilly]

Hi John,

I think that we will have to wait for the academics to properly tackle the excellent questions you have raised with respect to the validity and application of employee engagement. Consulting firms like Gallup and Towers Perrin do not worry as much about the "academic" validity of their work, but rather the "marketplace" validity. As long as there is strong face validity to their clients and they can credibly sell their engagements, that is good enough. If they have marketplace validity then they are motivated to keep their approaches, assumptions, weightings etc. to themselves.

I personally don't see what should be proprietary about their calculations...maybe they don't want to be criticized by academics and have their franchises threatened.

[John Kmiec]

Thanks, David. I agree that the differing agendas and perspectives of scholars, practioners and consultants don't always align as closely as we would like. Still, I embrace the differences and continue to learn from them. This blog is a great way to network and learn from one another. As for me, I'm pressing forward with my study of engagement in the hopes my dissertation and any follow-on articles may add a tiny bit more clarity to the topic.

John

[David Neilly]

John, I checked out the reference to "Utrecht Engagement Scale" and found some interesting stuff. There is a presentation and documents with the scales, as well as references to the professor's website. I find the English language wording of his scale a bit awkward in a few instances, probably reflecting that his mother tongue is Dutch and his translation into English business terminology is not perfect. However, I like the way he has constructed the questions and his scale...rarely, sometimes, etc.

In the first file, he makes references to the statistical validity of his scale and that might be worth checking out...

I am attaching the files for you and others that might be interested.

[Jean Douglas, CMC]

I agree with you on this.

When I was with Hewitt Associates (2001-2007), I made sure that we conducted reliability analyses, fine-tuned the surveys based on the literature reviews and analyses we conducted. I was very fortunate to have worked on the Best Employers in Canada study (and influenced the design of the studies in other countries) as the size of the database and hundreds of organizations contained in it meant that we could conduct really solid reliability and validity analyses.

In addition to the basic engagement and work environment items in the Best Employer survey, Hewitt (Canada) focused, annually, on a special topic of interest and then investigated the relationship between engagement and that topic - some of the topics I worked on (I designed the survey and analyses) included Leadership Development, HR, High Performing Cultures, Multi-Generational Workforce, etc. This provided an opportunity to see what if any differences emerged between high engagement vs low engagement organizations.

I am not with Hewitt Associates anymore, but say with confidence that employee engagement was approached from a scientific point of view, drawing from and adding to the current knowledge base of information.

[David Neilly]

Hi Jean,

Thanks for your comment. I used to be a client of Hewitt's and worked for Towers Perrin and don't doubt the effort, skill and desire for quality that goes into their engagement research. However, they don't publish their methodology for others to review/criticize, and ultimately they have a conflicted interest between their desire for reliability and their desire for commercial success. I was just stating the opinion in my previous post that there is a difference between academic work and commercial work, and that it would be nice to know more about their methodology.

[Jean Douglas, CMC]

I absolutely agree with you.
While I was there, Hewitt did share their methodology with others - it is just hard to find it. Mostly you have to know who to ask and you pretty much have to be a client.

Personally, I don't see the need for the secrecy. There are only so many ways you can create a measure - and clients need to be able to understand it - if it gets too complicated they don't believe it and the consultants doing follow-up work won't be able to capably explain it. The methodology isn't the key piece of the success - it is the advice one gives as a result of it.

[David Neilly]

Totally agree!! I assume you were a close colleague of Ted Emond's when you were at Hewitt? I saw him a few weeks ago and he is still going strong. Great guy.