How to measure employee engagement

[David Neilly]

I have been reviewing the literature on employee engagement and have not found any information about how the researchers...Towers Perrin, Gallup, Blessing White...actually measure and analyze engagement.

For example, Towers Perrin uses their 9 questions to measure engagement and then they put respondents into three "buckets"- Highly engaged, moderately engaged and disengaged. Does anyone know how the "buckets" are defined? For example, do they define highly engaged as scoring "Agree" or "Strongly Agree" on all 9 questions, or an average of 4 or better on a 5 point scale for the 9 questions?

I would appreciate any information that you can provide.

Many thanks,

Dave

[Jean Douglas, CMC]

I think that it would be best to place this topic in a separate area since it applies to surveys in general.

Please give me a few days to pull it together - I'm actually taking some vacation days (can't resist looking here though) and will reply to this post to let you know where to find it.

Jean

[Robert Morris]

Dave:

Hamish Deery is Regional Executive Director at Towers Perrin-ISR (Australia) and a staunch advocate of surveys to measure employee engagement. “If organizations are not measuring engagement they don’t know where to focus their efforts. What we often find is that the drivers of engagement vary from company to company and from operating unit to operating unit. So if you’re not measuring engagement and the things that drive it, you can be left focusing your interventions in the wrong areas.”

Deery strongly recommends the use of benchmarks when formulating a survey but only so long as the focus is on those issues that are (a) of greatest importance to employees and (b) require “actions in areas that you know are going to give you the biggest return on investment. Running a survey with benchmarks allows you to do that with confidence.”

I agree with Deery. If you wish to measure employee engagement, here is one approach to consider:

1. Conduct a survey among employees that asks them to rate a list of attributes on a scale of some kind. For example, a scale of 1-5 with 1 = Very Important, 2 = Important, 3 = Somewhat Important, 4 = Of Little Importance, and 5 = Of No Importance.

Here is a “typical list” of ten attributes provided by David Croston, author of Employee Engagement: “Senior Leadership behaviour, Relationship with immediate manager, Interesting and challenging work, Opportunities to grow and development, Involvement in decision-making, authority to exercise discretion, A collaborative corporate culture, Effective internal communications, Appropriate tools and training, and Supportive systems and structures.”

2. Compile the results to determine which “people issues” seem to be of greatest importance to respondents. Correlate responses with the highest priorities of the given organization as determined by its senior management. Then conduct another survey of employees in which the attributes consist of most important people issues in combination with issues of greatest importance to the organization such as productivity, elimination of waste, etc.

Hope that least some of this is helpful.

Best regards, Bob

[Lisa Wojtkowiak, SPHR]

Grab a cup of coffee - this is a way long response. ~:-)

Hi Dave,

I'm an employee engagement researcher and have spent a considerable amount of time learning the many different approaches. And I must say that there are many great models out there. But, if the needs of a particular company are not met in any research endeavor then the measure and analysis are useless. Furthermore, if an engagement study is conducted but the results are not communicated to employees or not acted upon then the research is also useless.

Ok, so now I'm done with my little soapbox and to try to address your question let me tell you about the one approach that I know the most about. I have been in research for over 12 years and most of it has been in Employee and Customer research. I specifically stay in Employee Research because of the people that my research affects. I see true improvement in a company after a project is completed and that part makes me feel good (actually great). The ORC approach takes employee research to the strategic level and helps to improve business performance. We all know that in our current economy, HR professionals are being asked and required to make that strategic contribution to earn a seat at the table (or remain at the table).

Here is a brief summary how ORC measures and analyzes employee research. Our method utilizes a database of over 200 questions to build and address certain needs/goals within an organization. The majority of our questions focus on three key concepts: Say, Stay and Stive.

“Say” describes an individual’s level of advocacy for an organization, both as a place to work and as a provider of services

“Stay” measures organizational commitment both in terms of intention to stay and desire to be a part of the organization

“Strive” represents discretionary effort – are the staff committed to ‘going the extra mile’ at work? Do they strive to improve the customer experience?

Employee engagement indexes can be established for key questions as well as the key concepts of Say, Stay and Strive. The indexes are very valuable in our trend comparison reports.

Benchmarking your staff survey results is also another valuable tool to manage engagement over time.

Other reporting options:
* A summary of the ‘highs’ and ‘lows’ giving top and bottom scores
* Results for each question asked, with positive, negative and neutral responses
* Comparisons against targets (KPI's)

But to try to answer your question about the actual calculations to measure the categories, I can only speak for our segmentation (buckets). We call it a cluster analysis and the parameters truly change based on the statistics of the data. Depending on what the data tells us the 'buckets' we use to identify what an engaged or disengaged employee looks like are: Highly Engaged, Engaged, On The Fence and Disengaged. We've often reference benchmarks to determine the validity of our conclusions.

I do take the time to review our methodology and other reporting calculations with my clients and if they're truly interested in the statistical calculations we have statisticians that will work with them personally.

Our methodology has been tested for over 20 years and the data is only data until our clients understand and act upon the results. Then we have successful research.

I hope this provided you with some information even if I couldn't answer your specific question. But I imagine that the calculation results for other companies are also varied based on the data. The calculations/methodology may be pure statistics (along with algorithms) like ours.

Please feel free to call me if you'd like more information. Good luck.

[John Kmiec]

This is a useful perspective, since I'll be conducting engagement research for my dissertation. Thanks John

[Jean Douglas, CMC]

Just wanted to point out that the "Say", "Stay" and "Strive" are the same three category names used by Hewitt. THey were "Say, Stay and Serve before 2001 but one client felt that "Serve" was subservient and so we came up with "Strive".

[Jason McPherson]

Hi Dave,
first of all I will declare my position. I am a Consultant and Account Director at Kenexa, we conduct Employee and Customer Engagement surveys, and prior to that I worked for Towers Perrin-ISR. The main method used for calculating Engagement scores is the percentage of agreeing responses (response of Agree or Strongly Agree using a 5 point scale) for the items in whichever Engagement index is being used. So if a person responded Agree or Strongly Agree to 4 out of 8 questions then that person would have a 50% Engagement score. However, we usually do not think of Engagement scores as belonging to a single individual and the score for the organization is the average level of agreement, defined in this way, across all employees. The buckets or clusters of different people mentioned by others here are calculated using a statistical clustering technique that usually uses the average responses to smaller groups of items within the Engagement index. Once the clusters or buckets have been identified then the percentage favourable scores for each of the Engagement components can be calculated much in the same way that the Engagement scores are. Generally the percentage favourable scores are used as people find them easier to understand than mean or average scores. Some companies do use non transparent calculations for their Engagement scores. I tend to think this practice is problematic and counter to open research policies as results cannot be checked independently. It can also be used to try and make it difficult for companies to change vendors which could be seen as a strategic rather than a best practice research policy.

I'd be happy to provide further details.

Kind regards,
Jason McPherson

[Deirdre Pickerell]

Greetings all...I just joined the network so am still finding my way around. This is a particularly interesting discussion for me as I look at engagement measures as part of PhD studies. My research is looking at the impact of career development/management on employee engagement. As part of my research I will pilot a tool to measure engagement. As noted by a number of contributors to this thread, many engagement tools are proprietary and it isn't clear how the tool was created. As such, I have little choice but to create my own tool...an interesting project in itself.

Anyway...great to be here.

Cheers,
Deirdre

[Stephen J. Gill]

I am very impressed with what network members are contributing to this forum. It's like a short course in survey research.

However, I want to point out that surveys are just one way of measuring employee engagement, and they are a limited way. If I want an estimate of how many people in an organization feel engaged and not engaged, I’ll do a survey; it’s an efficient and sufficiently effective way to get that estimate (assuming something close to a random sample of the target population). In terms of statistics, I wouldn’t rely on the average (mean) of responses to a scale as my primary indicator. That statistic is often a distortion of what is really happening in the organization. I would be more concerned with the distribution of responses across the scale. A response from a hundred people on a five-point scale that is fifty “1s” and fifty “5s” is very different from a response that is a hundred “3s”. In both cases the mean is 3.0, but each case tells a very different story.

If I want to know why employees feel engaged, how various organization interventions affect engagement, how engagement relates to performance, and what can be done in the future to enhance engagement, then I use a different method of measuring engagement. Usually, I will interview employees who are highly engaged and ask them these questions. A survey cannot answer these questions satisfactorily. I have to use interviewing to fully explore an employee’s thinking and experience. I find that ten to fifteen interviews of successful employees in an organization usually gives me the stories I need to draw reasonable conclusions about the links between the workplace experience and the performance of employees.

[Paul M. Mastrangelo]

Dave,

I am joining this conversation late in the game, but I agree with your thoughts. If engagement is important, then let's define it and measure it without any ambiguity. I was invited to this network because of a white paper that I shared, and it discusses three major issues regarding engagement: definition, measurement, and intervention. I am attaching it to this reply (I think), and it is also available at HR.com HERE.

Your comments welcomed.

Regards,

Paul