3 Fundamental Flaws of Employee Surveys and a Way Forward

Employee surveys—including annual employee satisfaction, culture, performance, engagement, and pulse surveys—are a great invention in many ways. Since industrial companies started to use employee surveys back in the 1920s, employee surveys have been an effective tool for evaluating organizational health and taking the pulse of company culture.

For managers who rely on gut feelings, employee surveys offer a huge step forward by providing evidence of how things are really going. A Harvard Business Review post written by Eric O’Rourke and Scott Judd of People Analytics at Facebook and Adam Grant, an organizational psychologist at Wharton, asserts that, even in the time of machine learning and data science, employee surveys are still one of the best ways to measure engagement.

Although employee surveys have been widely successful, it’s clear they have some fundamental flaws that have never been resolved. In addition to all the issues and potential errors accompanying their design and analysis, employee surveys have three fundamental shortcomings that apply to traditional paper formats and cutting-edge cloud engagement and experience solutions.

1. They Are Boring

Employee surveys are boring. If you take a walk around your organization during lunch to get feedback on a survey, you will likely perceive a lack of engagement. Although it’s possible to find a few feedback enthusiasts that suggest even more questions in the survey and love writing extensive comments, the list of complaints is usually long:

“Why should we answer all these questions?”
“It’s way too long.”
“It’s always the same questions.”
“Have you noticed that the questions are repetitive?”

Unfortunately, the introduction of online survey tools and analytics platforms hasn’t changed significantly. Companies are moving away from annual, bi-annual, or company-wide assessments that are extensive and time-consuming.

2. They Are "Not Understanding Us"

When conducted quantitatively, employee surveys describe the precise distribution of a few limited aspects or phenomena in a population. Unfortunately, surveys cannot detect social nuances of company culture and are incapable of understanding employees in a meaningful way. This lack of depth is another critical flaw of surveys. Employees do not feel that surveys truly ‘understand’ their circumstances, the type of work they do, their job, how they feel, and what it is like working for the organization. Quantitative surveys do not fully account for diverse employee experiences and various points of view, relying instead on standardized and structured numerical measurement.

But organizations are more than numbers. Researchers who have consistently advocated for using qualitative methods emphasize the need for detailed descriptions and contextual interpretations when studying companies. They argue that analysts must observe and get to know employees in their setting to truly grasp their thoughts, beliefs, and experiences in their organizational life.

Qualitative surveys effectively provide follow-up inquiries to get the heart of an issue with questions like, “I hear you say that communication is a major challenge for your team. Could you elaborate on why it is such a challenge?” Sometimes, a few well-planned, reflective questions are all it takes to uncover the most profound issue.

Many employee engagement and experience platforms rely heavily on comments to narrow down organizational challenges and prescribe solutions. Use of sentiment analysis and machine learning algorithms allow these platform solutions to uncover subtle nuances behind the survey results.

Additional data certainly supplements numerical survey results and provides a richer context to the findings. However, it can be problematic to rely heavily on comment fields, from a qualitative perspective. Comments can be unpredictable and random because only some employees voluntarily provide them. The accuracy of qualitative data relies on the qualitative researcher’s ability to carefully select interview participants and follow up on questions as needed.

The inconsistency of random, voluntary comments presents interpretative challenges. The choice between quantitative and qualitative methodologies depends on the organization’s feedback culture.

3. They Are Not Providing a Clear Foundation for Actions

Employee surveys do not provide a clear direction for action and improvement. While they are accurate, reliable, and valid regarding specific issues and contexts, they do not offer the depth and detail needed to take action.

Imagine a doctor who must provide a patient with an accurate diagnosis to prescribe proper medication. Delayed treatment, over-medicalization, or inaccurate prescription-writing can be fatal. What if the doctor relied on a one-size-fits-all, 20-question survey to diagnose his or her patients? Of course, this method would be too broad to make solid conclusions about an individual patient’s condition or medication.

Traditional annual satisfaction and engagement surveys may be thorough and accurate, but their infrequency can come across as tone-deaf. This may lead a disgruntled employee to state, “Thanks for highlighting our team conflict. It happened three months ago!” A year seems like a long time in dynamic organizations characterized by rapid innovation and rapid change.

Moreover, all quantitative surveys, annual as well as pulse surveys, are limited by the lack of depth and precision. Even an employee engagement survey with 70 questions is too short to cover major workplace characteristics meaningfully. Let’s say we would like to design a survey that covers ten metrics, which includes communication, collaboration, leadership, and other significant determinants for employee success and engagement. We develop seven questions for each of the ten metrics (70/10). For our communication metric, it makes sense to split the questions further into three categories: a) within our unit, b) across units, and c) with manager(s). We are left with only two or three questions (7/3) to explore each of these more specific areas in this level of detail.

To add a new wrinkle, question scores are also often difficult to interpret. The line manager is often presented with his or her unit results like this: “There are issues with your internal communications…your score is only 5.9, and the average is 6.2…Our survey is 100% valid and the survey benchmarks are based on the results of 12 million surveys!” This is hardly informative or actionable information.

For a researcher, a difference of 0.3 may seem significant. However, for the manager, what does it mean to be 0.3 below the average? And with only two or three questions covering internal communication, it’s hard to identify the precise issue and take explicit action. Is the internal communication problem related to meetings, work conversations, conflicts, online communication tools, or something else?

One solution to this confusion is to initiate focus groups, hold workshops, or invite HR business partners or external consultants to investigate the issue further. Failing to identify the root problem costs time and money in larger organizations. What’s worse, action is never taken.

Identifying a Way Forward

These major weaknesses of surveys can lead to problems related to survey fatigue. If a company scores inexplicably low on specific questions, some may simply remove those questions to obtain better scores next time!

Some companies might even give up asking employees for feedback. One startup made it their mission to rely on behavioral data and abandon surveys altogether. Although more extensive data is welcomed, abandoning employee surveys is not the best path forward. In reality, surveys effectively collect large amounts of employee feedback and go beyond behavioral data in numerous ways. Despite their setbacks, surveys are one of the best methods for ensuring that employee voices are heard.

Since we started Livingroom Analytics, one of our goals has been to resolve these problems with employee surveys. We encourage and believe in employee feedback, but do not want to burden company members with boring and irrelevant questions.

Over the past few years, we have implemented an innovative survey approach. Extending quantitative methods with qualitative interview techniques, our dynamic drill-down surveys are combining the best of both worlds.

Although innovative, our survey approach is not entirely new, as it has been successfully used to design learning tests for years. The concept of using the Dynamic Question Ordering framework in online surveys has been discussed in statistical journals (Early et al., 2017). Still, as far as we know, we are the first to apply the methodology to employee engagement and experience surveys. It’s not rocket science, but it has taken us years to develop and fine-tune our approach.

Unlike traditional employee surveys that typically include 20-60 fixed questions, our surveys choose from an extensive query bank containing approximately 1,700 research-based questions to cover ten key employee experience areas dynamically.

Our survey’s first 16 questions are the same for all employees, as fixed questions provide validity and reliability to the areas we are researching. The remaining 15-20 questions, however, are dynamic drill-down questions. Depending on the answers to the first 16 questions on ten primary areas, our survey bot functions like a trained qualitative interviewer by drilling deeper down with up to six inquiries covering just one to two key areas. Therefore, the first questions follow a quantitative methodology while the drill-down questions imitate a qualitative, semi-structured interview method.

In this way, each employee receives a custom survey. We limit the total number of questions to 35, with an overall average of 30. Our surveys are not time-consuming, tedious, or uniform, which allows the employee to feel understood and more engaged in the process.

Finally, our methodology provides a more straightforward and precise foundation for managerial action. The drill-down approach allows managers to generate a complete graphical overview of the strengths and challenges they are facing through an employee experience heatmap.

By highlighting key areas and specific challenges, our heatmap visually represents relationships between strengths and weaknesses, providing much more clarity than traditional survey techniques. This allows the manager to develop clear, corrective action plans for their organization.

Our survey methodology has shown reliability over time and effectively diagnoses organizational and employee experience challenges. We believe our dynamic drill-down approach is an innovative way forward for employee surveys and will likely be used more in the future.