Continuous listening and frequent pulse surveys have become popular topics in organizations. Companies feel the need to have frequent input from employees, so they design a process to continuously collect data on engagement and other key people metrics. Some have even said the use of continuous listening programs is like having a Fitbit for the company.

I have used a Fitbit. I loved the immediate feedback about the number of steps I had taken and how long I had been exercising. It made me aware of how much (or little) I had been moving, and that information encouraged me to get on my feet. I was motivated to take as many steps as I could, and to continue to increase my average of daily steps. It’s fair to say that I got a little obsessed with it. I wore my Fitbit as often as I could.

But it didn’t take long to see where my Fitbit was letting me down. When I triangulated my steps with other counting tools, I realized my Fitbit was inaccurate. It didn’t count a step if I didn’t use my arms, so all my steps in the grocery store or dragging my suitcase through an airport didn’t count. I learned that I could game the system, so when I felt my actual steps were undercounted I shook my wrist to activate the counter to accumulate the step count that I thought was correct.

While my Fitbit told me how many steps I had taken, it didn’t tell me the quality of those steps. Walking through a grocery store is different than hiking a rocky uphill trail. I could compare my step count to that of others, but was it benefiting me to have more steps than they did? Its measures were pretty myopic. All that mattered to the Fitbit were steps; it didn’t care that I was swimming laps or running or lifting weights. What difference did all those steps make to my health and my life?

I admit that my obsession waned after a while. In fact, I haven’t put on my Fitbit in years.

So how are pulse surveys and continuous listening different than a Fitbit? If they ask the same questions over and over, if the focus is solely on the “score” (and managers game the system to push that score to what they think it should be, and if little insight beyond the score is given, then the survey is a lot like a Fitbit. It’s valuable initially, and it may create a high level of interest (maybe even a little obsession). But at some point, it will lose its appeal.

Why? It isn’t accurate in its evaluation of organization health. The feedback it provides is narrow and incomplete. It doesn’t take into account strong headwinds or the steep slope of the trail ahead, which may contribute to the organization’s overall wellbeing—or lack thereof.

Numbers are great; tracking is great. But evaluating organizational health and effectiveness is complex and nuanced. One number doesn’t tell the tale, no matter how many times you collect the data or how continuously you listen.

It’s better to create a survey strategy. Ask the question: What’s important to the company and what will it take to succeed? Then build a survey plan to collect the right data—metrics, multiple populations, meaningful timing, and frequency—that provides insight and explanation and suggests action.

Who knows, I may get another Fitbit one day. I just won’t rely on it for the full story.

For advice on better ways to measure employee engagement, check out my new book Engaging the Workplace.