Forecasting Shouldn't Be a Gamble. 5 Things to Do to Succeed with Crowdsourced Forecasting.

By Vanessa Pineda on July 25, 2017

In more than a decade of working with large companies to building prediction markets and other crowdsourced forecasting mechanisms, we’ve seen one common thread with the projects that are unsuccessful. Project owners overestimate the technology (especially those that love to “geek out” on prediction market research), and underestimate what it takes to engage people to make the technology successful.

Though at our core, we are a startup that designs technology products, we know all too well that the success of our technology, in this case our Cultivate Forecasts platform, is a function of how people interact with it. In fact, since nearly 90% of our technology-only projects come back to us asking for consulting support, we now nearly insist that first-time projects include an up-front planning workshop at the very least. This way, we can work with clients to create a plan that will empower and better engage the people using the technology, so that it’s a successful outlet to acquire strategic input.

Perhaps counterintuitively, our project goal is never to achieve greater forecasting accuracy. Instead, our goal is to engage and recognize people—it just so happens that the byproduct of engaged people is more accurate and objective business forecasts.

Giving Away Our Secret Sauce

Here is a roundup of the top 5 project risks we typically encounter and how we mitigate them—yes, they are all people-related.

Risk #1. Engagement. Without an influential leader as a visible project champion, people won’t feel motivated to participate.

Like with any change-related project, leadership sponsorship is a critical component to driving employee behaviors. If you don’t have a leader to champion the effort - stop reading now, and forget about launching any project until you do. In our most successful projects, leaders actively sponsor predictive questions that are relevant to their decision-making needs, and they speak about the results they’re seeing in various forums (e.g., in one-on-one informal conversations, during team meetings or town halls, etc). In our analysis, participation is at its peak when leaders send out friendly reminders or executive reports of the forecasting.

Risk #2. Communication. Without regular feedback, people won’t feel or believe that their input is being heard – and has a purpose.

An executive at a large Australian corporation once very humbly asked us: “What do I communicate when what people predict is controversial, or we know we can’t take action on a prediction?” For executives, the rule is: Always, always communicate. You don’t always need an answer, but you still need to tell people you don’t have an answer.

You can say something as simple as, “We know the results are telling us X. We’re not able to say what we’re doing about it yet, but we’re listening.” This conveys that you’ve taken the time to look at results, and you’ve considered what can be done about it, even if it’s not appropriate for the business to take action for one reason or another. It also stops people from thinking that their forecasts aren’t serving a purpose, which results in declining participation. When leaders show their humility through authentic communication, it creates a feedback loop that keeps people interested and connected to the project and the organization at large.

Risk #3. Bad Results. Not getting enough people to sign up or come back impacts the validity and credibility of the results as there is more room for bias.

The status quo at most organizations is underrepresented opinions. The benefit of crowdsourcing forecasts is that it's a mechanism for capturing employees’ knowledge and expertise like never before. To get the most valuable results, though, you need enough people to make forecasts (always invite a much larger number than your target) and diversity in your participant pool. Diversity is important in terms of one or more of the following: level, experience, geography, division, interests. This provides unique perspectives, so you can get more objective consensus results and uncover contrarian viewpoints.

To continue to improve the organization’s forecasting accuracy, you will also want to provide employees feedback about how accurate they are (or not) and what biases exist in their forecasts, so they can become better calibrated over time. We work with clients to create this feedback both through the application and in custom reporting.

Risk #4. Content Management. Without interesting and relevant content, people won’t come back to forecast.

Similar to what attracts people to read an interesting newspaper article or blog post, compelling forecasting questions matter to draw people in. Of course, these will include questions that are directly related to the project, but it will also include “fun” questions pertaining to other company initiatives or recent news. We recommend publishing at least one new question per week to keep participants active. Additionally, questions should be a mix of short- and long-term forecasts. Though long-term questions may be of more valuable to the business strategy, short-term questions allow participants to see how accurate they are sooner (which is more motivating!).

Risk #5. Incentives. Selecting which behaviors to recognize and reward will impact engagement.

Finally, how you structure your incentives (intrinsic and extrinsic) should be based on what is most rewarding for your people. Maybe this includes posting a list of most accurate forecasters on the homepage of your Intranet, calling them out in a team meeting, throwing a pizza party for the most active regional office, etc. We encourage our clients to reward participants not only for accuracy, but other positive behaviors as well (e.g., contributing questions, helpful commentary, and suggesting resolutions once a question outcome is known).

Overall, there's usually a solution to any potential project risk; the bigger challenge is putting the solutions into action. But when achieved, you’ll see greater engagement and sense of purpose in employees, while giving leaders clearer line of sight for their most important decisions.

Are you interested in our help launching your crowdsourced forecasting effort? Contact us.

By Vanessa Pineda

Professional Services Director

prediction markets change management disruptive leadership crowdsourced forecasting