Cultivate Labs Blog

Discussing how prediction markets, internal crowdfunding, and other new workplace trends can be used to improve the #futureofwork

By Brian Leslie on Jun 12, 2017

Determining the Accuracy of Prediction Markets

So, you asked a prediction market question, and the outcome is now known. The question has been resolved and winnings have been disbursed to the forecasters who held winning positions. Forecasters know how well they did based upon their profits in the question, and you know who your good forecasters were too. But how accurate was your organization at answering the question itself? There are several things to consider when thinking about accuracy of the prediction market:

Read more

By Ben Golden on Sep 14, 2015

Evaluating Accuracy Effect By Percentile

In my last post analyzing my own forecasting history on Inkling Markets, I showed that I was consistently identifying long-shot bets that were more likely to pay off than their existing probability would suggest. In this post, I'll look at how my forecasts improve the accuracy of these markets, calculating how many the change in component Brier score within different percentiles.

Read more

By Ben Golden on Sep 11, 2015

Evaluating User Forecasts By Percentile

To analyze forecasts' accuracy, I looked specifically at forecasts that crossed percentile thresholds, and then examined how often the result being forecast actually occurred.

Read more

By Ben Golden on Sep 09, 2015

Usually Wrong Can Be Right

We're developing some new tools to analyze forecasters' performance, biases, and ways to help them improve, and I've been digging into my own forecasting history on Inkling. I've focused on a set of 3,343 forecasts I've made in questions that use an LMSR algorithm and have already resolved. The first interesting finding is that most of my forecasts have been wrong.

Read more

By Ben Golden on Aug 31, 2015

What Ashley Madison Can Teach About Prediction Markets

(You know, besides how to cheat...)The recent hack of Ashley Madison--a dating service marketed towards married people--revealed that almost no women were actively using the site. Rather, the site's almost entirely male userbase was paying to interact with non-existent, fake, or inactive female accounts.

Read more

By Ben Golden on Aug 20, 2015

Gaming Prediction Markets

Prediction markets are generally very good at generating accurate forecasts, but a key secondary challenge is to determine which forecasters are contributing most to forecast accuracy. User scores are closely linked to their accuracy because the underlying market mechanism rewards users when they move a forecast closer to its actual result and penalizes them when they move the forecast away from the result.

Read more

By Ben Golden on Jul 23, 2015

Pundits and Prediction Markets

As I've become more involved with prediction markets, I've grown increasingly frustrated with journalists who make predictions (aka pundits) without linking to prediction market questions. This is, in my opinion, lousy journalism, and insulting to readers.

Read more

By Ben Golden on Jul 23, 2015

When Intuition 'Trumps' Analysis

In growing my Inkling score from five thousand to ten million Inkles, one of the most important questions was related to the number of points each team would score in the most recent NBA season. The question asked about the difference between each team's points and the average of all teams.

Read more

By Ben Golden on Jul 16, 2015

The Best Is Not Enough

Barry Ritholtz has written a curious column titled The 'Wisdom of Crowds' Is Not That Wise for Bloomberg View, which criticizes prediction markets. This is not a new view for Ritholtz, as he reminds us by linking to six blog posts critical of prediction markets each written by...Barry Ritholtz. Indeed Ritholtz has made it his mission to find instances of prediction markets 'failing', and has found six of them.

Read more

By Ben Golden on Jul 02, 2015

I, Benthinkin

The challenge was daunting—Inkling users start with five thousand Inkles--while the top forecasters have accrued hundreds of millions (in one case billions).

Read more