RIP #15: A Formal Model of Post Effort and Learning on Stacker News^[1]

Previously, I posted a formal model of posting behavior on Stacker News. The model formalized the conditions that give rise to the phenomenon of higher posting costs leading to fewer posts, but higher quality posts.

In that model, quality was a random variable. People just randomly had a good quality idea or a bad quality idea. The mechanism driving the result was that if you just happened to get a low quality idea, you were less likely to post it if the posting cost was high.

However, one of the goals of my paper was to show that value-for-value incentives motivate people to make posts that other people like. That suggests some kind of effort goes into making the posts, not just randomly received post ideas. Therefore, I have now extended the model to incorporate effort, which is something @Undisciplined also suggested.

"You might work hard and the post ends up being a dud; or you might give only a little effort and have it be your highest zapped post of all time""You might work hard and the post ends up being a dud; or you might give only a little effort and have it be your highest zapped post of all time"

The model works much the same way as before. People randomly get an opportunity to post. When you have an opportunity to post, you first decide how much effort to exert in making the post. This decision is driven by your expected returns to effort and your cost of effort. Effort determines the quality of your post, but not entirely; there is still some randomness. For example, you might work hard and the post ends up being a dud; or you might give only a little effort and have it be your highest zapped post of all time.

The thing is, you don't know exactly the returns to effort. It could be that effort pays off well; it could also be not worth your time. This is something you must learn over time as you post more on Stacker News. You learn this based on your "zap surprises". That is, if you make a post that you put $X$ amount of effort into and expect to receive $Y$ zaps, but you receive more than $Y$, this is a positive signal on the return to effort, and you revise your expectations upwards. But if you receive less than $Y$, then this is a negative signal on the return to effort, and you revise your expectations downwards.

"The thing is, you don't know exactly the returns to effort. You learn this based on your "zap surprises""The thing is, you don't know exactly the returns to effort. You learn this based on your "zap surprises"

The main result of the model is to show that people who have accumulated more positive zap surprises over their posting history are more likely to subsequently make high quality posts. And vice versa: that people who accumulated more negative zap surprises over their post history are less likely to subsequently make high quality posts.

The result motivates an empirical framework in which I regress a user's next post's quality on their history of zap surprises up to the time of posting. That wasn't how I was doing it before, so the modeling exercise was useful. Haven't actually done the empirical tests yet, so it's an open question of whether I'll find the hypothesized result.

The full model and proof is stated below. As a flex.

ModelModel

We extend the model from the previous section to account for effort. As before, opportunities to post still arrive as a Poisson process with rate $\gamma$. But before observing $(q,ϵ)$, the user first decides how much effort to exert in crafting the post. Effort affects the distribution of post quality and intrinsic motivation, $f(q,ϵ;e)$ (users may be more motivated to post if they've exerted a lot of effort). The cost of exerting effort is $\kappa (e)$ with $\kappa (0)=0$ and ${\kappa }^{\prime }(e)>0$. The user's expected utility conditional on effort is:

Without uncertainty, the model would be equivalent to before, just with the distribution of $(q,ϵ)$ moderated by the optimal effort level. We therefore assume that users have uncertainty over the returns to quality. That is, let:

where users have uncertainty over $\theta$. Note that this is without loss of generality because $q$ has no natural units.

Given that users are uncertain about $\theta$, they must infer it from their received zaps relative to quality. We assume that users observe $q$. Let $z_i$ be the zaps received within a fixed time period, e.g. 48 hours, for a specific post $i$. $z_i$ is a noisy signal proportional to $V(q_i)$, and therefore:

Now let a user have a posting history of posts indexed by $i=1,\dots ,N$. An unbiased estimator for $\theta$ is:

That is, the user's unbiased estimate for $\theta$ is the average of their posts' zap "surprise", i.e. how much additional zapping it received over and above its baseline quality level.^[2]

We can now demonstrate that effort exertion is increasing in $\hat{\theta }$ and that, as a corollarly, the expected quality of the next post is increasing in $\hat{\theta }$.

Proposition 2.
Let $i=1,\dots ,N$ index a user's previously made posts. Let $q_i$ denote the quality of those posts, let $z_i\propto V(q_i)$ be a noisy signal proportional to the present value of zaps that post $i$ receives, and let $\hat{\theta }=\frac{1}{N}\sum \ln z_i-q_i$. Furthermore, assume that $F(q|ϵ;e)$ satisfies FOSD in $e$, conditional on $ϵ$. Then, optimal effort is weakly increasing in $\hat{\theta }$.

Proof.

Given $\hat{\theta }$, the user's expected utility from effort is:

It suffices to show that $U(e;\hat{\theta })$ is supermodular in $(e,\hat{\theta })$. For any ${\hat{\theta }}_2>{\hat{\theta }}_1$:

The integrand is supermodular in $(\hat{\theta },q)$ because $\exp (\hat{\theta }q)$ is supermodular in $(\hat{\theta },q)$ and because the $\max$ operator is convex and increasing. Furthermore, since $F(q|ϵ;e)$ satisfies FOSD in $e$, it follows immediately $U(e;{\hat{\theta }}_2)-U(e;{\hat{\theta }}_1)$ is increasing in $e$, and hence $U(e;\hat{\theta })$ is supermodular in $(e,\hat{\theta }).$

Q.E.D.

1. Note: This is a series in which I am publicly documenting the research process for an academic study into financial micro-incentives on discussion platforms, using data from Stacker.News. See here for a list of updates. ↩

2. This assumes diffuse priors; if the user had a more precise prior, their posterior mean would still be linear in $\frac{1}{N}\sum \ln z_i-q_i$. ↩