MathOverflow Studies

Below we describe the current studies involving MathOverflow, which are:

User Reputation and Perceived Quality of Submissions
Motivation

If you have any questions you can contact me, Yla Tausczik, at tausczik@mail.utexas.edu.

User Reputation and Perceived Quality of Submissions

Abstract: There are two perspectives on the role of reputation in collaborative online projects such as Wikipedia or Yahoo! Answers. One, user reputation should be minimized in order to increase the number of contributions from a wide user base. Two, user reputation should be used as a heuristic to identify and promote high quality contributions. The current study examined how offline and online reputations of contributors affect perceived quality in MathOverflow, an online community with 3470 active users. On MathOverflow, users post high-level mathematics questions and answers. Community members also rate the quality of the questions and answers. This study is unique in being able to measure offline reputation of users. Both offline and online reputations were consistently and independently related to the perceived quality of authors' submissions, and there was only a moderate correlation between established offline and newly developed online reputation.

To appear in:

Tausczik, Y. R. & Pennebaker, J. W. (2011). Predicting the perceived quality of online mathematics contributions from users' reputations. Proceedings of CHI 2011. (4 pages).

Tausczik, Y. R. & Pennebaker J. W. (2011, January). The effect of established and new reputation on influence in an online community, MathOverflow. Poster to be presented at the annual meeting of the Society for Personality and Social Psychology, San Antonio, TX.

Results

In this study we measured author reputation in four ways. We calculated to what extent these reputation variables were related to each other and to what extent they predicted the perceived quality of submissions. The data for the perceived quality of questions and answers was collected for the month of May, 2010. The data for reputation variables was collected for the period just prior to May, 2010.

Author Reputation

1	Offline reputation: Number of published math papers
2	MathOverflow Points: Quality ratings on past submissions (the reputation system as displayed on MO)
3	Authoritativeness: Linguistic markers of status (word count, first person plural, inverse of first person singular)
4	Social connectedness: Betweeness centrality in social network of past submissions. Two people are connected if one answered a question by another.

Are different forms of author reputation related?

Non-existent to moderate correlations between different forms of reputation. Significance of the correlations is given by '*' meaning p < 0.05, '**' meaning p < 0.01 and no star meaning not significant.

		2	3	4
Offline reputation	1	0.38**	0.06	-0.08*
MO Points	2		0.15**	-0.06
Authoritativeness	3			-0.00
Social connectedness	4

Does reputation predict the perceived quality of submissions?

Perceived quality of submissions was measured by the number of votes on MO.

Significant predictors (p < 0.05) from multi-level regression models for questions and answers. Colors represent the relative percent of variance explained by the four reputation variables, the gray represents the percent of variance in the perceived quality of submissions that is not explained.

How much variance does reputation explain?

40% of variance in perceived quality of questions
7% of variance in perceived quality of answers

Conclusions:

At most moderate correlations between offline and online reputation
All reputation variables predict some variance in perceived quality
Overall reputation explains a small percent of variance in perceived quality
Counterintuitively, reputation better predicts perceived quality of questions than answers

Motivation

User Survey

Participants were recruited through two methods during September, 2010. All active participants (at least 30 reputation points) who included a personal website with a valid email address in their profile were contacted to take the web survey via email. The survey was also advertised on the main MO website for four days. We have stopped recruiting MO users to take the survey, but if you want to complete the survey click here.

This project is ongoing and a paper is currently being prepared for publication. More information will be provided later. For now here is some basic demographic information:

Demographics:

A total of 403 participants completed the survey. Of the 217 users who included their username on MO the users sampled contributed at the time about 27.5% of the posts on the site. Of the 403 participants:

96.9% were male
90.1% are or were in a math degree program

Motivations:

Participants were asked to rate "Approximately how often do you use MO for satisfying the following needs". They answered this item on a 5 point Likert scale from 0% (almost none of the time) to 100% (almost all of the time) for 13 items. Similar items were combined into five factors: getting information, giving information, status seeking, collective math (that is doing math with others), and recreation (which included having fun and procrastination). There were significant differences in the reported levels of different motivations. Bars represent 95% confidence intervals.