Beyond the Code Itself: How Programmers Really Look at Pull Requests

Developers in open source projects must make decisions on contributions from other community members, such as whether or not to accept a pull request. However, secondary factors—beyond the code itself—can influence those decisions. For example, signals from GitHub profiles, such as a number of followers, activity, names, or gender can also be considered when developers make decisions. In this paper, we examine how developers use these signals (or not) when making decisions about code contributions. To evaluate this question, we evaluate how signals related to perceived gender identity and code quality influenced decisions on accepting pull requests. Unlike previous work, we analyze this decision process with data collected from an eye-tracker. We analyzed differences in what signals developers said are important for themselves versus what signals they actually used to make decisions about others. We found that after the code snippet (x = 57%), the second place programmers spent their time ixating is on supplemental technical signals (x = 32%), such as previous contributions and popular repositories. Diverging from what participants reported themselves, we also found that programmers ixated on social signals more than recalled.