This month, LinkedIn researchers revealed in Science that the company had spent five years quietly researching more than 20 million users. By adjusting the professional networking platform’s algorithm, the researchers were trying to determine through A/B testing whether users ended up with more job opportunities when they connected with known acquaintances or complete strangers.
To weigh the strength of connections between users as weak or strong, acquaintance or stranger, the researchers analyzed factors such as the number of messages they sent to each other or the number of mutual friends they shared, assessing how these factors have changed over time after logging on the social media platform. The researchers’ finding confirmed what they describe in the study as “one of the most influential social theories of the last century” on job mobility: the weaker the ties between users, the better the job mobility. While LinkedIn says these results will lead to changes in the algorithm to recommend more relevant connections to job seekers as “People You May Know” (PYMK) move forward, the New York Times reported that ethics experts said the study “raises questions about industry transparency.” and follow-up research.
One of the main concerns of the experts was that none of the millions of users analyzed by LinkedIn were directly informed that they were taking part in the study, which “could have affected the livelihoods of some people”, according to the NYT report.
Michael Zimmer, associate professor of computer science and director of the Center for Data, Ethics, and Society at Marquette University, told the NYT that “the results suggest that some users had better access to job opportunities or a significant difference in access to employment opportunities.”
LinkedIn clarifies A/B testing issues
A LinkedIn spokesperson told Ars that the company disputes this characterization of their research, saying no one was disadvantaged by the experiments. Since the NYT released its report, the LinkedIn spokesperson told Ars that the company has responded to questions due to “many inaccurate representations of the methodology” of its study.
Study co-author and LinkedIn data scientist Karthik Rajkumar told Ars that reports like the NYT report confuse “A/B testing and the observational nature of the data,” which “sounds like more to experimentation on people, which is incorrect”.
Rajkumar said the study was done because LinkedIn had noticed that the algorithm was already recommending more connections with weaker links with some users and more with stronger links with others. “Our A/B testing of PYMK was intended to improve the relevance of connection recommendations, not to study business results,” Rajkumar told Ars. Instead, his team’s goal was to find out “which relationships are most important to getting and securing jobs.”
Although it’s called “A/B testing”, suggesting it compares two options, the researchers didn’t just look at weak links versus strong links, exclusively testing a pair of algorithms that generated the either. Instead, the study experimented with seven different “processing variants” of the algorithm, noting that different variants yielded different results, such as users forming fewer weak links, building more links, building fewer links or creating the same number of weak or strong links. . Two variants, for example, caused users to form more links overall, including more weak links, while another variant caused users to form fewer links overall, including fewer weak links. A variant led to more links, but only strong links.
“We don’t randomly vary the proportion of weak and strong contacts suggested by PYMK,” a LinkedIn spokesperson told Ars. “We strive to make better recommendations to people, and some algorithms recommend more weak links than others. Because some people end up getting the best algorithms a week or two earlier than others during the testing period. , this creates enough variation in the data for us to apply the causal methods of observation to analyze it.No one is experienced in observing employment outcomes.