A critique of “Algorithmic Extremism”

This post is a cri­tique and re­con­tex­tu­al­iza­tion of a re­cent pre-print pa­per by Mark Ledwich and Anna Zaitsev, “Algorithmic Extremism: Examining YouTube’s Rabbit Hold of Radicalization.”

First, an al­ter­na­tive ab­stract:

The role that YouTube and its be­hind-the-scenes rec­om­men­da­tion al­go­rithm plays in en­cour­ag­ing on­line rad­i­cal­iza­tion has been sug­gest­ed by both jour­nal­ists and aca­d­e­mics alike. This study quantifies the ex­tent to which this is true for users who are new to the plat­form and who do not have a view­ing his­to­ry. After cat­e­go­riz­ing near­ly 800 po­lit­i­cal chan­nels, we were able to differentiate be­tween po­lit­i­cal schemas in or­der to an­a­lyze the al­go­rithm traffic flows out and be­tween each group. We an­a­lyzed the rec­om­men­da­tions that an anony­mous user was pro­vid­ed when vis­it­ing each chan­nel type. We ob­served that YouTube’s rec­om­men­da­tion al­go­rithm ac­tive­ly dis­cour­ages new and anony­mous view­ers from vis­it­ing rad­i­cal­iz­ing or ex­trem­ist con­tent. Instead, in this ini­tial ex­plorato­ry phase, the rec­om­men­da­tion al­go­rithm fa­vors main­stream me­dia and ca­ble news con­tent over in­de­pen­dent YouTube chan­nels. Our study thus sug­gests that YouTube’s rec­om­men­da­tion al­go­rithm does not pro­mote inflammatory or rad­i­cal­ized con­tent to users who are new to the plat­form.

That is how I would have framed this pa­per.

Insights into the black box of any pro­pri­etary rec­om­men­da­tion al­go­rithm are hard to come by. Ledwich and Zaitsev pro­vide a valu­able con­tri­bu­tion. While YouTube has pub­li­cized the tech­ni­cal de­tails of their rec­om­men­da­tion al­go­rithms, they pub­lish scant data on the na­ture of the re­sult­ing rec­om­men­da­tions.

My cri­tique of this pa­per is that Ledwich and Zaitsev don’t ad­e­quate­ly confine their con­clu­sions to the ex­pe­ri­ence of a new user. They use lan­guage that im­plies that their ob­ser­va­tions like­ly gen­er­al­ize to the rec­om­men­da­tion al­go­rithm as a whole:

• “YouTube’s rec­om­men­da­tion al­go­rithm fails to pro­mote inflammatory or rad­i­cal­ized con­tent”
• “YouTube’s rec­om­men­da­tion al­go­rithm ac­tive­ly dis­cour­ages view­ers from vis­it­ing rad­i­cal­iz­ing or ex­trem­ist con­tent”
• “The data shows that YouTube does the ex­act op­po­site of the rad­i­cal­iza­tion claims”

They in­clude only one para­graph that ex­press­es a very lim­it­ed doubt about whether their ob­ser­va­tions would gen­er­al­ize to logged-in users (em­pha­sis mine):

One should note that the rec­om­men­da­tions list pro­vid­ed to a user who has an ac­count and who is logged into YouTube might differ from the list pre­sent­ed to this anony­mous ac­count. However, we do not be­lieve that there is a dras­tic difference in the be­hav­ior of the al­go­rithm. Our confidence in the sim­i­lar­i­ty is due to the de­scrip­tion of the al­go­rithm pro­vid­ed by the de­vel­op­ers of the YouTube al­go­rithm [38]. It would seem counter-in­tu­itive for YouTube to ap­ply vast­ly different cri­te­ria for anony­mous users and users who are logged into their ac­counts, es­pe­cial­ly con­sid­er­ing how com­plex cre­at­ing such a rec­om­men­da­tion al­go­rithm is in the first place.

That para­graph ex­treme­ly un­der­states the role that per­son­al­iza­tion plays in YouTube’s rec­om­men­da­tion al­go­rithm. I rec­og­nize that Ledwich and Zaitsev haven’t col­lect­ed the data need­ed to confirm them­selves that per­son­al­ized rec­om­men­da­tions are different than anony­mous rec­om­men­da­tion, so per­haps they are just be­ing care­ful. But the whole point of a rec­om­men­da­tion al­go­rithm is to tai­lor the rec­om­men­da­tions to the in­di­vid­ual. And fur­ther, the very de­scrip­tion of the al­go­rithm pro­vid­ed by YouTube (Recommending What Video to Watch Next) ex­plic­it­ly men­tions per­son­al­ized rec­om­men­da­tions that take into ac­count a user’s watch his­to­ry, de­mo­graph­ics, time, and lo­ca­tion.

It’s also prob­a­bly not cor­rect to re­fer to “the YouTube al­go­rithm” as hav­ing been de­scribed in an in­di­vid­ual pa­per. YouTube is con­stant­ly run­ning ex­per­i­ments. Some im­prove­ments are pub­lished. Some but not all of those end up in the pro­duc­tion al­go­rithm.

So, let’s look at an­oth­er pa­per from YouTube: Deep Neural Networks for YouTube Recommendations. This pa­per also ac­knowl­edges that a user’s ac­tiv­i­ty his­to­ry is rel­e­vant for rec­om­men­da­tions.

So again, Ledwich and Zaitsev have made some in­ter­est­ing and im­por­tant ob­ser­va­tions about the be­hav­iour of YouTube’s al­go­rithm dur­ing its ini­tial, ex­plorato­ry, un-per­son­al­ized phase for a new user. But these ob­ser­va­tions should not be tak­en to demon­strate any­thing about the be­hav­iour of the al­go­rithm (or the be­hav­iour of the al­go­rithm mixed with user re­spons­es to that al­go­rithm) as it learns more and more about you.