Lightnews — Scholar-powered news

Tiago Ventura @tiagoventura.bsky.social · 13h

Thank YOU for the opportunity to work together on this, Jonathan!

Reposted by Tiago Ventura

Jonathan Nagler @jonathannagler.bsky.social · 14h

Proud to have worked on this paper with @tiagoventura.bsky.social, Bernhard Clemen von Hohenberg, Ericka Menchen-Trevino, and Mabdalena Wojcieszak: www.cambridge.org/core/journal... .

Survey Professionalism: New Evidence from Web Browsing Data | Political Analysis | Cambridge Core

Survey Professionalism: New Evidence from Web Browsing Data

www.cambridge.org

1 2 2

Tiago Ventura @tiagoventura.bsky.social · 2d

Yeah.. absolutely. We make our efforts to control for this by imposing a long break between repeated attempts (30min if I’m not mistaken). But substantive results don’t change much with longer break like 1h or even 24h

1

Tiago Ventura @tiagoventura.bsky.social · 2d

We use URLs from platforms we can identify a unique survey. So not all platforms we have in the paper and def not proprietary platforms. So think for example about qualtrics. In this case, a survey taker can try to take it more than once with the url, switching browsers, switching profiles, etc..

1

Tiago Ventura @tiagoventura.bsky.social · 2d

Me too! As someone who does most of my research outside of the U.S. I’m also very curious about this phenomenon in other markets! Hopefully our paper can help in the measurement challenges of survey taking for future research!

1

Tiago Ventura @tiagoventura.bsky.social · 2d

Yeah… that’s a great point we tried to be as thoughtful as possible! We provide a sensitivity analysis for this case, and show how our prevalence estimates would change. We discuss this limitation in detail in the paper!

Tiago Ventura @tiagoventura.bsky.social · 2d

Try to! We really do not know if they were able to complete the survey page they were visiting

1

Tiago Ventura @tiagoventura.bsky.social · 3d

Thank you, Amanda!

Tiago Ventura @tiagoventura.bsky.social · 3d

ohh very cool! Can you share the paper? I would love to read it!

Tiago Ventura @tiagoventura.bsky.social · 3d

The paper is co-authored with Bernhard Von Clemm, @ericka.bric.digital
@jonathannagler.bsky.social @magdalenawojciesza.bsky.social

This is also one of the projects I started at @csmapnyu.org ---- thanks to the entire lab involved!

The paper can be found here: www.cambridge.org/core/journal...

Survey Professionalism: New Evidence from Web Browsing Data | Political Analysis | Cambridge Core

Survey Professionalism: New Evidence from Web Browsing Data

www.cambridge.org

1 6

Tiago Ventura @tiagoventura.bsky.social · 3d

Huge caveat: our data comes from a period before the widespread use of Generative AI.

How professionals are using these tools, and how easy access to GenAI models might affect the quality of survey research, remains an open question for future research.

1 5

Tiago Ventura @tiagoventura.bsky.social · 3d

Our conclusion is that survey professionalism is widespread on online panels, but these do not, by and large, distort inferences, making us cautiously optimistic about research using these online samples!

1 7

Tiago Ventura @tiagoventura.bsky.social · 3d

But one concerning pattern emerges:

Professionals are considerably more likely to attempt to take the same survey repeatedly.

For example, roughly 85 % of Lucid professionals tried to take at least one survey more than once

2 6

Tiago Ventura @tiagoventura.bsky.social · 3d

c) When looking at the stability of responses over time, we find no evidence of professionals being less attentive or more likely to change their responses across survey waves.

We take this as evidence that they answer questions at least as attentively as non-professionals.

1 2

Tiago Ventura @tiagoventura.bsky.social · 3d

Our results show:

a) No robust demographic or political differences between professionals and non-professionals.

b)Professionals are more likely to speed and attempt repeats, but those are easy to screen out in surveys.

And..

1 4

Tiago Ventura @tiagoventura.bsky.social · 3d

Next, we ask: Do professionals risk survey quality and inference from survey research? We test this in three dimensions:

a) demographic & political differences;
b) low-effort behavior (speeding, straightlining),
c) survey stability across waves

1 2

Tiago Ventura @tiagoventura.bsky.social · 3d

Prevalence varies sharply across platforms. Our most conservative estimate shows:

- 1.7 % of Facebook respondents are professionals
- 7.6 % on YouGov
- 34.7 % on Lucid

Professionalism is a real phenomenon, but it varies widely across samples!

1 1 3

Tiago Ventura @tiagoventura.bsky.social · 3d

We analyze 3,886 respondents who donated their browsing histories (~96M web visits). We measure actual survey-taking, rather than relying on self-reports, as much of prior work on this topic does.

We focus on three outcomes: prevalence, data quality, and survey repetition.

1 3

Tiago Ventura @tiagoventura.bsky.social · 3d

How common are “survey professionals” - people who take dozens of online surveys for pay - across online panels, and do they harm data quality?

Our paper, FirstView at @politicalanalysis.bsky.social, tackles this question using browsing data from three U.S. samples (Facebook, YouGov, and Lucid):

4 52 130

Reposted by Tiago Ventura

Cambridge University Press Political Science & IR @cambup-polsci.cambridge.org · 3d

#OpenAccess from @polanalysis.bsky.social -

Survey Professionalism: New Evidence from Web Browsing Data - https://cup.org/3KWgqtg

- Bernhard Clemm von Hohenberg, @tiagoventura.bsky.social, Jonathan Nagler, @ericka.bric.digital & Magdalena Wojcieszak

#FirstView

Image featuring a yellow and red banner with the text "POLITICAL ANALYSIS" in large white letters. Below in a smaller font, it reads "#OpenAccess.

2 2

Tiago Ventura @tiagoventura.bsky.social · 23d

I use the YouTube API, but with someone’s else wrapper. To actually show how to query an api, I go with nyt. It is simple and easy to use.

2

Reposted by Tiago Ventura

NYU's Center for Social Media and Politics @csmapnyu.org · Jul 17

In the Global South, WhatsApp is more popular than X or Facebook.

New in @The_JOP, we ran a WhatsApp deactivation experiment during Brazil’s 2022 election to explore how the app facilitates the spread of misinformation and affects voters’ attitudes.

www.journals.uchicago.edu/doi/abs/10.1...

Abstract: In most advanced democracies, concerns about the spread of misinformation are typically associated with feed-based social media platforms like Twitter and Facebook. These platforms also account for the vast majority of research on the topic. However, in most of the world, particularly in Global South countries, misinformation often reaches citizens through social media messaging apps, particularly WhatsApp. To fill the resulting gap in the literature, we conducted a multimedia deactivation experiment to test the impact of reducing exposure to potential sources of misinformation on WhatsApp during the weeks leading up to the 2022 Presidential election in Brazil. We find that this intervention significantly reduced participants’ recall of false rumors circulating widely during the election. However, consistent with theories of mass media minimal effects, a short-term change in the information environment did not lead to significant changes in belief accuracy, political polarization, or well-being.

1 14 31

Tiago Ventura @tiagoventura.bsky.social · Jul 17

And here for a more general summary of the paper: csmapnyu.org/research/aca...

Misinformation Beyond Traditional Feeds: Evidence from a WhatsApp Deactivation Experiment in Brazil - NYU’s Center for Social Media and Politics

csmapnyu.org

Tiago Ventura @tiagoventura.bsky.social · Jul 17

Here the link for the JOP version: www.journals.uchicago.edu/doi/epdf/10....

University of Chicago Press Journals: Cookie absent

www.journals.uchicago.edu

1

Tiago Ventura @tiagoventura.bsky.social · Jul 17

🚨 Paper now as "just accepted" at @The_JOP. We ran the first WhatsApp deactivation experiment focused on multimedia content ahead of the 2022 election in Brazil. We find a reduction in users' recall of false rumors -- and, to a smaller degree, of true news. Null effects on attitudes. Full thread ⬇️

NYU's Center for Social Media and Politics @csmapnyu.org · Jul 17

In the Global South, WhatsApp is more popular than X or Facebook.

New in @The_JOP, we ran a WhatsApp deactivation experiment during Brazil’s 2022 election to explore how the app facilitates the spread of misinformation and affects voters’ attitudes.

www.journals.uchicago.edu/doi/abs/10.1...

1 4 25