Overall, the community has become more articulate about validity but still lacks shared ground on how empirical work should be designed, evaluated, and replicated. We hope this study helps strengthen discussions around empirical standards and reviewing practices.

💡 Replications are valued but rarely accepted.💡
There is broad support for more replications, yet little agreement on what makes them publishable. Some responses also reflect the perception that failed replications mean “no effect” and that they can be seen as confrontational.

💡 External validity is still widely favored, often blurred with ecological validity.💡
Many reviewers link external validity to realism and industrial relevance, even though these are distinct concepts. This conflation continues to influence expectations during review.

💡 Awareness has grown, but concerns about how methods are judged remain.💡
Extreme positions have disappeared, but studies using qualitative methods are still often evaluated with criteria that do not fit their methodology.

We revisited a cornerstone study from ten years ago. Back then, views on internal vs external validity were highly fragmented, leading to uneven and unpredictable reviews.
We asked today’s key contributors the same questions again.

➡︎ The paper?

sws.informatik.uni-leipzig.de/wp-content/u...

Thanks to our amazing participants
TU-Chemnitz.de ♥️, all the support from ScaDS.AI ♥️, and the amazing author team ♥️
(5/5)

➡︎ The conclusion?

✔️ Use chatbots to debug & test code
✔️ Teach prompt engineering early
✔️ Watch for prompts about basic concepts—these can signal struggling learners.

💡Chatbots can be great tools for beginners IFF used right!
(4/5)

➡︎ The challenge?

Beginners struggle with:
✘ Writing effective prompts
✘ Understanding what chatbots need to generate useful results

💡Let's teach working with LLMs and prompt engineering alongside coding skills.
(3/5)

➡︎ How do beginners use chatbots?

50% of prompts focus on code generation, but missing context often leads to poor results.

💡 Prompts used for debugging & testing code were game-changers: They boosted solution quality by 72%!
(2/5)