Lightnews — Scholar-powered news

Reposted by Mark Dredze

JHU Computer Science @jhucompsci.bsky.social · 29d

Congratulations to CS faculty @mdredze.bsky.social, Jason Eisner, Peter Kazanzides, and @tom-lippincott.bsky.social
on their @jhu.edu Nexus Awards! Learn more about their funded projects here: www.cs.jhu.edu/news/compute...

Headshots of Mark Dredze, Jason Eisner, Peter Kazanzides, and Tom Lippincott.

1 4

Reposted by Mark Dredze

William Jurayj @williamjurayj.bsky.social · Feb 20

🚨 You are only evaluating a slice of your test-time scaling model's performance! 🚨

📈 We consider how models’ confidence in their answers changes as test-time compute increases. Reasoning longer helps models answer more confidently!

📝: arxiv.org/abs/2502.13962

1 10 12

Mark Dredze @mdredze.bsky.social · Feb 19

Please read and share this excellent FAQ on University indirect costs by my friend @broniatowski.bsky.social

He explains why these funds are essential and a critical investment for research in the United States.

www.linkedin.com/posts/david-...

David Broniatowski on LinkedIn: NIH has just announced that it will save $4 billion by capping university… | 61 comments

NIH has just announced that it will save $4 billion by capping university indirect costs on federal grants. Will this actually save money? Bottom line: No. It… | 61 comments on LinkedIn

www.linkedin.com

3 3

Mark Dredze @mdredze.bsky.social · Feb 5

I know I can improve my ARR reviews, but there really is no need for name calling. 😁

1 18

Mark Dredze @mdredze.bsky.social · Jan 30

Helpful
Insightful
Probing
Valuable
Thoughtful
Illuminating
Constructive

In author feedback, these are synonyms for "we hate your review."

1

Mark Dredze @mdredze.bsky.social · Jan 27

Do reviewers purposely write confusing reviews with typos to demonstrate that the review wasn't written by a LLM?

2

Mark Dredze @mdredze.bsky.social · Jan 24

Golden idea for an NLP paper: a group of llamas is called a "cria herd".

That would make a great name for a LLM method, model, or paper.

Just remember to acknowledge me in your paper.

You're welcome.

9

Mark Dredze @mdredze.bsky.social · Jan 21

Idea for GenAI app: rewrite click bait headlines to normal headlines in the browser.

Input: you’ll never guess this one company organizing the best deals of the year

Output: Amazon has a modest sale on phone chargers

2 2 23

Mark Dredze @mdredze.bsky.social · Jan 20

Good idea!

Mark Dredze @mdredze.bsky.social · Jan 20

The ARR submission checklist is already pretty extensive, but I suggest we add an additional question:

"I certify that I know the difference between \citet and \citep."

1 1 22

Mark Dredze @mdredze.bsky.social · Jan 20

ARR: Reviews are due today.

Me:

1

Mark Dredze @mdredze.bsky.social · Jan 17

I feel seen. This is why I always access my API keys from my laptop.

1 10

Mark Dredze @mdredze.bsky.social · Jan 14

Do you have any of those fortune cookies that mock academics?

Sure!

1 6

Mark Dredze @mdredze.bsky.social · Jan 2

Starting a new year and reflecting on how lucky I am to work at @hopkinsengineer.bsky.social with amazing people @jhucompsci.bsky.social @jhuclsp.bsky.social.

I was promoted to full professor in 2023, and my students presented me with this amazing poster of current and former PhD students.

2 12

Mark Dredze @mdredze.bsky.social · Jan 2

Listen to @karaswisher.bsky.social's new podcast where she interviews @ruchowdh.bsky.social, @ghadfield.bsky.social and me about AI Ethics and Safety. The podcast was recorded before a live audience at @jhu.edu Bloomberg Center.

podcasts.apple.com/us/podcast/a...

AI Ethics and Safety — A Contradiction in Terms?

Podcast Episode · On with Kara Swisher · 01/02/2025 · 53m

podcasts.apple.com

2

Mark Dredze @mdredze.bsky.social · Dec 22

Examining the generated QA pairs, you can really see the difference. Our generations (bottom) look harder and more interesting.

Try our strategy for your synthetic generation task? Check out our paper, being presented at #ML4H2024 .
arxiv.org/abs/2412.04573

2

Mark Dredze @mdredze.bsky.social · Dec 22

Training a Clinical QA system on our data gives big improvements, whether we generate data from Llama or GPT-4o. These improvements are both in F1 and any overlap between the extracted and true answers.

1 1

Mark Dredze @mdredze.bsky.social · Dec 22

The generated pair has a lot of advantages: it doesn't use the same language as the report, it includes harder questions, and the answers are sometimes not in the report (unanswerable questions.) The result? Harder, more diverse and more realistic QA pairs.

1 1

Mark Dredze @mdredze.bsky.social · Dec 22

Second, we use a summarize-then-generate strategy. The LLM first summarizes a given clinical record in a structured format. The summary keeps the key points but loses the details, such as specific terminology and content. We then use the summary to generate a new QA pair.

1

Mark Dredze @mdredze.bsky.social · Dec 22

We explore two strategies. First, we craft instructions to encourage QA diversity. We formulate these as constraints on the answers to the questions. It helps, but we need more.

1

Mark Dredze @mdredze.bsky.social · Dec 22

We can ask an LLM to write QA pairs, but they turn out to be too easy and repetitive. They don't come close to what you can get with real data. We need more diverse data! Typical methods (e.g. annealing) don't work. What can we do?

1

Mark Dredze @mdredze.bsky.social · Dec 22

Paper at #ML42024!

Clinical QA can help doctors find critical information in patient records. But where do we get training data for these systems? Generating this data from an LLM is hard. 🧵

1 3

Mark Dredze @mdredze.bsky.social · Dec 22

Takeaways: If you can fine-tune a model on a specific clinical domain, that's great. If you can't, you should probably use models that are better overall, even if they aren't trained on clinical data.

Many more details in the paper!
arxiv.org/abs/2412.05845

Are Clinical T5 Models Better for Clinical Text?

Large language models with a transformer-based encoder/decoder architecture, such as T5, have become standard platforms for supervised tasks. To bring these technologies to the clinical domain, recent...

arxiv.org

1 3

Mark Dredze @mdredze.bsky.social · Dec 22

It turns out that when you have just a little supervised data, the models trained on more data and tasks, even when out of domain, do BETTER on the new clinical domain.

1

Mark Dredze @mdredze.bsky.social · Dec 22

Maybe the real advantage for domain-tuned models lies in the low resource setting. With lots of supervised data, an out of domain model can do well. What about with just a few training examples?

1