Lightnews — Scholar-powered news

David Berenstein @davidberenstein.bsky.social · Apr 10

🔥 Bespoke curator: Synthetic Data Curation for Post-Training & Structured Data Extraction

Create synthetic data pipelines with easy!
- Retries and caching included
- inference via LiteLLM, vLLM, and popular batch APIs
- asynchronous operations

🔗 URL: buff.ly/ajPRT1l

2

David Berenstein @davidberenstein.bsky.social · Apr 3

🔥One > token > at > a > time < a < at < token < One 🔥

token-explorer is a simple tool that lets you explore different possible paths that an LLM might sample!

- Arrow keys to navigate, pop and append tokens
- View the token probabilities and entropies.

GitHub: buff.ly/FQgsczM

2 11

David Berenstein @davidberenstein.bsky.social · Mar 7

🍽️ Let’s dissect the Synthetic Dataset Generator

💬 Natural language prompt to data

🦙 Ollama ensures secure local LLM inference

✍🏼 Argilla’s data curation capabilities complete the workflow

🔗 GitHub: buff.ly/5pX49Xc

GitHub - argilla-io/synthetic-data-generator: Build datasets using natural language

Build datasets using natural language. Contribute to argilla-io/synthetic-data-generator development by creating an account on GitHub.

buff.ly

5

David Berenstein @davidberenstein.bsky.social · Mar 5

🔥 Text2SQL, explore and share any data analysis!

🤗 Hugging Face - Dataset Studio is an amazing new feature.

🚀 Start yourself: buff.ly/pjpOKav

2

David Berenstein @davidberenstein.bsky.social · Mar 4

🔥 Vicinity: SEVEN semantic search BACK-ENDS, ONE single INTERFACE!

🫸 New release to push vector search to the Hub and work with any serialisable objects.

🧑‍🏫 KNN, HNSW, USEARCH, ANNOY, PYNNDESCENT, FAISS, and VOYAGER.

🔗 Library:

GitHub - MinishLab/vicinity: Lightweight Nearest Neighbors with Flexible Backends

Lightweight Nearest Neighbors with Flexible Backends - MinishLab/vicinity

buff.ly

3

David Berenstein @davidberenstein.bsky.social · Feb 27

🔥 NEW cool NO-CODE solution for clicking together AI WEB APPS!

🎨 Gradio released "gradio sketch"

🚼 Really easy way to create web apps with minimal code.

⚙️ Start with `pip install gradio` & `gradio sketch`

📒 Release: https://buff.ly/41aeLoA

David Berenstein @davidberenstein.bsky.social · Feb 27

Vector Search - let's keep it clean and lightweight! ⚡️

<100K records, no problem!
>100K, some scaling issues
ANN DuckDB index, sub-second response times

Notebook:

vector_search_with_hub_as_backend.ipynb

Run, share, and edit Python notebooks

buff.ly

2

David Berenstein @davidberenstein.bsky.social · Feb 25

🔥 The smolagents module has arrived in the agents course!

💻 Code agents optimised for software development
🔧 Tool calling agents that create modular, function-driven workflows
🔍 Retrieval agents designed to access and synthesise information

Course: https://buff.ly/4kcj6Ai

1 7

David Berenstein @davidberenstein.bsky.social · Feb 25

🧑‍🏫 Awesome. My talk for PyCon Italy 2025 got accepted!

Got data problems? Relax. Synthetic data is here to help.

Talk: https://buff.ly/3QzoZKj

2 3

David Berenstein @davidberenstein.bsky.social · Feb 21

🐳 Announcing docker support to Quickly set up your Synthetic Data Generator with (Gradio + Ollama + Argilla)!

🔥 Build genuinely useful datasets using natural language!

⚖️ Scale however you need.

🔐 Use them privately or share them with the world!

🧑‍💻 GitHub: https://buff.ly/49IDSmd

2 3

David Berenstein @davidberenstein.bsky.social · Feb 20

With 80K agent builders joining the agents course, it is time to make agents explorable on the Hub!

You can now search and find the perfect agents and tools for your needs!

Powered by @Gradio!

Start searching:

smolagents and tools gallery - a Hugging Face Space by davidberenstein1957

Discover amazing ML apps made by the community

buff.ly

1 3

David Berenstein @davidberenstein.bsky.social · Feb 19

Image Generation has landed in Arena form 🎨🤖!

1. Describe your desired image🎨
2. Two anonymous models output images
3. Vote for the winner!

Images have been sourced from our Open Image Preference dataset!
Dataset: https://buff.ly/4il0du9
Arena: https://buff.ly/4142NwH

2 1

David Berenstein @davidberenstein.bsky.social · Feb 18

Are you, the top of the Agents class?!

We just released a bonus unit on function calling (FC).

You will learn:
⑴ What is FC?
⑵ Thought → Act → Observe Cycle in FC
⑶ lightweight and efficient fine-tuning

Course: https://buff.ly/3Qn1DHB

7

David Berenstein @davidberenstein.bsky.social · Feb 14

📹 In case you've missed the hype around smolagents, here is a presentation I gave yesterday at an MLOps community event!

library: https://buff.ly/4hj6PrJ
slides: https://buff.ly/3WUzZ8D
video:

Smol Agents and Hugging Face - Anote AI Day Summit 2025

buff.ly

2 6

David Berenstein @davidberenstein.bsky.social · Feb 12

Slides for my MLOps community talk on smolagents!

Slides: https://buff.ly/3WUzZ8D

from bells and whistles to agents and tools

buff.ly

5

David Berenstein @davidberenstein.bsky.social · Feb 12

🚀 Find banger tools for your smolagents!

I created the Tools gallery, which makes tools specifically developed by/for smolagents searchable and visible. This will help with:
- inspiration
- best practices
- finding cool tools

Space: https://buff.ly/41cYctx

1

David Berenstein @davidberenstein.bsky.social · Feb 10

🔥 Come and get those AI agents certificates!

Join the cohort of 66K students: https://buff.ly/4hxb6rK

2

David Berenstein @davidberenstein.bsky.social · Feb 10

Documents or images to structured data using Vision Language Models

Outlines has an integration with transformers, which facilitates structured generation based on limiting token sampling probabilities.

Blog: https://buff.ly/4jFHMkr

2

David Berenstein @davidberenstein.bsky.social · Feb 10

Local docker deployments for the synthetic data generator 🫱🏾‍🫲🏼

We would love to hear your thoughts!

PR: https://buff.ly/4hRMny6

1

David Berenstein @davidberenstein.bsky.social · Feb 7

Spaces as tools: huggingface.co/spaces

Spaces - Hugging Face

Discover amazing ML apps made by the community

huggingface.co

David Berenstein @davidberenstein.bsky.social · Feb 7

Curious about "Why 🚀", you may wonder?

smolagents effortlessness combined with the power of 400,000 AI tools available on the Hub!

library: https://buff.ly/4hj6PrJ

1 5

David Berenstein @davidberenstein.bsky.social · Feb 6

WOW, this will rock the world! Hibiki is a model for simultaneous speech2speech translation.

And it actually works.

Available in French-English but super excited to see what the community will do.

Hub: https://buff.ly/3EtmM0f
Paper: https://buff.ly/4jIXNGd

2

David Berenstein @davidberenstein.bsky.social · Feb 6

Agentic RAG: Applied, visual, and step-by-step! 🐾

Get familiar with the Agents and tools, not the bells and whistles!

Retrieve - Augment and now GENERATE.

Parts:
1: https://buff.ly/40XNIxM
2: https://buff.ly/40HkB0x
3:

Agentic RAG Stack (3/5) - Generate responses using a SmolLM

A Blog post by David Berenstein on Hugging Face

buff.ly

David Berenstein @davidberenstein.bsky.social · Feb 6

🤯 Bring your own AI data, even if you have none!

Describe your dataset for RAG, LLMs or Text Classification
Bring your own context!
Press play and wait

Space: https://buff.ly/3Y1S99z
GitHub: https://buff.ly/49IDSmd

4

David Berenstein @davidberenstein.bsky.social · Feb 5

Anyone can create free hosted tools for their AI agents! 🔥

Agentic RAG stack part 2 - augment
Augment retrieval results by reranking optimises content without increasing time too much

part2: https://buff.ly/40HkB0x
part1: https://buff.ly/40XNIxM
code: https://buff.ly/4hEajpj

1