Lightnews — Scholar-powered news

David Gasquez

@davidgasquez.com

I've been using this pattern to "specialize" Codex for vaguely defined tasks like classification, filtering, soft sorting, ...

davidgasquez.com/specializing...

Made more than 10,000 invocations so far (reusing my ChatGPT subsciption) and am really happy with the pattern!

’ve been using Claude Code to take care of scrappy data cleaning tasks for a while. These days though, I’m using Codex as my coding agent. Similar to what I did with Claude, I’ve been “fine-tuning” Codex CLI to work on a few different vaguely defined tasks like classification, voting, filtering, or ranking.

The pattern in this post works surprisingly well when you have the following conditions:

Loosely defined open-ended tasks. e.g., tagging tweets with a set of predefined labels, extracting structured information from a GitHub issue, …
Powerful agentic capabilities. Doing the task requires something more than a simple llm call or PydanticAI script. e.g., using gh api CLI to get the number of stars of a repository.
Structured outputs. You need a response in a certain shape! This is something codex exec can do that claude couldn’t and is really powerful. e.g., return exactly True or False and nothing else.
Save money! Unlike llm or other tools/libraries that require an OPENAI_API_KEY, Codex can use your ChatGPT subscription, making things “free”.

October 22, 2025 at 6:38 PM

David Gasquez

@davidgasquez.com

Asked Claude to move my handbook notes from Obsdian to my Astro website. Very happy with the results!

davidgasquez.com/handbook/dat...

I'll keep the Obsidian Publish version around while I work on making it look as good.

Data Culture
The data team needs to be focus on delivering insights and supporting decisions. The outcome of the data team are decisions and a shared context across the organization that makes coordination easier.
Your goal as a data professional is to facilitate decision making and help surface/investigate the performance of a business (e.g. operational).
Learning to drive decisions quickly, a bias to action, is a critical competency for an analyst. Every skill you learn – communication, writing, experimentation, metric design – supports this.
If analysis is not actionable, it does not really matter. Analysis must drive to action. Clear results won’t spur action themselves. The organization needs to be ready to pivot when something isn’t working.
Data doesn’t make decisions, people do.
Data’s impact is tough to measure — it doesn’t always translate to value
The value of “insights” is often unknown.
The Data Team should be building and iterating the Data Product.
Notebooks are a workshop. Production systems are the factory. Not everything needs to be put into production. Not everything should be a notebook. You need both. Lean in to the strength of each.
Data is fundamentally a collaborative design process rather than a tool, an analysis, or even a product. Data works best when the entire feedback loop from idea to production is an iterative process.
To get buy in, explain how the business could benefit from better data (e.g: more and better insights). Start small and show value.
Run Purpose Meetings or Business Metrics Review.
Purpose Meetings are 30 min meetings in which stakeholders, engineers and data align on the goal of a release and what is the best way to evaluate the impact and understand its success. Align on the goal, commit on metrics and design the data.

July 23, 2025 at 10:23 AM

David Gasquez

@davidgasquez.com

The end dataset also serves as a much smoother API!

- No key required
- Batch export friendly
- Explorable in the browser

❯ curl -s "https://huggingface.co/datasets/datania/aemet/raw/main/valores-climatologicos/2025/07/16.json" | jq '.[] | select(.nombre == "GRANADA AEROPUERTO")'
{
"fecha": "2025-07-16",
"indicativo": "5530E",
"nombre": "GRANADA AEROPUERTO",
"provincia": "GRANADA",
"altitud": "560",
"tmed": "30,3",
"tmin": "18,8",
"horatmin": "05:08",
"tmax": "41,8",
"horatmax": "16:09",
"dir": "11",
"velmedia": "2,2",
"racha": "10,3",
"horaracha": "16:30",
"sol": "13,6",
"presMax": "955,3",
"horaPresMax": "Varias",
"presMin": "951,0",
"horaPresMin": "16",
"hrMedia": "29",
"hrMax": "60",
"horaHrMax": "Varias",
"hrMin": "13",
"horaHrMin": "16:02"
}
~

July 21, 2025 at 1:58 PM

David Gasquez

@davidgasquez.com

Updated linkweaver (custom notes & research helper) with a couple of new tricks.

1. Saves the URL content locally
2. Accepts any command to preprocess the generated Markdown. E.g: `llm "Clean the following Markdown"`.

github.com/davidgasquez...

Want to try it? `uvx linkweaver` and enjoy! 😀

Screenshot of linkweaver running. It is downloading a few urls

❯ uvx linkweaver --help
usage: linkweaver [-h] [--list-links] [--xml-cat] [-x COMMAND] [-v] [--retries N] [-q] [--dry-run] [--no-color] [-f] [input_files ...]

Extract URLs from markdown files and save each link as individual markdown files in resource folders.
By default, existing files are skipped to save time and bandwidth.

positional arguments:
input_files One or more markdown files to process

options:
-h, --help show this help message and exit
--list-links, -l List all unique URLs found in the files (no downloading)
--xml-cat Concatenate files with their resource folders in XML structure
-x, --exec COMMAND Execute a shell command on the markdown output before saving (e.g., 'llm -t mdclean')
-v, --verbose Enable verbose output with detailed progress information
--retries N Number of retry attempts for failed URL fetches (default: 3)
-q, --quiet Minimize output (opposite of verbose, but still show errors/warnings)
--dry-run Show what would be done without actually doing it
--no-color Disable colored output
-f, --force Force redownload of files even if they already exist

Examples:
linkweaver notes.md # Process file, skip existing resources
linkweaver --force notes.md # Force redownload all resources
linkweaver --list-links notes.md # List all URLs found
linkweaver --xml-cat notes.md # Concatenate with resources
linkweaver -x 'llm -t clean' notes.md # Process with custom command
linkweaver --retries 5 notes.md # Use 5 retry attempts for failed URLs
linkweaver --quiet --dry-run notes.md # Preview actions quietly
linkweaver -q -v notes.md # Quiet mode overrides verbose
linkweaver --no-color notes.md # Disable colored output

Common workflows:
linkweaver --list-links *.md | head -10 # Preview first 10 URLs

July 16, 2025 at 3:18 PM

David Gasquez

@davidgasquez.com

Now making the agents in the tool chose the final name of the tool using the tool itself. 😜

github.com/davidgasquez...

July 7, 2025 at 10:01 AM

David Gasquez

@davidgasquez.com

Finally had some time to publish a vibecoded tool I (and Claude Code) built to explore PydanticAI.

Replicates a contest where jurors evaluate candidates in pairwise comparisons that get turned into a leaderboard.

github.com/davidgasquez...

a contest...

❯ uv run examples/energy.py
2025-07-05 17:15:20 - arbitron.agent - INFO - Initialized agent safety_specialist
2025-07-05 17:15:20 - arbitron.agent - INFO - Initialized agent environmental_specialist
2025-07-05 17:15:20 - arbitron.agent - INFO - Initialized agent economic_specialist
2025-07-05 17:15:20 - arbitron.contest - INFO - Starting competition 'Arbitron Competition' with 5 items and 3 agents
2025-07-05 17:15:20 - arbitron.contest - INFO - Agent safety_specialist will perform 10 comparisons
2025-07-05 17:15:21 - arbitron.agent - INFO - Agent safety_specialist chose nuclear_fission over wind_turbines
2025-07-05 17:15:23 - arbitron.agent - INFO - Agent safety_specialist environmental_specialist chose wind_turbines over nuclear_fission
2025-07-05 17:15:40 - arbitron.agent - INFO - Agent environmental_specialist chose nuclear_fission over hydroelectric_dams
2025-07-05 17:15:41 - arbitron.agent - INFO - Agent environmental_specialist chose geothermal_power over nuclear_fission
2025-07-05 17:15:43 - arbitron.agent - INFO - Agent environmental_specialist chose geothermal_power over hydroelectric_dams
2025-07-05 17:15:44 - arbitron.agent - INFO - Agent environmental_specialist chose solar_photovoltaic over nuclear_fission
2025-07-05 17:15:46 - arbitron.agent - INFO - Agent environmental_specialist chose wind_turbines over geothermal_power
And you get a nice ranking!

1. solar_photovoltaic (score: 1.227)
2. wind_turbines (score: 1.127)
3. nuclear_fission (score: 1.127)
4. hydroelectric_dams (score: 0.801)
5. geothermal_power (score: 0.801)

July 5, 2025 at 3:34 PM

David Gasquez

@davidgasquez.com

Spent some time going through @cameron.pfiffer.org 's Comind project and @void.comind.network public code and had a lot of fun learning about them!

- comind.stream
- tangled.sh/@cameron.pfi...

Love Comind's components idea! Clean and simple.

Core Components
Blips: Atomic units of information formatted as ATProto records, such as thoughts, concepts, and emotions
Links: Typed relations between blips with semantic context
Cominds: Specialized AI agents that process network activity
Spheres: Workspaces defined by core directives that focus agent cognition
Melds: Interaction protocol for accessing sphere knowledge

July 4, 2025 at 9:24 AM

David Gasquez

@davidgasquez.com

Welcome to the dark side! We have the AUR and, eventually, a few interesting system updates! 😅

This is how my current setup (github.com/davidgasquez...) looks like after 9 years of tinkering.

A screenshot with 3 windows. VSCode, btop, and Claude Code

July 2, 2025 at 10:39 AM

David Gasquez

@davidgasquez.com

I'm also publishing a Parquet file joining all the datasets metadata. Use it from your favorite tool! 💃

sql-workbench.com#queries=v0,S...

June 16, 2025 at 7:08 PM

David Gasquez

@davidgasquez.com

Alright, this is a job for a "research" template + fragments + chain.

Is the `llm prompt` doing the same thing as model.chain() in Python?

$import httpx import llm def read_url(url: str) -> str: """ Use Jina Reader to convert a URL to Markdown text. Example usage: llm -f 'reader:https://simonwillison.net/tags/jina/' ... """ url = "https://r.jina.ai/" + url response = httpx.get(url) if response.status_code != 200: raise ValueError(f"Failed to load fragment from {url}: {response.status_code}") return response.text source_note = llm.Fragment( content=""" # Documentation - If your product/tool/process documentation is not good enough, people will not use it. - Documentation changes should be reviewed in the same way as your code. - [If someone's having to read your docs, it's not "simple"](https://justsimply.dev/). Remove filler words. - [Principles to keep in mind when writing documentation](https://mkaz.blog/misc/notes-on-technical-writing/). - The purpose of technical writing is to help users accomplish tasks as quickly and effectively as possible. """ ) model = llm.get_model("gpt-4.1-mini") chain_response = model.chain( """ Where is "You’re keen to share your excitement" mentioned in the sources? """, tools=[read_url], # after_call=print, fragments=[source_note], system="You are a helpful research assistant. ALWAYS read ALL the links with `read_url`. Once you've readed all the links in the note, help the user with their questions.", ) for chunk in chain_response: print(chunk, end="", flush=True)$

June 2, 2025 at 3:17 PM

David Gasquez

@davidgasquez.com

Ended up writting a small CLI that you can point to a Markdown note and get an "expanded" view into it.

It reads all the links, makes them Markdown and returns all of them under different XML tags for your favorite LLM to consume!

github.com/davidgasquez...

❯ lw Emergence.md | llm "Summary on one sentence"
Emergence describes the unpredictable, complex outcomes that arise from simple interactions, which, while not top-down plannable, can be provoked by designing systems with adaptable building blocks, multiple pathways to goals, a low barrier to entry, and multi-agent participation.

June 2, 2025 at 2:44 PM

David Gasquez

@davidgasquez.com

Wrote a post about some things I've been doing to make my projects more LLM friendly. Spoiler alert: it makes the projects more human friendly too!

davidgasquez.com/llm-friendly...

Any other ideas or suggestions?

Giving them proper context.
Using simple and specific prompts.
Describing small problems with clarity and boundaries.
Making things easy to run and test (Makefile, Docker, …).
Using well-known frameworks that appear in their training data.
With these basic ideas in mind, let’s see what we can do to make the most out of the current LLMs’ capabilities.

Since I’ve been doing Machine Learning projects recently (Kaggle-style competitions), I’ve developed a few extra things I do 1 on those projects.

Document the features you’re currently using and potential features you’d like to explore.
Keep track of open questions and EDA tasks you’d like to explore.
Maintain a DATA_DICTIONARY.md or similar with the terms you’re using. Context is king.
Rely on a functional pipeline style and modular feature engineering. Use “micro pipeline” scripts to generate features. If your dataset has granularity at the “user” level, create different scripts for different kinds of features that generate the feature_name.csv files. You can join them all later. This is one of the things I use a lot. Almost all the features I’ve added to my projects start like “Write a script that computes the user’s […]. Save as feature_name.csv with user, feature_name as headers.”
Track experiments. I’m bad at this one as I haven’t figured out a simple enough way to do this reliably. Ideally, keep track of input features, model parameters, preprocessing, postprocessing, local evaluation, and remote evaluation.
Have a temporary folder (listed in .gitignore) for the LLM to write small scripts and experiments. Here is where you’d have things like inspect_csv.py or other useful scripts that can give the LLMs context on the actual data (e.g., check columns, stats, …)

May 24, 2025 at 6:00 PM

David Gasquez

@davidgasquez.com

It is! Found this plugin that does exactly that.

github.com/jmdaly/llm-g...

Found it with this GitHub query:

github.com/search?q=%22...

Seems they're adding a custom prompt though!

❯ llm -m github-copilot/claude-sonnet-4 "Hello :)"
Hello! 👋 Great to meet you! I'm GitHub Copilot, and I'm here to help you with any programming questions, code reviews, debugging, or development tasks you might have.

Whether you're working on:
- Writing new code
- Debugging existing code
- Learning a new programming language
- Discussing software architecture
- Code optimization
- Or anything else development-related

I'm ready to assist! What are you working on today?

May 24, 2025 at 4:13 PM

David Gasquez

@davidgasquez.com

Some INE tables are large!

If you want to explore the "Travels, overnight stays and average stay by main features of the trips" dataset, you'd have to download a 37GB CSV!

The compressed Parquet file is less than 10MB and can be queried from your browser! 💃

shell.duckdb.org#queries=v0,s...

DuckDB WASM shell.

duckdb> select count(*) from 'https://huggingface.co/datasets/davidgasquez/ine/resolve/main/tablas/15797/datos.parquet';

March 20, 2025 at 12:56 PM

David Gasquez

@davidgasquez.com

Exactly! Will try as soon as it's available. 😀

Xet is an experimental feature that allows you to store large files efficiently. Xet is currently disabled.

March 18, 2025 at 4:57 PM

David Gasquez

@davidgasquez.com

Small update! Learned some things about INE's servers.

- Don't really support Range Requests.
- Don't officially support gzip Accept-Encodings, but they do. Shrinking the download sizes ~8 times.

That means we can loop through the compressed datasets and save them as Parquet files on GH Actions!

$This code. import duckdb c = duckdb.connect() c.sql(""" CREATE SECRET http (TYPE http, EXTRA_HTTP_HEADERS MAP {'Accept-Encoding': 'gzip'}); """) failed_tables = [] total_tables = len(tablas) for i, table in enumerate(tablas): # Si el directorio no existe, lo creamos table_dir = ine_dir / "tablas" / str(table["Id"]) table_dir.mkdir(exist_ok=True, parents=True) # Copiamos el CSV a Parquet try: c.sql(f""" copy ( from read_csv( 'https://www.ine.es/jaxiT3/files/t/en/csv_bdsc/{table["Id"]}.csv', delim=';', ignore_errors=true, normalize_names=true, null_padding=true, parallel=true, strict_mode=false, compression='gzip' ) ) to 'ine/tablas/{table["Id"]}/datos.parquet' ( format parquet, compression 'zstd', parquet_version v2, row_group_size 1048576 ); """) except Exception as e: print(f"Error processing table {table['Id']}: {str(e)}") failed_tables.append(table["Id"])$

March 18, 2025 at 10:25 AM

David Gasquez

@davidgasquez.com

Not sure why, it opened for a brief second and then went white.

The URL still is sql-workbench.com#config={%22p... so it doesn't seems to be truncating anything.

White screen on Brave with the Network tab open

White screen on Brave with the Console tab open. A couple of errors appear. They are syntax errors.

March 18, 2025 at 7:54 AM

David Gasquez

@davidgasquez.com

Wrote a post after spending one week using @zed.dev.

davidgasquez.com/trying-zed-e...

TLDR: Would love to switch once they work on a few extra features (Notebooks, Agent Mode, ...)

February 21, 2025 at 1:03 PM

David Gasquez

@davidgasquez.com

What a great definition of Data Engineering! 😅

The only thing I'd tweak is removing the "large amounts of data" requirement. Moving small/medium sized dataset can be as difficult/interesting!

ludic.mataroa.blog/blog/brainwa...

Screenshot of a section of the text with the following content hightlighted.

my specialty is building systems that move large amounts of data through companies, organize them in a way that is at least marginally less of a horrific clusterfuck than what random people without specific training will do when left to their own devices, and sometimes assist with statistics

February 13, 2025 at 8:58 AM

David Gasquez

@davidgasquez.com

A Python browser sandbox, by the @pydantic.dev folks!

github.com/pydantic/pyd...

Great to share small snippets of code (e.g: pydantic.run/store/c544b1...).

pydantic.run homepage wiht a simple polars example

January 30, 2025 at 10:43 AM

David Gasquez

@davidgasquez.com

Been binging that channel a lot lately! It is close to what I was looking for: ML stuff with modern tooling and interesting approaches!

January 29, 2025 at 2:57 PM

David Gasquez

@davidgasquez.com

Created a small repository as a personal template for Datathons / ML Competitions.

Contains a bunch of interesting resources! 📚

github.com/davidgasquez...