David Castillo-Bolado
@dcasbol.bsky.social
CS - PhD at SIANI-ULPGC. Researching LTM at GoodAI. LLM whisperer. #DeepLearning #AI #ALife #Memetics
También pongo boberías en español
También pongo boberías en español
Okay... this is interesting. I've found (probably rediscovered) a simple contrastive loss for reidentificación which has a trivial solution when the network outputs a constant embedding, but it doesn't seem to converge towards that!
The loss is:
max(0, |a_1 - a_2| - 0.9 * |a_1 - b|)
The loss is:
max(0, |a_1 - a_2| - 0.9 * |a_1 - b|)
September 1, 2025 at 3:02 PM
Okay... this is interesting. I've found (probably rediscovered) a simple contrastive loss for reidentificación which has a trivial solution when the network outputs a constant embedding, but it doesn't seem to converge towards that!
The loss is:
max(0, |a_1 - a_2| - 0.9 * |a_1 - b|)
The loss is:
max(0, |a_1 - a_2| - 0.9 * |a_1 - b|)
Everyone's using Adam or AdamW as optimizer, but I'm using ASGD and damn this thing is consistent! Talk me about monotinical decrease!
July 10, 2025 at 5:41 PM
Everyone's using Adam or AdamW as optimizer, but I'm using ASGD and damn this thing is consistent! Talk me about monotinical decrease!
Loss plots are fun... until it's your turn and it's not fun anymore!
July 8, 2025 at 11:27 AM
Loss plots are fun... until it's your turn and it's not fun anymore!
At this point I just get excited when there is no mention of LLMs in the abstract.
May 21, 2025 at 1:03 PM
At this point I just get excited when there is no mention of LLMs in the abstract.
La #HoraDelPlaneta fue el viernes pasado y no se le hizo mucho caso. No sé, yo lo veo claro... #apagon
es.wikipedia.org/wiki/Hora_de...
es.wikipedia.org/wiki/Hora_de...
Hora del Planeta - Wikipedia, la enciclopedia libre
es.wikipedia.org
April 28, 2025 at 2:58 PM
La #HoraDelPlaneta fue el viernes pasado y no se le hizo mucho caso. No sé, yo lo veo claro... #apagon
es.wikipedia.org/wiki/Hora_de...
es.wikipedia.org/wiki/Hora_de...
#apagon en toda península dicen, y yo no dejo de pensar en que por fin se va a implantar la jornada de 6 horas...
es.wikipedia.org/wiki/Huelga_...
es.wikipedia.org/wiki/Huelga_...
Huelga de La Canadiense - Wikipedia, la enciclopedia libre
es.wikipedia.org
April 28, 2025 at 11:01 AM
#apagon en toda península dicen, y yo no dejo de pensar en que por fin se va a implantar la jornada de 6 horas...
es.wikipedia.org/wiki/Huelga_...
es.wikipedia.org/wiki/Huelga_...
Ayer a las 21:29 (hora canaria) tuve la suerte de ver un meteorito entrar en la atmósfera terrestre 😍 es mi segundo avistamiento, aunque el primero es difícil de superar: fue tan grande que se vio desde aviones y salió en las noticias. Me trataron por loco durante unos días, no obstante 😂
April 4, 2025 at 9:28 AM
Ayer a las 21:29 (hora canaria) tuve la suerte de ver un meteorito entrar en la atmósfera terrestre 😍 es mi segundo avistamiento, aunque el primero es difícil de superar: fue tan grande que se vio desde aviones y salió en las noticias. Me trataron por loco durante unos días, no obstante 😂
Reposted by David Castillo-Bolado
En un momento en el que haría falta un debate científico sereno y maduro para afrontar una situación de auténtica emergencia en el Mediterráneo, lo que tenemos es a unos talibanes del pelotazo pactando con terraplanistas para seguir asfaltando zonas inundables mientras encubren juntos a un criminal.
March 27, 2025 at 11:55 AM
En un momento en el que haría falta un debate científico sereno y maduro para afrontar una situación de auténtica emergencia en el Mediterráneo, lo que tenemos es a unos talibanes del pelotazo pactando con terraplanistas para seguir asfaltando zonas inundables mientras encubren juntos a un criminal.
Reposted by David Castillo-Bolado
Haríamos bien en escuchar a los que entienden, aunque nos digan lo que no queremos escuchar.
En el mediterráneo tenemos una bomba metereológica en permanente ebullición.
La solución a largo plazo sería el decrecimiento, a corto, no votar a negacionistas.
Esto de @juanbordera.bsky.social
En el mediterráneo tenemos una bomba metereológica en permanente ebullición.
La solución a largo plazo sería el decrecimiento, a corto, no votar a negacionistas.
Esto de @juanbordera.bsky.social
March 24, 2025 at 10:51 AM
Haríamos bien en escuchar a los que entienden, aunque nos digan lo que no queremos escuchar.
En el mediterráneo tenemos una bomba metereológica en permanente ebullición.
La solución a largo plazo sería el decrecimiento, a corto, no votar a negacionistas.
Esto de @juanbordera.bsky.social
En el mediterráneo tenemos una bomba metereológica en permanente ebullición.
La solución a largo plazo sería el decrecimiento, a corto, no votar a negacionistas.
Esto de @juanbordera.bsky.social
Memento comment: a comment whose sole existence is to prevent your future self (or anyone else for that matter) to make the same mistake again.
February 14, 2025 at 10:09 AM
Memento comment: a comment whose sole existence is to prevent your future self (or anyone else for that matter) to make the same mistake again.
My today's realization: we think of ourselves as our body (what we control and feel), and that explains why we cherish so much our cars or good tools.
February 2, 2025 at 8:00 PM
My today's realization: we think of ourselves as our body (what we control and feel), and that explains why we cherish so much our cars or good tools.
I think the tech is ready for us to start training on chars instead of tokens. With a good binary encoding an LLM can learn more from the same data. Example:
- Punctuation or not
- Letter or number
- Upper case flag
- Tilde modifier
- Hat modifier
- ...
- Punctuation or not
- Letter or number
- Upper case flag
- Tilde modifier
- Hat modifier
- ...
January 28, 2025 at 10:14 AM
I think the tech is ready for us to start training on chars instead of tokens. With a good binary encoding an LLM can learn more from the same data. Example:
- Punctuation or not
- Letter or number
- Upper case flag
- Tilde modifier
- Hat modifier
- ...
- Punctuation or not
- Letter or number
- Upper case flag
- Tilde modifier
- Hat modifier
- ...
Reposted by David Castillo-Bolado
¡El éxodo ha comenzado!
A pesar de las zancadillas, el viaje a Bluesky y Mastodon ha comenzado gracias a la iniciativa HelloQuitteX, que permite a los usuarios de X conservar sus comunidades de seguidores en nuevos puertos más sanos.
(La herramienta es gratuita.)
Más info: ctxt.es/es/20250101/...
A pesar de las zancadillas, el viaje a Bluesky y Mastodon ha comenzado gracias a la iniciativa HelloQuitteX, que permite a los usuarios de X conservar sus comunidades de seguidores en nuevos puertos más sanos.
(La herramienta es gratuita.)
Más info: ctxt.es/es/20250101/...
January 21, 2025 at 6:24 PM
¡El éxodo ha comenzado!
A pesar de las zancadillas, el viaje a Bluesky y Mastodon ha comenzado gracias a la iniciativa HelloQuitteX, que permite a los usuarios de X conservar sus comunidades de seguidores en nuevos puertos más sanos.
(La herramienta es gratuita.)
Más info: ctxt.es/es/20250101/...
A pesar de las zancadillas, el viaje a Bluesky y Mastodon ha comenzado gracias a la iniciativa HelloQuitteX, que permite a los usuarios de X conservar sus comunidades de seguidores en nuevos puertos más sanos.
(La herramienta es gratuita.)
Más info: ctxt.es/es/20250101/...
I liked this paper a lot:
· Cross-entropy loss often improves by scaling weights.
· When softmax saturates, some gradients reach 0 and learning stops.
· Weight decay helps, but via side-effects.
· They propose a +stable softmax and orthogonal gradient that fix this!
arxiv.org/abs/2501.04697
· Cross-entropy loss often improves by scaling weights.
· When softmax saturates, some gradients reach 0 and learning stops.
· Weight decay helps, but via side-effects.
· They propose a +stable softmax and orthogonal gradient that fix this!
arxiv.org/abs/2501.04697
Grokking at the Edge of Numerical Stability
Grokking, the sudden generalization that occurs after prolonged overfitting, is a surprising phenomenon challenging our understanding of deep learning. Although significant progress has been made in u...
arxiv.org
January 15, 2025 at 10:05 AM
I liked this paper a lot:
· Cross-entropy loss often improves by scaling weights.
· When softmax saturates, some gradients reach 0 and learning stops.
· Weight decay helps, but via side-effects.
· They propose a +stable softmax and orthogonal gradient that fix this!
arxiv.org/abs/2501.04697
· Cross-entropy loss often improves by scaling weights.
· When softmax saturates, some gradients reach 0 and learning stops.
· Weight decay helps, but via side-effects.
· They propose a +stable softmax and orthogonal gradient that fix this!
arxiv.org/abs/2501.04697
My PhD thesis was affected by CoVID so I never attended a big conference until #NeurIPS2024, and I quickly recognized the main benefit of in-person events: being in a different environment and moving around make you think differently & learn better (@hubermanlab.com has a few episodes on this).
January 8, 2025 at 3:19 PM
My PhD thesis was affected by CoVID so I never attended a big conference until #NeurIPS2024, and I quickly recognized the main benefit of in-person events: being in a different environment and moving around make you think differently & learn better (@hubermanlab.com has a few episodes on this).
After some refactor and renovation, the layer-wise unsupervised training is running smoothly and, most importantly, predictably! I found out that I can just track the weights' std to detect when it has converged. Next steps will be adding biases and stacking layers! 🤘🏽
December 29, 2024 at 9:54 PM
After some refactor and renovation, the layer-wise unsupervised training is running smoothly and, most importantly, predictably! I found out that I can just track the weights' std to detect when it has converged. Next steps will be adding biases and stacking layers! 🤘🏽
Revisiting the unsupervised learning of weights with competing hidden units and giving the project some (deserved) love. See what happens when you train 100 hidden units on CIFAR10 (live, on CPU).
🐙 github.com/dcasbol/biol...
🐙 github.com/dcasbol/biol...
December 28, 2024 at 10:29 PM
Revisiting the unsupervised learning of weights with competing hidden units and giving the project some (deserved) love. See what happens when you train 100 hidden units on CIFAR10 (live, on CPU).
🐙 github.com/dcasbol/biol...
🐙 github.com/dcasbol/biol...
After 1 day back home (still jetlagged), here are my (personal) #NeurIPS2024 takeaways:
• All is about LLMs now
• Great interest + feedback from our LTM Benchmark poster 🥳
• Causality is hot
• xLSTM looks VERY promising
• Flow matching models 👌🏼
• Collective Intelligence is not dead!
• All is about LLMs now
• Great interest + feedback from our LTM Benchmark poster 🥳
• Causality is hot
• xLSTM looks VERY promising
• Flow matching models 👌🏼
• Collective Intelligence is not dead!
December 18, 2024 at 1:09 PM
After 1 day back home (still jetlagged), here are my (personal) #NeurIPS2024 takeaways:
• All is about LLMs now
• Great interest + feedback from our LTM Benchmark poster 🥳
• Causality is hot
• xLSTM looks VERY promising
• Flow matching models 👌🏼
• Collective Intelligence is not dead!
• All is about LLMs now
• Great interest + feedback from our LTM Benchmark poster 🥳
• Causality is hot
• xLSTM looks VERY promising
• Flow matching models 👌🏼
• Collective Intelligence is not dead!
We're here at #NeurIPS2024 presenting the LTM Benchmark! Come check it out, we're at the West Ballroom, poster 5407 😁✌🏽
December 13, 2024 at 7:01 PM
We're here at #NeurIPS2024 presenting the LTM Benchmark! Come check it out, we're at the West Ballroom, poster 5407 😁✌🏽
First #NeurIPS2024 talk "Collaborative AI" and find myself into a bitcoin trap, including serious accounting juggling by the talker to justify bitcoin. I guess it can only get better? 🤷🏽♂️
December 10, 2024 at 4:43 PM
First #NeurIPS2024 talk "Collaborative AI" and find myself into a bitcoin trap, including serious accounting juggling by the talker to justify bitcoin. I guess it can only get better? 🤷🏽♂️
Capitalism sets irrealistic goals, usually stemming from infinite growth. The only ways to survive as a business is therefore to either outsource your losses or detach your product or service from reality too.
November 27, 2024 at 1:19 PM
Capitalism sets irrealistic goals, usually stemming from infinite growth. The only ways to survive as a business is therefore to either outsource your losses or detach your product or service from reality too.
The often missed secret of successful ideas is that many people try to have one by thinking about potentially successful things, but it's the other way around. They start as niche hacks that end up being very useful or cherished by others, and nobody thought they'd be successful at all.
November 21, 2024 at 12:23 PM
The often missed secret of successful ideas is that many people try to have one by thinking about potentially successful things, but it's the other way around. They start as niche hacks that end up being very useful or cherished by others, and nobody thought they'd be successful at all.
"This is an engineering problem, and good engineers should be able to agree on it."
-- Bob Cabana (Retired NASA astronaut)
Me: spaces or tabs?
-- Bob Cabana (Retired NASA astronaut)
Me: spaces or tabs?
November 19, 2024 at 12:17 PM
"This is an engineering problem, and good engineers should be able to agree on it."
-- Bob Cabana (Retired NASA astronaut)
Me: spaces or tabs?
-- Bob Cabana (Retired NASA astronaut)
Me: spaces or tabs?
Hi all! First impressions are great 😁 I can tell that twitter minds were behind it, and finding so many mirrors from twitter accounts has been a cool surprise!
November 15, 2024 at 12:25 PM
Hi all! First impressions are great 😁 I can tell that twitter minds were behind it, and finding so many mirrors from twitter accounts has been a cool surprise!