ツ(psst also I am just a solo undergrad working on this, my fault if I get some stuff wrong)

Trained on a subset of AudioSet and CC3M for now, with lots of open questions to explore around architecture design and scaling before the full paper.
More experiments, improvements and 𝚙̶𝚛̶𝚘̶𝚙̶𝚎̶𝚛̶ evals coming soon.
HuggingFace: huggingface.co/SajayR/Triad
Github: github.com/SajayR/TRIAD

Current approaches learn global alignment between modalities (Think ImageBind or CLIP), Triad learns to map precise image regions to corresponding audio segments and text spans. So each patch/segment/span contains semantic concept information that are embedded in the same shared embedding space.

Nah, I've been following you for a decent while on there and just thought I'd say hi.

heyo :D

oh yeah you *totally* follow me on twitter
(but I will be just posting about cool papers I read in CV/NLP soooo :) )

Building off of DINO's teacher-student architecture (I'll should probably cover that too), they create multiple views of each frame by masking all but one tracked object. The student then has to match the teacher's predictions while only seeing these single-object views.

This tracking builds natural views that actually capture how objects transform in the real world, probably one of the major reasons for the SOTA evals.

They use the emergent property of transformer attention heads to focus on different parts of an image(as seen in DINO's paper), clean up these attention maps using Sinkhorn-Knopp to get non-overlapping object regions, and then track these objects temporally with cross-attention between frames.

Ah, was hoping for some deeper mech-interp insight but I guess it does make sense for the general user/devs.