The main skill needed here was pin pointing for the AI where the issue is exactly.
Then I just had to log absolutely everything that was happening, plug in one a good and one bad input sample, and give the logs/terminal output to cursor.

Left alone, cursor was searching in the wrong place and coming up with ridiculous theories like my embedding model randomly producing different vectors.