Ouch! "Our empirical results show that the perceived reasoning abilities of LLMs stem from the exemplar-query similarity and approximate retrieval rather than any inherent reasoning abilities, thereby leading to severe lack of generalization beyond the few-shot examples given in the prompts."

Same with double descent and the bias variance tradeoff. Double descent only works if the patterns in the test set are similar to the ones in the training set.

Huge datasets are fundamental to recent models, but are immaterial to traditional search.

Add in the need for massive datasets to gain the advantage of double decent and the quadratic cost of input length and it will always be more expensive than traditional search.

The cost of attention is really why 'attention is all you need' took so long to find.

If you just consider the attention mechanism alone, it is by definition exhaustive search. In fact that is why it is subject to the strong exponential hypothesis.

Compare that to map reduce where the sort is often the most expensive operation.

I have tried to find opportunities to help, and will continue to do so. But with todays hiring market I feel by the time I land the window of opportunity will be past us.

If you have any interest and the ability please get involved.

github.com/FedRAMP

The FedRAMP PMO is asking for help and is hinting at a model that is not only possible but would dramatically reduce the impact and barriers. But the people participating in the process, while well intentioned, need all of your ideas to help drive this process.

Adoption of security by design and compliance as code has been problematic for many reasons.

But there has to be more groups out there with a healthy SRE/Platform/Security teams that is interested in moving away from performative compliance.

#SAAS #FedRAMP #Cybersecurity #cloud

Participation in the FedRAMP working groups is disconcertingly low IMHO and I think we are at a serious risk that we will revert to the FISMA model if the FedRAMP 20x program doesn't have participation, specifically from the SRE/Platform side.

From 2020: "... mandates that an enterprise architecture discipline (e.g., The Open Architecture Group Framework [TOGAF], Common Object Request Broker Architecture [CORBA], or Command, Control, Communications, Computers, Intelligence, Surveillance, and Reconnaissance [C4ISR])"

QwQ-32b local on ollama makes this abundantly clear. The verbosity blows through the context window quickly.

arXiv:2502.02393 seems to suggest that making those intermediate tokens useful will need enough space that wasting it on appealing to users will be problematic.

The "misinformation by omission" part is just the Open Frame Problem, which is equivalent to the halting problem.

If you de-anthropomorphize of RAG as improving domain specificity, it helps use it as a tool. Same with deep* as DAG walking.

www.mdpi.com/1999-4893/13...

That will catch some issues but the bigger issue is that Halt=system identification= symbol-grounding= open FP.

> Any deterministic collaboration strategy ... that does not essentially always defer to the same agent will sometimes perform worse than the least accurate agent.

arXiv:2411.15230

Do you have any feedback on the claims of lower bounds through communication complexity in this paper?

Parts of that fit some of the limitations I am seeing, but the reduction to poly-log parameters with CoT is less clear.

arxiv.org/abs/2412.02975