yzma lets you use Go to perform local inference with Vision Language Models (VLMs) and Large Language Models (LLMs) using llama.cpp without CGo. - hybridgroup/yzma

yzma lets you use Go to perform local inference with Vision Language Models (VLMs) and Large Language Models (LLMs) using llama.cpp without CGo. - hybridgroup/yzma

This supersedes #6389 (now using a fully C++ approach), #5695 (first attempt at supporting Functionary) and #9592 (more recent Python wrapper). Which models are supported (in their native style)? W...

LlamaNet: 1~2줄의 코드 변환만으로 OpenAI 기반 애플리케이션을 llama.cpp 기반 모델로 쉽게 변경 가능한 라이브러리 소개 라마넷(Llamanet)은 OpenAI 기반의 앱을 llama.cpp 앱으로 단 한 줄의 코드만으로 전환할 수 있는 오픈소스 라이브러리/도구입니다. (모델도 변경해야 하므로 2줄) 별도의 설정 없이 그리고 서드파티 의존성 없이 작동하므로, 아무것도 몰라도 OpenAI API를 사용하는 것처럼 로컬 모델로 바로 변경하여 사용할 수 있습니다. LlamaNet이 해결하고자 하는 문제들은 다음과 같습니다: OpenAI API 기반의 LLM 앱을 로컬 LLM으로 즉시 포팅하고 싶으신가요? 앱 사용자가 서드파티 LLM 앱이나 서버를 다운로드할 필요 없이 앱을 사용하게 하고 싶으신가요? 서드파티 시스템에 의존하지 않고 앱 자체에서 LLM 관리를 처리하고 싶으신가요? LlamaNet은 상대적으로 가벼운 코드베이스를 가지고 있어, 설치와 실행이 용이...

Origin

As AI use cases continue to expand — from document summarization to custom software agents — developers and enthusiasts are seeking faster, more flexible ways to run large language models (LLMs). Running models locally on PCs with NVIDIA GeForce RTX GPUs enables high-performance inference, enhanced data privacy and full control over AI deployment and integration. Tools like LM Studio — free to try — make this possible, giving users an easy way to explore and build with LLMs on their own hardware. LM Studio has become one of the most widely adopted tools for local LLM inference. Built on the high-performance llama.cpp runtime, the app allows models to run entirely offline and can also serve as OpenAI-compatible application programming interface (API) endpoints for integration into custom workflows. The release of LM Studio 0.3.15 brings improved performance for RTX GPUs thanks to CUDA 12.8, significantly improving model load and response times. The update also introduces new developer-focused features, including enhanced tool use via the _“_ tool_choice” parameter and a redesigned system prompt editor. The latest improvements to LM Studio improve its performance and usability — delivering the highest throughput yet on RTX AI PCs. This means faster responses, snappier interactions and better tools for building and integrating AI locally. ## **Where Everyday Apps Meet AI Acceleration** LM Studio is built for flexibility — suited for both casual experimentation or full integration into custom workflows. Users can interact with models through a desktop chat interface or enable developer mode to serve OpenAI-compatible API endpoints. This makes it easy to connect local LLMs to workflows in apps like VS Code or bespoke desktop agents. For example, LM Studio can be integrated with Obsidian, a popular markdown-based knowledge management app. Using community-developed plug-ins like Text Generator and Smart Connections, users can generate content, summarize research and query their own notes — all powered by local LLMs running through LM Studio. These plug-ins connect directly to LM Studio’s local server, enabling fast, private AI interactions without relying on the cloud. Example of using LM Studio to generate notes accelerated by RTX. The 0.3.15 update adds new developer capabilities, including more granular control over tool use via the _“_ tool_choice” parameter and an upgraded system prompt editor for handling longer or more complex prompts. The tool_choice parameter lets developers control how models engage with external tools — whether by forcing a tool call, disabling it entirely or allowing the model to decide dynamically. This added flexibility is especially valuable for building structured interactions, retrieval-augmented generation (RAG) workflows or agent pipelines. Together, these updates enhance both experimentation and production use cases for developers building with LLMs. LM Studio supports a broad range of open models — including Gemma, Llama 3, Mistral and Orca — and a variety of quantization formats, from 4-bit to full precision. Common use cases span RAG, multi-turn chat with long context windows, document-based Q&A and local agent pipelines. And by using local inference servers powered by the NVIDIA RTX-accelerated llama.cpp software library, users on RTX AI PCs can integrate local LLMs with ease. Whether optimizing for efficiency on a compact RTX-powered system or maximizing throughput on a high-performance desktop, LM Studio delivers full control, speed and privacy — all on RTX. ## **Experience Maximum Throughput on RTX GPUs** At the core of LM Studio’s acceleration is llama.cpp — an open-source runtime designed for efficient inference on consumer hardware. NVIDIA partnered with the LM Studio and llama.cpp communities to integrate several enhancements to maximize RTX GPU performance. Key optimizations include: * **CUDA graph enablement:** Groups multiple GPU operations into a single CPU call, reducing CPU overhead and improving model throughput by up to 35%. * **Flash attention CUDA kernels** : Boosts throughput by up to 15% by improving how LLMs process attention — a critical operation in transformer models. This optimization enables longer context windows without increasing memory or compute requirements. * **Support for the latest RTX architectures:** LM Studio’s update to CUDA 12.8 ensures compatibility with the full range of RTX AI PCs — from GeForce RTX 20 Series to NVIDIA Blackwell-class GPUs, giving users the flexibility to scale their local AI workflows from laptops to high-end desktops. Data measured using different versions of LM Studio and CUDA backends on GeForce RTX 5080 on DeepSeek-R1-Distill-Llama-8B model. All configurations measured using Q4_K_M GGUF (Int4) quantization at BS=1, ISL=4000, OSL=200, with Flash Attention ON. Graph showcases ~27% speedup with the latest version of LM Studio due to NVIDIA contributions to the llama.cpp inference backend. With a compatible driver, LM Studio automatically upgrades to the CUDA 12.8 runtime, enabling significantly faster model load times and higher overall performance. These enhancements deliver smoother inference and faster response times across the full range of RTX AI PCs — from thin, light laptops to high-performance desktops and workstations. ## **Get Started With LM Studio** LM Studio is free to download and runs on Windows, macOS and Linux. With the latest 0.3.15 release and ongoing optimizations, users can expect continued improvements in performance, customization and usability — making local AI faster, more flexible and more accessible. Users can load a model through the desktop chat interface or enable developer mode to expose an OpenAI-compatible API. To quickly get started, download the latest version of LM Studio and open up the application. 1. Click the magnifying glass icon on the left panel to open up the **Discover** menu. 2. Select the **Runtime** settings on the left panel and search for the **CUDA 12 llama.cpp (Windows)** runtime in the availability list. Select the button to Download and Install. 3. After the installation completes, configure LM Studio to use this runtime by default by selecting **CUDA 12 llama.cpp (Windows)** in the Default Selections dropdown. 4. For the final steps in optimizing CUDA execution, load a model in LM Studio and enter the Settings menu by clicking the gear icon to the left of the loaded model. 5. From the resulting dropdown menu, toggle “Flash Attention” to be on and offload all model layers onto the GPU by dragging the “GPU Offload” slider to the right. Once these features are enabled and configured, running NVIDIA GPU inference on a local setup is good to go. LM Studio supports model presets, a range of quantization formats and developer controls like tool_choice for fine-tuned inference. For those looking to contribute, the llama.cpp GitHub repository is actively maintained and continues to evolve with community- and NVIDIA-driven performance enhancements. Each week, the RTX AI Garage blog series features community-driven AI innovations and content for those looking to learn more about NVIDIA NIM microservices and AI Blueprints, as well as building AI agents, creative workflows, digital humans, productivity apps and more on AI PCs and workstations. Plug in to NVIDIA AI PC on Facebook, Instagram, TikTok and X — and stay informed by subscribing to the RTX AI PC newsletter. Follow NVIDIA Workstation on LinkedIn and X. Categories: Corporate | Generative AI Tags: Artificial Intelligence | GeForce | NVIDIA RTX | RTX AI Garage

Origin

Microsoft's Clippy gets an AI makeover with new app, offering offline chat with local LLMs like Qwen3 and Llama 3.2.

PR by Xuan-Son Nguyen for `llama.cpp`: > This PR provides a big jump in speed for WASM by leveraging SIMD instructions for `qX_K_q8_K` and `qX_0_q8_0` dot product functions. > > …

YouTube video by Hassan Habib

We curated enough data to provide a foundational understanding of quantization principles, addressing common confusions and answering questions you might have hesitated to ask. So here’s everything I ...

Origin

PR by Xuan-Son Nguyen for `llama.cpp`: > This PR provides a big jump in speed for WASM by leveraging SIMD instructions for `qX_K_q8_K` and `qX_0_q8_0` dot product functions. > > …

This PR is still WIP (see todos at the bottom) but welcoming early feedback / testing Support streaming of tool calls in OpenAI format Improve handling of thinking model (DeepSeek R1 Distills, QwQ...

A first basic test putting my hands on LLaMA.CPP Introduction What is...