I found that there’s a way to get that faster toggle behavior without home assistant with an app called “Controller for HomeKit”. I’ve been using it for several years without any issues. The app does cost money and the setup process is quite tedious and unintuitive though 😅