I was reading into it and they can cause dermatitis in some people depending on the bite. Ugh! Thats a pain in the ass! I hope you're not having that reaction to them.

I wonder if you could somehow put some kind of diatemacous earth up there to try to kill them out? Or maybe patch those spots up with a wood filler until you can get pest control in there. I'm sorry youre dealing with this, bugs are really annoying to get rid of, and the stinging ones are never fun.

Evidently they go after some other beetle that eats away at wood. If you can I'd honestly do a full round of pest control, but getting rid of the beetles they eat is what will help drive them away.