It's possible the extra thickness of seam allowances is the reason the thread is a little loose at that end of the buttonhole.