I think you can "unfollow" any followers. (See your list of followers for the unfollow button.) I don't know much beyond that.