Because in comparison, the Wu family is obviously more powerful. If the Yao family denies it, it will undoubtedly mean that the Yao family will not climb up to the rich and nobles and is unwilling to deal with the Wu family. This way of refusing is a bit too blunt....