Are you upgrading to 2.3 by any chance? If so, then we removed the mutatingwebhookconfiguration (which still has an old CA in it). When doing upgrades you need to uninstall with the old tool version (which knows about the old resources), then install with the new one.
Over the past few releases I’ve been sneaking in annotations so we know the version something was installed with, at which point we can infer what needs to be deleted and do an automatic all in one upgrade command like magic. Keep an eye out for that in the next few releases…