Abstract: Adversarial attacks have challenged the security of deep neural networks (DNNs) recently. The most prominent adversarial attack methods include backdoor attacks, adversarial examples, etc.