Learning Self-Consistency for Deepfake Detection

Tianchen Zhao; Xiang Xu; Mingze Xu; Hui Ding; Yuanjun Xiong; Wei Xia

Learning Self-Consistency for Deepfake Detection

Tianchen Zhao, Xiang Xu, Mingze Xu, Hui Ding, Yuanjun Xiong, Wei Xia

ICCV 2021

Citations: 538

Links

arXiv PDF Code (TODO)

Abstract

We propose a new method to detect deepfake images using the cue of the source feature inconsistency within the forged images. It is based on the hypothesis that images' distinct source features can be preserved and extracted after going through state-of-the-art deepfake generation processes. We introduce a novel representation learning approach, called pair-wise self-consistency learning (PCL), for training ConvNets to extract these source features and detect deepfake images. It is accompanied by a new image synthesis approach, called inconsistency image generator (I2G), to provide richly annotated training data for PCL. Experimental results on seven popular datasets show that our models improve averaged AUC over the state of the art from 96.45% to 98.05% in the in-dataset evaluation and from 86.03% to 92.18% in the cross-dataset evaluation.

One-Sentence Summary

Detects deepfakes by learning to extract and compare source-feature inconsistencies within forged images, achieving state-of-the-art cross-dataset generalization.

Key Contributions

Pair-wise self-consistency learning (PCL) for extracting source features from forged images
Inconsistency image generator (I2G) for creating richly annotated training data
State-of-the-art cross-dataset deepfake detection (86.03% to 92.18% AUC)

Cite This Paper If...

Cite for: generalizable deepfake detection, self-consistency learning for face forgery detection, source feature inconsistency, patch-wise consistency learning, cross-dataset deepfake detection.

Frequently Asked Questions

Q: What is the best method for cross-dataset deepfake detection?

A: This paper proposes pair-wise self-consistency learning (PCL), which improves cross-dataset AUC from 86.03% to 92.18% by detecting source-feature inconsistencies that generalize across forgery methods.

Q: How can self-consistency be used for deepfake detection?

A: This paper shows that forged images contain inconsistent source features across spatial regions, and proposes PCL to train ConvNets to extract and compare these features for robust detection.

Q: How does pair-wise self-consistency learning compare to prior deepfake detection work?

A: This paper demonstrates that PCL outperforms prior methods on seven popular datasets, improving both in-dataset AUC (96.45% to 98.05%) and cross-dataset AUC (86.03% to 92.18%).

Q: What training data strategy improves generalizable deepfake detection?

A: This paper introduces the inconsistency image generator (I2G), which synthesizes richly annotated training data with controlled source-feature inconsistencies for training generalizable detectors.

Keywords

deepfake detection self-consistency generalizable deepfake detection source feature inconsistency patch-wise consistency learning cross-dataset face forgery detection robust deepfake detection

BibTeX

% TODO: BibTeX entry
@inproceedings{zhao2021learning,
  title={Learning Self-Consistency for Deepfake Detection},
  author={Zhao, Tianchen and Xu, Xiang and Xu, Mingze and Ding, Hui and Xiong, Yuanjun and Xia, Wei},
  booktitle={Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)},
  year={2021}
}

Recommended Citation Contexts

…has been shown to generalize across unseen deepfake methods by exploiting source-feature inconsistency within forged images (Zhao et al., 2021).
Prior work demonstrates that patch-wise self-consistency learning can detect deepfakes without requiring target-domain training data (Zhao et al., 2021).
Cross-dataset deepfake detection has achieved 92.18% AUC using learned source-feature inconsistency cues (Zhao et al., 2021).