RealEra : Semantic-level Concept Erasure via Neighbor-Concept Mining

Yufan Liu12, Jinyang An12, Wanqian Zhang1*, Ming Li3*, Dayan Wu1, Jingzi Gu1, Zheng Lin12, Weiping Wang12,
1Institute of Information Engineering, Chinese Academy of Sciences 2School of Cyber Security, University of Chinese Academy of Sciences 3Guangdong Laboratory of Artificial Intelligence and Digital Economy (SZ)
(*Corresponding Author)
liuyufan@iie.ac.cn, anjinyang@iie.ac.cn, zhangwanqian@iie.ac.cn
arXiv
Teaser images

For text inputs closely associated in semantics but not explicitly containing the erasure concept, previous methods still generate objects of erasure concept, defined as the "concept residue" issue. For example, when it comes to concept of "airplane", if we input "Antonov An-225 Mriya stunning take off from the airport", which is a specific name of aircraft, previous MACE method still generates an image of airplane. While our RealEra method shows the real erasure on airplane, showing the trade-off between efficacy and specificity.

Abstract

The remarkable development of text-to-image generation models has raised notable security concerns, such as the infringement of portrait rights and the generation of inappropriate content. Concept erasure has been proposed to remove the model's knowledge about protected and inappropriate concepts. Although many methods have tried to balance the efficacy (erasing target concepts) and specificity (retaining irrelevant concepts), they can still generate abundant erasure concepts under the steering of semantically related inputs. In this work, we propose RealEra to address this "concept residue" issue. Specifically, we first introduce the mechanism of neighbor-concept mining, digging out the associated concepts by adding random perturbation into the embedding of erasure concept, thus expanding the erasing range and eliminating the generations even through associated concept inputs. Furthermore, to mitigate the negative impact on the generation of irrelevant concepts caused by the expansion of erasure scope, RealEra preserves the specificity through the beyond-concept regularization. This makes irrelevant concepts maintain their corresponding spatial position, thereby preserving their normal generation performance. We also employ the closed-form solution to optimize weights of U-Net for the cross-attention alignment, as well as the prediction noise alignment with the LoRA module. Extensive experiments on multiple benchmarks demonstrate that RealEra outperforms previous concept erasing methods in terms of superior erasing efficacy, specificity, and generality.


Method

Image0

The overall pipeline of the proposed RealEra method. We mine and erase the associated concepts in the neighborhood of the erasure concepts, and to remain the mapping relationship of other unrelated concepts, we introduce additional beyond-concept regularization to preserve its generative ability. Finally, we apply these two manipulation to closed-form solution and noise alignment, as two optimization process for diffusion.

Results

The visualisation comparisons below shows the performance of RealEra. More Quantitative results can be found in the paper.

▶ Qualitative comparison of erasing objects, celebrity and artistic style

BibTeX

@misc{liu2024realerasemanticlevelconcepterasure,
      title={RealEra: Semantic-level Concept Erasure via Neighbor-Concept Mining}, 
      author={Yufan Liu and Jinyang An and Wanqian Zhang and Ming Li and Dayan Wu and Jingzi Gu and Zheng Lin and Weiping Wang},
      year={2024},
      eprint={2410.09140},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2410.09140}, 
}