Safeguarding Vision-Language Models: Mitigating Vulnerabilities to Gaussian Noise in Perturbation-based Attacks

Jiawei Wang, Yushen Zuo, Yuanjun Chai, Zhendong Liu, Yicheng Fu, Yichun Feng, Kin-Man Lam

Project |

• We propose the Robust-VLGuard dataset and DiffPure-VLM defense framework to address the problem that visual language models (VLMs) are vulnerable to Gaussian noise and adversarial perturbation attacks. Through fine-tuning with Gaussian noise enhancement, the attack success rate of VLMs on datasets such as CIFAR-10 and ImageNet is significantly reduced (for example, the attack success rate of MiniGPT-4 in the RealToxicityPrompts benchmark test dropped from 44.1% to 16.5%). Combined with the distribution conversion capability of the diffusion model DiffPure, adversarial noise is converted into Gaussian noise, further improving the defense effect, especially under strong attacks (for example, the attack success rate of InternVL2 dropped from 57.3% to 36.1% when ϵ=64/255).

IKC: Blind Super-Resolution With Iterative Kernel Correction

Jinjin Gu, Hannan Lu, Wangmeng Zuo, Chao Dong

Project |

• We propose an Iterative Kernel Correction (IKC) method for blur kernel estimation in blind SR problem, where the blur kernels are unknown. We draw the observation that kernel mismatch could bring regular artifacts (either over-sharpening or over-smoothing), which can be applied to correct inaccurate blur kernels. Thus we introduce an iterative correction scheme – IKC that achieves better results than direct kernel estimation. We further propose an effective SR network architecture using spatial feature transform (SFT) layers to handle multiple blur kernels, named SFTMD.

Kreado AI: AIGC Platform for Marketing Content Generation

Yuanjun Chai and YeahMobi.inc

• We propose a new AIGC platform for marketing content generation, named Kreado AI. Kreado AI is a hybrid worldwide AIGC platform that combines the strengths of so many AIGC functions:
  • Virtual Avatar (talking-face generation, speech synthesis, LLM)
  • AI model (text-to-image, LoRA, control net)
  • Custom clone serivces (image-to-video, voice clone)
  • Many AI tools and AI property
• Here I mainly focus on the Virtual Avatar and AI model algorithms improvement, as well as collaborate with system architect for entire architecture improvement. Users radiate to Europe, Africa, Southeast Asia, and the Americas, with quarterly revenue exceeding US$1 million.

h2o GPT: AIGC Platform for Marketing Content Generation

Yuanjun Chai

Project |

• We develop a new RAG-based LLM platform for AI cloud native and private AI. The platform could leverage diverse LLMs with extended dataset such as pdf, code base, dataset and internet links. Here I am responsible for all RAG-based LLM algorithm development, as well as industrial deployment. Functionality includes:
  • QA and chat with extended dataset extraction
  • Content summarization
  • Content generation
  • AI agent
• Next step we would take research about Multi-modal LLM for AI cloud native.

Face Control: Fine Facial Control Platform for High-paraplegia Disabled People

Yuanjun Chai, Ingrem.inc

• I co-founded a start-up Ingrem, with other hardcore guys. We aim to build up a entire bed for living and playing of high-paraplegia disabled people. Here, I am responsible for the development of the software – eyes&facial control platform. Based on computer vision algorithms, the system could help the diabled use their face details (such as eyebrow, eye, mouth, etc.) to control mouse and keyborad elaborately. Thus, our platform and our bed entirely enhance the accessibility of normal computer usage and social networks. We do believe tech make people’s lives better, and we do it!