Cognitive-structured Multimodal Agent

for Multimodal Understanding, Generation, and Editing

_{Long-horizon multimodal memory, retrieval, generation, and editing — with a tool-augmented deployment harness (CMA-Harness).}

Feng Wang^1,*   Canmiao Fu²   Zhipeng Huang²   Chen Li²   Jing LYU²   Ge Li¹

¹ Peking University   ² WeChat Vision, Tencent Inc.
_{* Work done during an internship at WeChat Vision, Tencent Inc.}

Adapted parts of this architecture have been rolled out to WeChat Xiaowei Agent for gray-scale testing. 该架构经调整的部分内容已上线到微信小微Agent中进行灰测。

TL;DR  ·  Results  ·  Demos  ·  Method  ·  Benchmark  ·  Citation

---

TL;DR

We introduce a memory-centric multimodal agent that externalizes visual history into Episodic Visual Memory (EVM), selectively retrieves relevant visual episodes, and plans understanding, generation, editing, and composition actions through a Multimodal Executive Controller (MEC). The same cognitive structure is instantiated as CMA-Harness, a tool-augmented, multi-session deployment.

Key Results

Numbers from paper Tables 3–4. Retrieval subsets follow Sec. 5.2.

Metric	English	Chinese	Setting
Retrieval (Full)	91.4%	89.6%	all 20 turns
Retrieval (Medium)	89.4%	85.6%	turns 11–20
Retrieval (Hard)	82.0%	79.0%	very_hard turns within 11–20
Gemini quality	8.49 / 10	8.53 / 10	overall generation score

Demos

Interactive multimodal sessions — search-driven generation, brand-fusion editing, cross-reference composition, and long-horizon visual recall.

Click any thumbnail to watch it on the live project page.

Brand Logo Fusion	AI Stock Watchlist	Pet Birthday Poster	Pet Birthday Composer
Bedroom Makeover	Weeknight Recipe	Bedroom Refresh	Listing Studio

Qualitative cases

_{A full multi-turn session. Branching dialogue spanning generation, editing, cross-reference composition, and long-horizon visual recall.}

_{Qualitative comparison. CMA (Ours) vs. an all-context baseline on cross-turn grounding, consistent editing, and long-range recall.}

_{Example Dialogue 1. Long-horizon generation, editing, topic switching, and visual question answering.}

_{Example Dialogue 2. Environmental transformations, long-range recall, and stable visual consistency.}

_{Example Dialogue 3. Object insertion, cross-scene composition, and visual comparison across distant turns.}

Method

A cognitive structure for long-horizon multimodal interaction:

• Structured visual memory — incoming and generated images are compressed into captions, tags, thumbnails, and metadata, so visual evidence persists without repeatedly occupying the model context window.
• Selective cross-turn retrieval — the Cognitive Retrieval Engine (CoRE) selects only the visual episodes relevant to the current user turn, improving grounding while reducing visual-token overhead.
• Executive task control — the Multimodal Executive Controller (MEC) infers whether a turn requires understanding, generation, editing, composition, or pure chat, then routes the task accordingly.
• Training for memory use — a Unified Scenario Engine generates structured multi-turn dialogues with retrieval annotations, enabling SFT and RL optimization for memory construction and retrieval.

M2CA-Bench

The Multi-turn Context Agent Benchmark (M2CA-Bench) is a held-out evaluation set of 100 sessions × 20 turns (2,000 turns) designed to stress-test long-horizon multimodal grounding.

2,000	100	55	4
evaluation turns	20-turn sessions	topics × 8 domains	difficulty strata

• Structured scenario representation — each turn is annotated as (uᵢ, τᵢ, Rᵢ*, dᵢ, fᵢ): user input, task type, ground-truth retrieval set, difficulty, and challenge tags. Topics span 8 domains with four task modes per topic — generate, edit, cross-reference-edit, understand.
• Four difficulty strata — stratified by topic shift, temporal span, multi-image interaction, and ambiguity: easy (same-topic sequential generation/editing), medium (high-similarity variants; simple topic switch), hard (cross-topic retrieval; long-range callback ≥ 8 turns), very_hard (multi-image comparison, fusion edits, ambiguous references).
• Hard-negative design — high-similarity confounders (near-duplicate images differing subtly) and negative retrieval samples (semantic negatives requiring no retrieval, structural negatives prompting new generation) block shortcut learning.
• Three evaluation subsets (paper Sec. 5.2) — Full (all 20 turns), Medium (turns 11–20), Hard (very_hard turns within 11–20).

Retrieval accuracy on M2CA-Bench (paper Table 3):

Method	EN Full	EN Med.	EN Hard	ZH Full	ZH Med.	ZH Hard
BAGEL	23.3	5.6	1.9	17.8	3.2	2.3
Agent 8B	78.9	67.0	59.5	76.7	66.0	60.8
Agent 32B	81.9	72.1	62.0	80.3	72.5	71.4
Multi 8B	81.8	75.2	63.6	82.5	77.6	77.8
Multi 32B	83.2	79.4	72.1	84.5	80.0	79.0
CMA 8B (Ours)	91.4	89.4	82.0	89.6	85.6	79.0

Citation

If you find this work useful, please consider citing:

@article{wang2026cognitive,
  title   = {Cognitive-structured Multimodal Agent for Multimodal Understanding, Generation, and Editing},
  author  = {Wang, Feng and Fu, Canmiao and Huang, Zhipeng and Li, Chen and LYU, Jing and Li, Ge},
  journal = {arXiv preprint arXiv:2607.08497},
  year    = {2026},
  eprint  = {2607.08497},
  archivePrefix = {arXiv},
  primaryClass = {cs.CV}
}

---

The code and M2CA-Bench dataset will be released here soon.

⭐ Star or 👀 watch this repository to get notified.