Design & media

bytedance-visual-recognition

Multimodal visual recognition via Doubao-Seed (6 models) + Zhipu GLM (2 free models). Image/video to text/code, auto-fallback, batch directory processing, follow-up conversations, local media caching (Temp/), persistent history (vision_history.json), auto IAM console sync. First-run privacy notice, plaintext config.json, cross-platform.

What it does

The skill document

ByteDance Visual Recognition — Doubao-Seed + GLM

Doubao-Seed (6 models) + Zhipu GLM (2 free models). First run auto-generates config.json, fill in one API Key to start. IAM console usage syncs automatically on each recognition.

Privacy & Data Notice

Network: Selected images/videos and prompts are base64-encoded and sent to Volcengine (Doubao) or Zhipu (GLM) cloud APIs.
Local cache: Media files temporarily copied to Temp/YYYYMMDD/, default 7-day retention (temp_retention_days in config.json, range 1-3650).
History: Recognition history stored in vision_history.json, follow-up context in .last_response, auto-cleaned after 7 days. GLM follow-up reuses full message history including base64 media data; Doubao follow-up uses previous_response_id without re-transmitting files.
Credentials: API Keys stored in plaintext config.json. Do not use personal keys on shared machines.
IAM sync: Automatically syncs token usage from Volcengine IAM console on each recognition when IAM credentials are configured.
First run: A privacy notice is displayed once. Continuing past it acknowledges data handling practices.

Setup (pick one)

Doubao (Volcengine)

Join the Collaboration Rewards Program for free quota, then get your API Key
Create inference endpoints, pick from:

config key	model	priority
`doubao_seed_21p_id`	Doubao-Seed-2.1-Pro	primary
`doubao_seed_21t_id`	Doubao-Seed-2.1-Turbo	secondary
`doubao_seed_20p_id`	Doubao-Seed-2.0-Pro	tertiary
`doubao_seed_20c_id`	Doubao-Seed-2.0-Code	code-first
`doubao_seed_20l_id`	Doubao-Seed-2.0-Lite	fallback
`doubao_seed_20m_id`	Doubao-Seed-2.0-Mini	low-cost

Edit config.json, replace "" with actual endpoint IDs.

GLM (Zhipu, free)

https://open.bigmodel.cn → get API Key
Edit config.json: "zhipu_api_key": "your-key"

Provider filter

provider_mode in config.json:

0 = all (default)
1 = Zhipu only
2 = Doubao only

Test

python doubao_vision_recognize.py --help
python doubao_vision_recognize.py status

Commands

command	purpose	example
`rec --image\|--video --text\|--code`	recognize	`rec a.jpg --image --text`
`rec --batch --image\|--video --text\|--code`	batch	`rec ./img/ --batch --image --text`
`ask --text\|--code --prompt "..."`	follow-up	`ask --text -p "details"`
`status`	usage stats
`sync`	manual console sync
`history`	7-day history

Parameters

param	desc
`--image`	image input
`--video`	video input
`--text`	text output
`--code`	code output
`--prompt` / `-p`	extra instruction
`--batch`	directory batch

Behavior Rules

1. Trigger only on listed patterns

Activate only when the user's message matches one of the trigger_patterns above. Do NOT activate on loosely related text. If uncertain, ask before executing.

2. Parameter inference

"recognize/analyze image" → --image --text
"recognize/analyze video" → --video --text
"convert to code / UI to code / design to code" → --code
extra requirements → --prompt "..."
unsure → ask once if image or video

3. Credential safety

API keys are stored in plaintext config.json. Warn users not to use high-value keys on shared machines. First-run privacy notice already discloses plaintext storage.

Limits

Doubao: 180W tokens per model per day, auto-fallback
GLM: free, auto-retry on failure (4.6V: 10 retries, 4.1V: 5 retries)
Image ≤ 15MB, Video ≤ 50MB

Related skills

vision-fallback

Vision/image understanding for agents whose model can't read images (returns "model does not support images", empty/unknown output, low confidence, or user-reported failure). Calls an OpenAI-compatible vision API (doubao or any OpenAI-compatible provider), returns structured JSON. Use whenever an image must be understood. Do NOT substitute with local OCR (tesseract) - OCR extracts text only, not layout/visual understanding.

1 stars