nyrn
/
openclaw
mirror of https://github.com/openclaw/openclaw.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
openclaw/docs/providers/comfy.md

4.7 KiB
Raw Blame History

title summary read_when
ComfyUI ComfyUI workflow image, video, and music generation setup in OpenClaw
You want to use local ComfyUI workflows with OpenClaw
You want to use Comfy Cloud with image, video, or music workflows
You need the bundled comfy plugin config keys

ComfyUI

OpenClaw ships a bundled comfy plugin for workflow-driven ComfyUI runs.

  • Provider: comfy
  • Models: comfy/workflow
  • Shared surfaces: image_generate, video_generate, music_generate
  • Auth: none for local ComfyUI; COMFY_API_KEY or COMFY_CLOUD_API_KEY for Comfy Cloud
  • API: ComfyUI /prompt / /history / /view and Comfy Cloud /api/*

What it supports

  • Image generation from a workflow JSON
  • Image editing with 1 uploaded reference image
  • Video generation from a workflow JSON
  • Video generation with 1 uploaded reference image
  • Music or audio generation through the shared music_generate tool
  • Output download from a configured node or all matching output nodes

The bundled plugin is workflow-driven, so OpenClaw does not try to map generic size, aspectRatio, resolution, durationSeconds, or TTS-style controls onto your graph.

Config layout

Comfy supports shared top-level connection settings plus per-capability workflow sections:

{
  models: {
    providers: {
      comfy: {
        mode: "local",
        baseUrl: "http://127.0.0.1:8188",
        image: {
          workflowPath: "./workflows/flux-api.json",
          promptNodeId: "6",
          outputNodeId: "9",
        },
        video: {
          workflowPath: "./workflows/video-api.json",
          promptNodeId: "12",
          outputNodeId: "21",
        },
        music: {
          workflowPath: "./workflows/music-api.json",
          promptNodeId: "3",
          outputNodeId: "18",
        },
      },
    },
  },
}

Shared keys:

  • mode: local or cloud
  • baseUrl: defaults to http://127.0.0.1:8188 for local or https://cloud.comfy.org for cloud
  • apiKey: optional inline key alternative to env vars
  • allowPrivateNetwork: allow a private/LAN baseUrl in cloud mode

Per-capability keys under image, video, or music:

  • workflow or workflowPath: required
  • promptNodeId: required
  • promptInputName: defaults to text
  • outputNodeId: optional
  • pollIntervalMs: optional
  • timeoutMs: optional

Image and video sections also support:

  • inputImageNodeId: required when you pass a reference image
  • inputImageInputName: defaults to image

Backward compatibility

Existing top-level image config still works:

{
  models: {
    providers: {
      comfy: {
        workflowPath: "./workflows/flux-api.json",
        promptNodeId: "6",
        outputNodeId: "9",
      },
    },
  },
}

OpenClaw treats that legacy shape as the image workflow config.

Image workflows

Set the default image model:

{
  agents: {
    defaults: {
      imageGenerationModel: {
        primary: "comfy/workflow",
      },
    },
  },
}

Reference-image editing example:

{
  models: {
    providers: {
      comfy: {
        image: {
          workflowPath: "./workflows/edit-api.json",
          promptNodeId: "6",
          inputImageNodeId: "7",
          inputImageInputName: "image",
          outputNodeId: "9",
        },
      },
    },
  },
}

Video workflows

Set the default video model:

{
  agents: {
    defaults: {
      videoGenerationModel: {
        primary: "comfy/workflow",
      },
    },
  },
}

Comfy video workflows currently support text-to-video and image-to-video through the configured graph. OpenClaw does not pass input videos into Comfy workflows.

Music workflows

The bundled plugin registers a music-generation provider for workflow-defined audio or music outputs, surfaced through the shared music_generate tool:

/tool music_generate prompt="Warm ambient synth loop with soft tape texture"

Use the music config section to point at your audio workflow JSON and output node.

Comfy Cloud

Use mode: "cloud" plus one of:

  • COMFY_API_KEY
  • COMFY_CLOUD_API_KEY
  • models.providers.comfy.apiKey

Cloud mode still uses the same image, video, and music workflow sections.

Live tests

Opt-in live coverage exists for the bundled plugin:

OPENCLAW_LIVE_TEST=1 COMFY_LIVE_TEST=1 pnpm test:live -- extensions/comfy/comfy.live.test.ts

The live test skips individual image, video, or music cases unless the matching Comfy workflow section is configured.

Related