​
What are ComfyUI and ComfyStream?

  • ComfyUI is a modular, node-based user interface for Stable Diffusion. It allows users to create and manage workflows using interconnected nodes, similar to node-based visual programming environments. This provides a flexible and powerful way to create AI-generated images and videos.
  • ComfyStream extends the capabilities of ComfyUI by enabling real-time streaming of video and image data. It introduces networking, GPU streaming, and live control features that go beyond the standard functionality of ComfyUI. ComfyStream does this by wrapping ComfyUI and running a special version of it.

By combining ComfyUI’s powerful workflow management with ComfyStream’s real-time capabilities, users can create and control AI-driven visual content with enhanced interactivity. That is what we are doing here!

​
Introduction to the Architecture

To understand the purpose and flow of this tutorial, it’s important to grasp the high-level architecture of the ComfyStream system. This document includes a setup that involves both a local environment and a remote GPU server environment, each playing a specific role in the overall system.

Note to those with powerful GPUs: Most of this document details processes that enable using a cloud GPU to do the heavy processing fast. If you have a good GPU (like a 4090, for instance), your setup process is far more straightforward. It will still be worth it to understand this document, but you can skip to the end for specific instructions to set this up locally.

​
Components Overview

  1. Local Machine (Windows)
    • Purpose: Running the ComfyStream front-end web application. The graphical interface that allows you to interact with the ComfyStream server from your browser.
    • Tools Used: PowerShell/Terminal, Node.js, SSH, Ncat.
    • Processes:
      • Set up SSH connections to the remote server.
      • Run a local development server to interact with ComfyStream’s user interface (UI).
      • Access ComfyStream’s front-end in a browser
  2. Remote GPU Server (RunPod or Remote server)
    • Purpose: Execute GPU-accelerated processes and serve the ComfyStream backend for real-time streaming.
    • Tools Used: RunPod, SSH, Jupyter Lab, Miniconda, Conda environments
    • Processes:
      • Deploy and configure a GPU server on RunPod.
      • Set up two Conda environments (comfyui and comfystream) to handle separate installations of ComfyUI and ComfyStream.
      • Install ComfyUI, ComfyStream, and ComfyUI-Manager.
      • Expose necessary ports for SSH tunneling and allow local access to the ComfyStream UI.

​
Why use both a local system and a Remote GPU Setup?

TaskLocal MachineRemote GPU Server
DevelopmentEdit/run, ComfyStream UIEdit/run ComfyUI workflows
GPU ProcessingNot applicable (no/weak GPU)Execute GPU-accelerated processes
SSH AccessEstablish tunnel to RunPod serverProvides SSH access for tunneling
Web UIRun npm to serve ComfyStream UIExpose ComfyStream API for UI connection
Port ForwardingUse SSH to forward remote portsOpen required ports to connect to local

Explanation:

  1. The local machine is where you configure your setup, and access the server’s GPU capabilities via SSH tunneling.
  2. The remote GPU server is where heavy GPU computations happen. Since your local machine may not have a GPU powerful enough for tasks like real-time diffusion, the remote server is used to offload these computations.
  3. The SSH Tunneling allows you to bridge the local machine with the remote server so you can work with ComfyStream as if it were running locally, while still taking advantage of the remote server’s GPU.

With this architectural understanding, you’re now ready to proceed with the setup instructions below!

Proceed to Install ComfyUI and ComfyStream

Install ComfyUI and ComfyStream