Hubbry Logo
ControlNetControlNetMain
Open search
ControlNet
Community hub
ControlNet
logo
7 pages, 0 posts
0 subscribers
Be the first to start a discussion here.
Be the first to start a discussion here.
ControlNet
ControlNet
ControlNet
View on Wikipedia
from Wikipedia

ControlNet is an open industrial network protocol for industrial automation applications, also known as a fieldbus. ControlNet was earlier supported by ControlNet International, but in 2008 support and management of ControlNet was transferred to ODVA, which now manages all protocols in the Common Industrial Protocol family.

Features which set ControlNet apart from other fieldbuses include the built-in support for fully redundant cables and the fact that communication on ControlNet can be strictly scheduled and highly deterministic. Due to the unique physical layer, common network sniffers such as Wireshark cannot be used to sniff ControlNet packets. Rockwell Automation provides ControlNet Traffic Analyzer software to sniff and analyze ControlNet packets.

Version 1, 1.25 and 1.5

[edit]

Versions 1 and 1.25 were released in quick succession when ControlNet first launched in 1997. Version 1.5 was released in 1998 and hardware produced for each version variant was typically not compatible. Most installations of ControlNet are version 1.5.[1]

Architecture

[edit]

Physical layer

[edit]

ControlNet cables consist of RG-6 coaxial cable with BNC connectors, though optical fiber is sometimes used for long distances. The network topology is a bus structure with short taps. ControlNet also supports a star topology if used with the appropriate hardware. ControlNet can operate with a single RG-6 coaxial cable bus, or a dual RG-6 coaxial cable bus for cable redundancy. In all cases, the RG-6 should be of quad-shield variety. Maximum cable length without repeaters is 1000m and maximum number of nodes on the bus is 99. However, there is a tradeoff between number of devices on the bus and total cable length. Repeaters can be used to further extend the cable length. The network can support up to 5 repeaters (10 when used for redundant networks). The repeaters do not utilize network node numbers and are available in copper or fiber optic choices.

The physical layer signaling uses Manchester code at 5 Mbit/s.

[edit]

ControlNet is a scheduled communication network designed for cyclic data exchange. The protocol operates in cycles, known as NUIs, where NUI stands for Network Update Interval. Each NUI has three phases, the first phase is dedicated to scheduled traffic, where all nodes with scheduled data are guaranteed a transmission opportunity. The second phase is dedicated to unscheduled traffic. There is no guarantee that every node will get an opportunity to transmit in every unscheduled phase. The third phase is network maintenance or "guardband". It includes synchronization and a means of determining starting node on the next unscheduled data transfer. Both the scheduled and unscheduled phase use an implicit token ring media access method. The amount of time each NUI consists of is known as the NUT, where NUT stands for Network Update Time. It is configurable from 2 to 100 ms. The default NUT on an unscheduled network is 5 ms.

The maximum size of a scheduled or unscheduled ControlNet data frame is 510 Bytes.

Application layer

[edit]

The ControlNet application layer protocol is based on the Common Industrial Protocol (CIP) layer which is also used in DeviceNet and EtherNet/IP.

References

[edit]
[edit]
Revisions and contributorsEdit on WikipediaRead on Wikipedia

ControlNet

View on Grokipedia
from Grokipedia
ControlNet is a architecture that adds spatial conditioning controls to large, pretrained text-to-image models, allowing users to guide image generation with precise inputs such as edge maps, human poses, depth maps, segmentation masks, and more, while preserving the original model's capabilities. Developed by researchers including Lvmin Zhang, it integrates these controls into models like by reusing their deep, robust encoding layers—pretrained on billions of images—as a backbone, without altering the core process. The architecture employs "zero convolutions," which are zero-initialized convolutional layers that connect the control modules to the pretrained model, enabling parameters to grow gradually from zero during training and preventing any disruptive noise from affecting the finetuning process. This design supports flexible conditioning, accommodating single or multiple control inputs alongside optional text prompts, and demonstrates robustness across diverse datasets, from small ones under 50,000 samples to large-scale sets exceeding 1 million. ControlNet's open-source implementation, available on , has facilitated widespread adoption in creative and technical applications, including pose-guided , edge-based sketch-to-image synthesis, and controlled scene composition. By decoupling control mechanisms from the generative backbone, ControlNet extends the utility of diffusion models beyond text prompts, enabling applications in fields like , , and tasks that require structured outputs. Its presentation at the 2023 IEEE/CVF International Conference on Computer Vision (ICCV), where it won the Marr Prize, underscores its contributions to controllable generative AI, with extensive evaluations showing superior performance in maintaining fidelity to conditions while generating high-quality images. Subsequent developments, such as ControlNet models for 3.5 Large released in November 2024, have further expanded its compatibility.

Overview

Definition and Purpose

ControlNet is a architecture that augments large pretrained text-to-image diffusion models, such as , with spatial conditioning controls. It enables precise guidance of image generation using additional inputs like edge maps, human poses, depth maps, and segmentation masks, while preserving the original model's capabilities. Developed by Lvmin Zhang and colleagues, ControlNet reuses the deep encoding layers of pretrained models—trained on billions of images—as a robust backbone for learning diverse conditional controls, without altering the core diffusion process. Its primary purpose is to provide controllable image synthesis beyond text prompts alone, supporting applications in , , scene composition, and tasks requiring structured outputs.

Key Features

ControlNet incorporates "zero convolutions," which are zero-initialized convolutional layers connecting the control modules to the locked pretrained model. This design allows parameters to grow gradually from zero during , preventing disruptive and enabling stable finetuning. The supports flexible conditioning with single or multiple control inputs, optionally combined with text prompts, and maintains robustness across scales from under 50,000 samples to over 1 million. Training converges rapidly, often within 10,000 steps on consumer hardware like an 3090Ti using 200,000 samples, achieving results competitive with larger-scale models. Evaluations demonstrate superior performance, with an Average User Ranking of 4.22 for image quality and 4.28 for fidelity to conditions on a 1-5 scale, outperforming baselines like PITI in user studies. Its open-source implementation on has driven adoption in creative and technical fields, as highlighted in its presentation at the 2023 IEEE/CVF International Conference on (ICCV).

History and Development

Origins and Standardization

ControlNet was introduced in a research paper titled "Adding Conditional Control to Text-to-Image Diffusion Models," published on arXiv on February 10, 2023, by Lvmin Zhang from , along with co-authors Anyi Rao and Maneesh Agrawala. The development stemmed from the need to enhance the controllability of large pretrained diffusion models like , which were limited to text prompts, by incorporating spatial conditions such as edge maps and poses without retraining the entire model from scratch. This approach reused the robust encoding layers of existing models, trained on billions of images, to maintain generative quality while adding flexible control modules. The architecture was open-sourced shortly after the paper's release via a GitHub repository by Lvmin Zhang (lllyasviel), enabling rapid community adoption and extensions. There is no formal standardization body for ControlNet, as it is a research-driven innovation rather than an industry protocol; however, its integration into popular frameworks like Automatic1111's Stable Diffusion WebUI and contributions from organizations like Stability AI have established it as a de facto standard for controllable image generation in the AI community. The paper was formally presented at the 2023 IEEE/CVF International Conference on Computer Vision (ICCV) in Paris, France, from October 2–6, 2023, highlighting its impact on generative AI.

Versions and Evolution

The initial version of ControlNet, often referred to as version 1.0, was released in early 2023 alongside the preprint, supporting a range of control types including Canny edges, depth maps, human poses, and segmentation, trained on datasets from under 50,000 to over 1 million samples. It featured "zero convolutions" for stable finetuning and was designed for compatibility with v1.5. Community features like low VRAM modes and non-prompt generation were added in February 2023 updates to the repository. In May 2023, ControlNet 1.1 was released as an improved iteration, focusing on better efficiency, reduced artifacts, and enhanced performance in multi-control scenarios, with pretrained models for 1.5 and 2.x. This version addressed limitations in the original by optimizing the trainable copy of the backbone, leading to higher fidelity in conditioned outputs. By November 2023, the paper reached version 3, incorporating minor revisions and supplementary materials. Evolution continued into 2024 with adaptations for newer base models. On November 26, 2024, Stability AI released three ControlNet models tailored for 3.5 Large: Blur for high-fidelity upscaling, Canny for edge-based structuring, and Depth for spatial guidance using depth maps generated by DepthFM. These extensions under the Stability AI Community License expanded ControlNet's applicability to advanced workflows like 8K image tiling and 3D texturing, while maintaining compatibility with the original architecture. As of November 2025, ongoing community contributions on continue to refine models for emerging diffusion systems like SDXL and , ensuring ControlNet's relevance in controllable generative AI.

Architecture

ControlNet is built upon a pretrained text-to-image , such as , by adding a trainable copy of the model's core components while locking the original pretrained weights. This design reuses the robust encoding layers of the base model—pretrained on billions of images—as a stable backbone, without modifying the diffusion process or text conditioning. The architecture primarily augments the denoising network, which consists of 25 blocks: 12 encoding blocks operating at resolutions of 64×64, 32×32, 16×16, and 8×8, followed by a middle block at 8×8 resolution, and 12 decoding blocks. ControlNet applies modifications only to the encoding blocks and the middle block to inject spatial controls efficiently.

Core Components

At the heart of ControlNet are the control models, which encode additional spatial conditions—such as edge maps (e.g., Canny edges), human poses (e.g., OpenPose), depth maps, segmentation masks, or scribbles—into feature maps compatible with the . Each control type uses a lightweight encoder network E(⋅)E(\cdot), typically comprising four 4×4 convolutional layers, to process the input condition into a 64×64 feature vector. This encoded control signal is then integrated into the via skip-connections, allowing the model to condition generation on both text prompts and spatial inputs simultaneously. Multiple controls can be combined by concatenating their feature maps channel-wise before injection.

Zero Convolutions and Trainable Copies

To connect the locked pretrained blocks with their trainable counterparts, ControlNet employs " convolutions"—1×1 convolutional layers initialized with all weights and biases set to . For each targeted block (encoder and middle), a copy is created with initialized parameters, and the zero convolution ensures that during initial training steps, the added branch outputs , preventing any disruptive from interfering with the pretrained model's behavior. As training progresses, the parameters of the trainable copy and zero convolution grow gradually, enabling the control signal to influence the denoising process without destabilizing the finetuning. This approach maintains the original model's text-to-image capabilities while adding precise spatial guidance. The integration preserves the base model's structure: the output of each pretrained block is added to the output of its trainable copy (after the zero convolution) before passing to the next block. Only the encoder and middle blocks are duplicated and controlled; the decoder blocks remain unchanged from the pretrained model. This selective augmentation reduces computational overhead and leverages the pretrained decoder for high-fidelity image synthesis. ControlNet supports flexible deployment, including multi-control setups and optional text prompts, and has been shown to robustly on datasets ranging from under 50,000 to over 1 million samples, often converging in fewer than 10,000 steps.

Implementation and Configuration

Note: ControlNet is a legacy network protocol still supported in Rockwell Automation products as of , but EtherNet/IP is recommended for new installations.

Network Topology and Redundancy

ControlNet networks support several configurations to accommodate diverse industrial environments, including bus (trunkline/dropline with terminators at both ends), star (using active hubs or taps for centralized connections), and hybrid combinations such as tree structures. These layouts allow flexibility in node placement, with a maximum of 99 nodes per network and up to 20 segments enabled by . Ring topologies can also be implemented using specialized fiber for enhanced connectivity in looped designs. Redundancy in ControlNet is achieved through dual-cable media, consisting of primary (Channel A) and backup (Channel B) or lines, which provide automatic upon detection of a cable fault. Fault detection occurs via continuous signal monitoring by network interfaces, enabling seamless switching typically within one or a few network update times (NUT) for minimal disruption in real-time operations. This mechanism ensures by isolating faults without halting network traffic, supporting up to 10 in redundant configurations compared to 5 in non-redundant setups. Sizing ControlNet networks involves calculating the network update time (NUT), the minimum repetitive cycle for data transmission, based on the number of nodes, scheduled data volume, and requested packet intervals (RPIs). Each node can transmit approximately 500 bytes of scheduled per NUT, with the total NUT determined using tools like RSNetWorx for ControlNet to balance throughput and latency; for example, a network with 50 nodes and moderate data exchange might require a 5 ms NUT to maintain . Repeater placement is limited to prevent excessive propagation delay, capping at 5 (or 10 in redundant mode) between any two nodes across segments. Installation best practices emphasize robust grounding to mitigate (), following guidelines that recommend single-point grounding for the entire network shield to avoid ground loops. Segments should be isolated using to limit fault propagation, while maximum stub lengths for drop cables are restricted to 30 m to minimize signal reflections and maintain integrity, particularly in bus topologies. For scalability, ControlNet trunklines can extend up to 1000 m using RG-6 , adjusted downward by 16.3 m for each tap beyond the first two to account for . Large-scale plants can bridge multiple ControlNet networks via gateways or modules in ControlLogix systems, enabling interconnection without exceeding per-network node limits and supporting expansion across facilities.

Communication Protocols and Tools

ControlNet employs two primary messaging types to facilitate real-time industrial communications: scheduled and unscheduled. Scheduled messaging supports cyclic (I/O) data exchange through a producer-consumer model, where producers broadcast such as status updates or control signals to multiple consumers at deterministic intervals defined by the Network Update Time (NUT). This ensures repeatable delivery for time-critical applications like and , utilizing up to 500 bytes per NUT per node via produced and consumed tags in controllers such as ControlLogix. In contrast, unscheduled messaging handles non-time-critical explicit communications, such as reading or writing device attributes, using the (CIP) message instructions; these transfers occur opportunistically during available bandwidth via the Unconnected Message Manager (UCMM), supporting operations like program uploads without disrupting scheduled traffic. Network configuration begins with node addressing, which can be set manually using rotary switches on modules (ranging from 01 to 99) or dynamically via software tools like RSNetWorx for ControlNet, ensuring unique identifiers across up to 99 nodes per segment. RSNetWorx facilitates scheduling by optimizing the NUT—the fundamental periodic cycle for data transfers, typically 2–100 ms—to balance scheduled data volume against available bandwidth; users define maximum scheduled (SMAX) and unscheduled (UMAX) node addresses, insert connections for produced/consumed tags, and generate a valid schedule file (*.xc) that is downloaded to the network keeper, such as a PLC-5C or ControlLogix controller. This process reserves bandwidth for unscheduled messaging, often set to 20–50% to prevent overruns, and includes auto-insertion of I/O connections for efficient setup. Diagnostic tools for ControlNet include Rockwell Automation's ControlNet Traffic Analyzer, a Windows-based application that captures and analyzes network packets in listen-only mode using a proprietary ControlNet ASIC and driver, displaying frames in MAC, LPacket, or interpreted formats with triggers and filters for targeted ; it is incompatible with due to its specialized hardware requirements, such as the 1784-PCC card. Module-level diagnostics rely on LED indicators: the Module Status (MS) LED shows solid green for normal I/O transfer, flashing green for operational but idle states, solid red for hardware faults or duplicate addresses, and flashing red for issues; Network (NET A/B) LEDs indicate steady green for active links, flashing red for no activity or media faults, and alternating red/green for configuration errors or self-test modes, aiding quick identification of link status and errors. ControlNet integrates CIP Safety extensions to enable fail-safe communications up to Safety Integrity Level 3 (SIL 3), allowing safety-rated devices like GuardLogix controllers to exchange verified data with integrity checks, preventing unsafe states during faults; this is achieved through CIP Safety profiles that embed safety parameters within standard CIP messages. Gateway support via CIP routing in devices like the ControlLogix ControlNet interface enables bridging to and networks, facilitating data exchange across heterogeneous CIP-based systems without protocol translation overhead. Common troubleshooting scenarios involve NUT overruns, where excessive scheduled data exceeds the cycle time, leading to missed updates—resolved by increasing the NUT or reducing connections in RSNetWorx to stay under 100% bandwidth utilization. Cable faults manifest as non-green NET LEDs or no activity, often due to improper termination, excessive length, or signal degradation; verification includes resistance checks (82–120 ohms) and segment isolation. Recovery from bandwidth constraints prioritizes reserving unscheduled capacity in RSNetWorx (e.g., via UMAX settings) to accommodate explicit messaging without impacting determinism, with tools like the Traffic Analyzer confirming resolution through packet analysis.

Applications and Comparisons

Applications in Creative and Technical Fields

ControlNet has seen widespread adoption in , where edges, poses, or depth maps are extracted from drawings and the AI follows these exactly (e.g., lineart dictates anatomy), enabling artists to generate images from sketches, edge maps, or segmentation masks while maintaining stylistic consistency with text prompts. For example, edge-based synthesis allows users to convert rough drawings into detailed illustrations, facilitating iterative creative workflows in tools like web UIs. In , ControlNet supports pose-guided by using OpenPose models to replicate human or creature poses in generated assets, improving pose control in Stable Diffusion image generation through precise guidance of human keypoints for applications in character design and animation; this aids in prototyping environments and cutscenes without manual keyframing. This is particularly useful for indie developers creating diverse character variations efficiently. In tasks, ControlNet generates structured outputs such as depth maps or normal maps from text descriptions, enhancing applications in and . Architectural visualization benefits from depth and segmentation controls to produce realistic building renders that adhere to spatial constraints. As of November 2024, Stability AI released ControlNet models for 3.5 Large, including Canny (), Depth, and Blur variants, expanding its utility in high-resolution image generation for professional design pipelines. Case studies highlight its impact: In , OpenPose integration allows generation of garment prototypes on virtual models, speeding up trend exploration. Animation studios have used it for storyboarding, combining pose and depth controls to visualize scenes rapidly. These applications demonstrate ControlNet's role in bridging generative AI with practical tools, supporting workflows from concept to final output. ControlNet shares conceptual similarities with other conditioning architectures for models but differs in and . Compared to T2I-Adapter, which adds lightweight adapters for controls like sketches or poses, ControlNet employs full copy-of-UNet modules with zero convolutions for deeper integration, offering greater flexibility and accuracy at the cost of higher computational demands—ControlNet processes every step, while T2I-Adapter runs once overall, making the latter faster for real-time applications. Evaluations show ControlNet superior in preserving fine details for complex conditions, though T2I-Adapter suffices for simpler tasks with reduced VRAM usage. Relative to IP-Adapter, which focuses on image-prompt conditioning for style or subject transfer without spatial maps, ControlNet excels in precise spatial guidance (e.g., edges, poses) but requires additional preprocessing for inputs. IP-Adapter, often combined with ControlNet in SDXL workflows, provides broader prompt adherence via CLIP features, achieving comparable quality in subject consistency while being lighter on resources. Both support variants, but ControlNet's robustness across datasets—from small pose sets to large scenic corpora—makes it preferable for controlled generation in technical domains. Other models like GLIGEN enable grounded text-to-image with location priors, contrasting ControlNet's non-textual controls; GLIGEN integrates directly with layouts but lacks ControlNet's modularity for multiple inputs. Overall, ControlNet's design balances power and preservation of pretrained capabilities, positioning it as a foundational tool for extensible conditioning in generative AI as of 2025.

References

  1. https://arxiv.org/abs/2302.05543
  2. https://github.com/lllyasviel/ControlNet
  3. https://openaccess.thecvf.com/content/ICCV2023/papers/Zhang_Adding_Conditional_Control_to_Text-to-Image_Diffusion_Models_ICCV_2023_paper.pdf
  4. https://lllyasviel.github.io/lvmin_zhang/
  5. https://stability.ai/news/sd3-5-large-controlnets
  6. https://github.com/lllyasviel/ControlNet-v1-1-nightly
  7. https://arxiv.org/abs/2302.05543v3
  8. https://www.rockwellautomation.com/en-us/support/product/product-compatibility-migration/product-lifecycle-status.html
  9. https://support.rockwellautomation.com/app/answers/answer_view/a_id/1139405
  10. https://literature.rockwellautomation.com/idc/groups/literature/documents/um/cnet-um001_-en-p.pdf
  11. https://literature.rockwellautomation.com/idc/groups/literature/documents/td/1786-td008_-en-p.pdf
  12. https://literature.rockwellautomation.com/idc/groups/literature/documents/in/cnet-in001_-en-p.pdf
  13. https://literature.rockwellautomation.com/idc/groups/literature/documents/in/cnet-in002_-en-p.pdf
  14. https://www.odva.org/wp-content/uploads/2020/05/PUB00200R1-Tech-Series-ControlNet.pdf
  15. https://literature.rockwellautomation.com/idc/groups/literature/documents/gr/cnet-gr001_-en-e.pdf
  16. https://literature.rockwellautomation.com/idc/groups/literature/documents/rm/9220-rm052_-en-p.pdf
  17. https://literature.rockwellautomation.com/idc/groups/literature/documents/um/recomm-um005_-en-e.pdf
  18. https://www.odva.org/technology-standards/distinct-cip-services/cip-safety/
  19. https://marketplace.odva.org/products/835-controllogix-controlnet-interface
  20. https://support.rockwellautomation.com/app/answers/answer_view_spider/a_id/22062
  21. https://stable-diffusion-art.com/controlnet/
  22. https://huggingface.co/lllyasviel/sd-controlnet-openpose
  23. https://www.tencentcloud.com/techpedia/125071
  24. https://www.reddit.com/r/StableDiffusion/comments/14p8a9f/interesting_use_cases_of_controlnet_besides_art/
  25. https://arxiv.org/abs/2302.08453
  26. https://comfyanonymous.github.io/ComfyUI_examples/controlnet/
  27. https://blog.finegrain.ai/posts/supercharge-stable-diffusion-ip-adapter/
Add your contribution
Related Hubs
User Avatar
No comments yet.