Video scaler
View on Wikipediafrom Wikipedia
A video scaler is a system that converts video signals from one display resolution to another; typically, scalers are used to convert a signal from a lower resolution (such as 480p standard definition) to a higher resolution (such as 1080i high definition), a process known as "upconversion" or "upscaling" (by contrast, converting from high to low resolution is known as "downconversion" or "downscaling").
Video scalers are typically found inside consumer electronics devices such as televisions, video game consoles, and DVD or Blu-ray players, but can also be found in other AV equipment (such as video editing and television broadcasting equipment). Video scalers can also be completely separate devices, often providing simple video switching capabilities. These units are commonly found as part of home theatre or projected presentation systems. They are often combined with other video processing devices or algorithms to create a video processor that improves the apparent definition of video signals.
Video scalers are primarily a digital device; however, they can be combined with an analog-to-digital converter (ADC, or digitizer) and a digital-to-analog converter (DAC) to support analog inputs and outputs.
Process
[edit]
The native resolution of a display is how many physical pixels make up each row and column of the visible area on the display's output surface.[1] There are many different video signals in use which are not the same resolution (neither are all of the displays), thus some form of resolution adaptation is required to properly frame a video signal to a display device. For example, within the United States, there are NTSC, ATSC, and VESA video standards, each with several different resolution video formats. Multiple common resolutions are also used for high-definition television; 720p, 1080i, and 1080p.
While scaling a video signal does allow it to match the size of a particular display, the process can result in an increased number of visual artifacts in the signal, such as ringing and posterization.[citation needed]
Scaling by television channels
[edit]Television channels that air a mixture of 16:9 (or high-definition) programming and 4:3 (or standard definition) programming may employ scaling or cropping in order to make the programming fill the entire screen, as opposed to pillarboxing the feed instead, in order to maintain consistency in format.[2][3] Likewise, broadcasters downscale programming produced in 16:9 for broadcast on their 4:3 feeds through letterboxing—either as a full 16:9 letterbox, or a partial 14:9 letterbox—a technique used primarily by European broadcasters during the transition to digital terrestrial television.[4][5] The Active Format Description standard is a system of variables defining various scaling, letterboxing, and pillarboxing states; broadcasting equipment and televisions can be configured to automatically switch to the appropriate state based on the AFD flag encoded in the content and the aspect ratio of the display.[6]
When the U.S. cable network TNT introduced an HD feed in 2004, it controversially employed a stretching system known as FlexView (which was also offered to other broadcasters). FlexView used a nonlinear method to stretch more near the edges of the screen than in the center of it. The practice was imposed by the senior vice president of broadcast engineering at TNT, Clyde D. Smith, who argued that pillarboxing led to inconsistency between programs for viewers, could cause burn-in on plasma televisions, some older HDTVs could not stretch 4:3 content automatically, and the quality of stretching on some displays was poor.[2] Despite TNT's intentions, the system was frequently criticized by viewers of high definition channels, with some nicknaming the effect "Stretch-O-Vision".[2][7]
In 2014, FXX faced similar criticism for its use of cropping and scaling on reruns of The Simpsons (which only started producing episodes in HD beginning in its 20th season), as its cropping method caused various visual gags to be lost. In February 2015, FXX announced that in response to these complaints, it would present these episodes in their original 4:3 aspect ratio on its video-on-demand service.[3][8]
Since about 2008, some networks and cable companies have run high-definition versions of old programming that was originally shot on film and shown in the 4:3 format. This format always cropped the sides of the image. For example, syndicated broadcast stations and the cable network TBS air Seinfeld in HD. The series producer, Sony Pictures, went to the 35 mm film source, making new HD masters and cropping the top and bottom parts of the frame, while restoring the sides.[9]
AI upscaling
[edit]In 2023, video upscaling products using generative artificial intelligence were previewed or released, such as NVIDIA's Video Super Resolution and Adobe's Project Res-Up. Unlike previous upscaling systems, these technologies generated new data based on provided video frames.[10][11][12]
See also
[edit]References
[edit]- ^ "Video Upscaling for better viewing experience". hometoys.com. 2011-05-25. Archived from the original on 2015-08-25. Retrieved 2013-03-07.
- ^ a b c "TNT Stretches for HD". TV Week. Retrieved 2008-08-10.
- ^ a b "FXX will finally stream The Simpsons in original 4:3 format". The Verge. 13 February 2015. Retrieved 13 February 2015.
- ^ Bob Wolfley (September 2, 2010). "Fox Sports taking a wider view of football". Milwaukee Journal Sentinel. Journal Communications. Archived from the original on September 5, 2010.
- ^ "Beyond HD". BBC Academy. Archived from the original on December 19, 2014. Retrieved 3 June 2015.
- ^ "Managing AFD: Keep image format under your control". TV Technology. January 2009. Retrieved 5 June 2016.
- ^ "Just say no to stretch-o-vision". EngadgetHD. Archived from the original on 2008-07-24. Retrieved 2008-08-10.
- ^ "FXX ruins the punchline by inexplicably cropping old standard definition 'Simpsons' episodes". The Verge. 24 August 2014. Retrieved 13 February 2015.
- ^ "Seinfeld goes HD on TBS HD". AOL. September 3, 2008. Archived from the original on January 16, 2013. Retrieved July 24, 2019.
- ^ Anderson, Ross (9 December 2023). "The Best AI Tool You're Not Using is Video Upscaling". The New York Sun.
- ^ Weatherbed, Jess (11 October 2023). "Adobe previews AI upscaling to make old, fuzzy videos and GIFs look fresh". The Verge.
- ^ Chacos, Brad (28 February 2023). "Tested: Nvidia's RTX Video Super Resolution is like going from VHS to Blu-ray". PC World.
Video scaler
View on Grokipediafrom Grokipedia
Overview
Definition and Purpose
A video scaler is a system or algorithm designed to convert video signals from one resolution to another by either upscaling—increasing the pixel count to a higher resolution—or downscaling—reducing the pixel count to a lower resolution—while striving to preserve overall image quality.[7][8] This process involves resampling the video frames to adjust their dimensions, ensuring the output aligns with the target display's capabilities without introducing excessive artifacts like blurring or aliasing.[7] The fundamental purpose of a video scaler is to bridge compatibility gaps between video sources and playback devices that operate at varying resolutions, such as adapting standard definition (SD) footage for high definition (HD) displays.[9] It enables enhanced viewing on higher-resolution screens by optimizing content delivery and supports seamless format conversions within production workflows, ultimately improving user experience across diverse platforms.[8] By facilitating these adjustments, scalers help maintain visual fidelity in applications ranging from broadcasting to personal media consumption.[10] Central to video scaling are concepts like resolution, which defines the number of pixels in a frame—for instance, 1080p resolution consists of 1920 pixels horizontally by 1080 pixels vertically—allowing precise specification of input and output frame sizes.[11] Scalers generally preserve the aspect ratio, the proportional width-to-height relationship of the video (commonly 16:9 for modern formats), to avoid stretching or squeezing the image, which could distort visual elements.[12] This preservation ensures that changes in frame dimensions do not compromise the intended composition of the content. In practical scenarios, video scalers are integral to streaming platforms, where they dynamically adjust resolutions to suit end-user devices, such as downscaling 4K videos for smartphones to conserve bandwidth or upscaling SD clips for 4K televisions to fill larger screens effectively.[8]Historical Development
The origins of video scaling trace back to analog video processing in the 1970s and 1980s, when broadcast equipment was developed to handle conversions between incompatible television standards like NTSC (used primarily in North America and Japan) and PAL (prevalent in Europe and elsewhere). These early systems, often comprising large, rack-mounted analog standards converters, performed essential adjustments to frame rates, line resolutions, and color encoding to enable international content exchange and compatibility in professional broadcasting workflows. The shift to digital video in the 1990s introduced the first dedicated digital scalers, coinciding with the commercialization of DVD players and the groundwork for high-definition television (HDTV). DVD players, first released in Japan in November 1996 and in the U.S. shortly thereafter, incorporated digital scaling circuitry to adapt standard-definition (SD) content—typically 480i or 576i—to various display outputs, marking a key step in consumer-level video processing. Early HDTV efforts, building on U.S. demonstrations from 1981, accelerated with the transition to all-digital systems by 1991, culminating in the FCC's approval of the ATSC digital broadcasting standard in December 1996. A foundational milestone was the ITU-R BT.601-5 recommendation in October 1995, which standardized digital sampling for studio television signals at 13.5 MHz for luminance (Y) and half that for color-difference components (Cb and Cr) in a 4:2:2 format, providing a common framework for accurate digital encoding, decoding, and scaling across 525-line and 625-line systems.[13][14] The 2000s saw widespread adoption of high-definition formats, driving further refinements in video scalers to support resolutions like 720p (progressive scan at 1280x720) and 1080i (interlaced at 1920x1080), which became core to ATSC broadcasts and consumer electronics. By 2002, an industry memorandum outlined phased rollouts for digital TV and HDTV compatibility, integrating scalers into set-top boxes and displays to upscale SD content for HD screens. In the 2010s, streaming services amplified these demands; Netflix's initiation of 4K Ultra HD streaming in early 2014, beginning with its original series House of Cards, required sophisticated scaling to deliver 2160p content adaptively across bandwidth-variable networks and heterogeneous devices, pushing the boundaries of real-time upscaling and downscaling algorithms.[14][15] Underlying this progression was a transition from specialized hardware-only scalers to integrated software solutions, propelled by Moore's Law—the observation that the number of transistors on a microchip roughly doubles every two years, enabling exponential gains in computational density and efficiency—and parallel advancements in graphics processing units (GPUs). These factors allowed video scaling to migrate toward programmable, GPU-accelerated implementations in the mid-2000s onward, facilitating more versatile handling of complex resolutions in both professional and consumer applications.Scaling Techniques
Spatial Scaling Methods
Spatial scaling methods in video processing adjust the resolution of individual frames by interpolating pixel values in the two-dimensional spatial domain, independent of temporal information across frames. These techniques are essential for upscaling low-resolution video to higher displays or downscaling high-resolution content for storage or transmission efficiency. Basic methods prioritize simplicity and speed, while advanced approaches aim for higher fidelity at the expense of computational resources. The simplest spatial scaling method is nearest-neighbor interpolation, which maps each output pixel to the nearest input pixel without blending, resulting in a blocky appearance particularly noticeable in upscaling scenarios due to aliasing and jagged edges.[16] This approach is computationally inexpensive, requiring only integer coordinate mapping, making it suitable for real-time applications where quality is secondary to performance.[17] Bilinear interpolation improves upon nearest-neighbor by averaging the intensities of the four nearest input pixels (a 2x2 neighborhood) weighted by their fractional distances, producing smoother transitions and reducing blockiness. The output value at coordinates (x, y) is calculated as:
where and are the fractional parts of the offsets in the x and y directions, respectively, and are the input pixel intensities.[16] This method offers a reasonable trade-off for general video scaling, though it can blur fine details during downscaling.[17]
Advanced techniques like bicubic interpolation extend bilinear by considering a 4x4 neighborhood of 16 pixels, using a cubic polynomial kernel to preserve sharper edges and finer details, which is particularly beneficial for upscaling. The interpolation kernel, often parameterized by a value (cubic convolution), is applied separably in x and y directions for efficiency.[18] Bicubic methods yield higher quality than bilinear but require significantly more computations.[17]
Lanczos resampling employs a sinc-based kernel for superior frequency response, making it ideal for downscaling video where anti-aliasing is critical to avoid ringing artifacts while retaining sharpness. The kernel is defined as for , where and (typically 2 or 3) controls the kernel width, with larger improving quality at higher cost. This method excels in preserving high-frequency content but introduces more ringing than polynomial-based approaches and demands significant computation due to the larger support.[17]
Overall, these methods balance computational cost against output quality: nearest-neighbor and bilinear favor low latency for real-time video playback, while bicubic and Lanczos prioritize perceptual sharpness, with Lanczos particularly suited to downscaling and bicubic to upscaling. Selection depends on the scaling factor, content type, and hardware constraints, as upscaling amplifies interpolation errors more than downscaling.[17]
Temporal Scaling Methods
Temporal scaling methods in video processing focus on adjusting the frame rate or temporal resolution by leveraging inter-frame dependencies, distinguishing them from spatial scaling that operates independently on individual frames. These techniques analyze motion across consecutive frames to maintain temporal consistency, enabling conversions such as from 24 frames per second (fps) in film content to 60 fps for smoother playback on modern displays. Basic temporal scaling can be achieved through frame duplication or dropping, where existing frames are repeated to increase the rate or selectively omitted to decrease it, providing a low-complexity solution but often resulting in visible artifacts like stuttering. More sophisticated approaches rely on motion-compensated interpolation, which estimates object motion between frames to synthesize new intermediate frames, thereby enhancing perceived smoothness without relying solely on repetition.[19] Motion estimation in these methods commonly employs block matching, where video frames are divided into small blocks (e.g., 8×8 or 16×16 pixels), and a motion vector is determined for each block by searching for the best match in a reference frame. The optimal vector minimizes the sum of absolute differences (SAD), defined as
where and represent pixel intensities in the current and reference frames, is the motion vector, and is the block dimension.[20] Once estimated, these vectors guide the warping of pixels to create interpolated frames, reducing discontinuities in motion. For finer-grained analysis, optical flow algorithms compute dense, pixel-level motion fields by solving equations that model brightness constancy and spatial smoothness, allowing for more accurate prediction of intermediate frame content in complex scenes.[21]
In practice, temporal scaling via motion compensation is widely applied to mitigate judder—the uneven motion perception arising from mismatched frame rates, such as when 24 fps cinematic content is displayed at 60 Hz—by inserting interpolated frames that align with display refresh rates. Similarly, these methods smooth the downscaling of high-frame-rate sources (e.g., 120 fps action footage to 30 fps) by adaptively blending frames, preserving dynamic details without abrupt drops.[19]
Implementation
Hardware Scalers
Hardware scalers refer to dedicated physical devices and integrated chipsets engineered for real-time video resolution conversion, leveraging specialized architectures to minimize latency and maximize throughput in demanding environments. These systems process video signals directly in hardware, bypassing the overhead of general-purpose computing, which enables seamless integration into pipelines where timing is critical, such as live production or embedded displays. Primary implementations include Application-Specific Integrated Circuits (ASICs) for fixed, high-volume applications and Field-Programmable Gate Arrays (FPGAs) for customizable, professional-grade setups.[22][23] ASIC-based scalers are commonly embedded in consumer television System-on-Chips (SoCs), where GPU hardware accelerators handle spatial upscaling tasks such as converting 1080p to 4K resolutions in embedded systems prioritizing power efficiency through minimized memory access. In contrast, FPGA implementations dominate professional broadcast gear, utilizing reconfigurable logic to execute polyphase filtering algorithms that support arbitrary scaling factors and resolutions up to 4K, as provided by Xilinx's Video Processing Subsystem IP cores. These FPGA solutions allow for on-the-fly adjustments via hardware description languages, making them suitable for dynamic broadcast workflows requiring custom video pipelines.[22][23] Historically, hardware scalers emerged in the 1990s with Faroudja Laboratories' line doublers, such as the LD200 model, which de-interlaced NTSC signals by doubling scan lines to produce progressive 480p output, significantly enhancing vertical resolution on early large-screen displays. Founded in 1971 in Silicon Valley, Faroudja's innovations, including the award-winning LD100 (1993 Advanced Technology Award), laid foundational techniques for motion-adaptive line doubling still influential today. Modern examples include HDMI scalers integrated into AV receivers, like those in Denon models, which upscale analog or lower-resolution HDMI inputs to 4K@60Hz for home theater compatibility.[24][25][26] Key features of hardware scalers emphasize real-time capabilities, such as processing 4K@60Hz streams with 10-bit color depth and latency of 1 second, as demonstrated by devices like Extron's Quantum Ultra videowall processor. However, these scalers are constrained by fixed functionality, offering less adaptability than software alternatives for iterative or non-real-time adjustments, and high-end professional units often cost over $500, such as Atlona's AT-HDVS-150-RX scaler-receiver.[27][28][29]Software Scalers
Software scalers perform video resizing using general-purpose processors, enabling algorithmic flexibility and integration into diverse applications without dedicated hardware. These implementations process frames through libraries that apply interpolation techniques to adjust resolution, aspect ratio, or format, often prioritizing computational efficiency for both real-time and offline use.[30] Prominent open-source frameworks include FFmpeg's libswscale library, which provides optimized scaling for video and image data, supporting algorithms such as bilinear, bicubic, Lanczos, and spline.[31] Libswscale handles pixel format conversions alongside scaling, making it suitable for media processing pipelines in tools like encoders and players.[32] For GPU acceleration within software environments, FFmpeg incorporates CUDA-based filters like scale_npp, which leverage NVIDIA hardware for parallelized resizing operations.[33] OpenCL support in similar libraries enables cross-vendor GPU utilization, allowing shaders to compute interpolated pixel values across frames in a distributed manner. Implementation contrasts CPU-based processing, which excels in sequential tasks like format adjustments, with GPU methods that parallelize pixel computations for faster throughput on high-resolution content.[34] In offline video editing, batch processing queues multiple frames for optimized execution, reducing overhead by amortizing initialization costs across operations.[35] Software scalers often include hooks for hardware acceleration, blending programmable logic with specialized pipelines for hybrid efficiency. Examples of software scalers in applications include Adobe Premiere Pro's scaling filters, which apply bicubic or adaptive methods accelerated by CUDA or OpenCL for real-time previews and exports, achieving up to 40 times faster performance than CPU-only modes in quality-critical tasks.[36] VLC media player integrates libswscale-derived scaling for playback, supporting modes like Lanczos (SincR) to resize videos dynamically while maintaining aspect ratios. Advantages of software scalers encompass customizability, permitting algorithm tweaks and updates via software releases without hardware redesigns, alongside seamless adaptability to emerging formats.[37] Drawbacks include elevated latency for live streams, where CPU reliance can bottleneck real-time demands compared to GPU-optimized paths.[38]Applications
In Television Broadcasting
In television broadcasting, video scalers are integral to production and distribution workflows, particularly for upconverting legacy standard-definition (SD) content to high-definition (HD) or 4K ultra-high-definition (UHD) formats to meet modern broadcast standards. Broadcasters routinely upscale archived SD footage—often stored in formats like 480i or 576i—to HD resolutions such as 1080p or 1080i, ensuring compatibility with contemporary transmission pipelines and viewer expectations for sharper imagery. This process is especially prevalent in file-based workflows, where scalers apply deinterlacing and spatial interpolation to preserve motion integrity while expanding resolution, allowing seamless integration of historical material into current programming.[39] Standards like ATSC 3.0 further emphasize scalable video feeds, enabling broadcasters to transmit layered content that supports multiple resolutions within a single stream, from SD base layers to enhanced HD or 4K extensions.[40] This scalability facilitates efficient delivery over terrestrial, cable, and satellite networks, where receivers can decode only the necessary layers based on device capabilities, reducing the need for separate SD and HD versions of the same program. Specific practices in broadcasting include channel-side scaling during live events, where incoming 1080p signals from cameras or remote feeds are upconverted to 4K for premium outputs while maintaining real-time processing. For instance, during major sports coverage, scalers handle resolution boosts to support multi-platform distribution, such as simulcasting in both HD and UHD.[41] In satellite and cable systems, multi-format output is common, with scalers generating simultaneous SD, HD, and 4K variants from a single master feed to accommodate diverse headends and subscriber equipment, optimizing bandwidth across transponders or multiplexes.[42] Challenges in these applications arise from bandwidth constraints and the demands of real-time operation, as upscaling high-bitrate live feeds can exceed available spectrum in traditional multiplexes, necessitating compression trade-offs that may introduce artifacts like aliasing. Economically, reusing scaled archives offers significant cost savings by avoiding the expense of new shoots, with broadcasters leveraging upconversion to monetize vast libraries—estimated to generate ROI through repurposed content streams—though it involves trade-offs in perceptual quality compared to native high-resolution productions.[43][44]In Consumer Devices
Video scalers are integral to consumer electronics, enabling seamless playback of content across diverse display sizes and resolutions in devices like smart TVs, smartphones, and streaming media players. These scalers automatically adjust video signals to match the native resolution of the device, preventing distortion and ensuring compatibility with varying hardware capabilities. For instance, in smart TVs, processors such as Samsung's NQ8 AI Gen3 in QLED models upscale lower-resolution sources like 1080p to 8K using AI-based algorithms that analyze and enhance image details for sharper output.[45] Similarly, mobile devices employ software scalers in video players to adapt content for screens ranging from compact smartphone displays (e.g., 1080x1920 pixels) to larger tablets, maintaining aspect ratios and fitting videos without black bars or cropping.[46] User-facing features in these devices often include customizable settings that influence scaling outcomes, allowing individuals to fine-tune visual quality. On smart TVs, options for sharpness and picture modes—accessible via menus like Settings > Picture > Expert Settings—let users adjust edge enhancement during upscaling, with recommendations to set sharpness to zero or low levels to avoid artificial halos and preserve natural detail.[47][48] Streaming integrations, such as the YouTube app, perform client-side scaling to the device's resolution (e.g., downscaling 4K streams to 1080p on mid-range phones), optimizing bandwidth while filling the screen appropriately.[49] In the 2020s, built-in scalers in consumer devices have evolved to handle not only resolution adjustments but also dynamic range conversions, such as tone mapping HDR content to SDR displays for broader compatibility. Modern smart TVs incorporate hardware-accelerated tone mapping to compress HDR's expanded brightness and color ranges into SDR without losing key visual intent, a feature standard in models from manufacturers like Samsung and LG since the mid-2010s.[50] This integration supports playback of mixed-format libraries on everyday hardware, reducing the need for external converters. Beyond technical performance, video scaling enhances accessibility by enabling universal content consumption, particularly through the preservation of subtitles during resizing. Closed caption tracks, rendered as overlays in players like VLC or native TV apps, scale proportionally with the video to maintain readability and positioning, ensuring that text remains legible on smaller mobile screens or during aspect ratio adjustments—critical for users with hearing impairments or in noisy environments.[51] This approach complies with guidelines from organizations like the W3C, promoting inclusive viewing without requiring separate content versions.[52]Quality Assessment
Metrics for Scaler Quality
The quality of video scalers is evaluated using both objective and subjective metrics to assess how well they preserve fidelity during resolution changes, such as upscaling or downscaling. Objective metrics provide quantifiable measures based on mathematical comparisons between original and scaled videos, while subjective metrics incorporate human perception to gauge visual acceptability. These evaluations help balance key factors like sharpness enhancement, reduction of aliasing artifacts, and computational efficiency in real-time applications.[53] Objective metrics are widely used for their reproducibility and automation. The Peak Signal-to-Noise Ratio (PSNR) is a fundamental full-reference metric that quantifies the difference between the original and scaled video frames by measuring pixel-level errors.
Here, is the maximum possible pixel value (e.g., 255 for 8-bit images), and is the mean squared error between corresponding pixels. Higher PSNR values indicate better fidelity, with typical targets exceeding 30 dB for upscaling scenarios like DVD (480p) to 4K, where bicubic interpolation often achieves around 28-30 dB and advanced methods surpass 32 dB on standard benchmarks.[54]
Another key objective metric is the Structural Similarity Index (SSIM), which evaluates perceived quality by comparing luminance, contrast, and structural features rather than just pixel errors. SSIM ranges from -1 to 1, with values closer to 1 denoting higher similarity; it correlates better with human judgment than PSNR for natural video content. Multi-scale variants like MS-SSIM further adapt it for varying resolutions.[55][53]
Video Multimethod Assessment Fusion (VMAF) is another prominent objective metric specifically designed for video quality assessment, combining multiple features such as detail loss, motion, and perceptual sensitivity. It shows strong correlation with subjective scores (Spearman rank correlation often above 0.9) and is widely used in streaming and scaling evaluations.[56]
Subjective metrics capture human visual preferences, essential for validating objective scores. The Mean Opinion Score (MOS) involves panels of viewers rating scaled videos on a scale of 1 (poor) to 5 (excellent), averaged to yield MOS; ITU recommendations standardize this process for multimedia, ensuring controlled viewing conditions. For edge preservation, the Modulation Transfer Function (MTF) measures contrast transfer at different spatial frequencies, with higher MTF curves indicating better sharpness retention post-scaling—often derived from slanted-edge analysis in video frames. MOS and MTF together highlight perceptual trade-offs, such as over-sharpened edges reducing overall MOS despite high PSNR.[57]
Testing standards employ specialized patterns for rigorous assessment. The ISO 12233 chart, featuring slanted edges and frequency wedges, enables measurement of spatial frequency response (SFR), from which MTF and resolution limits are computed; it is applied to individual video frames to evaluate scaler performance across scales. Benchmarks using such charts target balanced outcomes, prioritizing PSNR/SSIM above 30 dB/0.85 while minimizing aliasing through efficient algorithms like Lanczos filtering, which achieves this with lower computational cost than neural methods.[58]