Hubbry Logo
DTraceDTraceMain
Open search
DTrace
Community hub
DTrace
logo
8 pages, 0 posts
0 subscribers
Be the first to start a discussion here.
Be the first to start a discussion here.
DTrace
DTrace
DTrace
View on Wikipedia
from Wikipedia
DTrace
Original authorsBryan Cantrill, Adam Leventhal, Mike Shapiro (Sun Microsystems)
DevelopersSun Microsystems, Oracle, Microsoft
Initial releaseJanuary 2005; 20 years ago (2005-01)
Repositorygithub.com/opendtrace
Written inC
Operating systemSolaris, illumos, macOS, FreeBSD, NetBSD, Linux,[1] Windows[2]
TypeTracing
LicenseCDDL, GPLv2, UPL
Websitedtrace.org

DTrace is a comprehensive dynamic tracing framework originally created by Sun Microsystems for troubleshooting kernel and application problems on production systems in real time. Originally developed for Solaris, it has since been released under the free Common Development and Distribution License (CDDL) in OpenSolaris and its descendant illumos, and has been ported to several other Unix-like systems. Windows Server systems from Windows Server 2025 will have DTrace as part of the system.

DTrace can be used to get a global overview of a running system, such as the amount of memory, CPU time, filesystem and network resources used by the active processes. It can also provide much more fine-grained information, such as a log of the arguments with which a specific function is being called, or a list of the processes accessing a specific file.

In 2010, Oracle Corporation acquired Sun Microsystems and announced the discontinuation of OpenSolaris. As a community effort of some core Solaris engineers to create a truly open source Solaris, illumos operating system was announced via webinar on Thursday, 3 August 2010,[3] as a fork on OpenSolaris OS/Net consolidation, including DTrace technology.

In October 2011, Oracle announced the porting of DTrace to Linux,[4] and in 2019 official DTrace for Fedora is available on GitHub. For several years an unofficial DTrace port to Linux was available, with no changes in licensing terms.[5]

In August 2017, Oracle released DTrace kernel code under the GPLv2+ license, and user space code under GPLv2 and UPL licensing.[6] In September 2018 Microsoft announced that they had ported DTrace from FreeBSD to Windows.[2]

In September 2016 the OpenDTrace effort began on GitHub with both code and comprehensive documentation of the system's internals. The OpenDTrace effort maintains the original CDDL licensing for the code from OpenSolaris with additional code contributions coming under a BSD 2 Clause license. The goal of OpenDTrace is to provide an OS agnostic, portable implementation of DTrace that is acceptable to all consumers, including macOS, FreeBSD, OpenBSD, NetBSD, and Linux as well as embedded systems.

Description

[edit]

Sun Microsystems designed DTrace to give operational insights that allow users to tune and troubleshoot applications and the OS itself.

Testers write tracing programs (also referred to as scripts) using the D programming language (not to be confused with other programming languages named "D"). The language, inspired by C, includes added functions and variables specific to tracing. D programs resemble AWK programs in structure; they consist of a list of one or more probes (instrumentation points), and each probe is associated with an action. These probes are comparable to a pointcut in aspect-oriented programming. Whenever the condition for the probe is met, the associated action is executed (the probe "fires"). A typical probe might fire when a certain file is opened, or a process is started, or a certain line of code is executed. A probe that fires may analyze the run-time situation by accessing the call stack and context variables and evaluating expressions; it can then print out or log some information, record it in a database, or modify context variables. The reading and writing of context variables allows probes to pass information to each other, allowing them to cooperatively analyze the correlation of different events.

Special consideration has been taken to make DTrace safe to use in a production environment. For example, there is minimal probe effect when tracing is underway, and no performance impact associated with any disabled probe; this is important since there are tens of thousands of DTrace probes that can be enabled. New probes can also be created dynamically.

Command line examples

[edit]

DTrace scripts can be invoked directly from the command line, providing one or more probes and actions as arguments. Some examples: 

# New processes with arguments
dtrace -n 'proc:::exec-success { trace(curpsinfo->pr_psargs); }'

# Files opened by process
dtrace -n 'syscall::open*:entry { printf("%s %s",execname,copyinstr(arg0)); }'

# Syscall count by program
dtrace -n 'syscall:::entry { @num[execname] = count(); }'

# Syscall count by syscall
dtrace -n 'syscall:::entry { @num[probefunc] = count(); }'

# Syscall count by process
dtrace -n 'syscall:::entry { @num[pid,execname] = count(); }'

# Disk size by process
dtrace -n 'io:::start { printf("%d %s %d",pid,execname,args[0]->b_bcount); }'

# Pages paged in by process
dtrace -n 'vminfo:::pgpgin { @pg[execname] = sum(arg0); }'

Scripts can also be written which can reach hundreds of lines in length, although typically only tens of lines are needed for advanced troubleshooting and analysis. Over 200 examples of open source DTrace scripts can be found in the DTraceToolkit,[7] created by Brendan Gregg (author of the DTrace book[8]), which also provides documentation and demonstrations of each.

Supported platforms

[edit]

DTrace first became available for use in November 2003, and was formally released as part of Sun's Solaris 10 in January 2005. DTrace was the first component of the OpenSolaris project to have its source code released under the Common Development and Distribution License (CDDL).

DTrace is an integral part of illumos and related distributions.

DTrace is a standard part of FreeBSD[9] and NetBSD.[10]

Apple added DTrace support in Mac OS X 10.5 "Leopard", including a GUI called Instruments.[11] Over 40 DTrace scripts from the DTraceToolkit are included in /usr/bin,[12] including tools to examine disk I/O (iosnoop) and process execution (execsnoop). Unlike other platforms that DTrace is supported on, Mac OS X has a flag (P_LNOATTACH) that a program may set that disallows tracing of that process by debugging utilities such as DTrace and gdb. In the original Mac OS X DTrace implementation, this could affect tracing of other system information, as unrelated probes that should fire while a program with this flag set was running would fail to do so.[13] The OS X 10.5.3 update addressed this issue a few months later.[14] However, since El Capitan, System Integrity Protection prevents user from DTracing protected binary by default.

The Linux port of DTrace has been available since 2008;[15] work continues actively to enhance and fix issues. There is also an active implementation on github. Standard core providers are available (fbt, syscall, profile), plus a special "instr" provider (some of the Solaris providers are not yet available as of 2013). The Linux DTrace implementation is a loadable kernel module, which means that the kernel itself requires no modification, and thus allows DTrace to avoid CDDL vs. GPL licensing conflicts (in its source form, at least). However, once DTrace is loaded the kernel instance will be marked as tainted.

In 2007, a developer at QNX Software Systems announced on his blog that he and a colleague were working on incorporating DTrace into the QNX operating system.[16]

Oracle Corporation added beta DTrace support for Oracle Linux in 2011,[1] as a technology preview in the Unbreakable Enterprise Kernel release 2, which is under GPLv2 (the DTrace Linux kernel module was originally released under CDDL).[17] General availability was announced in December 2012.[18][19]

On March 11, 2019, Microsoft released a version of DTrace for Windows 10 insider builds.[20] Microsoft included DTrace as a built-in tool in Windows Server 2025.[21][22]

Language and application providers

[edit]

With a supported language provider, DTrace can retrieve context of the code, including function, source file, and line number location. Further, dynamic memory allocation and garbage collection can be made available if supported by the language.[23] Supported language providers include assembly language[clarification needed], C, C++, Java, Erlang, JavaScript, Perl, PHP, Python, Ruby, shell script, and Tcl.

Application providers allow DTrace to follow the operation of applications through system calls and into the kernel. Applications that offer DTrace application providers include MySQL, PostgreSQL, Oracle Database, Oracle Grid Engine, and Firefox.[23][24][25]

Authors and awards

[edit]

DTrace was designed and implemented by Bryan Cantrill, Mike Shapiro, and Adam Leventhal.

The authors received recognition in 2005 for the innovations in DTrace from InfoWorld and Technology Review.[26][27] DTrace won the top prize in The Wall Street Journal's 2006 Technology Innovation Awards competition.[28] The authors were recognized by USENIX with the Software Tools User Group (STUG) award in 2008.[29]

See also

[edit]
  • eBPF – Linux kernel tracing backend providing a set of features similar to DTrace[30] since kernel version 4.9
  • ftrace – a tracing framework for the Linux kernel, capable of tracing scheduling events, interrupts, memory-mapped I/O, CPU power state transitions, etc.
  • ktrace – a BSD Unix and macOS utility that traces kernel–program interactions
  • ltrace – a Linux debugging utility, displays the calls a userland application makes to shared libraries
  • strace – a debugging utility for Linux, monitors system calls used by a program and all received signals
  • SystemTap – a scripting language and utility used for instrumenting Linux installations
  • LTTng
  • IBM ProbeVue

References

[edit]
[edit]
Revisions and contributorsEdit on WikipediaRead on Wikipedia

DTrace

View on Grokipedia
from Grokipedia
DTrace is a comprehensive dynamic tracing framework designed for and of production software systems, enabling users to instrument both kernel and user-level code with minimal overhead and no need for reboots or source modifications. Developed by engineers , Adam Leventhal, and Mike Shapiro, it was first integrated into the Solaris 10 operating system upon its release in March 2005. As an open-source tool licensed under the (CDDL), DTrace has been ported to other platforms including FreeBSD, macOS (as part of OS X 10.5 Leopard and later), , and Windows. At its core, DTrace operates through a framework of probes, which serve as instrumentation points embedded in the operating system kernel, device drivers, and applications; these probes fire in response to system events and can be dynamically enabled to record detailed data such as function entry/exit points, variable values, timestamps, and stack traces. Users interact with DTrace via the D programming language—a C-like scripting language augmented with features for data aggregation, filtering, and printing—allowing the definition of predicates (conditional expressions to selectively activate probes) and actions (code executed upon probe firing, such as incrementing counters or logging state). This enables targeted exploration of system behavior, from identifying bottlenecks in web servers to debugging kernel panics, all while maintaining production stability through built-in safety mechanisms that prevent kernel crashes or infinite loops. The architecture of DTrace separates concerns into providers (kernel modules that expose probes, such as fbt for function boundary tracing or io for I/O events), a central kernel facility for managing probe activation and data buffering, and user-space consumers like the dtrace command-line utility for scripting and output processing. Its low-overhead design—achieved through just-in-time compilation of scripts and per-CPU buffering—makes it suitable for live systems, supporting use cases from real-time diagnostics to postmortem analysis of crash dumps. DTrace's innovation earned it the top prize in the 2006 Wall Street Journal Technology Innovation Awards, recognizing its impact on system observability.

Overview

Core Concept

DTrace is a comprehensive dynamic tracing framework originally developed by for the Solaris operating system, enabling troubleshooting of kernel and application issues on live production systems without modifying binaries or incurring downtime. It provides a unified mechanism to instrument both user-level processes and kernel code, allowing administrators and developers to observe system behavior in real time for performance analysis and . Central to DTrace's design is its non-intrusive nature, achieved through dynamic that imposes zero overhead when probes are disabled and minimal impact when active, as only enabled probes execute code. Safety is ensured by built-in features such as predicates—conditional expressions that limit tracing to specific conditions—preventing system instability or corruption even during intensive use on production environments. This addresses key limitations of static tracing tools like and gdb, which require process attachment, generate significant overhead, and lack systemic scope suitable for broad production diagnostics. The core workflow of DTrace involves identifying and enabling instrumentation points called probes, provided by kernel modules or applications, which fire in response to system events; upon firing, user-defined actions in the D programming language aggregate data—such as timestamps, arguments, or metrics—into per-CPU buffers for real-time examination and summarization. This enables scalable, on-the-fly insights into complex interactions across the entire software stack, from individual threads to global resource usage.

Key Features

DTrace enables dynamic , permitting the attachment of probes to any kernel or user-space function at runtime without access to or recompilation. This capability allows for on-the-fly exploration of system behavior in production environments, where traditional static would be impractical. When not enabled, DTrace imposes zero overhead, as probes are compiled into binaries but remain inactive until explicitly activated by the framework. To ensure safe operation on live systems, DTrace incorporates built-in protections against instability, including the deadman mechanism, which monitors for excessive CPU usage or unresponsiveness induced by tracing and automatically aborts tracing to prevent system hangs. Additional safeguards include safe compilation into an intermediate form that prevents runtime errors like or invalid pointer dereferences from crashing the system, instead disabling affected probes and reporting errors. These features collectively minimize the risk of performance degradation or system failure during tracing. DTrace supports rich data aggregation through arbitrary actions tied to probes, including built-in statistical functions such as @count for counting occurrences, @sum for accumulating values, and higher-level aggregations for quantizing or averaging data across system events. This allows users to perform complex, on-the-fly analysis without post-processing large trace logs. Integration with the D programming language further enables custom scripting for tailored queries, such as filtering events by process ID or aggregating metrics by thread, facilitating deep insights into system dynamics. The framework's portability stems from its provider abstraction layer, which standardizes probe interfaces across diverse operating systems—including Solaris, , macOS, distributions, and Windows—and CPU architectures like x86, , and ARM, ensuring consistent functionality without vendor-specific modifications. This design promotes widespread adoption by abstracting underlying kernel differences, allowing scripts written for one platform to run with minimal adaptation on others.

History

Origins at Sun Microsystems

DTrace's development commenced in 2001 at , spearheaded by kernel engineer to tackle persistent production debugging challenges in the Solaris operating system, where traditional tools often required service interruptions or risked system instability. The effort was driven by the need for a dynamic tracing framework that could safely instrument live systems without overhead or risk, enabling real-time analysis of complex, componentized environments. This initiative drew partial inspiration from Solaris's earlier Trace Normal Form (TNF) framework, a user-level tracing tool introduced in Solaris 2.5, but sought to overcome TNF's limitations, such as its restricted probe coverage, crude filtering mechanisms, and postmortem-only data handling, by providing a more comprehensive and scalable solution. Adam Leventhal and Mike Shapiro soon joined Cantrill, forming the core team that shaped DTrace's architecture within Sun's Solaris Kernel Development group. Central to the early design were principles of whole-system visibility, allowing tracing across kernel, device drivers, and user applications to uncover systemic behaviors and performance bottlenecks that process-centric tools could not address. The framework prioritized safety—ensuring probes could not crash the system—and minimal intrusion, with zero probe effect when tracing was disabled, facilitating its use in high-stakes production environments without requiring code recompilation or restarts. DTrace made its initial public appearance integrated into Solaris 10, which Sun Microsystems released in January 2005 as a standard OS component. From the outset of its external availability, Sun positioned DTrace as an open-source project under the Common Development and Distribution License (CDDL), encouraging broader adoption and contributions from the developer community while aligning with Sun's emerging initiative. This licensing choice reflected Sun's intent to extend DTrace's impact beyond proprietary Solaris deployments, laying the groundwork for its evolution into a versatile diagnostic tool.

Open-Sourcing and Expansion

DTrace was open-sourced by in 2005 as part of the project, licensed under the (CDDL), which facilitated its adoption and porting to other operating systems. This release enabled community contributions and led to early ports, including integration into starting with version 7.1 in 2009, following development efforts that began in 2007. Apple incorporated a port of DTrace into Mac OS X 10.5 in 2007, enhancing kernel and application diagnostics for its ecosystem. For , announced an official port in October 2011, initially as a kernel module for the Unbreakable Enterprise Kernel (UEK), alongside alternatives like SystemTap that drew inspiration from DTrace concepts. Sun's acquisition by in 2010 marked a transition in stewardship, with continuing active development of DTrace for Solaris and extending it to via UEK kernels, where it became a standard tool for real-time troubleshooting. Community efforts further expanded DTrace's reach, including ports to —fully integrated in 10.0 released in 2024—and , where an initial port was completed in 2007 to support its real-time kernel. Recent advancements include Microsoft's native support for DTrace in Windows Server 2025, announced in 2024 and shipped in November 2024, providing built-in diagnostics through a cross-platform port derived from the open-source OpenDTrace project. Oracle's ongoing enhancements culminated in DTrace 2.0.3-1 for Linux, released on June 10, 2025, which added support for User-space Statically Defined Tracing (USDT) probes in executables and shared libraries compiled with Link-Time Optimization (LTO). By 2025, DTrace's maturity is evident in its default inclusion across major operating system distributions, including Oracle Linux, FreeBSD, NetBSD, and Windows Server, diminishing the need for separate installations.

Technical Architecture

Probes and Providers

In DTrace, probes represent specific instrumentation points embedded within the operating system kernel, user applications, or libraries, designed to fire in response to particular events such as function entry or exit. These points are statically placed during compilation but remain inactive until dynamically enabled by a DTrace script, ensuring zero runtime overhead when not in use. Each probe is uniquely identified by a description consisting of four components in the provider:module:function:name, where the provider indicates the module responsible for the probe, the module specifies the kernel module or binary containing the probe, the function denotes the specific function or operation, and the name describes the event type (e.g., entry for function start or return for exit). This allows precise targeting of probes, with optional predicates—conditional expressions—to filter firings based on runtime criteria, and actions to define responses, such as recording data or aggregating statistics. Providers serve as the modular components that expose and manage sets of probes, functioning as kernel modules or user-level interfaces that implement particular types of . For instance, the fbt (Function Boundary Tracing) provider instruments kernel function boundaries to trace calls and returns; the syscall provider monitors entry and return; the proc provider tracks process lifecycle events like creation and execution; and the sdt (Statically Defined Tracing, often referred to as USDT) provider enables user-space applications to embed custom probes for application-specific tracing. includes numerous built-in providers—over two dozen core ones documented in official guides, with the total expanding based on installed software and kernel modules—allowing comprehensive coverage from hardware interrupts to network protocols. These providers are extensible, permitting developers to create custom ones for specialized , such as for third-party drivers or applications, thereby supporting virtually unlimited probe availability. A key architectural benefit of providers is their role in abstracting underlying hardware and operating differences, promoting portability of DTrace scripts across supported platforms. By standardizing probe interfaces and semantics, providers encapsulate platform-specific details—like instruction sets or kernel internals—enabling the same probe description to function consistently wherever the provider is implemented, though the exact set of available probes may vary by configuration. When enabled, probes introduce minimal computational overhead, typically equivalent to a few no-op instructions, on the order of nanoseconds per firing, which allows safe use even in production environments without significant impact.

D Programming Language

The D programming language is a designed specifically for writing DTrace scripts, enabling the definition of tracing actions in response to probe firings. It features a C-like syntax augmented with tracing-specific primitives, allowing scripts to execute safely in either kernel or user contexts without risking system instability. D programs are structured as a series of clauses that associate probe descriptions with optional predicates and actions, facilitating dynamic without recompilation or system . Key elements of D include probe clauses, which integrate probe descriptions with predicates and actions. A typical clause follows the form probe description /predicate/ { action statements; }, where the predicate is an optional conditional expression enclosed in slashes, such as /arg0 > 100/, that evaluates to true (non-zero) or false (zero) when the fires. Actions, enclosed in braces, consist of straight-line statements like trace(arg0); to output or printf("Value: %d\n", arg1); for formatted printing. Aggregations are defined using the @ prefix for associative arrays that summarize across probe firings, as in @foo[probefunc] = [count](/page/Count)(); to tally occurrences by function name. Built-in variables provide contextual information during execution. The variables arg0 through arg9 represent the first ten arguments passed to the probe as 64-bit integers, with their interpretation depending on the probe's ABI. Other variables include curthread, a pointer to the current kernel thread structure for thread-specific details, and timestamp, a nanosecond-resolution counter since an arbitrary , useful for relative timing measurements. Aggregation types in D leverage the @ operator to create thread-local, lock-free associative arrays for efficient summarization. Common aggregation functions include count() to track invocation frequency, avg(expression) to compute arithmetic means of scalar values, and lquantize(expression, lower_bound, upper_bound, step) to generate linear histograms distributing values into buckets of fixed width. These functions store intermediate results per CPU, enabling scalable analysis without global synchronization. D scripts are interpreted directly at runtime by the DTrace facility, requiring no separate compilation step; the dtrace command compiles and loads them on-the-fly into a safe intermediate form with built-in error handling for issues like . This supports both one-liner commands, such as dtrace -n 'syscall:::entry { trace(execname); }', and multi-clause scripts saved in .d files executed via dtrace -s script.d. Unlike full C, D omits control-flow constructs like loops and branches within action blocks to ensure deterministic, bounded execution and prevent infinite loops or resource exhaustion in kernel context. Instead, conditional logic relies on predicates, and post-processing is handled via special END clauses tied to the dtrace:::END probe, which fires after all other probes to format or output aggregated results, such as using printa(@foo); to display histogram data.

Usage and Examples

Command-Line Basics

DTrace is invoked from the command line using the dtrace utility, which serves as the primary interface for compiling, enabling, and executing D programs. The basic syntax allows for inline probe descriptions with the -n option or loading from a script file with -s. For example, dtrace -n 'probe /predicate/ { action }' executes a simple one-liner, where probe specifies the instrumentation point, predicate is an optional condition, and action defines the tracing behavior. Alternatively, dtrace -s script.d compiles and runs a D program stored in a file, enabling tracing unless overridden. Several common options control invocation and output. The -Z flag permits execution even if no probes match the description, allowing partial or experimental scripts to run without error. For quieter operation, -q suppresses non-data output, displaying only explicit print actions from the script. Output can be directed to a file with -o outputfile, useful for post-processing large traces. To attach to a running process, -p pid specifies the process ID for user-space tracing. Similarly, -c command executes and traces a specified command, such as dtrace -c 'ls -l', capturing its activity. Probe discovery is facilitated by -l, which lists available probes matching criteria, e.g., dtrace -l -n syscall:::entry to enumerate system call entry points. Tracing sessions can be managed programmatically within D scripts using built-in functions like stop() to halt the kernel temporarily for inspection and exit() to terminate the tracing run. By default, DTrace produces formatted ASCII output in a tabular style, showing timestamps, probe details, and variable values; the -x option enables extended variables, such as aggsize for aggregation buffer sizing, to customize this further. DTrace typically requires root privileges for kernel-level access, ensuring secure instrumentation of system-wide events, though user-space tracing is possible with appropriate permissions granted via tools like priv_set. For analysis, output integrates seamlessly with shell pipelines, such as piping to awk for filtering, e.g., dtrace -n 'syscall:::entry { trace(execname); }' | awk '{print $1}' | sort | uniq -c to count unique process names.

Practical Scripting Scenarios

DTrace scripts enable administrators and developers to diagnose performance issues in real-time without modifying applications or rebooting systems. Common scenarios include monitoring system calls to identify excessive kernel interactions, profiling functions to pinpoint CPU-intensive code paths, and tracing user-space events for application-specific insights. These scripts leverage DTrace's probes to collect data dynamically, often aggregating results for analysis. One practical scenario involves tracing system calls to understand process behavior and detect anomalies, such as a process making an unusually high number of calls. For example, the following one-liner counts system calls by executable name: dtrace -n 'syscall:::entry { @[execname] = count(); }'. This aggregates counts across all system calls, revealing which processes are most active at the kernel level, such as a database server dominating with thousands of reads. Function-level profiling helps identify "hot" functions consuming disproportionate within a binary, aiding in optimization during high-load . Using the fbt (Function Boundary Tracing) provider, a script can measure time spent in kernel functions; for instance: fbt::delay:entry, fbt::drv_usecwait:entry { self->in = timestamp; } fbt::delay:return, fbt::drv_usecwait:return /self->in/ { @snoozers[stack()] = quantize(timestamp - self->in); self->in = 0; }. This quantizes delays in device driver waits, showing stack traces and latency distributions to diagnose blocking operations during system boot or runtime. For user-space tracing, the USDT (User Statically Defined Tracing) provider allows instrumentation of applications like databases without recompilation. In , which embeds USDT probes, a script can monitor transaction activity: postgresql$1:::transaction-start { @start["Start"] = [count](/page/Count)(); self->ts = [timestamp](/page/Timestamp); } postgresql$1:::transaction-abort { @abort["Abort"] = [count](/page/Count)(); } postgresql&#36;1:::transaction-commit /self->ts/ { @commit["Commit"] = [count](/page/Count)(); @time["Total time (ns)"] = sum([timestamp](/page/Timestamp) - self->ts); self->ts=0; }, executed as ./txn_count.d <PID>. This counts starts, aborts, and commits while summing durations, helping troubleshoot transaction bottlenecks in production environments. Error handling in DTrace scripts ensures robust execution, particularly for long-running traces where predicates might fail or resources exhaust. The BEGIN probe initializes setup, such as printing headers or validating inputs: BEGIN { [printf](/page/Printf)("Tracing started\n"); }, while the END probe summarizes aggregations upon completion: END { printa(@); }. For runtime errors like null dereferences, the ERROR probe can log details: dtrace:::ERROR { trace("Error at probe %s\n", probefunc); }, preventing silent failures and aiding . Scripts can also profile I/O latency to isolate disk bottlenecks, a frequent cause of application slowdowns. A representative example tracks I/O latency by device: io:::start { self->ts = timestamp; self->dev = args[1]->dev_statname; } io:::done /self->ts/ { printf("%s: %d ms\n", self->dev, (timestamp - self->ts)/1000000); self->ts = 0; self->dev = 0; }. This prints completion times in milliseconds for each I/O operation, highlighting outliers like multi-second waits on overloaded volumes. An advanced technique for applications involves DTrace's built-in jstack() action to capture stack traces during profiling, focusing on core DTrace capabilities while complementing external tools. For instance: syscall::write:entry /execname == "java"/ { jstack(); } traces Java stacks on write system calls, revealing code paths like java.io.PrintStream.println leading to kernel I/O, thus identifying inefficient or file operations without relying solely on JVM-specific utilities.

Implementations and Platforms

Unix-Like Systems

DTrace was natively integrated into Solaris 10, released in 2005, providing comprehensive support for both kernel and user-space tracing with minimal overhead and no need for recompilation of the target system. This implementation includes built-in providers such as syscall, fbt (function boundary tracing), and proc for system-wide observability, and continues to deliver updates through releases, ensuring compatibility with modern hardware and security features. On , DTrace was initially ported starting in 2007 by developer John Birrell, with initial kernel support appearing in FreeBSD 7.1 in 2008 and userland support added in FreeBSD 10.0 in 2014, enabling it as a standard tool in the base system without additional packages. As of 2025, continues to enhance DTrace with support for architectures like ARMv8 and . The FreeBSD implementation offers robust kernel tracing via providers like fbt, syscall, and profile, alongside userland support that allows tracing of applications without kernel modifications, though it requires enabling specific kernel options such as DDB_CTF for full functionality. For , full DTrace support is available on through the Unbreakable Enterprise Kernel (UEK), starting with UEK3 in 2013, which provides native kernel and user-space tracing comparable to Solaris. On other distributions, partial functionality is achieved via tools like bpftrace, a high-level eBPF-based tracer inspired by DTrace syntax, or SystemTap for scriptable probing, though these lack the full probe ecosystem. As of 2025, DTrace 2.0 maintains compatibility with 6.x series on 9 via UEK7, supporting advanced features like USDT (User Statically-Defined Tracing) probes, which require applications to be compiled with specific flags such as including <sys/sdt.h> and using DTRACE_PROBE macros to embed probe points with negligible runtime overhead when disabled—similar to Solaris implementations. Community efforts like OpenDTrace provide alternative implementations, but Oracle's remains the most mature. Apple ported DTrace to macOS, integrating it starting with Mac OS X 10.5 Leopard in 2007 and continuing enhancements through later versions, enabling kernel and user tracing for performance analysis and debugging. It remained available until (10.13) in 2017, after which Apple deprecated certain APIs and restricted access in Mojave (10.14) and later releases, requiring (SIP) to be partially disabled for full use, though basic functionality persists. Community-maintained forks and patches, such as those via Homebrew or custom kernel extensions, allow limited DTrace operation on newer macOS versions like Ventura and Sequoia as of 2025. Among other Unix-like systems, DTrace support on NetBSD remains experimental, with a basic port available since 2012 that enables kernel tracing but lacks full userland integration and requires manual kernel configuration via options DTRACE. In contrast, illumos—a community fork of OpenSolaris—features active DTrace development, with ongoing enhancements in distributions like OpenIndiana 2025.10, preserving the original Solaris probe set and adding modern capabilities such as improved CTF (Compact C Type Format) support for debugging.

Windows and Other Ports

Microsoft initiated the porting of DTrace to Windows in 2019, basing it on the OpenDTrace implementation from , with the tool becoming available in Insider Program builds starting in March of that year. This adaptation leverages Windows-specific infrastructure, enabling dynamic tracing for performance analysis and debugging on the platform. The port reuses much of the user-mode components from OpenDTrace while incorporating a custom kernel to handle system-monitoring tasks not directly analogous to Unix kernels. It supports both x86 and x64 architectures. On Windows, DTrace integrates closely with Event Tracing for Windows (ETW), providing an ETW provider that allows tracing of kernel and user-mode events logged through this native facility. Equivalent providers to Unix's proc and syscall are supported, facilitating process-level and system call tracing, alongside user-mode and kernel-mode instrumentation. The Function Boundary Tracing (FBT) provider is also available, enabling probes on kernel function entry and return points. Features like Kernel Patch Protection (PatchGuard) may restrict certain kernel probes in secure environments. DTrace achieved full integration as a built-in tool in 2025, released on November 1, 2024, where dtrace.exe is included natively to support performance monitoring and issue diagnosis without requiring third-party utilities. This embedding enhances server troubleshooting capabilities, allowing administrators to run DTrace scripts directly for real-time insights into system behavior. Beyond Windows, DTrace has been ported to , a real-time , with an initial implementation completed in 2007 that encapsulates the tracing framework within QNX's resource manager architecture. This port provides comprehensive support for dynamic tracing in resource-constrained environments, making it suitable for automotive and industrial applications where low-overhead monitoring is essential.

Community and Recognition

Key Developers

Bryan Cantrill served as the lead architect for DTrace at Sun Microsystems, where he co-designed its core concepts, including probes and the D programming language, while also championing its development through advocacy and live debugging demonstrations. Michael W. Shapiro co-developed DTrace alongside Cantrill, with a primary focus on its kernel integration and implementation of safety features to ensure safe dynamic instrumentation in production environments. Adam Leventhal contributed significantly to DTrace's user-space tracing capabilities and aggregation mechanisms, enabling detailed analysis of application-level behaviors; he continued related work at after Sun's acquisition and later joined Oxide Computer Company. Cantrill's presentation of the seminal DTrace paper at the 2004 Annual Technical Conference played a pivotal role in its rapid adoption by demonstrating its power for production system diagnostics. For platform ports, Apple's engineering team—including Steve Peters, James McIlree, Terry Lambert, Tom Duffy, and Sean Callanan—led the integration of DTrace into Mac OS X, making it available starting with Mac OS X 10.5 Leopard. John Birrell spearheaded the initial port of DTrace to , achieving core functionality by 2006 and contributing to its inclusion in 7.1 in 2009. Microsoft's engineering team developed the Windows port based on the open-source OpenDTrace project, releasing it for Insider builds in 2019 and integrating it natively into 2025. Several original DTrace developers, including Cantrill and Leventhal, now work at Oxide Computer Company, where they continue to advance DTrace implementations for modern rack-scale systems.

Awards and Impact

DTrace received significant recognition for its innovative approach to dynamic and production debugging. In 2006, it was awarded the Wall Street Journal's Technology Innovation Award, highlighting its transformative impact on system observability. The framework has been prominently featured in ACM Queue, including Bryan Cantrill's 2006 article "Hidden in Plain Sight," which detailed its design principles, and a 2008 article "A Pioneer's Flash of Insight," which mentions its creators and the award. DTrace's design has profoundly influenced subsequent observability tools, particularly in Linux environments. It inspired the development of eBPF, a kernel-level technology for safe, dynamic tracing that extends similar capabilities to Linux systems. Similarly, bpftrace, a high-level tracing language for eBPF, draws direct inspiration from DTrace's scripting model, combining elements of awk, C, and DTrace to enable concise, powerful probes. These tools have positioned DTrace's principles as a foundational standard for cloud and observability, where dynamic tracing is essential for real-time performance analysis without system disruption. In broader applications, DTrace has enabled fault isolation and zero-downtime analysis in mission-critical environments. Its low-overhead supports detailed transaction timing across processes, facilitating rapid diagnosis in high-stakes operations. Industries such as have leveraged it for troubleshooting in production systems, while telecom deployments benefit from its ability to monitor complex, real-time interactions without halting services. As of 2025, DTrace maintains strong relevance through integrations with emerging technologies, including AI-assisted tracing. Oracle's advancements allow large language models to generate and interpret DTrace scripts via natural language prompts, enhancing accessibility for system diagnostics. Microsoft has incorporated DTrace as a built-in tool in 2025, signaling its role in future diagnostics roadmaps for cross-platform . The DTrace community continues to thrive, with events like dtrace.conf(24) held in December 2024 fostering ongoing development and adoption. DTrace's emphasis on safe, comprehensive tracing has contributed to the evolution of modern standards, with its core concepts influencing frameworks like OpenTelemetry for distributed system observability.

References

  1. https://www.cs.princeton.edu/courses/archive/fall05/cos518/papers/dtrace.pdf
  2. https://www.oracle.com/solaris/technologies/releases.html
  3. https://dtrace.org/about/
  4. https://docs.oracle.com/en/operating-systems/oracle-linux/6/adminsg/ol_about_dtrace.html
  5. https://learn.microsoft.com/en-us/windows-hardware/drivers/devtest/dtrace
  6. https://docs.oracle.com/cd/E19253-01/819-5488/gbwaz/index.html
  7. https://docs.oracle.com/en/operating-systems/oracle-linux/dtrace-guide/dtrace-ref-AboutDTrace.html
  8. https://docs.freebsd.org/en/books/handbook/dtrace/
  9. https://docs.oracle.com/en/operating-systems/oracle-linux/6/porting/ch04s06.html
  10. https://docs.oracle.com/cd/E37838_01/pdf/E61035.pdf
  11. https://www.brendangregg.com/dtrace.html
  12. https://www.blackhat.com/presentations/bh-usa-08/Beauchamp_Weston/BH_US_08_Beauchamp-Weston_DTrace.pdf
  13. https://www.usenix.org/conference/2004-usenix-annual-technical-conference/dynamic-instrumentation-production-systems
  14. https://bcantrill.dtrace.org/2008/09/03/happy-5th-birthday-dtrace/dtrace_ktd.pdf
  15. https://www.bsdcan.org/2017/schedule/attachments/432_dtrace_internals.pdf
  16. https://www.crn.com/news/components-peripherals/57703039/source-sun-to-release-first-bits-of-open-solaris-tuesday
  17. https://adtmag.com/articles/2005/01/24/sun-launches-opensolaris-project-with-dtrace-code.aspx
  18. https://www.cnet.com/tech/tech-industry/sun-logs-thousands-of-dtrace-downloads/
  19. https://www.oracle.com/technetwork/server-storage/9520-dtrace-158978.pdf
  20. https://oss.oracle.com/ossh.html
  21. https://www.freebsd.org/status/report-2007-10-2007-12.html
  22. https://klarasystems.com/articles/freebsd-history-understanding-the-origins-of-dtrace/
  23. https://papers.put.as/papers/macosx/2009/Debugging-Cocoa-with-DTrace.pdf
  24. https://docs.oracle.com/en/operating-systems/oracle-linux/dtrace-relnotes/news-010Oct20th2011.html
  25. https://developers.lseg.com/en/article-catalog/article/using-dtrace-with-elektron-sdk
  26. https://docs.oracle.com/en/operating-systems/oracle-linux/dtrace-tutorial/dtrace-tutorial-IntroducingDTrace.html
  27. https://wiki.netbsd.org/tutorials/how_to_enable_and_run_dtrace/
  28. https://netbsd.org/releases/formal-10/NetBSD-10.0.html
  29. https://www.osnews.com/story/18901/dtrace-ported-to-qnx/
  30. https://learn.microsoft.com/en-us/windows-server/administration/windows-commands/dtrace
  31. https://learn.microsoft.com/en-us/windows-server/get-started/whats-new-windows-server-2025
  32. https://docs.oracle.com/en/operating-systems/oracle-linux/dtrace-relnotes/news-2_0_3_1.html
  33. https://docs.oracle.com/en/operating-systems/oracle-linux/dtrace-relnotes/OL-DTRACE-RELNOTES.pdf
  34. https://wiki.freebsd.org/DTrace
  35. https://man.netbsd.org/dtrace.1
  36. https://docs.oracle.com/en/operating-systems/solaris/oracle-solaris/11.4/dtrace-guide/providers-and-probes.html
  37. https://docs.oracle.com/cd/E23824_01/html/E22973/gkyal.html
  38. https://docs.oracle.com/en/operating-systems/oracle-linux/dtrace-guide/dtrace-ref-StaticallyDefinedTracingofUserApplications.html
  39. https://docs.oracle.com/en/operating-systems/oracle-linux/dtrace-guide/dtrace-ref-TheDProgrammingLanguage.html
  40. https://docs.oracle.com/cd/E53394_01/html/E53395/gkwlh.html
  41. https://docs.oracle.com/cd/E18752_01/html/819-5488/gcggh.html
  42. https://docs.oracle.com/en/operating-systems/oracle-linux/dtrace-guide/dtrace-ref-TheDProgrammingLanguage.html#dt_builtin_dlang
  43. https://docs.oracle.com/en/operating-systems/oracle-linux/dtrace-v2-guide/aggregation_lquantize.html
  44. https://docs.oracle.com/cd/E19253-01/817-6223/chp-dtrace-end/index.html
  45. https://docs.oracle.com/en/operating-systems/oracle-linux/dtrace-guide/dtrace-ref-dtraceCommandReference.html
  46. https://docs.huihoo.com/opensolaris/dtrace-user-guide/html/ch02.html
  47. https://www.oracle.com/solaris/technologies/dtrace-tutorial.html
  48. https://docs.oracle.com/cd/E18752_01/html/819-5488/gcfqr.html
  49. https://docs.oracle.com/cd/E53394_01/html/E53395/gkyaz.html
  50. https://www.postgresql.org/docs/current/dynamic-trace.html
  51. https://docs.oracle.com/cd/E19253-01/817-6223/chp-script/index.html
  52. https://docs.oracle.com/cd/E19253-01/817-6223/chp-dtrace-error/index.html
  53. https://docs.oracle.com/cd/E19253-01/817-6223/chp-io-examples/index.html
  54. https://www.brendangregg.com/Slides/dtrace_topics_java.pdf
  55. https://www.oracle.com/technetwork/server-storage/solaris10/solaris-dtrace-wp-167895.pdf
  56. https://blogs.oracle.com/linux/dtrace-on-linux-an-update
  57. https://www.brendangregg.com/blog/2018-10-08/dtrace-for-linux-2018.html
  58. https://www.oracle.com/linux/downloads/linux-dtrace.html
  59. https://blogs.oracle.com/linux/usdt-tracing-across-different-runtimes
  60. https://ahl.dtrace.org/2008/06/07/apple-updates-dtrace/
  61. https://stackoverflow.com/questions/60908765/mac-osx-using-dtruss
  62. https://jade.fyi/blog/misadventures-in-dtrace/
  63. https://www.osnews.com/story/143655/openindiana-2025-10-released/
  64. https://www.phoronix.com/news/Microsoft-DTrace-Windows
  65. https://arstechnica.com/gadgets/2019/03/microsoft-brings-oracles-open-source-dtrace-tracing-to-windows/
  66. https://www.osnews.com/story/129587/microsoft-ports-dtrace-to-windows/
  67. https://learn.microsoft.com/en-us/windows-hardware/drivers/devtest/dtrace-etw
  68. https://virtualizationreview.com/articles/2019/04/19/using-dtrace-for-windows.aspx
  69. https://github.com/microsoft/DTrace-on-Windows
  70. https://learn.microsoft.com/en-us/lifecycle/products/windows-server-2025
  71. https://www.osnews.com/story/139740/windows-server-2025-to-ship-with-dtrace-by-default/
  72. https://community.qnx.com/sf/projects/dtrace/
  73. https://ahl.dtrace.org/2005/07/20/dtrace-user-land/
  74. https://www.usenix.org/legacy/event/usenix04/tech/general/cantrill.html
  75. https://bcantrill.dtrace.org/2006/08/07/dtrace-on-mac-os-x/
  76. https://bcantrill.dtrace.org/2006/05/25/dtrace-on-freebsd-update/
  77. https://oxide.computer/blog/dtrace-conf24
  78. https://www.oreilly.com/pub/au/5315
  79. https://queue.acm.org/detail.cfm?id=1117401
  80. https://queue.acm.org/detail.cfm?id=1413260
  81. https://www.tigera.io/learn/guides/ebpf/
  82. https://github.com/bpftrace/bpftrace
  83. https://ahl.dtrace.org/2004/10/29/dtrace-time/
  84. https://blogs.oracle.com/linux/dtrace-and-llms
Add your contribution
Related Hubs
User Avatar
No comments yet.