From: dale.hamel@... Date: 2019-08-01T02:12:49+00:00 Subject: [ruby-core:94089] [Ruby master Feature#16027] Update Ruby's dtrace / USDT API to match what is exposed via the TracePoint API Issue #16027 has been updated by dalehamel (Dale Hamel). Thanks for the reply and your work on TracePoints, Koichi. > I need to study USDT... I can recommend some resources, I might suggest this excerpt of [my own writing on this](https://bpf.sh/usdt-report-doc/index.html#adding-usdt-support-to-a-dynamic-language), as I've been researching this topic a great deal lately. That document is from a larger 'work in progress' of my findings, though I wouldn't recommend reading all of it just yet as it needs more editing. Another good spot is https://awesome-dtrace.com/, which details more traditional and conventional uses of dtrace for USDT. > I think dtrace feature is useful because they can operate outside of process Yes, this is the key advantage of USDT tracing versus other approaches. A standardized debugging approach can be used for different programming languages (V8 and Python have built-in dtrace support for instance), allowing for the same tools to be used by experts who work on different languages. This is less true of Java, which has its own debugging ecosystem as I understand it, but even Java has USDT support to a degree. > do not need to modify observed process This is not exactly true, the process has hooks built-into it, which indicate addresses of statically defined tracepoints. The kernel can inject a debugging instruction (int3) at this address. When the code is executed, it will trigger a handler, uprobes in the case of linux, which can read data that the program emits to the breakpoint. Unless the program is under observation, there will be no breakpoint instruction at this address. It depends on how the program handles this behavior, but usually it will noop and the program executes unmodified. In the casy of the RubyVM, these tracepoints are embedded in the code using the sys/sdt.h header macros for defining USDT probes for dtrace. For the case of dynamic code, we must prepare a stub executable, which can contain the address for the kernel to insert its debugging instruction. So the process isn't exactly unmodified, but the modifications should be minimal. At the time when a TracePoint enabled, a static tracepoint would be generated and loaded into memory, which cats as the bridge between ruby code and the kernel. > However, your proposal seems to introduce new Ruby APIs to enable them. What do you think about it? I want to try and keep the API change as clean as possible, so that static tracepoints are complementary to the builtin tracepoints, as an extension to use them as a way to attach an external debugger like dtrace or bpftrace to the process. I hope my explanation above is what you're looking for, but I'd be happy to clarify if not. > Could you show some example scenario with dtrace command line? For basic usage of a static tracepoint: ```ruby require 'ruby-static-tracing' t = StaticTracing::Tracepoint.new('global', 'hello_test', String) puts t.provider.enable Signal.trap('USR2') do puts "TRAP #{t.enabled?}" t.fire('Hello world') if t.enabled? sleep 2 end loop { puts t.enabled?; sleep 1 } ``` We can then attach to the process by specifying the PID, and using the following dtrace script: ``` global*:::hello_test { printf("%s\n", copyinstr(arg0)) } ``` Or the following bpftrace script (linux): ``` usdt::global:hello_test { printf("%s\n", str(arg0)); } ``` This is from one of the integration tests. It will do the following: - Create an ELF (or DOF) stub matching the namespace specified, above I used 'global'. This corresponds to the name of the shared library stub, which will be like `global-stub.so`. - The stub will have notes that can be read with `readelf --notes` indicating the address of the tracepoint function and the tracepoint arguments - A debugger can be attached by specifying "global:hello_test" to find the probe information - Once attached, the loop above will indicate the tracepoint is enabled, as the kernel will have overwritten the first byte (safely) with an int3/ 0xCC debug instruction (on x86, other platforms support different instructions). - To simulate the tracepoint being hit, we can send `kill -USR2 $(pidof ruby)` to enter the trap handler, which will fire the probe - Note that it only fires the probe data if it is enabled, meaning if no debugger is attached it doesn't fire the probe. This can be used to guard against expensive logic - The debugger (dtrace/bpftrace) will then output "Hello world", showing the transmission of debugging data from ruby to the debugger through the Kernel. The call to `fire` is done like a method call, in the case of ruby like a method call to a native extension. The arguments are thus put on the stack, and the kernel can grab them in a predictable way. This is why we must indicate the type when a tracepoint is fired, so the kernel has a way to read off the arguments and give them to the debugger program. > I can't understand how to use usdt: keyword. I have rethought the API to try and simplify and clarify. Maybe a better name for it would by "tracepoint_types" or "arg_types". Currently on linux USDT probes are limited to 6 arguments, but I think some platforms support more than 6. The static tracepoint must emit a type that matches what was declared, and where to read the data for these arguments from. So, the way this might look if builtin to ruby could be: ``` trace.enable(target: nil, target_line: nil, target_thread: nil: static_tracepoints: nil) ``` If static_tracepoints (above i had said just usdt, but maybe static_tracepoints is more descriptive?) is set, then if we enable a tracepoint without a block, such as: ```ruby trace.enable(target: method(:foo), target_line: 5) ``` Then we could default to emitting the same tracepoint arguments as are [already emitted](https://github.com/ruby/ruby/blob/master/doc/dtrace_probes.rdoc), so a tracepoint could be created and enabled as a member of the TracePoint instance: ```ruby @static_tracepoint = StaticTracing::TracePoint.new(:method_foo, :line_5, String, String, String, Integer) ``` When the tracepoint is hit, it would by default fire off the data that matches this signature for the default handler: ```ruby ... if @static_tracepoint.enabled? @static_tracepoint.fire(tp.classname, tp.methodname, tp.filename, tp.lineno) end ... ``` Note that @static_tracepoint.enabled? is checking if an address in memory is a noop (0x90) or breakpoint (0xCC), and guarding actually firing the tracepoint on this. If a block is specified when creating the tracepoint, then we must specify the signature of the tracepoint. This is more powerful, as it allows for *any* data in the tracepoint debugging context to be emitted to the user. For instance, enable may instead look like: ```ruby trace.enable(target: method(:foo), target_line: 5, arg_types: [Integer, String]) do |tp| tp.fire(tp.lineno, "Any String I want!") end ``` Which would result the dtrace/bpftrace getting output like: ``` 5 Any String I want! ``` I hope this helps to clarify and improve on my original suggestions, I am sorry for yet another wall of text, and appreciate the time you've taken in considering this feature. ---------------------------------------- Feature #16027: Update Ruby's dtrace / USDT API to match what is exposed via the TracePoint API https://bugs.ruby-lang.org/issues/16027#change-80321 * Author: dalehamel (Dale Hamel) * Status: Assigned * Priority: Normal * Assignee: ko1 (Koichi Sasada) * Target version: ---------------------------------------- # Abstract I propose that Ruby's "dtrace" support be extended to match what is available in the TracePoint API, as was the case until feature [Feature #15289] landed. # Background I will refer to Ruby's "dtrace" bindings as USDT bindings for simplicity, as this is the typo of dtrace probe that they support. Prior to [Feature #15289] being merged, Ruby's tracepoint API was able to trace only 'all' instances of a type of event. Ruby added support for tracing ruby with dtrace, and so Ruby's USDT Ruby TracePoint API were "in sync". Once the Ruby TracePoint API recently added the ability to do filtered tracing in [Feature #15289], it added new functionality but brought the TracePoint and USDT API out of sync. Currently the TracePoint API is ahead of the USDT API, which presents the problem. There is valuable debug information available, but we do not have a way to access it with dtrace instrumentation. Additionally, the recent release of bpftrace adds support for USDT tracing on linux, which makes this a valuable opportunity to be able to use Ruby's TracePoint API in an efficient and targeted way for production tracing. To achieve this, we must synchronize the features of the USDT and TracePoint API. What is currently lacking is the ability to do filtered, selective tracing as the `TracePoint#enable` call now supports as per [prelude.rb#L141](https://github.com/ruby/ruby/blob/master/prelude.rb#L141) # Proposal When enabling a TracePoint, users can specify a flag: `usdt: [LIST_OF_SIMPLE_TYPES]`, which will trigger Ruby to also enable the USDT API for when it enables TracePoints. Within the TracePoint block, users can call `tp.fire` to send USDT data. So the new default API is: ```ruby trace.enable(target: nil, target_line: nil, target_thread: nil: usdt: nil) ``` And the usage might look like: ```ruby trace.enable(target: method(:foo), target_line: 5, usdt: [Integer, String]) do |tp| tp.fire(tp.lineno, "Any String I want to send") end ``` The types specified must be simple types such as `Integer` or `String`, given by their names as constants. When data to the tracepoint, the types must match. If they don't, the tracer won't be able to interpret them properly, but nothing should crash. # Details I propose that Ruby optionally generate ELF (Linux) or DOF (Darwin) annotations for TracePoint targets when they are enabled. As ruby is a dynamic language, it cannot do this natively (yet) though Ruby JIT may make this easier, but for now it is not suitable for production use. To get around this, Ruby can either generate the DOF or ELF stub shared library itself, for example it may do one per class, treating the class as the "provider" for the USDT API, and the methods as tracepoints. This is the approach used by [libusdt](https://github.com/chrisa/libusdt), which generates DOF usable on Darwin, BSD, and other platforms, and [libstapsdt](https://github.com/sthima/libstapsdt), which generates ELF stubs for use on linux. When a tracepoint is triggered, the user may be able to call a new API `TracePoint#fire`, to send data to the Kernel via the USDT API, using the generated ELF stub as a bridge, giving the kernel an address to target in order to receive this data. Upon enabling a tracepoint, we can either generate these stubs internally, or by linking to an external library that must be enabled at configure time (without this, USDT tracing wouldn't be enabled at all). It may be possible to use the existing bridge that is used by ruby jit, or have an experimental flag such as `--usdt` that enables support for generating these stubs. It may be more consistent with the future Ruby JIT to do this, or else Ruby can generate these stubs by its own native code, but this will require a sort of merging of libusdt and libstapsdt. This would add a dependency to the libelf development header, but that is probably not a problem on Linux platforms. I would suggest the first approach, if this feature is accepted, would be to try and implement the ELF / DOF generation directly in Ruby. What libstapsdt and libusdt do isn't that complex and could be done in its own C file that probably wouldn't be too large. Failing that approach, it may be worth investigating the Ruby JIT code to see if a compiler can generate these stubs for us easily. This approach would be to have ruby generate C code that results in the necessary DOF/ELF annotations, and have the compiler pipeline used by ruby JIT to generate the file. This couples the feature to ruby jit though. # Usecase This feature would be used by dtrace / bpftrace users to debug ruby applications. It may be possible for other platforms to benefit from this too, but I think the main use case is for Linux system administrators and developers to use external debuggers (dtrace/bpftrace) to introspect Ruby's behavior. # Discussion ## Pros: * Syncs the Ruby TracePoint and USDT API * Allows for much more dynamic and targeted USDT tracing * Can help to find problems in both development and production * Can be used for performance and error analysis * Is better than printing, as emitting/collecting data is only done while a "debugger is attached" ## Cons: * Complexity introduced, in order to generate the ELF/DOF stub files * Not easily ported to other platforms * Isn't fully consistent with the current dtrace functionality of Ruby, which is built-in to the VM # Limitation This will only work on *Nix platforms, and probably just on Linux to start, as that is where most of the benefits are. If the Ruby JIT approach is preferred or much simpler, then that functionality will be tied to the Ruby JIT functionality. # See also * https://bpf.sh/usdt-report-doc/index.html a document describing my experimental gem ruby-static-tracing, which prototypes this functionality outside of the RubyVM * https://bpf.sh/production-breakpoints-doc/index.html a work-in-progress on adding more dynamic method and line based USDT tracing to ruby, built atop ruby-static-tracing now using the ruby tracepoint API. -- https://bugs.ruby-lang.org/ Unsubscribe: