From: dale.hamel@...
Date: 2019-08-03T02:41:28+00:00
Subject: [ruby-core:94124] [Ruby master Feature#16027] Update Ruby's dtrace / USDT API to match what is exposed via the TracePoint API

Issue #16027 has been updated by dalehamel (Dale Hamel).


> Could you show us how V8 and Python provide USDT features?

V8 has ustack helpers (which I'd love to try and figure out how to do for MRI) https://www.joyent.com/blog/understanding-dtrace-ustack-helpers which make the type of probing i'm proposing here a little moot. It is still possible though, here's an example https://www.npmjs.com/package/usdt

Python also had some work on a ustack helper, I'm not sure if it landed though as i can only find an old repo for it. It has other dtrace probes available too https://docs.python.org/3/howto/instrumentation.html but these seem more analagous to what's already in ruby. It is however possible to do this type of dynamic probing in python, like in node, with https://pypi.org/project/stapsdt/ which also uses the ELF stubbing approach.

I think what makes ruby different is that it seems like TracePoint is a natural hook-point for debugging, and I'm not aware if either of python or V8 has something like this exactly.


> If my understanding is correct, dtrace does not need to modify original source code.
> I think your extension requires to modify ruby (.rb) source code.
> (people need to write TracePoint code)
> Is it acceptable?
> I don't against your proposal, but I'm afraid how they are useful.

I think it's desirable, but I think it might need some further extensions of what is exposed within the tracepoint context (eg, some way to get at more VM internals like what the value the previous statement evaluated to, look at the stack, etc).

I would anticipate that the default block handler for a static tracepoint, extending a similar API to the RubyVM dtrace handlers, would provide some decent value on their own.

I suspect that the real utility of this feature will be to have a userspace libraries of tracepoint helpers, to build up a variety of debugging tools for the automatic handling of a tracepoint. For example, a "recipe" to inspect the value at a particular line, have "entry" and "exit" tracepoints that can be used to measure latency between the execution of lines within the same thread, getting bits of the stack trace, and a general "hook point" for extremely targeted debugging. Some probes may be more expensive to run and comprehensive, and others may be very fast to gather simple data from local variables and fire it off to be measured. 

> For this purpose, it should be good but maybe TracePoint is not good name...

Can you elaborate?

> Maybe this is bad design.

Which aspect of it do you think should change?

> How about to introduce fire method in some class?

You mean that the static tracepoint would not be a private member of TracePoint, but instead should be a different class? I propose the class be a member so that it when a tracepoint is enabled, the stub library can be loaded. If this is done inside the tracepoint block, i'm not sure where the stub would be initiated.

I think I may need you to elaborate on your concern to understand the problem, if you would please.


> BTW, on the Linux system when I checked implementation, they used global variable to detect enable/disable and the checking code was slow.
> This is why I don't introduce them aggressively (or I don't introduce them).
> Is that changed?

I'm not sure I know what you are referring to with the global variable / slow check. A reference implementation is here https://github.com/sthima/libstapsdt/blob/master/src/libstapsdt.c#L185-L193

You can see that it is comparing the address of the function pointer to a mask, to ensure that the expected 0x90 is there, as the definition of _fire is done in assembly. dtrace probes can also use a semaphore instead of this check, which should also be a fast operation.

This speed also matters only in so much as 'returning fast' while tracepoints are enabled, but not attached. This enabled check will also of course only be in effect when the ruby TracePoint that they are associated with has been enabled.



----------------------------------------
Feature #16027: Update Ruby's dtrace / USDT API to match what is exposed via the TracePoint API
https://bugs.ruby-lang.org/issues/16027#change-80371

* Author: dalehamel (Dale Hamel)
* Status: Assigned
* Priority: Normal
* Assignee: ko1 (Koichi Sasada)
* Target version: 
----------------------------------------
# Abstract

I propose that Ruby's "dtrace" support be extended to match what is available in the TracePoint API, as was the case until feature [Feature #15289] landed.

# Background

I will refer to Ruby's "dtrace" bindings as USDT bindings for simplicity, as this is the typo of dtrace probe that they support.

Prior to [Feature #15289] being merged, Ruby's tracepoint API was able to trace only 'all' instances of a type of event.

Ruby added support for tracing ruby with dtrace, and so Ruby's USDT Ruby TracePoint API were "in sync".

Once the Ruby TracePoint API recently added the ability to do filtered tracing in [Feature #15289], it added new functionality but brought the TracePoint and USDT API out of sync.

Currently the TracePoint API is ahead of the USDT API, which presents the problem. There is valuable debug information available, but we do not have
a way to access it with dtrace instrumentation.

Additionally, the recent release of bpftrace adds support for USDT tracing on linux, which makes this a valuable opportunity to be able to use Ruby's TracePoint API in an efficient and targeted way for production tracing. To achieve this, we must synchronize the features of the USDT and TracePoint API.

What is currently lacking is the ability to do filtered, selective tracing as the `TracePoint#enable` call now supports as per [prelude.rb#L141](https://github.com/ruby/ruby/blob/master/prelude.rb#L141)

# Proposal

When enabling a TracePoint, users can specify a flag: `usdt: [LIST_OF_SIMPLE_TYPES]`, which will trigger Ruby to also enable the USDT API for when it enables TracePoints.

Within the TracePoint block, users can call `tp.fire` to send USDT data. So the new default API is:

```ruby
trace.enable(target: nil, target_line: nil, target_thread: nil: usdt: nil)
```

And the usage might look like:

```ruby
trace.enable(target: method(:foo), target_line: 5, usdt: [Integer, String]) do |tp|
  tp.fire(tp.lineno, "Any String I want to send")
end
```

The types specified must be simple types such as `Integer` or `String`, given by their names as constants. When data to the tracepoint, the types must match. If they don't, the tracer won't be able to interpret them properly, but nothing should crash.

# Details

I propose that Ruby optionally generate ELF (Linux) or DOF (Darwin) annotations for TracePoint targets when they are enabled.

As ruby is a dynamic language, it cannot do this natively (yet) though Ruby JIT may make this easier, but for now it is not suitable for production use.

To get around this, Ruby can either generate the DOF or ELF stub shared library itself, for example it may do one per class, treating the class as the "provider" for the USDT API, and the methods as tracepoints. This is the approach used by [libusdt](https://github.com/chrisa/libusdt), which generates DOF usable on Darwin, BSD, and other platforms, and [libstapsdt](https://github.com/sthima/libstapsdt), which generates ELF stubs for use on linux.

When a tracepoint is triggered, the user may be able to call a new API `TracePoint#fire`, to send data to the Kernel via the USDT API, using the generated ELF stub as a bridge, giving the kernel an address to target in order to receive this data.

Upon enabling a tracepoint, we can either generate these stubs internally, or by linking to an external library that must be enabled at configure time (without this, USDT tracing wouldn't be enabled at all).

It may be possible to use the existing bridge that is used by ruby jit, or have an experimental flag such as `--usdt` that enables support for generating these stubs.

It may be more consistent with the future Ruby JIT to do this, or else Ruby can generate these stubs by its own native code, but this will require a sort of merging of libusdt and libstapsdt. This would add a dependency to the libelf development header, but that is probably not a problem on Linux platforms.

I would suggest the first approach, if this feature is accepted, would be to try and implement the ELF / DOF generation directly in Ruby. What libstapsdt and libusdt do isn't that complex and could be done in its own C file that probably wouldn't be too large.

Failing that approach, it may be worth investigating the Ruby JIT code to see if a compiler can generate these stubs for us easily. This approach would be to have ruby generate C code that results in the necessary DOF/ELF annotations, and have the compiler pipeline used by ruby JIT to generate the file. This couples the feature to ruby jit though.

# Usecase

This feature would be used by dtrace / bpftrace users to debug ruby applications. It may be possible for other platforms to benefit from this too, but I think the main use case is for Linux system administrators and developers to use external debuggers (dtrace/bpftrace) to introspect Ruby's behavior.

# Discussion

## Pros:

* Syncs the Ruby TracePoint and USDT API
* Allows for much more dynamic and targeted USDT tracing
* Can help to find problems in both development and production
* Can be used for performance and error analysis
* Is better than printing, as emitting/collecting data is only done while a "debugger is attached"

## Cons:

* Complexity introduced, in order to generate the ELF/DOF stub files
* Not easily ported to other platforms
* Isn't fully consistent with the current dtrace functionality of Ruby, which is built-in to the VM

# Limitation

This will only work on *Nix platforms, and probably just on Linux to start, as that is where most of the benefits are.

If the Ruby JIT approach is preferred or much simpler, then that functionality will be tied to the Ruby JIT functionality.

# See also

* https://bpf.sh/usdt-report-doc/index.html a document describing my experimental gem ruby-static-tracing, which prototypes this functionality outside of the RubyVM
* https://bpf.sh/production-breakpoints-doc/index.html a work-in-progress on adding more dynamic method and line based USDT tracing to ruby, built atop ruby-static-tracing now using the ruby tracepoint API.




-- 
https://bugs.ruby-lang.org/

Unsubscribe: <mailto:ruby-core-request@ruby-lang.org?subject=unsubscribe>
<http://lists.ruby-lang.org/cgi-bin/mailman/options/ruby-core>