[ruby-core:33982] Re: [Ruby 1.9-Feature#4211][Open] Converting the Ruby and C API documentation to YARD syntax

On Dec 27, 2010, at 19:03, Loren Segal wrote:
> On 12/27/2010 5:03 PM, Eric Hodel wrote:
>> I highly support having a unified documentation tool and style for Ruby, but I personally dislike an enforced template like doxygen requires.
> 
> What do you mean by an "enforced template"?

If we're building a documentation standard for ruby we'll need an example of good documentation, a "template" which we will enforce when documentation is submitted.

> Doxygen (or an alternative @tag based syntax) doesn't "require" anything from you, you're free to add the information you choose, just as with any documentation. @tag syntaxes are merely an *equivalent* to the existing documentation. The requirements come from a style guide, and that's part of documentation process, not some tool. However, since the Ruby C API is already starting to make use of @tag syntax with Doxygen, there is little reason not to leverage this syntax as part of the unified style. Furthermore, YARD already supports this syntax. I'd have no problems with RDoc adding support, too. I don't know of any tools that can document the C end of things, unless RDoc plans on adding support for the C APIs (which I guess I would not object to).

It would not be difficult to add to RDoc.

>> I've read documentation generated by such tools many times and have always found them difficult to navigate as there is not enough context to understand the interaction between all the components of an API.
> 
> What does RDoc do better, here? Again, this is not the fault of @tag based syntaxes-- it is the fault of the tools. For instance, doxygen has some rough edges in displaying documentation, but this is the fault of the tool, not the use of @tags. Keep in mind that I'm not proposing a tool, just a *unified documentation style*. RDoc could certainly implement this if you think it does the best job at displaying information to a user.

Good documentation is independent of the tools used to process it or the syntax used to write it.  I don't think a different tool can make better documentation without human intervention.  Some characteristics of good documentation are consistent wording, approachable writing, comprehensive coverage and useable examples.  I'm unsure of how a tool can help with anything other than comprehensive coverage.

>> If ruby adopts a documentation minimum standard I don't think an enforced template should be our only requirement.
> 
> I agree. As mentioned, the goal is to supply a unified *style*, not just a "template". @tag based syntax is merely a part of this style,

We are in agreement here.

> since it's already being used in the C API, and there is no alternative to properly documenting the C end of things than @tag based tools. It therefore makes sense to continue in this vein and unify the documentation in this way. 

Granted.

> However, it's important to realize that supplying a "bare minimum" guideline to a documenter does a lot to improve the quality of documentation. Obviously, manual checks would have to be done to make sure people are not just "checking off boxes", but having those tags there lets us know what to look for when reviewing documentation, and lets *them* know what they should be writing. This is an improvement over the current adhoc "just-write-something" methodology. Tags also give us the ability to implement tooling later, because the docs are much more easily parse-able.

A common feature of documentation in ruby core is to "tag" method parameters in ++, <tt></tt> or </code></code>.  Matching the "tagged" items in a comment to the method's arguments is trivial way to implement an enhanced coverage report without requiring @tags.

One of the greatest joys I've found in programming is the ability to have the computer do work for me.  I would prefer to have a tool that could figure out what I haven't documented, or documented well enough, with the fewest possible concessions.

>> I question merging YARD into Ruby and replacing RDoc.
>> 
>> Why aren't extensions or enhancements to RDoc being made instead?
> 
> YARD has fairly significant architectural differences that don't lend well to simply writing extensions. YARD is actually a superset of RDoc, and not the other way around, so a merge of functionality wouldn't really work. For instance, YARD is not only a "parse and generate" tool. It is also often used as a 2 phase: "parse and store", "generate from stored data" tool (as exemplified by the 'yard server' tool).

Sounds like ri -f html.  ri uses RDoc::RI::Store to load data from a stored documentation tree.

> RDoc does not have this phase, so much of the functionality YARD has could not be implemented easily.

Loading an RDoc::RI::Store and running it back through an HTML generator could be implemented very easily.  The HTML generator may need to be adapted to handle pre-parsed comments which would involve adding two lines.

> YARD's parsing architecture also allows much more leeway to customizing the parsing phase than RDoc's limited markup preprocessing APIs. And yes, these APIs are used in at least a few plugins I know of.
> 
>> RDoc has several APIs for adding such extensions, including plugins allowing alternate generators and additional directives:
>> 
>> http://rdoc.rubyforge.org/RDoc/Generator.html
>> http://rdoc.rubyforge.org/RDoc/RDoc.html
>> http://rdoc.rubyforge.org/RDoc/Markup/PreProcess.html#method-c-register
>> 
>> (I do not know if the APIs are sufficient as I've never received any communication from anyone attempting to use them for this task.  I would love some feedback on this matter.)
>> 
> It's great that RDoc has these APIs. I've never used any of them myself, so I can't comment on their usefulness (at first glance they look good). However, a few questions/notes:
> 
> - How is support for new constructs (DSLs) added in Ruby code? This one isn't very specific to this discussion, since Ruby core code doesn't use any non-standard DSLs, but I've found that this is one of the most powerful YARD APIs.

I built a proof-of-concept parser for Rakefiles:

https://github.com/rdoc/rdoc-rake

Since I am still supporting Ruby 1.8.7 I haven't been able to drop the irb-based pure-ruby parser.

Additionally there's Hugh Sasse's perl/pod parser:

https://github.com/rdoc/rdoc-perl_pod

And a fortran parser.

All these parsers build an RDoc::CodeObject tree which the generator handles transparently.  rdoc-rake subclasses RDoc::Include, RDoc::NormalModule and RDoc::AnyMethod to have a better-looking implementation.

> - Are there any ways to hook into the information generated by directives *after* they've been declared? For instance, can you check for the existence of a specific directive inside of a custom Generator? Again, this kind of functionality is used extremely often in YARD, and is extremely important to outputting documentation for arbitrary @tag-style metadata.

There is no restriction.  If a directive is unhandled it's stored in the code object's metadata hash.

> - Can the preprocessing API handle blocks of text (ones including newlines)? Certain tags require this (@example, for instance).

Currently, no.  Calling a registered handler with a third parameter (the comment's text) could add this functionality.

>> Finally, I'm unsure why you've never emailed me or otherwise contacted me to propose adding extensions to RDoc to support YARD-style comments.  I am not opposed to an extension that would support it.
> 
> This is the first time I've proposed creating a unified style. This feature came at the request of Yugui to have a formalized discussion of the proposal I made in the ruby-core list linked above. It originally had nothing to do with RDoc specifically (and still doesn't). I have no problem adding this support to RDoc (in fact I suggested it in the previous email). I don't care which tool is ultimately used for generation, be it RDoc or YARD. However, would RDoc be generating documentation for the C API?

It wouldn't be much work to add, perhaps a week to fully polish.  My boss says I can work on this in place of company business, so I'll give it a shot.