[ruby-core:33949] Re: [Ruby 1.9-Feature#4211][Open] Converting the Ruby and C API documentation to YARD syntax

On 12/27/2010 5:03 PM, Eric Hodel wrote:
> I highly support having a unified documentation tool and style for Ruby, but I personally dislike an enforced template like doxygen requires.

What do you mean by an "enforced template"? Doxygen (or an alternative @tag based syntax) doesn't "require" anything from you, you're free to add the information you choose, just as with any documentation. @tag syntaxes are merely an *equivalent* to the existing documentation. The requirements come from a style guide, and that's part of documentation process, not some tool. However, since the Ruby C API is already starting to make use of @tag syntax with Doxygen, there is little reason not to leverage this syntax as part of the unified style. Furthermore, YARD already supports this syntax. I'd have no problems with RDoc adding support, too. I don't know of any tools that can document the C end of things, unless RDoc plans on adding support for the C APIs (which I guess I would not object to).

> I find the doxygen-style template leads to stilted wording and low-quality documentation as the author merely has to check all the boxes to consider their work "complete".  

I'm not sure what documentation you're basing this on, so I'll simply say this: the quality of any documentation ultimately depends on the effort that is put into writing it. People who "check all the boxes" are going to write poor documentation in any form, be it tag based or any other form. I've seen plenty of *good* documentation using tag based syntax, and I'm sure we can share plenty more anecdotes. One thing I know, though, of the *well documented* code I know, almost all of them use tag based syntax-- this includes Java's stdlib, Python's stdlib, Cocoa's APIS, many well documented JS libs like jQuery, and more. You can interpret this as you wish, but I see this as a positive trend. 

> I've read documentation generated by such tools many times and have always found them difficult to navigate as there is not enough context to understand the interaction between all the components of an API.

What does RDoc do better, here? Again, this is not the fault of @tag based syntaxes-- it is the fault of the tools. For instance, doxygen has some rough edges in displaying documentation, but this is the fault of the tool, not the use of @tags. Keep in mind that I'm not proposing a tool, just a *unified documentation style*. RDoc could certainly implement this if you think it does the best job at displaying information to a user.

> If ruby adopts a documentation minimum standard I don't think an enforced template should be our only requirement.

I agree. As mentioned, the goal is to supply a unified *style*, not just a "template". @tag based syntax is merely a part of this style, since it's already being used in the C API, and there is no alternative to properly documenting the C end of things than @tag based tools. It therefore makes sense to continue in this vein and unify the documentation in this way. 

However, it's important to realize that supplying a "bare minimum" guideline to a documenter does a lot to improve the quality of documentation. Obviously, manual checks would have to be done to make sure people are not just "checking off boxes", but having those tags there lets us know what to look for when reviewing documentation, and lets *them* know what they should be writing. This is an improvement over the current adhoc "just-write-something" methodology. Tags also give us the ability to implement tooling later, because the docs are much more easily parse-able.

> I question merging YARD into Ruby and replacing RDoc.
> 
> Why aren't extensions or enhancements to RDoc being made instead?

YARD has fairly significant architectural differences that don't lend well to simply writing extensions. YARD is actually a superset of RDoc, and not the other way around, so a merge of functionality wouldn't really work. For instance, YARD is not only a "parse and generate" tool. It is also often used as a 2 phase: "parse and store", "generate from stored data" tool (as exemplified by the 'yard server' tool). RDoc does not have this phase, so much of the functionality YARD has could not be implemented easily. YARD's parsing architecture also allows much more leeway to customizing the parsing phase than RDoc's limited markup preprocessing APIs. And yes, these APIs are used in at least a few plugins I know of.

> 
> Based on download numbers from rubygems.org it appears that a minority (about 25%) of rubyists prefer YARD to rdoc. 

I'm not sure it's exactly fair to correlate popularity with preference here. PHP is probably downloaded more than Ruby, but I really don't believe programmers prefer PHP to Ruby. 

Certainly RDoc has more users, as more people know about RDoc's existence than YARD, since it comes packaged with Ruby. In addition, most users simply don't have the need for YARD's extra features, so RDoc is "good enough". I highly doubt any of this correlates with "preference", though. The http://rubydoc.info project Nick Plante and I have been running for the last little while has been greatly successful and has been receiving quite a lot of traffic and good feedback from users. Given more even exposure to both tools, I'd bet the real numbers would be fairly different. 

> RDoc has several APIs for adding such extensions, including plugins allowing alternate generators and additional directives:
> 
> http://rdoc.rubyforge.org/RDoc/Generator.html
> http://rdoc.rubyforge.org/RDoc/RDoc.html
> http://rdoc.rubyforge.org/RDoc/Markup/PreProcess.html#method-c-register
> 
> (I do not know if the APIs are sufficient as I've never received any communication from anyone attempting to use them for this task.  I would love some feedback on this matter.)
It's great that RDoc has these APIs. I've never used any of them myself, so I can't comment on their usefulness (at first glance they look good). However, a few questions/notes:

- How is support for new constructs (DSLs) added in Ruby code? This one isn't very specific to this discussion, since Ruby core code doesn't use any non-standard DSLs, but I've found that this is one of the most powerful YARD APIs.
- Are there any ways to hook into the information generated by directives *after* they've been declared? For instance, can you check for the existence of a specific directive inside of a custom Generator? Again, this kind of functionality is used extremely often in YARD, and is extremely important to outputting documentation for arbitrary @tag-style metadata.
- Can the preprocessing API handle blocks of text (ones including newlines)? Certain tags require this (@example, for instance).

> 
> Finally, I'm unsure why you've never emailed me or otherwise contacted me to propose adding extensions to RDoc to support YARD-style comments.  I am not opposed to an extension that would support it.

This is the first time I've proposed creating a unified style. This feature came at the request of Yugui to have a formalized discussion of the proposal I made in the ruby-core list linked above. It originally had nothing to do with RDoc specifically (and still doesn't). I have no problem adding this support to RDoc (in fact I suggested it in the previous email). I don't care which tool is ultimately used for generation, be it RDoc or YARD. However, would RDoc be generating documentation for the C API?

- Loren