Re: Ruby.bah! (was Re: XMLParser, NQXML... and also RAA.succ)

> >> Ruby, on the other hand, has two going on three different
> >> formats, if I'm understanding what Sean is saying correctly.
> > Sadly yes... however there is a saving grace with rubydoc, it's
> > standard is XML and the other utilities export XML... which means
> > that anyone can write a stylesheet that'll convert rdoc-> rubydoc
> > or rd-> rubydoc. These stylesheets will be included in the base
> > rubydoc installation.
> 
> In some ways, though, I still think that this is probably the wrong
> approach. IMO, either rdoc or rd needs to die. Ideally, rdoc will
> pick up the ability to parse rd comments cleanly (perhaps spitting
> out warnings) so that there only needs to be one primary
> documentation tool.

Agreed, but I don't have any interest in what happens in the embedded
documentation space.  If something new comes along, great, I'll use
that too.  After rubynet and rubydoc have reached critical mass and I
have free time to work on something else, I may spend some time
flushing out an inline documentation format of my own that fits in
nicely with rubydoc that isn't XML.  XML's great for machines, but
when it comes to writing stuff in emacs or any text editor, writing
XML from scratch is the perfect way to develop carpal tunnel.  See the
ruby-doc@ mailing list for details about what I've quasi envisioned,
however I need to support some kind of linking of variables, methods,
and classes.  After spending so much time in the libxml source, I
actually kinda like gnome's doc format... granted it's a rip off of
javadoc and wherever the inventive folks at Sun ripped it off from.

> Okay ... so ... what's the point of rubydoc? IMO, rdoc is quite
> sufficient to produce the necessary documentation for the code. Is
> rubydoc intended to massage that documentation into a nicer format
> that could, theoretically, look good in PDF?

To encapsulate and provide a transformable, publishable (www, TeX,
PDF, text, nroff), includable (I _REALLY_ want to be able to offer
documentation sets/examples that are dynamically driven from the
rubynet.org and rubydoc.org sites and also have static content.  Think
php.net + PostgreSQL's idocs + docbook + perldoc) and indexable (from
the CLI: rubydoc [term/class]).  Nothing anywhere in the programming
world is coming close to offering that.  I as a developer want that.
Others likely do too.

> Don't get me wrong -- I'm not overly enamoured of the default rdoc
> template, but at the moment I'm just too lazy to change it for my
> own purposes and (later) offer it as a replacement template.

Good point, I'll make a not of that request and let you specify a
different txt/nroff stylesheet in your ~/.rubynet/rubydoc.cfg.

> I have to be honest and say that I don't *get* why rubydoc *as a
> tool* is necessary. If you're going to make it an aggregator so that
> the user can have the equivalent of a single interface like users of
> the ActiveState Perl package do ... that's cool.

Basically, yeah.

> If you're going to make it a super-duper version of ri with Rimport,
> even better! (See?  I forgot another Ruby documentation tool.) But
> if it's just going to be Yet Another Documentation Tool, I've got to
> say that I don't see the point.

Naw, that's why I'm incessant on standardizing around XML.  Any of the
Yet-Another-Ruby-Doc formats can be massaged into rubydoc's XML spec
with a stylesheet.  From there, any number of documents can be
made/indexed.

> IMO, rubydoc should be a shell and transformation agent on top of
> rdoc, not a whole new documentation system. But that's just IMO.

Bingo, that's exactly what it's going to do.

> > :-/ Working on stuff by yourself and being in a perpetual state of
> > over commitment and everywhere-all-at-once kinda bites when you're
> > trying to design something "right" and aren't hacking it together.
> > Help wanted/appreciated.
> 
> That's not a criticism, Sean. However, I can't at the moment really
> provide any assistance with the code side -- job searches tend to
> take a lot of time, and I have my own coding projects that I'm
> attacking[snip].

It's alright, I know how that goes.  I have to talk fault in it moving
slow since I'm doing this work outside of the limelight of -talk.
I've actually completely disengaged from the -talk community and only
read through this mailbox when I get a nudge from someone on IRC
hinting that there's something worth checking out.

> I have joined rubynet-devel, though, so I can possibly provide some
> design commentary.

I welcome the addition, there's a good crew of some 30+ lurkers now.
Put enough of them on a list and someone's bound to chirp up with a
patch every now and then.  8-)

> >> It supports -- at least experimentally and I haven't been able to
> >> get it to work yet -- automatic diagram creation.
> > Speaking of, does ruby have a DOT interface? I've wanted to use
> > this for class diagramming on rubynet but haven't looked into it.
> 
> Look at rdoc; there's a dot/ directory in the distribution. I'm not
> exactly sure the best way to use it -- part of the problem could be
> that I use Ruby from Windows, and I don't have dot itself installed
> (: As I said above, rdoc should be the documentation system for
> Ruby; rubydoc should be a way of transforming rdoc output into ri
> database information (a la Rimport), creating a unified API
> reference for Ruby and all the modules installed on the user's
> system, etc.

That's the short term plan.  Glad to know my ideas aren't existing in
a total void.

> >> else. Want to create PDFs instead of HTML files? Write the
> >> appropriate generator and/or template.
> > The joy of XSLT and flow objects.
> 
> Certainly -- and this is probably what rubydoc should do. (Of
> course, I've now tried rdoc's CHM -- Windows HTMLHelp -- output and
> have had mixed results with it. It seems to ignore #:nodoc:
> directives; I may look at that to provide a patch for Dave when he
> gets back, like another patch that I've made to a patch that he made
> in response to a patch that I gave him.)

So long as rdoc does the right thing when it comes to generating XML,
then the stylesheet should handle this correctly.... though I'm not a
Win32 guy atm, but I'm doing an increasing amount of Win32 GUI clients
in FOX so who knows, I'll likely transgress into the depths of MS hell
again at some point in the not too distant future.

> > I'm not a yaml lover, personally. Culturally I think YAML exists
> > as a counter movement to Java/XML and XMLs tendency to get bundled
> > with Java. I can't say as I disagree with the dislike of the
> > Java/M$ developer sentiment, Sun hasn't done much in the way of
> > innovative computing in a while and I wish would just curl up and
> > flop. Tandem makes better hardware anyway. :)
> 
> I think that your cultural analysis is correct. I still think that
> YAML is a useful lightweight format.

No disagreements there, but I just hacked out libxml which seems to do
everything that I need it to at the moment... short of generating DTDs
on the fly, but that'll come here sometime this week.  :)

> > Here's the dilly with supporting multiple files and formats: I
> > don't know what your preference is.
> 
> One format, one file. Period. Don't give me the option of multiple
> formats and files to describe a single package. I picked YAML
> because it appears to be close to the .rubynet_ format that you
> specified. Perhaps as follows (note that I use '...' when there
> could be other attributes or I'm eliding):

Ooooh!  

[snipped the wonderful yaml example]

> IMO, in a single file, this provides everything that you would have
> in the multiple file form -- and it's easier to edit. XML would be
> appropriate, too.

There is no perfect format, IMHO.  XML's good for machines and for
transferring between automated processes.  Yaml's good if you need to
have a human touch down and edit the file and don't want to barrage
him with a zillion characters that the eye has to parse through.
Multiple plain text files, however, are shell friendly.  I honestly
see room for having all three as valid formats for describing a
package.  I actually wonder if I couldn't generate the formats for the
plain text files and the YAML from the XML spec...  hrm, that'd be an
interesting exercise in code generation.... any XSLT/YAML buffs out
there that'd want to take that on?

> I'll re-present this on rubynet-devel with further commentary in a
> couple of days, but I think that there are mistakes being made in
> the design as expressed by the package information files.

!  Excellent!  I look forward to the discussion.

> > Contrary to my sentiment about YAML, I'll likely support a YAML
> > interface for configuring packages just because that's a format
> > that some developers prefer. I personally favor having simple and
> > small files each with a specific format. Makes it easier to
> > manipulate/create the files with sed(1) and find(1). I'm a die
> > hard UNIX guy at heart, what can I say. It showing?
> 
> As I said before, one format and one file. Too many options or files
> will make the project unmaintainable. If you keep the existing
> dot-rubynet files, I can guarantee that I won't be packaging things
> that way -- it's too much work for me, the maintainer. (Ideally,
> even, you would have a cross-platform GUI- or TUI-based interface
> for building and maintaining the file.)

The dot files will get compiled into XML.  From there, a GUI can
operate on the XML.  The dot files are being generated at the moment
with some crude guesses so for me, it's actually really nice.

$ rubynet --generate

[edit dot files and delete false positives/add missing entries]

$ rubynet --compile-module

> > By the time the data hits the rubynet server, the data will be
> > serialized into an XML file. The dot files are used only by an
> > author for describing their package, not for use in the rubynet
> > system. Once things hit the rubynet system, it's XML. Period. For
> > those that are curious, binary data is MIME64 encoded in an
> > element in the rubynet file.
> 
> Blechhhh. base64 encoding is evil unless it's unavoidable. It adds
> an unnecessary 30% or more to the size of the file.

No arguments from me here, but do you know of another way to encode
binary data in an XML file?  I'd love to use something more efficient,
but don't know of anything.  Fortunately with the compression set at 9
the size drops quite dramatically as it seems there are enough
patterns in the mime encoded file.  I'll add support to compress files
individually before they get encoded in the event that this pattern
doesn't hold true.

> It also seems that here, you're planning on mixing metadata and
> content

Correctomundo, that's exactly what I'm doing... and I'm being
quasi-clever about it for several reasons.  I'm trying to create a
format that fullfills three purposes.

1) Is indexable.  XML+XPath takes care of this.
2) Contains meta data for the module that way a skeleton module can be
   distributed instead of the full blown tarball.  Think FreeBSD ports
   here.  The reason for doing this is for ports conformity, and
   because I'd like to make this system attractive to commercial
   vendors who need to distribute modules and have restricted downloads.
3) Can be used as a ubiquitous format that stores all of the packaging
   and file content.  I've got an idea in the back of my head for how
   to convert a tarball into a rubynet package, for example.  As a
   user, I only want to have to download one thing.  Think of this a
   JAR file on steroids, if you will.  While I agree with your next
   point that it's a no-no, by and large, I do think that this is
   miles better than what Perl has and certainly better than the
   simple zip format that JAR files employ.

> -- which I consider a major no-no when it comes to data
> modeling.

See #3 above.

> Frankly, I think that this is a task for which XML is uniquely
> UNSUITED. Sure, XPath helps here, but XML will *never* match the
> power of a properly coded relational database for this sort of
> problem space.

Agreed... there's an element of usability though that can't be matched
if you split things into two files.  Hrm... maybe I should just
concatenate a tarball with the rubynet skeleton file with a small
contents that gives the version of rubynet needed and the sizes of the
XML and package...  hrm.... to be continued on [e-mail].  :)
-sc

-- 
Sean Chittenden