From: Timothy Goddard <interfecus@...>
Date: 2006-10-12T04:45:13+09:00
Subject: Re: Script to fetch Wikipedia text

Kyrre Nyg�rd wrote:
> Hey!
>
> 	I'm involved in a few research projects, and like to keep my
> information well organized. I usually get most of it from Wikipedia,
> however, I hate printing HTML articles to PDF. I'd rather want them
> in pure, well laid out text. And I'm sure others would too. Being
> able to master ones knowledge provides a warm inner peace.
>
> 	Hence I've tried dumping the output from text browsers such as w3m,
> elinks, lynx etc. I am, however, only interested in the articles
> themselves, not their links, views, toolboxes, search bars, other
> available languages and so on. I tried running a whole bunch of
> regular expressions over the output, but that really felt like the hard way.
>
> 	So some guy gave me this:
>
> #!/usr/bin/env ruby
>
> require 'rexml/document'
> require 'cgi'
> require 'tempfile'
> require 'open-uri'
>
> url = 'http://en.wikipedia.org/wiki/Special:Export/' +
> CGI::escape(ARGV.join(" ").strip.squeeze(' ').tr(' ',
> '_')).gsub(/%3[Aa]/,':').gsub(/%2[Ff]/,'/').gsub(/%23/,'#')
>
> open(url) { |f|
>    puts REXML::XPath.first(REXML::Document::new(f.class == Tempfile ?
> f.open : f), '//text').text
> }
>
> 	Which seem to take advantage of Wikipedia's special export feature,
> which really seems cool. However there's a few issues. First, the
> script looks kinda complex. I'm sure there's a simpler way of writing
> it. Second, it does not yet output the kind of pure and well laid out
> text as it should. For instance, on
> http://en.wikipedia.org/wiki/GNU_Hurd, it outputs:
>
> ########## BEGIN
>
> {{Infobox_Software
> | name = GNU Hurd
> | logo = [[Image:Hurd-logo.png]]<br />
> | developer = [[Thomas Bushnell| Michael (now Thomas) Bushnell]]
> (original developer) and various contributors
> | latest_release_version =
> | latest_release_date =
> | operating_system = [[GNU]]
> | genre = [[Kernel (computer science)|Kernel]]
> | family = [[POSIX]]-conformant [[Unix]]-Clones
> | kernel_type = [[Microkernel]]
> | license = [[GNU General Public License|GPL]]
> | source_model = [[Free software]]
> | working_state = In production / development
> | website = [http://www.gnu.org/software/hurd/hurd.html www.gnu.org]
> }}
> {{redirect|Hurd}}
> '''The GNU Hurd''' is a computer operating system [[Kernel (computer
> science)|kernel]]. It consists of a set of [[Server
> (computing)|servers]] (or [[daemon (computer software)|daemons]], in
> [[Unix]]-speak) that work on top of either the [[GNU Mach]]
> [[microkernel]] or the [[L4 microkernel family|L4 microkernel]];
> together, they form the [[kernel (computer science)|kernel]] of the
> [[GNU]] [[operating system]].  It has been under development since
> [[1990]] by the [[GNU]] Project and is distributed as [[free
> software]] under the [[GNU General Public License|GPL]].  The Hurd
> aims to surpass [[Unix]] kernels in functionality, security, and
> stability, while remaining largely compatible with them. This is done
> by having the Hurd track the [[POSIX]] specification, while avoiding
> arbitrary restrictions on the user.
>
> "HURD" is an indirectly [[recursive acronym]], standing for "HIRD of
> [[Unix]]-Replacing [[Daemon (computer software)|Daemons]]", where
> "HIRD" stands for "HURD of Interfaces Representing Depth". It is also
> a play of words to give "[[herd]] of [[wildebeest|gnus]]" reflecting
> how it works.
>
> ==Development history==
> Development on the GNU operating system began in 1984 and progressed
> rapidly. By the early 1990s, the only major component missing was the kernel.
>
> Development on the Hurd began in [[1990]], after an abandoned kernel
> attempt started from the finished research [[Trix (kernel)|Trix]]
> operating system developed by Professor [[Steve Ward (Computer
> Scientist)| Steve Ward]] and his group at [[Massachusetts Institute
> of Technology| MIT]]'s [[Laboratory for Computer Science]] (LCS).
> According to [[Thomas Bushnell| Michael (now T
> homas) Bushnell]], the initial Hurd architect, their early plan was
> to adapt the [[BSD]] 4.4-Lite kernel and, in hindsight, "It is now
> perfectly obvious to me that this would have succeeded splendidly and
> the world would be a very different place today".<ref>{{cite web |
> url = http://www.groklaw.net/article.php?story=20050727225542530 |
> title = The Hurd and BSDI|accessdate = 2006-08-08 | author = Peter H.
> Salus | work = The Daemon, the GNU and the Penguin}}</ref> However,
> due to a lack of cooperation from the [[University of California,
> Berkeley|Berkeley]] programmers, [[Richard Stallman]] decided instead
> to use the [[Mach microkernel]], which subsequently proved
> unexpectedly difficult, and the Hurd's development proceeded slowly.
>
> ########## END
>
> This should instead be something like:
>
> ########## BEGIN
>
> http://en.wikipedia.org/wiki/GNU_Hurd
>
> Name = GNU Hurd
> Developer = Thomas Bushnell (original developer) and various contributors
> Operating_system = GNU
> Genre = Kernel (computer science)
> Family = POSIX-conformant Unix-Clones
> Kernel type = Microkernel
> License = GNU General Public License
> Source model = Free software
> Working state = In production / development
> Website = http://www.gnu.org/software/hurd/hurd.html
>            http://www.gnu.org
>
>
> The GNU Hurd is a computer operating system. It consists of a set of
> servers (or daemons, in Unix-speak) that work on top of either the
> GNU Mach microkernel or the L4 microkernel; together, they form the
> kernel of the GNU operating system.  It has been under development
> since 1990 by the GNU Project and is distributed as free software
> under the GPL. The Hurd aims to surpass Unix kernels in
> functionality, security, and stability, while remaining largely
> compatible with them. This is done by having the Hurd track the POSIX
> specification, while avoiding arbitrary restrictions on the user.
>
> ``HURD'' is an indirectly recursive acronym, standing for ``HIRD of
> Unix-Replacing Daemons", where ``HIRD'' stands for ``HURD of
> Interfaces Representing Depth". It is also a play of words to give
> ``herd of gnus'' reflecting how it works.
>
> Development history
>
> Development on the GNU operating system began in 1984 and progressed
> rapidly. By the early 1990s, the only major component missing was the kernel.
>
> Development on the Hurd began in 1990, after an abandoned kernel
> attempt started from the finished research Trix operating system
> developed by Professor Steve Ward and his group at MIT's Laboratory
> for Computer Science (LCS). According to Michael (now Thomas)
> Bushnell, the initial Hurd architect, their early plan was to adapt
> the BSD 4.4-Lite kernel and, in hindsight, "It is now perfectly
> obvious to me that this would have succeeded splendidly and the world
> would be a very different place today". However, due to a lack of
> cooperation from the Berkeley programmers, Richard Stallman decided
> instead to use the Mach microkernel, which subsequently proved
> unexpectedly difficult, and the Hurd's development proceeded slowly.
>
> ########## END
>
> 	Looks real gorgeous doesn't it? Had I only been skilled enough to do
> this myself. Which brings me to my question: Is anybody out there
> willing to help me fix my script?
>
> Thanks a lot,
> Kyrre

It sounds like the Wikipedia API would be best for this. Visit the
script without parameters at http://en.wikipedia.org/w/query.php for
instructions on its use. For Ruby programs you're probably best off
retrieving YAML data as it's easiest to work with.

require 'open-uri'
require 'yaml'

module Wikipedia
  class WikipediaException < Exception
  end

  def self.query(*titles)
    result =
open("http://en.wikipedia.org/w/query.php?what=content&format=yaml&titles=#{titles.flatten.join('|')}")
    data = YAML.load(result)
    if data['error']
      raise WikipediaException, data['error'].values.first
    end
    data
  end
end