From: Mark Thomas <mrt@...>
Date: 2008-03-01T06:19:55+09:00
Subject: Re: Need a regex searching html code

On Feb 29, 2:03 pm, William James <w_a_x_...@yahoo.com> wrote:
> On Feb 29, 7:50 am, Mark Thomas <m...@thomaszone.com> wrote:
>
> > All the regex solutions provided will break with the following
> > perfectly valid HTML:
>
> > <div class="info">
> > <h5 >Tagline:</h5>
> > Yippee Ki Yay Mo - John 6:27
> > </div>
>
> Easily fixed.

> def find_header header, html
>   html.scan( %r{<div.*?>(.*?)</div\s*>}im ).flatten.
>   each{|s|
>     return $1.strip if s =~ %r{<h5\s*>#{header}</h5\s*>(.*)}im }
>   return nil
> end

Easily broken again.

<div class="info">
<h5 class="header">Tagline:</h5>
Yippee Ki Yay Mo - John 6:27
</div>

The point is, regex-based parsing is fragile, and is provably
incomplete for parsing arbitrarily nested structures like HTML. A real
parser (such as a recursive descent parser) is needed. I use regular
expressions often, but when parsing HTML, XML, or other nested data, I
reach for other tools.

> > This is one of many reasons it is a BAD idea to use regexes to parse
> > HTML. Regular expressions are simply not the right tool for the job.

> Who told you that they are not?  And why did you take his word for it?

Experience, for one. Until I really understood parsers, I tended to
use regular expressions for everything. I've been using regular
expressions for a LONG time, and I am very comfortable with them. But
parsing HTML was always troublesome.

This has been discussed for years e.g. in Perl circles (PerlMonks,
etc) where it is well known that regexes do not fit nested data.
People with questions asking how to parse HTML with a regex will get
chided, especially with so many good parsers available in Perl. There
are good parsers available in Ruby now too, so people should be
encouraged to use them.

> Does hpricot use regular expressions?

Of course not.