From: Lance Bradley <l@...>
Date: 2009-02-12T11:22:57+09:00
Subject: Re: hpricot or nokogiri?

I've been going through a similar situation with my current project. I 
was initially using Hpricot, and was very frustrated by the lack of 
documentation and some of the lingering bugs. I've now switched to 
nokogiri and have been very impressed with it.

I'm now running into some of the robustness issues that are faced when 
you process data from the open web, like Dan alluded to. I'm using 
nokogiri's sax implementation, and I've ran into some problems with 
handling html entities, rather they are preserved or decoded into utf-8. 
In both cases, nokogiri will quit calling my start and end element 
handlers, but continue to call my character handler after an entity is 
seen. Specifically, I've noticed this behavior when it sees &nbsp; and 
&#8230;. Has anyone else experienced this and have any advice to share? 
I appreciate it!
-lance

(here's my code)

class Nokogiri::XML::SAX::Document
  attr_accessor :rhtml
  def initialize
    @rhtml = ""
    @keep_text = true
    @keep_elements = %w{ br p img ul ol title li div table head body 
meta base blockquote }
  end

  def start_element name, attrs = []
    puts "start element called: " + name
    if @keep_elements.include?(name)
      puts "keeping: #{name}"
      @rhtml << "<#{name}>\n"
    end
    if ['script', 'style'].include? name
      @keep_text = false
    end
  end

  def characters text
    #@rhtml << @coder.decode( text ) if @keep_text
    @rhtml << text if @keep_text
    puts text
  end

  def end_element name
    puts "end element called: " + name
    if @keep_elements.include?(name)
      @rhtml << "</#{name}>\n"
    end
    if ['script', 'style'].include? name
      @keep_text = true
    end
  end

end

html = open(ARGV[0], 'r').collect { |l| l }.join

#coder = HTMLEntities.new
#html = coder.decode(html)

Tidy.path = '/usr/lib/libtidy-0.99.so.0'
xml = Tidy.open(:show_warnings=>true) do |tidy|
  tidy.options.output_xml = true
  #tidy.options.char_encoding = 'utf8'
  tidy.options.preserve_entities  = true
  xml = tidy.clean(html)
end

doc = Nokogiri::XML::SAX::Document.new
parser = Nokogiri::XML::SAX::Parser.new(doc)

parser.parse(xml)

puts "doc:"
puts doc.rhtml.gsub(/\n+/, "\n")
-- 
Posted via http://www.ruby-forum.com/.