[ ruby-Bugs-3654 ] Syck fails with parse error on parsing multiple documents

Bugs item #3654, was opened at 2006-02-24 15:45
You can respond by visiting: 
http://rubyforge.org/tracker/?func=detail&atid=1698&aid=3654&group_id=426

Category: Standard Library
Group: None
Status: Open
Resolution: None
Priority: 3
Submitted By: Hannes Wyss (hwyss)
Assigned to: Nobody (None)
Summary: Syck fails with parse error on parsing multiple documents

Initial Comment:
195-documents.yaml is a File (12M) containing 195 complex YAML-Documents (I can provide the file if necessary)

Without custom domain-types
ruby -r'yaml' -e 'docs = []; File.open("195-documents.yaml") { |fh| YAML.each_document(fh) { |doc| docs.push(doc) } }; puts docs.size'
-> 18

With custom domain-types
ruby -r'yaml' -r'domain-types' -e 'docs = []; File.open("195-documents.yaml") { |fh| YAML.each_document(fh) { |doc| docs.push(doc) } }; puts docs.size'
-> /usr/lib/ruby/1.8/yaml.rb:217:in `load_documents': syntax error on line 23806, col 36: `on (<0,1%). Rarement peuvent se' (ArgumentError)
        from /usr/lib/ruby/1.8/yaml.rb:217:in `each_document'
        from -e:1
        from -e:1


With a backport from ruby 1.9 the Problem goes away. Here is a minimal patch:

diff -Nbr -U4 ruby-1.8.4/ext/syck/syck.c yarv-0.4.0/ext/syck/syck.c
--- ruby-1.8.4/ext/syck/syck.c  2005-09-13 05:58:32.000000000 +0200
+++ yarv-0.4.0/ext/syck/syck.c  2006-02-20 07:16:38.000000000 +0100
@@ -1,9 +1,9 @@
 /*
  * syck.c
  *
- * $Author: why $
- * $Date: 2005/09/13 03:58:32 $
+ * $Author: ocean $
+ * $Date: 2006/02/03 06:10:21 $
  *
  * Copyright (C) 2003 why the lucky stiff
  */
 #include "ruby.h"
@@ -409,11 +409,8 @@
     if ( p->token == NULL )
         return 0;

     skip = p->limit - p->token;
-    if ( skip < 1 )
-        return 0;
-
     if ( ( count = p->token - p->buffer ) )
     {
         S_MEMMOVE( p->buffer, p->token, char, skip );
         p->token = p->buffer;



Without custom domain-types
ruby -r'yaml' -e 'docs = []; File.open("195-documents.yaml") { |fh| YAML.each_document(fh) { |doc| docs.push(doc) } }; puts docs.size'
-> 195

With custom domain-types
ruby -r'yaml' -r'domain-types' -e 'docs = []; File.open("195-documents.yaml") { |fh| YAML.each_document(fh) { |doc| docs.push(doc) } }; puts docs.size'
-> 195


Request: 

----------------------------------------------------------------------

You can respond by visiting: 
http://rubyforge.org/tracker/?func=detail&atid=1698&aid=3654&group_id=426