From: Krzysieq <krzysieq@...>
Date: 2008-09-19T20:07:04+09:00
Subject: Re: How to clean an xml files from non-utf-8 chars?

------=_Part_20648_30327322.1221822874462
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit
Content-Disposition: inline

Ok, I tried all previous suggestions, neither worked (gsub idea, TRANSLIT,
IGNORE or the one from the link posted by Mark Thomas). In fact, the last
two don't seem to have done anything, while gsub seems to do too much -
seems like it has damaged the xml structure in some way, which seems very
strange to me. I don't really care about the data inside, but I need the xml
to remain valid.

@Gregory - that's true, it may not. However, the places where I found the
funny characters are text nodes inside xml documents, and there aren't that
many of them. Surely, one is many enough to break the whole thing, but
typically there's very few and it seems more like corrupted database data. I
think they store some newspaper articles there or pieces of news. I learned
from the team who maintain that database in their app, that typically it
should all be ISO-8859-1, but for some reason it's not always the case.
Hence the idea with corrupted data seems quite likely.

Thanks for any help You can provide me with :)
Cheers,
Chris

2008/9/18 Mark Thomas <mark@thomaszone.com>

> On Sep 18, 9:25 am, Krzysieq <krzys...@gazeta.pl> wrote:
> > [Note:  parts of this message were removed to make it a legal post.]
> >
> > Unfortunately, there's no way telling the original encoding. I would
> rather
> > go for some method of removing / substituting the chars that don't belong
> > there, but the method first suggested by Brian doesn't seem to work for
> some
> > reason. Does anyone have another option?
>
> Try the iconv solutions with latin-1 (iso-8859-1) as the From. That's
> as close as you can get to a one-byte "anything-goes" encoding.
>
> -Mark.
>
>

------=_Part_20648_30327322.1221822874462--