From: "Jesse P." <j.prabawa@...>
Date: 2008-01-06T00:39:58+09:00
Subject: Re: Regexp for matching UTF-8 characters without close tag

Hi Tiziano,

My apologies. It seems that I have oversimplied the problem due to my
lack of understanding for UTF-8.

The actual string is an xml file I obtained from flickr at
http://api.flickr.com/services/rest/?method=flickr.people.getInfo&api_key=44dfd94b104d544f8f80b521a70429e3&user_id=55669962%40N00&api_sig=6a39aab2fb665e24d2b6e1cef9d0be27:
An excerpt is as follows:
<?xml version="1.0" encoding="utf-8" ?>
<rsp stat="ok">
<person id="55669962@N00" nsid="55669962@N00" isadmin="0" ispro="0"
iconserver="136" iconfarm="1">
	<username>(_.·´¯`·â"¢â(tm) â'ª Emirates Wizard â'ªâ(tm) â"¢Â·Â</
username>
	<realname />
	<mbox_sha1sum>0b88a178b28c40ff81d44c5ae475438abec2009c</mbox_sha1sum>
	<location />
	<photosurl>http://www.flickr.com/photos/emirates_wizard/</photosurl>
	<profileurl>http://www.flickr.com/people/emirates_wizard/</
profileurl>
	<mobileurl>http://m.flickr.com/photostream.gne?id=5467956</mobileurl>
	<photos>
		<firstdatetaken>2006-07-16 15:22:42</firstdatetaken>
		<firstdate>1162548449</firstdate>
		<count>36</count>
	</photos>
</person>
</rsp>

The part of the xml that is causing the problem is in the <username>
tag which if in ruby, is represented with octals as:
"<username>(_.\302\267\302\264\302\257`\302\267\342\204\242\342\231
\342\202\252 Emirates Wizard \342\202\252\342\231
\342\204\242\302\267\302</username>"

Note that the XML says that the contents are in UTF-8. So when I use
REXML to process this xml, after it processes the the tag
"<username>", it is left with
string = "(_.\302\267\302\264\302\257`\302\267\342\204\242\342\231
\342\202\252 Emirates Wizard \342\202\252\342\231
\342\204\242\302\267\302</username>"

I just checked and if I match this string with TEXT_PATTERN = /
\A([^<]*)/um, I get the text and also the close tag.
TEXT_PATTERN = /\A([^<]*)/um
text_data = string.match(TEXT_PATTERN).to_s
=> "(_.\302\267\302\264\302\257`\302\267\342\204\242\342\231
\342\202\252 Emirates Wizard \342\202\252\342\231
\342\204\242\302\267\302</username>"

Assuming that the XML has some malformed data (some are not actually
UTF-8), is there anyway that I can process the xml as it is and only
treat the malformed data differently? (e.g. you mentioned that the
\302 character is not a UTF-8 character)

Best regards,

Jesse



On Jan 5, 10:45 pm, Tiziano Merzi <giua...@gmail.com> wrote:
> Jesse P. wrote:
> > Hi all,
>
> > Im trying to solve this problem:
>
> > string = "\302</u>"
> > TEXT_PATTERN = /\A([^<]*)/um
> > text_data = string.match(TEXT_PATTERN).to_s
> > => "\302</u>"
>
> A solution may be :
>
> require 'iconv'
> string = "\302</u>"  # string isn't in utf-8 \302 in utf is \303\202
> string = Iconv.conv("UTF-8","ISO-8859-1",string)
> TEXT_PATTERN = /\A([^<]*)/um
> text_data = string.match(TEXT_PATTERN).to_s
> text_data = Iconv.conv("ISO-8859-1","UTF-8",text_data)
>
> puts text_data
>
> \A -> beginig of line
> --
> Posted viahttp://www.ruby-forum.com/.