From: ruby talk <rubytalk@...>
Date: 2006-01-23T07:43:37+09:00
Subject: Re: remove non-ASCII characters in a string

What is considered an ascii char? i used the
http://www.lookuptables.com/ chart from ! to ~ or 33 .. 127

class String
def remove_nonascii(replacement)
n=self.split("")
self.slice!(0..self.size)
n.each{|b|
 if b[0].to_i< 33 || b[0].to_i>127 then
	 self.concat(replacement)
	 else
	 self.concat(b)
 end
 }
 self.to_s
end
end
 require 'test/unit'

 class TestAsciify < Test::Unit::TestCase
   def test_asciify
     assert_equal "I�t�rn�ti�n�liz�ti�n".remove_nonascii("?"),
"I?t?rn?ti?n?liz?ti?n"
     assert_equal "M�torhead".remove_nonascii("(removed)"), "M(removed)torhead"
   end
 end


On 1/22/06, Ezra Zygmuntowicz <ezmobius@gmail.com> wrote:
>         I have a need for something like this as well. But I need to
> replace the chars with something plain ascii besides a placeholder.
> Any ideas how to do that?
>
>         I ended up finding the escape codes for all the chars like "\322"
> and friends so I could replace say curly quotes with standard quotes
> and stuff like that.
>
>         I will play with your code a bit and see if I can make it do what I
> want. Thanks for sharing it though.
>
> Cheers-
> -Ezra
>
>
>
> On Jan 22, 2006, at 10:41 AM, Levin Alexander wrote:
>
> > Hi,
> >
> > i needed a method to convert a piece of text to plain ascii and
> > replace all non-ascii chars with a placeholder.  I could not find
> > anything in the stdlib so I wrote one.
> >
> > I'd love to hear your comments. (or pointers to existing libraries for
> > this task)
> >
> > -Levin
> >
> >
> > #!/usr/bin/ruby
> >
> > require 'iconv'
> >
> > class String
> >
> >   # removes all characters which are not part of ascii
> >   # and replaces them with +replacement+
> >   #
> >   # +replacement+ is supposed to be the same encoding as +source+
> >   #
> >   def asciify(replacement = "?", target = "ASCII", source = "UTF-8")
> >     intermediate = "UCS-4"
> >     pack_format = "N*"
> >     i = Iconv.new(intermediate, source)
> >
> >     u16s = i.iconv(self)
> >     repl = i.iconv(replacement).unpack(pack_format)
> >
> >     s = u16s.unpack(pack_format).collect { |codepoint|
> >       codepoint < 128 ? codepoint : repl
> >     }.flatten.pack(pack_format)
> >
> >     return Iconv.new(target, intermediate).iconv(s)
> >   end
> > end
> >
> > if __FILE__ == $0
> >   require 'test/unit'
> >
> >   class TestAsciify < Test::Unit::TestCase
> >     def test_asciify
> >       assert_equal "I�t�rn�ti�n�liz�ti�n".asciify, "I?t?rn?ti?n?liz?
> > ti?n"
> >       assert_equal "M�torhead".asciify("(removed)"), "M(removed)
> > torhead"
> >     end
> >   end
> > end
>
> -Ezra Zygmuntowicz
> WebMaster
> Yakima Herald-Republic Newspaper
> http://yakimaherald.com
> ezra@yakima-herald.com
> blog: http://brainspl.at
>
>
>
>