From: Brian Candler Date: 2009-04-17T04:44:50+09:00 Subject: Re: querying using HTTP James Gray wrote: > I'm pretty sure you are in the minority with this opinion. Quite possibly :-) > You really like this? > > $ ruby -e 'p "R�sum�"[0..1]' > "R\303" > > How often is that going to be the desired result? Well, if I were extracting the first two bytes from a JPEG header, that would be exactly what I'd expect. I've very rarely wanted to extract the first two *characters* from a string. I can think of one example: a string truncation helper in a web page. def trunc(string, maxlen=50) if string.length > maxlen string = string[0,maxlen-3] + "..." end string end I'll certainly agree that's something you'd want to do, and /.{,50}/u is an ugly way of doing it. In any case, I'm not saying there shouldn't be any m17n support, or even that tagging strings with encodings is in itself wrong, as long as the semantic implications are made clear. The number one bugbear I have is that (unless you take a number of specific steps to avoid it), program behaviour is inconsistent. You can run the *same* program with exactly the *same* input data on two different machines, and they will process it differently, possibly even crashing in one case. If someone has a problem running your app, it's now insufficient just to ask what O/S and ruby version they are running in order to be able to replicate the problem. Consider an app which is bundled with HTML templates, which the app reads using File.read(). The templates happen to be written using, say, UTF-8. It all works fine on my machine, and passes all tests. However it barfs when run on someone else's machine, because their environment variables are different. I think that LC_ALL is a very poor predictor of what encoding a specific file is in. Ruby doesn't trust it for source files (it uses #encoding tags instead), so why trust it for data? Now, if the default external encoding were fixed as (say) UTF-8, that would be more sane. The default behaviour would then be the same on any machine where ruby is installed: - File#gets returns a string with encoding='UTF-8' - File#read returns a string with encoding='BINARY' unless explicitly overridden, e.g. when the file is opened. So if these hypothetical HTML templates are written in ISO-8859-15, you would be forced to declare this in your program. In any case, I'm used to having my data treated as binary unless I explicitly ask otherwise. e.g. $ echo "ßßß" | wc 1 1 7 $ echo "ßßß" | wc -m 4 [Ubuntu Hardy, default setup with LANG=en_GB.UTF-8] > Can you list what's not yet covered in my blog series? I've posted a bunch of lists before. Every time I try out some feature, because it's undocumented, the test turns up more questions than it answers. Maybe I really should go ahead and document it all, but that would be a very large project. Trying out in irb used to be a good way to test ruby, but that's no good in ruby 1.9 because it's not consistent with script behaviour. For example: $ irb19 irb(main):001:0> "foo".encoding => # irb(main):002:0> /foo/.encoding => # irb(main):003:0> "fooß".encoding => # irb(main):004:0> /fooß/.encoding => # Now try running this program: p "foo".encoding p /foo/.encoding p "fooß".encoding p /fooß/.encoding It barfs on the multi-byte chars. That's reasonable in the absence of knowledge about the source file, so now add an #encoding line: #encoding: UTF-8 p "foo".encoding p /foo/.encoding p "fooß".encoding p /fooß/.encoding and you still get a different answer to IRB. The first string gets an encoding as UTF-8 instead of US-ASCII; and yet the /foo/ regexp gets an encoding of US-ASCII in both cases. This is compounded by the hidden state which remembers whether a particular string is all 7-bit characters or not. That is, although "foo" and "fooß" are both marked as having identical encoding UTF-8, they are actually treated *differently* by the encoding rules. You have to test using the #ascii_only? method. And yet a regexp literal apparently follows a different rule. Except when you are in IRB. > It means that I think your comments are doing harm to the 1.9 > migration and I can't find the good you are doing to balance that. I don't think what I'm saying would stop any library author from modifying their library to work with 1.9 if they so wish. They have to make up their own minds. I believe the worst long-term problems are likely to be C extensions. I have seen no hints at all for C extension writers on how to handle strings properly (especially the hidden ascii_only? state) so I believe these are likely to have obscure bugs for some time. Regards, Brian. -- Posted via http://www.ruby-forum.com/.