From: Jeremy Hinegardner Date: 2009-08-24T13:25:20+09:00 Subject: OSX -flat_namespace issues show up in hitimes, nokogiri and amalgalite I'm not sure if it is the problem, but it appears that because ruby is compiled with -flat_namespace on Mac OS X that unexpected behavior can occur if an extension links against CoreServices. Here's a fairly indepth example, using ruby 1.8.6. The exact same thing happens on 1.9.1. % ./bin/ruby -v ruby 1.8.6 (2009-08-04 patchlevel 383) [i686-darwin9.8.0] % ./bin/ruby -rrbconfig -e 'puts Config::CONFIG["LDSHARED"]' cc -dynamic -bundle -undefined suppress -flat_namespace % ./bin/ruby -rubygems -e "require 'hitimes'" -e "require 'nokogiri'" -e 'puts "libxml version : #{Nokogiri::LIBXML_PARSER_VERSION}"' WARNING: Nokogiri was built against LibXML version 2.7.2, but has dynamically loaded 2.6.16 HI. You're using libxml2 version 2.6.16 which is over 4 years old and has plenty of bugs. We suggest that for maximum HTML/XML parsing pleasure, you upgrade your version of libxml2 and re-install nokogiri. If you like using libxml2 version 2.6.16, but don't like this warning, please define the constant I_KNOW_I_AM_USING_AN_OLD_AND_BUGGY_VERSION_OF_LIBXML2 before requring nokogiri. libxml version : 20616 Change the order of the requires and all is good: % ./bin/ruby -rubygems -e "require 'nokogiri'" -e "require 'hitimes'" -e 'puts "libxml version : #{Nokogiri::LIBXML_PARSER_VERSION}"' libxml version : 20702 A similar effect happens when using the Amalgalite gem, that case complains about the SQLite version. Both hitimes and amalgalite are my gems, that's how I found this situation, but it could be exposed by someone elses gems too. Why does this happen? The answer is, hitimes, on OSX links against '-framework CoreServices' in order to get access to the 'AbsoluteToNanoseconds()' and 'UpTime()' functions. When hitimes loads and therefore the CoreServices framework loads, CoreServices brings along with it the system libxml and the sytem libsqlite3. We can see this by setting DYLD_PRINT_LIBRARIES=1: % DYLD_PRINT_LIBRARIES=1 ./bin/ruby -rubygems -e "require 'hitimes'" -e "require 'nokogiri'" -e 'puts "libxml version : #{Nokogiri::LIBXML_PARSER_VERSION}"' dyld: loaded: /Users/jeremy/pkgs/ruby-flat_namespace/./bin/ruby dyld: loaded: /usr/lib/libSystem.B.dylib dyld: loaded: /usr/lib/libobjc.A.dylib dyld: loaded: /usr/lib/libgcc_s.1.dylib dyld: loaded: /usr/lib/system/libmathCommon.A.dylib dyld: loaded: /usr/lib/libauto.dylib dyld: loaded: /usr/lib/libstdc++.6.dylib dyld: loaded: /Users/jeremy/pkgs/ruby-flat_namespace/lib/ruby/1.8/i686-darwin9.8.0/thread.bundle dyld: loaded: /Users/jeremy/pkgs/ruby-flat_namespace/lib/ruby/1.8/i686-darwin9.8.0/etc.bundle dyld: loaded: /Users/jeremy/pkgs/ruby-flat_namespace/lib/ruby/1.8/i686-darwin9.8.0/stringio.bundle dyld: loaded: /Users/jeremy/pkgs/ruby-flat_namespace/lib/ruby/1.8/i686-darwin9.8.0/syck.bundle dyld: loaded: /Users/jeremy/pkgs/ruby-flat_namespace/lib/ruby/gems/1.8/gems/hitimes-1.0.4/lib/hitimes/1.8/hitimes_ext.bundle dyld: loaded: /System/Library/Frameworks/CoreServices.framework/Versions/A/CoreServices dyld: loaded: /System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/CarbonCore.framework/Versions/A/CarbonCore dyld: loaded: /System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/CFNetwork.framework/Versions/A/CFNetwork dyld: loaded: /System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/Metadata.framework/Versions/A/Metadata dyld: loaded: /System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/OSServices.framework/Versions/A/OSServices dyld: loaded: /System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/SearchKit.framework/Versions/A/SearchKit dyld: loaded: /System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation dyld: loaded: /System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/AE.framework/Versions/A/AE dyld: loaded: /System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/LaunchServices.framework/Versions/A/LaunchServices dyld: loaded: /System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/DictionaryServices.framework/Versions/A/DictionaryServices dyld: loaded: /System/Library/Frameworks/IOKit.framework/Versions/A/IOKit dyld: loaded: /usr/lib/libicucore.A.dylib dyld: loaded: /System/Library/Frameworks/DiskArbitration.framework/Versions/A/DiskArbitration dyld: loaded: /usr/lib/libbsm.dylib dyld: loaded: /usr/lib/libz.1.dylib dyld: loaded: /System/Library/Frameworks/Security.framework/Versions/A/Security dyld: loaded: /System/Library/Frameworks/SystemConfiguration.framework/Versions/A/SystemConfiguration --> dyld: loaded: /usr/lib/libsqlite3.0.dylib dyld: loaded: /usr/lib/libresolv.9.dylib --> dyld: loaded: /usr/lib/libxml2.2.dylib dyld: loaded: /usr/lib/libxslt.1.dylib dyld: loaded: /Users/jeremy/pkgs/ruby-flat_namespace/lib/ruby/gems/1.8/gems/nokogiri-1.3.3/lib/nokogiri/nokogiri.bundle dyld: loaded: /opt/local/lib/libexslt.0.dylib dyld: loaded: /opt/local/lib/libxslt.1.dylib --> dyld: loaded: /opt/local/lib/libxml2.2.dylib dyld: loaded: /opt/local/lib/libz.1.dylib dyld: loaded: /opt/local/lib/libiconv.2.dylib WARNING: Nokogiri was built against LibXML version 2.7.2, but has dynamically loaded 2.6.16 HI. You're using libxml2 version 2.6.16 which is over 4 years old and has plenty of bugs. We suggest that for maximum HTML/XML parsing pleasure, you upgrade your version of libxml2 and re-install nokogiri. If you like using libxml2 version 2.6.16, but don't like this warning, please define the constant I_KNOW_I_AM_USING_AN_OLD_AND_BUGGY_VERSION_OF_LIBXML2 before requring nokogiri. dyld: loaded: /Users/jeremy/pkgs/ruby-flat_namespace/lib/ruby/1.8/i686-darwin9.8.0/racc/cparse.bundle dyld: loaded: /Users/jeremy/pkgs/ruby-flat_namespace/lib/ruby/1.8/i686-darwin9.8.0/strscan.bundle libxml version : 20616 You can see that the system libxml2, which is version 20616 is loaded first and then the version 20702 that we want to use, is loaded second. So it appears that the -flat_namespace compiler paramter when compiling ruby, and the binary extensions,is shooting us in the foot. Lets make sure. So editing the distributed 'configure' script line 16991 from darwin*) : ${LDSHARED='cc -dynamic -bundle -undefined suppress -flat_namespace'} to darwin*) : ${LDSHARED='cc -dynamic -bundle -undefined dynamic_lookup'} we have to change -undefined too or else we'll get : ld: can't use -undefined warning or suppress with -twolevel_namespace Now we have a new ruby that uses the two_level namespace on OS X % ./bin/ruby -v -rrbconfig -e 'puts Config::CONFIG["LDSHARED"]' ruby 1.8.6 (2009-08-04 patchlevel 383) [i686-darwin9.8.0] cc -dynamic -bundle -undefined dynamic_lookup And when we run the same thing that before blew up before all is well. % DYLD_PRINT_LIBRARIES=1 ./bin/ruby -rubygems -e "require 'hitimes'" -e "require 'nokogiri'" -e 'puts "libxml version : #{Nokogiri::LIBXML_PARSER_VERSION}"' dyld: loaded: /Users/jeremy/pkgs/ruby-nonflat/./bin/ruby dyld: loaded: /usr/lib/libSystem.B.dylib dyld: loaded: /usr/lib/libobjc.A.dylib dyld: loaded: /usr/lib/libgcc_s.1.dylib dyld: loaded: /usr/lib/system/libmathCommon.A.dylib dyld: loaded: /usr/lib/libauto.dylib dyld: loaded: /usr/lib/libstdc++.6.dylib dyld: loaded: /Users/jeremy/pkgs/ruby-nonflat/lib/ruby/1.8/i686-darwin9.8.0/thread.bundle dyld: loaded: /Users/jeremy/pkgs/ruby-nonflat/lib/ruby/1.8/i686-darwin9.8.0/etc.bundle dyld: loaded: /Users/jeremy/pkgs/ruby-nonflat/lib/ruby/1.8/i686-darwin9.8.0/stringio.bundle dyld: loaded: /Users/jeremy/pkgs/ruby-nonflat/lib/ruby/1.8/i686-darwin9.8.0/syck.bundle dyld: loaded: /Users/jeremy/pkgs/ruby-nonflat/lib/ruby/gems/1.8/gems/hitimes-1.0.4/lib/hitimes/1.8/hitimes_ext.bundle dyld: loaded: /System/Library/Frameworks/CoreServices.framework/Versions/A/CoreServices dyld: loaded: /System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/CarbonCore.framework/Versions/A/CarbonCore dyld: loaded: /System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/CFNetwork.framework/Versions/A/CFNetwork dyld: loaded: /System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/Metadata.framework/Versions/A/Metadata dyld: loaded: /System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/OSServices.framework/Versions/A/OSServices dyld: loaded: /System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/SearchKit.framework/Versions/A/SearchKit dyld: loaded: /System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation dyld: loaded: /System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/AE.framework/Versions/A/AE dyld: loaded: /System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/LaunchServices.framework/Versions/A/LaunchServices dyld: loaded: /System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/DictionaryServices.framework/Versions/A/DictionaryServices dyld: loaded: /System/Library/Frameworks/IOKit.framework/Versions/A/IOKit dyld: loaded: /usr/lib/libicucore.A.dylib dyld: loaded: /System/Library/Frameworks/DiskArbitration.framework/Versions/A/DiskArbitration dyld: loaded: /usr/lib/libbsm.dylib dyld: loaded: /usr/lib/libz.1.dylib dyld: loaded: /System/Library/Frameworks/Security.framework/Versions/A/Security dyld: loaded: /System/Library/Frameworks/SystemConfiguration.framework/Versions/A/SystemConfiguration --> dyld: loaded: /usr/lib/libsqlite3.0.dylib dyld: loaded: /usr/lib/libresolv.9.dylib --> dyld: loaded: /usr/lib/libxml2.2.dylib dyld: loaded: /usr/lib/libxslt.1.dylib dyld: loaded: /Users/jeremy/pkgs/ruby-nonflat/lib/ruby/gems/1.8/gems/nokogiri-1.3.3/lib/nokogiri/nokogiri.bundle dyld: loaded: /opt/local/lib/libexslt.0.dylib dyld: loaded: /opt/local/lib/libxslt.1.dylib --> dyld: loaded: /opt/local/lib/libxml2.2.dylib dyld: loaded: /opt/local/lib/libz.1.dylib dyld: loaded: /opt/local/lib/libiconv.2.dylib dyld: loaded: /Users/jeremy/pkgs/ruby-nonflat/lib/ruby/1.8/i686-darwin9.8.0/racc/cparse.bundle dyld: loaded: /Users/jeremy/pkgs/ruby-nonflat/lib/ruby/1.8/i686-darwin9.8.0/strscan.bundle libxml version : 20702 This time around even though the 20616 version is loaded first, and the correct one, 20702, is what is used by the extension. I guess my question is, what is the proper way to resolve this: 1) should the ./configure script that is distributed with ruby be altered to use the two level namespaces? 2) or should nokogiri and amalgalite and possibly others, warn the user when the library they are uses is not the one the extension expects? Thoughts? enjoy, -jeremy -- ======================================================================== Jeremy Hinegardner jeremy@hinegardner.org