From: flebber Date: 2010-10-12T19:15:11+09:00 Subject: Re: Ruby Sanity Check On Oct 12, 4:41 pm, Josh Cheek wrote: > [Note:  parts of this message were removed to make it a legal post.] > > > > > > On Mon, Oct 11, 2010 at 6:25 PM, flebber wrote: > > HI > > > Just a quick check. I am starting into Ruby with little interest in > > Ruby on Rails (its good to know its there though). I want to use Ruby > > as a more data centric means. So in my first case I want to extract > > data from websites and put it in a suitable format xml, csv etc for > > use in excel or later for use in a db application such as mysql or > > postgre. > > > I am not nuts am I? > > > I was looking at several comparison and I thought that Ruby dealt with > > lists and data in a logical fashion. > > By "extract data from websites" I assume you mean screen scraping. Here are > two Railscasts about ithttp://railscasts.com/episodes/173-screen-scraping-with-scrapihttp://railscasts.com/episodes/190-screen-scraping-with-nokogiri > > Some Nokogiri tutorials about ithttp://nokogiri.org/tutorials > > Some Mechanize tutorials about it (you will only need to use Mechanize if > you need to interact with the site, it uses Nokogiri under the covers. Note > that it can't handle Javascript, and there are some alternatives if you need > that)http://mechanize.rubyforge.org/mechanize/EXAMPLES_rdoc.html > > Depending on what site you're trying to get info from, it might have an API, > and there might even be a gem for interacting with that API, and saving > yourself the headache and brittleness of screen scraping. > > For outputing to XML, Nokogiri can do that. If you have difficulty getting > it installed, I've also enjoyed using Hpricot (aside from api, the biggest > difference is that Nokogiri is built on libxml2, an open source very popular > C library, while Hpricot is built on a Ragel parser), and if you have > difficulty with that as well, the standard library provides one called > REXML. > > Also consider YAML, which is built into the stdlib, (but has difficulty, I > found, dealing with huge data sets). > > There are a couple of gems for JSON, I can't remember which one I've used. > > For CSV, the fastercsv gem. > > I am almost certain there are tools for interacting with Excel, but I'm on a > Mac, so not able to really help there. > > Depending on what you're doing, you may not need the intermediate form to be > human readable (maybe you just need to perpetuate an array of strings > between runnings of your script, or something like that). If that is the > case, you can just marshall the data.http://ruby-doc.org/core/classes/Marshal.htmlProbably the easiest solution, > and really fast, but it means your data is Ruby. > > For dealing with databases, ActiveRecord, DataMapper, and Sequel should be > able to help you out. > > ActiveRecord is extremely mature as it's the de facto Rails M in its MVC, > but it requires a little bit of infrastructure to get going outside of > Rails. If you want to use it,http://guides.rubyonrails.org/is, IMO, the > best resource. There are also lots of Railscasts that deal with it (note > that AR3 just released, so the interface is a little different). > > DataMapper is another nice project, I like it because you can do it all in > one file without migrations (easy to get up and going) you literally define > your schema in your code. It has some other nice features such as > guaranteeing that there will only ever be one instance of your DB rows in > memory at a time (you can find yourself in some wonky situations with AR, > where it has cached results, or you load the same data twice, and the one is > unaware of the other). It also has a cool solution to the n+1 problem, where > it will preload data as soon as it recognizes you're going to query for it > in a loop. Unfortunately, it's nowhere near as mature as ActiveRecord. I > finally ended up switching my last project off of DataMapper and onto > ActiveRecord after too many headaches dealing with polymorphism, immature > libraries for it (I needed tagging), and dissatisfaction with the IRC > channel. If you don't need external libraries like that, you probably won't > experience such frustrations. If you're interested in it, it has some good > tutorials on its sitehttp://datamapper.org/docs/I also really liked the $9 > Peepcode about Sinatra, which uses DataMapper to talk to its database.http://peepcode.com/products/sinatra > > I've not used Sequel, but I've seen its creator present at Ruby Midwest. He > _really_ knows his stuff. I've also only heard good things about the > project, such as actively developed, and easy to get support for. But my > understanding is that it's main strength is in connecting to "non > opinionated" (what AR would call "legacy") databases. If you have the > ability to design yours from the beginning, some of it's strengths might be > necessary.- Hide quoted text - > > - Show quoted text - Wow thank you!!! Totally beyond expectation. Lets me spend more time learning than searching, very much appreciated. I will update when I understand better the tools and what flow I am going to use.