From: Robert Klemme Date: 2011-05-02T23:07:09+09:00 Subject: Re: Question on the best approach for gathering tweets based on time On Mon, May 2, 2011 at 3:22 PM, Panagiotis Atmatzidis wrote: > Hello, > > I want to know what is the best way to gather tweets from a specific > date till 'time.now'. > > I have a database which I dumped all user tweet history. Tweets are > dumped in a sqlite3 database. My db fields are tweet.created_at, > tweet.text and tweet.id plus  an integer as key. > > I use tweet.id to perform a match test before accepting new tweets on > the database. > > However, now the script tries to dump all possible > tweets from twitter's API every time, do the match and add the ones > that are missing (which are the ones of course). The procedure, as you > imagine, causes big delays. > > The created_at date string is like this: "Tue Jul 06 10:08:23 +0000 2010" > > Time matters, I can't deal only with dates. > > I have a couple of solutions in mind, but I'd like to know from more > experienced users which way to approach this: > > 1) Convert the 'created_at' string to YYYY-MM-DD date? This could be > tricky because there's also the exact time of the tweet to consider. (didn't try it out yet) > > 2) Using sqlite3's "id integer primary key" which uses the biggest > number for the latest entry and extract date from there? > > 3) Any smarter way? It seems this is rather a question for the twitter API. If that API provides some id for every tweed and if it provides a mechanism to query "all tweets after " then the solution is obvious: store the twitter id in your table and fetch do the fetch accordingly. http://apiwiki.twitter.com/w/page/22554679/Twitter-API-Documentation Turns out that id does exist (see XML format): http://dev.twitter.com/doc/get/statuses/public_timeline And there is also the "since" query type: http://dev.twitter.com/doc/get/statuses/user_timeline Happy coding! Kind regards robert -- remember.guy do |as, often| as.you_can - without end http://blog.rubybestpractices.com/