From: Wilson Bilkovich Date: 2005-12-01T02:08:39+09:00 Subject: Re: Ruby Enterprise App Design Advice I'll insert my comments inline. Pardon any mangling I manage to do to your message. On 11/30/05, TeslaOMD wrote: > In a way I'm building > a framework, but a project specific one for an enterprise app, not for > the average personal dynamic website. I may release it after the > project is complete. This is a good idea in general, because it forces you to see the problem in an abstract way, and to make sure you have full test coverage. Rails is much better as an open project than as an in-house framework, from a code quality perspective. > I know that many do not consider Ruby to be an enterprise level > language, but I am a believer in rapid development and scaling through > good design/programming. I've been using Ruby in production in an enterprise environment for some time. Most enterprise (read, Java) experts like to ignore the amount of code tied up in "non-enterprise" languages like shell script. If Ruby isn't an enterprise language, is Perl? It sounds like you're not too worried about the viewpoint of CIO magazine, so I think you can go ahead and use Ruby if you want. Heh. > Users will have lots of preferences/settings to > manage/configure, most of which will be stored to disk as xml > (serialized objects perhaps?) rather than the database. Do you have a scalability plan for this? Databases are mostly a 'solved problem', but persisting things to disk in a cluster is much harder. Are you worried about the overhead of the database for this task? > We also want to try to give > back some of our end results to the Ruby community in terms of code. Very cool. > > 3. Database -- MySQL or Postgres - We need transaction support and data > integrity -- doubts about MySQL in these areas. We also need good join > performance - database is heavily normalized, may denormalize if > needed. Separate DBs for things like Logging/Auditing vs. Content. Want > to cluster DBs. We want connection pooling. Considering making some > modifications to DBI. Use stored procedures in the database, no sql in > the code/dynamic sql. I loathe O/R mappers for complex databases and > they would likely bring our system to a screeching halt. Unfortunately > we cannot afford to use Oracle right now. If your database is heavily normalized, isn't that precisely where ORM systems shine? It's usually with complex 'legacy' schema that you run into trouble. Object mappers save so much developer time, I'd rather spend the extra time writing better business logic, and optimizing the database design. Slogging through endless "stuff these fields into these properties" modules is a recipe for difficult refactoring. Personally, I've really only used Oracle with Ruby. Between MySQL and Postgres, my choice would certainly be the latter. I much prefer sequences to autoincrements, and it's nice to have subselects not be a recently-added feature. That being said, many huge systems are running on MySQL. You just need to be familiar with its limitations while designing your app. > > 4. Caching -- We'll cache user settings, DB data, etc. Need a good ruby > caching solution. Considering using memcached or something else > existing. Rails comes with pretty slick caching. Even if you don't want to use Rails itself, you could at least borrow their caching code. Remember, also, that you can use Rails without ActiveRecord, if you don't like ORM libraries. > > 5. Templating -- Not satisfied with anything. Considered Clearsilver > but now developing our own system in mainly C++ that is more > accomodating to the way our site works/AJAX. What exactly is ERb not doing for you? I've never been a big template fan, myself, except maybe for what Wicket uses: (Warning: Java!) http://wicket.sourceforge.net/ > 6. Unit Testing and Performance numbers for every procedure. Is there a > good code profiling tool for Ruby? Check out Ryan Davis's awesome work, in the form of ZenProfiler, RubyInline, Ruby2c, etc. http://rubyforge.org/projects/zenhacks/ > 7. Sessions - Separate server for managing Sessions if possible. I > would like to persist things in memory and share if possible. If not > possible, we'll settle for persisting info to the database. Been > looking at Session affinity for Ruby some. You can do this with DRb, which is extremely fast and handy. Alternately, you can just store sessions in the DB, which is convenient because it yields fewer 'things' to optimize and profile. > 8. AJAX - Polling and Queuing system for AJAX interactions in > Javascript. Queing server side. Likely we will use and possibly extend > an existing AJAX library such as Mochikit or base it off another > existing one. Have you taken a look at Zimbra? They have a very cool implementation of this. http://www.zimbra.com/ > 10. External Services - We may expose some web services in the future > or have internal web services. We may also be providing RSS feeds from > things like blogs or lists. We'll use ReXML for our xml needs, but we > may switch to c++. I've heard performance concerns about REXML but have > yet to test for myself. Even if ReXML didn't serve your needs, you could use RubyInline to re-implement slow parts of it in C, without tossing out all your Ruby-ness. > > 11. Events/Threading - I am worried about Ruby here, especially after > reading about Myriad's problems with libevent. We will definitely need > something similar to delegates and events and some good queueing and > threading functionality. In my opinion, you should stay away from this kind of code in Ruby until 2.0 is out. Luckily, it is easy to hook Ruby up to C code, and you can write your performance-critical code in that language instead. I'm glad that Myriad thread made it to the list. Very illuminating. > We looked at Rails like everyone else but after using it a bit, reading > the author's blog, and from previous experience in other languages it > is clearly not suitable for the size and complexity of our application. > I also will reiterate that I hate O/R mapping unless it's for a quick > personal app. Well, the authors of Hibernate don't agree with you. Heh. Are you sure this personal preference is worth the development time cost? > Concerns: Our main concern is of course scalability. Our AJAX controls > (many already finished) will need to poll the server in some cases over > a specified interval. This is pretty much a non-issue in the CGI model. Just add more webservers as needed. There are Rails sites doing 1M+ page requests a day on only three cheap rackmount boxes. You should focus your concern on the database. If your database can handle the load, scaling Ruby / CGI is super easy. > For instance, a purely hypothetical example might be we have a control > that lists online users along with status information.The controls > would need to requery information every 20 seconds to obtain fresh > information about the users (are they online? what is their current > mood? what did they last do?). This means that our server is going to > get hit a lot harder than your typical web application that serves up a > dynamic page, then sits idle until the user moves on to a new page. Be very careful with this. There was a good article in ACM Queue about this recently: http://acmqueue.org/modules.php?name=Content&pa=showpage&pid=337 Google's advice is to add as much delay between your client requests as possible, and they use the auto-refresh concept as an example. > We need to support a lot of simultaneous users > and I'm worried the process model needs to be perhaps replaced with a > threadpool/queing model (one that responds quick though). Remember, Apache HTTPd uses the process model as well, and scales pretty much as far as you want. In fact, I'd say that the process model has in general been shown to be an easier scaling problem than threading. It's threads that I worry about. Processes are easy, and can even be migrated to other systems at runtime, without stopping them. Let me leave you with the old adage: Don't optimize prematurely. Developers are, in general, NOT good at predicting which pieces of code will end up being hit hardest. Don't spend 6 months implementing an event queue only to find it using 1% of the total resources. Write the code in Ruby, profile it, rewrite the parts that are too slow. Best of luck, --Wilson.