Using Amazon S3, EC2, SQS, Lucene, and Ruby for Web Spidering
http://developer.amazonwebservices.com/connect/entry.jspa?externalID=1182
Each Amazon Web Services (AWS) product is, in and of itself, very powerful. By combining them, you can produce true enterprise-class web applications. This article guides you through building one of these applications. We will build a web-spidering solution using Ruby, Ruby on Rails, and Lucene. The application will run on Amazon Elastic Compute Cloud Beta (Amazon EC2™), will use Amazon Simple Queue Service (Amazon SQS) for job management, and will store any persistent data (in addition to backups) on Amazon Simple Storage Service (Amazon S3).
