ec2base

http://groups.drupal.org/node/36750

Very similar to setting up an instance with a single EBS drive, you can also fairly easily do a RAID configuration. This leads to better performance usually and higher reliability.

http://blog.reddit.com/2010/01/why-did-we-take-reddit-down-for-71.html

Part of our setup uses what we call a “permacache”, which uses Memcachedb. Memcachedb is Memcached with a built-in permanent storage system using BDB. One of the “features” of this system is that it saves up its disk writes and then bursts them to the disk. Unfortunately, the single EBS volumes they were on could not handle these bursting writes. Memcachedb also has another feature that blocks all reads while it writes to the disk. These two things together would cause the site to go down for about 30 seconds every hour or so lately.

http://news.ycombinator.com/item?id=1037993

http://aws-musings.com/how-to-create-an-ebs-image-from-an-existing-ec2-instance/

Amazon recently announced a new feature which allows you to boot from an ebs volume. But it doesn’t provide any tools to convert your existing AMIs to this new type of image. There is no easy way to create an ebs image from scratch. There are some posts that explain how to convert your existing AMI into this new type of image using ec2-unbundle and dd (a linux utility). I am going to take a little different route and explain how we can create an ebs image from an existing instance. It’s fairly simple to create a new image using dd from an existing instance.

http://stu.mp/2009/12/disk-io-and-throughput-benchmarks-on-amazons-ec2.html

We’ve been ironing out kinks in our production environment for the last few weeks and one of the things that worried me was if these assertions were true. So, I set out to run a fairly comprehensive test of Disk IO and throughput. I ran hdparm -t, bonnie++, and iozone against ephemeral drives in various configurations along with EBS volumes in various configurations.

For all of my tests I tested the regular ephemeral drives as they were installed (”Normal”), LVM in a JBOD setup (”LVM”), the two ephemeral drives in a software RAID0 setup (”RAID0″), a single 100GB EBS volume (”EBS”), and two 100GB EBS volumes in a software RAID0 setup (”EBS RAID0″). All of the tests were ran on large instances (m1.large). All tests were ran using the XFS file system.

http://news.ycombinator.com/item?id=986445 – YC thread

http://blogs.adobe.com/livecycle/2009/12/running_livecycle_on_amazon_ec.html

One of the major annoyances with Amazon’s Elastic Compute Cloud (EC2) has been its behavior when you shut down one of the instances. All changes you made were immediately and irretrievably lost. You had to be a programmer to backup your changes to Amazon’s Simple Storage Service (S3). If you decided to keep your instance running overnight so that you could continue working the next day, you incurred hefty charges for the 8-12 hours of night time during which you do not even use it.

Amazon then introduced the Elastic Block Store (EBS) service which essentially is your own hard disk where you could save things that would survive a machine shutdown. However, up until a week ago, you could not boot from your EBS volume. Now you can.

http://blog.shutupandcode.net/?p=374

Solr, Zabbix, Selenium

http://debianzone.org/raid-and-lvm-on-amazon-ec2-part-i/

This is the first part of three articles I’m posting for a great storage solution using RAID, LVM and Amazon Elastic Block Store (EBS).

First, you need to choose your RAID implementation. Personally, I prefer to use RAID 5 on Amazon EC2, combined with LVM2.

http://docs.google.com/View?docID=dhh4z6n4_96w387mqhn&revision=_latest

Using LVM + DRBD + NFS + Heartbeat + VTun To Gain Data Persistence, Redundancy, Automatic Fail-Over, and Read/Write Disk Access Across Multiple EC2 Nodes

http://vehera.jsn-server7.com/LiddleBlog/?p=518

Amazon Elastic Block Storage (Amazon EBS) is a new type of storage designed specifically for Amazon EC2 instances. Amazon EBS allows you to create volumes that can be mounted as devices by EC2 instances. Amazon EBS volumes behave as if they were raw unformatted external hard drives and can be formatted using a file system such as ext3 (Linux) or NTFS (Windows) and mounted on an EC2 instance; files are accessed through the file system . They have user supplied device names and provide a block device interface.

http://biodivertido.blogspot.com/2009/11/amazon-ebs-raid-0-postgis-build.html

EC2’s dirty secret

Javier’s post was a great tutorial on building out a PostGIS database on Amazon EC2. We all know EC2, but it does have it’s drawbacks and they are mainly related to disk IO. When using EC2 & EBS with large datasets you can easily run into IO bottlenecks. Individually these are not such a big deal, but when you are conducting global analyses poor disk IO on EC2 & EBS can quickly become a problem.

http://news.ycombinator.com/item?id=943046

We run all of reddit on ec2, which includes a bunch of postgres servers. Each one is running with a single EBS. However, I’ve heard horror stories of people trying to run much less busy databases and having lots of problems, usually with MySQL.

Edit: I forgot to mention that on all the database disks, we use ext2 and noatime. Both decrease the total number of writes necessary, and have very little downside (the biggest being that you have to fsck on a crash).

http://tech.backtype.com/increasing-io-performance-on-ec2ebs

When you start working with large datasets that don’t fit in memory, improving IO performance on EC2 becomes a priority. To do that at BackType, we use software RAID with EBS volumes; with EBS you don’t need added redundancy so you can settle with RAID0: