... more stuff
at
php-app-engine.com

Archive for the ‘EBS’ Category

Setting up EBS as a RAID

Sunday, January 24th, 2010

http://groups.drupal.org/node/36750

Very similar to setting up an instance with a single EBS drive, you can also fairly easily do a RAID configuration. This leads to better performance usually and higher reliability.

Why did we take reddit down for 71 minutes?

Thursday, January 7th, 2010

http://blog.reddit.com/2010/01/why-did-we-take-reddit-down-for-71.html

Part of our setup uses what we call a “permacache”, which uses Memcachedb. Memcachedb is Memcached with a built-in permanent storage system using BDB. One of the “features” of this system is that it saves up its disk writes and then bursts them to the disk. Unfortunately, the single EBS volumes they were on could not handle these bursting writes. Memcachedb also has another feature that blocks all reads while it writes to the disk. These two things together would cause the site to go down for about 30 seconds every hour or so lately.

http://news.ycombinator.com/item?id=1037993

How to create an ebs image from an existing ec2 instance

Tuesday, December 29th, 2009

http://aws-musings.com/how-to-create-an-ebs-image-from-an-existing-ec2-instance/

Amazon recently announced a new feature which allows you to boot from an ebs volume. But it doesn’t provide any tools to convert your existing AMIs to this new type of image. There is no easy way to create an ebs image from scratch. There are some posts that explain how to convert your existing AMI into this new type of image using ec2-unbundle and dd (a linux utility). I am going to take a little different route and explain how we can create an ebs image from an existing instance. It’s fairly simple to create a new image using dd from an existing instance.

Disk IO and throughput benchmarks on Amazon’s EC2

Wednesday, December 9th, 2009

http://stu.mp/2009/12/disk-io-and-throughput-benchmarks-on-amazons-ec2.html

We’ve been ironing out kinks in our production environment for the last few weeks and one of the things that worried me was if these assertions were true. So, I set out to run a fairly comprehensive test of Disk IO and throughput. I ran hdparm -t, bonnie++, and iozone against ephemeral drives in various configurations along with EBS volumes in various configurations.

For all of my tests I tested the regular ephemeral drives as they were installed (”Normal”), LVM in a JBOD setup (”LVM”), the two ephemeral drives in a software RAID0 setup (”RAID0″), a single 100GB EBS volume (”EBS”), and two 100GB EBS volumes in a software RAID0 setup (”EBS RAID0″). All of the tests were ran on large instances (m1.large). All tests were ran using the XFS file system.

http://news.ycombinator.com/item?id=986445 – YC thread

Running LiveCycle on Your Own Amazon EC2 Cloud Machine Instance

Wednesday, December 9th, 2009

http://blogs.adobe.com/livecycle/2009/12/running_livecycle_on_amazon_ec.html

One of the major annoyances with Amazon’s Elastic Compute Cloud (EC2) has been its behavior when you shut down one of the instances. All changes you made were immediately and irretrievably lost. You had to be a programmer to backup your changes to Amazon’s Simple Storage Service (S3). If you decided to keep your instance running overnight so that you could continue working the next day, you incurred hefty charges for the 8-12 hours of night time during which you do not even use it.

Amazon then introduced the Elastic Block Store (EBS) service which essentially is your own hard disk where you could save things that would survive a machine shutdown. However, up until a week ago, you could not boot from your EBS volume. Now you can.

Amazon EC2 – Fedora/Linux Startup Guide

Wednesday, December 9th, 2009

http://blog.shutupandcode.net/?p=374

Solr, Zabbix, Selenium

RAID and LVM on Amazon EC2 (part I)

Wednesday, December 9th, 2009

http://debianzone.org/raid-and-lvm-on-amazon-ec2-part-i/

This is the first part of three articles I’m posting for a great storage solution using RAID, LVM and Amazon Elastic Block Store (EBS).

First, you need to choose your RAID implementation. Personally, I prefer to use RAID 5 on Amazon EC2, combined with LVM2.

Preparing For EC2 Persistent Storage

Saturday, November 28th, 2009

http://docs.google.com/View?docID=dhh4z6n4_96w387mqhn&revision=_latest

Using LVM + DRBD + NFS + Heartbeat + VTun To Gain Data Persistence, Redundancy, Automatic Fail-Over, and Read/Write Disk Access Across Multiple EC2 Nodes

Difference between Amazon S3 and Amazon EBS on the Elastic Cloud

Tuesday, November 17th, 2009

http://vehera.jsn-server7.com/LiddleBlog/?p=518

Amazon Elastic Block Storage (Amazon EBS) is a new type of storage designed specifically for Amazon EC2 instances. Amazon EBS allows you to create volumes that can be mounted as devices by EC2 instances. Amazon EBS volumes behave as if they were raw unformatted external hard drives and can be formatted using a file system such as ext3 (Linux) or NTFS (Windows) and mounted on an EC2 instance; files are accessed through the file system . They have user supplied device names and provide a block device interface.

Amazon EC2, EBS RAID-0 & PostGIS build script

Monday, November 16th, 2009

http://biodivertido.blogspot.com/2009/11/amazon-ebs-raid-0-postgis-build.html

EC2’s dirty secret

Javier’s post was a great tutorial on building out a PostGIS database on Amazon EC2. We all know EC2, but it does have it’s drawbacks and they are mainly related to disk IO. When using EC2 & EBS with large datasets you can easily run into IO bottlenecks. Individually these are not such a big deal, but when you are conducting global analyses poor disk IO on EC2 & EBS can quickly become a problem.

Anyone used Amazon EC2 with a database or other high IO operation?

Sunday, November 15th, 2009

http://news.ycombinator.com/item?id=943046

We run all of reddit on ec2, which includes a bunch of postgres servers. Each one is running with a single EBS. However, I’ve heard horror stories of people trying to run much less busy databases and having lots of problems, usually with MySQL.

Those databases are all on XLarge instances, so there is minimal sharing, and we’ve also gone to great lengths to make sure all of our normal queries are in indexes, so the disk gets hit less.

We also have a read slave for every database to alleviate read loads.

One thing you might want to do is run ‘iostat -xtc’ on your current box and put that in a log file. Then go back and analyze it and see what your average and peak reads and writes are. Amazon’s max for a single EBS appears to be about 1000 ops / second (at least, that is what we were doing when they told us we maxed out the performance of the disk).

Good luck!

Edit: I forgot to mention that on all the database disks, we use ext2 and noatime. Both decrease the total number of writes necessary, and have very little downside (the biggest being that you have to fsck on a crash).

Increasing IO Performance with RAID on EC2

Monday, November 9th, 2009

http://tech.backtype.com/increasing-io-performance-on-ec2ebs

When you start working with large datasets that don’t fit in memory, improving IO performance on EC2 becomes a priority. To do that at BackType, we use software RAID with EBS volumes; with EBS you don’t need added redundancy so you can settle with RAID0: