Archive for the ‘EBS’ Category
Sunday, January 24th, 2010
http://groups.drupal.org/node/36750
Very similar to setting up an instance with a single EBS drive, you can also fairly easily do a RAID configuration. This leads to better performance usually and higher reliability.
Posted in EBS, HowTos, LoadBalancing | No Comments »
Thursday, January 7th, 2010
http://blog.reddit.com/2010/01/why-did-we-take-reddit-down-for-71.html
Part of our setup uses what we call a “permacache”, which uses Memcachedb. Memcachedb is Memcached with a built-in permanent storage system using BDB. One of the “features” of this system is that it saves up its disk writes and then bursts them to the disk. Unfortunately, the single EBS volumes they were on could not handle these bursting writes. Memcachedb also has another feature that blocks all reads while it writes to the disk. These two things together would cause the site to go down for about 30 seconds every hour or so lately.
http://news.ycombinator.com/item?id=1037993
Posted in Caching, EBS, HowTos, Performance | No Comments »
Tuesday, December 29th, 2009
http://aws-musings.com/how-to-create-an-ebs-image-from-an-existing-ec2-instance/
Amazon recently announced a new feature which allows you to boot from an ebs volume. But it doesn’t provide any tools to convert your existing AMIs to this new type of image. There is no easy way to create an ebs image from scratch. There are some posts that explain how to convert your existing AMI into this new type of image using ec2-unbundle and dd (a linux utility). I am going to take a little different route and explain how we can create an ebs image from an existing instance. It’s fairly simple to create a new image using dd from an existing instance.
Posted in EBS, HowTos, Imaging | No Comments »
Wednesday, December 9th, 2009
http://stu.mp/2009/12/disk-io-and-throughput-benchmarks-on-amazons-ec2.html
We’ve been ironing out kinks in our production environment for the last few weeks and one of the things that worried me was if these assertions were true. So, I set out to run a fairly comprehensive test of Disk IO and throughput. I ran hdparm -t, bonnie++, and iozone against ephemeral drives in various configurations along with EBS volumes in various configurations.
For all of my tests I tested the regular ephemeral drives as they were installed (”Normal”), LVM in a JBOD setup (”LVM”), the two ephemeral drives in a software RAID0 setup (”RAID0″), a single 100GB EBS volume (”EBS”), and two 100GB EBS volumes in a software RAID0 setup (”EBS RAID0″). All of the tests were ran on large instances (m1.large). All tests were ran using the XFS file system.
http://news.ycombinator.com/item?id=986445 – YC thread
Posted in EBS, Performance | No Comments »
Wednesday, December 9th, 2009
http://blogs.adobe.com/livecycle/2009/12/running_livecycle_on_amazon_ec.html
One of the major annoyances with Amazon’s Elastic Compute Cloud (EC2) has been its behavior when you shut down one of the instances. All changes you made were immediately and irretrievably lost. You had to be a programmer to backup your changes to Amazon’s Simple Storage Service (S3). If you decided to keep your instance running overnight so that you could continue working the next day, you incurred hefty charges for the 8-12 hours of night time during which you do not even use it.
Amazon then introduced the Elastic Block Store (EBS) service which essentially is your own hard disk where you could save things that would survive a machine shutdown. However, up until a week ago, you could not boot from your EBS volume. Now you can.
Posted in EBS, HowTos, Imaging | No Comments »
Wednesday, December 9th, 2009
http://debianzone.org/raid-and-lvm-on-amazon-ec2-part-i/
This is the first part of three articles I’m posting for a great storage solution using RAID, LVM and Amazon Elastic Block Store (EBS).
First, you need to choose your RAID implementation. Personally, I prefer to use RAID 5 on Amazon EC2, combined with LVM2.
Posted in Cluster Architecture, EBS, HowTos, LoadBalancing | No Comments »
Saturday, November 28th, 2009
http://docs.google.com/View?docID=dhh4z6n4_96w387mqhn&revision=_latest
Using LVM + DRBD + NFS + Heartbeat + VTun To Gain Data Persistence, Redundancy, Automatic Fail-Over, and Read/Write Disk Access Across Multiple EC2 Nodes
Posted in Cluster Architecture, EBS, HowTos | No Comments »
Tuesday, November 17th, 2009
http://vehera.jsn-server7.com/LiddleBlog/?p=518
Amazon Elastic Block Storage (Amazon EBS) is a new type of storage designed specifically for Amazon EC2 instances. Amazon EBS allows you to create volumes that can be mounted as devices by EC2 instances. Amazon EBS volumes behave as if they were raw unformatted external hard drives and can be formatted using a file system such as ext3 (Linux) or NTFS (Windows) and mounted on an EC2 instance; files are accessed through the file system . They have user supplied device names and provide a block device interface.
Posted in EBS, HowTos, Money, Performance | No Comments »
Monday, November 16th, 2009
http://biodivertido.blogspot.com/2009/11/amazon-ebs-raid-0-postgis-build.html
EC2’s dirty secret
Javier’s post was a great tutorial on building out a PostGIS database on Amazon EC2. We all know EC2, but it does have it’s drawbacks and they are mainly related to disk IO. When using EC2 & EBS with large datasets you can easily run into IO bottlenecks. Individually these are not such a big deal, but when you are conducting global analyses poor disk IO on EC2 & EBS can quickly become a problem.
Posted in EBS, HowTos, Performance | No Comments »
Sunday, November 15th, 2009
http://news.ycombinator.com/item?id=943046
We run all of reddit on ec2, which includes a bunch of postgres servers. Each one is running with a single EBS. However, I’ve heard horror stories of people trying to run much less busy databases and having lots of problems, usually with MySQL.
Those databases are all on XLarge instances, so there is minimal sharing, and we’ve also gone to great lengths to make sure all of our normal queries are in indexes, so the disk gets hit less.
We also have a read slave for every database to alleviate read loads.
One thing you might want to do is run ‘iostat -xtc’ on your current box and put that in a log file. Then go back and analyze it and see what your average and peak reads and writes are. Amazon’s max for a single EBS appears to be about 1000 ops / second (at least, that is what we were doing when they told us we maxed out the performance of the disk).
Good luck!
Edit: I forgot to mention that on all the database disks, we use ext2 and noatime. Both decrease the total number of writes necessary, and have very little downside (the biggest being that you have to fsck on a crash).
Posted in EBS, HowTos, MySQL, Performance | No Comments »
Monday, November 9th, 2009
http://tech.backtype.com/increasing-io-performance-on-ec2ebs
When you start working with large datasets that don’t fit in memory, improving IO performance on EC2 becomes a priority. To do that at BackType, we use software RAID with EBS volumes; with EBS you don’t need added redundancy so you can settle with RAID0:
Posted in EBS, HowTos, Performance | No Comments »
Monday, November 9th, 2009
http://jframeworks.com/blog/2009/11/08/running-mysql-on-amazon-ec2-with-elastic-block-store-another-take/
I have been following ‘Eric Hammod’ article on how to setup MySQL so that, it survives EC2 instance termination or restarts. That article is one of best article but it’s too much for my requirements. So I came up with some more simple steps to host MySQL files on EBS volume. Many parts of Eric’s article still applies like how to create EBS volume and how to create snapshot of EBS.
Posted in EBS, HowTos, MySQL | No Comments »
Saturday, November 7th, 2009
http://craiccomputing.blogspot.com/2009/08/creating-and-attaching-aws-ebs-volumes.html
AWS Elastic Block Store (EBS) Volumes are arbitrary blocks of storage that can be mounted on EC2 nodes and used like regular filesystems. Here are the basic steps needed to set them up (and the associated gotchas to avoid).
Posted in EBS, HowTos | No Comments »
Friday, October 30th, 2009
http://www.mooreds.com/wordpress/archives/000549
The biggest attraction of the cloud was the ability to spin up and spin down extra servers as the expected traffic on the site increased or decreased. We choose Amazon’s EC2 for hosting. They seem a bit like the IBM of the cloud–no one ever got fired, etc. They have a rich set of offerings and great documentation.
Exactly.
Fork your AMI
Use Capistrano
Use EBS
EC2 Persistence
Use ElasticFox
Consider CloudFront
Use Internal Network Addressing where possible
Consider reserved instances
Some right, some not.
Posted in Cluster Architecture, EBS, GettingStarted | 3 Comments »
Friday, October 30th, 2009
http://www.ardentsoft.com/blog/tag/lamp-on-ec2
Part 1: Setting up Amazon Web Services
Part 2: Setting up a Persistent Volume
Part 3: Starting and Configuring the Instance
Part 4: Mounting a Persistent Volume
Part 5: Configuring MySQL
Part 6: Configuring Apache
Part 7: Configuring PHP
Part 8: Setting up an Elastic IP Address
Part 9: Setting up Snapshots
Part 10: Generating a Custom AMI
Posted in Cluster Architecture, EBS, HowTos, Imaging, LAMP, MySQL | No Comments »
Friday, October 23rd, 2009
http://serverfault.com/questions/77535/what-is-the-fastest-filesystem-to-use-under-linux-on-ec2-for-mysql
EXT3 will work just fine, but you may want to use EBS volumes instead of the space that comes with your EC2 instance, as that may be faster.
Edit: You can create a software RAID on EBS to boost performance (I’d recommend RAID-10) by using mdadm.
Posted in EBS, HowTos, MySQL, Performance | No Comments »
Wednesday, October 21st, 2009
http://www.floydprice.com/2009/10/automatic-elastic-block-snapshots-with-a-cron-job/
Amazon EC2 really is amazing, and the Elastic Block storage is pretty darn good too, however I wish you could automate the snapshot process form the EC2 console.
It is however pretty easy to do yourself using a simple cron job.
Before you start make sure you have a JRE:
Posted in Backup, EBS, HowTos | No Comments »
Wednesday, October 21st, 2009
http://pigeonflight.blogspot.com/2009/08/growing-xfs-partition-on-amazon-ebs.html
Amazon EBS is Amazon’s Persistent Storage service. I think of it as the ability to have a “thumbdrive in the clouds”. I can then “attach” that drive to an Amazon EC2 instance (a machine in the clouds).
Today I needed to add space to a filesystem. The filesystem was 10G XFS partition and I wanted to make it 30G. (don’t try this if you’re not using XFS, some of these notes are XFS specific).
Posted in EBS, HowTos | No Comments »
Monday, October 19th, 2009
Posted in EBS, HowTos, LAMP, MySQL | No Comments »
Monday, October 19th, 2009
http://vastpark.posterous.com/moving-to-amazon-web-services
I have been administrating LAMP web servers in some form or another for more than a decade either in support of a business or for personal use. I seem to never have so few projects that shared hosting made sense. So instead, solutions like virtual private servers, managed dedicated hosting and, in some cases, my own rack of equipment have served up my projects over the years.
Most recently I had been using dedicated managed servers on Rackspace and GoDaddy to host a few dozen sites. Because I no longer run my own DNS or mail servers, it made my migration focused on making MySQL and Apache work on AWS. (I no longer host DNS or mail because as I have found services like EveryDNS and Gmail to be far superior.)
Posted in EBS, GettingStarted, HowTos, LAMP, MySQL | No Comments »
Sunday, October 18th, 2009
http://blog.labnotes.org/2009/10/13/mysql-backups-with-ebs-snapshots/
Amazon EBS snapshots are awesome. You can take snapshots to backup your database, quickly move data between staging and production, run analytics on the latest data, resize volumes and much more.
For Apartly, I’m using MySQL, storing data and log files on a mounted EBS volume. EBS offers persistent, reliable storage (with replication for fault tolerance). Still, nothing can prevent data from disappearing, hence backups.
Posted in Backup, EBS, HowTos, MySQL | No Comments »
Sunday, October 18th, 2009
http://www.john-nousis.com/development/tutorial-running-a-phpmysql-server-on-ec2-with-ebs/
During the presentation I showed how you can start an AMI of fedora 8 base, install apache, php and mySQL and make them all store their data on an EBS volume. This attracted a lot of attention and participants asked me to post this as an online tutorial. This tutorial requires basic knowledge of AWS like running an EC2 instance, creating an EBS volume and assigning an elastic IP to the instance. During the presentation I used ElasticFox which is a firefox plugin that allows you to perform several operations on Amazon through a graphical user interface.
Posted in EBS, HowTos, LAMP, MySQL | No Comments »
Friday, October 16th, 2009
http://ianso.blogspot.com/2009/10/ec2-now-having-actually-played-with-it.html
Consistent IO on EBS is apparently not an option. This is something I think Amazon should fix toute suite, because things like RackSpace (maybe) and NewServers (h.t. etbe) seem to be to stomping all over the EBS I/O figures. In a different context, James Hamilton says “it makes no sense to allow a lower cost component impose constraints on the optimization of a higher cost component”, and assuming that the servers are the expensive part, this is what (IMHO) may make using RDBMSs on EC2 a bit of a PIA long-term.
Posted in Cluster Architecture, EBS, HowTos, Performance | No Comments »
Thursday, October 15th, 2009
http://innergysoftware.com/?p=17
Recently I’ve been using a lot Amazon’s Cloud Computing services in the form of EC2 and S3. I’ve set up several scripts that make life easier by automating some of the tasks that need to be done regularly when administering EC2 virtual instances. In the hope that they will be useful to other people, I’ll be posting here some of them. Here is the first in the series.
Posted in EBS, HowTos | No Comments »
Wednesday, October 14th, 2009
http://www.freemindsystems.com/amazon-ec2-introduction-useful-info-and-tricks/
I’m sure many people think the cloud is the future, and more or less have migration plannings. Also, some of these people point to Amazon as supplier for their price, stability and capabilities. This first post of EC2 are things you need to know from the standpoint of a sysadmin.
For a System Administrator, Amazon EC2 is a big challenge, mostly because their ephemeral nature:
Posted in DNS, EBS, GettingStarted, HowTos, Misc | No Comments »
Wednesday, October 14th, 2009
http://developer.amazonwebservices.com/connect/thread.jspa?threadID=25071
I’ll assume you’ve already set everything up:
1) created the volume (ec2-create-volume)
2) you’ve attached it to your instance
3) Logged into your instance and ran mkfs and mounted the volume
Now to set it up so that it comes up automatically on boot:
1) Load the API tools on your instance and your pk and cert files
2) Load the script I’ve attached (mountec2vol) into /etc/init.d
3) Modify the script variables at the top of the script (VOL, DEV, MOUNT_POINT, JAVA_HOME, EC2 vars)
4) Tell Linux to add mountec2vol as a service
/sbin/chkconfig –add mountec2vol
Posted in EBS, HowTos | No Comments »