PHP App Engine

http://www.mnxsolutions.com/blog/linux/amazon-ec2-benchmark-pystone.html

The pricing model for Amazon EC2 looks attractive from the surface, but when you get down to it — monthly pricing for similar performing hardware can be much cheaper at a dedicated server provider like Softlayer.com. The use of EC2 is climbing, but I am concerned that many of the current uses of EC2 are better served using leased hardware.

http://alan.blog-city.com/has_amazon_ec2_become_over_subscribed.htm

One of our aw2.0 portfolio companies, has been a long term user of Amazon EC2 running a sizable 24×7 of core instances with a number of instances going up and down as scale demands it. Our monthly bill gets us the dubious honor of a first point of contact with an Amazon Account Manager (not that that has been much use). We’ve pushed the limits of many of their services and continue to do so.

After 3 years of production usage what we can tell you is this .. Amazon do have a breaking point.

Visual evidence of Amazon EC2 network issues

https://www.cloudkick.com/blog/2010/jan/12/visual-ec2-latency/

http://news.ycombinator.com/item?id=1048873

http://news.ycombinator.com/item?id=1048694

As my mother would say; “misery enjoys company”

http://alan.blog-city.com/amazon_ec2_latency_the_pretty_graphs.htm

Amazon: We Don’t Have Cloud Capacity Issues

http://www.datacenterknowledge.com/archives/2010/01/14/amazon-we-dont-have-cloud-capacity-issues/?utm-source=feedburner&utm-medium=feed&utm-campaign=Feed:%20DataCenterKnowledge%20%28Data%20Center%20Knowledge%29&utm-content=Google%20Reader

PLEASE SAY IT AIN’T SO, CLOUD OVER-SUBSCRIPTION?

http://www.examiner.com/x-33449-Chicago-Cloud-Computing-Examiner~y2010m1d16-PLEASE-SAY-IT-AINT-SO-CLOUD-OVERSUBSCRIPTION

http://vuksan.com/blog/2009/12/04/cloud-cartography-load-co-residence-detection/

I was actually quite surprised at the magnitude of degradation. I’d say this may be even a more successful co-residence detection attack than network probing since you could generate legitimate HTTP traffic to a site of interest (or a node of interest), throw tons of load at it and see if you notice response degradation.

What Clouds Can Learn From Airlines

http://consultingblogs.emc.com/simonmunro/archive/2010/01/18/what-clouds-can-learn-from-airlines.aspx

http://www.infoq.com/news/2010/01/ec2-oversubscribed

There have been various reports from the community of Amazon EC2 users, that their instances are suffering poor performance, as the result of high internal network latency. This has led to speculations that Amazon’s Cloud might be getting oversubscribed.

http://tech.slashdot.org/story/10/01/15/1350213/Amazon-EC2-May-Be-Experiencing-Growing-Pains

“Some developers using Amazon EC2 are wondering aloud whether the popularity of the cloud computing service is beginning to affect its performance. Amazon this week denied speculation that it was experiencing capacity problems after a veteran developer reported performance issues and suggested that EC2 might be oversubscribed. Meanwhile, a cloud monitoring service published charts showing increased latency on EC2 in recent weeks. The reports follow an incident over the holidays in which a DDoS on a DNS provider slowed Amazon’s retail and cloud operations.”

I have observed “noisy neighbor” effects.

Our scripts use c1.medium instances for the separate Zookeeper quorum ensemble as they need fewer resources in terms of RAM but are still sensitive to io and cpu scheduling latencies.

http://www.jonathanboutelle.com/mt/archives/2010/01/hadoophackday_w.html

-Hadoop is very resource-intensive! We started out using 1-node clusters to run our jobs against small subsets of data. Very quickly teams started upgrading to 5-node clusters due to the amount of time they were having to wait for results. Final runs against full data sets were powered by 10-node clusters of “medium” ec2 servers. You have no choice but to use cloud computing for these kinds of jobs, because it seems to me that production use could easily require 100s of nodes, and no one would want to buy that many servers for machines that they only use one hour a day.

http://searchcloudcomputing.techtarget.com/news/article/0,289142,sid201_gci1378426,00.html?track=NL-1329&ad=743755&asrc=EM_NLN_10614277&uid=1914599#

Teich also said that all of Heroku’s m2.2xlarge instances were running in a single availability zone, which was a mistake. He stressed that Heroku had failover built in already — if 21 instances had failed instead of 22, or if it had spread instances across several zones, “we wouldn’t be talking [about the outage],” he said.

An event like this was an “unknown unknown” that nobody planned for because nobody imagined it. He chalked it up to the learning process and pointed out that everybody in Amazon Web Services was flying by the seat of the pants at least part of the time.

HN thread:
http://news.ycombinator.com/item?id=1046500

http://moussadao.com/2009/12/30/amazon-ec2-pricings-demystified/

Amazon Elastic Compute Cloud (Amazon EC2) is the leading provider for cloud based computes resources. Amazon EC2 allows users to provision compute resources on-demand. With On-demand Instances, Amazon EC2 does not require any commitment, sign-up fees, and upfront payments. Amazon Web Services (AWS) uses a monthly billing cycle; therefore users get charged at end of each month for their EC2 usages.

http://www.tarsnap.com/

Online backups for the truly paranoid

At the present time, tarsnap does not support Windows (except via Cygwin) and does not have a graphical user interface.

http://aws.typepad.com/aws/2009/12/thirdparty-aws-tracking-sites.html

A couple of really cool third-party AWS tracking sites have sprung up lately. Some of these sites make use of AWS data directly and others measure it using their own proprietary methodologies. I don’t have any special insight in to the design or operation of these sites, but at first glance they appear to be reasonably accurate.

http://zuzara.com/blog/2009/11/01/tpcc-mysql-rough-benchmark-for-amazon-rds/

I tried to do tpcc-mysql benchmark for Amazon RDS. Before do that, did the same test for EC2 small instance.

This is a pretty rough benchmark, but I can say EC2 small instance and RDS small instance have the same performances as CPU and memory are the same spec. RDS is about 30% expensive. (EC2=$0.085, RDS=$0.11/h)

http://img.skitch.com/20091212-8mfp3jihiuts3diifk5e62py56.jpg

Totally. Need to know what u’re doing, man. And how.

http://www.allthingsdistributed.com/2009/12/amazon_ec2_spot_instances.html

Spot instances. Yeah.

# On-Demand Instances – On-Demand Instances let you pay for compute capacity by the hour with no long-term commitments or upfront payments. You can increase or decrease your compute capacity depending on the demands of your application and only pay the specified hourly rate for the instances you use. These instances are used mostly for short term workloads and for workloads with unpredictable resource demand characteristics.
# Reserved Instances – Reserved Instances let you make a low, one-time, upfront payment for an instance, reserve it for a one or three year term, and pay a significantly lower rate for each hour you run that instance. You are assured that your Reserved Instance will always be available in the Availability Zone in which you purchased it. These instances are used for longer running workloads with predictable resource demands.
# Spot Instances – Spot Instances allow you to specify the maximum hourly price that you are willing to pay to run a particular instance type. We set a Spot Price for each instance type in each region, which is the price all customers will pay to run a Spot Instance for that given hour. The Spot Price fluctuates based on supply and demand for instances, but customers will never pay more than the maximum price they have specified. These instances are used for workloads with flexible completion times.

http://news.ycombinator.com/item?id=984691

Using EC2 for anything but overflow is silly. The costs are extraordinarily high if you are doing decent volume ( read- if you have more than 3-5 servers )

The only real advantage to using ec2 is the hourly billing. Make its perfect for overflow, but thats about it. I read about alot of startups that use ec2 for things like webservers or other servers that have 100% reliance.

http://aws.typepad.com/aws/2009/12/the-economics-of-aws.html

For the past several years, many people have claimed that cloud computing can reduce a company’s costs, improve cash flow, reduce risks, and maximize revenue opportunities. Until now, prospective customers have had to do a lot of leg work to compare the costs of a flexible solution based on cloud computing to a more traditional static model. Doing a genuine “apples to apples” comparison turns out to be complex — it is easy to neglect internal costs which are hidden away as “overhead”.

http://developer.amazonwebservices.com/connect/thread.jspa?messageID=155720

Sorry if I am posting in the wrong place, I cannot see anywhere else I can take this issue up. I have emailed webservices@amazon.com a few times, and I have not received any response. Does anyone know how long I shoudl expect to wait for a response?

I created my EC2 instance, and have had no problems accessing it. I spent some time initially setting it up, but did not complete installing my rails app there. So its basically just got apache, mysql, rails, etc there, not much else. I got distracted, and had to put it on hold.

…

sounds like you started a extra large instance and left it running. $0.80/hr * 24hours = $19.20 per day or $576 per month.

Going to the cloud?
Learn to count

http://wi.nr/4G

Launching a new webapp is never easy – even one as simple as a URL shortener. Will it catch on? If it does, what does that mean in terms of traffic? 10, 100, 1000 requests per second?

A few weeks ago we did some back of the envelope calculations for how big wi.nr could get in the best case. The calculation went something like: 10,000 active users x 10 shortened URLs per day x 100 people clicking on each of those URLs = 10M shortened URLs clicked a day or ~120 per sec.

http://i.justrealized.com/2009/12/03/price-evaluation-of-amazon-ec2/

I’ve been looking at the pricing of Amazon EC2 (Amazon Elastic Compute Cloud) running as if it were a VPS. I intend to run it continuously for as long as possible. I likely only need a small instance as described here.

http://kontrollsoft.com/archives/551?utm_source=feedburner&utm_medium=feed&utm_campaign=Feed%3A+Kontrollsoft+%28kontrollsoft%29

This is a follow up on a previous post about Amazon’s EC2 cloud services. You may recall that I had the Kontrollbase demo server hosted there until I was hit with a >$370 bill for less than 2 weeks of service. Now, you may think you want to say “hey you should have known the cost..” or something similar and you would be partially correct. However, now that I have the breakdown in daily usage I see that it is in fact not the CPU/resource usage or having the larger VM instances running like I previously thought but all due to data transfer. EC2 does not include data with their servers – it’s all separate and it’s a complete rip off when compared to any major hosting provider.

http://kontrollsoft.com/archives/554?utm_source=feedburner&utm_medium=feed&utm_campaign=Feed%3A+Kontrollsoft+%28kontrollsoft%29

Updated: Looks like Amazon is listening

http://kovshenin.com/archives/optimizing-your-amazon-web-services-costs/

I’ve been with Amazon for quite a long time now and you must have heard that their web hosting services aren’t very cheap. The average total of one instance per month (including EBS, S3 and all the others) was around $120 at the start. That was back in July 2009 when I had no idea about how all this stuff works. With a lot of experimenting I managed to drop my instance per month costs down by around 40%. Below are a few tips that can help you lower your Amazon Web Services charges:

http://cloudsplit.com/

CloudSplit is the first company to offer a real time view on what is happening on your Amazon grid from a cost perspective. We can all understand that cloud computing can significantly reduce our infrastructure spend, but even cloud costs can mount up if we use our clouds carelessly.

CloudSplit will ensure you don’t accidentally overspend, by tracking your cloud spending in real-time and giving you clear graphical breakdowns of how those costs were accrued.

http://www.elusive-cloud.com/go/module/pages/page/106/

The requirement of the concept, in addition to relocating the web project to the EC2 platform, was to ensure that, with the help of AutoScaling, there was always an
optimal number of servers in use.

http://www.vincestross.com/2009/04/upgrade-an-ec2-instance/

I’ve been using the Amazon Elastic Computing Cloud (EC2) for running our web servers at NetCrafters for almost a year now. It’s been an amazing experience and I’ve kept a detailed account of many of the lessons learned.

Amazon Web ServicesThe most recent challenge came when our servers just started locking-up for no reason. The sites would still be responding so we knew the LAMP stack was still limping along, but they were completely unresponsive via SSH. The only way to regain control was to issue a reboot through the Amazon API command line tools. Even this would sometimes take two or three times before the server would cycle.

kernel BUG at arch/i386/mm/pgtable-xen.c:306!

http://vehera.jsn-server7.com/LiddleBlog/?p=518

Amazon Elastic Block Storage (Amazon EBS) is a new type of storage designed specifically for Amazon EC2 instances. Amazon EBS allows you to create volumes that can be mounted as devices by EC2 instances. Amazon EBS volumes behave as if they were raw unformatted external hard drives and can be formatted using a file system such as ext3 (Linux) or NTFS (Windows) and mounted on an EC2 instance; files are accessed through the file system . They have user supplied device names and provide a block device interface.

http://php-app-engine.com/static/cloudbench.html

Picked ‘wrong’ EC2 image – you will be punished.

http://www.gogrid.com/cloud-hosting/compare-gogrid-to-amazon-ec2.php

Features and Pricing Comparison

Heavily Windows oriented

http://www.structuredthought.org/?tag=reseller

Part 1 – Overview
Part 2 – Initial Server Setup
Part 3 – EBS Volumes and Elastic IPs

http://www.techkismet.com/systems-admin/automating-ec2-ebs-snapshot-cleanup.html

So the problem in a nutshell is I have 10 volumes, each of which is cron’ed to be snapshotted at various times of the day (depends on the specific volume as to how often it is backed up). With 10 volumes, my S3 storage costs can get out of hand quite quickly. So I needed to develop a set of scripts that would scan my snapshots – and remove the oldest ones – so I’m not paying for that storage.

http://blog.getasysadmin.com/2009/02/more-apache-benchmarks-using-ec2.html

So to sum things up:
c1.medium can serve about 32 requests per second for 0.20$ per hour.
m1.large can serve less than c1.medium, around 24 requests per second for 0.40$ per hour … not nice
m1.xlarge can serve 47 requests per second, but it will cost you 0.80$ per hour.
The champion is c1.xlarge can serve 142 requests per second at same price as m1.xlarge, 0.80$ pe hour.

My advice would be to go with c1.medium instances since they offer best price/performance ratio. But do the math for yourself!

His numbers correspond to mine. But it is not so simple.

http://alestic.com/2009/10/ec2-4xlarge

Amazon Web Services just announced the release of two new instance types for EC2. These new types have 34.2 GB and 68.4 GB of RAM with a decent amount of CPU capacity on modern CPUs to go along with it.

sitting on my couch with my laptop watching an episode of “Lie to me” on TiVo I just typed:

http://community.zenoss.org/blogs/mikelunt/2009/10/26/qa-testing–to-cloud-or-not-to-cloud

We are considering supplementing our engineering lab via the use of cloud services such Amazon’s EC2 or Rackspace’s Cloud Servers. Currently, we use a large set of VMWare servers to host our QA testing environment, where we perform installations of Zenoss servers as well as setup target systems (Windows, Linux, etc.) to monitor. In addition, these VMWare servers are connected to the same private network as various routers, storage devices, and other types of OS’s, such as HP-UX, Solaris, etc. All of this is done to simulate what Zenoss users have in their IT environments, and for the most part, the setup works quite well except that there’s never enough. The QA engineers are constantly finding new combinations of OS’s, applications, etc. that require more VM guests to be created. Here is a quick brain dump of the comparisons thus far.

http://www.cloudhosting.co.uk/content/amazon-aws-what-it-good

What exactly can you do with it and how can it help your business? 6 areas come to mind ­
• Testing environments
• Temporary hosting solutions
• Unknown capacity solutions
• Uneven web traffic patterns
• Web Start up businesses
• Large processing requirements
• Software and vendor evaluation

http://www.s3rsync.com/index.php/Rsync_to_Amazon_S3

The problem:
S3 storage protocol is “all or nothing” meaning that you can not modify a file on S3 and even for a minor change you have to upload (S3 PUT) the whole file again. This implies that backup and synchronization to S3 are inefficient. Any file modification forces you to upload the whole new file again, not just what was changed. So you are wasting bandwidth and the backup process is significantly slower!

Our solution:
S3rsync resolves this limitation and allows you to use Rsync bandwidth efficient algorithm that enables to upload only partial files that were changed. This is done by connecting to our Rsync servers located inside Amazon facility (Ec2). Using this functionality enables you to fully benefit from Rsync power.