PHP App Engine

http://loadstorm.com/

Type Monthly Max Concurrent Users
Breeze FREE (forever) 25

http://winnersdontlose.com/?tag=ec2-filesystem-performance

I’ve been posting lately about a tool named bonnie++, which will run a suite of tests against your linux filesystem to determine metrics in 3 important areas: data read/write speed, max random seeks, and max metadata operations. Last time I posted about profiling one of Linode.com’s “Linode 360″ instances. In this article I will profile a m1.small instance on Amazon Web Services’ (AWS) Elastic Compute Cloud (EC2) service.

http://www.mnxsolutions.com/blog/linux/amazon-ec2-benchmark-pystone.html

The pricing model for Amazon EC2 looks attractive from the surface, but when you get down to it — monthly pricing for similar performing hardware can be much cheaper at a dedicated server provider like Softlayer.com. The use of EC2 is climbing, but I am concerned that many of the current uses of EC2 are better served using leased hardware.

http://code.google.com/p/redis/wiki/Benchmarks

Redis includes the redis-benchmark utility that simulates SETs/GETs done by N clients at the same time sending M total queries (it is similar to the Apache’s ab utility). Below you’ll find the full output of the benchmark executed against a Linux box.

http://www.ruturaj.net/redis-memcached-tokyo-tyrant-mysql-comparison

I wanted to compare the following DBs, NoSQLs and caching solutions for speed and connections. Tested the following

* Redis
* Memcached
* Tokyo Tyrant / Tokyo Cabinet
* MySQL 5.1.40 (MyISAM)
* MySQL 5.1.40 (with Innodb Plugin 1.0.4), compiled into source of MySQL

http://bjclark.me/2009/08/04/nosql-if-only-it-was-that-easy/

So, does RDBMS scale? I would say the answer is: not any worse than lots of other things. Most of what doesn’t scale in a RDBMS is stuff people don’t use that often anyway. And does NoSQL scale: a couple solutions do, most don’t. You might even argue that it’s just as easy to scale mysql (with sharding via mysql proxy) as it is to shard some of these NoSQL dbs. And I think it’s a pretty far leap to declare the RDBMS dead.

http://nosql.mypopescu.com/post/335520054/memcachedb-bursting-blocking-writes

I’d really love to hear how other NoSQL solutions are behaving in this particular scenario. Based on the characteristics of Memcachedb, I’m looking for an answer at least from the key-value stores: Project Voldemort, Redis, SimpleDB, Tokyo Cabinet, M/DB, etc. But if others want to jump in and present their solution that would be great. You can either post your reply as a comment or send it over an emai and I’ll make sure it will get included here.

http://anyall.org/blog/2009/04/performance-comparison-keyvalue-stores-for-language-model-counts/

Here are timings for a single counting process: iterate over 45,000 short text messages, tokenize them, then increment counters for their unigrams and bigrams. (The speed of the data store is only one component of performance.) There are about 17 increments per tweet: 400k unique terms and 750k total count. This is substantially smaller than what I need, but it’s small enough to easily test. I used several very different architectures and packages, explained below.

http://www.mysqlperformanceblog.com/2009/10/15/mysql-memcached-or-nosql-tokyo-tyrant-part-1/

A classic example is a simple online game. An online game may only require that an application retrieve a single record from the database. The record may contain all the vital stats for the game, be updated and stuffed back into the database. You would be surprised how many people use this type of system as I run into this type of application frequently. Keeping it simple, ensures that application is generally mean and lean and performs well. The issue is even this simple design can start to have issues as the data size increases and you blow through your available memory. Is there a better architecture? Is there a way to get more scalability out of your database? Is the database even the best place for this data?

percona guys (one of the highest mysql authorities) put some nails in mysql coffin (innocently?)

http://alan.blog-city.com/has_amazon_ec2_become_over_subscribed.htm

One of our aw2.0 portfolio companies, has been a long term user of Amazon EC2 running a sizable 24×7 of core instances with a number of instances going up and down as scale demands it. Our monthly bill gets us the dubious honor of a first point of contact with an Amazon Account Manager (not that that has been much use). We’ve pushed the limits of many of their services and continue to do so.

After 3 years of production usage what we can tell you is this .. Amazon do have a breaking point.

Visual evidence of Amazon EC2 network issues

https://www.cloudkick.com/blog/2010/jan/12/visual-ec2-latency/

http://news.ycombinator.com/item?id=1048873

http://news.ycombinator.com/item?id=1048694

As my mother would say; “misery enjoys company”

http://alan.blog-city.com/amazon_ec2_latency_the_pretty_graphs.htm

Amazon: We Don’t Have Cloud Capacity Issues

http://www.datacenterknowledge.com/archives/2010/01/14/amazon-we-dont-have-cloud-capacity-issues/?utm-source=feedburner&utm-medium=feed&utm-campaign=Feed:%20DataCenterKnowledge%20%28Data%20Center%20Knowledge%29&utm-content=Google%20Reader

PLEASE SAY IT AIN’T SO, CLOUD OVER-SUBSCRIPTION?

http://www.examiner.com/x-33449-Chicago-Cloud-Computing-Examiner~y2010m1d16-PLEASE-SAY-IT-AINT-SO-CLOUD-OVERSUBSCRIPTION

http://vuksan.com/blog/2009/12/04/cloud-cartography-load-co-residence-detection/

I was actually quite surprised at the magnitude of degradation. I’d say this may be even a more successful co-residence detection attack than network probing since you could generate legitimate HTTP traffic to a site of interest (or a node of interest), throw tons of load at it and see if you notice response degradation.

What Clouds Can Learn From Airlines

http://consultingblogs.emc.com/simonmunro/archive/2010/01/18/what-clouds-can-learn-from-airlines.aspx

http://www.infoq.com/news/2010/01/ec2-oversubscribed

There have been various reports from the community of Amazon EC2 users, that their instances are suffering poor performance, as the result of high internal network latency. This has led to speculations that Amazon’s Cloud might be getting oversubscribed.

http://tech.slashdot.org/story/10/01/15/1350213/Amazon-EC2-May-Be-Experiencing-Growing-Pains

“Some developers using Amazon EC2 are wondering aloud whether the popularity of the cloud computing service is beginning to affect its performance. Amazon this week denied speculation that it was experiencing capacity problems after a veteran developer reported performance issues and suggested that EC2 might be oversubscribed. Meanwhile, a cloud monitoring service published charts showing increased latency on EC2 in recent weeks. The reports follow an incident over the holidays in which a DDoS on a DNS provider slowed Amazon’s retail and cloud operations.”

I have observed “noisy neighbor” effects.

Our scripts use c1.medium instances for the separate Zookeeper quorum ensemble as they need fewer resources in terms of RAM but are still sensitive to io and cpu scheduling latencies.

http://www.jonathanboutelle.com/mt/archives/2010/01/hadoophackday_w.html

-Hadoop is very resource-intensive! We started out using 1-node clusters to run our jobs against small subsets of data. Very quickly teams started upgrading to 5-node clusters due to the amount of time they were having to wait for results. Final runs against full data sets were powered by 10-node clusters of “medium” ec2 servers. You have no choice but to use cloud computing for these kinds of jobs, because it seems to me that production use could easily require 100s of nodes, and no one would want to buy that many servers for machines that they only use one hour a day.

http://blog.reddit.com/2010/01/why-did-we-take-reddit-down-for-71.html

Part of our setup uses what we call a “permacache”, which uses Memcachedb. Memcachedb is Memcached with a built-in permanent storage system using BDB. One of the “features” of this system is that it saves up its disk writes and then bursts them to the disk. Unfortunately, the single EBS volumes they were on could not handle these bursting writes. Memcachedb also has another feature that blocks all reads while it writes to the disk. These two things together would cause the site to go down for about 30 seconds every hour or so lately.

http://news.ycombinator.com/item?id=1037993

http://support.citrix.com/article/CTX120300

by citrix

http://victortrac.com/EC2_Ephemeral_Disks_vs_EBS_Volumes

Amazon’s EC2 service is really neat, but its disk subsystem has some peculiarities that are not initially obvious. Up until very recently, root directories (’/') at EC2 were limited to 10Gb, a limit defined by the maximum size of an Amazon Machine Image (AMI), essentially a template of an EC2 instance. In order to use more disk space, Amazon provides ephemeral disks that one can format and mount anywhere on the file system. However, in order to get persistent storage, one has to use network-attached EBS volumes, a sort of limitless in capacity but bound in I/O wonder of Amazon architecture. There are clear performance implications in choosing how to configure an EC2 instance’s disk subsystem, so I recently benchmarked some various ephemeral and EBS RAID configurations.

http://www.slideshare.net/gear6memcached/cloud-scaling-with-memcached-2678002

# Memcached and the “Cloud”
Presented by: Bill Takacs Director, Product Management December 8th 2009

# Agenda • Dynamic Web • Memached • Why leverage the cloud? • Where to start? • Use cases • Gear6 and the Cloud 2 : Copyright 2009 Gear6 Inc.

http://developer.amazonwebservices.com/connect/thread.jspa?threadID=34938

I would like to know two things below, because I am observing an elb which has got only one IP address (namely one virtual load balancer) is giving us lots of timeouts under a certain load which a single c1.medium Apache server can process without any timeouts.

(1) basic performance info of a typical virtual load balancer under an elb, including rate per second and concurrent connections. In our case, http.
(2) what triggers “ramping up” elb

http://issues.apache.org/jira/secure/attachment/12413737/hive_benchmark_2009-07-12.pdf

A Benchmark for Hive, PIG and Hadoop

http://www.initsix.co.uk/content/how-make-xfs-4-disk-raid-0-partition-ec2-mysql-maximum-disk-io-performance

Use this technique at your own risk. You need to be hot on snapshots, backups and restores to ensure you dont lose everything.

* Raid 0 offers no fault tolerance
* Disk reads and writes can be potentially 400% faster
* Loss of one disk will destroy the entire dataset

http://zuzara.com/blog/2009/11/01/tpcc-mysql-rough-benchmark-for-amazon-rds/

I tried to do tpcc-mysql benchmark for Amazon RDS. Before do that, did the same test for EC2 small instance.

This is a pretty rough benchmark, but I can say EC2 small instance and RDS small instance have the same performances as CPU and memory are the same spec. RDS is about 30% expensive. (EC2=$0.085, RDS=$0.11/h)

http://img.skitch.com/20091212-8mfp3jihiuts3diifk5e62py56.jpg

Totally. Need to know what u’re doing, man. And how.

http://stu.mp/2009/12/disk-io-and-throughput-benchmarks-on-amazons-ec2.html

We’ve been ironing out kinks in our production environment for the last few weeks and one of the things that worried me was if these assertions were true. So, I set out to run a fairly comprehensive test of Disk IO and throughput. I ran hdparm -t, bonnie++, and iozone against ephemeral drives in various configurations along with EBS volumes in various configurations.

For all of my tests I tested the regular ephemeral drives as they were installed (”Normal”), LVM in a JBOD setup (”LVM”), the two ephemeral drives in a software RAID0 setup (”RAID0″), a single 100GB EBS volume (”EBS”), and two 100GB EBS volumes in a software RAID0 setup (”EBS RAID0″). All of the tests were ran on large instances (m1.large). All tests were ran using the XFS file system.

http://news.ycombinator.com/item?id=986445 – YC thread

http://news.ycombinator.com/item?id=984691

Using EC2 for anything but overflow is silly. The costs are extraordinarily high if you are doing decent volume ( read- if you have more than 3-5 servers )

The only real advantage to using ec2 is the hourly billing. Make its perfect for overflow, but thats about it. I read about alot of startups that use ec2 for things like webservers or other servers that have 100% reliance.

http://www.gear6.com/memcached-product/gear6-webcache-aws-ec2-getting-started

Gear6. The mission critical Memcached distribution

https://issues.apache.org/jira/browse/PIG-200

To benchmark Pig performance, we need to have a TPC-H like Large Data Set plus Script Collection. This is used in comparison of different Pig releases, Pig vs. other systems (e.g. Pig + Hadoop vs. Hadoop Only).

I am currently running long-running Pig scripts over data-sets in the order of tens of TBs. Next step is hundreds of TBs.

http://wi.nr/4G

Launching a new webapp is never easy – even one as simple as a URL shortener. Will it catch on? If it does, what does that mean in terms of traffic? 10, 100, 1000 requests per second?

A few weeks ago we did some back of the envelope calculations for how big wi.nr could get in the best case. The calculation went something like: 10,000 active users x 10 shortened URLs per day x 100 people clicking on each of those URLs = 10M shortened URLs clicked a day or ~120 per sec.

http://www.vincestross.com/2009/04/upgrade-an-ec2-instance/

I’ve been using the Amazon Elastic Computing Cloud (EC2) for running our web servers at NetCrafters for almost a year now. It’s been an amazing experience and I’ve kept a detailed account of many of the lessons learned.

Amazon Web ServicesThe most recent challenge came when our servers just started locking-up for no reason. The sites would still be responding so we knew the LAMP stack was still limping along, but they were completely unresponsive via SSH. The only way to regain control was to issue a reboot through the Amazon API command line tools. Even this would sometimes take two or three times before the server would cycle.

kernel BUG at arch/i386/mm/pgtable-xen.c:306!

http://serverfault.com/questions/87504/is-cpu-actually-occupied-during-iowait-wa-in-top-on-linux-ec2

On an 8-way Amazon EC2 instance (running Linux 2.6.21) with 8 EBS volumes and a lot of disk traffic, we see high %wa in top (30-40%), and high load average (8-9). My understanding is that processes waiting on I/O from the EBS volumes are counted in the load average (a ps shows several processes in the D state, about as many as the load average).

http://www.ruby-forum.com/topic/150120

When running nginx on Amazon’s EC2 and using this same test file and
nginx conf, I’m only getting 4K requests/sec and am trying to
understand why.

Any thoughts as to what is the bottleneck? Hardware? OS? Network?
nginx misconfiguration?

http://vehera.jsn-server7.com/LiddleBlog/?p=518

Amazon Elastic Block Storage (Amazon EBS) is a new type of storage designed specifically for Amazon EC2 instances. Amazon EBS allows you to create volumes that can be mounted as devices by EC2 instances. Amazon EBS volumes behave as if they were raw unformatted external hard drives and can be formatted using a file system such as ext3 (Linux) or NTFS (Windows) and mounted on an EC2 instance; files are accessed through the file system . They have user supplied device names and provide a block device interface.

http://blog.readytocloud.com/2009/03/configuring-squid-in-amazon-ec2-cloud.html

If you have a web server and you are suddenly faced with an overload of traffic to your site and you want to buy some time before you re-engineer your site with or without the cloud computing options in mind, you can build a quick reverse proxy squid server to help you out. It will take one hour of your time (assuming you already have an account with Amazon, if not then about 2 hours) and you can have a full reverse proxy server running. This will permit your site more breathing room by caching all requests and serving them to subsequent users. It is a powerful mechanism but needs to be thought out completely if used as a long term solution.