PHP App EngineClouds and NoSQL – by Smart Robot |
|
http://groups.drupal.org/node/36750
Very similar to setting up an instance with a single EBS drive, you can also fairly easily do a RAID configuration. This leads to better performance usually and higher reliability.
http://mutexlock.blogspot.com/2010/01/resetting-overwriting-etcresolvconf-in.html
For our purposes – we had set up an internal dns server running (bind – process ‘named’ on a particular host).
When we were launching our pool of servers – we wanted to make sure that the new instances fall under the same domain that we specify it to be.
We also wanted to set the nameserver of the newly created instances pointing to the internal DNS server we have to resolve the ambiguity we have.
Before making the change – the file /etc/resolv.conf was looking as follows.
http://monitorcloud.com/monitorcloud
The main monitoring features of Monitis Amazon EC2 (Elastic Compute Cloud) and S3 cloud storage on-demand cloud monitoring service are: external monitoring of launched Amazon virtual servers and generate notifications to users upon breaking the threshold level, dynamically and adaptively monitor web servers, mail servers, databases etc without the need of manually adding each monitor and rules per server, automatically install Monitis agent on new servers and monitor performance metrics plus generate notifications when resources are detected to be low, proactive notification to users if a server is lost in the Amazon cloud.
http://www.lindstromconsulting.com/node/8
In the previous article on the subject of cloud computing using AWS, we setup a simple LAMP application that used a single web server to present data that was queried from a single RDS instance. In this guide we will see how to save the changes we made to the EC2 instance, create more EC2 instances, and setup load balancing across our web servers.
http://www.mysqlperformanceblog.com/2009/10/15/mysql-memcached-or-nosql-tokyo-tyrant-part-1/
A classic example is a simple online game. An online game may only require that an application retrieve a single record from the database. The record may contain all the vital stats for the game, be updated and stuffed back into the database. You would be surprised how many people use this type of system as I run into this type of application frequently. Keeping it simple, ensures that application is generally mean and lean and performs well. The issue is even this simple design can start to have issues as the data size increases and you blow through your available memory. Is there a better architecture? Is there a way to get more scalability out of your database? Is the database even the best place for this data?
percona guys (one of the highest mysql authorities) put some nails in mysql coffin (innocently?)
http://www.umiacs.umd.edu/~jimmylin/cloud9/docs/content/start-EC2.html
This tutorial will get you started with Cloud9 on Amazon’s EC2 (running the simple word count demo). For a gentler introduction to Hadoop, or if you don’t feel like experimenting with EC2, try my tutorial on getting started with Cloud9 in standalone mode. This tutorial assumes you’ve already downloaded Cloud9 and gotten it set up. Otherwise, see my tutorial on that.
http://www.michael-noll.com/wiki/Hadoop
* Writing An Hadoop MapReduce Program In Python
* Running Hadoop On Ubuntu Linux (Single-Node Cluster)
* Running Hadoop On Ubuntu Linux (Multi-Node Cluster)
Teich also said that all of Heroku’s m2.2xlarge instances were running in a single availability zone, which was a mistake. He stressed that Heroku had failover built in already — if 21 instances had failed instead of 22, or if it had spread instances across several zones, “we wouldn’t be talking [about the outage],” he said.
Nevertheless, on Friday, January 2, every m2.2xlarge instance in that availability zone suddenly vanished, despite all other types of EC2 instances running as normal. That’s unheard of in traditional hosting. It would be like every server with a given amount of RAM suddenly shutting down, regardless of operating system, age, brand, hardware or location in the data center, with no effect on its neighbors.
“For us, there’s the stuff you plan for and then there’s the stuff you don’t even know about,” Teich said.
An event like this was an “unknown unknown” that nobody planned for because nobody imagined it. He chalked it up to the learning process and pointed out that everybody in Amazon Web Services was flying by the seat of the pants at least part of the time.
HN thread:
http://news.ycombinator.com/item?id=1046500
A while ago one of the major problems people faced to use Amazon EC2 into production environments was the dynamic state of the instances IPs. Every time one instance was started it was getting a new, dynamic IP. This has been addressed with the introduction of Amazon Elastic IP Addresses, but even when using this, the private IPs are still dynamic and most of the time people will want to communicate between several instances on the private allocated IPs and not on the public ones. This article will show how you can easily automate the process to update DNS hostnames for your EC2 instances, by adding to the AMI’s the logic for this. I will use for this a master DNS server running bind9, but this can be adapted to any other DNS server.
http://bret.appspot.com/entry/how-friendfeed-uses-mysql
We use MySQL for storing all of the data in FriendFeed. Our database has grown a lot as our user base has grown. We now store over 250 million entries and a bunch of other data, from comments and “likes” to friend lists.
As our database has grown, we have tried to iteratively deal with the scaling issues that come with rapid growth. We did the typical things, like using read slaves and memcache to increase read throughput and sharding our database to improve write throughput. However, as we grew, scaling our existing features to accomodate more traffic turned out to be much less of an issue than adding new features.
http://www.powercram.com/2010/01/amazon-ec2-instance-types.html
Amazon EC2 Instance Types.
Standard Instances
Instances of this family are well suited for most applications …
http://blog.reddit.com/2010/01/why-did-we-take-reddit-down-for-71.html
Part of our setup uses what we call a “permacache”, which uses Memcachedb. Memcachedb is Memcached with a built-in permanent storage system using BDB. One of the “features” of this system is that it saves up its disk writes and then bursts them to the disk. Unfortunately, the single EBS volumes they were on could not handle these bursting writes. Memcachedb also has another feature that blocks all reads while it writes to the disk. These two things together would cause the site to go down for about 30 seconds every hour or so lately.
http://wagerlabs.com/setting-up-erlang-on-amazon-ec2
Amazon does not provide tools to cluster your instances or replicate data among them. This is a task that Erlang copes with extremely well so Amazon EC2 and Erlang are a match made in haven!
http://blog.kenweiner.com/2009/09/amazon-elb-capturing-client-ip-address.html
If you’re using Amazon EC2’s Elastic Load Balancer (ELB) for load balancing web applications, you may have noticed that in your web access logs, the remote host IP address is the same for every request. The IP address you see is the private IP address of the load balancer.
If you want to see the IP address of the client (called remote host in access log documentation), you’ll need to look at the value of the X-Forwarded-For request header which ELB populates when it forwards the request.
http://moussadao.com/2009/12/30/amazon-ec2-pricings-demystified/
Amazon Elastic Compute Cloud (Amazon EC2) is the leading provider for cloud based computes resources. Amazon EC2 allows users to provision compute resources on-demand. With On-demand Instances, Amazon EC2 does not require any commitment, sign-up fees, and upfront payments. Amazon Web Services (AWS) uses a monthly billing cycle; therefore users get charged at end of each month for their EC2 usages.
http://www.lyquidity.com/devblog/?p=56
As a result of finding out that our emails supplier may be bouncing valid emails just because a sender’s DNS settings are not just-so we decided to create our own mail server. It was surprisingly easy and, for us, cost effective.
http://aws-musings.com/how-to-create-an-ebs-image-from-an-existing-ec2-instance/
Amazon recently announced a new feature which allows you to boot from an ebs volume. But it doesn’t provide any tools to convert your existing AMIs to this new type of image. There is no easy way to create an ebs image from scratch. There are some posts that explain how to convert your existing AMI into this new type of image using ec2-unbundle and dd (a linux utility). I am going to take a little different route and explain how we can create an ebs image from an existing instance. It’s fairly simple to create a new image using dd from an existing instance.
http://cheat.errtheblog.com/s/ec2/
ec2 cheat sheet
The Zend Framework contains several classes that make using cloud-based storage services easy. Part 1 of this “Cloud computing with PHP” series looks at using Zend classes with Amazon’s S3 cloud storage service. This article covers the Zend classes that make it easy to work with virtual machines in Amazon’s Elastic Compute Cloud (EC2).
http://stackoverflow.com/questions/1967452/ec2-and-login-logging
Hello – I have set up several EC2 instances and intend to give multiple users access to them via SSH. Is there any file on the server that monitors who logs in and when?
SSH logins are usually logged automatically in /var/log/auth.log. You can grep this file to check who logged in and when.
http://tech.backtype.com/swap-space-on-ec2
Recently, while investigating some mysterious crashes we were experiencing, I discovered that certain EC2 instance types have no allocated swap space. This post covers some basics of swap, what I learned about swap partitions on EC2 instances, and what you can to create swap on the machines that do not have any.
As cloud computing continues to evolve many questions arise around the best way to design a highly reliable site on the cloud with failover and recovery. As a system administrator you must always plan for failure. This document will help explain the best way to create affordable and reliable sites on EC2 using Elastic IPs (EIP) and availability zones.
http://solutions.amazonwebservices.com/connect/message.jspa?messageID=133726
I’m considering moving some of the web sites I run to EC2, but I haven’t yet managed to find out information about how often the virtual machines crash/vanish/go down etc…
Also, what happens when they do? Does the machine get automatically restarted with the same virtual hard disk with data as-saved before the crash/vanish happens?
What about mapped ips, etc?
As you can tell, I’m pretty new at this stuff, but just trying to get a feel for how things work when virutal machines or their physical hosts crash, etc…
http://elwoodicious.com/2009/12/23/using-elb-to-serve-multiple-domains-over-ssl-on-ec2-for-giggles/
One of the complaints about EC2 is that you only have one IP address allocated per instance which makes it difficult to host, in a clean manner, multiple domains that require SSL certs. Where you have control over IP allocation you could punch down a couple for one server and then set up your domains and SSL certs by IPs. That method is a no go so you are left with the ugly method of allocating those certs by port, something that Joe and Jane public are a little skittish about (https://superawesomefuntime.com is cool but https://superawesomefuntime.com:8443 smells phishy). Thankfully, ELB makes for a great proxy to hide the hideousness of the port-based workaround.
http://developer.amazonwebservices.com/connect/thread.jspa?threadID=34938
I would like to know two things below, because I am observing an elb which has got only one IP address (namely one virtual load balancer) is giving us lots of timeouts under a certain load which a single c1.medium Apache server can process without any timeouts.
(1) basic performance info of a typical virtual load balancer under an elb, including rate per second and concurrent connections. In our case, http.
(2) what triggers “ramping up” elb
http://www.higherpass.com/linux/Tutorials/Building-Hadoop-Clusters-On-Linux-In-Ec2/
Learn to build and use multi-node Hadoop clusters running in Amazon EC2. A few bits of knowledge are assumed in this article, first a basic knowledge of Hadoop. If you haven’t used hadoop before you probably want to read Intro To Hadoop Article first.
http://www.higherpass.com/java/Tutorials/Building-Hadoop-Mapreduce-Jobs-In-Java/
Hadoop is a parallel job processing framework from the Apache foundation. The hadoop framework is written in java and supports jar files for job execution. This tutorial is going to cover building a MapReduce job in java. The dataset being used will be the 2000 US Census available as an EBS volume snapshot on Amazon EC2. The census dataset is extemely large, and only a small part of the overall dataset will be explained.
http://blogs.23.nu/c0re/2009/12/running-a-couchdb-cluster-on-amazon-ec2/
CouchDB is a nearly zero-configuration multi-master document oriented database. It is a awsome product build by an awsome team.
So far I have been using CouchDB like we would have used any other modern Document Datastore: in a centraized fashion. One Server at our premises. For backup purposes we replicated on a second couchdb instance running on our backup server.
Hosting about 300 GB of data a small 2.6 GHz Server with consumer-grade disks we started seeing preformance issues. Also we see latency issues since we are hosting some application at Amazon EC2 “in the cloud” which results an an addiotional 40 ms delay for all queries to our locally hosted server.
Use this technique at your own risk. You need to be hot on snapshots, backups and restores to ensure you dont lose everything.
* Raid 0 offers no fault tolerance
* Disk reads and writes can be potentially 400% faster
* Loss of one disk will destroy the entire dataset
http://www.guidelightsolutions.com/blog/aws-managers
When explaining AWS for the first time to managers (or anyone, for that matter) it is best to talk in concepts rather than in concrete terms. I’ve also noticed it is beneficial to try to tangiblize the discussion with familiar terms. Using terms like “Elastic IP” gets quizical looks, but calling it a publicly accessible IP address helps people to get a better overall grasp of concepts before using the AWS terms for things.
http://zuzara.com/blog/2009/11/01/tpcc-mysql-rough-benchmark-for-amazon-rds/
I tried to do tpcc-mysql benchmark for Amazon RDS. Before do that, did the same test for EC2 small instance.
This is a pretty rough benchmark, but I can say EC2 small instance and RDS small instance have the same performances as CPU and memory are the same spec. RDS is about 30% expensive. (EC2=$0.085, RDS=$0.11/h)
Wordpress Ec2 Heroku Jack of all Clouds