PHP App EngineClouds and NoSQL – by Smart Robot |
|
http://www.lindstromconsulting.com/node/8
In the previous article on the subject of cloud computing using AWS, we setup a simple LAMP application that used a single web server to present data that was queried from a single RDS instance. In this guide we will see how to save the changes we made to the EC2 instance, create more EC2 instances, and setup load balancing across our web servers.
Teich also said that all of Heroku’s m2.2xlarge instances were running in a single availability zone, which was a mistake. He stressed that Heroku had failover built in already — if 21 instances had failed instead of 22, or if it had spread instances across several zones, “we wouldn’t be talking [about the outage],” he said.
Nevertheless, on Friday, January 2, every m2.2xlarge instance in that availability zone suddenly vanished, despite all other types of EC2 instances running as normal. That’s unheard of in traditional hosting. It would be like every server with a given amount of RAM suddenly shutting down, regardless of operating system, age, brand, hardware or location in the data center, with no effect on its neighbors.
“For us, there’s the stuff you plan for and then there’s the stuff you don’t even know about,” Teich said.
An event like this was an “unknown unknown” that nobody planned for because nobody imagined it. He chalked it up to the learning process and pointed out that everybody in Amazon Web Services was flying by the seat of the pants at least part of the time.
HN thread:
http://news.ycombinator.com/item?id=1046500
A while ago one of the major problems people faced to use Amazon EC2 into production environments was the dynamic state of the instances IPs. Every time one instance was started it was getting a new, dynamic IP. This has been addressed with the introduction of Amazon Elastic IP Addresses, but even when using this, the private IPs are still dynamic and most of the time people will want to communicate between several instances on the private allocated IPs and not on the public ones. This article will show how you can easily automate the process to update DNS hostnames for your EC2 instances, by adding to the AMI’s the logic for this. I will use for this a master DNS server running bind9, but this can be adapted to any other DNS server.
http://www.powercram.com/2010/01/amazon-ec2-instance-types.html
Amazon EC2 Instance Types.
Standard Instances
Instances of this family are well suited for most applications …
http://wagerlabs.com/setting-up-erlang-on-amazon-ec2
Amazon does not provide tools to cluster your instances or replicate data among them. This is a task that Erlang copes with extremely well so Amazon EC2 and Erlang are a match made in haven!
As cloud computing continues to evolve many questions arise around the best way to design a highly reliable site on the cloud with failover and recovery. As a system administrator you must always plan for failure. This document will help explain the best way to create affordable and reliable sites on EC2 using Elastic IPs (EIP) and availability zones.
http://blog.shutupandcode.net/?p=374
Solr, Zabbix, Selenium
http://debianzone.org/raid-and-lvm-on-amazon-ec2-part-i/
This is the first part of three articles I’m posting for a great storage solution using RAID, LVM and Amazon Elastic Block Store (EBS).
First, you need to choose your RAID implementation. Personally, I prefer to use RAID 5 on Amazon EC2, combined with LVM2.
http://www.elastician.com/2009/12/creating-ebs-backed-ami-from-s3-backed.html
The recent introduction of Boot From EBS for EC2 opens up a lot of new possibilities. But there are some bootstrapping issues to deal with. There aren’t many EBS-backed AMI’s available yet and, given the rather complex process involved in porting them, it may take a while for them to show up. This article will walk through the process of converting a popular S3-based AMI to an EBS-backed AMI.
Launching a new webapp is never easy – even one as simple as a URL shortener. Will it catch on? If it does, what does that mean in terms of traffic? 10, 100, 1000 requests per second?
A few weeks ago we did some back of the envelope calculations for how big wi.nr could get in the best case. The calculation went something like: 10,000 active users x 10 shortened URLs per day x 100 people clicking on each of those URLs = 10M shortened URLs clicked a day or ~120 per sec.
http://web.mit.edu/stardev/cluster/
Multiple Clusters – Currently, StarCluster only supports launching a single cluster on ec2. In theory, the software should be able to START multiple clusters but it’s not equipped to handle the accounting on which nodes belong to which cluster after the initial startup. This means things like listing the nodes, terminating a particular cluster, etc will not work. Support for the correct account of multiple clusters should come in future versions.
Dynamic Load Balancing – Support for a dynamically resizing cluster on ec2. Integrating the Service Domain Manager (SDM) and Hedeby software products from SUN into the AMI will allow ec2 nodes to easily be added to the Sun Grid Engine queue. This means you could theoretically start a single node cluster and as the load increases, ec2 nodes would be launched, added to the cluster, used for computation, and then removed when they’re idle. The impact of this would be to significantly lower the cost of using EC2 by only having one node up 24/7 and adding/removing nodes as needed.
http://docs.google.com/View?docID=dhh4z6n4_96w387mqhn&revision=_latest
Using LVM + DRBD + NFS + Heartbeat + VTun To Gain Data Persistence, Redundancy, Automatic Fail-Over, and Read/Write Disk Access Across Multiple EC2 Nodes
This sounds like a dream: you are running out of CPU power? Easy, within 10 minutes you have another server ready to roll. However, it’s not really that simple. Unless you did things right, a new server won’t go very far — it needs to be configured to do the right thing. For example if your web server is under too much stress, and you add a server, the second one needs to be in the pool of daemons which serve web pages; more importantly, it needs to be able to access the data (database and files) in order to do that. So, realistically you will have a database server, a file server, a main web server which will act as a proxy, and a pool of web servers which will access data from the shared file system and the database server(s). Such a setup is common but it’s not as straightforward as it might sound. You might decide to have only one database server, and use memcached instead — therefore having a second pool of servers. No, it’s not for everybody. I pointed this out to Whichard. He said “Yeah, you are totally right. You definitely will need a script that will configure the new server and will get it to do something useful. The setup you mentioned [with a reverse proxy server dispatching HTTP requests to sub-servers] is the most common one; thing is, now there is the technology to do this. It might not be easy (well, not yet) but it’s definitely possible.
http://www.elusive-cloud.com/go/module/pages/page/106/
The requirement of the concept, in addition to relocating the web project to the EC2 platform, was to ensure that, with the help of AutoScaling, there was always an
optimal number of servers in use.
http://www.slideshare.net/gnap/big-data-on-ec2-mashing-technology-in-the-cloud
# Scenario… ✤ Given: >10^6 publishers, >10^9 users, >10^10 urls ✤ Early-stage start-up, < 25 people, seeking to minimize spend on Ops and capex for data centers ✤ Serving widgets on page views for popular online publications: ESPN, HuffPost, FOX, CS Monitor, CBS Marketwatch, Wired, TechCrunch, etc. ✤ Spikes in popularity of stories leads to elastic demands throughout the system architecture: serving API, logging, DW, BI, etc. ✤ Business needs to improve user experience by analyzing how people share online content ScaleCamp 2009-06-09
# System Architecture ✤ 100% infrastructure based on Amazon AWS ✤ Each component designed for cost-effective, horizontal scale-out ✤ AsterData: infrastructure based on a “hub-and-spoke” pattern of batch jobs and data consolidation ✤ Cascading: abstraction layer for tying together system components ✤ Batch jobs run on Amazon Elastic MapReduce and AsterData SQL/MR ✤ Vertical search based on Bixo and Katta ScaleCamp 2009-06-09
http://blog.readytocloud.com/2009/03/configuring-squid-in-amazon-ec2-cloud.html
If you have a web server and you are suddenly faced with an overload of traffic to your site and you want to buy some time before you re-engineer your site with or without the cloud computing options in mind, you can build a quick reverse proxy squid server to help you out. It will take one hour of your time (assuming you already have an account with Amazon, if not then about 2 hours) and you can have a full reverse proxy server running. This will permit your site more breathing room by caching all requests and serving them to subsequent users. It is a powerful mechanism but needs to be thought out completely if used as a long term solution.
http://zapnap.github.com/presentations/ec2-rubber/#0
Ruby kids like picturez.
Rubber.
Moonshine.
PoolParty.
http://mattlazycat.livejournal.com/357669.html
WhoKnowsWho runs on a scalable farm that responds to increased load by starting new servers. It’s a service that Amazon (as part of their CloudWatch monitoring service), and Scalr both offer.
http://www.reddit.com/r/IAmA/comments/a2zte/i_run_reddits_servers_and_do_a_bunch_of_other/
I made a blog post today about our move to the cloud, and thought I would give you all the chance to ask me questions, too. I’ll answer anything I can, and if I can’t, I’ll let you try to let you know.
To get the discussion going, here are some fun stats about our servers:
218 Virtual CPUs 380GB of RAM
9TB of Block Storage
2TB of S3 Storage
6.5 TB of Data Out / mo
2TB of Data In / mo
156M+ Pageviews
http://support.rightscale.com/03-Tutorials/02-AWS/02-Website_Edition/2.1_MySQL_Setup
There are two different ways of setting up a redundant MySQL database on Amazon EC2. In both cases, there will be a Master-DB and Slave-DB with replication for failover and recovery. The core difference between each setup is how the backups are saved.
http://www.mooreds.com/wordpress/archives/000549
The biggest attraction of the cloud was the ability to spin up and spin down extra servers as the expected traffic on the site increased or decreased. We choose Amazon’s EC2 for hosting. They seem a bit like the IBM of the cloud–no one ever got fired, etc. They have a rich set of offerings and great documentation.
Exactly.
Fork your AMI
Use Capistrano
Use EBS
EC2 Persistence
Use ElasticFox
Consider CloudFront
Use Internal Network Addressing where possible
Consider reserved instances
Some right, some not.
http://www.ardentsoft.com/blog/tag/lamp-on-ec2
Part 1: Setting up Amazon Web Services
Part 2: Setting up a Persistent Volume
Part 3: Starting and Configuring the Instance
Part 4: Mounting a Persistent Volume
Part 5: Configuring MySQL
Part 6: Configuring Apache
Part 7: Configuring PHP
Part 8: Setting up an Elastic IP Address
Part 9: Setting up Snapshots
Part 10: Generating a Custom AMI
http://wiki.github.com/kirktrue/voldemort/ec2-testing-infrastructure
An open source clone of Amazon’s Dynamo
Goals and Deliverables
The primary goals of this project are as follows:
1. Ability to initialize EC2 instances for use with Voldemort
2. Cluster deployment, configuration
3. Individual node start/stop
4. Ability to leverage above for performance and correctness tests
http://community.zenoss.org/blogs/mikelunt/2009/10/26/qa-testing–to-cloud-or-not-to-cloud
We are considering supplementing our engineering lab via the use of cloud services such Amazon’s EC2 or Rackspace’s Cloud Servers. Currently, we use a large set of VMWare servers to host our QA testing environment, where we perform installations of Zenoss servers as well as setup target systems (Windows, Linux, etc.) to monitor. In addition, these VMWare servers are connected to the same private network as various routers, storage devices, and other types of OS’s, such as HP-UX, Solaris, etc. All of this is done to simulate what Zenoss users have in their IT environments, and for the most part, the setup works quite well except that there’s never enough. The QA engineers are constantly finding new combinations of OS’s, applications, etc. that require more VM guests to be created. Here is a quick brain dump of the comparisons thus far.
http://www.cloudhosting.co.uk/content/amazon-aws-what-it-good
What exactly can you do with it and how can it help your business? 6 areas come to mind
• Testing environments
• Temporary hosting solutions
• Unknown capacity solutions
• Uneven web traffic patterns
• Web Start up businesses
• Large processing requirements
• Software and vendor evaluation
http://www.mikebrittain.com/blog/2008/07/19/web-hosting-on-ec2/
In the months prior to leaving Heavy, I led an exciting project to build a hosting platform for our online products on top of Amazon’s Elastic Compute Cloud (EC2). We eventually launched our newest product at Heavy using EC2 as the primary hosting platform.
I’ve been following a lot of what other people have been doing with EC2 for data processing and handling big encoding or rendering jobs. This is not one of those projects.
We set out to build a fairly standard LAMP hosting infrastructure where we could easily and quickly add additional capacity. In fact, we can add new servers to our production pool in under 20 minutes, from the time we call the “run instance” API at EC2, to the time when public traffic begins hitting the new server. This includes machine startup time, adding custom server config files and cron jobs, rolling out application code, running smoke tests, and adding the machine to public DNS.
What follows is a general outline of how we do this.
http://stackoverflow.com/questions/1597930/architecture-of-a-php-app-on-amazon-ec2
But what’s the next logical step that is going to yield good results for scalability? Do I fire up an AMI instance for the MySQL and one for the Apache service? Or do I just replicate the instances out as many times as I need them and then do some sort of load balancing on the front end? Ideally, I’d like to have a centralized database because I do aggregate statistics across all database rows, however, this is not a hard requirement (there are probably some application specific solutions I could come up with to work around this)
http://www.kaavo.com/blog/-/blogs/building-a-private-cloud-within-a-public-cloud
One of our customers wanted to establish a site to site connectivity between their datacenter and public cloud (Amazon EC2) and then have a private network within Amazon EC2 with their own custom IP addresses for their servers in the cloud. Basically idea here is to augment the internal datacenter resources with the resources in the public cloud securely so that the servers in the cloud appear as if they are part of their own private corporate network. The idea here is to isolate the servers used by the customer in the cloud from the rest of the servers in the cloud using private network, just like the corporate internal datacenters are isolated using private network with private routers routing the internal traffic.
http://developer.amazonwebservices.com/connect/entry.jspa?externalID=1074
A Redundant Array of Independent Drives (RAID) is a method of grouping multiple drives together to improve performance or fault tolerance. Because m1.large and m1.xlarge instances have multiple ephemeral stores, you can now use RAID with these instance types.
Important: Using RAID does not guarantee against instance failure and is not a replacement for a sound backup policy. If an underlying drive fails, you should immediately migrate to a new instance.
http://aws.amazon.com/solutions/case-studies/postrank/
AideRSS, Inc., a Waterloo, Ontario based technology company, provides a leading edge service, PostRankTM, that uses social engagement data to help publishers identify and build community, and readers to focus attention on the content that matters. PostRank collects a wide variety of social media metrics on a real-time basis (comments, bookmarks, tweets, etc.) to analyze the level of social engagement with online news posts and stories. Publishers benefit from understanding and being able to actively participate in community reaction to every piece of content, and readers can filter and rank the content they want to read.
Wordpress Ec2 Heroku Jack of all Clouds