PHP App Engine

http://cheat.errtheblog.com/s/ec2/

ec2 cheat sheet

http://www.ibm.com/developerworks/views/opensource/libraryview.jsp?search_by=cloud+computing+with+php

The Zend Framework contains several classes that make using cloud-based storage services easy. Part 1 of this “Cloud computing with PHP” series looks at using Zend classes with Amazon’s S3 cloud storage service. This article covers the Zend classes that make it easy to work with virtual machines in Amazon’s Elastic Compute Cloud (EC2).

http://stackoverflow.com/questions/1967452/ec2-and-login-logging

Hello – I have set up several EC2 instances and intend to give multiple users access to them via SSH. Is there any file on the server that monitors who logs in and when?

SSH logins are usually logged automatically in /var/log/auth.log. You can grep this file to check who logged in and when.

http://tech.backtype.com/swap-space-on-ec2

Recently, while investigating some mysterious crashes we were experiencing, I discovered that certain EC2 instance types have no allocated swap space. This post covers some basics of swap, what I learned about swap partitions on EC2 instances, and what you can to create swap on the machines that do not have any.

http://blogs.23.nu/c0re/2009/12/running-a-couchdb-cluster-on-amazon-ec2/

CouchDB is a nearly zero-configuration multi-master document oriented database. It is a awsome product build by an awsome team.

Hosting about 300 GB of data a small 2.6 GHz Server with consumer-grade disks we started seeing preformance issues. Also we see latency issues since we are hosting some application at Amazon EC2 “in the cloud” which results an an addiotional 40 ms delay for all queries to our locally hosted server.

http://www.linuxjournal.com/article/6675

Link your home and office securely with a virtual private network.

Can use for office/cloud link

http://blog.shutupandcode.net/?p=374

Solr, Zabbix, Selenium

http://sderot.dotanmazor.com/index.php?option=com_content&view=article&id=96:ec2-ami-move&catid=16:2008-11-14-08-43-34&Itemid=48

Who are you? You are running a VPS server (AMI, image, running as an instance) in the US location, of the Amazon EC2 service. And you want to perform a simple task: have the instance run in the EU (European Union) location.

Why are you here? Because this ’simple task’ can drive a normal person insane. There are bits of answers that are scattered all over the Internet, and you can’t find your head from your tail in all that noise.

http://flurdy.com/docs/ec2/

I use Amazon’s ec2 for most of my server needs and then some.
And have decided to document some of these uses, for others benefit.

http://tech.mangot.com/roller/dave/entry/on_running_terracotta_on_ec2

One recent project he asked me to do was run a large test of Terracotta on a cloud. Any cloud. So I set out to find an appropriate cloud environment and a way to run one of our session clustering tests. What I wound up with was a framework that can be used to scale anything on EC2 with relative ease.

http://www.vincestross.com/2009/04/upgrade-an-ec2-instance/

I’ve been using the Amazon Elastic Computing Cloud (EC2) for running our web servers at NetCrafters for almost a year now. It’s been an amazing experience and I’ve kept a detailed account of many of the lessons learned.

Amazon Web ServicesThe most recent challenge came when our servers just started locking-up for no reason. The sites would still be responding so we knew the LAMP stack was still limping along, but they were completely unresponsive via SSH. The only way to regain control was to issue a reboot through the Amazon API command line tools. Even this would sometimes take two or three times before the server would cycle.

kernel BUG at arch/i386/mm/pgtable-xen.c:306!

http://stackoverflow.com/questions/1761609/whats-a-good-way-to-collect-logs-from-amazon-ec2-instances

I haven’t implemented it yet, but I came across Facebook Scribe, and it seems like a good idea. http://developers.facebook.com/scribe/

Have a look at the free version of Splunk – it will handle the remote log file collection and also give you some really nice search and analysis tools.

http://clouddevelopertips.blogspot.com/2009/11/how-to-work-with-contractors-on-aws-ec2.html

1. Experience: experience solving real world problems is probably more important than anything else.
2. Code: someone who can produce running code is often more useful than someone who just makes recommendations for others to follow.
3. Community Engagement: discussion boards are a great way to gauge experience, and provide insight into the capabilities of the candidate.
4. Blogs & Whitepaper: another good way to determine a candidate’s insight and capabilities.
5. Interview: ask the candidate questions to gauge their qualifications.
6. References: do your homework and make sure the candidate really did what s/he claims to have done.

http://blog.readytocloud.com/2009/03/configuring-squid-in-amazon-ec2-cloud.html

If you have a web server and you are suddenly faced with an overload of traffic to your site and you want to buy some time before you re-engineer your site with or without the cloud computing options in mind, you can build a quick reverse proxy squid server to help you out. It will take one hour of your time (assuming you already have an account with Amazon, if not then about 2 hours) and you can have a full reverse proxy server running. This will permit your site more breathing room by caching all requests and serving them to subsequent users. It is a powerful mechanism but needs to be thought out completely if used as a long term solution.

http://www.cultofgary.com/2007/10/05/ec2-centos-and-pptp/

Continuing on my EC2 journey.. I wanted to allow my ec2 hosts to talk to a private network at the office. We use a Microsoft PPTP vpn — it’s the quickest way for me to set up the VPN.

I’ve been the Rightscale CentOS 5 AMI for my testing. In order to get my PPTP vpn connection up and running, I used pptpclient. To install it, I mostly followed their instructions.

http://atbrox.com/2009/11/11/how-to-combine-elastic-mapreducehadoop-with-other-amazon-web-services/

Elastic Mapreduce default behavior is to read from and store to S3. When you need to access other AWS services, e.g. SQS queues or database services SimpleDB and RDS (MySQL) the best approach from Python is to use Boto. To get Boto to work with Elastic Mapreduce you need to dynamically load boto on each mapper and reducer, Cloudera’s Jeff Hammerbacher outlined how to do that using Hadoop Distributed Cache and Peter Skomorroch suggested how to load Boto to access Elastic Blockstore (EBS), this posting is based on those ideas and gives a detailed description how to do it.

This posting shows how to load boto in an Elastic Mapreduce mapper and gives a simple example how to use simpledb from the same mapper. For accessing other AWS services, e.g. SQS from Elastic Mapreduce check out the Boto documentation (it is quite easy when the boto + emr integration is in place).

http://browsermob.com/performance-testing

Website Monitoring

* Control 2K+ real browsers; 25K+ virtual users
* Extends popular Selenium open source tooklit
* No lock-in: direct MySQL access to your data
* Provides real-time reporting for fast debugging
* Captures screenshots of failures to help debug
* Affordable: costs as little as 7.5 ¢ per user

http://developer.amazonwebservices.com/connect/entry.jspa?externalID=2241

The environment of choice for most programmers currently developing Java web applications with Tomcat is the Eclipse integrated development environment (IDE) in conjunction with the built-in Web Tools Platform (WTP) that comes with most distributions of Eclipse for Java Platform, Enterprise Edition (Java EE) development. In the past, developing these Java web applications with Tomcat and deploying them through Amazon EC2 was a considerably irritating process that involved multiple steps and a number of separate tools. In other words, there was no simple, streamlined solution. As a result, developers had to find or build their own AMI, use a different tool to start an instance based on that AMI, write their own deployment script, then configure Tomcat on Amazon EC2 so that it connected to the debugging session. Then, after perfecting the application, developers had to use different tools to bundle the AMI.

The AWS Toolkit for Eclipse solves this problem by moving all those tasks into Eclipse. It also automates most of the steps required, making it easier to develop, deploy, debug, and bundle Java applications in Amazon EC2. It is important to note that the toolkit is very well documented. After installing it, I strongly encourage you to explore the documentation. Simply access it by clicking Help > Help Contents in Eclipse. If you have installed the toolkit properly, you will find a new topic in the left pane of the Eclipse Help titled AWS Toolkit User Guide.

Also: http://kopfkind.blogspot.com/2009/11/amazon-ec2-first-steps.html

Steps in short
- Install the AWS Toolkit for Eclipse using their update site
- Provide the account credentials in the Eclipse settings (account number, access key id, access secret)
- Create a dynamic web project using the Eclipse project wizard
- Create a servlet for the web project, also using the Eclipse wizard
- Edit the servlet (details below)
- Deploy to local Tomcat for testing, verify functionality
- Create EC2 Tomcat instance in Eclipse’s “Server” view
- Deploy to EC2, test on remote host

http://www.foamz.za.net/?p=121

One of my colleagues pointed out that there was no swap on one of our EC2 servers. I missed it because we make use of predefined images from Rightscale and usually they define all that jazz for you.

I also noticed there was a /dev/sdc drive attached to those instances. This gave me the opportunity to create a swap partition from that drive.

http://www.freemindsystems.com/amazon-ec2-introduction-useful-info-and-tricks/

I’m sure many people think the cloud is the future, and more or less have migration plannings. Also, some of these people point to Amazon as supplier for their price, stability and capabilities. This first post of EC2 are things you need to know from the standpoint of a sysadmin.

For a System Administrator, Amazon EC2 is a big challenge, mostly because their ephemeral nature:

http://sprayfly.com/2009/06/28/shell-script-for-automated-amazon-ec2-personal-proxy-server/

I then came accross a post on Duane Storey’s blog which described how he created an Amazon EC2 instance which could be used to make a proxy server available that was only to Iranian web users. Duane’s solution involves his own custom Fedora image which has Apache setup to run as a web proxy. This setup was pretty much exactly what I needed – except that I would need to make sure the Proxy server only allowed my own computer to connect rather than making it a public proxy server accessible to Iranians. After a bit of testing, I managed to get something going and was pleased to see that the proxy was very speedy indeed!

http://atbrox.com/2009/10/07/how-to-use-c-compiled-python-for-amazons-elastic-mapreduce-hadoop/

Sometimes it can be useful to compile Python code for Amazon’s Elastic Mapreduce into C++ and then into a binary. The motivation for that could be to integrate with (existing) C or C++ code, or increase performance for CPU-intensive mapper or reducer methods. Here follows a description how to do that:

Based on Shedskin

http://jimmyg.org/blog/2007/amazon-ec2-for-people-who-prefer-debian-and-python-over-fedora-and-java.html

British. Python.

EC2 API reference
Dev Guide
User Guide

http://awsdocs.s3.amazonaws.com/EC2/latest/ec2-api.pdf
http://awsdocs.s3.amazonaws.com/EC2/latest/ec2-ug.pdf
http://awsdocs.s3.amazonaws.com/EC2/latest/ec2-dg.pdf

http://alestic.com/2009/06/ec2-ami-bundle

There are two primary ways to create an image for EC2:

1. Create an EC2 image from scratch. This process lets you control every detail of what goes into the image and is the easiest way to automate image creation.
2. Rebundle a running EC2 instance into a new image. This approach is the topic of the rest of this article.

http://developer.amazonwebservices.com/connect/entry.jspa?externalID=2865

In order to upgrade your AMI to the latest kernel you will have to launch your existing AMI and rebundle it explicitly overriding the default bundling tools behavior – which is to use the instance meta-data service to determine your current kernel and ramdisk combination OR overriding the launch time kernel and ramdisk parameters.

http://till.klampaeckel.de/blog/archives/55-CouchDB-on-Ubuntu-on-AWS.html

Here’s a little HowTo on how to setup CouchDB on an AWS EC2 instance. But outside of AWS (and EC2), this setup works on any other Ubuntu server, and I suppose Debian as well.