My definition of a build server is a standalone box that has been configured so that it can build a release of your software with one command. The server typically has compilers, install packing software, and is dedicated to the task of building a release. Questions from the Joel Test that a build server needs to answer in the affirmative:
Build in one step?
My build scripts, that run on the build server, perform these tasks:
- Clean the Build Directories
- Checkout all the source files from source control.
- Compile all the source.
- Package the binaries and assemblies into installations (setup.exe)
- Deploy the installations to the download location on the web sever.
- Update a database (web content server) with version information, download location etc…
Over the years, I built different types of build servers, and written many build scripts. Ten years ago, I had dedicated build servers. Lately, I use virtual machines hosted on top of Virtual Server 2005. Since I am moving everything to the cloud, I wanted to use Amazon EC2 for Windows as my build server.
One reason I choose Amazon EC2 is that my source control hosting company is unfuddle.com and they use Amazon EC2 and S3. Because they are on S3, data transfer between my EC2 server and S3 is free and is very fast. The installations in step 5 are going back to my S3 account (for download by my users) so those are also free of transfer charges. However, regardless of where you are hosting your source control,
“If your source control is externally hosted on the Internet, an EC2 build server is the next step.”
I tried to do this several months ago, however the 10 Gigabyte limitation on the Amazon Machine Image (AMI) and the slow boot times of a saved AMI made it impossible. Several days ago Amazon announced that you could create a 30 Gigabyte Elastic Block Store (ESB) instance on EC2. The ESB start and stop much easier then the AMI. This is the feature I needed to make this project work. 10 Gigabytes is enough to run the IIS web server and a web site, but in order to get everything else (Visual Studio installed, my source, and have room to compile), I need that 30 Gigabyte ESB instance. It took me 2 days to create the build server. Most of the time was spent installing software and configuring my build script to run remotely.
Here are the steps:
Creating a Key Pair
- Open the AWS Management Console
- Choose Key Pairs.
- Press the button to Create a Key Pair
- Enter a name, like Build, and press Create.
- If you are like me and running Internet Explorer, you will get a download file yellow warning ribbon at the top of your browser. Choose to Download File, however the Ajax of the AWS Management Console doesn’t work well here, which means that this Key Pair is lost (you need to get that file)
- Delete the key pair you just created.
- Without closing the browser, repeat the steps again. This time the file will download. Save it to a very safe place.
Once you create a key pair, you will not have to do it again. Now you need to create an instance.
Launch a Default Instance
- Choose AMIs and select the filter at the top to view “ESB images” (with operating system already installed)
- Choose “ami-ba22c0d3”, which is labeled: amazon/Windows-Server2003R2-i386-Base-v108
- Right click and choose “Launch Instance”.
- Walk through the wizard using the key pair you created.
- You want 1 instance of the High-CPU Medium instance type.
- You do not need CloudWatch monitoring. Amazon CloudWatch is a web service that provides monitoring for AWS cloud resources, starting with Amazon. For a build server where it is required that it not always be up CloudWatch is not needed.
- Press the Launch Button.
Once you launch your instance, you need to wait for it to start. Go back to the instance list in the AWS Management console and look for your booting instance. Keep checking the System Log (right click System Log) until it says Windows is Ready to use and has a password line. It is not ready if it is blank – all black.
Getting You Administrator Password
Once the instance is booted, it has a separated virtualized disk just for you.
“The local Administrator password that is just for your instance.”
This password is not what you see in the System Log. The password in the System Log plus your key pair decrypt into the Administrator password.
- In the AWS Management console right click on your instance and choose “Get Windows Password”.
- Open your key pair file that you downloaded and copy the whole contents into your clipboard.
- Paste the clipboard in the text area that is presented in the window from the AWS Management Console.
- Press decrypt password
If you reset your password the next time you login, that will be the last time you have to do that.
Logging in With Remote Desktop
“All access to a Windows EC2 image is done via remote desktop connection.”
Remote desktop is a utility that you have on your Windows operating system in the Accessories folder. The first thing is figure out the DNS that Amazon has allocated for your instance. It is one of the columns in the instance list, or if you want to copy it into your clipboard:
- In the AWS Management Console, right click on the instance.
- Choose Connect from the drop down menu.
- Highlight the DNS Name and copy it into your clipboard
- Once you have the DNS name in your clipboard, open a session of Remote Desktop and paste in the DNS name.
- Login with “Administrator” and the password you decrypted.
Change the Password
The next step is to change the password so you don’t have to remember or decrypt the default Amazon password. Once you are logged in, you can do this by clicking on the Windows Security option from the Start menu.
Run Windows Updates
First thing you want to do with every new machine is run Windows updates. There have been many security updates since Windows 2003 R2. This is a good time to mention: when you shutdown your ESB Windows you are not charged for CPU time (however you are charged for the storage of the image). Rebooting your instance (like you have to do when you run Windows Updates) doesn’t make it disappear.
“The only way to lose the ESB image is to Terminate it via the AWS Management console.”
You do not want to terminate unless you want to start again from scratch. Shutting down Windows makes your instance shutdown. So when you are done for the day, just shut Windows down and the ESB instance will be there for you to launch the next day. You are billed by the whole hour, so it doesn’t make sense (with a ten minute boot up time) to shut it down when you go to lunch.
Remember that after each reboot forced on you by Windows update, you need to rerun the Windows updates until it tells you there is no more updates. This is Windows 2003 and it will not do all the updates at once.
First thing I need to create a build machine is an install of Visual Studio 2008. However, there is no CD-ROM drive.
“Everything that you want to install on your EC2 instance needs to be downloaded to it.”
Since I am an MSDN subscriber I can download the Visual Studio 2008 .iso directly from the Microsoft.com web site to the EC2 instance and save it on a file on the hard disk. Downloads are amazingly fast to the EC2 instance. Amazon has a great pipe to the Internet and the ESB image writes fast, much faster than a .vhd in Virtual Server 2005.
Once I have the .iso on the hard disk, I need to mount it to install. I am using a free installation of DAEMON tools lite. If you use the custom install option you can opt out of the installation changing your home page, your default search provider and install a toolbar in Internet explorer. Once you have their software installed and you reboot, you can mount your .iso and it appears like a DVD under your computer in Windows explorer.
The next thing I need to install is InstallShield 2008, which I have on DVD. To do this, I use Amazon S3 Organizer, a plug-in for Firefox and upload the whole DVD to a bucket on Amazon S3 from my desktop computer. Then I install Firefox and Amazon S3 Organizer on the running EC2 instance and download the whole DVD to a directory on the C:\ drive. The only issue is that S3 Organizer won’t transfer zero byte files from S3 to the hard drive and the InstallSheild installation has about 20 files with nothing in them. I solve this problem by taking the error list of zero byte files from the S3 Organizer and creating 20 files by hand on the hard drive – that are empty. Fortunately the install works from the directory without having to be run from a root.
Getting the Source Files
Checking out the source from my build script (a batch file) requires that I have an SVN client installed. This is because Unfuddle.com, my source control repository, supports SVN. I use Silk SVN 1.6 which is free and I can download directly to the EC2 instance. The command I add to the batch file look like this:
svn.exe update --non-interactive
Since I have already checked out the repository to the build root, svn.exe knows what repository to pull the files from. Because Unfuddle.com uses S3, updating the files is amazingly fast.
Moving the Installations Off the Server
Once I have the installations created using the InstallShield product, I need to move them to a location where my users can download them. Since I want to shut down the build machine after I am done building, they need to move off of this machine. I chose to place them in my Amazon S3 bucket, and make them as publically available. Anyone can download them directly from Amazon’s servers. To do this, I needed to write some C# code that copied the files to the right location. I created a C# console application called BuildHelper.exe that took in a number of parameters and created a unique file path and name for each installation that the build script created. The BuildHelper.exe project was added to the list of projects I was building and is compiled every time the build script ran.
The command line in the build script batch file looks like this:
echo "BuildHelper.exe" /publish /filename "bigDrive_S3_%1.%2.%3.%4_0600_x86.exe" /filepath "%INSTALLDIR%\Installation\AmazonS3\AmazonS3\WindowsVista\DiskImages\DISK1\setup.exe" /os "Windows Vista" /platform "x86" /productline "BigDrive" /product "BigDrive for Amazon S3" /major %1 /minor %2 /build %3 /intraday %4
%1 %2 %3 %4 are the version parameters passed into the batch file. Each build is has a different version number. /filepath is the location of the setup.exe file. /filename is the new name in the Amazon S3 bucket. I have one of these lines for each operating system and installation created.
Updating The Version Database
At the end of my build script, I am in a really good spot. The assemblies are compiled. I have a setup.exe and it has been deployed where users can download it. The build script “knows” that everything has happened successfully, the version of the release, and what was released. Now I need it to “tell” my version database this information so that the external web site is updated correctly. However, the build server can not access the version database. The version database is behind my firewall in the company data center, and the EC2 image is out on the Internet and is going to be shut down.
To update the database I added another method to BuildHelper.exe that makes an HTTP REST API call to our external website. The REST API is configured to accept the version number, product name, platform etc… and update the internal version database. The database is updated with a public column bit turned off, so that the release doesn’t appear on the web site until we are ready to deploy it.
Once the build is completed, I shut down the Windows 2003 Server, which shuts down the EC2 instance and the clock stops running. The whole process takes about 30 minutes.