Explore Continuous Delivery in AWS with the Pipeline Starter Kit
Continuous delivery, automating your software delivery process from
code to build to deployment, is a powerful development technique and the
ultimate goal for many development teams. AWS provides services,
including AWS CodePipeline (a continuous delivery service) and AWS CodeDeploy
(an automated application deployment service) to help you reach this
goal. With AWS CodePipeline, any time a change to the code occurs, that
change runs automatically through the delivery process you’ve defined.
If you've ever wanted to try these services, but not wanted to set up
the resources, we’ve created a starter kit you can use. This starter kit
sets up a complete pipeline that builds and deploys a sample
application in just a few steps. The starter kit includes an AWS CloudFormation
template to create the pipeline and all of its resources in the US East
(N. Virginia) Region. Specifically, the CloudFormation template
creates:
Let’s get started.
Decide how you want to source the provided sample application. AWS CodePipeline currently allows you use either an Amazon S3 bucket or a GitHub repository as the source location for your application.
The CloudFormation template allows you to choose either of these methods. If you choose to use a GitHub repository, you will have a little more set up work to do, but you will be able to easily test modifying the application and seeing the changes run automatically through the pipeline. If you choose to use the Amazon S3 bucket already configured as the source in the startup kit, set up is simpler, but you won’t be able to modify the application.
Follow the steps for your choice:
GitHub:
to launch the starter kit template directly in the AWS CloudFormation console. Make sure that you are in the US East (N. Virginia) region.
Note: If you want to download the template to your own computer and then upload it directly to AWS CloudFormation, you can do so from this Amazon S3 bucket. Save the aws-codedeploy-codepipeline-starter-kit.template file to a location on your computer that’s easy to remember.
Choose Next.
On the Specify Details page, do the following:
(Optional) On the Options page, in Key, type Name. In Value, type a name that will help you easily identify the resources created for the starter kit. This name will be used to tag all of the resources created by the starter kit. Although this step is optional, it’s a good idea, particularly if you want to use or modify these resources later on. Choose Next.
On the Review page, select the I acknowledge that this template might cause AWS CloudFormation to create IAM resources check box. (It will.) Review the other settings, and then choose Create.
It will take several minutes for CloudFormation to create the resources on your behalf. You can watch the progress messages on the Events tab in the console.
When the stack has been created, you will see a CREATE_COMPLETE message in the Status column of the console and on the Overview tab.
Congratulations! You’ve created your first pipeline, complete with all required resources. The pipeline has four stages, each with a single action. The pipeline will start automatically as soon as it is created.
(If CloudFormation fails to create your resources and pipeline, it will roll back all resource creation automatically. The most common reason for failure is that you specified a stack name that is allowed in CloudFormation but not allowed in Amazon S3, and you chose Amazon S3 for your source location. For more information, see the Troubleshooting problems with the starter kit section at the end of this post.)
To view your pipeline, open the AWS CodePipeline console at http://console.aws.amazon.com/codepipeline. On the dashboard page, choose the name of your new pipeline (for example, StarterKitDemo-Pipeline). Your pipeline, which might or might not have started its first run, will appear on the view pipeline page.
You can watch the progress of your pipeline as it completes the action configured for each of its four stages (a source stage, a build stage, and two deployment stages).
The pipeline flows as follows:
You can view the details of any stage except the source stage by choosing the Details link for that stage. For example, choosing the Details link for the Jenkins build action in the build stage opens the status page for that Jenkins build:
To view the details of the build, choose the link to the log file. To
view the Maven project created in Jenkins to build the application,
choose Back to Project.
While you’re in Jenkins, we strongly encourage you to consider securing it if you’re going to keep the resource for any length of time. From the Jenkins dashboard, choose Manage Jenkins, choose Setup Security, and choose the security options that are best for your organization. For more information about Jenkins security, see Standard Security Setup.
When Succeeded is displayed for the pipeline status, you can view the application you built and deployed:
Now that you have a pipeline, try experimenting with it. You can release a change, disable and enable transitions, edit the pipeline
to add more actions or change the existing ones – whatever you want to
do, you can do it. It’s yours to play with. You can make changes to the
source in your GitHub repository (if you chose GitHub as your source
location) and watch those pushed changes build and deploy automatically.
You can also explore the links to the resources used by the pipeline,
such as the application and deployment groups in AWS CodeDeploy and the
Jenkins server.
If you decide to keep the pipeline and some or all of its resources, here are some things to consider:
Problem: The stack creation fails when trying to create the custom action in AWS CodePipeline.
Possible Solution: You or someone who shares your AWS account number might have used the starter kit once and chosen the same name for the application. Custom actions must have unique names within an AWS account. Another possibility is that you or someone else then deleted the resources, including the custom action. You cannot create a custom action using the name of a deleted custom action. In either case, delete the failed stack, and then try to create the stack again using a different application name.
Problem: The stack creation fails in AWS CloudFormation without any error messages.
Possible Solution: You’re probably missing one or more required permissions. Creating resources with the template in AWS CloudFormation requires the following policy or its equivalent permissions:
Problem: Deleting the stack fails when trying to delete the Amazon S3 bucket created by the stack.
Possible solution: One or more files or folders might be left in the bucket created by the stack. To delete this bucket, follow the instructions in Delete or Empty a Bucket, and then delete the stack in AWS CloudFormation.
Problem: I want to use my own Amazon S3 bucket as the source location for a pipeline, not the bucket pre-configured in the template.
Possible solution: Create your own bucket, following these steps:
- An AWS Virtual Private Cloud (VPC), including all the necessary routing tables and routes, an Internet gateway, and network ACLs for EC2 instances to be launched into.
- An Amazon EC2 instance that hosts a Jenkins server (also installed and configured for you).
- Two AWS CodeDeploy applications, each of which contains a deployment group that deploys to a single Amazon EC2 instance.
- All IAM service and instance roles required to run the resources.
- A pipeline in AWS CodePipeline that builds the sample application and deploys it. This includes creating an Amazon S3 bucket to use as the artifact store for this pipeline.
- An AWS account. (Sign up for one here if you don’t have one already.)
- An Amazon EC2 key pair in the US East (N. Virginia) Region. (Learn how to create one here if you don’t have one.)
- Administrator-level permissions in IAM, AWS CloudFormation, AWS CodeDeploy, AWS CodePipeline, Amazon EC2, and Amazon S3. (Not sure how to set permissions in these services? See the sample policy in Troubleshooting Problems with the Starter Kit.)
- Optionally, a GitHub account so you can fork the repository for the sample application. Alternatively, if you do not want to create a GitHub account, you can use the Amazon S3 bucket configured in the starter kit template, but you will not be able to edit the application or see your changes automatically run through the pipeline.
Note: The resources created in the starter kit exceed
what’s included in the AWS Free Tier so the use of the kit will result
in charges to your account. The cost will depend on how long you keep
the CloudFormation stack and its resources.
Decide how you want to source the provided sample application. AWS CodePipeline currently allows you use either an Amazon S3 bucket or a GitHub repository as the source location for your application.
The CloudFormation template allows you to choose either of these methods. If you choose to use a GitHub repository, you will have a little more set up work to do, but you will be able to easily test modifying the application and seeing the changes run automatically through the pipeline. If you choose to use the Amazon S3 bucket already configured as the source in the startup kit, set up is simpler, but you won’t be able to modify the application.
Follow the steps for your choice:
GitHub:
- Sign in to GitHub and fork the sample application repository at https://github.com/awslabs/aws-codedeploy-sample-tomcat.
- Navigate to https://github.com/settings/tokens and generate a token to use with the starter kit. The token requires the permissions needed to integrate with AWS CodePipeline: repo and admin:repo_hook. For more information, see the AWS CodePipeline User Guide. Make sure you copy the token after you create it.
- If you’re using the bucket configured in the starter kit, there’s nothing else for you to do but continue on to step 3. If you want to use your own bucket, see Troubleshooting Problems with the Starter Kit.
to launch the starter kit template directly in the AWS CloudFormation console. Make sure that you are in the US East (N. Virginia) region.Note: If you want to download the template to your own computer and then upload it directly to AWS CloudFormation, you can do so from this Amazon S3 bucket. Save the aws-codedeploy-codepipeline-starter-kit.template file to a location on your computer that’s easy to remember.
Choose Next.
On the Specify Details page, do the following:
- In Stack name, type a name for the stack. Choose something short and simple for easy reference.
-
In AppName, you can leave the default as-is, or you can type a name of no more than 15 characters (for example, starterkit-demo). The name has the following restrictions:
- The only allowed characters are lower-case letters, numbers, periods, and hyphens.
- The name must be unique in your AWS account, so be sure to choose a new name each time you use the starter kit.
-
In AppSourceType, choose S3 or GitHub, depending on your preference for a source location, and then do the following:
-
If you want to use the preconfigured Amazon S3 bucket as the source
for your starter kit, leave all the default information as-is. (If you
want to use your own Amazon S3 bucket, see Troubleshooting Problems with
the Starter Kit.)
-
If you want to use a GitHub repo as the source for your starter kit, in Application Source – GitHub, type the name of your user account in GitHubUser. In GitHubToken, paste the token you created earlier. In GitHubRepoName, type the name of the forked repo. In GitHubBranchName, type the name of the branch (by default, master).
-
If you want to use the preconfigured Amazon S3 bucket as the source
for your starter kit, leave all the default information as-is. (If you
want to use your own Amazon S3 bucket, see Troubleshooting Problems with
the Starter Kit.)
- In Key Name, choose the name of your Amazon EC2 key pair.
-
In YourIP, type the IP address from which you will
access the resources created by this starter kit. This is a recommended
security best practice.
(Optional) On the Options page, in Key, type Name. In Value, type a name that will help you easily identify the resources created for the starter kit. This name will be used to tag all of the resources created by the starter kit. Although this step is optional, it’s a good idea, particularly if you want to use or modify these resources later on. Choose Next.
On the Review page, select the I acknowledge that this template might cause AWS CloudFormation to create IAM resources check box. (It will.) Review the other settings, and then choose Create.
It will take several minutes for CloudFormation to create the resources on your behalf. You can watch the progress messages on the Events tab in the console.
When the stack has been created, you will see a CREATE_COMPLETE message in the Status column of the console and on the Overview tab.
Congratulations! You’ve created your first pipeline, complete with all required resources. The pipeline has four stages, each with a single action. The pipeline will start automatically as soon as it is created.
(If CloudFormation fails to create your resources and pipeline, it will roll back all resource creation automatically. The most common reason for failure is that you specified a stack name that is allowed in CloudFormation but not allowed in Amazon S3, and you chose Amazon S3 for your source location. For more information, see the Troubleshooting problems with the starter kit section at the end of this post.)
To view your pipeline, open the AWS CodePipeline console at http://console.aws.amazon.com/codepipeline. On the dashboard page, choose the name of your new pipeline (for example, StarterKitDemo-Pipeline). Your pipeline, which might or might not have started its first run, will appear on the view pipeline page.
You can watch the progress of your pipeline as it completes the action configured for each of its four stages (a source stage, a build stage, and two deployment stages).
The pipeline flows as follows:
- The source stage contains an action that retrieves the application from the source location (the Amazon S3 bucket created for you to store the app or the GitHub repo you specified).
- The build stage contains an action that builds the app in Jenkins, which is hosted on an Amazon EC2 instance.
- The first deploy stage contains an action that uses AWS CodeDeploy to deploy the app to a beta website on an Amazon EC2 instance.
- The second deploy stage contains an action that again uses AWS CodeDeploy to deploy the app, this time to a separate, production website on a different Amazon EC2 instance.
You can view the details of any stage except the source stage by choosing the Details link for that stage. For example, choosing the Details link for the Jenkins build action in the build stage opens the status page for that Jenkins build:
Note: The first time the pipeline runs, the link to
the build will point to Build #2. Build #1 is a failed build left over
from the initial instance and Jenkins configuration process in AWS
CloudFormation.
While you’re in Jenkins, we strongly encourage you to consider securing it if you’re going to keep the resource for any length of time. From the Jenkins dashboard, choose Manage Jenkins, choose Setup Security, and choose the security options that are best for your organization. For more information about Jenkins security, see Standard Security Setup.
When Succeeded is displayed for the pipeline status, you can view the application you built and deployed:
- In the status area for the ProdDeploy action in the Prod stage, choose Details. The details of the deployment will appear in the AWS CodeDeploy console.
- In the Deployment Details section, in Instance ID, choose the instance ID of the successfully deployed instance.
- In the Amazon EC2 console, on the Description tab, in Public DNS, copy the address, and then paste it into the address bar of your web browser. The web page opens on the application you built:
Tip: You can also find the IP addresses of each instance in AWS CloudFormation on the Outputs tab of the stack.
What to Do Next
After you’ve finished exploring your pipeline and its associated resources, you can do one of two things:- Delete the stack in AWS CloudFormation, which deletes the pipeline, its resources, and the stack itself. This is the option to choose if you no longer want to use the pipeline or any of its resources. Cleaning up resources you’re no longer using is important, because you don’t want to be charged for resources you no longer using.
To delete the stack:
- Delete the Amazon S3 bucket used as the artifact store in AWS CodePipeline. Although this bucket was created as part of the CloudFormation stack, Amazon S3 does not allow CloudFormation to delete buckets that contain objects. To delete this bucket, open the Amazon S3 console, select the bucket whose name starts with demo and ends with the name you chose for your stack, and then delete it. For more information, see Delete or Empty a Bucket.
- Follow the steps in Delete the stack.
- Change the pipeline and its resources to start building applications you actually care about. Maybe you’re not ready to get into the business of creating bespoke suits for dogs. (We understand that dogs can be difficult clients to dress well, and that not everyone wants to be paid in dog treats.) However, perhaps you do have an application or two that you would like to set up for continuous delivery with AWS CodePipeline. AWS CodePipeline integrates with other services you might already be using for your software development, as well as GitHub. You can edit the pipeline to remove the actions or stages and add new actions and stages that more accurately reflect the delivery process for your applications. You can even create your own custom actions, if you want to integrate your own solutions.
If you decide to keep the pipeline and some or all of its resources, here are some things to consider:
- Review the IAM policies and roles created for you by the starter kit and modify their permissions, if necessary.
- Use one of the managed policies provided by AWS to give permissions for AWS CodePipeline to IAM users and groups you want to access the pipeline.
- If you plan to continue using AWS CodeDeploy, add or remove permissions for AWS CodeDeploy, review the IAM instance profile and service role that were created for you, and review or change the deployment group settings.
- If you plan to continue using the Jenkins server or the Amazon EC2 instance it is hosted on, make sure you secure both Jenkins and the EC2 instance appropriately.
Troubleshooting Problems with the Starter Kit
You can use the events on the Events tab of the CloudFormation stack to help you troubleshoot problems if the stack fails to complete creation or deletion.Problem: The stack creation fails when trying to create the custom action in AWS CodePipeline.
Possible Solution: You or someone who shares your AWS account number might have used the starter kit once and chosen the same name for the application. Custom actions must have unique names within an AWS account. Another possibility is that you or someone else then deleted the resources, including the custom action. You cannot create a custom action using the name of a deleted custom action. In either case, delete the failed stack, and then try to create the stack again using a different application name.
Problem: The stack creation fails in AWS CloudFormation without any error messages.
Possible Solution: You’re probably missing one or more required permissions. Creating resources with the template in AWS CloudFormation requires the following policy or its equivalent permissions:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"cloudformation:*",
"codedeploy:*",
"codepipeline:*",
"ec2:*",
"iam:AddRoleToInstanceProfile",
"iam:CreateInstanceProfile",
"iam:CreateRole",
"iam:DeleteInstanceProfile",
"iam:DeleteRole",
"iam:DeleteRolePolicy",
"iam:GetRole",
"iam:PassRole",
"iam:PutRolePolicy",
"iam:RemoveRoleFromInstanceProfile",
"s3:*"
],
"Resource": "*"
}
]
}
Problem: Deleting the stack fails when trying to delete the Amazon S3 bucket created by the stack.
Possible solution: One or more files or folders might be left in the bucket created by the stack. To delete this bucket, follow the instructions in Delete or Empty a Bucket, and then delete the stack in AWS CloudFormation.
Problem: I want to use my own Amazon S3 bucket as the source location for a pipeline, not the bucket pre-configured in the template.
Possible solution: Create your own bucket, following these steps:
- Download the sample application from GitHub at https://github.com/awslabs/aws-codedeploy-sample-tomcat and upload the suitsfordogs.zip application to an Amazon S3 bucket that was created in the US East (N. Virginia) Region.
- Sign into the AWS Management Console and open the Amazon S3 console at https://console.aws.amazon.com/s3.
- Choose your bucket from the list of buckets available, and on the Properties tab for the bucket, choose to add or edit the bucket policy.
-
Make sure that your bucket has the following permissions set to Allow:
- s3:PutObject
- s3:List*
- s3:Get*
For more information, see Editing Bucket Permissions. - When configuring details in CloudFormation, on the Specify Details page, in AppSourceType, choose S3, but then replace the information in Application Source – S3 with the details of your bucket and object.
No comments: