CI/CD pipeline for AWS Glue

2019-11-26T04:08:49

I am fairly a new user of AWS Glue, which is one of new AWS managed services to orchestrate batch job workflows with ease.

  • I have 3 different AWS IAM account (Dev, Test, Prod). Separate login for each account.
  • Glue, Scala Jobs are version controlled in my Github repository.
  • I want to build a CI/CD pipeline to automate the Test, Build and deployment of my Glue job as I commit and push in my Github repository.

I went through number of blogs and article describing CI/CD best practices. I found a good article (specifically on CI/CD in Data Pipeline workflows) on AWS Blog here. It is quite simple and has details of building CodePipeline using CodeCommit & AWS Cloudformation. But all the phases of CI/CD initiate and ends in the same AWS IAM account.

I have two questions:

  1. Is separating Dev, Test and Prod account (different IAM) for building and managing Data Pipelines/ Data warehouse, a good practice?
  2. If yes, how would I design CI/CD pipeline using 3 different IAM account?

Any suggestion on best practices to design CI/CD pipeline for AWS-Glue ?

Copyright License:
Author:「Raxit Solanki」,Reproduced under the CC 4.0 BY-SA copyright license with link to original source & disclaimer.
Link to:https://stackoverflow.com/questions/59039385/ci-cd-pipeline-for-aws-glue

About “CI/CD pipeline for AWS Glue” questions

I am fairly a new user of AWS Glue, which is one of new AWS managed services to orchestrate batch job workflows with ease. I have 3 different AWS IAM account (Dev, Test, Prod). Separate login for ...
I am writing down the scope of our CI/CD pipeline to be developed using AWS native tools. What do you recommend? discussion I am in the process of finalizing the scope of our CI/CD pipeline that wi...
I need to create CI/CD pipeline in AWS cloud for a pyspark application , finally this py-spark is to be invoked through a airflow DAG.
From my research so far, it seems Atlassian is the only one that supports cloud formation stacks as part of the CI/CD pipeline. Most other tools like Drone, CodeShip etc seem to work with S3 for dr...
Has anybody built a CI CD pipeline for pyspark jobs? Is there any sample project that someone can share? Or any ideas on how to do that. We currently deploy all our code is AWS S3 and run spark-sub...
I'm trying to setup a CI/CD pipeline for a dotnet app which uses AWS Lambda and AWS CDK for infrastructure. The source is on github and we have a Jenkins pipeline which runs the tests and publishes...
I am trying to implement CI/CD pipeline in bitbucket for the project. Basically whenever new code will pushed to the branch 'XamDev', I want to deploy the web api on aws elastic beanstalk using bit...
My AWS devops concepts are still fuzzy and I could use some help. Here is what I am tyring to do - I have a CloudFormation nested stack which contains few lambdas, eventbridge, api gateways etc. I
I am currently looking to create a CI/CD workflow in AWS using AWS CodePipeline using AWS CodeBuild and AWS CloudFormation with Bitbucket Cloud as the SCM. I am looking to implement two pipelines f...
I am trying to implement Bitbucket CI/CD pipeline to deploy .net core application on AWS elastic beanstalk. The .net core application getting successfully published, but when I am trying to deploy ...

Copyright License:Reproduced under the CC 4.0 BY-SA copyright license with link to original source & disclaimer.