Sane way to find AWS Glue triggers

2022-11-10T04:01:29

My account has hundreds of Glue triggers. I need to find one by name, but the Web UI is terribly - the search only seems to look for exact matches (I need to do substring match) and makes you flip pages for every 100 or so. There is also no way to jump to a specific page, only next/prev šŸ™„

I tried using the aws glue list-triggers. But it turns out this is limited to 200 at a time, and then you have to paginate.

However, it turns out there is --query parameter. So I thought I could add --query 'TriggerNames[?contains(@, `foo`)]' it would return only triggers containing foo, which is certainly less than 200. However, it turns out that this filters the response, so it only filters down the one page. I really have no idea why you would support all the complexity of JMESPath for something that returns a list of at most 200 strings, when I could just grep it myself, so maybe I'm missing something. But sounds like --query won't help either.

I could obviously write a script with boto that paginates through the full list, applies the search query, etc. Thereby senselessly reinventing multiple wheels. Is this really the best I can do, when searching for a Glue trigger containing a string? Or is there a better way?

Copyright Licenseļ¼š
Author:怌Dommondke怍,Reproduced under the CC 4.0 BY-SA copyright license with link to original source & disclaimer.
Link toļ¼šhttps://stackoverflow.com/questions/74380729/sane-way-to-find-aws-glue-triggers

About ā€œSane way to find AWS Glue triggersā€ questions

My account has hundreds of Glue triggers. I need to find one by name, but the Web UI is terribly - the search only seems to look for exact matches (I need to do substring match) and makes you flip ...
I have a table which contains few schedules for various jobs. I want to process the records and create Triggers via AWS Glue API. http://docs.aws.amazon.com/glue/latest/dg/aws-glue-api.html The...
I'm trying to figure out how to orchestrate a job that has upstream dependencies on crawlers as well as glue jobs. Currently, AWS Glue Job Triggers support completion of other jobs, but not crawle...
I am creating a cloudwatch event rule which triggers when there is a "Glue job state change". All works good but the payload from cloud watch event doesnt seem to have the custom parameters I have ...
Currently I'm using an AWS Glue job to load data into RedShift, but after that load I need to run some data cleansing tasks probably using an AWS Lambda function. Is there any way to trigger a Lambda
I am using AWS Glue ETL scripts and triggers to run a number of jobs on data in s3. I have written a total of four jobs that will take specific parameters based on the data we want to run the jobs
I have created an AWS Glue Trigger as part of the AWS Glue Workflow that runs on a periodic basis. I have successfully set the periodic schedule via the trigger with no problems, but now I need to ...
We know that, the procedure of writing from pyspark script (aws glue job) to AWS data catalog is to write in s3 bucket (eg.csv) use a crawler and schedule it. Is there any other way of writing to aws
I have created Stored procedures on Redshift and need to orchestrate it. The SP contains the DML statements for SCD creation and is limited to Redshift. Is there a way on AWS to run the SP on Redsh...
I have been searching for an example of how to set up Cloudformation for a glue workflow which includes triggers, jobs, and crawlers, but I haven't been able to find much information on it. This ...

Copyright Licenseļ¼šReproduced under the CC 4.0 BY-SA copyright license with link to original source & disclaimer.