AWS Glue Job logs still going to output and error logs

2022-09-24T03:18:56

I have configured a AWS Glue Job in which I have enabled continuous-logging and then defined the continuous-logging in the job-parameters.

enter image description here

Parameters:

  1. --enable-continuous-cloudwatch-log - true
  2. --enable-continuous-log-filter - false
  3. --continuous-log-logStreamPrefix - test
  4. --continuous-log-logGroupName or --continuous-log-logGroup - custom_log_group

But still my Glue Job log group points to "/aws-glue/jobs" and the links under Cloudwatch logs enter image description here

  1. 'All Logs' opens default log group - /aws-glue/jobs/logs-v2. Though the l,ink opens wrong location but its actually writing to my custom_log_group. But it only contains glue context logs. Not my java library logs which is being referenced in glue job.
  2. 'Output Logs' goes to - /aws-glue/jobs/output
  3. 'Error Logs' goes to - /aws-glue/jobs/error - This shows all my java library logs.

I want all these logs to go my custom_log_group. What am I missing here.

Copyright License:
Author:「Niddhi Garg」,Reproduced under the CC 4.0 BY-SA copyright license with link to original source & disclaimer.
Link to:https://stackoverflow.com/questions/73832090/aws-glue-job-logs-still-going-to-output-and-error-logs

About “AWS Glue Job logs still going to output and error logs” questions

I have configured a AWS Glue Job in which I have enabled continuous-logging and then defined the continuous-logging in the job-parameters. Parameters: --enable-continuous-cloudwatch-log - true --...
Need to pass the Glue job logs to Custom Log group (aws-glue/schema) as by default the Glue job logs go into /aws-glue/jobs/output. Found this on aws documentation '--continuous-log-logGroup': '
The AWS CLI is very good for managing AWS Glue jobs. But if a job fails, I may not see anything more useful than something like: "JobRunState": "FAILED", ...
So my problem is sometimes my Glue job takes a lot of time to run or even fails all together because of some inexplicable issues. Is there a way to get messages printed out to the output or error l...
I am running glueetl (Glue version 2.0) job using Python with the below configuration for logging. I have continuous logging enabled. I get INFO entries from the driver present in /aws-glue/jobs/ou...
I am trying to use AWS Glue to run an ETL job that fetches data from Redshift to S3. When I run a crawler it successfully connects to Redshift and fetches schema information. Relevant logs are cre...
I have some "Python Shell" type Glue jobs and I want to send the job logs to a custom CloudWatch log group instead of the default log group. I am able to achieve this for "Spark" type glue jobs by
I've set up an AWS Glue connection to an RDS database (in the same account and region). When doing the test connection I get the following error rds-prod-snapshot test connection failed. For more
I am new to AWS GLUE and just want to solve a particular problem. Currently i have only Glue service available only and no EC2 node no lambda. I am trying to run a AWS spark glue job from Aws python
I have inherited a python script that I'm trying to log in Glue. Originally it had prints, but they were only sent once job finished, but it was not possible to see the status of the execution in r...

Copyright License:Reproduced under the CC 4.0 BY-SA copyright license with link to original source & disclaimer.