How to Use IAM authentication for RDS PostgreSQL with Glue ETL Jobs

Amazon RDS enables you to use AWS Identity and Access Management (IAM) to manage database access for Amazon RDS for PostgreSQL DB instances. It’s possible use the IAM authentication with Glue connections but it is not documented well, so I will demostrate how you can do it. In your Glue job, you can import boto3 library to call “generate_db_auth_token” method to generate a token and use it when connecting.

Here’s a simple Glue ETL script I wrote for testing. It connects to PostgreSQL using IAM authentication, reads data from a table and writes the output to S3:

As you can see, I use “create_dynamic_frame_from_options” to create the connection. I generate a token, and then use it as password. Please note that your Glue ETL role should have required permissions to be able to use IAM authentication. So I added an inline policy to the role I used with Glue ETL:

The resource describes one database account in one DB instance. The ARN format is as follows:

arn:aws:rds-db:region:account-id:dbuser:DbiResourceId/your-db-user-name

Please do not forget that you need to grant “rds_iam” role to your database user to use IAM authentication with PostgreSQL:

It’s also possible to use from_jdbc_conf method of DynamicFrameWriter to write data to RDS database which you connected by IAM authentication.

Although you use create_dynamic_frame_from_options and from_jdbc_conf, you may still need to create a Glue connection (even a dummy one) for your Glue ETL job to access your RDS database. So you can set up your security groups and allow Glue to connect your RDS in a secure way.

Hope it helps. If you have any questions, please write into comments, and I’ll try to answer.

Please share
  •  
  •  
  •  
  •  
  •  
  •  

AWS Big Data Specialist. Oracle Certified Professional (OCP) for EBS R12, Oracle 10g and 11g. Co-author of "Expert Oracle Enterprise Manager 12c" book published by Apress. Awarded as Oracle ACE (in 2011) and Oracle ACE Director (in 2016) for the continuous contributions to the Oracle users community. Founding member, and vice president of Turkish Oracle User Group (TROUG). Presented at various international conferences including Oracle Open World.

Leave Comment

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.