Arjoika Paul

ETL for data from API to ODS (June 2024 – Dec 2024)

• Worked with the client to understand the requirements for the platform updates.
• Worked on the development of PySpark code to fetch data via an API and store the data in the S3 raw bucket.
• Created stone-branch jobs to run the AWS Glue job that loads data from the API.
• Implemented firewall changes to allow traffic from the API to the AWS server.
• Designed Type 1 and Type 2 tables, and developed PySpark code to load data from raw files onto these tables.

Stone-branch migration (March 2024 – May 2024).

• Worked on the migration of AWS Glue jobs from Autosys to stone-branch across all environments.
• Worked on creating workflows, triggers, task monitors, etc. for AWS glue jobs on stone-branch as well as scheduling these jobs.
• Worked on resolving issues with AWS glue jobs to ensure smooth execution of stone-branch jobs.

Master data management for Reltio (April 2023 – February 2024)

• Worked on creating and optimizing inbound materialized views to retrieve data from multiple ODS tables.
• Focused on query optimization to ensure faster refresh rates.
• Used reltio to visualize the mastered data fetched through the views.
• Worked on the development and enhancement of PySpark code to load incremental and historical data from reltio to ODS tables.
• Leveraged AWS glue jobs to run the code using autosys.
• Engaged with clients to gather requirements, and collaborated with the team to implement changes and improvements at the database level.

Data modeling and data interpretation (Dec 2022 – March 2023)

• Worked on interpreting and analyzing insurance data, and preparing data models using Erwin Data Modeler.
• Worked on the automation of STTM for landing to stage, and stage to ODS, in order to minimize manual intervention and errors.
• Ensured that the framework could dynamically adapt to changes in source and target schemas, based on the transformation rules.