Change Data/Schema capture in AWS Glue

By



In today’s world, organizations are collecting an unprecedented amount of data from all kinds of different data sources, such as transactional data stores, clickstreams, log data, IoT data, and more.

In many use cases, I have found that the data teams responsible for building the data pipeline don’t have any control of the source schema, and they need to build a solution to identify changes in the source schema in order to be able to build the process or automation around it. This might include various challenges as :

  • Sending notifications of changes to the teams dependent on the source schema, building an auditing solution to log all the schema changes.
  • Building an automation or change request process to propagate the change in the source schema to downstream applications such as an ETL tool or BI dashboard.
  • Sometimes, to control the number of schema versions, you may want to delete the older version of the schema when there are no changes detected between it and the newer schema.

there many more ways and challenges to change data or schema capture.

For example, assume you’re receiving claim files from different external partners in the form of flat files, and you’ve built a solution to process claims based on these files. However, because these files were sent by external partners, you don’t have much control over the schema and data format. For example, columns such as customer_id and claim_id were changed to customerid and claimid by one partner, and another partner added new columns such as customer_age and earning and kept the rest of the columns the same. You need to identify such changes in advance so you can edit the ETL job to accommodate the changes, such as changing the column name or adding the new columns to process the claims.

To all these changes Glue is smart enough to fix it at crawler level. So while defining crawler for our source we just need to do some config changes as :

So we have various options in crawler based on your requirement. Hope you have learnt something new today. Enjoy Glueing to your data!

Comments

Post a Comment

Popular posts from this blog

Do you know over logging can cost you most?

By
  Short description To identify your CloudWatch usage charges, review your  AWS Cost and Usage Reports . To reduce and prevent unnecessary charges, review the following CloudWatch services, and implement cost-effective methods: CloudWatch custom metrics CloudWatch metric API calls CloudWatch alarms CloudWatch dashboards CloudWatch Logs CloudWatch Contributor Insights CloudWatch Synthetics CloudWatch Internet Monitor CloudWatch Container Insights CloudWatch Application Signals CloudWatch RUM CloudWatch Evidently Note:  To monitor future AWS costs, it's a best practice to  turn on billing alerts . Resolution CloudWatch custom metrics All CloudWatch detailed monitoring and custom metrics are prorated by the hour and metered when metrics are sent to CloudWatch. To reduce CloudWatch metrics charges, take the following actions: Turn off detailed monitoring for instances, Auto Scaling group launch configurations, and API gateways. Turn off monitoring for custom metrics when...
Read more

Diagonal Scaling concept

By
Image
Diagonal scaling as it relates to a database and/or cloud computing is precisely what it sounds like - combining the best of both vertical and horizontal scaling to better suit the needs of the organization. When implemented correctly, diagonal scaling enables you to handle more data and more users, ultimately enhancing efficiency and reducing bottlenecks.. In the database world, scaling typically refers to the process of adding more resources (such as servers, storage, or processing power) to handle increased data and workload demands. Common approaches to scaling databases include vertical scaling (adding more resources to a single server) and horizontal scaling (distributing the database across multiple servers). ‍ Diagonal or hybrid scaling can be a game-changer, offering a dynamic solution to the challenges related to fluctuating workloads. Unlike traditional vertical and horizontal scaling approaches, diagonal scaling provides the flexibility to adapt to changing demands seamless...
Read more

Netflix on AWS

By
Image
  Netflix is one of the world’s leading entertainment services with over 260 million members in more than 190 countries. Netflix uses AWS for nearly all its computing and storage needs, including databases, analytics, recommendation engines, video transcoding, and more—hundreds of functions that in total use more than 100,000 server instances on AWS. Key highlights of Netflix in AWS : 1)  Netflix Connects with 100 Million Customers in 190 Countries Using Amazon Simple Email Service. 2) Netflix Realizes Multi-Region Resiliency on Amazon Route 53. 3) Netflix Uses NICE DCV on AWS to Build VFX Studio in the Cloud for Artists Globally. 4) Netflix Empowers Remote Artistry with Low-Latency Workstations Using AWS Local Zones. With the new Amazon EC2 G5 instances, we can provision higher-end graphics workstations that offer up to 3x higher performance compared to workstations with EC2 G4dn instances. With G5 instances, content creators have the freedom to create more complex ...
Read more