Serverless ETL and Analytics with AWS Glue PDF Download

Author: Vishal Pathak
Publisher: Packt Publishing Ltd
ISBN: 1800562551
Category : Computers
Languages : en
Pages : 435

View

Book Description
Build efficient data lakes that can scale to virtually unlimited size using AWS Glue Key Features Book DescriptionOrganizations these days have gravitated toward services such as AWS Glue that undertake undifferentiated heavy lifting and provide serverless Spark, enabling you to create and manage data lakes in a serverless fashion. This guide shows you how AWS Glue can be used to solve real-world problems along with helping you learn about data processing, data integration, and building data lakes. Beginning with AWS Glue basics, this book teaches you how to perform various aspects of data analysis such as ad hoc queries, data visualization, and real-time analysis using this service. It also provides a walk-through of CI/CD for AWS Glue and how to shift left on quality using automated regression tests. You’ll find out how data security aspects such as access control, encryption, auditing, and networking are implemented, as well as getting to grips with useful techniques such as picking the right file format, compression, partitioning, and bucketing. As you advance, you’ll discover AWS Glue features such as crawlers, Lake Formation, governed tables, lineage, DataBrew, Glue Studio, and custom connectors. The concluding chapters help you to understand various performance tuning, troubleshooting, and monitoring options. By the end of this AWS book, you’ll be able to create, manage, troubleshoot, and deploy ETL pipelines using AWS Glue.What you will learn Apply various AWS Glue features to manage and create data lakes Use Glue DataBrew and Glue Studio for data preparation Optimize data layout in cloud storage to accelerate analytics workloads Manage metadata including database, table, and schema definitions Secure your data during access control, encryption, auditing, and networking Monitor AWS Glue jobs to detect delays and loss of data Integrate Spark ML and SageMaker with AWS Glue to create machine learning models Who this book is for ETL developers, data engineers, and data analysts

Author: Vishal Pathak
Publisher: Packt Publishing Ltd
ISBN: 1800562551
Category : Computers
Languages : en
Pages : 435

View

Book Description
Build efficient data lakes that can scale to virtually unlimited size using AWS Glue Key Features Book DescriptionOrganizations these days have gravitated toward services such as AWS Glue that undertake undifferentiated heavy lifting and provide serverless Spark, enabling you to create and manage data lakes in a serverless fashion. This guide shows you how AWS Glue can be used to solve real-world problems along with helping you learn about data processing, data integration, and building data lakes. Beginning with AWS Glue basics, this book teaches you how to perform various aspects of data analysis such as ad hoc queries, data visualization, and real-time analysis using this service. It also provides a walk-through of CI/CD for AWS Glue and how to shift left on quality using automated regression tests. You’ll find out how data security aspects such as access control, encryption, auditing, and networking are implemented, as well as getting to grips with useful techniques such as picking the right file format, compression, partitioning, and bucketing. As you advance, you’ll discover AWS Glue features such as crawlers, Lake Formation, governed tables, lineage, DataBrew, Glue Studio, and custom connectors. The concluding chapters help you to understand various performance tuning, troubleshooting, and monitoring options. By the end of this AWS book, you’ll be able to create, manage, troubleshoot, and deploy ETL pipelines using AWS Glue.What you will learn Apply various AWS Glue features to manage and create data lakes Use Glue DataBrew and Glue Studio for data preparation Optimize data layout in cloud storage to accelerate analytics workloads Manage metadata including database, table, and schema definitions Secure your data during access control, encryption, auditing, and networking Monitor AWS Glue jobs to detect delays and loss of data Integrate Spark ML and SageMaker with AWS Glue to create machine learning models Who this book is for ETL developers, data engineers, and data analysts

Author: Anthony Virtuoso
Publisher: Packt Publishing Ltd
ISBN: 1800567863
Category : Computers
Languages : en
Pages : 438

View

Book Description
Get more from your data with Amazon Athena's ease-of-use, interactive performance, and pay-per-query pricing Key FeaturesExplore the promising capabilities of Amazon Athena and Athena's Query Federation SDKUse Athena to prepare data for common machine learning activitiesCover best practices for setting up connectivity between your application and Athena and security considerationsBook Description Amazon Athena is an interactive query service that makes it easy to analyze data in Amazon S3 using SQL, without needing to manage any infrastructure. This book begins with an overview of the serverless analytics experience offered by Athena and teaches you how to build and tune an S3 Data Lake using Athena, including how to structure your tables using open-source file formats like Parquet. You'll learn how to build, secure, and connect to a data lake with Athena and Lake Formation. Next, you'll cover key tasks such as ad hoc data analysis, working with ETL pipelines, monitoring and alerting KPI breaches using CloudWatch Metrics, running customizable connectors with AWS Lambda, and more. Moving on, you'll work through easy integrations, troubleshooting and tuning common Athena issues, and the most common reasons for query failure. You will also review tips to help diagnose and correct failing queries in your pursuit of operational excellence. Finally, you'll explore advanced concepts such as Athena Query Federation and Athena ML to generate powerful insights without needing to touch a single server. By the end of this book, you'll be able to build and use a data lake with Amazon Athena to add data-driven features to your app and perform the kind of ad hoc data analysis that often precedes many of today's ML modeling exercises. What you will learnSecure and manage the cost of querying your dataUse Athena ML and User Defined Functions (UDFs) to add advanced features to your reportsWrite your own Athena Connector to integrate with a custom data sourceDiscover your datasets on S3 using AWS Glue CrawlersIntegrate Amazon Athena into your applicationsSetup Identity and Access Management (IAM) policies to limit access to tables and databases in Glue Data CatalogAdd an Amazon SageMaker Notebook to your Athena queriesGet to grips with using Athena for ETL pipelinesWho this book is for Business intelligence (BI) analysts, application developers, and system administrators who are looking to generate insights from an ever-growing sea of data while controlling costs and limiting operational burden, will find this book helpful. Basic SQL knowledge is expected to make the most out of this book.

Author: Anthony Virtuoso
Publisher: Packt Publishing Ltd
ISBN: 1800567863
Category : Computers
Languages : en
Pages : 438

View

Book Description
Get more from your data with Amazon Athena's ease-of-use, interactive performance, and pay-per-query pricing Key FeaturesExplore the promising capabilities of Amazon Athena and Athena's Query Federation SDKUse Athena to prepare data for common machine learning activitiesCover best practices for setting up connectivity between your application and Athena and security considerationsBook Description Amazon Athena is an interactive query service that makes it easy to analyze data in Amazon S3 using SQL, without needing to manage any infrastructure. This book begins with an overview of the serverless analytics experience offered by Athena and teaches you how to build and tune an S3 Data Lake using Athena, including how to structure your tables using open-source file formats like Parquet. You'll learn how to build, secure, and connect to a data lake with Athena and Lake Formation. Next, you'll cover key tasks such as ad hoc data analysis, working with ETL pipelines, monitoring and alerting KPI breaches using CloudWatch Metrics, running customizable connectors with AWS Lambda, and more. Moving on, you'll work through easy integrations, troubleshooting and tuning common Athena issues, and the most common reasons for query failure. You will also review tips to help diagnose and correct failing queries in your pursuit of operational excellence. Finally, you'll explore advanced concepts such as Athena Query Federation and Athena ML to generate powerful insights without needing to touch a single server. By the end of this book, you'll be able to build and use a data lake with Amazon Athena to add data-driven features to your app and perform the kind of ad hoc data analysis that often precedes many of today's ML modeling exercises. What you will learnSecure and manage the cost of querying your dataUse Athena ML and User Defined Functions (UDFs) to add advanced features to your reportsWrite your own Athena Connector to integrate with a custom data sourceDiscover your datasets on S3 using AWS Glue CrawlersIntegrate Amazon Athena into your applicationsSetup Identity and Access Management (IAM) policies to limit access to tables and databases in Glue Data CatalogAdd an Amazon SageMaker Notebook to your Athena queriesGet to grips with using Athena for ETL pipelinesWho this book is for Business intelligence (BI) analysts, application developers, and system administrators who are looking to generate insights from an ever-growing sea of data while controlling costs and limiting operational burden, will find this book helpful. Basic SQL knowledge is expected to make the most out of this book.

Author: Joe Minichino
Publisher: John Wiley & Sons
ISBN: 1119909252
Category : Computers
Languages : en
Pages : 426

View

Book Description
A comprehensive and accessible roadmap to performing data analytics in the AWS cloud In Data Analytics in the AWS Cloud: Building a Data Platform for BI and Predictive Analytics on AWS, accomplished software engineer and data architect Joe Minichino delivers an expert blueprint to storing, processing, analyzing data on the Amazon Web Services cloud platform. In the book, you’ll explore every relevant aspect of data analytics—from data engineering to analysis, business intelligence, DevOps, and MLOps—as you discover how to integrate machine learning predictions with analytics engines and visualization tools. You’ll also find: Real-world use cases of AWS architectures that demystify the applications of data analytics Accessible introductions to data acquisition, importation, storage, visualization, and reporting Expert insights into serverless data engineering and how to use it to reduce overhead and costs, improve stability, and simplify maintenance A can't-miss for data architects, analysts, engineers and technical professionals, Data Analytics in the AWS Cloud will also earn a place on the bookshelves of business leaders seeking a better understanding of data analytics on the AWS cloud platform.

Author: Brij Kishore Pandey
Publisher: Packt Publishing Ltd
ISBN: 1804615536
Category : Computers
Languages : en
Pages : 246

View

Book Description
Develop production-ready ETL pipelines by leveraging Python libraries and deploying them for suitable use cases Key Features Understand how to set up a Python virtual environment with PyCharm Learn functional and object-oriented approaches to create ETL pipelines Create robust CI/CD processes for ETL pipelines Purchase of the print or Kindle book includes a free PDF eBook Book DescriptionModern extract, transform, and load (ETL) pipelines for data engineering have favored the Python language for its broad range of uses and a large assortment of tools, applications, and open source components. With its simplicity and extensive library support, Python has emerged as the undisputed choice for data processing. In this book, you’ll walk through the end-to-end process of ETL data pipeline development, starting with an introduction to the fundamentals of data pipelines and establishing a Python development environment to create pipelines. Once you've explored the ETL pipeline design principles and ET development process, you'll be equipped to design custom ETL pipelines. Next, you'll get to grips with the steps in the ETL process, which involves extracting valuable data; performing transformations, through cleaning, manipulation, and ensuring data integrity; and ultimately loading the processed data into storage systems. You’ll also review several ETL modules in Python, comparing their pros and cons when building data pipelines and leveraging cloud tools, such as AWS, to create scalable data pipelines. Lastly, you’ll learn about the concept of test-driven development for ETL pipelines to ensure safe deployments. By the end of this book, you’ll have worked on several hands-on examples to create high-performance ETL pipelines to develop robust, scalable, and resilient environments using Python.What you will learn Explore the available libraries and tools to create ETL pipelines using Python Write clean and resilient ETL code in Python that can be extended and easily scaled Understand the best practices and design principles for creating ETL pipelines Orchestrate the ETL process and scale the ETL pipeline effectively Discover tools and services available in AWS for ETL pipelines Understand different testing strategies and implement them with the ETL process Who this book is for If you are a data engineer or software professional looking to create enterprise-level ETL pipelines using Python, this book is for you. Fundamental knowledge of Python is a prerequisite.

Author: Documentation Team
Publisher:
ISBN: 9789888407699
Category : Computers
Languages : en
Pages : 432

View

Book Description
AWS Glue is a fully managed ETL (extract, transform, and load) service that makes it simple and cost-effective to categorize your data, clean it, enrich it, and move it reliably between various data stores. AWS Glue consists of a central metadata repository known as the AWS Glue Data Catalog, an ETL engine that automatically generates Python or Scala code, and a flexible scheduler that handles dependency resolution, job monitoring, and retries. AWS Glue is serverless, so there's no infrastructure to set up or manage. Use the AWS Glue console to discover data, transform it, and make it available for search and querying. The console calls the underlying services to orchestrate the work required to transform your data. You can also use the AWS Glue API operations to interface with AWS Glue services. Edit, debug, and test your Python or Scala Apache Spark ETL code using a familiar development environment.

Author: Scott Bateman
Publisher: Packt Publishing Ltd
ISBN: 1804610577
Category : Computers
Languages : en
Pages : 276

View

Book Description
Build an end-to-end geospatial data lake in AWS using popular AWS services such as RDS, Redshift, DynamoDB, and Athena to manage geodata Purchase of the print or Kindle book includes a free PDF eBook. Key Features Explore the architecture and different use cases to build and manage geospatial data lakes in AWS Discover how to leverage AWS purpose-built databases to store and analyze geospatial data Learn how to recognize which anti-patterns to avoid when managing geospatial data in the cloud Book DescriptionManaging geospatial data and building location-based applications in the cloud can be a daunting task. This comprehensive guide helps you overcome this challenge by presenting the concept of working with geospatial data in the cloud in an easy-to-understand way, along with teaching you how to design and build data lake architecture in AWS for geospatial data. You’ll begin by exploring the use of AWS databases like Redshift and Aurora PostgreSQL for storing and analyzing geospatial data. Next, you’ll leverage services such as DynamoDB and Athena, which offer powerful built-in geospatial functions for indexing and querying geospatial data. The book is filled with practical examples to illustrate the benefits of managing geospatial data in the cloud. As you advance, you’ll discover how to analyze and visualize data using Python and R, and utilize QuickSight to share derived insights. The concluding chapters explore the integration of commonly used platforms like Open Data on AWS, OpenStreetMap, and ArcGIS with AWS to enable you to optimize efficiency and provide a supportive community for continuous learning. By the end of this book, you’ll have the necessary tools and expertise to build and manage your own geospatial data lake on AWS, along with the knowledge needed to tackle geospatial data management challenges and make the most of AWS services.What you will learn Discover how to optimize the cloud to store your geospatial data Explore management strategies for your data repository using AWS Single Sign-On and IAM Create effective SQL queries against your geospatial data using Athena Validate postal addresses using Amazon Location services Process structured and unstructured geospatial data efficiently using R Use Amazon SageMaker to enable machine learning features in your application Explore the free and subscription satellite imagery data available for use in your GIS Who this book is forIf you understand the importance of accurate coordinates, but not necessarily the cloud, then this book is for you. This book is best suited for GIS developers, GIS analysts, data analysts, and data scientists looking to enhance their solutions with geospatial data for cloud-centric applications. A basic understanding of geographic concepts is suggested, but no experience with the cloud is necessary for understanding the concepts in this book.

Author: Asif Abbasi
Publisher: John Wiley & Sons
ISBN: 1119819458
Category : Computers
Languages : en
Pages : 416

View

Book Description
Virtual, hands-on learning labs allow you to apply your technical skills in realistic environments. So Sybex has bundled AWS labs from XtremeLabs with our popular AWS Certified Data Analytics Study Guide to give you the same experience working in these labs as you prepare for the Certified Data Analytics Exam that you would face in a real-life application. These labs in addition to the book are a proven way to prepare for the certification and for work as an AWS Data Analyst. AWS Certified Data Analytics Study Guide: Specialty (DAS-C01) Exam is intended for individuals who perform in a data analytics-focused role. This UPDATED exam validates an examinee's comprehensive understanding of using AWS services to design, build, secure, and maintain analytics solutions that provide insight from data. It assesses an examinee's ability to define AWS data analytics services and understand how they integrate with each other; and explain how AWS data analytics services fit in the data lifecycle of collection, storage, processing, and visualization. The book focuses on the following domains: • Collection • Storage and Data Management • Processing • Analysis and Visualization • Data Security This is your opportunity to take the next step in your career by expanding and validating your skills on the AWS cloud. AWS is the frontrunner in cloud computing products and services, and the AWS Certified Data Analytics Study Guide: Specialty exam will get you fully prepared through expert content, and real-world knowledge, key exam essentials, chapter review questions, and much more. Written by an AWS subject-matter expert, this study guide covers exam concepts, and provides key review on exam topics. Readers will also have access to Sybex's superior online interactive learning environment and test bank, including chapter tests, practice exams, a glossary of key terms, and electronic flashcards. And included with this version of the book, XtremeLabs virtual labs that run from your browser. The registration code is included with the book and gives you 6 months of unlimited access to XtremeLabs AWS Certified Data Analytics Labs with 3 unique lab modules based on the book.

Author: Mohit Gupta
Publisher: Packt Publishing Ltd
ISBN: 1789802237
Category : Computers
Languages : en
Pages : 226

View

Book Description
Serverless Architectures with AWS teaches you how to build serverless applications on AWS—applications that do not require the developer to provision, scale, or manage any servers. Using an event-driven approach and AWS Lambda as the primary service, the book explains the many benefits of serverless architectures. By the end of the book, you ...