Building Batch Data Pipelines on Google Cloud

This course is part of multiple programs.

Instructor: Google Cloud Training

What you'll learn

Review different methods of data loading: EL, ELT and ETL and when to use what
Run Hadoop on Dataproc, leverage Cloud Storage, and optimize Dataproc jobs
Build your data processing pipelines using Dataflow
Manage data pipelines with Data Fusion and Cloud Composer

Skills you'll gain

Details to know

Shareable certificate

Add to your LinkedIn profile

Assessments

4 assignments

Taught in English

See how employees at top companies are mastering in-demand skills

Learn more about Coursera for Business

Build your subject-matter expertise

This course is available as part of

When you enroll in this course, you'll also be asked to select a specific program.

Learn new concepts from industry experts
Gain a foundational understanding of a subject or tool
Develop job-relevant skills with hands-on projects
Earn a shareable career certificate

Earn a career certificate

Add this credential to your LinkedIn profile, resume, or CV

Share it on social media and in your performance review

There are 6 modules in this course

Data pipelines typically fall under one of the Extract and Load (EL), Extract, Load and Transform (ELT) or Extract, Transform and Load (ETL) paradigms. This course describes which paradigm should be used and when for batch data. Furthermore, this course covers several technologies on Google Cloud for data transformation including BigQuery, executing Spark on Dataproc, pipeline graphs in Cloud Data Fusion and serverless data processing with Dataflow. Learners get hands-on experience building data pipeline components on Google Cloud using Qwiklabs.

In this module, we introduce the course and agenda

What's included

1 video

This module reviews different methods of data loading: EL, ELT and ETL and when to use what

What's included

6 videos1 assignment

This module shows how to run Hadoop on Dataproc, how to leverage Cloud Storage, and how to optimize your Dataproc jobs.

What's included

11 videos1 assignment1 app item

11 videosTotal 48 minutes

Module introduction0 minutesPreview module
The Hadoop ecosystem4 minutes
Running Hadoop on Dataproc11 minutes
Cloud Storage instead of HDFS6 minutes
Optimizing Dataproc2 minutes
Optimizing Dataproc Storage9 minutes
Optimizing Dataproc Templates and Autoscaling4 minutes
Optimizing Dataproc Monitoring3 minutes
Lab Intro: Running Apache Spark jobs on Dataproc0 minutes
Getting Started with Google Cloud and Qwiklabs4 minutes
Summary0 minutes

1 assignmentTotal 6 minutes

Executing Spark on Dataproc6 minutes

1 app itemTotal 90 minutes

Running Apache Spark jobs on Dataproc90 minutes

This module covers using Dataflow to build your data processing pipelines

What's included

13 videos1 reading1 assignment6 app items

13 videosTotal 34 minutes

Module introduction0 minutesPreview module
Introduction to Dataflow5 minutes
Why customers value Dataflow2 minutes
Building Dataflow Pipelines in code3 minutes
Key considerations with designing pipelines2 minutes
Transforming data with PTransforms3 minutes
Lab Intro: Building a Simple Dataflow Pipeline0 minutes
Aggregate with GroupByKey and Combine5 minutes
Lab Intro: MapReduce in Beam0 minutes
Side Inputs and Windows of data4 minutes
Lab Intro: Serverless Data Analysis with Dataflow: Side Inputs0 minutes
Creating and re-using Pipeline Templates3 minutes
Summary2 minutes

1 readingTotal 1 minute

Completing Labs in this course1 minute

1 assignmentTotal 4 minutes

Serverless Data Processing with Dataflow4 minutes

6 app itemsTotal 540 minutes

A Simple Dataflow Pipeline (Python)90 minutes
Serverless Data Analysis with Dataflow: A Simple Dataflow Pipeline (Java)90 minutes
MapReduce in Beam (Python)90 minutes
Serverless Data Analysis with Beam: MapReduce in Beam (Java)90 minutes
Serverless Data Analysis with Dataflow: Side Inputs (Python)90 minutes
Serverless Data Analysis with Dataflow: Side Inputs (Java)90 minutes

This module shows how to manage data pipelines with Cloud Data Fusion and Cloud Composer.

What's included

13 videos1 assignment2 app items

13 videosTotal 29 minutes

Module introduction0 minutesPreview module
Introduction to Cloud Data Fusion3 minutes
Components of Cloud Data Fusion1 minute
Cloud Data Fusion UI1 minute
Build a pipeline4 minutes
Explore data using wrangler1 minute
Lab Intro: Building and executing a pipeline graph in Cloud Data Fusion0 minutes
Orchestrate work between Google Cloud services with Cloud Composer1 minute
Apache Airflow Environment1 minute
DAGs and Operators4 minutes
Workflow scheduling5 minutes
Monitoring and Logging3 minutes
Lab Intro: An Introduction to Cloud Composer0 minutes

1 assignmentTotal 2 minutes

Manage Data Pipelines with Cloud Data Fusion and Cloud Composer2 minutes

2 app itemsTotal 240 minutes

Building and Executing a Pipeline Graph with Data Fusion150 minutes
Lab: An Introduction to Cloud Composer90 minutes

Course Summary

What's included

1 video

Instructor

Instructor ratings

4.6 (248 ratings)

Google Cloud Training

Google Cloud

1,734 Courses2,981,115 learners

Offered by

Google Cloud

Why people choose Coursera for their career

Felipe M.

Learner since 2018

"To be able to take courses at my own pace and rhythm has been an amazing experience. I can learn whenever it fits my schedule and mood."

Jennifer J.

Learner since 2020

"I directly applied the concepts and skills I learned from my courses to an exciting new project at work."

Larry W.

Learner since 2021

"When I need courses on topics that my university doesn't offer, Coursera is one of the best places to go."

Chaitanya A.

"Learning isn't just about being better at your job: it's so much more than that. Coursera allows me to learn without limits."

Learner reviews

4.5

1,697 reviews

5 stars
65.88%
4 stars
25.75%
3 stars
6.06%
2 stars
1.47%
1 star
0.82%

Showing 3 of 1697

Reviewed on May 28, 2020

A great course to help understand the various wonderful options Google Cloud has to offer to move on-premise Hadoop workload to Google Cloud Platform to leverage scalability of clusters.

Reviewed on Jun 14, 2020

Good course covering Dataproc, Dataflow, Dataprep and the labs ofcourse..great way to get introduced to batch data pipelines in GCP.

Reviewed on Sep 30, 2020

Good contents. Some lab questions do not have answers. Hope provide them to know how I understand the knowledge.

View more reviews