HDFS Architecture and Programming

HDFS Architecture and Programming

This course is part of Big Data Processing Using Hadoop Specialization

Instructor: Karthik Shyamsunder

What you'll learn

Understand HDFS architecture, components, and how it ensures scalability and availability for big data processing.
Learn to configure Hadoop for Java programming and perform file CRUD operations using HDFS APIs.
Master advanced HDFS programming concepts like compression, serialization, and working with specialized file structures like Sequence and Map files.

Details to know

Shareable certificate

Add to your LinkedIn profile

Assessments

9 assignments

Taught in English

See how employees at top companies are mastering in-demand skills

Learn more about Coursera for Business

Build your subject-matter expertise

This course is part of the Big Data Processing Using Hadoop Specialization

When you enroll in this course, you'll also be enrolled in this Specialization.

Learn new concepts from industry experts
Gain a foundational understanding of a subject or tool
Develop job-relevant skills with hands-on projects
Earn a shareable career certificate

Earn a career certificate

Add this credential to your LinkedIn profile, resume, or CV

Share it on social media and in your performance review

There are 4 modules in this course

The course “HDFS Architecture and Programming” offers a comprehensive understanding of the Hadoop Distributed File System (HDFS) architecture, components, and advanced programming techniques. You will gain practical experience in setting up and configuring Hadoop for Java development, while mastering key concepts such as file and directory CRUD operations, data compression, and serialization. By the end of the course, you will be proficient in using HDFS to handle large-scale data processing, enabling you to build scalable, high-availability solutions.

What sets this course apart is its hands-on approach, where you will work directly with HDFS, writing client programs and applying advanced techniques such as using Sequence and Map Files for specialized data storage. Whether you're new to Hadoop or looking to refine your existing skills, this course equips you with the tools and knowledge to become proficient in HDFS programming, making you a valuable asset in the field of Big Data.

This course provides a comprehensive understanding of Hadoop Distributed File System (HDFS) architecture and its key components. Students will gain hands-on experience with HDFS, learning how to set up Java programming environments and configure Hadoop. The course covers essential topics such as the HDFS programming model, file and directory CRUD operations, and compression techniques. You will also explore serialization, deserialization, and specialized file structures like Sequence and Map Files. By the end of the course, You will be equipped to leverage HDFS for scalable, highly available big data solutions.

What's included

2 readings

In this module, we will cover the working model and architecture behind Hadoop Distributed File System (HDFS) 1.0 and the capabilities and deficiencies of HDFS 1.0 architecture.

What's included

6 videos4 readings3 assignments

6 videosTotal 103 minutes

Introduction - HDFS Architecture1 minutePreview module
HDFS Architecture32 minutes
Hands-on HDFS Architecture33 minutes
Block Size, Replication, and Rack Awareness17 minutes
HDFS Architectural Capabilities17 minutes
Summary - HDFS Architecture0 minutes

4 readingsTotal 85 minutes

HDFS Architecture15 minutes
Block Size, Replication, and Rack Awareness15 minutes
HDFS Architectural Capabilities15 minutes
Self-Reflective Reading: Comparing Hadoop's HDFS and MapR's FileSystem40 minutes

3 assignmentsTotal 90 minutes

HDFS Architecture60 minutes
Mastering HDFS: Architecture Overview and Hands-On Experience15 minutes
Deep Dive into HDFS: Block Size, Replication, and Architectural Capabilities15 minutes

In this module, we will cover HDFS programming concepts, HDFS API, and steps to write an HDFS client program for CRUD (Create, Read, Update and Delete) on files.

What's included

6 videos5 readings3 assignments

6 videosTotal 106 minutes

Introduction - HDFS Programming Basics2 minutesPreview module
HDFS C API Overview11 minutes
Hadoop Configuration API and Dev Env Setup37 minutes
HDFS Java API Overview19 minutes
File CRUD33 minutes
Summary - HDFS Programming Basics2 minutes

5 readingsTotal 105 minutes

HDFS C API Overview15 minutes
Hadoop Configuration API and Dev Env Setup20 minutes
HDFS Java API Overview15 minutes
File CRUD15 minutes
Self-Reflective Reading: Design Reflection: Error Handling in File Deletion40 minutes

3 assignmentsTotal 90 minutes

HDFS Programming Basics60 minutes
Getting Started with HDFS Programming: Basics, C API, and Configuration15 minutes
HDFS Programming Essentials: Java API, File CRUD Operations, and Summary15 minutes

In this module, we will cover HDFS advanced programming concepts, such as CRUD on directories, compression, serialization and deserialization, and file-based data structures like sequence files.

What's included

6 videos5 readings3 assignments

6 videosTotal 76 minutes

Introduction - HDFS Advanced Programming2 minutesPreview module
Directory CRUD13 minutes
File Compression Decompression19 minutes
Serialization Deserialization17 minutes
File-Based Data Structures21 minutes
Summary - HDFS Advanced Programming2 minutes

5 readingsTotal 80 minutes

Directory CRUD15 minutes
File Compression Decompression10 minutes
Serialization Deserialization10 minutes
File-Based Data Structures15 minutes
Self-Reflective Reading: Hadoop's Serialization Decision30 minutes

3 assignmentsTotal 90 minutes

HDFS Programming Advanced60 minutes
Advanced HDFS Programming: Directory CRUD and File Compression/Decompression15 minutes
Advanced HDFS Programming: Serialization, File-Based Data Structures, and Summary15 minutes