Implementation:Heibaiying BigData Notes HdfsUtils HDFS Operations

Knowledge Sources	Heibaiying_BigData_Notes Hadoop HDFS API
Domains	HDFS, Hadoop, File_Operations
Last Updated	2026-02-10 10:30 GMT

Overview

Concrete utility class for performing common HDFS file system operations (mkdir, read, write, copy, list, delete) via the Hadoop Java API.

Description

HdfsUtils is a utility class that wraps the Hadoop FileSystem API to provide simplified static and instance methods for interacting with HDFS. It initializes a singleton FileSystem instance in a static block using a hardcoded HDFS URI and user, then delegates each operation (directory creation, file reading, file writing, renaming, local-to-HDFS copy, HDFS-to-local copy, file listing, block location queries, and deletion) to the corresponding FileSystem method.

Usage

Use this class when you need to perform programmatic HDFS file operations from a Java application, such as creating directories, uploading files, downloading files, reading file content, or listing directory entries. It serves as a reference implementation for wrapping the Hadoop FileSystem API into a reusable utility.

Code Reference

Source Location

• Repository: Heibaiying_BigData_Notes
• File: code/Hadoop/hdfs-java-api/src/main/java/com/heibaiying/utils/HdfsUtils.java
• Lines: 1-180

Signature

public class HdfsUtils {

    // Static initialization of FileSystem
    private static final String HDFS_PATH = "hdfs://192.168.0.107:8020";
    private static final String HDFS_USER = "root";
    private static FileSystem fileSystem;

    public static FileSystem getFileSystem()
    public static boolean mkdir(String path) throws Exception
    public static String text(String path, String encode) throws Exception
    public void createAndWrite(String path, String context) throws Exception
    public boolean rename(String oldPath, String newPath) throws Exception
    public void copyFromLocalFile(String localPath, String hdfsPath) throws Exception
    public void copyToLocalFile(String hdfsPath, String localPath) throws Exception
    public FileStatus[] listFiles(String path) throws Exception
    public RemoteIterator<LocatedFileStatus> listFilesRecursive(String path, boolean recursive) throws Exception
    public BlockLocation[] getFileBlockLocations(String path) throws Exception
    public boolean delete(String path) throws Exception
}

Import

import com.heibaiying.utils.HdfsUtils;
// Requires Hadoop dependencies:
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.*;

I/O Contract

Inputs

Name	Type	Required	Description
path	String	Yes	HDFS path for the target file or directory
encode	String	No	Character encoding for reading file content (defaults to "utf-8")
context	String	Yes (createAndWrite)	Content string to write to the new file
oldPath	String	Yes (rename)	Source file path for rename operation
newPath	String	Yes (rename)	Destination file path for rename operation
localPath	String	Yes (copy ops)	Local filesystem path for copy to/from HDFS
hdfsPath	String	Yes (copy ops)	HDFS path for copy to/from operations
recursive	boolean	Yes (listFilesRecursive)	Whether to list files recursively

Outputs

Name	Type	Description
mkdir returns	boolean	True if directory was created successfully
text returns	String	File content as a string
rename returns	boolean	True if rename succeeded
listFiles returns	FileStatus[]	Array of file/directory status objects
listFilesRecursive returns	RemoteIterator<LocatedFileStatus>	Iterator over file statuses with block locations
getFileBlockLocations returns	BlockLocation[]	Array of block location information
delete returns	boolean	True if deletion succeeded

Usage Examples

Basic HDFS Operations

import com.heibaiying.utils.HdfsUtils;
import org.apache.hadoop.fs.FileStatus;

// 1. Create a directory on HDFS
boolean created = HdfsUtils.mkdir("/user/heibaiying/test");

// 2. Read file content from HDFS
String content = HdfsUtils.text("/user/heibaiying/input/wordcount.txt", "utf-8");
System.out.println(content);

// 3. Create a file and write content
HdfsUtils hdfsUtils = new HdfsUtils();
hdfsUtils.createAndWrite("/user/heibaiying/test/hello.txt", "Hello HDFS!");

// 4. Copy a local file to HDFS
hdfsUtils.copyFromLocalFile("/home/user/data.txt", "/user/heibaiying/test/data.txt");

// 5. List files in a directory
FileStatus[] statuses = hdfsUtils.listFiles("/user/heibaiying/test");
for (FileStatus status : statuses) {
    System.out.println(status.getPath().getName() + " - " + status.getLen() + " bytes");
}

// 6. Delete a file
boolean deleted = hdfsUtils.delete("/user/heibaiying/test/hello.txt");

Related Pages

• Environment:Heibaiying_BigData_Notes_Java_8_Maven_Environment
• Environment:Heibaiying_BigData_Notes_Hadoop_CDH_Environment