0

My goal is to check if a Hadoop path exists and if yes, then it should download some file. Now in every five minutes, I want to check if the file exists in the Hadoop path. currently, my code is checking if the file exists but not on an interval basis.

public boolean checkPathExistance(final org.apache.hadoop.fs.Path hdfsAbsolutePath)
{
    final Configuration configuration = getHdfsConfiguration();
    try
    {
        final FileSystem fileSystems = hdfsAbsolutePath.getFileSystem(configuration);
        if (fileSystems.exists(hdfsAbsolutePath))
        {
            return true;
        }
    }
    catch (final IOException e)
    {
        e.printStackTrace();
    }
    return false;
}

The criteria are this checkPathExistance method should be called in every five minutes to check if the file exists. And when it will return true, the file should be downloaded.

 public void download(final String hdfs, final Path outputPath)
{
    final org.apache.hadoop.fs.Path hdfsAbsolutePath = getHdfsFile(hdfsLocalPath).getPath();
    logger.info("path check {}", hdfsAbsolutePath.getName());
    final boolean isPathExist =  checkPathExistance(hdfsAbsolutePath);
    downloadFromHDFS(hdfsAbsolutePath, outputPath);
}

Can I please get some help here ?

Anonymous Asked question May 14, 2021