Scan a directory for files and load it in memory efficiently

I am working on a little project where I need to scan all the files present in a folder on the disk and load it in memory. Below is my code which does that exactly and works fine.

Here are the steps:

• On the disk there is already a default Records folder which has all the default config files present. This is to fallback if in case something gets wrong or the loadDefaultFlag is enabled.
• There are also new config files present as a tar.gz file (max 100 MB size) in a remote url location which I need to download and store it on disk in _secondaryLocation if loadDefaultFlag is disabled.
• Depending on whether loadDefaultFlag is present or not – we will either load default local files already present on the disk or load it from _secondaryLocation (after downloading it from the remote url location).
• During server startup call goes to my RecordManager constructor where it checks whether loadDefaultFlag is enabled or not and basis on that it loads the file either from Records folder as mentioned in point 1 or download new configs from url as mentioned in point 2 and then load it in memory.

I get the json value of configKey from IConfiguration object in my constructor which has all the details whether to use default configs or download files from a remote url and store it on disk. Sample content of configKey object is –

{"loadDefaultFlag": "false", "remoteFileName":"data-1234.tgz", ...}

Basis on the above json value I figure out what to do as outlined in above series of points.

Below is my code:

using System;
using System.Collections.Generic;
using System.IO;
using System.Threading;
using System.Threading.Tasks;
using System.Net.Http;
using ICSharpCode.SharpZipLib.GZip;
using ICSharpCode.SharpZipLib.Tar;
using Polly;
using Microsoft.Extensions.Configuration;
using Newtonsoft.Json;

public class RecordManager
{
    private readonly string _remoteUrl = "remote-url-from-where-to-download-new-configs";
    private readonly string _secondaryLocation = "SecondaryConfigs";
    private readonly string _localPath = null;
    private readonly IConfiguration _configuration;

    private static HttpClient _httpClient = new HttpClient()
    {
        Timeout = TimeSpan.FromSeconds(3)
    };

    public RecordManager(IConfiguration configuration, string localPath = "Records")
    {
        _localPath = localPath ?? throw new ArgumentNullException(nameof(localPath));
        _configuration = configuration;
        ChangeToken.OnChange(configuration.GetReloadToken, _ => ConfigChanged(), new object());

        string jsonValue = configuration["configKey"];
        if (!string.IsNullOrWhiteSpace(jsonValue))
        {
            RecordPojo dcc = JsonConvert.DeserializeObject<RecordPojo>(jsonValue);
            Boolean.TryParse((string)dcc.loadDefaultFlag, out bool loadDefaultFlag);
            string remoteFileName = dcc.remoteFileName;
            if (!loadDefaultFlag && !string.IsNullOrWhiteSpace(remoteFileName))
            {
                // get all the configs from the url and load it in memory
                if (!LoadAllConfigsInMemory(_url, remoteFileName, _secondaryLocation).Result) throw new ArgumentNullException(nameof(_records));
            }
            else
            {
                var recordsList = LoadDefaultConfigsInMemory() ?? throw new ArgumentNullException("recordsList");
                if (recordsList.Count == 0) throw new ArgumentNullException("recordsList");

                if (!UpdateRecords(recordsList)) throw new ArgumentNullException(nameof(_records));
            }
        }
        else
        {
            var recordsList = LoadDefaultConfigsInMemory() ?? throw new ArgumentNullException("recordsList");
            if (recordsList.Count == 0) throw new ArgumentNullException("recordsList");

            if (!UpdateRecords(recordsList)) throw new ArgumentNullException(nameof(_records));
        }
    }

    // This method will load all the configs downloaded from the url in memory
    private async Task<bool> LoadAllConfigsInMemory(string url, string fileName, string directory)
    {
        IList<RecordHolder> recordsList = new List<RecordHolder>();
        try
        {
            recordsList = GetRecords(url, fileName, directory);
            if (recordsList == null || recordsList.Count == 0)
            {
                throw new ArgumentException("No config records loaded from remote service.");
            }
            return UpdateRecords(recordsList);
        }
        catch (Exception ex)
        {
            // log error
        }
        // falling back to load default configs
        recordsList = LoadDefaultConfigsInMemory();

        return UpdateRecords(recordsList);
    }

    // This will return list of all the RecordHolder by iterating on all the files.
    private IList<RecordHolder> GetRecords(string url, string fileName, string directory)
    {
        var recordsList = new List<RecordHolder>();
        var recordPaths = GetAllTheFiles(url, fileName, directory);
        for (int i = 0; i < recordPaths.Count; i++)
        {
            var configPath = recordPaths[i];
            if (File.Exists(configPath))
            {
                var fileDate = File.GetLastWriteTimeUtc(configPath);
                string fileContent = File.ReadAllText(configPath);
                var pathPieces = configPath.Split(System.IO.Path.DirectorySeparatorChar, StringSplitOptions.RemoveEmptyEntries);
                var fileName = pathPieces[pathPieces.Length - 1];
                recordsList.Add(new RecordHolder()
                {
                    Name = fileName,
                    Date = fileDate,
                    JDoc = fileContent
                });
            }
        }
        return recordsList;
    }

    // This method will return list of all the files by downloading a tar.gz file
    // from a url and then extracting contents of tar.gz into a folder.
    // Maybe this code can be simplified better - I am doing lot of boolean checks here
    // not sure if that's good.
    private IList<string> GetAllTheFiles(string url, string fileName, string directory)
    {
        IList<string> allFiles = new List<string>();
        bool isDownloadSuccessful = DownloadConfigs(url, fileName).Result;
        if (!isDownloadSuccessful)
        {
            return allFiles;
        }
        bool isExtracted = ExtractTarGz(fileName, directory);
        if (!isExtracted)
        {
            return allFiles;
        }
        return GetFiles(directory);
    }

    // This method will download a tar.gz file from a remote url and save it onto the disk
    // in a particular folder
    private async Task<bool> DownloadConfigs(string remoteUrl, string fileName)
    {
        var policyResult = await Policy
           .Handle<TaskCanceledException>()
           .WaitAndRetryAsync(retryCount: 5, sleepDurationProvider: i => TimeSpan.FromMilliseconds(500))
           .ExecuteAndCaptureAsync(async () =>
           {
               using (var httpResponse = await _httpClient.GetAsync(remoteUrl + fileName).ConfigureAwait(false))
               {
                   httpResponse.EnsureSuccessStatusCode();
                   return await httpResponse.Content.ReadAsByteArrayAsync().ConfigureAwait(false);
               }
           }).ConfigureAwait(false);

        if (policyResult.Outcome == OutcomeType.Failure || policyResult.Result == null)
            return false;
        try
        {
            // write all the content of tar.gz file onto the disk
            File.WriteAllBytes(fileName, policyResult.Result);
            return true;
        }
        catch (Exception ex)
        {
            // log error
            return false;
        }
    }

    // This method extracts contents of tar.gz file in a directory
    private bool ExtractTarGz(string fileName, string directory)
    {
        try
        {
            Stream inStream = File.OpenRead(fileName);
            Stream gzipStream = new GZipInputStream(inStream);

            TarArchive tarArchive = TarArchive.CreateInputTarArchive(gzipStream);
            tarArchive.ExtractContents(directory);
            tarArchive.Close();

            gzipStream.Close();
            inStream.Close();
        }
        catch (Exception ex)
        {
            // log error
            return false;
        }
        return true;
    }

    // This method gets list of all files in a folder matching particular suffix
    private IList<string> GetFiles(string path)
    {
        var allFiles = new List<string>();
        try
        {
            var jsonFiles = Directory.GetFiles(path, "*.json", SearchOption.AllDirectories);
            var testFiles = Directory.GetFiles(path, "*.txt", SearchOption.AllDirectories);
            allFiles.AddRange(jsonFiles);
            allFiles.AddRange(testFiles);
        }
        catch (UnauthorizedAccessException ex)
        {
            // log error
        }
        return allFiles;
    }

    // This method will load all the default local configs in memory
    // if  `loadDefaultFlag` is enabled or cannot talk to remote url location
    private IList<RecordHolder> LoadDefaultConfigsInMemory()
    {
        var configs = new List<RecordHolder>();
        var recordPaths = GetFiles(_localPath);
        for (int i = 0; i < recordPaths.Count; i++)
        {
            var configPath = recordPaths[i];
            if (File.Exists(configPath))
            {
                var fileDate = File.GetLastWriteTimeUtc(configPath);
                string fileContent = File.ReadAllText(configPath);
                var pathPieces = configPath.Split(System.IO.Path.DirectorySeparatorChar, StringSplitOptions.RemoveEmptyEntries);
                var fileName = pathPieces[pathPieces.Length - 1];
                configs.Add(new RecordHolder()
                {
                    Name = fileName,
                    Date = fileDate,
                    JDoc = fileContent
                });
            }
        }
        return configs;
    }

    private bool UpdateRecords(IList<RecordHolder> recordsHolder)
    {
        // leaving out this code as it just updates the config in memory
    }

}

Opting for a code review here. I am specifically interested in the way I have designed and implemented my code. I am sure there must be a better way to rewrite this whole class efficiently with clear design and implementation. Also there are few methods above which could be written in a better and efficient way as well.

The idea is very simple – During server startup, either load default local configs already present on the disk or load it from a secondary folder on the disk after downloading it from remote url location.