TransWikia.com

Improving efficiency of download code

Code Review Asked on December 10, 2021

I’m in the process of reviewing some old code released by an old colleague.

We have a cronjob that executes once in an hour to download some thumbnails: the paths to the thumbnails are stored in a plain array.

// $totalThumbs is usually less than 2000 (each about 3kb)
while ($i < $totalThumbs) {

  $imgName = basename($thumbs_array[$i]);

  $fgc = file_get_contents($thumbs_array[$i]);

  $currentFile = __DIR__ . "/" . $imgName;

  // if file_get_contents doesn't return false
  if ($fgc !== false) {

    // if the file is not in that directory, write the file
    if (!file_exists($currentFile)) {

      file_put_contents($currentFile, $fgc);
      clearstatcache(true, $currentFile);
    
    }
  
  }

  $i++;

  sleep(1);
}

This code works but, for example, we can’t use CURL multi_exec because of the limited resources of our server.

Is there a way to improve it (more efficient and/or more secure), considering our hardware limits? We don’t need speed, but eventually less memory consumption because the same server is at the same time busy with many other ‘jobs’.

Thanks

EDIT (for Mast): one important thing to say is the current idea is to remove this part and use a cronjob to directly store an array in a file, so that the file we are talking about has only to read that array

$dir = "https://mylocaldir";
$thumbs_array = [];

// this one returns JSONP
$raw_json = 'https://endpoint';
$json = file_get_contents($raw_json);

// JSONP removal
$json = str_replace("jsonOutreachFeed(", "", $json);
$json = substr($json, 0, -1);

$decoded_json = json_decode($json);

$itm = $decoded_json->items;

$totalThumbs = count($itm);

for ($i = 0; $i < $totalThumbs; $i++) {
  $thumbs_array[] = $itm[$i]->media->m;
}

One Answer

Since the code is only reading the contents of the file to see if there are contents in the file, an optimization for both speed and memory usage would be to use the filesize() function and not read the contents. Reading the contents of the file consumes both time and memory.

Rather than writing the contents read in, copy or move the original file to the new location using either rename() or move_uploaded_file().

Answered by pacmaninbw on December 10, 2021

Add your own answers!

Ask a Question

Get help from others!

© 2024 TransWikia.com. All rights reserved. Sites we Love: PCI Database, UKBizDB, Menu Kuliner, Sharing RPP