How to syncronize XML files and store them as .gz at destination using rsync

Question

I'm using rsync -z to backup a large amount of XML files. The source is a huge machine with thousands of GBs available on disk, but the destination is not that powerful.
What I want to do is to compress XML files during transfer and keep them compressed at the destination.
The option -z does the compression only during the transit.
Is it possible to do it using just one crontab command?
This is the command I am currently using:
*/30 * * * * rsync -rz -e 'sshpass -p "MY_PASSWORD" ssh -p 5022' --ignore-existing --quiet --delete REMOTE_USER@REMOTE_HOST:/path/to/remote/ /path/to/local/

optimus_prime · Answer

I managed to solve this with a workaround. On local host, I just compress the syncronized file and overwrite the original file with the string "0".
It prevents rsync from synchronizing the same file again and significantly reduces disk load.
exec('find /path/to/local/ -name "*.xml" -size +10k',$output);
foreach ($output as $key => $filepath) {`
        
        //gzip the xml file
        $gz = gzopen("/path/to/gz/".basename($filepath).".gz", "wb9");
        gzwrite($gz, file_get_contents($filepath));
        gzclose($gz);
        
        //overwrite xml file
        $xml = fopen($filepath,"w");
        fwrite($xml, "0");
        fclose($xml);
}

How to syncronize XML files and store them as .gz at destination using rsync

One Answer

Add your own answers!

Ask a Question