TransWikia.com

How to write gdal_translate output from memory directly to s3

Geographic Information Systems Asked on May 7, 2021

I’m trying to convert a .tif to .mrf format using Python gdal bindings. I’m doing the conversion in memory using /vsimem then attempting to write directly to s3. The problem is only the .mrf file gets copied to s3 but none of the auxiliary side car files.

When I write to disk it writes a few side car files like .idx, lrc, .mrf.aux.xml which are necessary. It does this without me having to explicitly tell it to output those files.

How would I go about accomplishing this in s3? How do I know what files need to be copied to s3? This works when I just have a single output file.

from osgeo import gdal
import boto3

s3 = boto3.resource('s3')
object = s3.Object('test-bucket', 'testing/output.mrf')

tif_file = r"C:new_foldertest.tif"
ds = gdal.Open(tif_file)

gdal.Translate('/vsimem/temp_file.mrf', ds, format="MRF", creationOptions=['COMPRESS=NONE', 'BLOCKSIZE=10'])


f = gdal.VSIFOpenL('/vsimem/temp_file.mrf', 'rb')
gdal.VSIFSeekL(f, 0, 2)  # seek to end
size = gdal.VSIFTellL(f)
gdal.VSIFSeekL(f, 0, 0)  # seek to beginning
data = gdal.VSIFReadL(1, size, f)
gdal.VSIFCloseL(f)

object.put(Body=data)

ds = None

One Answer

Got some help and figured it out. Changed some things around and basically need to

use gdal.ReadDir('/vsimem/temp/') to get a list of all files created and then add those files one by one to s3.

Answered by fpolig01 on May 7, 2021

Add your own answers!

Ask a Question

Get help from others!

© 2024 TransWikia.com. All rights reserved. Sites we Love: PCI Database, UKBizDB, Menu Kuliner, Sharing RPP