Get metadata for cloud datafiles accessed via a signed URL

[cortadocodes]

We'll soon release the ability to access cloud datasets via a signed URL. While we can access the dataset metadata via these URLs (because it's kept in a file and not in object metadata), we can't currently access the datafile metadata (which is kept in object metadata).

[thclark]

It's just a nightmare. Who'd have thought all this was so hard to work with?!

I don't want to have a central system orchestrating all this signing but I'm really coming close, TBH.

[thclark]

Have you raised an issue on the google python SDK for this?

[thclark]

And have you checked the headers in the response? If you put on debug mode, use requests to fetch the object at the signed URL, then squirrel around in the response object, you might find metadata in the headers??

[cortadocodes]

It turns out we can get metadata for signed URL datafiles. It's just the storage emulator that needs updating. I've raised an issue here.