Monitor iCloud Export Directory

[thoughtgap]

As a user I would like to have the server running and detect automatically when new (daily) json exports are added to the iCloud Export Directory (the so-called Layer 1). New or updated files should be auto-provisioned to arc-data-server's data.

• Find a way for files to get from Layer 1 to Layer 2 (unzip to Layer2-directory)
• Trigger the update via api endpoint
• Possibility for an "overwrite all" approach
• Monitor the iCloud Directory and trigger the move from Layer 1 to 2 automatically (e.g. with https://github.com/paulmillr/chokidar

[PalminX]

I have some preliminary code what can extract the gz files from Layer 1 to a tmp directory.
It doesn't do any error handling or check if files are already there, but will just overwrite them.

If you are interested, I could provide a patch, it is really too rough to commit.

[PalminX]

I started the Layer1 extraction here:
https://github.com/PalminX/arc-data-server/tree/Layer1ExtractionWIP

This is not ready yet, this is just the first try, it will extract to a tmp directory, but it handles duplicate json.gz files already and will just extract the latest file (see #5)

[thoughtgap]

Thank you so much for the approach! I have copied some of your coding but modified it quite heavily for it to:

• Handle duplicate files (see Use latest of multiple daily export files #5)
• Copy .json files, but extract .json.gz files
• Handle the files only if the files don't exist or are older in the target directory
• work with monthly export files inside the layer1 directory

[thoughtgap]

Added todos to

• Trigger the update via api endpoint
• Possibility for an "overwrite all" approach

[thoughtgap]

Ah, right now it's only working if the autoLoad of layer2 directory is set to false