MetaGen

A Filesystem Indexing Tool Designed to Support the Preservation of Projects

What is 'MetaGen'?

MetaGen is a tool written in Python with a GUI in PySide that scans file directories, collects metadata about every file it finds, and compiles that information into a large .csv file called an "Index". It uses an Extensions system to compartmentalize and execute per-file type processing logic to collect unique file metadata that would be otherwise be tedious to acquire.

Setup

Setting up the Development Environment

1. Download the required version of Python (See the version badge at the top of this README for the exact version)

2. Clone the MetaGen repository by running:

   git clone https://github.com/sonyinteractive/metagen.git
   

3. Follow the steps in the Downloading Prerequisite Tools section

4. Open a Terminal and CD into the metagen folder

5. Run the following to get the required Python libraries:

   pip install -r requirements.txt
   

   It is recommended to use a Virtual Environment (venv) to isolate the MetaGen dependencies from your system installation, but this is merely a suggestion.

Downloading Prequisite Tools

MetaGen's 'Extensions' system relies on third-party binaries to process and interpret file types. However, due to their potential size, they are not kept within MetaGen's source code, and must be downloaded separately to allow those plugins to function.

Please refer to the Extensions section for what tools need to be downloaded for each Extension, and the installation instructions for them.

Note: This only applies to local development workflows. Compiled releases of MetaGen come with the third-party binaries for ease of use.

Getting Started - Using the Interface

Opening MetaGen

1. Either Click on the metagen.exe if using a compiled release, or by running the following if you have a development environment set up

   python metagen.py
   

Creating a New Project

1. Go to File->New Project
2. Choose a location and project name, and click 'Ok'

Generating an Index

1. Fill out the fields in the Configuration tab

2. Go to Index->Generate Index.

   Note: Depending on the amount of data to process, this may take a long, long time. Please monitor the Output window. When the processing is complete, you should see something like the following:

   

View a Generated Index

Indexes are simple .csv files which can be opened in a variety of methods. A simple way of previewing an index file is using MetaGen's built-in viewer. Keep in mind this will only load a small portion of the file.

1. Go to Index->Load Index

Getting Started - Using the CLI

Note: Not all features are available through the CLI as it was designed with the sole function of generating indexes. If you want access to the full suite of features, please follow the steps in the Using the U.I section.

1. To access the help text of the CLI:

   python metagen.py --help
   

   

2. To generate an index:

   python metagen.py --use_cli -p <metagen_project_path>
   

   

Extensions - The 'Plugin' System

MetaGen's core purpose is to serve as an orchestrator for file processing and analysis. To support this and future growth, a plugin-like system called the 'Extensions' feature was added.

Extensions define what file formats they support, and the unique processing logic that is ran to extract metadata from those formats. They also support executing third-party tools which need to be downloaded separately in order for MetaGen to function correctly.

Please refer to the following table for all native Extensions and their respective download instructions

Extensions List

Name	Third-Party Tools Download Link	Instructions
AudioVisual	https://www.gyan.dev/ffmpeg/builds/	1. Download the latest 'Full' build zip 2. Extract the .zip contents add it to the Extensions/AudioVisual/Tools folder under the folder name FFMPEG