Hjm/doc 1.x to 2.x

README.md

@@ -52,36 +52,29 @@ CoAL provides comprehensive tools for interacting with the CosmoTech API, allowi
 
 - Authenticate with different identity providers (API Key, Azure Entra, Keycloak)
 - Manage workspaces and files
-- Work with the Twin Data Layer for graph data
 - Handle runners and runs
 - Process and transform data
 - Build end-to-end workflows
 
-```python
-import os
-from cosmotech.coal.cosmotech_api.connection import get_api_client
-
-# Set up environment variables for authentication
-os.environ["CSM_API_URL"] = "https://api.cosmotech.com"  # Replace with your API URL
-os.environ["CSM_API_KEY"] = "your-api-key"  # Replace with your actual API key
-
-# Get the API client
-api_client, connection_type = get_api_client()
-print(f"Connected using: {connection_type}")
+CoAL use a Configuration that loads environement variables. This Allows to integrate authentication directly in api clients. Below the intantiation of OrganizationApi integrate the authentiation via the configuration file.
 
+```python
 # Use the client with various API instances
-from cosmotech_api.api.organization_api import OrganizationApi
-org_api = OrganizationApi(api_client)
+from cosmotech.coal.cosmotech_api.apis import OrganizationApi
+org_api = OrganizationApi()
 
 # List organizations
 organizations = org_api.find_all_organizations()
 for org in organizations:
     print(f"Organization: {org.name} (ID: {org.id})")
 
-# Don't forget to close the client when done
-api_client.close()
 ```
 
+### Configuration
+
+CoAL uses a Configuration, it contains default configuration (all environment variable loaded for a Cosmotech RUN) and can be augmented by loading a .toml file.
+The configuration is the tool that allows the seemless connection to the cosmotech API.
+
 ### Other Components
 
 - **coal**: Core library with modules for API interaction, data management, etc.

docs/.nav.yml

@@ -6,3 +6,4 @@ nav:
   - Tutorials: tutorials/
 #  - References: references/
   - csm-data: csm-data/
+  - Helpers: helpers/

docs/csm-data/store/load-parquet-folder.md

@@ -0,0 +1,11 @@
+---
+hide:
+  - toc
+description: "Command help: `csm-data store load-parquet-folder`"
+---
+# load-parquet-folder
+
+!!! info "Help command"
+    ```text
+    --8<-- "generated/commands_help/csm-data/store/load-parquet-folder.txt"
+    ```

docs/csm-data/store/output.md

@@ -0,0 +1,11 @@
+---
+hide:
+  - toc
+description: "Command help: `csm-data store output`"
+---
+# output
+
+!!! info "Help command"
+    ```text
+    --8<-- "generated/commands_help/csm-data/store/output.txt"
+    ```

docs/helpers/.nav.yml

@@ -0,0 +1,9 @@
+nav:
+   - aws: 'aws.md'
+   - azure: 'azure.md'
+   - cosmotech_api: 'cosmotech_api.md'
+   - csm: 'csm.md'
+   - postgresql: 'postgresql.md'
+   - singlestore: 'singlestore.md'
+   - store: 'store.md'
+   - utils: 'utils.md'

docs/helpers/aws.md

@@ -0,0 +1,23 @@
+---
+description: "Helpers"
+---
+# **cosmotech.coal.aws**
+
+::: cosmotech.coal.aws.s3.S3
+    options:
+      members:
+        - file_prefix
+        - use_ssl
+        - ssl_cert_bundle
+        - access_key_id
+        - endpoint_url
+        - bucket_name
+        - secret_access_key
+        - output_type
+        - client
+        - resource
+        - upload_file
+        - upload_folder
+        - download_files
+        - upload_data_stream
+        - delete_objects

docs/helpers/azure.md

@@ -0,0 +1,85 @@
+---
+description: "Helpers"
+---
+# **cosmotech.coal.azure**
+
+::: cosmotech.coal.azure.adx.auth
+    options:
+      members:
+        - create_kusto_client
+        - create_ingest_client
+        - initialize_clients
+        - get_cluster_urls
+
+<br/>
+
+::: cosmotech.coal.azure.adx.ingestion
+    options:
+      members:
+        - ingest_dataframe
+        - send_to_adx
+        - check_ingestion_status
+        - monitor_ingestion
+        - handle_failures
+        - clear_ingestion_status_queues
+
+<br/>
+
+::: cosmotech.coal.azure.adx.query
+    options:
+      members:
+        - run_query
+        - run_command_query
+
+<br/>
+
+::: cosmotech.coal.azure.adx.runner
+    options:
+      members:
+        - prepare_csv_content
+        - construct_create_query
+        - insert_csv_files
+        - send_runner_data
+
+<br/>
+
+::: cosmotech.coal.azure.adx.store
+    options:
+      members:
+        - send_table_data
+        - process_tables
+        - send_pyarrow_table_to_adx
+        - send_store_to_adx
+
+<br/>
+
+::: cosmotech.coal.azure.adx.tables
+    options:
+      members:
+        - table_exists
+        - check_and_create_table
+        - create_table
+
+<br/>
+
+::: cosmotech.coal.azure.adx.utils
+    options:
+      members:
+        - create_column_mapping
+        - type_mapping
+
+<br/>
+
+::: cosmotech.coal.azure.blob
+    options:
+      members:
+        - dump_store_to_azure
+        - delete_azure_blobs
+
+<br/>
+
+::: cosmotech.coal.azure.storage
+    options:
+      members:
+        - upload_file
+        - upload_folder

docs/helpers/cosmotech_api.md

@@ -0,0 +1,54 @@
+---
+description: "Helpers"
+---
+# **cosmotech.coal.cosmotech_api**
+
+::: cosmotech.coal.cosmotech_api.apis.DatasetApi
+    options:
+      members:
+        - download_dataset
+        - download_parameter
+        - path_to_parts
+        - upload_dataset
+        - upload_dataset_parts
+
+<br/>
+
+::: cosmotech.coal.cosmotech_api.apis.RunApi
+    options:
+      members:
+        - get_run_metadata
+
+<br/>
+
+::: cosmotech.coal.cosmotech_api.apis.RunnerApi
+    options:
+      members:
+        - get_runner_metadata
+        - download_runner_data
+
+<br/>
+
+::: cosmotech.coal.cosmotech_api.apis.WorkspaceApi
+    options:
+      members:
+        - list_filtered_workspace_files
+        - download_workspace_file
+        - upload_workspace_file
+
+<br/>
+
+::: cosmotech.coal.cosmotech_api.objects.Connection
+    options:
+      members:
+        - get_api_client
+
+<br/>
+
+::: cosmotech.coal.cosmotech_api.objects.Parameters
+    options:
+      members:
+        - format_parameters_list
+        - write_parameters_to_json
+        - write_parameters_to_csv
+        - write_parameters

docs/helpers/csm.md

@@ -0,0 +1,9 @@
+---
+description: "Helpers"
+---
+# **cosmotech.coal.csm**
+
+::: cosmotech.coal.csm.engine
+    options:
+      members:
+        - apply_simple_csv_parameter_to_simulator

docs/helpers/postgresql.md

@@ -0,0 +1,45 @@
+---
+description: "Helpers"
+---
+# **cosmotech.coal.postgresql**
+
+::: cosmotech.coal.postgresql.runner
+    options:
+      members:
+        - send_runner_metadata_to_postgresql
+        - remove_runner_metadata_from_postgresql
+
+<br/>
+
+::: cosmotech.coal.postgresql.store
+    options:
+      members:
+        - dump_store_to_postgresql
+        - dump_store_to_postgresql_from_conf
+
+<br/>
+
+::: cosmotech.coal.postgresql.utils.PostgresUtils
+    options:
+      members:
+        - table_prefix
+        - db_name
+        - db_schema
+        - host_uri
+        - host_port
+        - user_name
+        - user_password
+        - password_encoding
+        - full_uri
+        - metadata_table_name
+        - get_postgresql_table_schema
+        - send_pyarrow_table_to_postgresql
+        - add_fk_constraint
+        - is_metadata_exists
+
+<br/>
+
+::: cosmotech.coal.postgresql.utils
+    options:
+      members:
+        - adapt_table_to_schema

docs/helpers/singlestore.md

@@ -0,0 +1,9 @@
+---
+description: "Helpers"
+---
+# **cosmotech.coal.singlestore**
+
+::: cosmotech.coal.singlestore.store
+    options:
+      members:
+        - load_from_singlestore