Revamp archiving#1229
Draft
ThomasKroes wants to merge 309 commits into
Draft
Conversation
ThomasKroes
added
enhancement
Standardize workflow error handling and improve lifecycle/reporting. Replaced _error with _errorMessage in WorkflowContextBase, ProjectOpen/Save contexts and workflows; updated catch/finalize checks to use the new field. Added handleDone implementations for ProjectOpen/ProjectSave to validate results and post user notifications with icons, and tightened null handling (throw on unexpected null result). Ensure results are initialized by calling initResult() in AbstractWorkflow::start and switch to result.reset(new ...) in initResult implementations. Converted several context structs to classes with explicit public: sections and made minor ProjectManager changes to manage and start active workflows.
Include ModalTask and add a _task member to AbstractWorkflow to provide a modal progress task for workflows. Initialize _task in the constructor with the workflow title, mark it running in start() and finished in handleDone(). Also mark the setupTree parameter as unused to avoid warnings. This allows workflows to display modal progress and ensures the task lifecycle is tracked.
Introduce conditional verbose debug logging across workflow classes. ProjectOpenWorkflow now defines PROJECT_OPEN_WORKFLOW_VERBOSE in debug builds and adds printLine calls to key stages (makeRecipe, setup, extract, finalize, handleDone) with structured labels and verbosity levels. ProjectSaveWorkflow printLine labels/levels were standardized ("Workflow" / "Recipe stage") and indentation levels adjusted. AbstractWorkflow now logs construction and includes verbosity levels on existing printLine calls. Changes are diagnostic only and do not alter workflow logic.
Split and clean up the project save flow: add Archiver to ProjectSaveContext, replace a single save() with saveProjectJson, saveProjectMetaJson, saveWorkspaceJson and archive() stages, and update the recipe to return per-stage success/failure. Use UniqueTemporaryDir for temporary directory handling, compress via Archiver, and improve finalize() to add the recent file, set the project's file path and update contributors (throws if no current project). Remove debug qDebug logging from Task::updateProgress and Task::privateKill and delete obsolete commented-out save logic in ProjectManager.
Introduce explicit workflow support and task propagation across the project opening/saving pipeline. Key changes: - AbstractProjectManager: added workflow API (getActiveWorkflow, hasActiveWorkflow, setActiveWorkflow, resetActiveWorkflow) and removed legacy getOpenTask usage. - ProjectManager: implement workflow getters/setters/reset, store active workflow as util::UniqueAbstractWorkflow, start/reset workflows via the new API, remove _openTask member, and clean up includes. - AbstractWorkflow: expose getTask() to let recipe tasks update workflow progress and introduce UniqueAbstractWorkflow alias. - Project/ProjectOpen/ProjectSave workflows: wire workflow tasks into project/data-hierarchy serialization and emit finished signals from concrete workflows (moved emission out of base class). - DataHierarchyManager and DatasetsLoadRecipeBuilder: update progress reporting to use task subtasks (setRunning, setSubtasks, setSubtaskStarted/Finished) and remove direct references to the old open task. - util::Serializable: add QPointer<Task> member and accessors (getTask, hasTask, setTask) so serializable objects can report progress. - Misc: minor formatting, include cleanups, and small whitespace/ordering tweaks. Purpose: unify progress reporting under the new workflow/task model so recipes and serializable objects can report subtasks and overall workflow progress consistently.
Introduce src/actions/ActionOperation.h which defines the mv::ActionOperation enum (Save, Load, Duplicate, Export, Validate) and includes SPDX/license header. Register the new header in cmake/CMakeMvSourcesPublic.cmake under PUBLIC_MISCELLANEOUS_ACTIONS_HEADERS so the file is picked up by the build.
Introduce a SerializationPlan API and executor to support staged/parallel serialization. Adds mv::util::SerializationPlan, AbstractSerializationPlanExecutor and a TaskTreeSerializationPlanExecutor (stubbed). Wire the executor into ProjectManager and AbstractProjectManager, update DataHierarchyManager to build/execute serialization plans, and add makeFromPlan/makeToPlan helpers to Serializable. Also add several placeholder action headers/sources and update CMake lists to include the new files. TaskTreeSerializationPlanExecutor currently contains a debug stub and should be implemented to run plan stages/jobs.
Add a new SerializePlanWorkflow (SerializePlanWorkflow.{h,cpp}) and wire it into TaskTreeSerializationPlanExecutor to run serialization plans via a workflow. Refactor SerializationPlan (header + cpp) to provide Job/Stage getters, Job::run, result storage, and a Stage::Mode enum so stages can be parallel or sequential. Make AbstractSerializationPlanExecutor inherit QObject and add a _workflow unique_ptr to the executor. Update DataHierarchyManager to collect per-job results into a final datasets map and add some debug logging. Update CMake lists to include the new files.
Add robust error/exception support and API to SerializationPlan jobs, capture exceptions from job executions, and make stage creation more flexible. Changes: - SerializationPlan: expose Job::getFunction by const-ref, clear errors before run, and add result/error helpers (clearResult, setError, hasError, getError, clearError, setException, setUnknownException, fail). Add optional _error member. - SerializationPlan::addSequentialStage: replace old overload with a template that accepts callables taking either Job& or no arguments, constructing appropriate JobFunction wrappers. - SerializePlanWorkflow: execute jobs via shared_ptr<SerializationPlan::Job>, wrap job.run() in try/catch to set job errors on exceptions, and update task completion handling to report failures/cancellations. Remove verbose onGroupDone logging. - TaskTreeSerializationPlanExecutor: add conditional verbose logging macro for debug builds and guard debug output. - DataHierarchyManager: clean up debug prints and log final serialization result, ensure job results are cleared when appropriate. Overall this makes the serialization workflow safer (exceptions turn into job errors), more flexible in how sequential stages are provided, and cleans up noisy debug output.
Introduce Serializable::reportSerializationWarning/Error/Fatal helpers that forward messages to the active OperationContext (with qWarning fallback in debug). Include OperationContextScope in Serializable.cpp. Update callsites: report a dataset serialization error in DataHierarchyManager and adjust SerializePlanWorkflow lambdas to capture this and use the instance operation context for OperationContextScope.
Introduce a shared OperationContext and propagate it across workflows and tasks so operation state/errors can be reported and aggregated. Key changes: - Add SharedOperationContext alias in OperationContext.h and refactor OperationContextScope to hold a thread-local SharedOperationContext (getShared/currentShared) and accept a SharedOperationContext in its ctor. - Update AbstractWorkflow to accept an optional SharedOperationContext, store it, and expose getOperationContext() returning the shared pointer. - Update SerializePlanWorkflow (header + ctor) to accept and forward an operation context; use OperationContextScope(getOperationContext()) inside job task lambdas. - TaskTreeSerializationPlanExecutor now constructs SerializePlanWorkflow with OperationContextScope::getShared(). - ProjectSaveWorkflow uses OperationContextScope in key stages and reports combined errors/notifications when save completes with errors. - DataHierarchyManager adds Serializable::reportSerializationError calls at several failure points during serialization. - Minor includes and API adjustments to match shared-context model. These changes centralize error handling and allow sharing the operation context across threads/tasks during serialization and project save workflows.
Introduce a shared state to SerializationPlan and refactor data-hierarchy serialization to use parallel job creation and a final assembly stage. DataHierarchyManager::toVariantMap now creates per-dataset jobs that populate a shared QVariantMap with item maps (including SortIndex), then assembles a nested variant map (with Children entries) from that shared state; added error handling and logging. Removed child traversal/SortIndex emission from DataHierarchyItem::toVariantMap to avoid double-serialization. Make TaskTreeSerializationPlanExecutor synchronous by waiting on the SerializePlanWorkflow via a QEventLoop.
Convert the assembled QVariantMap to a QJsonObject and QJsonDocument in DataHierarchyManager::toVariantMap(), preparing the data for JSON serialization. Also remove a qDebug() line that logged the plan stages and job count to reduce noisy debug output.
Introduce prettyPrintVariantMap(QVariantMap) to print a QVariantMap as indented JSON (implementation and header declaration). Remove an unused QJsonObject/QJsonDocument allocation from DataHierarchyManager::toVariantMap to avoid creating unnecessary temporary objects. No behavior changes to data serialization logic; this adds a small utility for debugging/printing maps.
Introduce a SerializationPlan-based workflow in DataHierarchyManager::fromVariantMap with three sequential stages ("Enumerate datasets", "Populate data hierarchy", "Populate datasets") and execute it via the project's serialization plan executor; stage bodies are scaffolded with error reporting. Add guards in ProjectManager (openProject, saveProject, publishProject) to throw if another workflow is active to prevent concurrent workflows. Add ProjectOpenWorkflow::openPojectJson (and its declaration) to load a project JSON into the current project with basic existence/error checks.
Rework DataHierarchyManager::fromVariantMap to build hierarchy items eagerly and load dataset contents in parallel. Introduces loadDataHierarchyItem and recursive populateDataHierarchy (with SortIndex ordering and abort check), collects dataset metadata, reorders to load non-derived datasets first, and schedules dataset load jobs via a parallel stage. Also add improved logging (qDebug/qWarning) for project open/save workflows, fix openPojectJson -> openProjectJson, add a stub openWorkspaceJson, and invoke openProjectJson in the open recipe.
Wrap ProjectOpenWorkflow and ProjectSaveWorkflow result handling with a check for mv::projects().getCurrentProject() and use currentProject->getFilePath() for success messages instead of result->_filePath. This prevents null dereferences when no active project is present and ensures notifications show the current project's path. Also move save-error notifications inside the same guard and apply a minor whitespace fix.
Remove the saveToDisk boolean parameter from rawDataToVariantMap and related encodeBlock API, simplifying serialization to always use blob files (encodeToFile) and return URIs/compressed sizes. Update header docs and call sites across LinkedData, ClusterData, ImageData, PointData and Serialization to match the new signature and behavior. Also add a debug prettyPrintVariantMap call in ClusterData::fromVariantMap and minor cleanup (whitespace/alignment and comment removal). Note: this is a behavioral change — raw data is no longer optionally inlined via the old flag, callers now always produce external blob references.
Add user notifications and debug logging when project open/save operations complete with errors. ProjectOpenWorkflow.cpp now posts a notification and qDebugs the combined error message if the operation context has errors. ProjectSaveWorkflow.cpp reorders the debug log to follow the notification for consistent behavior and formatting. Notifications use the "exclamation-triangle" StyledIcon and display the combined error message.
Replace manual allocation via UniqueTemporaryDir(new QTemporaryDir(...)) with std::make_unique<QTemporaryDir>(...) in ProjectSaveWorkflow::setup for clearer, exception-safe ownership semantics. No functional change; still creates the same temporary directory path.
Fix user-facing messages and workspace loading: ensure the success text consistently reports the project path and uses "opened" (not "saved") for longer loads; include a prefixed error message in the notification and debug output when the operation context has errors. Replace commented-out project JSON handling with a direct call to workspaces().loadWorkspace(context._workspaceJsonPath) to actually load the workspace JSON.
Fix inconsistent file path shown in the save success message when the duration is >= 1000 ms. The seconds-formatted branch previously used result->_filePath, which could display a different or incorrect path; both branches now use currentProject->getFilePath() so the displayed path is consistent. No other behavior or formatting changes were made.
Pass ConcurrencyMode::Sequential to rawDataToVariantMap calls and add minor debug/cleanup. Updated ClusterData::toVariantMap and Points::toVariantMap to call rawDataToVariantMap(..., nullptr, ConcurrencyMode::Sequential) for serial processing of raw buffers (indices, selection, dimension names, clusters). Removed a debug prettyPrintVariantMap call in ClusterData::fromVariantMap and added a qDebug log in Points::fromVariantMap. Also includes small whitespace/formatting cleanup.
Write the assembled hierarchy into the SerializationPlan's shared state instead of using an internal QVariant result. Remove the obsolete _result member and getResult() accessor from SerializationPlan. Update DataHierarchyManager to set (*toPlan.getSharedState())["Hierarchy"] and to return that map. In ProjectSaveWorkflow re-enable saveProjectMetaJson and add an archive QSyncTask (with exception handling) before finalize. These changes centralize serialization outputs in the shared state and simplify result handling.
Introduce structured lifecycle reporting and console formatting for workflows. Adds WorkflowExecutionLifecycleScope, WorkflowStageSummary and WorkflowConsoleFormatter, and wires lifecycle reporting into WorkflowExecutionContext (types, depth, lifecycle/detail helpers, typed child creators) and TaskflowWorkflowPlanExecutor (root lifecycle, nested workflow scope, job/stage reporting and error handling). Refactor dataset/item serialization to produce UniqueWorkflowPlan workflows (ToVariantMapWorkflowContext helper) and adapt callers to execute plans. Other changes: expose new util headers in CMake, minor cleanups (Cluster serializer type, debug macro toggle, DataHierarchyManager logging removal) and various API adjustments to support lifecycle reporting.
Convert dataset mapping jobs into nested workflow stages and improve lifecycle reporting and console output. - DataHierarchyItem/DataHierarchyManager: replace inline "Dataset" jobs with nested workflow stages so datasets are compiled as sub-workflows. - TaskflowWorkflowPlanExecutor: use WorkflowExecutionLifecycleScope for job lifecycle; compileStage now creates timed stage contexts (sequential/parallel children), measures elapsed time via QElapsedTimer, and reports duration on finish. - WorkflowConsoleFormatter: enhance formatting to include severity, icons, labels, aligned duration column, and clearer failed/skipped messages; removed special-case summary formatting. - WorkflowExecutionContext: remove stray debug logging.
Simplify and harden serialization/workflow code: DataHierarchyItem::toVariantMap now builds and returns a QVariantMap directly (removed its workflow helper and header declaration). Introduced ToVariantMapWorkflowContext (mutex-protected) for DataHierarchyManager and switched the save workflow to spawn item creation in parallel then assemble maps. Make WorkflowExecutionContext::info/warning/error thread-safe by serializing qDebug calls with a static mutex. Adjust Serializable::toVariantMapWorkflow to publish results via executionContext->publishResultValue using the serialization name. Also commented-out legacy workflow branches in ClusterData and Points (pre-1.5.0 / raw-data load paths) and applied minor formatting/cleanup across affected files.
Introduce getResultValues() and takeResultValues() on WorkflowExecutionState (thread-safe access and consume semantics) to allow retrieving all key/value result pairs at once. Update TaskflowWorkflowPlanExecutor to consume the aggregated result values via takeResultValues() and pass them to WorkflowResult::setResultValues(), replacing the previous single-value call. This change centralizes result handling and clears stored result values after consumption.
Introduce scoping for workflow result values so contexts can get/take only their own namespaced results. Added getResultValues(), takeResultValues() and getResultScope() to WorkflowExecutionContext and updated TaskflowWorkflowPlanExecutor to use state->takeResultValues(rootContext->getResultScope()) and set the result via setValue(). WorkflowExecutionState now exposes getResultValues(const QString& scope) and takeResultValues(const QString& scope), implementing prefix-based filtering/removal (protected by the existing mutex) so only keys in the given scope (scope + '/') are returned or erased. This is an API change—callers that previously used the unscoped variants must supply a scope where appropriate.
Make result scopes unique by using QUuid strings: the root context now uses its UUID as the result scope, and child contexts build scoped result paths via a new makeChildResultScope helper. Mark takeResultValues() as const in both header and implementation. Also set NestedWorkflow children to initialize their result scope. Add qDebug() logging in WorkflowExecutionState::setResultValue to dump the result map for debugging.
Use scoped result keys in WorkflowExecutionContext so result lookups/consumers use a fully-qualified key (scopedResultKey) to avoid collisions across nested workflows. Make WorkflowExecutionState::getResultValues const and adjust its implementation (and takeResultValues) to correctly handle an empty scope by returning/removing all entries; normalize prefix handling when a non-empty scope is provided. Add [[nodiscard]] qualifiers and documentation to several getters and to scopedResultKey to improve API clarity and const-correctness.
Replace string-based result scoping with per-context published-key tracking and explicit cleanup. WorkflowExecutionContext now tracks published result keys (with a mutex), exposes takeResultValues() and releasePublishedResultValues(), and calls release on destruction. Removed the old _resultScope and scopedKey helpers and removed corresponding state APIs (getResultValues/takeResultValues(scope)). Call sites updated: Project now composes scoped keys itself via a scopeResultKey lambda, and TaskflowWorkflowPlanExecutor uses rootContext->takeResultValues() without a scope argument. The change centralizes lifecycle management of published results, simplifies scoping semantics, and ensures published values are removed when a context is destroyed.
Remove several debug/log statements and simplify result-value lifecycle: drop qDebug output in Project::toVariantMap and WorkflowExecutionState::setResultValue; remove WorkflowExecutionContext destructor, published-result-keys tracking, mutex, and releasePublishedResultValues (publishResultValue no longer records keys; takeResultValues delegates to state). Add codec initialization for blob encoding in bytesToBlobVariantMap. Also clean up/refactor PointData/Points serialization (flatten else block, rename indices -> indicesMap, remove commented code) and apply minor formatting/whitespace fixes.
Introduce WorkflowExecutionNodeType (header + impl) and wire it into WorkflowExecutionContext. Rework WorkflowProgressNode to track type, name, status and timing, add thread-safe child management (createRoot/createChild/addChild) and lifecycle state methods. Simplify cluster serialization: ClusterData::toVariantMap now returns ClustersSerializer::toVariantMap(QVector<Cluster>) (synchronous QVariantMap) and remove the previous workflow-plan based toVariantMapWorkflow path. Update DataHierarchyManager to use datasetId as job key, adjust TaskflowWorkflowPlanExecutor formatting, and add new sources to CMake. Note: public API changes include removed toVariantMapWorkflow overloads and new progress/node-type APIs.
Introduce a Snapshot struct to capture a node's immutable state (type, status, name, progress, weight, child counts, elapsed time and nested children) and add createSnapshot() which builds a recursive snapshot while minimizing lock hold by copying child pointers and using an unlocked elapsed-time helper. Change getElapsedMilliseconds() to return std::int64_t and add getElapsedMillisecondsUnlocked() to allow calling the elapsed calculation without re-locking. These changes enable safe, consistent snapshotting of workflow progress nodes for reporting or UI use.
Introduce formatting support for workflow progress trees. Adds WorkflowConsoleFormatter::formatProgressTree and helpers appendProgressNode and statusName to produce a column-aligned, multi-line textual representation of a WorkflowProgressNode::Snapshot (kind, name, status, progress, children, time). Includes necessary headers and declares the new API in the header. Uses getWorkflowExecutionNodeTypeName and rounds progress for display; recursion prints child nodes with indentation.
Refactor console progress utilities into a WorkflowConsoleDashboard class. Replaced the WorkflowExecutionNodeType enum and its string conversion helpers with a dashboard that holds a WorkflowExecutionState::Ptr, uses WorkflowConsoleFormatter to format a progress snapshot, and prints it to stdout using ANSI sequences to clear and position the cursor. Updated includes and removed unused QHash/QString helpers.
Introduce a scoped RAII helper and async console dashboard to show live workflow progress. - Add WorkflowConsoleDashboardScope (header + cpp) that starts/stops a WorkflowConsoleDashboard for a given WorkflowExecutionState. - Make WorkflowConsoleDashboard run in a background thread with start/stop/run/render and ensure proper cleanup in its destructor. - Integrate scope into TaskflowWorkflowPlanExecutor::executeRoot to display the dashboard for root workflows. - Update WorkflowExecutionContext to update progress nodes on lifecycle events (markRunning, setProgress+markCompleted, markFailed, markSkipped). - Simplify WorkflowExecutionLifecycleScope destructor to call finish(). - Register new header/source files in CMakeMvSourcesPublic.cmake.
Change WorkflowExecutionContext::createChild to accept a Type parameter and propagate it through createTypedChild helpers (nested/sequence/parallel/job). Update call sites (WorkflowPlanExecutor) to use createNestedWorkflowChild. Initialize _type earlier when constructing children. Remove redundant _progressNode->setProgress(1.0) from reportFinished since markCompleted handles completion. These changes ensure child contexts are created with the correct Type and avoid duplicate progress updates.
Only create the WorkflowConsoleDashboard for root executions and when explicitly enabled via WorkflowExecutionOptions._enableConsoleDashboard. Added a new _enableConsoleDashboard option to WorkflowExecutionOptions. Introduced WorkflowExecutionContext::isRootExecution() and WorkflowProgressNode::isRoot() to detect root nodes. Also added a process-wide QMutex (workflowConsoleMutex) and QMutexLocker usage in WorkflowConsoleDashboard to serialize console output and avoid interleaved prints from multiple threads. Minor refactors: conditional dashboard scope creation in TaskflowWorkflowPlanExecutor and small rendering loop/locking changes in WorkflowConsoleDashboard.
Replace the WorkflowPlan-based serialization with a direct implementation of DatasetImpl::toVariantMap. The method now builds and returns a QVariantMap constructed from WidgetAction::toVariantMap, proxy members, linked data, dataset metadata and PropertiesSerializer::toVariantMap(_properties). The UniqueWorkflowPlan/toVariantMapWorkflow implementation and its declaration in the header were removed.
Replace workflow-based serialization with scoped variants that accept an execution context. Serializable now exposes fromVariantMapScoped() and toVariantMapScoped(), and JSON helpers accept an optional SharedWorkflowExecutionContext. Project uses these scoped methods (fromVariantMapScoped/toVariantMapScoped) and simplifies loading/saving logic by directly composing QVariantMaps instead of building nested WorkflowPlans. DataHierarchyManager now loads datasets via dataset->fromVariantMap(...) in parallel jobs and returns the hierarchy map directly. PointData (Points) now calls DatasetImpl::fromVariantMap(...) and includes legacy/version handling during load. Project open/save workflow plans were updated to call the scoped APIs and to pass execution contexts into projects().toJsonFile(). These changes simplify control flow, integrate execution context propagation, and require updated function signatures across callers.
Replace many fromVariantMapWorkflow/UniqueWorkflowPlan patterns with fromVariantMapScoped(...) void APIs to load serialized state directly within a provided parent execution context. Update Project, DataHierarchyManager, PointData, ClusterData, and related classes to use the scoped loaders (including Points) and pass parentExecutionContext into raw-data population where appropriate. Add defensive error logging in Plugin::fromVariantMap. Rename serialized key "Hierarchy" -> "DataHierarchy" in Project::toVariantMapScoped. Improve TaskflowWorkflowPlanExecutor::executeChild to require a valid context, create a nested workflow child, execute the taskflow synchronously with proper lifecycle, capture results/duration, and propagate/report exceptions. Miscellaneous formatting and minor debug/info logging added.
Introduce WorkflowRuntimeScoped singleton and expose it in CMake to centralize execution of workflow plans. Use it from populateBytesFromBlobMap (add null/size guards and delegate execution) and include the new header in Serialization. Simplify Plugin::fromVariantMap by checking for serialized keys instead of relying on exceptions. Update PointData to call populateBytesFromBlobMap with the corrected signature. Harden TaskflowWorkflowPlanExecutor: early-null check for plans, simplify child execution flow, ensure lifecycle.finish()/fail() are invoked correctly and propagate exceptions consistently. Fix ZstdBlobCodec to create/free ZSTD_DCtx and check decompression errors properly. Improve WorkflowResultFuture exception rethrowing with logging for std::exception and unknown exceptions.
Improve robustness and logging across serialization, codec, and data-hierarchy code. Key changes: - Serialization: introduce ActiveDecodeGuard (with global mutex/set) to detect/avoid duplicate concurrent decodes and prevent overlapping decode races; add guard usage and expanded exception logging. Implement batching of decode jobs with a max concurrency (4) to limit parallel decodes. - ZstdBlobCodec: validate ZSTD frame size (error/unknown/mismatch), check ZSTD_createDCtx() result, and add debug output when reading zstd entries. - DataHierarchyManager: remove unused getter helpers, add debug logging when building plan and traversing items, collect dataset ids for debug, and fix child map insertion to use child's dataset id. - PointData & DataHierarchyItem: minor formatting/whitespace tweaks and added debug output in Points::fromVariantMap. Overall these changes add defensive checks and diagnostic output to help detect and prevent decoding/frame errors and to trace data-hierarchy processing.
Introduce WorkflowPlan::addBatchedParallelStage to split job lists into parallel batches (validates batchSize>0, no-op for empty jobs) and create per-batch parallel stages. Update DataHierarchyManager to use batched stages with a batch size of 8 when loading datasets to limit concurrency. Re-enable PropertiesSerializer deserialization in DatasetImpl::fromVariantMap and comment out a legacy version-check block in Clusters::fromVariantMapScoped.
Replace manual batching of decode jobs with WorkflowPlan::addBatchedParallelStage. The change renames the plan variable, removes the flushBatch helper and maxConcurrentDecodeJobs constant, and builds a single jobs list which is submitted to addBatchedParallelStage with a batch size of 8. This simplifies concurrency handling and improves readability while preserving per-block bounds checks and decode logic.
Re-enable the previously commented logic in Clusters::fromVariantMapScoped to restore the intended deserialization flow. Refactor ClustersSerializer to remove the FromVariantMapWorkflowContext dependency: use local headers/allIndices variables, have buildIndexBuffer return allIndices, and call rebuildClusters(headers, allIndices). Change DataHierarchyManager to load datasets sequentially (commenting out the batched parallel stage) to ensure deterministic dataset loading.
In DataHierarchyManager.cpp switch dataset loading from a sequential stage to a batched parallel stage (batch size 32) to improve loading performance and concurrency. Comment out the previous sequential stage. Remove several qDebug debug prints (dataset ID loop, traversing item, datasetMap logging) and perform minor whitespace cleanup to reduce noisy logs.
Add configurable maximum console log depth and thread through to formatter and execution. - Introduce _maxConsoleLogDepth (std::uint32_t) to WorkflowExecutionOptions with default = max uint to allow limiting how deep workflow progress nodes are logged to the console. - Extend WorkflowConsoleFormatter::format to accept an optional maxDepth parameter and early-return when a node's depth exceeds it. - Pass getState()->getExecutionOptions()._maxConsoleLogDepth into WorkflowConsoleFormatter::format from WorkflowExecutionContext::info/warning/error so runtime config controls logging depth. - Set _maxConsoleLogDepth = 2 when creating execution options in ProjectManager::openProject (and enable the chrome trace sink there). - Add [[nodiscard]] annotations on several formatter helpers and minor signature/formatting tweaks. This reduces noisy console output for large workflow graphs and makes max log depth configurable per execution.
Replace int depth with an explicitly cast std::uint32_t (using auto and static_cast) so the variable matches the unsigned maxDepth type and avoids signed/unsigned comparison warnings. Keeps behavior intact while improving type-safety.
Revamp cluster serialization to a compact, robust binary format and apply several performance and API improvements. Key changes: - Replace legacy QDataStream vector helpers with a new binary header format (RawClusterHeader / RawSingletonClusterHeader) and string blob layout. Add general vs singleton storage modes to embed single-point clusters directly into headers to reduce index buffer overhead. - Add thorough bounds/size checks and exceptions for corrupt/oversized data; split serialization/deserialization into serializeGeneral/serializeSingleton and corresponding deserializeGeneral/deserializeSingleton flows. Introduce DecodedHeaders that can carry embedded indices. - Rename toVariantMap methods to toVariantMapScoped and add SharedWorkflowExecutionContext propagation for blob IO and warnings. Update ClusterData/Clusters callers accordingly. - Optimize Cluster API: take name/id/color/indices by value and move into members; remove commented legacy stream operators. - Improve diagnostics and perf tracing: add QElapsedTimer debug logs during cluster read/write phases and a user warning for extremely large cluster counts when saving. - Minor fixes and consistency improvements: use std::move for indices, tighten string handling, and small code style tweaks. - Reduce parallel batch sizes from 32 to 4 for dataset and blob decoding stages to limit concurrency; increase ProjectManager console log depth and disable a local trace sink. - Fix getElapsedTimeHumanReadable formatting and small serialization workflow adjustments. These changes aim to improve serialization performance, memory usage, and robustness when handling large cluster datasets, while also providing better diagnostics and controlled parallelism during load/save workflows.
Add context-aware serialization by renaming toVariantMap to toVariantMapScoped and passing a parent WorkflowExecutionContext through DataHierarchyItem and callers. Improve robustness of WorkflowExecutionContext logging (guard _state, use configurable max depth) and tighten createChild null-check. Add additional qDebug diagnostics for workflow messages and task execution, and reduce default console log depth in ProjectManager. Revise ClustersSerializer: streamline blob reads, lower the large-header warning threshold, emit context warnings/info and print collected messages for easier debugging. Temporarily disable level-based filtering in WorkflowMessagesFilterModel by commenting out related checks.
Introduce scoped serialization APIs and refactor workflow executor and callsites. - Serializable: rename toJsonDocument -> toJsonDocumentScoped, add toJsonFileScoped(…) and keep toJsonFile(path) delegating to scoped variant. Update header accordingly. - Replace numerous toJsonFile/toVariantMap usages with their "Scoped" variants to accept/propagate a SharedWorkflowExecutionContext (Project, WorkspaceManager, PresetsAction, ApplicationConfigurationAction, ProjectSaveWorkflowPlan, etc.). Remove legacy toVariantMap overrides where appropriate. - DataHierarchyManager: convert toVariantMapWorkflow -> toVariantMapScoped, build workflow plan, execute it with the provided parent execution context and return assembled "Hierarchy" map. - ProjectManager: adjust toVariantMapScoped signature and forward to project->toVariantMapScoped(parentContext). - TaskflowWorkflowPlanExecutor: large refactor to better handle contexts and async execution — reformat signatures, add executeWithContext(), improve executeBlocking/executeAsync behavior, support nested/root contexts safely, add assertions, centralize execution logic, adjust result/message handling so root context populates messages/notifications, and tighten exception handling and logging. Also various small fixes in job execution, lifecycle handling and notification formatting. - Minor fixes: uncommented filtering logic in WorkflowMessagesFilterModel, small formatting/whitespace cleanups and header updates to reflect API changes. These changes propagate an explicit WorkflowExecutionContext through serialization and workflow plan execution paths, improving control over nested executions and ensuring correct result/message propagation.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This pull request introduces a new workflow-based architecture for project operations (such as open, save, import, and publish), adds support for Zstandard (zstd) compression, and refactors the CMake build system to register new codecs and workflow components. It also removes the legacy
ProjectSerializationTasksystem and adds new utility and action classes to support the updated design.Core architecture and workflow changes:
WorkflowPlan,WorkflowExecutionOptions,AbstractWorkflowPlanExecutor,BlobCodec, and related classes) to the build system (ManiVault/cmake/CMakeMvSourcesApplication.cmake,ManiVault/cmake/CMakeMvSourcesPublic.cmake). [1] [2] [3] [4]ProjectOpenParameters,ProjectSaveParameters, etc.) and removed the legacyProjectSerializationTaskand its related code fromAbstractProjectManager. [1] [2] [3] [4] [5] [6]getWorkflowPlanExecutor()toAbstractProjectManagerto provide access to the new workflow plan executor.Codec and compression support:
ManiVault/CMakeLists.txt). [1] [2]ZstdBlobCodec,PassthroughBlobCodec, their factories, and settings actions) in the build system. [1] [2] [3] [4]Actions and utility improvements:
CodecSettingsAction,ActionOperation, and several action recipe classes. [1] [2] [3] [4] [5]Build system and source organization:
These changes modernize project serialization and operation handling, add robust compression support, and prepare the codebase for future extensibility.