Feature: clsi cache to accelerate compile speed - #79
Conversation
- Add IngestManager.js to manage the ingestion of build files from CLSI. - Implement file validation and ingestion logic, including local disk imports and downloads from CLSI. - Introduce metadata writing for ingested builds. - Create Persistor.js for object persistence settings. - Add utility functions in utils.js for filename validation and project key generation. - Establish settings defaults for clsi-cache configuration. - Create buildscript.txt for service configuration. - Implement tests for IngestManager and utility functions. - Update CLSICacheHandler to notify about builds based on new settings. - Modify ClsiManager and related components to integrate with the new caching mechanism.
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 4ef43d9f20
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| splitEditorBuildId, | ||
| } from './utils.js' | ||
|
|
||
| const LOCATION = Settings.path.cacheDir |
There was a problem hiding this comment.
When CLSI_CACHE_BACKEND is s3 or gcs, LOCATION is passed to object-persistor as the bucket name, but it comes from cacheDir and defaults to an absolute filesystem path such as /overleaf/services/clsi-cache/cache. Both cloud backends therefore issue every list/read/write against an invalid bucket name; setting it to a real bucket is also unsafe because app.js calls mkdirSync on the same value. Introduce a separate bucket setting and reserve cacheDir for the fs backend so the advertised shared-storage deployment can ingest or retrieve builds.
Useful? React with 👍 / 👎.
| secret: process.env.AWS_SECRET_ACCESS_KEY, | ||
| endpoint: process.env.AWS_S3_ENDPOINT, | ||
| pathStyle: process.env.AWS_S3_PATH_STYLE, | ||
| partSize: process.env.AWS_S3_PARTSIZE || 100 * 1024 * 1024, |
There was a problem hiding this comment.
With the S3 backend, every successful lookup calls S3Persistor.getRedirectUrl, which calculates expiresIn from settings.signedUrlExpiryInMs; this S3 configuration never supplies that value, unlike the GCS block above. Consequently the presigner receives a NaN expiry and produces an unusable or rejected signed URL, so cached outputs cannot be downloaded even after the bucket configuration is corrected. Add signedUrlExpiryInMs to the S3 settings as well.
Useful? React with 👍 / 👎.
| sweepIntervalMs: | ||
| parseInt(process.env.CLSI_CACHE_SWEEP_INTERVAL_MS, 10) || 10 * 60 * 1000, |
There was a problem hiding this comment.
When an operator sets CLSI_CACHE_SWEEP_INTERVAL_MS=0 as documented to disable expiry sweeping, parseInt(...) || 10 * 60 * 1000 replaces zero with the default interval. ExpiryManager.init() therefore still starts the timer and continues deleting expired builds, making it impossible to disable the sweeper through the advertised setting. Handle an explicitly parsed zero separately from an unset or invalid value.
Useful? React with 👍 / 👎.
|
We may not merge this pr, but this is really interesting!
Very smart way to avoid cache miss! 
|
Musicminion
changed the title
|
This one is really useful. |
2 participants
Description
Before: every time you open a project, it recompiles.
202608238010235.mp4
After: (Same as SaaS, need paid subscriptions)
202608238005949.mp4
Related issues / Pull Requests
Contributor Agreement