FreshLit

An automated literature-monitoring agent: a four-stage ETL pipeline that ingests newly published papers and preprints, filters out duplicates and off-topic items, performs LLM-driven structured synthesis, and writes a formatted Markdown digest into a local folder (including an Obsidian vault). Obsidian is optional.

FreshLit 0.3.0 is the provider-neutral release. It supports ChatGPT/Codex, OpenAI, OpenRouter, OpenCode Go, local Ollama, and explicit OpenAI-compatible endpoints. Provider, model, endpoint, and credential selection never fall back silently.

See literature_agent_architecture.md for the full architecture specification.

INGESTION  ->  DEDUP & FILTERING  ->  LLM SYNTHESIS  ->  MARKDOWN DELIVERY
OpenAlex        SQLite ID check       Pydantic schema     YAML frontmatter
Europe PMC      Title matching        Field pulse         Vault writer
                Vector (SPECTER2)
                LLM scoring + journal weight

AI Disclosure

FreshLit's concept, architecture specification, and research profile are the author's own work. The implementation — code, debugging, and documentation — was developed with AI assistance (opencode, an AI coding agent) under human supervision.

Installation for new users

FreshLit requires Python 3.11+, Git, and macOS or Linux/POSIX. Native Windows is not currently supported by the hardened Codex launcher or secure digest writer. The current release runs from a source checkout; it is not yet published as a standalone wheel or pipx application.

1. Clone FreshLit

git clone https://github.com/natevmp/FreshLit.git
cd FreshLit

All commands below run from this checkout. FreshLit keeps its configuration and local state here even when its launcher is invoked from another directory.

2. Create a virtual environment and install

For OpenAI, OpenRouter, OpenCode Go, Ollama, or another OpenAI-compatible endpoint:

python3 -m venv .venv
.venv/bin/python -m pip install -e .

To use a ChatGPT subscription through the hardened Codex adapter, install the optional Codex extra instead:

python3 -m venv .venv
.venv/bin/python -m pip install -e ".[codex]"

The editable install creates .venv/bin/freshlit and keeps it pointed at this checkout. API and local-model installations do not include the coding-agent runtime. The Codex extra pins the official openai-codex SDK to 0.147.0.

Confirm that the command is available:

.venv/bin/freshlit --help

3. Create private user configuration

.venv/bin/freshlit init

This creates .env and config/research_profile.md from their neutral examples only when those files are missing; rerunning it never overwrites your work. The private .env is created with owner-only permissions.

4. Choose research inputs and an output folder

Edit config/research_profile.md and add your own keywords and/or OpenAlex topic IDs plus a nonblank research description. The unfinished template intentionally cannot run. See Research profile for the complete format and the optional topic lookup command.

Set the normal digest destination in config/settings.yaml:

obsidian:
  vault_path: "~/Documents/FreshLit"
  digest_folder: ""

Any ordinary Markdown folder works; Obsidian and its plugins are optional. Run the following if the configured output root does not exist yet:

.venv/bin/freshlit init --create-output-dir

Optionally put OPENALEX_API_KEY in .env for a larger OpenAlex request budget. FRESHLIT_VAULT_PATH is an advanced scheduler/deployment override and should be left unset for a normal installation. Its precedence is process environment, then .env, then YAML. doctor and every run display the resolved destination and warn when an override masks a different YAML path.

5. Select and configure one LLM provider

Run the guided setup:

.venv/bin/freshlit configure-llm

Provider	New-user requirement
codex	Install .[codex], prepare a dedicated profile, and authenticate with a ChatGPT subscription
openai	Add OPENAI_API_KEY to .env
openrouter	Add OPENROUTER_API_KEY to .env
opencode	Add OPENCODE_GO_API_KEY to .env
ollama	Run Ollama locally with the selected model; no API key is used
openai_compatible	Supply an explicit HTTPS endpoint, or a loopback HTTP endpoint, and choose API-key or no-auth mode

The legacy gateway provider remains available only for existing configurations. FreshLit never silently falls back to another provider, model, endpoint, or credential.

The setup can also be scripted with flags, for example:

.venv/bin/freshlit configure-llm --provider ollama --model qwen3:8b
.venv/bin/freshlit configure-llm --provider openrouter --model provider/model-id

configure-llm writes connection choices, but never API-key values, to .env. Add any required key directly to that private file after configuration. The command rejects symlinks, non-files, and unsafe .env ownership or permissions. Do not pass secrets on the command line or put them in config/settings.yaml.

6. Complete Codex setup, if selected

API and Ollama users can skip this step. Codex users must create a dedicated profile outside the checkout:

.venv/bin/freshlit init --prepare-codex-home

The command creates or validates an owner-only mode-0700 directory and prints the exact login command. Complete ChatGPT OAuth, not API-key authentication, with the official Codex tooling; see the Codex authentication documentation. Install the official codex command separately if it is not already on PATH; the Python extra provides FreshLit's runtime integration. With the default profile path, the login command is:

CODEX_HOME="$HOME/.freshlit-codex" codex login

Do not use sudo, reuse ~/.codex, or share the dedicated profile with another Codex installation. FreshLit never initiates login or reads authentication files.

7. Validate and run

# Local, read-only checks; no network requests or model loading
.venv/bin/freshlit doctor

# Optional online compatibility check using bundled synthetic papers
.venv/bin/freshlit llm-check --yes

# Run the complete pipeline
.venv/bin/freshlit run

llm-check makes no request without --yes. With confirmation, it performs three structured checks; retries or output-mode negotiation can add provider calls, so hosted usage may be billed. It checks protocol/schema compatibility, not model or scientific quality.

The first full run initializes SQLite state and builds the research-profile vector automatically. Building it may download the configured embedding model. Later profile or embedding-model changes trigger a rebuild. Manual init-db and build-profile commands remain available and do not require LLM credentials.

Updating an existing installation

New users should follow Installation for new users instead. Before updating an older checkout, preserve local source changes and back up config/settings.yaml, config/journal_tiers.json, .env, the private profile, and data/. Do not replace those files with public examples.

The current public defaults no longer contain the original author's topics or venue weights. If your profile predates structured front matter, migrate while your old local research settings are still present (restore them from your copies if needed):

.venv/bin/freshlit migrate-profile          # preview only, printed to stdout
.venv/bin/freshlit migrate-profile --apply  # writes profile + exclusive .bak

Migration copies effective topics, Europe PMC choices, and venue weights into the profile front matter without altering the existing Markdown body. It makes config/research_profile.md.bak before the first write, refuses to overwrite an existing backup, and does not touch the database, credentials, or other settings. An already-modern profile is not rewritten. Conflicting settings keyword overrides require manual resolution; they are not silently merged.

After migration, remove the matching research-only settings from the old YAML and venue JSON. Nonempty conflicting legacy values fail with guidance. Retain your output path, provider/model, thresholds, and other operational choices.

After updating the checkout with your normal Git workflow, refresh the editable installation from the project root. Use one install command, matching the selected provider:

# API and local providers:
.venv/bin/python -m pip install -e .

# Codex provider (use instead of the command above):
.venv/bin/python -m pip install -e ".[codex]"

.venv/bin/python -m pip check
.venv/bin/freshlit doctor

This refreshes FreshLit's dependency metadata and console command without a blanket dependency upgrade. Preserve local source changes, the existing environment configuration, data, and authenticated Codex profile; there is no need to recreate them. pyproject.toml is the dependency source of truth; requirements.txt is now a compatibility entry point for the same editable installation, not a second list to maintain.

The Codex dependency is now optional; API and local-model users should use the core install. Existing provider settings remain explicit and are not migrated silently. Run freshlit configure-llm to select a different provider, and use freshlit llm-check --yes only when you intend to make the synthetic online test calls. doctor reports whether the digest destination comes from YAML, .env, or the process environment; remove an old FRESHLIT_VAULT_PATH override only when you want YAML to control output.

FreshLit launches the SDK's bundled, version-matched Codex runtime. Updating a separate global codex command does not update that runtime. Keep the SDK pin until its private-router compatibility fix has been reviewed against a newer release.

Usage

An existing PATH launcher pointing to this checkout's virtual environment can be used from any directory: freshlit run. Configuration and data are located from the source checkout, not the terminal's working directory. Bare freshlit still requires an explicit subcommand; it does not start the pipeline automatically.

# Full pipeline
.venv/bin/freshlit run

# Provider setup and explicit online compatibility check
.venv/bin/freshlit configure-llm
.venv/bin/freshlit llm-check --yes

# Useful flags
.venv/bin/freshlit run --provider codex
.venv/bin/freshlit run --provider ollama --model qwen3:8b
.venv/bin/freshlit run --model gpt-5.6-luna
.venv/bin/freshlit run --lookback 14
.venv/bin/freshlit run --limit 50
.venv/bin/freshlit run --max-candidates 20
.venv/bin/freshlit run --skip-llm
.venv/bin/freshlit run --from-cache
.venv/bin/freshlit run --reconsider-rejected
.venv/bin/freshlit run --dry-run
.venv/bin/freshlit run --force

Provider choices are codex, openai, openrouter, opencode, ollama, openai_compatible, and the legacy gateway alias. Other supported overrides include --output-name. --from-cache reuses data/last_ingest.json; --skip-llm stops after vector ranking. CLI provider and model options override their configured values for that run. --from-cache fails if no cache exists rather than silently fetching. Legacy caches and caches from other queries/date windows can be replayed explicitly, with a warning.

Changing the profile does not delete paper history. If FreshLit detects history from another or unknown profile/selection context, it warns. Use --reconsider-rejected to re-evaluate rejected papers present in the current ingestion or replay cache. It retains passed-paper history and within-batch deduplication; it does not fetch all previously rejected papers. Use a suitable lookback or an explicit cache replay to supply them. Mixed-history warnings remain conservative because older rejected rows are not erased.

run --dry-run skips paper-disposition updates and digest delivery, but it can still initialize the database, update ingestion/profile caches, download embeddings, and submit LLM requests. To avoid ingestion and LLM requests while testing local ranking with an existing ingestion cache, combine --from-cache --skip-llm --dry-run; this still needs the embedding model and may download its weights if they are not cached.

Output: {vault_path}/{digest_folder}/YYYY-Www.md (ISO year + week), with YAML frontmatter for Dataview, including the LLM provider, model, and safe endpoint identity used. Custom endpoint URLs are represented only by a one-way hash. At startup, FreshLit reports the resolved output directory and whether its vault root came from YAML, .env, or the process environment.

The vault must already exist. Delivery opens it as the trusted filesystem boundary, securely creates nested digest directories, and uses directory-relative atomic replacement without following swapped symlinks. In-process and advisory locks serialize FreshLit rerun merges; the content-free .freshlit-delivery.lock entry is retained in the digest folder. These guarantees require POSIX directory-descriptor operations and advisory locking; unsupported platforms fail closed. Other editors must cooperate with the lock to avoid conflicting edits.

Pipeline and LLM boundary

The generative LLM runs only after retrieval and deterministic/local reduction:

1. FreshLit builds user-authored keyword/topic queries and retrieves records from OpenAlex and, when enabled, Europe PMC.
2. SQLite history plus exact/fuzzy matching removes duplicates and prior papers.
3. Local SPECTER2 embeddings rank title/abstract records against the research profile and retain the configured candidate budget.
4. The selected LLM assigns validated relevance scores to those candidates.
5. FreshLit applies score thresholds and configured venue weights.
6. The same selected LLM extracts structured summaries for qualifying records and generates the three-item field pulse.
7. Ordinary application code renders and safely writes the Markdown digest.

The configurable LLM therefore does not choose search terms or retrieve papers. It receives retrieved metadata and abstracts, not complete paper PDFs. One model currently serves scoring and synthesis; changing it changes both stages.

Forgetting accepted papers

Deleting a Markdown digest does not reset paper history, which is stored in data/cache.db. To reconsider recently accepted papers:

.venv/bin/freshlit forget --dry-run        # preview the default seven-day window
.venv/bin/freshlit forget                  # forget acceptances from the last 7 days
.venv/bin/freshlit forget --days 14        # choose a different window
.venv/bin/freshlit forget --days 14 --dry-run

The window is the last N × 24 hours, inclusive, measured in UTC against the time each paper was recorded as passed, not its publication date. --days must be a positive integer. A passed paper was selected by filtering; this does not necessarily mean synthesis and delivery completed successfully.

The command removes those accepted records plus directly matching dropped_dedup records (same normalized DOI or the existing title/first-author fuzzy rule, using filtering.dedup_title_similarity). Those duplicate records may be older than the requested window; retaining them could otherwise keep the same papers suppressed. It does not follow chains of fuzzy matches. Older accepted papers, independent vector/LLM rejections, unrelated duplicate history, and profile-context history are retained. It warns if preserved records still match a selected paper or if acceptance timestamps cannot be interpreted safely.

Before any deletion, FreshLit creates an exclusive, owner-only SQLite backup at data/cache.backup-<UTC timestamp>-<unique suffix>.db, then deletes the selected records in one transaction. Backup failure prevents deletion; deletion failure rolls back and retains the completed backup. Missing databases and no matches are harmless no-ops without a backup. forget --dry-run is read-only and creates no backup, unlike the pipeline's run --dry-run.

If you have manually switched the database to SQLite WAL mode, preview refuses to open it: even a read-only SQLite connection can create or update WAL sidecars. FreshLit's normal rollback-journal database supports preview. Apply mode supports WAL and backs up its committed data; preview never checkpoints or changes the journal mode automatically.

forget never fetches papers, loads models, makes LLM requests, or modifies Markdown digests, the ingestion cache, or the profile vector. It needs no research profile or credentials. Do not run it concurrently with freshlit run: its database transaction is locked, but an already-running pipeline could record papers again after it completes.

After forgetting, rerun normally or reuse the cached input:

.venv/bin/freshlit run --from-cache
# If the old weekly digest still exists and should be replaced:
.venv/bin/freshlit run --from-cache --force

The papers must still be in the retrieved/cached input and pass filtering again; scores and summaries can differ. --force only controls digest replacement—it does not forget history. Keep backups if you may need to restore the previous state; restoring a full database backup also restores all other history from that snapshot, not just the forgotten papers.

Configuration files

File	Purpose
config/settings.yaml	Output paths, lookback, thresholds, LLM provider/model, timeout/retries, batching/concurrency
config/research_profile.md	User-chosen search selectors, optional venue preferences, and research context; gitignored
config/journal_tiers.json	Legacy compatibility only; new venue preferences live in the profile
data/cache.db	SQLite dedup/disposition history (state)
data/profile_exemplars.json	Generated profile vector (from build-profile)

Research profile

Retrieval is not AI-generated. You select the keywords and topic IDs. FreshLit constructs API queries deterministically: quoted keywords joined by OR, plus a separate OpenAlex topic query when topics are selected. These result streams are combined, not intersected. Europe PMC is an explicit opt-in for biomedical coverage; it uses the same keywords. Topic-only profiles work with OpenAlex, but Europe PMC needs keywords to search.

The profile's YAML front matter contains structured choices; the Markdown body contains the keywords and research context. For example:

---
version: 1
search:
  openalex_topics: []
  europe_pmc:
    enabled: false
    sources: [PPR]
venue_weights: []
---
# Research Profile

## Keywords
- exoplanet atmospheres
- transmission spectroscopy

## Research Description
I study exoplanet atmospheres using observational spectroscopy. Include new
observational constraints and transferable retrieval methods. Exclude papers
concerned only with Solar System missions.

Optional topic lookup uses OpenAlex directly, not an LLM, and never edits your profile automatically:

.venv/bin/freshlit topics "exoplanet atmospheres"

Copy selected {id, label} entries from its output into search.openalex_topics. IDs control retrieval; labels are for you. Canonical OpenAlex topic URLs are also accepted. Optional venue_weights entries have issn, weight, and an optional name; unlisted venues use filtering.default_journal_weight (normally 1.0).

The Markdown body drives the vector profile and both relevance/synthesis prompts. The AI evaluates retrieved papers against your stated interests and exclusions; it does not choose initial search terms. Front matter is excluded from embedding and LLM text so changing a display label cannot change ranking. For compatibility, embedding inputs retain the complete Markdown body and a separate keyword text; editing that body can therefore change ranking. Never put credentials in it.

Both ## Keywords and ## Research Description are required for new profiles. Use - keyword bullets; provide at least one keyword or topic and a nonblank research description. The unfinished example is never used as a fallback.

Tuning

• Research focus — edit config/research_profile.md; the next run detects stale vectors and rebuilds them. build-profile can refresh explicitly.
• Selectivity — filtering.vector_threshold (floor), score_cutoff (qualify at >= 7.0), max_llm_candidates (LLM-scoring budget).
• Venue weighting — edit the optional named venue_weights in the profile.
• LLM latency — llm.model, filtering.llm_batch_size, llm.workers, llm.timeout_seconds, and llm.max_retries. The CLI can override the model, provider, and candidate count without changing configuration.

Scheduling

FreshLit does not install a background service. After a successful manual run, schedule the checkout's absolute .venv/bin/freshlit run path using cron on Linux/macOS or launchd on macOS. For example, a cron entry running Mondays at 08:00 (scheduler-local time) is:

0 8 * * 1 /absolute/path/to/FreshLit/.venv/bin/freshlit run >> /absolute/path/to/freshlit.log 2>&1

Quote executable/log paths if they contain spaces. Configuration is located from the source checkout, not the scheduler's working directory. For Codex, run as the same user who owns the private Codex home. Ensure no overlapping runs and arrange log rotation. Ingestion dates use ingestion.timezone (UTC by default), independently of the scheduler's timezone. Moving the checkout may require recreating its virtual environment and updating the scheduled executable path.

LLM providers and security model

Shared guarantees

• Pipeline nodes use one provider-neutral structured-generation interface. Only the explicitly selected provider, endpoint, credential, and model are built. FreshLit never silently changes any of them, so a failure cannot redirect research text or billing to another service.
• A malformed batch may be retried paper by paper against the same client. Structured-output mode negotiation may try JSON, tool-call, and parse-only JSON formats against the same endpoint. Neither behavior is provider/model fallback.
• Paper titles, venues, abstracts, and intermediate summaries are marked as untrusted data. Every response must pass strict Pydantic validation and the batch ID/cardinality contracts before entering the pipeline.
• Only the effective provider's credential variable is retrieved. Fixed presets ignore stale endpoint/auth overrides, preventing their credentials from being redirected. Custom URLs reject embedded credentials, query strings, fragments, and cleartext HTTP except on loopback.
• API clients use a direct environment-independent transport: ambient SDK headers, organization/project settings, and HTTP proxy variables are ignored; redirects are not followed; and a final canonical-origin/header check sends only the explicitly selected key (or no authorization header for no-auth endpoints). Proxy configuration is not currently supported.
• API/local adapters are ordinary inference clients: they expose no local file, shell, MCP, agent, or tool-execution capability. Provider exceptions, response bodies, prompts, URLs, and keys are replaced by bounded sanitized errors, and raw SDK/transport diagnostic logger trees are disabled.

There is no automatic provider fallback or hidden billed API route. Explicit same-model retries remain bounded by llm.max_retries and the request timeout.

Provider presets

Provider	Protocol and endpoint policy	Default credential
codex	Hardened Codex app-server using an existing ChatGPT subscription login	Dedicated Codex profile
openai	Fixed official OpenAI Chat Completions API root	OPENAI_API_KEY
openrouter	Fixed OpenRouter OpenAI-compatible root	OPENROUTER_API_KEY
opencode	Fixed OpenCode Go OpenAI Chat Completions root	OPENCODE_GO_API_KEY
ollama	Fixed 127.0.0.1 Ollama OpenAI-compatible root	None
openai_compatible	Explicit validated HTTPS or loopback HTTP root	FRESHLIT_LLM_API_KEY, configurable by name

Chat Completions is an API request/response format, not a measure of model quality. It is sufficient for FreshLit's text-analysis tasks, but a model exposed only through OpenAI Responses, Anthropic Messages, Gemini, or another native API needs a future adapter. In particular, choosing opencode does not make an OpenCode model on a Responses-only route compatible; use llm-check --yes with the exact endpoint/model combination.

A custom no-auth loopback server can be configured explicitly:

.venv/bin/freshlit configure-llm \
  --provider openai_compatible \
  --model local-model \
  --base-url http://localhost:8000/v1 \
  --auth none

Remote custom endpoints require HTTPS. FreshLit records only a stable hash of a custom endpoint in history/digest provenance, never its raw URL or credential.

OpenAI-compatible response-mode negotiation accepts Pydantic validation failures or HTTP 400/404/422 errors whose bounded structured metadata identifies a mode-specific field such as response_format or tool_choice. Generic request, model, endpoint, authentication, rate-limit, and transport failures stop after one HTTP attempt. An explicit parse-only retry policy prevents older Instructor versions from retrying those terminal failures internally. Creating this client suppresses Instructor's raw retry diagnostics process-wide because they can contain provider response details; FreshLit's sanitized logs remain enabled.

Optional hardened Codex adapter

• codex and gpt-5.6-luna remain the checked-in defaults for existing users. Production startup forces ChatGPT login and the built-in OpenAI provider, then fails closed unless the effective safety configuration, account, and model pass preflight. The app-server starts through an allowlisted, empty environment, so inherited credentials cannot select an API-key-backed billed OpenAI API path.
• Each run uses one official Codex runtime and fresh ephemeral threads in fresh temporary working directories outside the repository. Approval is deny-all, the sandbox is read-only, and tools, shell execution, network/web search, MCP, agents, skills, hooks, apps, memories, history persistence, telemetry, feedback, and update checks are explicitly disabled. Temporary request directories are deleted and the shared runtime is closed when the LLM context exits.
• The low-level client explicitly declines approval requests rather than using the SDK's permissive default. A local compatibility override buffers early turn-completion notifications and replays pending events atomically. This override depends on the pinned SDK's private router and must be reviewed and regression-tested before upgrading openai-codex.
• llm.timeout_seconds is a total deadline for each structured Codex request, including startup and retries. An over-deadline turn is interrupted; if bounded cancellation fails, that runtime is terminated and permanently marked unusable. Successful cancellation fails only that request. The hardened launcher requires POSIX and the standard /usr/bin/env utility.

Legacy gateway migration

Existing provider: "gateway", custom base_url, and OPENCODE_GO_API_KEY configuration remain accepted as an explicit API-key-only compatibility alias; auth: none is rejected. The alias is never selected automatically. New configurations should use a named preset or openai_compatible; a gateway failure cannot fall back to Codex and a Codex failure cannot invoke the gateway.

Notes

• LLM latency varies by model and batch content; bounded concurrency and batching keep weekly runs practical without weakening request isolation.
• OpenAlex anonymous requests have a small daily budget; add an API key for reliable weekly runs.