A zero-cost, automated daily intelligence pipeline that collects, filters, classifies and ranks high-value developments across economics, markets, business, geopolitics, AI, technology, startups, Europe, Italy and the Milan/Bocconi ecosystem.
Instead of manually checking many separate publications and institutions every day, the system produces one bounded Markdown report containing the most relevant developments, their sources, relevance scores and source-provided context.
The system runs automatically every day and stores each generated intelligence report in the public repository.
View the latest daily intelligence reports
Open the most recent dated report to see the current output.
Each report includes the selected developments, original sources, relevance scores, source-provided context, and source-health information for that run.
The project is still evolving. If you use the reports, critical feedback is especially useful — including missing sources, weak classifications, ranking issues, information gaps, or changes that would make the system more useful.
This project was built with substantial AI assistance, but it deliberately does not use an AI-heavy production architecture.
The current pipeline relies on deterministic Python logic for filtering, classification, ranking, deduplication and report generation. More complex components such as agents, RAG, embeddings, vector databases or machine-learning ranking are not excluded; they are deferred until real use demonstrates a problem they would meaningfully solve.
The principle is simple:
Use AI as leverage, but let the problem earn the complexity.
For this project, explainability, source transparency, zero recurring monetary cost, maintainability and predictable behaviour currently matter more than architectural sophistication.
The system is therefore intended to evolve from evidence rather than from feature accumulation.
This README is intentionally divided into two parts.
A short overview for anyone who wants to understand:
- what the system is;
- why it exists;
- what it produces;
- what information it covers;
- what makes the project different.
If you only want the high-level explanation, Part I is enough.
A technical deep dive covering:
- repository structure;
- configuration;
- collection;
- normalisation;
- validation;
- time-window filtering;
- deduplication;
- classification;
- ranking;
- storage;
- report generation;
- automation;
- failure handling;
- testing;
- local execution;
- design boundaries.
Important professional information is distributed across many separate places:
- newspapers;
- central banks;
- public institutions;
- technology companies;
- research organisations;
- startup publications;
- policy organisations;
- local and professional ecosystem sources.
Following all of these manually creates several problems:
- fragmentation — relevant information is spread across many websites;
- information overload — too much content is published to review efficiently;
- duplication — the same event may appear repeatedly;
- uneven source quality — primary evidence, reporting and low-value material are mixed together;
- weak prioritisation — most feeds are not tailored to this project's information needs;
- thin context — a headline alone often does not explain what actually happened;
- limited historical memory — useful developments are easily forgotten after daily reading.
The Daily Intelligence System turns this fragmented workflow into one repeatable process.
Every day, the system automatically:
collects public structured sources
→ normalises article metadata
→ removes invalid or out-of-window records
→ reduces exact duplicates
→ classifies useful stories into domains
→ calculates transparent relevance scores
→ selects a bounded set of stories
→ generates a Markdown intelligence report
→ stores the underlying structured records
→ records the operational run status
→ persists the outputs through GitHub
The intended user workflow is:
open one report
→ understand the important developments
→ selectively open original sources when deeper reading is useful
rather than:
check many websites
→ scan many headlines
→ manually decide what matters
→ repeatedly open articles just to understand the basic development
The system is designed to improve high-value awareness with negligible daily maintenance.
It can help a reader:
- stay informed across several professional domains without monitoring each source manually;
- distinguish higher-value developments from routine content;
- see where each story came from;
- understand why a story was selected;
- receive enough source-provided context to reduce unnecessary click-through;
- preserve a historical archive of processed information;
- identify gaps or failures in the information workflow rather than silently assuming coverage is complete.
The system is intentionally selective.
A short report containing only genuinely useful developments is preferable to a large report filled with low-value material.
The system currently monitors ten strategic domains:
- Global Politics and Geopolitics
- Economics and Macroeconomics
- Financial Markets
- Companies and Corporate Strategy
- Artificial Intelligence
- Technology and Software
- Startups and Venture Capital
- Europe and the European Union
- Italy
- Milan and Bocconi Ecosystem
All ten domains are implemented.
They are not equally mature, and the project does not claim comprehensive coverage.
Known residual gaps include:
- broader global Companies and Corporate Strategy reporting;
- Financial Markets coverage beyond monetary-policy and rates evidence;
- independent AI and technology reporting;
- independent European economic-policy interpretation;
- Startups/VC diversification;
- Milan/Bocconi recruiting, employer-event and established-company coverage.
These gaps are documented rather than hidden.
The current production system uses thirteen public RSS sources:
- BBC News World
- BBC News Business
- European Central Bank
- European Commission Highlighted News
- Istat Press Releases
- OpenAI News
- Tech.eu
- Tech Europe Foundation
- Federal Reserve Board Monetary Policy
- MIMIT News
- Lavoce.info Imprese
- Google DeepMind News
- ISPI Geoeconomics
The source universe is deliberately curated.
A source is not added merely because it is prestigious or technically accessible.
It must contribute a useful information function while remaining compatible with:
- zero recurring cost;
- automated access;
- public-repository persistence;
- source transparency;
- reasonable maintenance;
- copyright and credential boundaries.
A normal report contains:
- report date;
- generation timestamp;
- monitored time window;
- run status;
- source-health summary;
- number of collected items;
- number of displayed items;
- domain sections;
- story headline and original link;
- source;
- publication time;
- relevance score;
- secondary domains when relevant;
- bounded source-provided context.
Example structure:
## Artificial Intelligence
### Story headline
Source: Example Source
Published: ...
Relevance score: 7
Source context: Source-provided description explaining the core development.
The system does not generate independent AI summaries.
Source context is explicitly source-provided metadata.
The report is designed to provide enough initial context to understand the core development when the source metadata permits it.
Current source-context behaviour:
description available and <= 500 characters
→ render unchanged
description > 500 characters
→ prefer a complete sentence within the limit
→ otherwise truncate at a word boundary
description missing
or description duplicates the title
→ show an explicit no-context fallback
Fallback text:
No additional source-provided context available.
The report-context limit is currently:
500 characters
The system deliberately does not use:
- article-body scraping;
- generic RSS body-content ingestion;
- LLM summarisation;
- generated filler text.
If the source provides weak metadata, that limitation remains visible.
Current validated MVP checkpoint:
13 active public RSS sources
10 active domains
daily GitHub Actions automation
zero recurring monetary cost
deterministic classification
deterministic ranking
JSONL historical storage
Markdown daily reports
structured JSON run summaries
source-level failure isolation
122 automated tests passed at the 2026-08-19 checkpoint
The core pipeline is operational.
Current development is evidence-driven rather than feature-driven: new complexity should be introduced only when real report use demonstrates a meaningful limitation.
The project follows several fixed principles.
The core system does not require:
- paid news APIs;
- paid automation platforms;
- paid hosting;
- OpenAI API credits;
- recurring GitHub AI or Copilot credits.
The production pipeline uses ordinary deterministic Python logic for:
- filtering;
- classification;
- ranking;
- deduplication;
- reporting.
Preferred inputs are:
RSS / Atom
→ official free APIs
→ other explicitly permitted structured public endpoints
Every displayed story retains a direct link to its original source.
The production repository must not contain:
- credentials;
- authenticated premium article bodies;
- private Bocconi systems;
- private email;
- restricted database exports;
- private Career OS content.
Normal daily operation should require no manual report construction or source checking.
The project deliberately avoids introducing:
- agents;
- RAG;
- embeddings;
- vector databases;
- machine-learning ranking;
- a cloud database;
- a complex frontend;
unless real evidence later demonstrates a need.
PUBLIC STRUCTURED SOURCES
│
▼
COLLECTION
│
▼
NORMALISATION
│
▼
VALIDATION
│
▼
24-HOUR FILTER
│
▼
DEDUPLICATION
│
▼
CLASSIFICATION
│
▼
RANKING
│
▼
STRUCTURED STORAGE
│
▼
REPORT SELECTION
│
▼
MARKDOWN REPORT
│
▼
GITHUB HISTORY
Technical deep dive
Everything above is sufficient if you only wanted to understand what the project does and why it exists.
The following sections explain the complete implementation and production workflow.
The application pipeline is:
Configuration
→ Collection
→ Normalisation
→ Validation
→ Collection-window filtering
→ Exact deduplication
→ Classification
→ Ranking
→ Processed-record storage
→ Report selection
→ Report rendering
→ Run-summary generation
→ Output persistence
Production wraps the same application pipeline with GitHub Actions:
GitHub trigger
→ checkout repository
→ set up Python
→ install dependencies
→ run automated tests
→ run production pipeline
→ validate generated outputs
→ stage generated output directories
→ detect whether anything changed
→ create bot commit when needed
→ push
There is no separate local pipeline and production pipeline.
The same Python application is used in both environments.
The main repository structure is:
Daily-Intelligence-System/
│
├── .github/
│ └── workflows/
│ └── daily-intelligence.yml
│
├── config/
│ ├── domains.yaml
│ ├── settings.yaml
│ └── sources.yaml
│
├── data/
│ ├── processed/
│ └── runs/
│
├── docs/
│ └── project/
│
├── reports/
│ └── daily/
│
├── src/
│ └── daily_intelligence/
│ ├── __init__.py
│ ├── cli.py
│ ├── classify.py
│ ├── collect.py
│ ├── config.py
│ ├── deduplicate.py
│ ├── filter_window.py
│ ├── models.py
│ ├── normalize.py
│ ├── pipeline.py
│ ├── rank.py
│ ├── report.py
│ ├── run_summary.py
│ ├── storage.py
│ └── validate.py
│
├── tests/
│
├── LICENSE
├── pyproject.toml
└── README.md
The main responsibilities are:
| Path | Responsibility |
|---|---|
config/ |
Sources, domains, ranking and report configuration |
src/daily_intelligence/ |
Production Python pipeline |
tests/ |
Automated deterministic tests and controlled fixtures |
data/processed/ |
Historical processed article records |
data/runs/ |
Structured operational run summaries |
reports/daily/ |
Human-readable daily intelligence reports |
.github/workflows/ |
Scheduled and manual production automation |
docs/project/ |
Canonical project/product/architecture documentation |
The project separates configuration from processing logic.
This allows many source, domain and ranking changes to be made without rewriting pipeline code.
Defines the production source registry.
Typical source configuration includes:
id
name
feed_url
source_type
source_tier
default_domains
language
geographic_scope
active
A source may have one or more default domains when its entire information function genuinely supports that classification.
Examples:
OpenAI News
→ Artificial Intelligence
Google DeepMind News
→ Artificial Intelligence
Federal Reserve Monetary Policy
→ Economics and Macroeconomics
MIMIT News
→ Italy
Lavoce.info Imprese
→ Italy
Tech Europe Foundation
→ Milan and Bocconi Ecosystem
Broad sources such as BBC, Tech.eu or ISPI Geoeconomics do not receive blanket defaults simply because they often publish stories in certain domains.
Defines the topic taxonomy.
Each domain contains:
id
name
keywords
active
Most domains use deterministic lexical evidence.
Some domains intentionally contain no keywords.
For example:
Italy
Milan and Bocconi Ecosystem
can be classified through validated source defaults instead of generic words that would create excessive false positives.
Defines ranking and report behaviour.
Current ranking settings conceptually implement:
Tier 1 = 4 points
Tier 2 = 3 points
Tier 3 = 2 points
Tier 4 = 1 point
domain match = 2 points
keyword match = 1 point
Current report limits:
maximum items per domain = 5
maximum total items = 30
maximum source context = 500 characters
These are upper bounds, not targets.
The article is the core production record.
The pipeline normalises collected feed entries into a consistent ArticleRecord.
Conceptually, a processed record contains information such as:
record identity
source identity
title
article URL
publication timestamp
retrieval timestamp
normalized description
assigned domains
matched keywords
relevance score
The system intentionally keeps one article-oriented production model.
It does not currently maintain separate models for:
- events;
- opportunities;
- application deadlines;
- statistical releases;
- enriched AI summaries.
Those would require separate validated product needs.
The collector retrieves configured RSS or Atom feeds.
The collection layer handles:
- HTTP/HTTPS retrieval;
- explicit request headers;
- RSS/Atom parsing;
- redirects;
- source-level result reporting;
- source-level failure isolation.
Current network behaviour includes:
10-second request timeout
normal TLS verification
explicit User-Agent
explicit Accept header
Each source produces a structured outcome.
A source may therefore be:
success
empty
failed
without automatically terminating the entire pipeline.
External feeds can fail independently because of:
- network errors;
- HTTP errors;
- endpoint changes;
- malformed feeds;
- temporary publisher outages.
The system is designed so that one failed source does not normally destroy all successful information collected from other sources.
RSS feeds expose metadata in slightly different shapes.
Normalisation converts those differences into one consistent article representation.
The normaliser handles:
- title extraction and cleanup;
- article URL cleanup;
- publication timestamps;
- description extraction;
- HTML-to-text conversion;
- whitespace cleanup;
- deterministic record identity;
- preservation of source metadata.
Publication timestamps are converted to timezone-aware UTC values when suitable structured timestamps exist.
Some RSS descriptions contain markup.
The generic normalisation path is:
feed description
→ HTML parser
→ visible text extraction
→ whitespace cleanup
→ normalized description
The logic is generic rather than publisher-specific.
The system does not attempt to infer missing words or repair arbitrary malformed publisher text.
Validation checks whether normalized records are structurally usable.
It verifies required fields and record integrity.
A structurally valid record is not automatically a report item.
A valid record may later be:
- outside the current reporting window;
- removed as an exact duplicate;
- left unclassified;
- excluded by report limits.
This distinction is important:
valid record
≠
displayed story
The system currently uses a rolling reporting window:
actual execution time
minus the previous 24 hours
A record must contain a usable publication timestamp to enter this stage successfully.
The pipeline deliberately does not substitute retrieval time when publication time is unavailable.
That prevents an old article retrieved today from being treated as newly published.
GitHub Actions may begin later than its nominal scheduled minute.
Because the reporting window is based on actual execution time:
later GitHub start
→ slightly later 24-hour cutoff
This is a known limitation.
A fixed daily cutoff has not been introduced because the current behaviour has not demonstrated enough product harm to justify additional complexity.
The system performs deterministic exact duplicate reduction.
Current duplicate evidence includes:
normalized article URL
normalized title
When an exact duplicate is found, the first deterministic occurrence is retained.
The system does not currently perform:
- semantic similarity;
- embeddings;
- fuzzy clustering;
- multi-source story clustering.
These remain deferred until real reports show that exact deduplication is insufficient.
Classification determines which strategic domains apply to each article.
The system currently uses two evidence types:
source-default domains
+
keyword matches against title and description
A record can receive multiple domains.
Source defaults are used only when the source itself provides reliable domain evidence.
Examples:
Istat
→ Economics and Macroeconomics
OpenAI News
→ Artificial Intelligence
Google DeepMind News
→ Artificial Intelligence
Federal Reserve Monetary Policy
→ Economics and Macroeconomics
MIMIT News
→ Italy
Lavoce.info Imprese
→ Italy
Tech Europe Foundation
→ Milan and Bocconi Ecosystem
Broad publications are intentionally not assigned blanket defaults.
Other classification evidence comes from configured keywords found in:
title
+
normalized description
Keyword sets are deliberately conservative.
A new keyword is normally added only after:
- examining real missed stories;
- checking likely false positives;
- inspecting historical processed records;
- evaluating ranking consequences.
The classifier uses a simple deterministic case convention:
all-lowercase configured keyword
→ case-insensitive matching
configured keyword containing uppercase characters
→ case-sensitive matching
This became important for:
AI
because lowercase Italian:
ai
is an ordinary word.
Keeping AI uppercase avoids large numbers of false Artificial Intelligence matches in Italian-language articles.
The same principle supports:
IA
as an intentional Italian Artificial Intelligence acronym.
The system does not try to classify every collected article.
A record can remain:
unclassified
and still be retained in the processed historical dataset.
Unclassified records are omitted from the main report by default.
This is deliberate.
The system prefers:
smaller accurate report
over:
larger report created by forcing ambiguous classifications
After classification, the system calculates a deterministic relevance score.
Current formula:
source-tier score
+ 2 × number of assigned domains
+ 1 × number of matched keywords
Current source-tier scores:
| Source Tier | Score |
|---|---|
| Tier 1 | 4 |
| Tier 2 | 3 |
| Tier 3 | 2 |
| Tier 4 | 1 |
This ranking model is intentionally simple.
Its advantages are:
- transparency;
- reproducibility;
- easy testing;
- easy debugging.
A higher score does not guarantee that a story is objectively more important.
The score represents the current configured evidence available to the deterministic system.
The project deliberately avoids:
- machine-learned ranking;
- LLM relevance scoring;
- semantic embeddings;
- source-specific penalties;
- opaque publisher adjustments.
If poor source or classification evidence creates inflated scores, the preferred response is:
correct the evidence
rather than:
add another ranking exception
Processed records are stored as:
JSON Lines
under date-based paths:
data/processed/YYYY/MM/YYYY-MM-DD.jsonl
JSONL was chosen because it is:
- human-inspectable;
- simple;
- Git-friendly;
- deterministic;
- database-free;
- zero-cost.
The stored historical records also provide a regression corpus for later taxonomy changes.
For example:
proposed keyword
→ replay/search historical records
→ inspect changed classifications
→ detect false positives
→ retain or reject change
The report displays at most:
500 characters
of source context.
That does not mean the stored normalized description is capped at 500 characters.
The 500-character limit belongs to the report presentation layer.
Storage semantics remain independent.
Not every classified record is displayed.
The report uses bounded selection rules:
maximum 5 stories per primary domain
maximum 30 stories overall
These are maximums.
The system does not attempt to fill every section.
If only three stories deserve inclusion, a three-story report is acceptable.
Selection uses deterministic ordering based on factors including:
- relevance score;
- publication time;
- source tier;
- normalized title;
- record identity.
This keeps repeated processing reproducible.
A story appears once.
Its first eligible domain becomes its primary report section.
Additional domains are displayed as:
Also:
Example:
Primary:
Economics and Macroeconomics
Also:
Artificial Intelligence
This avoids repeating the same story in several sections.
Selected stories receive a bounded source-context line.
The context comes from the existing:
ArticleRecord.description
No separate enrichment field is created.
No additional source-provided context available.
No additional source-provided context available.
Render it unchanged.
The formatter:
- looks for the latest complete sentence that fits;
- returns the complete sentence when one is available;
- otherwise finds the last usable word boundary;
- appends
...when word-boundary truncation is required.
The text may be a publisher:
- summary;
- abstract;
- description;
- teaser.
The system therefore labels it:
Source context
rather than implying that it is an independently written or AI-generated summary.
Some feeds expose richer content fields containing thousands of characters.
These can behave more like article bodies than compact metadata.
Using them generically could:
- increase copyright and persistence risk;
- inject much more incidental text into classification;
- distort relevance scores;
- increase report size;
- create source-specific maintenance.
The current implementation therefore uses the existing normalized description only.
Each execution generates a machine-readable operational summary.
Stored path:
data/runs/YYYY/MM/YYYY-MM-DD.json
The summary records information such as:
- run identifier;
- timestamps;
- monitored window;
- run status;
- active sources;
- successful sources;
- failed sources;
- empty sources;
- raw items;
- valid items;
- invalid items;
- duplicates;
- displayed items;
- warnings.
This provides a machine-readable operational record separate from the human report.
pipeline.py coordinates the complete application flow.
Conceptually:
load configuration
→ collect all active sources
→ normalize successful entries
→ validate records
→ keep records inside the current publication window
→ deduplicate
→ classify
→ rank
→ write processed JSONL
→ build operational run summary
→ render Markdown report
→ persist report and run summary
The pipeline is responsible for coordinating components.
Individual modules remain focused on their own deterministic responsibilities.
The application exposes one production-equivalent command:
python -m daily_intelligence.cli runThe same command is used by GitHub Actions.
This means local testing and production execute the same application path.
Production automation is defined in:
.github/workflows/daily-intelligence.yml
The workflow supports:
manual execution
+
daily scheduled execution
Current intended schedule:
06:05 Europe/Rome
1. Check out repository
2. Set up Python 3.12
3. Install package and development/test dependencies
4. Run automated tests
5. Run the production pipeline
6. Validate that the three expected output files exist and are non-empty
7. Stage:
data/processed/
data/runs/
reports/daily/
8. Check whether generated outputs actually changed
9. If nothing changed:
stop without creating a commit
10. If something changed:
create a github-actions[bot] commit
11. Push the new outputs
The automated commit message is:
data: update daily intelligence outputs
The workflow uses a production concurrency group.
A second scheduled execution does not cancel an already-running production execution.
This reduces the risk of competing runs writing the same daily outputs.
The system distinguishes three operational states:
success
degraded
failure
All required pipeline components complete successfully.
Individual source outcomes and counts remain visible.
One or more sources may fail while enough successful information remains to create a useful report.
The successful sources continue through the pipeline.
The report/run summary makes the degraded state visible.
Critical configuration, orchestration or pipeline errors prevent a valid production result.
Critical failures should not silently publish output as successful.
The system prefers:
useful partial information
+
visible degradation
over:
discard all successful information because one external source failed
while still preserving strict failure behaviour for critical internal errors.
Each normal production run writes three main output types.
data/processed/YYYY/MM/YYYY-MM-DD.jsonl
Contains structured processed article records.
Use cases:
- historical archive;
- inspection;
- debugging;
- taxonomy regression;
- future deterministic quality analysis.
reports/daily/YYYY/MM/YYYY-MM-DD.md
Contains the human-readable intelligence report.
This is the main daily consumption layer.
data/runs/YYYY/MM/YYYY-MM-DD.json
Contains operational execution metadata.
The project uses pytest.
Tests cover important deterministic behaviour including:
- source configuration;
- domain configuration;
- settings;
- RSS fixtures;
- collection;
- normalisation;
- validation;
- publication-window filtering;
- deduplication;
- classification;
- multilingual/case-sensitive keyword behaviour;
- ranking;
- storage;
- report selection;
- source-context rendering;
- run summaries;
- pipeline integration.
At the 2026-08-19 richer-report implementation checkpoint:
20 feed-fixture tests passed
14 report tests passed
122 tests passed in the complete suite
The test count is a checkpoint, not a fixed project requirement.
Future development may change the total.
Passing tests is necessary but not sufficient.
Every meaningful product change should also inspect real output.
The project evaluates questions such as:
- Were important stories included?
- Were useful stories missed?
- Were weak stories promoted?
- Did classification become misleading?
- Did relevance scores become inflated?
- Is the report repetitive?
- Is the report too long?
- Is the report too sparse?
- Does source context actually explain the development?
- Are source-quality defects visible?
- Did a source fail silently?
- Are generated outputs understandable to a human reader?
A technically successful pipeline can still produce a poor intelligence product.
Python >= 3.12
Git
Clone the repository and move into it.
Create a virtual environment:
python -m venv .venvActivate it on macOS/Linux:
source .venv/bin/activateInstall the package and development dependencies:
python -m pip install -e ".[dev]"Run the test suite:
python -m pytestRun the production-equivalent pipeline:
python -m daily_intelligence.cli runAfter execution, inspect:
data/processed/
data/runs/
reports/daily/
A new source should not begin with configuration editing.
The preferred workflow is:
identify a real information gap
→ evaluate the source's information function
→ verify public structured access
→ verify persistence/copyright compatibility
→ test with the real collector
→ inspect normalized records
→ inspect metadata quality
→ test existing classification
→ test ranking consequences
→ determine whether keywords/defaults are justified
→ make the smallest configuration/code change
→ run targeted tests
→ run full tests
→ run production-equivalent pipeline
→ inspect generated report
→ inspect Git diff
Possible source outcomes are:
Active
Standby
Rejected
Manual/research layer
Deferred production-readiness candidate
A technically functional RSS feed can still be rejected.
Domain changes normally belong in:
config/domains.yaml
Before adding broad keywords:
- inspect real missed records;
- simulate the proposed keyword;
- search historical processed records;
- check false positives;
- inspect relevance-score effects.
Some domains may deliberately use:
keywords: []when validated source defaults provide better evidence.
The project does not optimise for maximum classification coverage.
The project prefers information-function coverage over publisher accumulation.
Useful source roles include:
primary institutional evidence
high-quality reporting
independent analysis
specialist intelligence
frontier-lab primary evidence
professional ecosystem information
Adding several publishers that all perform the same role can increase:
- duplication;
- noise;
- failures;
- maintenance;
without materially improving intelligence quality.
The smallest strong source universe is preferred.
The repository is public.
Automated production may use:
- public RSS;
- public Atom;
- official free APIs;
- approved public structured metadata.
The production system must not:
- store credentials;
- automate Bocconi authentication;
- scrape authenticated premium publications;
- bypass paywalls;
- automate
yoU@B; - automate JobGate;
- bulk-ingest licensed databases;
- store complete premium articles;
- store private email/newsletter content;
- bypass anti-bot or access-control systems.
The user may legitimately have access to premium publications through institutional services.
That does not automatically give the automated pipeline permission to retrieve or store the premium article body.
The project deliberately separates:
what a person can legitimately read
from:
what a public automated repository can retrieve and persist
The project uses AI during development and reasoning, but AI is not part of the production dependency chain.
The production problem can currently be solved through:
- structured feeds;
- deterministic rules;
- transparent ranking;
- bounded source metadata.
Adding production LLM calls would introduce:
- recurring cost risk;
- nondeterminism;
- provenance complexity;
- new failure modes;
- evaluation burden.
The current system therefore deliberately contains:
no LLM classification
no LLM ranking
no LLM summaries
no agents
no RAG
no embeddings
no vector database
These are not missing features.
They are intentionally deferred technologies whose value has not been demonstrated for the current workflow.
Current storage uses Git + JSONL + JSON + Markdown.
That is sufficient for the current scale.
Benefits:
- zero external infrastructure;
- transparent history;
- inspectable data;
- easy version control;
- no database administration;
- no recurring hosting cost.
A database should be introduced only if repository-native storage becomes a demonstrated limitation.
The current delivery interface is Markdown inside the repository.
This keeps the project:
- simple;
- transparent;
- free;
- easy to maintain.
Possible future interfaces could include:
- a stable latest-report view;
- GitHub Pages;
- GitHub Issues;
- another lightweight delivery layer.
None is justified until the existing Markdown workflow creates meaningful recurring friction.
The current system deliberately accepts several limitations.
All ten domains exist, but some are stronger than others.
Different headlines describing the same event may still appear separately.
The 24-hour window moves with actual execution time.
Some publishers provide little or no source context.
Malformed source-provided descriptions may remain visible.
The system does not yet identify structured companies, people or institutions.
The system does not separately classify:
- analysis;
- news;
- event;
- opportunity;
- press release.
Publication date is not always equivalent to application deadline or event date.
Structured statistical databases are not currently converted into synthetic intelligence events.
The system does not use semantic similarity or embeddings.
These are tracked limitations, not automatic implementation tasks.
The project evolves through a controlled evidence loop:
observe real problem
→ identify exact cause
→ test the simplest explanation
→ choose smallest sufficient change
→ implement
→ run targeted validation
→ run full validation
→ inspect real output
→ inspect Git diff
→ commit only after evidence supports the change
The project deliberately avoids building a large speculative roadmap.
A future feature should answer:
- What real problem does it solve?
- Has the problem actually occurred?
- Can configuration solve it?
- Can a simpler deterministic change solve it?
- What maintenance burden does it add?
- What new failure modes appear?
- Does it preserve zero recurring monetary cost?
- Does it preserve negligible daily manual work?
- Does it preserve source transparency?
- How will success be validated?
The major completed phases are:
Phase 0
Project definition and repository setup
→ complete
Phase 1
Local vertical pipeline
→ complete
Phase 2
Real-source production readiness
→ complete
Phase 3
GitHub Actions production automation
→ complete
Phase 4
Source and domain correction / expansion
→ complete for current MVP boundary
Phase 5
Richer-report product design
→ complete
Phase 6
Richer-report implementation and evaluation
→ implemented and locally validated
The current system is therefore operational rather than a prototype awaiting its first complete pipeline.
Future development should be driven by evidence from normal report use.
The following decisions are intentionally preserved:
Python
→ core implementation language
RSS / Atom
→ primary source-delivery mechanism
YAML
→ source/domain/settings configuration
ArticleRecord
→ core production data model
deterministic rules
→ classification and ranking
JSONL
→ processed historical records
JSON
→ operational run summaries
Markdown
→ daily human-readable reports
GitHub Actions
→ production automation
Git repository
→ history and persistence
No additional infrastructure is required for the current MVP.
This README explains the repository for an external or first-time reader.
The detailed project decisions remain in the canonical project documents.
Defines:
- why the project exists;
- strategic scope;
- hard constraints;
- success criteria;
- major non-goals.
Defines:
- required user behaviour;
- report requirements;
- information coverage requirements;
- richer-context acceptance criteria;
- product boundaries.
Defines:
- component responsibilities;
- processing architecture;
- configuration architecture;
- automation architecture;
- persistence;
- failure behaviour;
- technical gates;
- deferred architecture.
Defines:
- domain taxonomy;
- source-selection policy;
- source tiers;
- keyword/default-domain policy;
- access and copyright rules;
- source-audit conclusions;
- metadata policy.
Defines:
- completed phases;
- latest validation state;
- current checkpoint;
- deferred work;
- next development action.
The repository and these documents together are the source of truth.
This repository is licensed under the terms of the included LICENSE file.
The Daily Intelligence System is intentionally a small, transparent intelligence pipeline rather than a sophisticated AI application.
Its current production model is:
13 curated public RSS sources
→ deterministic collection
→ normalization
→ validation
→ rolling-window filtering
→ exact deduplication
→ deterministic domain classification
→ deterministic relevance ranking
→ repository-native storage
→ bounded source-context rendering
→ daily Markdown report
→ GitHub Actions persistence
Its design philosophy is:
useful before sophisticated
deterministic before AI
structured sources before scraping
transparent before opaque
evidence before complexity
The system is designed to make daily information consumption more selective, reproducible and useful while preserving:
zero recurring monetary cost
negligible daily manual work
public-source transparency
public-repository safety
simple maintainable architecture