diff --git a/README.md b/README.md index 8c3338d..aa47727 100644 --- a/README.md +++ b/README.md @@ -4,47 +4,55 @@ A mobile-first, browser-only tracker for public X timelines, styled and structur Live dashboard: -## How the free collector works - -X's free embedded timeline is unreliable and frequently returns HTTP 429. X's official data API is pay-per-use. This project therefore provides an optional logged-in-browser collector: - -1. Install [Tampermonkey](https://www.tampermonkey.net/). -2. **Allow Chrome to execute userscripts:** - - Right-click Tampermonkey → **Manage extension**. - - Turn on **Allow User Scripts**. Chrome 138+ requires this even when Tampermonkey and the script both appear enabled. - - If the toggle is unavailable, enable **Developer mode** at `chrome://extensions`. - - Set Tampermonkey site access to **On all sites**, or at least allow `x.com`. - - See [Tampermonkey FAQ Q209](https://www.tampermonkey.net/faq.php?locale=en#Q209). -3. Install or update [`collector.user.js`](https://daggerok.github.io/twitter/collector.user.js). Confirm version **1.1.0**. -4. Log into X normally and open [`@I_Am_The_ICT` Posts & replies](https://x.com/I_Am_The_ICT/with_replies). -5. Hard-reload the X tab after installation: **Ctrl+Shift+R** on Windows/Linux or **Cmd+Shift+R** on macOS. -6. Confirm that the floating **𝕏 ICT Collector** panel appears at the bottom-right: - - **Start Manual** records rendered tweet cards while you scroll. - - **Start + Auto** performs slow, capped assisted scrolling. - - **Pause** stops immediately. -7. Choose **Copy New JSON** or **Download JSON**. -8. Return to the dashboard and use **📋 Paste** or the **📄 JSON import** dialog. - -No X password, session cookie, authorization token, or private browser storage is copied to the dashboard. The collector observes only tweet cards rendered in the current X page. +## Free logged-in Chrome collector + +X's free embedded timeline is unreliable and frequently returns HTTP 429. X's official data API is pay-per-use. This project therefore provides an unpacked Chromium extension that collects tweet cards rendered in your already logged-in X tab. + +The Chrome extension replaces the earlier Tampermonkey userscript. + +## Install the extension + +1. Disable or remove the old **daggerok X Timeline Collector** userscript from Tampermonkey. +2. Download [`twitter-collector-extension.zip`](https://daggerok.github.io/twitter/twitter-collector-extension.zip). +3. Unzip it. The extracted `extension` folder must directly contain `manifest.json`. +4. Open `chrome://extensions`. +5. Enable **Developer mode** in the top-right. +6. Click **Load unpacked**. +7. Select the extracted `extension` folder—not the ZIP and not the repository root. +8. Pin **daggerok X Timeline Collector** to the browser toolbar. + +The source is also available in [`extension/`](./extension/). + +## Daily usage + +1. Log into X normally and open [`@I_Am_The_ICT` Posts & replies](https://x.com/I_Am_The_ICT/with_replies). +2. Click the **daggerok X Timeline Collector** extension icon while the X tab is active. +3. The popup shows whether the collector content script is loaded. It automatically injects/reinjects the script when needed. +4. Choose: + - **Show panel** — display/reopen the floating controls. + - **Start Manual** — collect rendered tweet cards while you scroll. + - **Start Auto** — perform slow, capped assisted scrolling. + - **Pause** — stop immediately. +5. Choose **Copy New JSON** in the popup or floating panel, or download JSON from the panel. +6. Return to and use **📋 Paste** or the **📄 JSON import** dialog. + +No X password, session cookie, authorization token, or private X browser storage is exported. The extension observes only tweet cards rendered in the current X page. ## If the panel is missing Check these in order: -1. Tampermonkey's global status is **Enabled**. -2. The `daggerok X Timeline Collector` toggle is green. -3. Chrome's Tampermonkey extension details have **Allow User Scripts** enabled. -4. Tampermonkey site access allows `https://x.com/*`. -5. The installed collector version is **1.1.0**. -6. Hard-reload the X tab after installing or updating the script. -7. Open Tampermonkey on the X tab and run **Show / reopen ICT Collector panel**. -8. In DevTools Console, successful execution logs: +1. The unpacked extension is enabled at `chrome://extensions`. +2. The folder loaded into Chrome directly contains `manifest.json`. +3. The extension's site access permits `https://x.com/*`. +4. You are on the X profile tab—not the dashboard—when opening the extension popup. +5. Click **Show panel**. The popup attempts a direct content-script reinjection when no receiver is detected. +6. Reload the X tab once after first installing or reloading the unpacked extension. +7. Open DevTools Console and look for: ```text - [daggerok X Collector] userscript v1.1.0 loaded + [daggerok X Collector] Chrome extension v1.0.0 loaded ``` -The collector now mounts its panel before loading Tampermonkey storage, so a storage error is displayed in the panel rather than causing an invisible startup failure. - ## Dashboard features - Defaults to `@I_Am_The_ICT`, but supports other public profile handles. @@ -64,7 +72,7 @@ Assisted scrolling is deliberately conservative: - configurable scroll cap, defaulting to 80 - stops after repeated scans with no new rendered posts - stops after reaching posts seen in an earlier collector run -- stores only post IDs in Tampermonkey storage to recognize previous captures +- stores only post IDs in `chrome.storage.local` to recognize previous captures ## Important limitations and Terms @@ -81,4 +89,7 @@ Open **⚙️ Settings** in the dashboard and expand **📋 Google Sheets Setup ## Files - [`index.html`](./index.html) — dashboard, cache, classification, filtering, CSV, and Google Sheets sync -- [`collector.user.js`](./collector.user.js) — Tampermonkey companion that observes rendered X tweet cards +- [`extension/manifest.json`](./extension/manifest.json) — Manifest V3 extension configuration +- [`extension/content.js`](./extension/content.js) — collector injected into X profile pages +- [`extension/popup.html`](./extension/popup.html) and [`popup.js`](./extension/popup.js) — toolbar controls and reinjection diagnostics +- `twitter-collector-extension.zip` — downloadable unpacked-extension package diff --git a/extension/README.md b/extension/README.md new file mode 100644 index 0000000..17fe186 --- /dev/null +++ b/extension/README.md @@ -0,0 +1,23 @@ +# daggerok X Timeline Collector — Chrome Extension + +## Install + +1. Download and unzip `twitter-collector-extension.zip` from the dashboard/repository. +2. Open `chrome://extensions` (or your Chromium browser's extension-management page). +3. Enable **Developer mode**. +4. Click **Load unpacked**. +5. Select the unzipped `twitter-collector-extension` folder containing `manifest.json`. +6. Pin **daggerok X Timeline Collector** to the toolbar. + +## Use + +1. Open and log in normally. +2. Click the extension toolbar icon. +3. Click **Show panel**, **Start Manual**, or **Start Auto**. +4. Keep the X tab visible while collecting. +5. Use **Copy New JSON** in either the extension popup or floating panel. +6. Open and press the purple clipboard button. + +The popup automatically injects `content.js` if X's single-page navigation or browser startup prevented the manifest content script from loading. + +No password, session cookie, authorization token, or private X storage is exported. diff --git a/collector.user.js b/extension/content.js similarity index 84% rename from collector.user.js rename to extension/content.js index 8e1fca3..d6c3d3b 100644 --- a/collector.user.js +++ b/extension/content.js @@ -1,27 +1,12 @@ -// ==UserScript== -// @name daggerok X Timeline Collector -// @namespace https://daggerok.github.io/twitter/ -// @version 1.1.0 -// @description Collect tweet cards rendered while you browse a public X profile; exports JSON for daggerok/twitter. -// @author daggerok -// @homepageURL https://daggerok.github.io/twitter/ -// @supportURL https://github.com/daggerok/twitter/issues -// @updateURL https://daggerok.github.io/twitter/collector.user.js -// @downloadURL https://daggerok.github.io/twitter/collector.user.js -// @match https://x.com/* -// @match https://twitter.com/* -// @run-at document-idle -// @noframes -// @grant GM_getValue -// @grant GM_setValue -// @grant GM_deleteValue -// @grant GM_setClipboard -// @grant GM_registerMenuCommand -// ==/UserScript== - -(function () { +(() => { 'use strict'; + if (globalThis.__DAGGEROK_X_COLLECTOR_EXTENSION_LOADED__) { + globalThis.__DAGGEROK_X_COLLECTOR_EXTENSION_API__?.showPanel('Panel reopened after reinjection'); + return; + } + globalThis.__DAGGEROK_X_COLLECTOR_EXTENSION_LOADED__ = true; + const SCHEMA = 'daggerok-x-collector/v1'; const DEFAULT_MAX_SCROLLS = 80; const DEFAULT_DELAY_MS = 2500; @@ -42,12 +27,22 @@ let lastReason = 'Ready'; let ui = null; - const gmGet = async (key, fallback) => typeof GM_getValue === 'function' ? await Promise.resolve(GM_getValue(key, fallback)) : fallback; - const gmSet = async (key, value) => { if (typeof GM_setValue === 'function') await Promise.resolve(GM_setValue(key, value)); }; + const storageGet = (key, fallback) => new Promise(resolve => { + chrome.storage.local.get([key], result => { + if (chrome.runtime.lastError) return resolve(fallback); + resolve(result[key] ?? fallback); + }); + }); + const storageSet = (key, value) => new Promise(resolve => { + chrome.storage.local.set({ [key]: value }, () => resolve()); + }); + const storageRemove = key => new Promise(resolve => { + chrome.storage.local.remove(key, () => resolve()); + }); const knownKey = () => `known:${(targetHandle || 'unknown').toLowerCase()}`; - console.info('[daggerok X Collector] userscript v1.1.0 loaded', location.href); - document.documentElement.dataset.daggerokXCollector = '1.1.0'; + console.info('[daggerok X Collector] Chrome extension v1.0.0 loaded', location.href); + document.documentElement.dataset.daggerokXCollectorExtension = '1.0.0'; function getProfileHandle() { const part = location.pathname.split('/').filter(Boolean)[0] || ''; @@ -141,14 +136,14 @@ } async function loadKnown() { - const values = await gmGet(knownKey(), []); + const values = await storageGet(knownKey(), []); knownIds = new Set(Array.isArray(values) ? values.map(String) : []); initialKnownIds = new Set(knownIds); } async function persistKnown() { const values = Array.from(knownIds); - await gmSet(knownKey(), values.slice(Math.max(0, values.length - MAX_KNOWN_IDS))); + await storageSet(knownKey(), values.slice(Math.max(0, values.length - MAX_KNOWN_IDS))); } async function scanVisible() { @@ -251,8 +246,17 @@ return; } const json = JSON.stringify(payload(), null, 2); - if (typeof GM_setClipboard === 'function') GM_setClipboard(json, 'text'); - else await navigator.clipboard.writeText(json); + try { + await navigator.clipboard.writeText(json); + } catch { + const textarea = document.createElement('textarea'); + textarea.value = json; + textarea.style.cssText = 'position:fixed;left:-9999px;top:-9999px'; + document.body.appendChild(textarea); + textarea.select(); + document.execCommand('copy'); + textarea.remove(); + } lastReason = `Copied ${session.size} new post${session.size === 1 ? '' : 's'} as JSON`; updateUi(); } @@ -281,7 +285,7 @@ session.clear(); knownIds.clear(); initialKnownIds.clear(); - if (typeof GM_deleteValue === 'function') await Promise.resolve(GM_deleteValue(knownKey())); + await storageRemove(knownKey()); updateUi(); } @@ -387,12 +391,43 @@ updateUi(); } - if (typeof GM_registerMenuCommand === 'function') { - GM_registerMenuCommand('Show / reopen ICT Collector panel', () => showPanel('Panel reopened from Tampermonkey')); - GM_registerMenuCommand('Start manual collection', async () => { await showPanel('Starting manual collection'); await start(false); }); - GM_registerMenuCommand('Start assisted auto-scroll', async () => { await showPanel('Starting assisted scrolling'); await start(true); }); + function collectorStatus() { + return { + ok: true, + version: '1.0.0', + handle: targetHandle, + collecting, + autoScrolling, + captured: session.size, + scrollCount, + known: initialKnownIds.size, + reason: lastReason, + panelVisible: Boolean(document.getElementById(PANEL_ID)) + }; } + globalThis.__DAGGEROK_X_COLLECTOR_EXTENSION_API__ = { showPanel, start, stop, copyJson, downloadJson, status: collectorStatus }; + + chrome.runtime.onMessage.addListener((message, _sender, sendResponse) => { + if (!message || message.source !== 'daggerok-x-collector-popup') return false; + (async () => { + if (message.command === 'show') await showPanel('Panel opened from extension'); + else if (message.command === 'start-manual') { await showPanel('Starting manual collection'); await start(false); } + else if (message.command === 'start-auto') { await showPanel('Starting assisted scrolling'); await start(true); } + else if (message.command === 'pause') stop('Paused from extension'); + else if (message.command === 'copy') await copyJson(); + else if (message.command === 'get-json') { + const status = collectorStatus(); + status.json = session.size ? JSON.stringify(payload(), null, 2) : ''; + if (!status.json) status.error = 'No newly captured posts are available to copy yet.'; + sendResponse(status); + return; + } + sendResponse(collectorStatus()); + })().catch(error => sendResponse({ ok: false, error: error?.message || String(error) })); + return true; + }); + // Mount the UI first so a storage permission/error cannot make the collector fail invisibly. createPanel(); loadKnown().then(() => { diff --git a/extension/manifest.json b/extension/manifest.json new file mode 100644 index 0000000..42e41ee --- /dev/null +++ b/extension/manifest.json @@ -0,0 +1,33 @@ +{ + "manifest_version": 3, + "name": "daggerok X Timeline Collector", + "version": "1.0.0", + "description": "Collect tweet cards rendered in your logged-in X profile timeline and export JSON to daggerok/twitter.", + "minimum_chrome_version": "102", + "permissions": [ + "activeTab", + "clipboardWrite", + "scripting", + "storage" + ], + "host_permissions": [ + "https://x.com/*", + "https://twitter.com/*" + ], + "action": { + "default_title": "X Timeline Collector", + "default_popup": "popup.html" + }, + "content_scripts": [ + { + "matches": [ + "https://x.com/*", + "https://twitter.com/*" + ], + "js": [ + "content.js" + ], + "run_at": "document_idle" + } + ] +} diff --git a/extension/popup.html b/extension/popup.html new file mode 100644 index 0000000..d17ed82 --- /dev/null +++ b/extension/popup.html @@ -0,0 +1,24 @@ + + + + + +X Timeline Collector + + + +

𝕏 Timeline Collector

Chrome extension v1.0.0
+
+
Checking current tab…
Open an X profile page.
+
+
+
Open ICT
+ Open dashboard +
Assisted scrolling may trigger X anti-automation limits. Keep the X tab visible and pause if X warns or rate-limits you.
+
+
No passwords, cookies, or X authorization tokens are exported.
+ + + diff --git a/extension/popup.js b/extension/popup.js new file mode 100644 index 0000000..5ac14d0 --- /dev/null +++ b/extension/popup.js @@ -0,0 +1,81 @@ +(() => { + 'use strict'; + + const statusEl = document.getElementById('status'); + let activeTab = null; + + function isXUrl(url) { + try { + const host = new URL(url).hostname; + return host === 'x.com' || host === 'www.x.com' || host === 'twitter.com' || host === 'www.twitter.com'; + } catch { + return false; + } + } + + function renderStatus(status) { + if (!status?.ok) { + statusEl.innerHTML = `Collector unavailable
${status?.error || 'Open an X profile and click Show panel.'}
`; + return; + } + const mode = status.collecting ? (status.autoScrolling ? 'Assisted auto-scroll' : 'Manual collection') : 'Paused'; + statusEl.innerHTML = `${mode} · @${status.handle || 'no profile'}
${status.captured} captured · ${status.known} known · ${status.scrollCount} scrolls
${status.reason || ''}
`; + } + + async function findActiveTab() { + const tabs = await chrome.tabs.query({ active: true, currentWindow: true }); + activeTab = tabs[0] || null; + return activeTab; + } + + async function sendRaw(command) { + return await chrome.tabs.sendMessage(activeTab.id, { source: 'daggerok-x-collector-popup', command }); + } + + async function ensureCollector() { + const tab = await findActiveTab(); + if (!tab || !isXUrl(tab.url || '')) { + throw new Error('Current tab is not x.com. Open the ICT profile first.'); + } + try { + return await sendRaw('status'); + } catch { + await chrome.scripting.executeScript({ target: { tabId: tab.id }, files: ['content.js'] }); + await new Promise(resolve => setTimeout(resolve, 100)); + return await sendRaw('status'); + } + } + + async function command(name) { + try { + await ensureCollector(); + const result = await sendRaw(name); + renderStatus(result); + } catch (error) { + renderStatus({ ok: false, error: error?.message || String(error) }); + } + } + + document.getElementById('show').addEventListener('click', () => command('show')); + document.getElementById('manual').addEventListener('click', () => command('start-manual')); + document.getElementById('auto').addEventListener('click', () => command('start-auto')); + document.getElementById('pause').addEventListener('click', () => command('pause')); + document.getElementById('copy').addEventListener('click', async () => { + try { + await ensureCollector(); + const result = await sendRaw('get-json'); + if (!result.json) throw new Error(result.error || 'No newly captured posts are available to copy yet.'); + await navigator.clipboard.writeText(result.json); + result.reason = `Copied ${result.captured} new post${result.captured === 1 ? '' : 's'} as JSON`; + renderStatus(result); + } catch (error) { + renderStatus({ ok: false, error: error?.message || String(error) }); + } + }); + + ensureCollector().then(renderStatus).catch(error => renderStatus({ ok: false, error: error?.message || String(error) })); + setInterval(() => { + if (!activeTab) return; + sendRaw('status').then(renderStatus).catch(() => {}); + }, 1000); +})(); diff --git a/index.html b/index.html index dfc9b91..ed5e7e8 100644 --- a/index.html +++ b/index.html @@ -114,58 +114,62 @@

📋 Core Architecture &
-
Logged-in X Browser Collector (free companion)
+
Logged-in X Chrome Extension Collector (free companion)
- 📋 Tampermonkey Collector Setup (assisted scrolling) + 📋 Chrome Extension Setup (assisted scrolling)
-
1. Install Tampermonkey
-

Install the Tampermonkey browser extension.

+
1. Remove or disable the old Tampermonkey collector
+

Open Tampermonkey and turn off/delete daggerok X Timeline Collector. The new Chrome extension replaces it and prevents two collectors from conflicting.

-
2. Allow Chrome to execute userscripts — required
-

Right-click the Tampermonkey toolbar icon → Manage extension → turn on Allow User Scripts. On Chrome 138+, an installed/enabled script can silently do nothing until this is enabled. If that toggle is unavailable, open chrome://extensions and enable Developer mode. Also set Tampermonkey’s site access to On all sites or allow x.com.

-

Tampermonkey’s official permission instructions ↗

+
2. Download and unzip the extension
+

Download twitter-collector-extension.zip, unzip it, and locate the extracted extension folder containing manifest.json.

-
3. Install or update the collector
-

Open collector.user.js and approve Install/Update. The current script version is 1.1.0.

+
3. Load it into Chrome
+
    +
  1. Open chrome://extensions.
    2. +
  2. Enable Developer mode in the top-right.
    3. +
  3. Click Load unpacked.
    4. +
  4. Select the extracted extension folder—not the ZIP and not the repository root.
    5. +
  5. Pin daggerok X Timeline Collector to the toolbar.
    6. +
-
4. Reload X after installation
-

Click the 𝕏 toolbar button to open @I_Am_The_ICT Posts & replies, log in normally, then perform a hard reload with Ctrl+Shift+R (Windows/Linux) or Cmd+Shift+R (macOS). A userscript is not injected retroactively into a tab that was already open before installation.

+
4. Open the ICT profile and extension popup
+

Click the dashboard’s 𝕏 toolbar button, log into X normally, then click the newly installed extension icon. Its popup reports whether the collector is loaded and can inject/reinject it if necessary.

-
5. Verify the floating panel
-

A dark 𝕏 ICT Collector panel must appear at the bottom-right of the X page. If it does not, click Tampermonkey and run Show / reopen ICT Collector panel. In DevTools Console, a successful injection also logs [daggerok X Collector] userscript v1.1.0 loaded.

+
5. Start collection
+

In the extension popup click Show panel, Start Manual, or Start Auto. The dark floating panel also provides Pause, delay, scroll-cap, copy, download, and history controls. Keep the X tab visible during assisted scrolling.

-
6. Collect and transfer
-

Press Start + Auto for slow assisted scrolling, or Start Manual and scroll yourself. Keep the tab visible. When finished, press Copy New JSON, return here, and press the purple 📋 toolbar button. You can alternatively download JSON and use the 📄 import dialog.

+
6. Transfer to this dashboard
+

Press Copy New JSON in the extension popup or floating panel. Return here and press the purple 📋 toolbar button. You can alternatively download JSON and use the 📄 import dialog.

- What is collected: only tweet cards rendered in the visible X page—post ID/URL/time/text, author, visible reply/repost context, visible metrics, and media URLs. Classification is heuristic because the web page exposes less metadata than an API. + What is collected: only tweet cards rendered in the visible X page—post ID/URL/time/text, author, visible reply/repost context, visible metrics, and media URLs. The extension never exports passwords, session cookies, or X authorization tokens.
Panel still missing? Check these in order
    -
  1. Tampermonkey’s global status says Enabled.
    2. -
  2. The collector’s own toggle is green.
    3. -
  3. Allow User Scripts is enabled in Chrome’s Tampermonkey extension details.
    4. -
  4. Tampermonkey site access permits https://x.com/*.
    5. -
  5. You installed/updated collector version 1.1.0.
    6. -
  6. You hard-reloaded the X tab after installation.
    7. -
  7. Try the Tampermonkey command Show / reopen ICT Collector panel.
    8. +
  8. The unpacked extension is enabled at chrome://extensions.
    9. +
  9. You selected the folder that directly contains manifest.json.
    10. +
  10. The extension’s site access permits https://x.com/*.
    11. +
  11. You are on an X tab—not the dashboard—when opening the extension popup.
    12. +
  12. Click Show panel; the popup automatically reinjects the content script if it is missing.
    13. +
  13. Reload the X tab once after the first extension installation.
-
Important: X’s Terms prohibit scraping without express permission. Assisted scrolling can also trigger anti-automation limits. Use conservative settings, stop if X warns or rate-limits you, and proceed only if you accept the account/Terms risk.
+
Important: X’s Terms prohibit scraping without express permission. Assisted scrolling can trigger anti-automation limits. Use conservative settings and proceed only if you accept the account/Terms risk.
-
✓ No paid X data API and no proxy
-

Collection happens locally in the already logged-in X tab. The dashboard receives only the JSON you explicitly copy or import. Google Sheets authentication remains separate and uses Google’s OAuth popup.

+
✓ No paid X data API, proxy, or Tampermonkey dependency
+

Collection happens locally in the logged-in X tab through an explicitly installed Chrome extension. The dashboard receives only JSON you deliberately copy or import. Google Sheets authentication remains separate.

@@ -305,7 +309,7 @@

📋 Timeli
📄 Collector JSON Import
-
Paste JSON copied or downloaded by collector.user.js. Expected schema: daggerok-x-collector/v1.
+
Paste JSON copied or downloaded by the Chrome collector extension. Expected schema: daggerok-x-collector/v1.