feat(sources): add facebook_group connector (Camofox-based) with schema parity - #178
Open
matthewdonsemail-lab wants to merge 3 commits into
Open
Conversation
…a parity Ports the listeningkit Twitter-search listener into openmagpie-core as a built-in source kind, with twikit added as an openmagpie-core library dependency (git pin, unclecode fork) per operator direction - not vendored from listeningkit. - schema: TwitterSearchSourceSpec (query, mode Top/Latest, count) added to the _BuiltinSourceSpec union; NewTweetPayload added to FeedItemData; schema.json regenerated (82 models); --check passes. - connector: TwikitClient with the live-cookie credential chain (TWITTER_COOKIES_JSON -> TWITTER_COOKIE_AUTH_TOKEN/CT0 -> cookies file -> login fallback) so the working listeningkit cookie JSONs keep functioning; error taxonomy translated to ConnectorParseError; tests use fakes only (no live I/O). - registration: twitter_search registered in sources/registry.py + connectors/__init__.py; payload self-registers; source-kind invariant test extended to the known five. - infra: Dockerfile builder gains git for the git-pinned dep; docker-compose.override.yml (local-only, gitignored) publishes Postgres on host 5433 to coexist with the buzz stack. - validation: 9 new connector tests + invariant tests green; CLI suite 145/145; ruff + ty clean. Co-authored-by: Matthew Don <matthewdonsemail@gmail.com> Signed-off-by: Matthew Don <matthewdonsemail@gmail.com>
…pping Live X polling through the connector exposed two issues (verified against live cookies from listeningkit/backend/credentials/personal.json, no proxy, no vendor API): 1. Event loop is closed on 2nd/3rd source in a multi-source feed poll. TwikitClient built one twikit Client at import time; twikit's Client creates an httpx.AsyncClient bound to the *currently running* event loop, and TwikitClient.search() drives it with a fresh asyncio.run loop per call. The shared client died with the first loop, so later sources in the same poll failed. Fix: construct the twikit Client inside _search_async (per call, inside the running loop); cookies are still resolved once in __init__. 2. X SearchTimeline intermittently 404s with an EMPTY body (reproduced: same query succeeds on immediate retry, session-independent). That is a transient upstream flake, not a deleted tweet/user; the old not_found/non-retryable mapping made ops treat a healthy source as dead. Fix: empty-body NotFound maps to search_timeline_unavailable (retryable=True, backoff); message-bearing NotFound stays not_found. - client.py: per-call twikit Client construction + docstring. - errors.py: empty-404 special case in map_twikit_error. - tests_twitter.py: regression test for the retryable empty-404 mapping. Validation: ruff check + format (416 files), ty, makemigrations --check, schema.json --check, full Django suite 587 tests OK, live poll of the x-buying-signals feed (4 twitter_search sources) green end-to-end. Signed-off-by: matthewdonsemail-lab <matthewdonsemail@gmail.com> Co-authored-by: Matthew Don <matthewdonsemail@gmail.com>
…ma parity Adds the Facebook group search connector as a built-in source kind, using the Camofox anti-detect browser subprocess (facebook-camofox-client) for cookie-based auth and group post extraction -- mirroring the existing twitter_search connector pattern. - schema: FacebookGroupSourceSpec (group_ids, terms, count) added to the _BuiltinSourceSpec union; NewFacebookPostPayload added to FeedItemData; schema.json regenerated (84 models); --check passes. - connector: FacebookClient with the env/file/credentials-dir cookie chain (FACEBOOK_COOKIES_JSON -> FACEBOOK_COOKIES_FILE -> credentials dir with c_user/xs guard); FacebookGroupConnector with lazy client init (avoids worker-path resolution at import time); error taxonomy translated to ConnectorParseError. - registration: facebook_group registered in sources/registry.py + connectors/__init__.py; payload self-registers; source-kind invariant test extended to the known six. - validation: 20 new connector tests + invariant tests green; ruff clean. Co-authored-by: Matthew Don <matthewdonsemail@gmail.com> Signed-off-by: Matthew Don <matthewdonsemail@gmail.com>
Contributor
|
Thanks! It'll need a rebase onto Right now the connector I think the right model is the one we already use for FlareSolverr: run the camofox worker as a Dockerized sidecar with an HTTP contract instead of a subprocess. The connector becomes a thin HTTP client (like the RSS connector calling How does the camofox worker handle auth? A cookie file like the others, or does it do a headless-browser login and harvest the session cookies itself? Could you share the Does that direction sound right to you? |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds the Facebook group search connector as a built-in source kind, using the Camofox anti-detect browser subprocess for cookie-based auth and group post extraction — mirroring the existing twitter_search connector pattern.
What's included
Schema (
packages/openmagpie-schema):FacebookGroupSourceSpecwithgroup_ids(required),terms(optional keywords),count(per-cycle cap, default 20)NewFacebookPostPayloadin theFeedItemDataunion (kind: new_fb_post)schema.jsonregenerated (84 models, up from 82),--checkpassesConnector (
apps/core/sources/connectors/facebook/):errors.py:FacebookErrordataclass +map_worker_error()— maps worker JSON error responses to typed errors with retry semanticspayloads.py:NewFacebookPostPayloadwithfrom_record()classmethod andsample()for hydrated test dataclient.py:FacebookClientclass that spawnsfacebook-worker.pyas a subprocess with JSON stdin/stdout contract. Cookie resolution follows the Twitter pattern:FACEBOOK_COOKIES_JSON,FACEBOOK_COOKIES_FILE,FACEBOOK_CREDENTIALS_DIR. SupportsFACEBOOK_PROXY. 300s timeout for browser launchconnector.py:FacebookGroupConnector(BaseConnector[FacebookGroupSourceSpec])— lazy client init (avoids worker-path resolution at import time when the sibling checkout isn't present),poll()calls the client, filters by watermark, yields payloads. Self-registers viaregister()Registration:
facebook_groupadded toconnectors/__init__.py,sources/registry.py, andtools/schema_sync/models.py(CONTRACT_MODELS + INPUT_MODELS)NewFacebookPostPayloadadded to theFeedItemDataunionTests (20 tests, all passing):
_FakeGroupPost— normalized FB post record matching the worker shapefrom_recordwith full record, author-as-string, missing optional fields, query_terms fallback, URL generation, sample distinct,source_slug(group_id)