feat: multi-step cron architecture — check-research route + ingest refactor + pg_cron#603
Closed
codercatdev wants to merge 9 commits intodevfrom
Closed
feat: multi-step cron architecture — check-research route + ingest refactor + pg_cron#603codercatdev wants to merge 9 commits intodevfrom
codercatdev wants to merge 9 commits intodevfrom
Conversation
…undle Move deploySite/deployFunction/getOrCreateBucket into remotion-deploy.ts (one-time CLI setup). remotion.ts now only imports renderMediaOnLambda + getRenderProgress — no @rspack/binding dependency chain in serverless routes. Also remove @remotion/bundler, @remotion/cli, @rspack/core, @rspack/binding from serverExternalPackages in next.config.ts (no longer needed).
The main @remotion/lambda entry point re-exports from @remotion/bundler which pulls in @rspack/core → @rspack/binding (native binary). The /client subpath only exports renderMediaOnLambda + getRenderProgress without the bundler dependency chain. This is the official Remotion approach for serverless environments.
All cron jobs are triggered by Supabase pg_cron + pg_net calling the HTTP endpoints directly. Removes vercel.json cron config to avoid confusion and potential double-triggering. Co-authored-by: content <content@miriad.systems>
Adds 002_cron_schedules.sql with idempotent schedules for: - ingest-daily (10:00 UTC) - check-research (every 5 min) - check-renders (every 5 min) - sponsor-outreach (Mon/Thu 09:00 UTC) Uses DO blocks for safe unschedule on re-runs. Co-authored-by: research <research@miriad.systems>
New cron route that polls docs in "researching" status: - Queries Sanity for docs with researchNotebookId - Polls NotebookLM research status - On completion: imports sources, generates infographics, gets summary, re-generates enriched Gemini script, runs critic - Updates Sanity doc to "script_ready" with enriched data - Stuck detection: flags docs >30min in "researching" - Follows check-renders patterns (auth, fetchCache, maxDuration) Co-authored-by: research <research@miriad.systems>
When ENABLE_NOTEBOOKLM_RESEARCH is set: - Creates notebook + adds sources + starts research (~10s) - Does NOT poll for completion (was blocking 10+ min) - Generates basic script without research data - Creates Sanity doc with status "researching" - Stores researchNotebookId + researchTaskId on doc - check-research cron will poll and enrich later When research is NOT enabled: - Behavior unchanged — straight to "script_ready" Removes conductResearch() import (blocking call). Co-authored-by: research <research@miriad.systems>
…hema Adds: - "researching" status option (between draft and script_ready) - researchNotebookId field (hidden, stores NotebookLM notebook UUID) - researchTaskId field (hidden, stores deep research task UUID) - trendScore field (0-100 from trend discovery) - trendSources field (comma-separated signal sources) Co-authored-by: research <research@miriad.systems>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Splits the blocking 10+ minute NotebookLM research step into a fire-and-forget + polling architecture, matching the existing
check-renderspattern. All cron schedules moved to Supabase pg_cron.New Status Flow
When
ENABLE_NOTEBOOKLM_RESEARCHis NOT set, ingest goes straight to"script_ready"— no change to existing behavior.Files Changed
NEW:
app/api/cron/check-research/route.ts(690 lines)Polls NotebookLM research status for docs in
"researching"state. When research completes:"script_ready"with enriched contentPatterns from
check-renders:CRON_SECRETbearer authfetchCache = 'force-no-store'+maxDuration = 60"researching"→ flagged)MODIFIED:
app/api/cron/ingest/route.tsconductResearch()call (was 10+ minutes)researchNotebookIdandresearchTaskIdon Sanity doc"researching"when NotebookLM enabled,"script_ready"when notMODIFIED:
sanity/schemas/documents/automatedVideo.ts"researching"to pipeline status optionsresearchNotebookIdfield (hidden)researchTaskIdfield (hidden)trendScoreandtrendSourcesfieldsNEW:
supabase/migrations/002_cron_schedules.sqlIdempotent pg_cron migration for all pipeline schedules:
ingest-daily0 10 * * */api/cron/ingestcheck-research*/5 * * * */api/cron/check-researchcheck-renders*/5 * * * */api/cron/check-renderssponsor-outreach0 9 * * 1,4/api/cron/sponsor-outreachPrerequisites: Set
app.site_urlandapp.cron_secretas Supabase config vars.Testing
npx tsc --noEmitpasses (only pre-existing errors in unrelatedanalytics.tsx)CRON_SECRETauth