Skip to main content

Orchestration

Overview​

Orchestration fans out a post submission to provider-specific publish functions using Cloud Tasks. Three orchestrators exist: publish (multi-target), repost (automated repurposing), and clipper (media transformation). Media is normalized once (e.g., video rehost) before fan-out.

Related: Public API | Functions API | Cloud Tasks Topology

Publish Orchestrator​

Entry: apps/functions/orchestrators/publish-orchestrator.js Core logic: apps/functions/orchestrators/publishing-service.js

  • Input: { userId, postSubmissionId, request } (from Public API /v1/posts)
  • Target resolution: from request.post.content.platform normalized to lowercase; twitter → x.

Account Resolution​

Multi-account support is built into the orchestration layer:

  • If user has multiple accounts for a platform, accountId is required in the request
  • If user has exactly one account, it's automatically selected
  • If no connected accounts exist, orchestration fails immediately with missing_integrations error
  • Account metadata is validated before task dispatch:
    • integrations/{platform}/{accountId} must exist
    • accessToken and required credentials must be present
    • Account must not be in error state

Media Normalization​

Before platform fan-out, media may be normalized to meet platform requirements:

  • Video rehosting: Videos are rehosted to Cloud Storage to ensure all platforms can access the media URL
    • Original video is fetched and streamed to a new Cloud Storage object
    • New public URL is generated and used across all platforms
    • Prevents issues with platform-specific download restrictions
  • Media validation: File types and sizes are checked against platform limits
  • URL accessibility: All media URLs are validated for public accessibility before dispatch

Task Dispatch​

For each platform, enqueues a Cloud Task to the provider function:

  • facebook → publishFacebook
  • instagram → publishInstagram
  • threads → publishThreads
  • x → publishX
  • tiktok → publishTiktok
  • youtube → publishYoutube
  • linkedin → publishLinkedin
  • snapchat → snapPost

Each task includes:

  • userId, postSubmissionId
  • Resolved accountId for the platform
  • Normalized media URLs
  • Platform-specific content (text, hashtags, etc.)
  • Distributed trace context (traceId, spanId)

State Management​

Writes lifecycle to Firestore:

  • postSubmissions/{id}:
    • status: 'orchestrating' → 'dispatched'
    • targets array with resolved platforms and accountIds
    • traceId for distributed tracing
  • postSubmissions/{id}/runs/{platform}_{accountId}:
    • status: 'queued'
    • taskName (Cloud Tasks task identifier)
    • accountId (resolved account)
    • spanId (for distributed tracing)
    • Timestamps: createdAt, queuedAt

Distributed Tracing​

Every post submission generates a unique traceId that flows through all operations:

  • Created at API entry point
  • Passed to orchestrator
  • Propagated to all platform publish tasks
  • Logged in all structured log entries
  • Enables end-to-end request tracing across async boundaries

Provider Publish Functions​

Each provider validates connectivity under users/{uid}/integrations/{provider}, transitions run state to processing, performs the API call, then writes succeeded or failed with results/errors. On success, a user-visible post record is created under users/{uid}/posts.

References:

  • Facebook: integrations/facebook/{auth.js,post.js} — Pages feed posting
  • Instagram: integrations/instagram/{auth.js,post.js} — Container + publish for images/video
  • Threads: integrations/threads/{auth.js,post.js} — Text and image posts
  • X: integrations/x/{auth.js,post.js} — Tweet creation with PKCE + refresh
  • TikTok: integrations/tiktok/{auth.js,post.js} — Direct publish for video/photo with status polling
  • YouTube: integrations/youtube/{auth.js,post.js} — Video upload via Data API v3
  • LinkedIn: integrations/linkedin/{auth.js,post.js} — UGC Posts API
  • Snapchat: integrations/snapchat/{auth.js,post.js} — Snap post creation

See Provider Integrations for detailed per-platform documentation.

Repost Orchestrator​

Entry: apps/functions/orchestrators/repost-orchestrator.js

  • Triggered by two paths:
    • Firestore trigger: onOrganicPostCreated watches users/{uid}/organicPosts/* for new content from any polled platform and enqueues the repost orchestrator
    • Polling: pollerTiktok, pollerInstagram, pollerX, pollerYoutube detect new posts and create organicPosts docs
  • Reads workflow definitions from users/{uid}/automations_workflows/* to determine target platforms and actions
  • Supports both direct reposting and template-based repurposing with AI-generated content
  • Passes through trial_reel from workflow repost targets and repurpose actions to the Instagram provider. When trial_reel is true on a target, the resulting Instagram Reel is published with limited visibility. The flag is ignored for non-Instagram targets and non-video content.
  • Enqueues per-platform repost jobs through the same Cloud Tasks fan-out

Clipper Orchestrator​

Entry: apps/functions/orchestrators/clipper-orchestrator.js

  • Handles media transformation workflows (e.g., image-to-video conversion)
  • Called when a template or repurpose action requires media format conversion
  • Uses FFmpeg and Sharp for transcoding
  • Results are stored in Cloud Storage and referenced in the post submission

Cloud Tasks​

Helpers:

  • Helper: apps/functions/tasks.js — builds function URLs and enqueues tasks.

Queues:

  • orchestrate-posts, publish-*, repost-*. Names are configured via env in shared config.

Queue setup​

Cloud Tasks queues are regional and must exist before tasks can be enqueued. Create them in the same region as your Functions (default: us-central1). The names are platform‑agnostic — create only the ones you use today, and add new queues as you onboard more providers.

Basic creation:

REGION=us-central1

# Orchestrator (dispatches per-target publish tasks)
gcloud tasks queues create orchestrate-posts --location="$REGION"

# Provider publish queues (create the ones you need)
gcloud tasks queues create publish-facebook   --location="$REGION"
gcloud tasks queues create publish-instagram  --location="$REGION"
gcloud tasks queues create publish-threads    --location="$REGION"
gcloud tasks queues create publish-x          --location="$REGION"
gcloud tasks queues create publish-tiktok     --location="$REGION"
gcloud tasks queues create publish-youtube    --location="$REGION"

# Example for a new platform (e.g., LinkedIn)
gcloud tasks queues create publish-linkedin   --location="$REGION"

# Optional repost queues (only if you enable repost flows)
gcloud tasks queues create repost-instagram   --location="$REGION"
gcloud tasks queues create repost-youtube     --location="$REGION"

Recommended tuning (optional):

# Example conservative defaults; adjust to your provider rate limits
gcloud tasks queues update publish-x \
  --location="$REGION" \
  --max-dispatches-per-second=10 \
  --max-concurrent-dispatches=5 \
  --max-attempts=5 \
  --min-backoff=10s \
  --max-backoff=600s

Verification:

gcloud tasks queues describe orchestrate-posts --location="$REGION"
gcloud tasks queues describe publish-x        --location="$REGION"

Notes:

  • Keep --location aligned with FUNCTION_REGION/deployment region. Queues are regional.
  • Our enqueue helper posts unauthenticated HTTP requests to Cloud Functions URLs. If you restrict invocation, extend the task to include an OIDC token bound to a service account with Cloud Functions Invoker and update the function to require auth.
  • Queue names are platform‑agnostic. For any new provider, create a publish-<provider> queue and wire it in the orchestrator.

Observability​

State Tracking​

  • Firestore serves as the source of truth for submission and per-platform run states
  • users/{uid}/apiRequests logs Public API v1 calls with durationMs and inferred targets
  • postSubmissions/{id} tracks orchestration status and target platforms
  • postSubmissions/{id}/runs/{platform}_{accountId} tracks individual platform execution

Distributed Tracing​

The system implements distributed tracing across all async operations:

  • Trace context: Every request generates a traceId (UUID) at the API layer
  • Span hierarchy: Each operation (orchestration, platform publish) creates child spans with spanId
  • Depth tracking: depth field tracks nesting level (0 = root, 1 = orchestrator, 2 = platform)
  • Flow correlation: flowId (typically postSubmissionId) groups related operations
  • Structured logs: All log entries include trace context for correlation

Query patterns for debugging:

// Find all logs for a specific post submission
db.collection('logs')
  .where('flowId', '==', postSubmissionId)
  .orderBy('timestamp')

// Find failed operations in a trace
db.collection('logs')
  .where('traceId', '==', traceId)
  .where('spanStatus', '==', 'error')

// Time-series queries with bucketHour
db.collection('logs')
  .where('bucketHour', '>=', '2025-01-15-10')
  .where('bucketHour', '<=', '2025-01-15-12')
  .orderBy('bucketHour')
  .orderBy('timestamp')

See also: Tracing & Logging