Core Concepts

Copy as MarkdownOpen in...

This guide explains the key concepts and architecture of Zero Sources.

Overview

Zero Sources implements a three-container architecture for real-time data synchronization:

┌─────────────────┐     ┌─────────────┐     ┌────────────┐
│  MongoDB        │────▶│  Change     │────▶│  Zero      │
│  Database       │     │  Source     │     │  Cache     │
└─────────────────┘     └─────────────┘     └────────────┘
                                                    │
                                                    ▼
                                            ┌────────────┐
                                            │  Client    │
                                            │  Apps      │
                                            └────────────┘

Change Sources

A Change Source watches MongoDB for changes and streams them to Zero clients.

How It Works

1. Watch: Monitors MongoDB change streams
2. Transform: Converts MongoDB changes to Zero protocol
3. Stream: Pushes changes via WebSocket

Key Features

• Real-time: Sub-second latency from database to client
• Scalable: Handles thousands of concurrent connections
• Resilient: Automatic reconnection and error handling

Schemas

Schemas define your data structure in a database-agnostic way.

const userSchema = createTableSchema({
    tableName: 'user',
    columns: {
        id: { type: 'string' },
        name: { type: 'string' },
        email: { type: 'string' }
    },
    primaryKey: ['id'],
    relationships: {
        messages: {
            sourceField: ['id'],
            destSchema: () => messageSchema,
            destField: ['userId']
        }
    }
});

Why Schemas?

• Type Safety: TypeScript types generated automatically
• Validation: Data validated at runtime
• Relationships: Query across related data
• Portability: Same schema works with any database

Table Mappings

Table Mappings connect Zero schemas to MongoDB collections.

{
    "user": {
        "collection": "users",
        "fields": {
            "id": "_id",
            "name": "fullName",
            "email": "emailAddress"
        }
    }
}

Field Mapping

Map MongoDB fields to Zero columns:

• Transform field names
• Handle nested documents
• Apply filters

Discriminated Unions

Discriminated Unions handle polymorphic MongoDB collections.

{
  "content:article": {
    "collection": "content",
    "filter": { "type": "article" },
    "fields": { ... }
  },
  "content:video": {
    "collection": "content",
    "filter": { "type": "video" },
    "fields": { ... }
  }
}

One MongoDB collection → Multiple Zero tables

Watermarks

Watermarks track sync progress for each client.

Why Watermarks?

• Resume: Continue sync after disconnect
• Consistency: Guarantee no missed changes
• Efficiency: Only send new changes

Storage Backends

• ZQLite: SQLite-based storage
• NATS KV: Distributed key-value store

WebSocket Gateway

The WebSocket Gateway provides real-time bidirectional communication.

Message Flow

Client ──▶ Subscribe ──▶ Gateway ──▶ Change Source
       ◀── Changes  ◀──         ◀──

Features

• Multiplexing: Multiple subscriptions per connection
• Compression: Efficient binary protocol
• Backpressure: Flow control and buffering

Change Events

Change events represent database operations:

• Insert: New document created
• Update: Existing document modified
• Delete: Document removed
• Replace: Document replaced entirely

Event Structure

interface ChangeEvent {
    type: 'insert' | 'update' | 'delete' | 'replace';
    table: string;
    key: Record<string, any>;
    value?: Record<string, any>;
    timestamp: number;
}

Client Caching

Zero clients maintain a local cache:

1. Initial Load: Fetch current data
2. Subscribe: Watch for changes
3. Apply: Update cache incrementally
4. Query: Query local cache instantly

Benefits

• Instant: Zero-latency reads
• Offline: Works without network
• Optimistic: Update UI immediately

Best Practices

Schema Design

• Keep schemas simple and flat
• Use relationships for complex data
• Avoid deep nesting

Table Mappings

• Map only needed fields
• Use filters for large collections
• Document complex transformations

Performance

• Index MongoDB fields used in filters
• Enable MongoDB replica set
• Monitor change stream lag