Skip to content

Commit 5f0f0ed

Browse files
icecrasher321waleedlatif1claude
authored
improvement(repo): separate realtime into separate app (#4262)
* improvement(repo): restructuring to make realtime image narrower scoped * improvements * chore(repo): rebase fixes and quality improvements for realtime split Addresses merge-time issues and gaps from the realtime app split: - Retarget stale vi.mock paths to @sim/workflow-persistence/subblocks - Restore README branding, fix AGENTS.md script reference - Restore TSDoc on workflow-persistence subblocks helpers - Use toError() from @sim/utils/errors in save.ts - Add vitest config + local mocks so @sim/audit tests run standalone - Move socket.io-client to devDependencies in apps/realtime - Add missing package COPY steps to docker/app.Dockerfile - Add check:boundaries/check:realtime-prune scripts and wire into CI Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * refactor(security): consolidate crypto primitives into @sim/security Move general-purpose crypto primitives out of apps/sim into the @sim/security package so both apps/sim and apps/realtime can share them. @sim/security exports (all pure, dependency-free): ./compare safeCompare (constant-time HMAC-wrapped equality) ./encryption encrypt/decrypt (AES-256-GCM, iv:cipher:tag format) ./hash sha256Hex ./tokens generateSecureToken (base64url) Migrate apps/sim call sites to use these + @sim/utils helpers: crypto.randomUUID() -> generateId() from @sim/utils/id createHash('sha256').digest -> sha256Hex timingSafeEqual on hashed hex -> safeCompare new Promise(setTimeout) -> sleep from @sim/utils/helpers No behavior change: encryption format, digest output, and token length are preserved exactly. * refactor(copilot): use toError in remaining otel/finalize sites Replace the last two `error instanceof Error ? error : new Error(String(error))` patterns with toError from @sim/utils/errors. Completes the sweep of clean candidates — no behavior change. * refactor(security): consolidate HMAC-SHA256 primitives into @sim/security Adds hmacSha256Hex and hmacSha256Base64 to @sim/security/hmac and migrates 15 webhook providers plus 5 other hot paths (deployment token signing, outbound webhook requests, workspace notification delivery, notification test route, Shopify OAuth callback) off bare `createHmac` calls. Secret parameter accepts `string | Buffer` to cover base64-decoded Svix-style secrets (Resend) and MS Teams' HMAC scheme. AWS SigV4 signing in S3 and Textract tools intentionally retains direct `createHmac` usage — its multi-step key derivation chain doesn't fit a generic helper. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * chore(packages): post-audit test + packaging polish - Add safeCompare unit tests (identity, length mismatch, hex-nibble diff). - Add Buffer-secret cases to hmac tests to lock in Svix/MS-Teams contract. - Declare `reactflow` as a peerDependency on @sim/workflow-types — only used for type imports. - Add a barrel export to @sim/workflow-persistence for consumers that prefer package-level imports; subpath exports retained. - Document the data-field invariant in load.ts for loop/parallel subflow patching. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * chore(realtime): address PR review feedback - Remove redundant SOCKET_PORT=3002 env from Dockerfile runner stage (env.PORT already defaults to 3002 via zod schema). - Reorder PORT fallback so an explicitly-set SOCKET_PORT wins over the schema default for PORT; keeps SOCKET_PORT functional as an override instead of dead code. - Add dedicated type-check CI step for @sim/realtime so TS errors surface pre-deploy (the Dockerfile runs source TS via Bun and has no implicit build-time type check). Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * chore(realtime): remove unused SOCKET_PORT env var SOCKET_PORT has lived in the socket server since the June 2025 refactor but was never actually set in any deploy config — docker-compose.prod, helm values/templates, .env.example, and docs all use PORT or the 3002 default exclusively. No self-hoster was ever pointed at SOCKET_PORT, so removing it is safe. Simplifies realtime port resolution to `env.PORT` (zod-validated with a 3002 default) and drops the orphaned sim-side schema entry. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> --------- Co-authored-by: Waleed Latif <walif6@gmail.com> Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
1 parent bed5e95 commit 5f0f0ed

326 files changed

Lines changed: 2927 additions & 1687 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.claude/rules/sim-testing.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -144,7 +144,7 @@ vi.useFakeTimers()
144144
| `@/app/api/auth/oauth/utils` | `authOAuthUtilsMock`, `authOAuthUtilsMockFns` | `vi.mock('@/app/api/auth/oauth/utils', () => authOAuthUtilsMock)` |
145145
| `@/app/api/knowledge/utils` | `knowledgeApiUtilsMock`, `knowledgeApiUtilsMockFns` | `vi.mock('@/app/api/knowledge/utils', () => knowledgeApiUtilsMock)` |
146146
| `@/app/api/workflows/utils` | `workflowsApiUtilsMock`, `workflowsApiUtilsMockFns` | `vi.mock('@/app/api/workflows/utils', () => workflowsApiUtilsMock)` |
147-
| `@/lib/audit/log` | `auditMock`, `auditMockFns` | `vi.mock('@/lib/audit/log', () => auditMock)` |
147+
| `@sim/audit` | `auditMock`, `auditMockFns` | `vi.mock('@sim/audit', () => auditMock)` |
148148
| `@/lib/auth` | `authMock`, `authMockFns` | `vi.mock('@/lib/auth', () => authMock)` |
149149
| `@/lib/auth/hybrid` | `hybridAuthMock`, `hybridAuthMockFns` | `vi.mock('@/lib/auth/hybrid', () => hybridAuthMock)` |
150150
| `@/lib/copilot/request/http` | `copilotHttpMock`, `copilotHttpMockFns` | `vi.mock('@/lib/copilot/request/http', () => copilotHttpMock)` |

‎.cursor/rules/sim-testing.mdc‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -144,7 +144,7 @@ vi.useFakeTimers()
144144
| `@/app/api/auth/oauth/utils` | `authOAuthUtilsMock`, `authOAuthUtilsMockFns` | `vi.mock('@/app/api/auth/oauth/utils', () => authOAuthUtilsMock)` |
145145
| `@/app/api/knowledge/utils` | `knowledgeApiUtilsMock`, `knowledgeApiUtilsMockFns` | `vi.mock('@/app/api/knowledge/utils', () => knowledgeApiUtilsMock)` |
146146
| `@/app/api/workflows/utils` | `workflowsApiUtilsMock`, `workflowsApiUtilsMockFns` | `vi.mock('@/app/api/workflows/utils', () => workflowsApiUtilsMock)` |
147-
| `@/lib/audit/log` | `auditMock`, `auditMockFns` | `vi.mock('@/lib/audit/log', () => auditMock)` |
147+
| `@sim/audit` | `auditMock`, `auditMockFns` | `vi.mock('@sim/audit', () => auditMock)` |
148148
| `@/lib/auth` | `authMock`, `authMockFns` | `vi.mock('@/lib/auth', () => authMock)` |
149149
| `@/lib/auth/hybrid` | `hybridAuthMock`, `hybridAuthMockFns` | `vi.mock('@/lib/auth/hybrid', () => hybridAuthMock)` |
150150
| `@/lib/copilot/request/http` | `copilotHttpMock`, `copilotHttpMockFns` | `vi.mock('@/lib/copilot/request/http', () => copilotHttpMock)` |

‎.devcontainer/post-create.sh‎

Lines changed: 13 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -71,14 +71,26 @@ fi
7171

7272
# Set up environment variables if .env doesn't exist for the sim app
7373
if [ ! -f "apps/sim/.env" ]; then
74-
echo "📄 Creating .env file from template..."
74+
echo "📄 Creating apps/sim/.env from template..."
7575
if [ -f "apps/sim/.env.example" ]; then
7676
cp apps/sim/.env.example apps/sim/.env
7777
else
7878
echo "DATABASE_URL=postgresql://postgres:postgres@db:5432/simstudio" > apps/sim/.env
7979
fi
8080
fi
8181

82+
# Set up env for the realtime server (must match the shared values in apps/sim/.env)
83+
if [ ! -f "apps/realtime/.env" ] && [ -f "apps/realtime/.env.example" ]; then
84+
echo "📄 Creating apps/realtime/.env from template..."
85+
cp apps/realtime/.env.example apps/realtime/.env
86+
fi
87+
88+
# Set up packages/db/.env for drizzle-kit and migration scripts
89+
if [ ! -f "packages/db/.env" ] && [ -f "packages/db/.env.example" ]; then
90+
echo "📄 Creating packages/db/.env from template..."
91+
cp packages/db/.env.example packages/db/.env
92+
fi
93+
8294
# Generate schema and run database migrations
8395
echo "🗃️ Running database schema generation and migrations..."
8496
echo "Generating schema..."

‎.github/workflows/test-build.yml‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -103,6 +103,15 @@ jobs:
103103
- name: Lint code
104104
run: bun run lint:check
105105

106+
- name: Enforce monorepo boundaries
107+
run: bun run check:boundaries
108+
109+
- name: Verify realtime prune graph
110+
run: bun run check:realtime-prune
111+
112+
- name: Type-check realtime server
113+
run: bunx turbo run type-check --filter=@sim/realtime
114+
106115
- name: Run tests with coverage
107116
env:
108117
NODE_OPTIONS: '--no-warnings --max-old-space-size=8192'

‎AGENTS.md‎

Lines changed: 34 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -20,19 +20,42 @@ You are a professional software engineer. All code must follow best practices: a
2020

2121
### Root Structure
2222
```
23-
apps/sim/
24-
├── app/ # Next.js app router (pages, API routes)
25-
├── blocks/ # Block definitions and registry
26-
├── components/ # Shared UI (emcn/, ui/)
27-
├── executor/ # Workflow execution engine
28-
├── hooks/ # Shared hooks (queries/, selectors/)
29-
├── lib/ # App-wide utilities
30-
├── providers/ # LLM provider integrations
31-
├── stores/ # Zustand stores
32-
├── tools/ # Tool definitions
33-
└── triggers/ # Trigger definitions
23+
apps/
24+
├── sim/ # Next.js app (UI + API routes + workflow editor)
25+
│ ├── app/ # Next.js app router (pages, API routes)
26+
│ ├── blocks/ # Block definitions and registry
27+
│ ├── components/ # Shared UI (emcn/, ui/)
28+
│ ├── executor/ # Workflow execution engine
29+
│ ├── hooks/ # Shared hooks (queries/, selectors/)
30+
│ ├── lib/ # App-wide utilities
31+
│ ├── providers/ # LLM provider integrations
32+
│ ├── stores/ # Zustand stores
33+
│ ├── tools/ # Tool definitions
34+
│ └── triggers/ # Trigger definitions
35+
└── realtime/ # Bun Socket.IO server (collaborative canvas)
36+
└── src/ # auth, config, database, handlers, middleware,
37+
# rooms, routes, internal/webhook-cleanup.ts
38+
39+
packages/
40+
├── audit/ # @sim/audit — recordAudit + AuditAction + AuditResourceType
41+
├── auth/ # @sim/auth — @sim/auth/verify (shared Better Auth verifier)
42+
├── db/ # @sim/db — drizzle schema + client
43+
├── logger/ # @sim/logger
44+
├── realtime-protocol/ # @sim/realtime-protocol — socket operation constants + zod schemas
45+
├── security/ # @sim/security — safeCompare
46+
├── tsconfig/ # shared tsconfig presets
47+
├── utils/ # @sim/utils
48+
├── workflow-authz/ # @sim/workflow-authz — authorizeWorkflowByWorkspacePermission
49+
├── workflow-persistence/ # @sim/workflow-persistence — raw load/save + subflow helpers
50+
└── workflow-types/ # @sim/workflow-types — pure BlockState/Loop/Parallel/... types
3451
```
3552

53+
### Package boundaries
54+
- `apps/* → packages/*` only. Packages never import from `apps/*`.
55+
- Each package has explicit subpath `exports` maps; no barrels that accidentally pull in heavy halves.
56+
- `apps/realtime` intentionally avoids Next.js, React, the block/tool registry, provider SDKs, and the executor. CI enforces this via `scripts/check-monorepo-boundaries.ts` and `scripts/check-realtime-prune-graph.ts`.
57+
- Auth is shared across services via the Better Auth "Shared Database Session" pattern: both apps read the same `BETTER_AUTH_SECRET` and point at the same DB via `@sim/db`.
58+
3659
### Naming Conventions
3760
- Components: PascalCase (`WorkflowList`)
3861
- Hooks: `use` prefix (`useWorkflowOperations`)

‎apps/realtime/.env.example‎

Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,30 @@
1+
# Environment variables required by the @sim/realtime (Socket.IO) server.
2+
# These MUST match the corresponding values in apps/sim/.env for auth to work.
3+
# See apps/realtime/src/env.ts for the full zod schema.
4+
5+
# Core
6+
NODE_ENV=development
7+
PORT=3002
8+
9+
# Database — must point at the same Postgres as the main app
10+
DATABASE_URL=postgresql://postgres:postgres@localhost:5432/simstudio
11+
12+
# Auth — shared with apps/sim (Better Auth "Shared Database Session" pattern)
13+
BETTER_AUTH_URL=http://localhost:3000
14+
BETTER_AUTH_SECRET=your_better_auth_secret_min_32_chars
15+
16+
# Internal RPC — shared with apps/sim
17+
INTERNAL_API_SECRET=your_internal_api_secret_min_32_chars
18+
19+
# Public app URL — used for CORS allow-list and base URL resolution
20+
NEXT_PUBLIC_APP_URL=http://localhost:3000
21+
22+
# Optional: Redis for cross-pod room management
23+
# Leave unset for single-pod / in-memory rooms
24+
# REDIS_URL=redis://localhost:6379
25+
26+
# Optional: extra Socket.IO CORS allow-list (comma-separated)
27+
# ALLOWED_ORIGINS=https://embed.example.com,https://admin.example.com
28+
29+
# Optional: disable auth entirely for trusted private networks
30+
# DISABLE_AUTH=true

‎apps/realtime/package.json‎

Lines changed: 48 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,48 @@
1+
{
2+
"name": "@sim/realtime",
3+
"version": "0.1.0",
4+
"private": true,
5+
"license": "Apache-2.0",
6+
"type": "module",
7+
"engines": {
8+
"bun": ">=1.2.13",
9+
"node": ">=20.0.0"
10+
},
11+
"scripts": {
12+
"dev": "bun --watch src/index.ts",
13+
"start": "bun src/index.ts",
14+
"type-check": "tsc --noEmit",
15+
"lint": "biome check --write --unsafe .",
16+
"lint:check": "biome check .",
17+
"format": "biome format --write .",
18+
"format:check": "biome format .",
19+
"test": "vitest run",
20+
"test:watch": "vitest"
21+
},
22+
"dependencies": {
23+
"@sim/audit": "workspace:*",
24+
"@sim/auth": "workspace:*",
25+
"@sim/db": "workspace:*",
26+
"@sim/logger": "workspace:*",
27+
"@sim/realtime-protocol": "workspace:*",
28+
"@sim/security": "workspace:*",
29+
"@sim/utils": "workspace:*",
30+
"@sim/workflow-authz": "workspace:*",
31+
"@sim/workflow-persistence": "workspace:*",
32+
"@sim/workflow-types": "workspace:*",
33+
"@socket.io/redis-adapter": "8.3.0",
34+
"drizzle-orm": "^0.45.2",
35+
"postgres": "^3.4.5",
36+
"redis": "5.10.0",
37+
"socket.io": "^4.8.1",
38+
"zod": "^3.24.2"
39+
},
40+
"devDependencies": {
41+
"@sim/testing": "workspace:*",
42+
"@sim/tsconfig": "workspace:*",
43+
"@types/node": "24.2.1",
44+
"socket.io-client": "4.8.1",
45+
"typescript": "^5.7.3",
46+
"vitest": "^3.0.8"
47+
}
48+
}

‎apps/realtime/src/auth.ts‎

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,17 @@
1+
import { createVerifyAuth } from '@sim/auth/verify'
2+
import { env } from '@/env'
3+
4+
export const ANONYMOUS_USER_ID = '00000000-0000-0000-0000-000000000000'
5+
6+
export const ANONYMOUS_USER = {
7+
id: ANONYMOUS_USER_ID,
8+
name: 'Anonymous',
9+
email: 'anonymous@localhost',
10+
emailVerified: true,
11+
image: null,
12+
} as const
13+
14+
export const auth = createVerifyAuth({
15+
secret: env.BETTER_AUTH_SECRET,
16+
baseURL: env.BETTER_AUTH_URL,
17+
})

apps/sim/socket/config/socket.ts renamed to apps/realtime/src/config/socket.ts

Lines changed: 1 addition & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -3,9 +3,7 @@ import { createLogger } from '@sim/logger'
33
import { createAdapter } from '@socket.io/redis-adapter'
44
import { createClient, type RedisClientType } from 'redis'
55
import { Server } from 'socket.io'
6-
import { env } from '@/lib/core/config/env'
7-
import { isProd } from '@/lib/core/config/feature-flags'
8-
import { getBaseUrl } from '@/lib/core/utils/urls'
6+
import { env, getBaseUrl, isProd } from '@/env'
97

108
const logger = createLogger('SocketIOConfig')
119

apps/sim/socket/database/operations.ts renamed to apps/realtime/src/database/operations.ts

Lines changed: 15 additions & 36 deletions
Original file line numberDiff line numberDiff line change
@@ -1,15 +1,7 @@
1+
import { AuditAction, AuditResourceType, recordAudit } from '@sim/audit'
12
import * as schema from '@sim/db'
2-
import { webhook, workflow, workflowBlocks, workflowEdges, workflowSubflows } from '@sim/db'
3+
import { workflow, workflowBlocks, workflowEdges, workflowSubflows } from '@sim/db'
34
import { createLogger } from '@sim/logger'
4-
import { and, eq, inArray, isNull, or, sql } from 'drizzle-orm'
5-
import { drizzle } from 'drizzle-orm/postgres-js'
6-
import postgres from 'postgres'
7-
import { AuditAction, AuditResourceType, recordAudit } from '@/lib/audit/log'
8-
import { env } from '@/lib/core/config/env'
9-
import { cleanupExternalWebhook } from '@/lib/webhooks/provider-subscriptions'
10-
import { getActiveWorkflowContext } from '@/lib/workflows/active-context'
11-
import { loadWorkflowFromNormalizedTables } from '@/lib/workflows/persistence/utils'
12-
import { mergeSubBlockValues } from '@/lib/workflows/subblocks'
135
import {
146
BLOCK_OPERATIONS,
157
BLOCKS_OPERATIONS,
@@ -19,7 +11,14 @@ import {
1911
SUBFLOW_OPERATIONS,
2012
VARIABLE_OPERATIONS,
2113
WORKFLOW_OPERATIONS,
22-
} from '@/socket/constants'
14+
} from '@sim/realtime-protocol/constants'
15+
import { getActiveWorkflowContext } from '@sim/workflow-authz'
16+
import { loadWorkflowFromNormalizedTablesRaw } from '@sim/workflow-persistence/load'
17+
import { mergeSubBlockValues } from '@sim/workflow-persistence/subblocks'
18+
import { and, eq, inArray, isNull, or, sql } from 'drizzle-orm'
19+
import { drizzle } from 'drizzle-orm/postgres-js'
20+
import postgres from 'postgres'
21+
import { env } from '@/env'
2322

2423
const logger = createLogger('SocketDatabase')
2524

@@ -182,7 +181,7 @@ export async function getWorkflowState(workflowId: string) {
182181
throw new Error(`Workflow ${workflowId} not found`)
183182
}
184183

185-
const normalizedData = await loadWorkflowFromNormalizedTables(workflowId)
184+
const normalizedData = await loadWorkflowFromNormalizedTablesRaw(workflowId)
186185

187186
if (normalizedData) {
188187
const finalState = {
@@ -915,30 +914,10 @@ async function handleBlocksOperationTx(
915914
}
916915
}
917916

918-
// Clean up external webhooks
919-
const webhooksToCleanup = await tx
920-
.select({
921-
webhook: webhook,
922-
workflow: {
923-
id: workflow.id,
924-
userId: workflow.userId,
925-
workspaceId: workflow.workspaceId,
926-
},
927-
})
928-
.from(webhook)
929-
.innerJoin(workflow, eq(webhook.workflowId, workflow.id))
930-
.where(and(eq(webhook.workflowId, workflowId), inArray(webhook.blockId, blockIdsArray)))
931-
932-
if (webhooksToCleanup.length > 0) {
933-
const requestId = `socket-batch-${workflowId}-${Date.now()}`
934-
for (const { webhook: wh, workflow: wf } of webhooksToCleanup) {
935-
try {
936-
await cleanupExternalWebhook(wh, wf, requestId)
937-
} catch (error) {
938-
logger.error(`Failed to cleanup webhook ${wh.id}:`, error)
939-
}
940-
}
941-
}
917+
// Webhook rows are only created at deploy time (saveTriggerWebhooksForDeploy in
918+
// lib/webhooks/deploy.ts) with deploymentVersionId set; their external-subscription
919+
// lifecycle is managed by deploy.ts, lifecycle.ts, and the /api/webhooks/[id] route.
920+
// Removing a trigger block from the draft canvas does not touch any webhook rows.
942921

943922
// Delete edges connected to any of the blocks
944923
await tx

0 commit comments

Comments
 (0)