benchmarking results

Author: surbhigarg92

.gitignore

@@ -3,4 +3,5 @@ gorm/
 .DS_Store
 node_modules
 build/
-package-lock.json
+package-lock.json
+googleapis/

spannerlib/.gitignore

@@ -10,3 +10,5 @@ vendor/bundle
 *.swp
 ext/
 Gemfile.lock
+/googleapis/
+package-lock.json

spannerlib/wrappers/spannerlib-nodejs-poc/benchmark/README.md

@@ -0,0 +1,79 @@
+# Spanner Node.js Wrappers Benchmark
+
+This directory contains a benchmark suite to compare the performance, throughput, and memory usage of three different Node.js wrappers for the Spanner Shared C-Library:
+1.  **Koffi** (Fast FFI)
+2.  **N-API** (Native C++ Addon)
+3.  **IPC** (gRPC Server/Client)
+
+## Prerequisites
+Ensure you have the following installed on the machine:
+*   **Node.js** (v22 recommended)
+*   **Go** (v1.22 recommended)
+*   **Build Tools** (`gcc`, `g++`, `make`)
+
+---
+
+## Setup Instructions
+
+### 1. Fetch Google API Protos (For IPC)
+The IPC wrapper depends on standard Google API protos. You can fetch them by running the build script in the `grpc-server` directory:
+
+```bash
+cd spannerlib/grpc-server
+./build-protos.sh
+```
+*(Note: This script also attempts to generate Go code using `protoc`. If you do not have `protoc` installed, it may show an error on the generation step, but it will still have successfully cloned the `googleapis` folder that Node.js needs).*
+
+### 2. Build the Go Shared Library
+Navigate to the shared library directory and build the `.so` file:
+```bash
+cd spannerlib/shared
+go build -o libspanner.so -buildmode=c-shared shared_lib.go
+```
+
+### 3. Install Dependencies for Wrappers
+Navigate to each wrapper directory and install its specific dependencies:
+
+**Koffi:**
+```bash
+cd spannerlib/wrappers/spannerlib-nodejs-poc/koffi
+npm install
+```
+
+**N-API:**
+```bash
+cd spannerlib/wrappers/spannerlib-nodejs-poc/napi
+npm install # This will also automatically build the C++ addon
+```
+
+**IPC:**
+```bash
+cd spannerlib/wrappers/spannerlib-nodejs-poc/ipc
+npm install
+```
+
+### 4. Start the gRPC Server (Required for IPC)
+The IPC wrapper requires the Go gRPC server to be running. 
+Navigate to `spannerlib/grpc-server` and run it in the background (or in a `tmux` session):
+```bash
+cd spannerlib/grpc-server
+nohup go run server.go localhost:50051 tcp > server.log 2>&1 &
+```
+
+---
+
+## Running the Benchmark
+
+Once all setups are complete, navigate to this directory:
+
+```bash
+cd spannerlib/wrappers/spannerlib-nodejs-poc/benchmark
+npm install
+node --expose-gc benchmark.js
+```
+
+## Results
+The script will generate a `RESULTS.md` file in this directory containing:
+*   Latency statistics from `mitata`.
+*   Throughput (ops/sec) and Heap memory delta for 5 runs of each wrapper.
+*   A 1-minute cooldown sleep is automatically applied between wrappers to ensure DB cooling and isolation.

spannerlib/wrappers/spannerlib-nodejs-poc/benchmark/RESULTS.md

@@ -4,58 +4,52 @@
 - **Database**: `projects/span-cloud-testing/instances/gargsurbhi-testing/databases/test-db-koffi-136`
 
 ## Mitata Latency Benchmark
-
-```
-cpu: Apple M4 Pro
-runtime: node v22.17.1 (arm64-darwin)
-
 benchmark                    time (avg)             (min … max)       p75       p99      p999
 --------------------------------------------------------------- -----------------------------
-Koffi Single Point Read  66'049 µs/iter (62'838 µs … 69'484 µs) 68'705 µs 69'484 µs 69'484 µs
-N-API Single Point Read  71'349 µs/iter (66'112 µs … 83'420 µs) 76'782 µs 83'420 µs 83'420 µs
-IPC Single Point Read    75'546 µs/iter (66'407 µs … 97'423 µs) 91'583 µs 97'423 µs 97'423 µs
-```
+Koffi Single Point Read   4'228 µs/iter   (2'763 µs … 7'062 µs)  4'795 µs  6'234 µs  7'062 µs
+N-API Single Point Read   4'223 µs/iter   (2'745 µs … 6'530 µs)  4'777 µs  5'694 µs  6'530 µs
+IPC Single Point Read     6'744 µs/iter  (5'166 µs … 11'075 µs)  7'296 µs 11'075 µs 11'075 µs
 
 ## Summary of Manual Benchmark (Average of 5 Runs)
 
 | Wrapper | Throughput | Avg Latency | Min Latency | Max Latency | Heap Diff (Avg) |
 | :--- | :--- | :--- | :--- | :--- | :--- |
-| **Koffi** | 13.83 ops/sec | 72.48 ms | 60.38 ms | 159.86 ms | 0.73 MB |
-| **N-API** | 13.97 ops/sec | 71.81 ms | 59.34 ms | 267.20 ms | 0.55 MB |
-| **IPC** | 13.24 ops/sec | 75.80 ms | 63.78 ms | 215.10 ms | 13.28 MB |
+| **Koffi** | 247.50 ops/sec | 4.04 ms | 2.19 ms | 120.20 ms | 0.48 MB |
+| **N-API** | 251.79 ops/sec | 3.97 ms | 2.08 ms | 114.84 ms | 0.29 MB |
+| **IPC** | 178.48 ops/sec | 5.60 ms | 3.55 ms | 62.48 ms | 1.11 MB |
 
 ## Detailed Runs
 
 ### Koffi
 
 | Run | Start Time | End Time | Throughput | Avg Latency | Min Latency | Max Latency | Heap Diff |
 | :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- |
-| Run 1 | 2026-04-13T06:26:00.195Z | 2026-04-13T06:26:35.666Z | 14.10 ops/sec | 70.94 ms | 60.02 ms | 369.14 ms | 0.68 MB |
-| Run 2 | 2026-04-13T06:26:37.677Z | 2026-04-13T06:27:11.543Z | 14.76 ops/sec | 67.73 ms | 59.99 ms | 95.69 ms | 0.76 MB |
-| Run 3 | 2026-04-13T06:27:13.556Z | 2026-04-13T06:27:51.062Z | 13.33 ops/sec | 75.01 ms | 61.05 ms | 108.63 ms | 0.72 MB |
-| Run 4 | 2026-04-13T06:27:53.073Z | 2026-04-13T06:28:31.692Z | 12.95 ops/sec | 77.24 ms | 61.14 ms | 106.80 ms | 0.71 MB |
-| Run 5 | 2026-04-13T06:28:33.703Z | 2026-04-13T06:29:09.433Z | 13.99 ops/sec | 71.46 ms | 59.70 ms | 119.04 ms | 0.78 MB |
-| **Average** | - | - | 13.83 ops/sec | 72.48 ms | 60.38 ms | 159.86 ms | 0.73 MB |
+| Run 1 | 2026-04-13T16:09:56.696Z | 2026-04-13T16:13:17.847Z | 248.57 ops/sec | 4.02 ms | 2.10 ms | 211.72 ms | 0.63 MB |
+| Run 2 | 2026-04-13T16:13:19.850Z | 2026-04-13T16:16:39.698Z | 250.19 ops/sec | 4.00 ms | 2.21 ms | 50.32 ms | 0.60 MB |
+| Run 3 | 2026-04-13T16:16:41.700Z | 2026-04-13T16:20:04.955Z | 246.00 ops/sec | 4.06 ms | 2.22 ms | 46.13 ms | 0.50 MB |
+| Run 4 | 2026-04-13T16:20:06.957Z | 2026-04-13T16:23:30.452Z | 245.71 ops/sec | 4.07 ms | 2.23 ms | 83.57 ms | 0.59 MB |
+| Run 5 | 2026-04-13T16:23:32.455Z | 2026-04-13T16:26:54.852Z | 247.04 ops/sec | 4.05 ms | 2.18 ms | 209.29 ms | 0.05 MB |
+| **Average** | - | - | 247.50 ops/sec | 4.04 ms | 2.19 ms | 120.20 ms | 0.48 MB |
 
 ### N-API
 
 | Run | Start Time | End Time | Throughput | Avg Latency | Min Latency | Max Latency | Heap Diff |
 | :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- |
-| Run 1 | 2026-04-13T06:30:09.445Z | 2026-04-13T06:30:45.556Z | 13.85 ops/sec | 72.22 ms | 59.20 ms | 633.28 ms | 0.56 MB |
-| Run 2 | 2026-04-13T06:30:47.561Z | 2026-04-13T06:31:20.317Z | 15.26 ops/sec | 65.51 ms | 59.25 ms | 131.44 ms | 0.53 MB |
-| Run 3 | 2026-04-13T06:31:22.329Z | 2026-04-13T06:31:57.787Z | 14.10 ops/sec | 70.92 ms | 58.95 ms | 129.25 ms | 0.56 MB |
-| Run 4 | 2026-04-13T06:31:59.797Z | 2026-04-13T06:32:36.084Z | 13.78 ops/sec | 72.57 ms | 59.58 ms | 192.24 ms | 0.55 MB |
-| Run 5 | 2026-04-13T06:32:38.096Z | 2026-04-13T06:33:17.018Z | 12.85 ops/sec | 77.84 ms | 59.74 ms | 249.79 ms | 0.55 MB |
-| **Average** | - | - | 13.97 ops/sec | 71.81 ms | 59.34 ms | 267.20 ms | 0.55 MB |
+| Run 1 | 2026-04-13T16:27:56.860Z | 2026-04-13T16:31:15.625Z | 251.55 ops/sec | 3.97 ms | 2.07 ms | 212.07 ms | -1.32 MB |
+| Run 2 | 2026-04-13T16:31:17.628Z | 2026-04-13T16:34:37.128Z | 250.63 ops/sec | 3.99 ms | 2.07 ms | 211.28 ms | 0.60 MB |
+| Run 3 | 2026-04-13T16:34:39.131Z | 2026-04-13T16:37:58.728Z | 250.50 ops/sec | 3.99 ms | 2.10 ms | 49.52 ms | -0.15 MB |
+| Run 4 | 2026-04-13T16:38:00.729Z | 2026-04-13T16:41:19.177Z | 251.96 ops/sec | 3.97 ms | 2.10 ms | 48.44 ms | -0.19 MB |
+| Run 5 | 2026-04-13T16:41:21.180Z | 2026-04-13T16:44:37.807Z | 254.29 ops/sec | 3.93 ms | 2.07 ms | 52.89 ms | 2.50 MB |
+| **Average** | - | - | 251.79 ops/sec | 3.97 ms | 2.08 ms | 114.84 ms | 0.29 MB |
 
 ### IPC
 
 | Run | Start Time | End Time | Throughput | Avg Latency | Min Latency | Max Latency | Heap Diff |
 | :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- |
-| Run 1 | 2026-04-13T06:34:17.030Z | 2026-04-13T06:34:58.429Z | 12.08 ops/sec | 82.80 ms | 63.94 ms | 608.08 ms | 8.40 MB |
-| Run 2 | 2026-04-13T06:35:00.440Z | 2026-04-13T06:35:39.422Z | 12.83 ops/sec | 77.96 ms | 64.92 ms | 134.13 ms | 15.16 MB |
-| Run 3 | 2026-04-13T06:35:41.437Z | 2026-04-13T06:36:17.121Z | 14.01 ops/sec | 71.37 ms | 62.60 ms | 112.61 ms | 15.49 MB |
-| Run 4 | 2026-04-13T06:36:19.130Z | 2026-04-13T06:36:54.629Z | 14.09 ops/sec | 71.00 ms | 62.87 ms | 118.40 ms | 13.68 MB |
-| Run 5 | 2026-04-13T06:36:56.642Z | 2026-04-13T06:37:34.589Z | 13.18 ops/sec | 75.89 ms | 64.54 ms | 102.27 ms | 13.66 MB |
-| **Average** | - | - | 13.24 ops/sec | 75.80 ms | 63.78 ms | 215.10 ms | 13.28 MB |
+| Run 1 | 2026-04-13T16:45:39.821Z | 2026-04-13T16:50:15.184Z | 181.58 ops/sec | 5.51 ms | 3.54 ms | 100.94 ms | 7.74 MB |
+| Run 2 | 2026-04-13T16:50:17.187Z | 2026-04-13T16:54:54.569Z | 180.26 ops/sec | 5.55 ms | 3.56 ms | 53.70 ms | -0.07 MB |
+| Run 3 | 2026-04-13T16:54:56.572Z | 2026-04-13T16:59:37.603Z | 177.92 ops/sec | 5.62 ms | 3.57 ms | 52.15 ms | -0.16 MB |
+| Run 4 | 2026-04-13T16:59:39.606Z | 2026-04-13T17:04:22.795Z | 176.56 ops/sec | 5.66 ms | 3.57 ms | 55.32 ms | -5.46 MB |
+| Run 5 | 2026-04-13T17:04:24.798Z | 2026-04-13T17:09:08.731Z | 176.10 ops/sec | 5.68 ms | 3.50 ms | 50.27 ms | 3.48 MB |
+| **Average** | - | - | 178.48 ops/sec | 5.60 ms | 3.55 ms | 62.48 ms | 1.11 MB |
 

spannerlib/wrappers/spannerlib-nodejs-poc/benchmark/benchmark.js

@@ -63,10 +63,8 @@ async function measure(name, conn, executeMethod, iterations = 500, runs = 5, sl
     const runResults = [];
 
     for (let r = 0; r < runs; r++) {
-        if (r > 0) {
-            console.log(`  Sleeping for ${sleepMs}ms to let DB cool down...`);
-            await new Promise(resolve => setTimeout(resolve, sleepMs));
-        }
+        console.log(`  Sleeping for ${sleepMs}ms to let DB cool down...`);
+        await new Promise(resolve => setTimeout(resolve, sleepMs));
 
         console.log(`  Run ${r + 1}/${runs}...`);
         if (global.gc) global.gc();
@@ -213,16 +211,16 @@ async function runMitataBenchmark(connections) {
         });
     }
 
-    const originalLog = console.log;
+    const originalWrite = process.stdout.write;
     let mitataOutput = '';
-    console.log = (...args) => {
-        mitataOutput += args.join(' ') + '\n';
-        originalLog.apply(console, args);
+    process.stdout.write = function(chunk, encoding, callback) {
+        mitataOutput += chunk.toString();
+        return originalWrite.call(process.stdout, chunk, encoding, callback);
     };
 
     await run();
 
-    console.log = originalLog;
+    process.stdout.write = originalWrite;
     return mitataOutput;
 }
 
@@ -239,10 +237,8 @@ async function runManualBenchmark(connections, iterations = 500) {
     if (connections.ipc) wrappers.push({ key: 'Ipc', name: 'IPC', conn: connections.ipc.conn, method: 'execute' });
 
     for (let i = 0; i < wrappers.length; i++) {
-        if (i > 0) {
-            console.log(`\nSleeping for 1 minute before ${wrappers[i].name} to let DB cool down...`);
-            await new Promise(resolve => setTimeout(resolve, 60000));
-        }
+        console.log(`\nSleeping for 1 minute before ${wrappers[i].name} to let DB cool down...`);
+        await new Promise(resolve => setTimeout(resolve, 60000));
         const w = wrappers[i];
         results[w.key] = await measure(w.name, w.conn, w.method, iterations);
     }
@@ -258,11 +254,14 @@ function saveResults(mitataOutput, results, iterations) {
     let report = `# Benchmark Results\n\n`;
     report += `## Environment\n`;
     report += `- **Database**: \`${dbPath}\`\n\n`;
+    report += `**Action Benchmarked**: A Single Point Read operation involving executing a SQL query (\`SELECT ... WHERE SingerId = 1\`), reading the first result row, and closing the result stream.\n\n`;
 
     report += `## Mitata Latency Benchmark\n\n`;
     report += `\`\`\`\n${mitataOutput}\`\`\`\n\n`;
 
     report += `## Summary of Manual Benchmark (Average of 5 Runs)\n\n`;
+    report += `This table summarizes a manual benchmark designed to measure throughput and memory efficiency under sustained load.\n\n`;
+    report += `Each run consists of 50,000 iterations of this action. The table displays the average results across 5 independent runs for each wrapper.\n\n`;
     report += `| Wrapper | Throughput | Avg Latency | Min Latency | Max Latency | Heap Diff (Avg) |\n`;
     report += `| :--- | :--- | :--- | :--- | :--- | :--- |\n`;
 
@@ -306,7 +305,7 @@ async function main() {
 
     const mitataOutput = await runMitataBenchmark(connections);
 
-    const iterations = 500;
+    const iterations = 50000;
     const results = await runManualBenchmark(connections, iterations);
 
     saveResults(mitataOutput, results, iterations);