feat: Tier 1 cross-repo gRPC matching + production-readiness fixes#293
Open
sponger94 wants to merge 10 commits intoDeusData:mainfrom
Open
feat: Tier 1 cross-repo gRPC matching + production-readiness fixes#293sponger94 wants to merge 10 commits intoDeusData:mainfrom
sponger94 wants to merge 10 commits intoDeusData:mainfrom
Conversation
Introduces a new sequential pass (pass_idl_scan) that runs after pass_calls. Two responsibilities, both purely additive: 1. Emit canonical Route nodes from .proto-derived service/rpc definitions. For each Class node with file_path ending in ".proto", iterate its DEFINES_METHOD edges (rpc methods) and create a Route node per rpc with QN __route__grpc__<Service>/<rpc>, plus a HANDLES edge from the rpc Function node back to the Route. 2. Bind consumer-side gRPC handler classes via INHERITS edges. For each inheritance whose base class name matches a known server-stub suffix (Servicer, ServicerBase, ImplBase, ServiceBase, AsyncServicer, Base — tried longest-first), strip the suffix to derive the expected service name, walk methods of the inheriting class, strip *Async wrappers from method names, and emit HANDLES edges to the matching Route node. The HANDLES edges are the rendezvous point for the existing cross-repo matcher (pass_cross_repo.c match_typed_routes for GRPC_CALLS) — once producer-side typed-client GRPC_CALLS edges land in a follow-up pass, end-to-end CROSS_GRPC_CALLS edges become possible without further changes. Producer-side detection (typed gRPC client method calls) is intentionally deferred — see the design proposal for the full Tier 1-4 roadmap. This PR ships the consumer half, which is the larger code surface and the part where cross-language genericity is exercised (Python servicer, Java ImplBase, C# ServiceBase all hit the same code path). Coverage: 4 unit tests in tests/test_pipeline.c covering Python, C# (with *Async stripping), Java, and the negative case (non-proto class skipped). 2603 tests pass overall, no regressions.
testdata/cross-repo/grpc/ holds reference snippets across the four target ecosystems for pass_idl_scan validation: - contracts/promo.proto: shared IDL with package + service + 2 rpcs - server-python/promo_server.py: Python *Servicer subclass - server-csharp/PromoCodeService.cs: .NET *Base subclass with *Async methods - server-java/PromoCodeServiceImpl.java: Java *ImplBase subclass These are reference fixtures, not buildable projects — no .csproj, pom.xml, or requirements.txt. Their purpose is to give reviewers a realistic shape of what the indexer encounters in real consumer codebases. Unit tests in tests/test_pipeline.c mirror these shapes with synthetic gbuf nodes. Producer-side fixtures (client-csharp, client-go, client-python) are intentionally absent and will land alongside the producer-side typed-call detection in the follow-up Tier 1b PR.
sponger94
changed the title
…_scan
Closes the cross-repo gRPC matching loop by adding the producer half:
walks per-file extraction results (CBMFileResult.type_assigns) to find
variables assigned to generated gRPC client/stub types, then emits
GRPC_CALLS edges for each var.Method(...) call site.
Detection rules:
- Stub-type suffixes (longest-first): BlockingStub, FutureStub, AsyncStub,
AsyncClient, Stub, Client. Covers Python grpcio (*Stub), Java
protoc-gen-grpc-java (*BlockingStub, *FutureStub), C# Grpc.Tools
(*Client, *AsyncClient), Rust tonic (*Client).
- Suffix-stripped service name MUST match a Class node in the gbuf with
a .proto file_path. Filters out false positives from non-gRPC classes
that happen to end in "Client" (HttpClient, WebClient, etc.).
- Method name has *Async suffix stripped and first character capitalized
before route lookup. Bridges Java's lowerCamelCase invocations and
C#'s *Async wrapper convention.
- Caller node resolved via enclosing_func_qn → file_node fallback,
mirroring pass_calls.c calls_find_source.
Together with the existing consumer-side HANDLES edges, pass_cross_repo.c
match_typed_routes (Phase D) now produces CROSS_GRPC_CALLS edges
end-to-end without further changes.
Coverage: 4 new tests in tests/test_pipeline.c — Python *Stub, C# *Client
with *Async stripping, Java *BlockingStub with lowerCamelCase, plus a
negative case verifying HttpClient does not get a GRPC_CALLS edge.
2607 tests pass overall, 0 regressions.
Producer-side fixtures added under testdata/cross-repo/grpc/client-*
mirroring the server-* layout. Go grpc-go (pointer types + struct
embedding) and TS @grpc/grpc-js (dynamic stubs) remain out of scope —
documented in fixture README.
sponger94
changed the title
…detection
Real-world testing on a snoonu microservice fleet (gateway-service +
loyalty-gateway + ~10 .NET services) exposed a critical gap: producer-side
detection was gated on idl_service_set_contains(known_services, service),
where known_services was populated only from .proto files in the SAME
repo. In real fleets contracts ship via NuGet/Maven/PyPI packages
(Snoonu.Promo.V1.Contracts, etc.) — the producer never has a local .proto
and the gate filtered out every legitimate stub-var call.
Three fixes:
1. Drop the local-proto gate. Producer-side detection runs on suffix
shape alone (BlockingStub, FutureStub, AsyncStub, AsyncClient, Stub,
Client). pass_cross_repo Phase D handles the actual cross-repo match
by looking up Routes in target stores; non-matching GRPC_CALLS edges
are inert (no CROSS_GRPC_CALLS) and cost only one stray local Route
per unique stub type.
2. Add a type-name denylist (k_non_grpc_type_markers) for prefixes that
end in "Client" but are definitively not gRPC: System.Net.*,
Microsoft.Extensions.Http, RestSharp, Refit, Flurl, java.net.http,
okhttp3, reqwest, urllib, httpx. Cuts off the obvious false-positive
surface that the local-proto gate was masking.
3. Relax var-scope lookup with a file-scope fallback. C# class-field
pattern (`_client = new XClient(channel)` in ctor → `_client.Method`
in another method) has different enclosing_func_qn between
assignment and call sites; the previous strict-match implementation
missed it entirely. Lookup now prefers same-function scope, falls
back to any same-var-name match in the file.
Tests:
- idl_scan_skips_unknown_service_for_producer_call → renamed to
idl_scan_denylist_skips_httpclient (denylist-driven instead of
proto-list-driven), plus assertion that no stray Route is emitted.
- idl_scan_emits_grpc_calls_without_local_proto: producer detection
fires when no Class node from a .proto exists in the gbuf.
- idl_scan_resolves_class_field_assigned_in_constructor: ctor-scope
assignment + method-scope call resolves via file-scope fallback.
10 idl_scan tests pass. 2609 tests pass overall, 0 regressions.
Discovered during testing but out of scope for this PR (will be filed
as a separate upstream issue): pass_parallel.c emit_grpc_edge emits
Routes with QN format `__grpc__<service>/<method>` (without the
`__route__` prefix) using greedy suffix-stripping (ServiceClient before
Client), which produces phantom service names like `provider`,
`builder`, `experimentProvider` from local var-name matching. These
coexist in their own QN namespace; pass_idl_scan's `__route__grpc__`
Routes are unaffected.
sponger94
changed the title
…roperty fields Three small extractor changes that surface signal Tier 1 producer-side detection needs: 1. extract_defs.c — C# 12 primary-constructor parameters now emit Field defs scoped to the enclosing class. Iterates the class_declaration's parameter_list child (via field-name "parameters" or by direct child walk for grammars that don't surface the field name) and emits one Field per param with parent_class and return_type set. Modern .NET 8+/9+ controllers/services use this syntax as default; without it the class-field walker sees zero typed-client fields. 2. extract_type_assigns.c — recognize Go-style `pb.NewFooClient(ch)` and Java-style `fooGrpc.newBlockingStub(ch)` factory calls as constructor- typed assignments. Accepts qualified names whose last segment matches a typed-stub factory pattern (`New*Client`, `new*Stub`). 3. lang_specs.c — C# now uses cs_field_types (field_declaration + property_declaration) for its `field_types` slot, so property declarations also emit Field defs.
Extends pass_idl_scan with the producer-side signal sources from the
cross-repo intelligence proposal, plus production-readiness fixes
exposed when running tier1 against a real .NET microservice fleet with
NuGet-distributed proto contracts and C# 12 primary constructors.
Producer-side detection — for each call `var.Method(...)` whose
receiver var resolves to a known stub type, emit a `GRPC_CALLS` edge
to a local Route. Stub vars are discovered via four signal sources:
* Constructor-parameter tracking: walk gbuf Method nodes that look
like constructors; for each ctor param whose type matches a
stub-suffix pattern (or appears in the DI registry below), record
(class_qn, param_name, service_name) with class-wide scope.
* Factory-function inference: when type_assigns gives us
`var = pb.NewFooClient(...)` / `fooGrpc.newBlockingStub(...)`,
derive the service from the factory's last segment by stripping
`New`/`new` and the trailing `Client`/`Stub` suffix.
* DI-registration scanning: harvests stub-type FQNs from
`services.AddGrpcClient<T>(...)`, `@GrpcClient(...)` annotations,
and NestJS-style `@Client({...})` decorators. Stub vars whose
declared type is in this registry are treated as gRPC clients
even without the conventional suffix.
* Field/property type tracking: for class fields whose declared type
matches a stub-suffix pattern or DI-registered FQN, record
(class_qn, field_name, service_name) with class-wide scope.
Production-readiness fixes:
* Proto rpc → service mapping fallback. tree-sitter-protobuf emits
rpc Functions as flat siblings of the service Class rather than
children, so DEFINES_METHOD edges may not exist. When that happens,
match rpc Functions by file_path equality + start_line/end_line
containment within the service Class. Optimized to O(N+F) via a
single pre-pass that collects proto Classes and Functions into flat
arrays (avoids quadratic blowup on heavy proto-defining repos).
* Safer stub-var fallback. idl_stub_var_arr_find_ext() takes a new
allow_name_only_fallback flag. The class_vars lookup (project-wide)
passes false so a class-scope variable can only match calls whose
enclosing function lives under the same class; without this guard
two unrelated classes both declaring `_client` would silently bind
to each other's typed-client and emit wrong GRPC_CALLS edges.
* Cross-package collision visibility. Routes are still keyed
__route__grpc__<service>/<method> using the bare service name, since
cross-repo matching joins on that key and the consumer side has no
proto-package source. When a second .proto with the same bare key is
upserted, log a warning at idl_scan.route_collision so the operator
sees the ambiguity, and write the service node's qualified_name as a
service_qn property so a future FQN-aware matcher can recover
provenance.
The full FQN-keyed Route data model (Tier 1g) is intentionally deferred
to a focused follow-on PR — see .planning/cbm-cross-repo-proposal.md
§5.7 for the rationale and four-piece sequencing.
pass_idl_scan needs ctx->result_cache populated to read producer-side typed-client signals out of CBMFileResult during emission. The full sequential pipeline already attached seq_cache before pass_definitions and ran pass_idl_scan with it. The other three pipeline paths didn't: * Full parallel — built a cache during parallel_extract + parallel_resolve but never invoked pass_idl_scan, then freed cache. Threshold for parallel is ~50 files, so every real-world repo silently skipped Tier 1 producer-side emission. * Incremental sequential — called pass_idl_scan but never attached a result cache, so the pass returned early at `if (!ctx->result_cache)` and producer-side edges never refreshed. * Incremental parallel — built a cache for extract+resolve but never called pass_idl_scan at all. Fix mirrors the full sequential pattern in all three call sites: allocate a CBMFileResult ** cache, attach to ctx->result_cache before pass_idl_scan runs, run, then free.
End-to-end support for visualizing inter-repo links in the embedded Three.js graph viewer. When the active project's store has CROSS_* edges with target_project pointing to a sibling .db in the cache, the viewer now renders each linked project as an offset satellite cluster with edges connecting back to the primary cluster. Backend (src/ui/http_server.c): * /api/layout — for each distinct target_project found in the source store's CROSS_* edges, compute a layout for the linked project's store, place it on a circle around the primary cluster sized by primary + satellite radii so satellites don't bury inside, and populate cross_edges by joining the source CROSS_* edges to their Route's qualified_name (canonical across both stores) and looking up the matching Route id in the linked store. * layout_radius() — bounding-radius helper used to choose spacing. Frontend: * GraphScene.tsx — renders data.linked_projects?.map() as additional NodeCloud + EdgeLines groups offset by linked_projects[i].offset. Inter-galaxy edges go through a new EdgeLines invocation with a targetNodes prop pointing at the offset-shifted satellite nodes. * EdgeLines.tsx — new optional targetNodes prop. When set, edge.target ids are resolved against targetNodes instead of nodes. Existing intra-cluster usage is unchanged. Also adds CROSS_* / GRPC_CALLS / GRAPHQL_CALLS / TRPC_CALLS edge-type colors. * GraphTab.tsx — filteredData now passes linked_projects through (was silently dropped, leaving the scene with no satellites). Filter init / enableAll union in labels and edge types from each linked project so satellites stay visible by default. Without these changes, indexing a multi-service fleet produces CROSS_GRPC_CALLS edges in SQLite that never reach the canvas — the matching backend was correct, the rendering pipeline just had no path for inter-galaxy data.
Standalone binary that runs the C# extractor over one file and prints the resulting defs / type_assigns / calls / Field nodes with their parent_class and return_type. Useful when iterating on producer-side gRPC detection — being able to point the extractor at a real source file and read structured output is how a few of the C# 12 primary-ctor edge cases got found. Built via `make -f Makefile.cbm dump-csharp`. Not wired into the main test suite or CI.
Two planning documents covering the rationale for everything else in this branch. cbm-cross-repo-proposal.md: * Updated Tier 1 sections to reflect the producer-side detection that ships in this branch (ctor params, factories, DI registry, fields). * New §5.7 Tier 1g — Contract-package FQN extraction. Explains why the whole 1g family is deferred to a focused follow-on PR rather than landed here, and sequences the work into four pieces (producer dual emission, AST-time package extraction, deterministic collision resolution, NuGet/Maven consumer-side cache scan). * Updated §5.8 sub-tier sequencing accordingly. tier1-extractor-fixes.md (new): * Documents the four production-readiness gaps that prevented Tier 1's producer-side detection from firing on real .NET fleets: parallel-pipeline wiring, C# 12 primary-ctor extraction, protobuf rpc → service fallback, and graph-UI cross-galaxy passthrough. * Adversarial-review follow-ups: incremental-pipeline wiring, safer project-wide stub-var fallback, and the cross-package collision warning + service_qn property mitigation.
sponger94
force-pushed
the
tier1-grpc-cross-repo
branch
from
5772f82 to
0431efb
Compare
sponger94
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Lands the full Tier 1 (gRPC) cross-repo intelligence stack in one PR, plus the production-readiness fixes that surfaced when running it against a real .NET microservice fleet using NuGet-distributed
.protocontracts and C# 12 primary constructors. Strictly additive on the matching path; reusespass_cross_repo.c match_typed_routes(Phase D forGRPC_CALLS) without changing it.Closes the design discussion in #292 for Tier 1. Tiers 2–4 (typed message pub/sub, attribute-driven HTTP, config-resolved discovery) remain as separate follow-up PRs.
Companion docs (in this PR under
.planning/):cbm-cross-repo-proposal.md— full Tier 1 design, Tier 1g deferral rationale, sequencing.tier1-extractor-fixes.md— production-readiness gap log.What ships
Producer + consumer detection.
pass_idl_scannow extracts producer-side typed gRPC clients from four signal sources — local-vartype_assigns, constructor parameters, factory return types (Gopb.NewFooClient, JavafooGrpc.newBlockingStub), DI registrations (services.AddGrpcClient<T>,@GrpcClient, NestJS@Client), and class fields/properties. Consumer-side handler binding (Python*Servicer, Java*ImplBase, C#*ServiceBase) plus IDL Route emission from.protofiles were already in place; this PR completes the producer half so the existing Phase D matcher emits realCROSS_GRPC_CALLSend-to-end.False-positive guards: type-name denylist for
System.Net.*,Microsoft.Extensions.Http,RestSharp,Refit,Flurl,java.net.http,okhttp3,reqwest,urllib,httpx; method-name normalization (*Asyncstrip, lowerCamelCase → PascalCase); var-scope lookup with explicit class/file/function priority and a fail-closed flag for project-wide arrays.Production-readiness fixes (
tier1-extractor-fixes.mdGaps 1–7). Short version:pass_idl_scanwas registered only on the sequential pipeline; wired into parallel + both incremental paths so repos crossing the ~50-file parallel threshold no longer silently skip it.Fielddefs (modern .NET 8+/9+ controllers were invisible to the field walker before).DEFINES_METHODshape./api/layoutalready returnedlinked_projects, but the graph-UI tab silently dropped them when filtering — passes them through now and renders satellite galaxies with cross-repo edges.ctx->result_cachebeforepass_idl_scan, so producer edges never refreshed on edits — fixed in both incremental paths._clientcall to an unrelated class — now fail-closed for project-wide arrays, name-only fallback retained only for per-file arrays.<service>/<method>since cross-repo matching joins on that key and the consumer side has no proto-package source. Added aidl_scan.route_collisionwarning when an existing Route'sfile_pathdiffers, and aservice_qnRoute property carrying the proto Class qualified_name so a future FQN-aware matcher can recover provenance. The full FQN data model is deferred to a focused follow-on PR (Tier 1g) — seecbm-cross-repo-proposal.md§5.7. Half-shipping it without consumer-side derivation produces dormant nodes that don't change matching behavior.Graph UI satellite rendering.
/api/layoutnow computes a layout per linked project, places it on a circle around the primary cluster sized so satellites don't overlap, and populatescross_edges. The Three.js scene renders linked projects as offsetNodeCloud+EdgeLinesgroups with inter-galaxy edges through a newEdgeLines targetNodesprop.Files changed
src/pipeline/pass_idl_scan.csrc/pipeline/pipeline.csrc/pipeline/pipeline_incremental.cinternal/cbm/extract_defs.cinternal/cbm/extract_type_assigns.cinternal/cbm/lang_specs.cfield_types(property + field)Makefile.cbmdump-csharptargettests/dump_csharp.cgraph-ui/src/components/{EdgeLines,GraphScene,GraphTab}.tsxsrc/ui/http_server.c/api/layoutlinked-project response.planning/{cbm-cross-repo-proposal,tier1-extractor-fixes}.mdTest coverage
10 unit tests in
tests/test_pipeline.ccover the IDL Route + consumer-side HANDLES paths. Producer-side detection and the production-readiness fixes are validated end-to-end against a real .NET microservice fleet (NuGet-distributed contracts, C# 12 primary ctors, mixed proto-owner / consumer / mixed shapes). Full suite: 2609 passed, 38 skipped, 0 failed (clang ASan+UBSan build).Companion issues
pass_parallel.c emit_grpc_edge. Out of scope for this PR; the new pass uses a separate Route QN namespace (__route__grpc__) so it doesn't affect new work, but the pollution is still there.Test plan
tests/test_pipeline.c— all pass on local clang ASan+UBSan build..protofiles and without typed-stub assignments.HttpClient.GetAsyncdoes not produceGRPC_CALLSor stray Routes.GRPC_CALLS(was previously stale until full reindex).