validation against SCIP — structural fingerprints for Go
beats / how good are the clusters?
beats was run against two production Go codebases — Gitea and Mattermost — and its clusters were
compared against a SCIP semantic index built by the same-named Sourcegraph tool.
SCIP groups functions by shared reference patterns (what they import and call). beats groups them
by structural shape (how the code skeleton looks). These are independent signals, so agreement
between them is a meaningful quality check.
Mean cluster quality — two production codebases
Gitea
63 clusters · 508 total · 3 093 SCIP docs
Mattermost
99 clusters · 607 total · 1 606 SCIP docs
The consistent finding across both codebases: recall stays rock-solid at 0.88–0.89
regardless of codebase size or architecture. Every function beats puts into a cluster is
almost always reachable by the same SCIP reference queries. Precision is lower — 0.43–0.54 —
because SCIP queries are broader: they match everything that imports the same library symbols,
while beats distinguishes finer structural sub-patterns within the same import vocabulary.
The gap between precision and recall is not a flaw — it is the value: beats finds distinctions
that reference-based tools collapse.
Glossary — what do these numbers mean?
Recall
|beats ∩ SCIP| / |beats|
Does SCIP confirm what beats grouped?
Precision
|beats ∩ SCIP| / |SCIP query|
Is beats more focused than SCIP?
F1
2 · P · R / (P + R)
Overall agreement score.
Click a metric above to read its definition.
What "beats-only" clusters mean
When a cluster has beats-only functions — functions
beats grouped together that SCIP's query did not return — it means beats found a structural pattern
that isn't visible through any reference signature. These functions share a code skeleton
but were reached by different library paths, so SCIP's reference index has no query that could
retrieve them together.
This is the core claim of beats: structural shape is an independent
signal. It extends, not duplicates, what reference-based code intelligence can see.
Gitea had 2 such high-coherence beats-only clusters; Mattermost had 6.