ready
exact-url-duplicateDetect identical canonical URLs returned by multiple providers or repeated searches.
A repeated URL is one source, not multiple confirmations.DISCOVERY DEDUPLICATION READINESS LAYER
Repetition is not verification.
A read-only deduplication readiness layer that defines how TheoB will separate copied repetition from independent corroboration before scoring, Vault ingestion, or capsule compression.
Universal Time Scroll
Every TheoB pathway can move through Past, Present, and Future without losing context.
Present
Read current signals, conditions, and live context.
🎙
Universal Voice Orb
Voice ready
SYSTEM STATE
StableFounder ControlledHuman Reviewed
DEDUPLICATION READINESS
Ten copies of one claim are still one claim.
Discovery Deduplication Readiness prepares TheoB to cluster exact URLs, canonical variants, syndicated content, repeated claims, datasets, images, CAD, schematics, and visual near-duplicates without erasing source trails.
trueDeduplication Ready
falseLive Deduplication
8Deduplication Types
falseAuto Deletion
Deduplication readiness is active as a non-destructive policy layer. TheoB can define duplicate clustering logic, but it cannot process live provider results, delete sources, write Vault records, or compress capsules yet.
ready
canonical-url-variantCluster URLs that differ only by tracking parameters, mobile versions, trailing slashes, fragments, or protocol variants.
Strip noise carefully without destroying meaningful URL identity.foundation-ready
syndicated-content-duplicateDetect when the same article or release appears across multiple publishers, mirrors, or syndication networks.
Syndication should not be mistaken for independent corroboration.foundation-ready
near-text-duplicateDetect pages that are not identical but substantially repeat the same language, summary, or claim structure.
Near duplicates should lower independence, not erase all records.foundation-ready
claim-level-duplicateCluster sources repeating the same claim even when the article, page, or file is different.
Claims need independent-source context before confidence increases.review-required
dataset-version-duplicateDetect when datasets represent different versions, revisions, snapshots, or mirrors of the same underlying data.
Dataset revisions must preserve version, timestamp, schema, and source provenance.review-required
image-visual-duplicateDetect reused images, resized images, cropped images, screenshots, and visual near-duplicates.
Visual similarity must preserve uncertainty and source trail.review-required
diagram-schematic-duplicateDetect reused diagrams, CAD exports, schematics, floor plans, maps, and engineering visual variants.
Schematic differences may be meaningful and cannot be flattened blindly.ready
Deduplicate Before TrustTheoB must cluster repeated sources before scoring confidence.
Repetition is not verification.ready
Preserve Original Source TrailEvery duplicate cluster must preserve all member sources and original provider trails.
Deduplication must compress noise, not erase provenance.ready
Canonicalization Must Be ExplainableURL normalization and duplicate clustering must show why records were grouped.
No invisible source merging.ready
Independent Corroboration Must Be SeparatedTheoB must distinguish copied repetition from independent support.
A copied article is not the same as a second witness.ready
No Live Deduplication YetThis layer defines readiness only and does not process live provider results.
No provider queries, Vault writes, or automatic source changes.ready
No Automatic Source DeletionDuplicates may be clustered, but records should not be deleted automatically.
Keep auditability and reversibility.review-required
Claim-Level Deduplication Needs Scoring ContextDuplicate claims should eventually connect to source scoring and conflict detection.
A repeated claim can still be wrong.review-required
Multimodal Deduplication Requires Separate MethodsImages, diagrams, maps, schematics, CAD, and datasets require specialized duplicate logic.
Do not use text-only duplicate logic for visual or structured files.review-required
Capsule Compression Depends On DeduplicationIntelligence Capsules should be built from deduplicated source clusters, not raw repeated noise.
A capsule must preserve enough truth to be reawakened faithfully.Allowed NowRender deduplication readiness.Define duplicate types.Define duplicate cluster shape.Define canonicalization shape.Separate repetition from independent corroboration.Keep live deduplication disabled.Keep automatic source deletion disabled.
Not Allowed YetDeduplicate live provider results.Query discovery providers.Delete duplicate sources automatically.Write duplicate clusters to the Vault.Compress duplicate clusters into capsules.Treat repetition as verification.Apply text-only deduplication to images, CAD, schematics, maps, or datasets.
duplicateClusterId: stable duplicate cluster idclusterType: exact-url/canonical-url/syndicated-content/near-text/claim-level/dataset-version/image-visual/diagram-schematiccanonicalReferenceId: preferred reference card idmemberReferenceIds: array of all linked reference idsproviderIds: array of discovery provider idssourceUrls: array of source URLs or safe source IDsduplicateReason: human-readable explanation of why records were clusteredindependenceScore: 0-100repetitionCount: numberindependentSourceCount: numberfirstSeenAt: ISO timestamplastSeenAt: ISO timestampconflictStatus: none/partial/strong/unknownsourceTrailPreserved: trueproductionMutation: false
canonicalizationId: stable canonicalization idoriginalUrl: original source URLcanonicalUrl: normalized canonical URLremovedParameters: tracking or non-semantic parameters removednormalizationRules: array of applied normalization rulesconfidence: low/medium/highreviewRequired: true/falseexplanation: human-readable canonicalization explanation
Discovery Conflict Detection ReadinessDiscovery Vault Ingestion ReadinessTheoB Intelligence Capsule Engine FoundationImage Duplicate Detection ReadinessDiagram And Schematic Deduplication ReadinessVisual Semantics Color Intelligence Registry