Types & Metadata

One declaration, every subsystem reads it

Protobuf messages are described by descriptors: the runtime blueprint of a message type, available even when no generated code is present. ProtoMolt adds custom options to that blueprint, so a field can state, in the schema file itself, how it should be validated, displayed, indexed, and even how a language model should fill it. Everything operates on the descriptor (a dynamically resolved message gets identical treatment to a compiled one), so the declarations are the single source of truth for every consumer.

The example below is a real field from the toolkit's example corpus: the name of a court in a legal opinion. It carries three option families at once. Try switching between the subsystems on the right, and switching option families off on the left, to see who reads what.

(ai.protomolt.proto.meta.v1.field) = {
  display_name: "Court"
  description: "The issuing court, as it appears in the caption."
  sensitivity: "pii"   // a free-form class string
}

(ai.protomolt.proto.validate.v1.field) = {
  required: true
  string: { max_len: 200 }
}

(ai.protomolt.proto.index.hints.v1.index) = {
  type: INDEX_FIELD_TYPE_TEXT
}

(ai.protomolt.proto.llm.v1.field) = {
  instruction: "Name the court exactly as it appears in the caption."
  safeguards: "Do not abbreviate."
}

The full declaration, exactly as it appears in the example corpus:

dev-tools/protomolt protobuf llm examples, court/v1/opinion_metadata.proto

string court = 1 [
  (ai.protomolt.proto.validate.v1.field) = { required: true  string: { max_len: 200 } },
  (ai.protomolt.proto.meta.v1.field)     = { description: "The issuing court, as it appears in the caption." },
  (ai.protomolt.proto.llm.v1.field)      = { instruction: "Name the court exactly as it appears in the caption."
                                            safeguards: "Do not abbreviate." }
];

The validator: 25 named formats, zero regular expressions

The validate.v1 option dialect is enforced by ProtoValidator, a dialect-neutral engine. It never reads an annotation dialect directly: every dialect implements a rule-source interface and translates into one neutral constraint model, and all violations from all sources merge. ProtoMolt implements the supported Buf Protovalidate specification in its own Java engine. The buf.validate module supplies the annotation reader and Buf's attributed schema definition; rule evaluation runs inside ProtoMolt. Rules compile and cache when a descriptor's constraints are first assembled.

The string rule surface has 36 knobs. Eleven are structural (min and max length, pattern, prefix, suffix, contains, and friends). The other 25 are named format switches, and each is backed by a hand-written parser in the core/formats package, a zero-dependency library with no java.util.regex anywhere in it. This is a deliberate choice: a regular expression checks shape, a parser checks meaning. The calendar parser refuses 2026-02-30, a date any digit-pattern would accept. The gtin format verifies the GS1 check digit, not just the digit count:

// Value must be a GS1 Global Trade Item Number: 8, 12, 13, or 14
// digits whose final digit is the GS1 mod-10 check digit. Stricter
// than a digit-count pattern: an invalid check digit refuses.
optional bool gtin = 33;

The 25 named formats:

Vocabulary formats use data that ships with the JDK itself: strict BCP 47 language tags, ISO 4217 currency codes, ISO 3166 region codes via CLDR, so nothing needs bundling or attribution. endpoint_address is a precedence grammar: it parses host:port before considering URI forms, so localhost:99999 refuses on the out-of-range port instead of passing as a plausible URI. Violation rule ids deliberately align with protovalidate's naming (string.min_len, timestamp.within), so error vocabulary transfers.

Conformance is measured, not asserted. The toolkit drives ProtoValidator against buf's protovalidate conformance suite pinned at v1.2.2: 2872 of 2872 cases pass with no skip list, and the full suite is re-scored with buf's own runner so the score cannot silently regress.

One declaration fires at every boundary: the gRPC interceptor (server and client side), the Kafka serde's write and read paths, the registry's config gate, the search service's document gate, and the Connect transforms. A message that validates at intake cannot go stale on its way through the pipeline.

Types that extend the basics: ranges and paths

Sometimes a string or an integer is the wrong abstraction. A contract period, a page span, a shelf location in a category tree: these want their own types, but a new message type should never force every client to upgrade. The governing philosophy is wire-plain, server-strict: every value stays a plain protobuf scalar or message on the wire, so any client in any language reads it untouched, and the platform's gates enforce the semantics server-side with structured INVALID_ARGUMENT rejections.

The library lives in protobuf/types: DateRange, LongRange, and DoubleRange share one four-field shape (begin, include_head, end, include_tail), over ISO date strings, 64-bit integers, and doubles. TreePath carries a taxonomy position as an ordered list of segments. The include flags end the closed-versus-half-open ambiguity that bites every range API: an unset flag means the bound is included, and an unset bound means the range is open on that side. Move the sliders:

The range types carry their invariants with them, as message-level CEL rules that enforce at-least-one-bound, begin-not-after-end, and refusal of an include flag without its bound:

// A calendar-day range. Bounds are strict ISO dates, so the
// whole range stays wire-plain: any proto client can read it.
message DateRange {
  option (ai.protomolt.proto.validate.v1.message) = {
    cel: { id: "date_range.ordered"
           message: "begin must not be after end"
           expression: "!has(this.begin) || !has(this.end) || this.begin <= this.end" }
  };
  optional string begin = 1 [(ai.protomolt.proto.validate.v1.field) = { string: { date: true } }];
  optional string end   = 2 [(ai.protomolt.proto.validate.v1.field) = { string: { date: true } }];
  optional bool include_head = 3;  // Absent = included
  optional bool include_tail = 4;
}

The indexing SPI recognizes these types by name: a field of ai.protomolt.proto.types.v1.LongRange indexes as a LONG_RANGE with no hint options at all, and TreePath likewise needs nothing extra. Any other message that happens to declare a matching (gte, lte) or (min, max) bound pair duck-types into a range too.

Sensitivity: one string, three subsystems act on it

meta.v1 metadata includes a sensitivity class: a free-form string ("public", "internal", "pii", "secret" are examples, not an enum). Three independent consumers read that one string, and each acts on it in a different direction.

flowchart LR
          s["meta.v1 sensitivity: a free-form class string"] --> e["Embedding gate: refuses classified TEXT fields by default"]
          s --> r["Index rendering: encrypt, mask, or exclude classes"]
          s --> c["Screening: model-driven spans, masked and tagged"]

The embedding gate. The default VectorizationPolicy refuses to embed any TEXT field that carries a sensitivity class. The reasoning is physical: an embedding leaves the process, is retained by whatever serves the model, and cannot be un-sent. A vector is also not the harmless summary it looks like, since inversion attacks recover a usable approximation of the source text. Deployments opt named classes in, or declare the question moot for a field that carries no class.

The index security renderer. The render-index-mappings tool takes a sensitivity object such as {"encrypt": ["pii"]} and renders those fields as store-only ciphertext containers (type: keyword, index: false, doc_values: false): semantic search runs over content the engine never sees in the clear. Sibling rules render mask and exclude fragments, or a complete security role ready to install. The boundary is stated plainly in the docs: the vector is derived from plaintext and leaks through neighborhood and inversion channels, so treat vectors as confidential in their own right.

The screening layer. A mounted ScreeningConfig (class, model reference, threshold, policy) is applied to every string field whose sensitivity equals the screened class: findings are masked with *** and tagged, or, with explicit opt-in only, the document is refused. Below-threshold detections are dropped and never recorded, because an unactioned finding would itself be evidence of the text's content. One design line matters here: screening is not validation. Schema validity is deterministic, the same input always gives the same verdict. Model detection is probabilistic and carries its model version as evidence on every mask, so it must never masquerade as schema truth.

Index hints: 19 field kinds and the vector lane

The index.hints.v1 option family tells the index layer what each field becomes. There are 19 field kinds: full-text TEXT, exact-value KEYWORD, the numeric point kinds, DATE, BINARY, VECTOR, OBJECT, NESTED, SKIP, the five *_RANGE kinds (which the canonical range types select by name), and TREE_PATH. A hint with no explicit kind is not an error: the hint source infers the field type from the proto descriptor, and any attributes you do set still win over the inferred defaults.

dev-tools/protomolt search index SPI, the 19 field kinds

INDEX_FIELD_TYPE_TEXT  = 1;   // full-text, analyzed
INDEX_FIELD_TYPE_KEYWORD = 2; // exact value
INDEX_FIELD_TYPE_VECTOR = 10; // KNN vector over a repeated numeric field
// ... INT32, INT64, FLOAT, DOUBLE, BOOLEAN, DATE, BINARY,
//     OBJECT, NESTED, SKIP, five *_RANGE kinds ...
INDEX_FIELD_TYPE_TREE_PATH = 19; // taxonomy path, ancestor-chain terms

Beyond the kind, one FieldIndexHint can carry stored and indexed flags, a name override, vector dimensions, similarity function, and element type, HNSW graph parameters (m, ef_construction), sub-fields, analyzers, a null value, sortable and facetable flags, map and date-resolution modes, and a per-engine escape hatch keyed by engine id for the cases the common model does not cover. The chunk-and-embed policy that turns TEXT into vectors has a deterministic digest: two indexes agree on their chunk boundaries and vector spaces exactly when their policy digests agree, so a changed policy is visible as a changed digest rather than silent drift.

Two further option families exist in the dialects but have no runtime reader yet: mesh.v1 and metric.v1.