Connectors: getting data in

The front door: intake for authenticated ingestion

The document store behind everything (repo-service, covered on its own page) can guard its gRPC and HTTP listeners with PROTOMOLT_API_TOKEN. When that setting is absent, it is an open trusted-network deployment. Intake adds a different boundary for document ingestion: a caller proves itself, gets bound to a narrow allowance, and hands over bytes.

The proof of identity never travels inside a message field. It rides in gRPC metadata, the standard x-api-key header or the usual authorization header, and the server resolves it to an IntakeScope: an account id, the datasource ids this key may use, the drives it may write, the MIME types it may send, and a payload size cap. The contract says it in a header comment worth reading whole:

dev-tools/protomolt intake module, intake_service.proto, the auth rule stated in the contract

// AUTH IS NOT A PROTO FIELD. The credential rides gRPC metadata
// (x-api-key, or the standard authorization header) and is resolved
// server-side to a scope {account_id, allowed datasource_ids,
// allowed drives, permissions}; the request messages below only
// NARROW within that scope, never widen it.

The narrowing rule has two visible consequences. A request aimed at a datasource or drive outside its scope fails with PERMISSION_DENIED: the server knows who you are, and this is not yours. A key that resolves to nothing fails earlier with UNAUTHENTICATED. The error split is deliberate: "who are you?" and "not for you" are different answers, and callers should be able to tell them apart.

One more rule falls out of the scope model: ownership fields the scope dictates win over whatever the caller supplied. If the scope says this key writes to the intake drive, a request claiming the pipeline drive is rewritten, not executed.

Three lanes, one door

The same gate serves three shapes of upload. IngestDocument is a single unary call for a program that already has a complete document or a raw payload (bytes, a filename, and a declared MIME type that is treated as a hint; the parsing coordinator sniffs the real content type from magic bytes later). IngestStream is a client-streaming call for large payloads, with strict frame discipline: the first frame must carry metadata and every later frame must carry data, and breaking that order is an INVALID_ARGUMENT. And POST /v1/intake:upload is a plain HTTP route for the simplest possible client, a curl one-liner that sends the key as a header:

The HTTP lane; identical scope rules, no gRPC needed

curl -X POST http://intake-host/v1/intake:upload \
  -H 'x-api-key: <your key>' \
  --data-binary @quarterly-plan.pdf

All three lanes return the same receipt vocabulary: a doc_id, a node_id and address inside the store, the drive, the size, a server-computed SHA-256, and a deduplicated flag. And all three save through repo-service's wire contract; intake itself never touches object storage.

The document id is the content hash

This is the detail that makes re-uploads safe. The doc id is derived from the payload's SHA-256, so identical bytes always map to the same document. Re-running a crawler that already ran yesterday does not create duplicates; the second upload converges on the same id and the receipt says deduplicated: true. Deterministic ids turn "import everything again" from a cleanup chore into a no-op.

Where keys live

Three interchangeable stores sit behind the key-resolution seam. An in-memory store covers tests, demos, and single-tenant runs. A JDBC store covers the real case: keys live in PostgreSQL, and the table is keyed by the lowercase hex SHA-256 of the credential, never the credential itself, so a dump of the key table authenticates nobody. Revocation is a revoked_at timestamp kept as an audit row rather than a delete, which makes key rotation with a grace period just two live rows resolving to the same account. An OIDC introspection store (RFC 7662) treats an API key as an identity-provider-issued opaque token, with the scope riding the token's claims and revocation taking effect on the next call. One error discipline binds all three: a broken key store throws an internal error rather than answering like a bad key, because an outage must never masquerade as the caller's fault.

The reach-in side: acquire, pulling from sources that keep their own documents

Push is not always the right shape. A bucket full of PDFs, a table with a LONGVARBINARY column, a wiki that already organizes a company's knowledge: these sources already hold the documents, so ProtoMolt's acquire family reaches in instead. It splits into three groups with very different levels of maturity. A web crawler for public sites and a Google Drive connector are planned but do not exist yet.

Verbs for buckets and databases

Two contributed verbs cover the simplest sources. pull-s3 pulls objects from an S3 bucket and pull-jdbc pulls rows over JDBC. Both are one bounded pass per call: there is no crawl engine and no scheduler anywhere in the tree, so each invocation copies a defined slice and finishes. The verbs ship and work, but nothing schedules them: they run when a caller invokes them, not on their own.

The gRParse fleet proxies: Confluence and Microsoft Graph as typed gRPC

Hosted knowledge bases are harder than buckets: they paginate, they update, they carry attachments, and their native APIs are REST with vendor quirks. The gRParse fleet provides two proxies that speak a vendor REST API upstream and typed gRPC downstream. Its Confluence service exposes spaces, pages, blog posts, comments, and attachments, including optional attachment bytes up to a 25 MiB cap. A cursor-based Sync stream sends a bounded incremental crawl, and completed runs can write an Open Knowledge Format bundle, a sibling WARC 1.1 archive, a protobuf Kafka stream, and sync-ledger records. Its Microsoft Graph service exposes sites, drives, and files through GetItem, DownloadItem, and Sync, with SharePoint columns mapped to typed fields. The fleet also has Streamable HTTP MCP tools for connection setup, output settings, and syncs. These are independently deployable gRParse fleet services; the Java acquire/msgraph library is a separate framework building block.

flowchart TD
          a["Request arrives, key in metadata"] --> k["Does the key resolve?"]
          k -- "no: UNAUTHENTICATED" --> r1["Refused: unknown key"]
          k -- "yes" --> s["Targets inside the scope?"]
          s -- "no: PERMISSION_DENIED" --> r2["Refused: outside this key's scope"]
          s -- "yes" --> p["Within the payload cap?"]
          p -- "no: RESOURCE_EXHAUSTED" --> r3["Refused: over the cap"]
          p -- "yes" --> d["doc_id = SHA-256 of the payload"]
          d --> v["Bytes already known?"]
          v -- "yes" --> dup["Same id, receipt says deduplicated"]
          v -- "no" --> save["Saved through repo-service, receipt returned"]