Skip to content
ClickHouse Docs
ClickHouse DocsClickHouse Docs
49.6kGet Started

Discover ClickPipe source schema

Beta
POST/v1/organizations/{organizationId}/services/{serviceId}/clickpipes/schemaDiscovery
API playground

This endpoint is in beta. API contract is stable, and no breaking changes are expected in the future.

Infers the schema (field names and ClickHouse data types) of a ClickPipe source without creating a pipe. Supported for Kafka, Kinesis, Pub/Sub, and object storage sources. Object storage inference runs on the destination service, which must be running.

Permission

The API key must have the control-plane:service:manage-clickpipes permission.

Authorizations

Path parameters

  • organizationIdstringrequired

    ID of the organization that owns the service.

    format: uuid
  • serviceIdstringrequired

    ID of the service to run schema discovery against.

    format: uuid

Request bodyJSON

  • sourceoptionalobject
    4 properties
    • kafkaoptionalClickPipePostKafkaSourceornull
      2 variants

      One of the following:

      • ClickPipePostKafkaSource
        View ClickPipePostKafkaSource schema →
        15 properties
        • typeoptionalkafkaorredpandaormskorgcmkorconfluentorwarpstream+2 more

          Type of the Kafka source.

        • formatoptionalJSONEachRoworAvroorAvroConfluentorProtobuf

          Format of the Kafka source.

        • brokersoptionalstring

          Brokers of the Kafka source.

        • topicsoptionalstring

          One or more Kafka topics as a comma-separated string. All topics must have the same schema and are ingested into the same destination table by a single ClickPipe.

          Example: "topic1,topic2"
        • consumerGroupoptionalstring | null

          Consumer group of the Kafka source. If not provided "clickpipes-<>" will be used.

          Example: "my-clickpipe-consumer-group"
        • authenticationoptionalPLAINorSCRAM-SHA-256orSCRAM-SHA-512orIAM_ROLEorIAM_USERorMUTUAL_TLS+1 more

          Authentication method of the Kafka source. SERVICE_ACCOUNT_WORKLOAD_IDENTITY is in Private Preview. ClickPipes uses the GCP service account returned in gcpWorkloadIdentity.principal by the operation with operationId clickPipesServiceContextGet; grant it access to the source resources. Supported authentication methods: kafka: PLAIN, SCRAM-SHA-256, SCRAM-SHA-512, MUTUAL_TLS, msk: SCRAM-SHA-512, IAM_ROLE, IAM_USER, MUTUAL_TLS, gcmk: PLAIN, MUTUAL_TLS, SERVICE_ACCOUNT_WORKLOAD_IDENTITY, confluent: PLAIN, MUTUAL_TLS, warpstream: PLAIN, azureeventhub: PLAIN, redpanda: SCRAM-SHA-256, SCRAM-SHA-512, MUTUAL_TLS, dokafka: SCRAM-SHA-256, MUTUAL_TLS

        • iamRoleoptionalstring | null

          IAM role for the Kafka source. Use with IAM role authentication. Read more in ClickPipes documentation: https://clickhouse.com/docs/en/integrations/clickpipes/kafka#iam

          Example: "arn:aws:iam::123456789012:role/MyRole"
        • offsetoptionalClickPipeKafkaOffsetornull
          2 variants

          One of the following:

        • schemaRegistryoptionalClickPipeMutateKafkaSchemaRegistryornull
          2 variants

          One of the following:

        • caCertificateoptionalstring | null

          PEM encoded CA certificates to validate the broker's certificate.

        • reversePrivateEndpointIdsoptionalarray ofstring

          Reverse private endpoint UUIDs used for a secure private connection to the Kafka source.

        • exactlyOnceoptionalboolean | null

          Enable exactly-once delivery. Guarantees every Kafka record is inserted exactly once across restarts and rebalances. Can only be set at pipe creation.

        • tombstoneModeoptionaldeleteorsoft_delete

          How Kafka tombstone records are handled. Set to "delete" to delete the matching destination row; this requires exactly-once delivery. Set to "soft_delete" to write a row with the _is_deleted virtual column set to true; exactly-once delivery is not required. Can only be set at pipe creation.

          Example: "soft_delete"
        • credentialsoptionalPLAINorMskIamUserorAzureEventHuborMutualTLS

          Credentials for Kafka source. Choose one that is supported by the authentication method.

          4 variants

          One of the following:

        • protobufSchemaoptionalstring

          Base64-encoded .proto source or serialized FileDescriptorSet. Supported only with Protobuf format and cannot be combined with schemaRegistry.

          maxLength: 1048576, minLength: 1
          Example: "c3ludGF4ID0gInByb3RvMyI7IG1lc3NhZ2UgRXZlbnQge30="
      • null
    • kinesisoptionalClickPipePostKinesisSourceornull
      2 variants

      One of the following:

      • ClickPipePostKinesisSource
        View ClickPipePostKinesisSource schema →
        11 properties
        • formatoptionalJSONEachRoworAvroorAvroConfluentorProtobuf

          Format of the Kinesis stream.

        • streamNameoptionalstring

          Name of the Kinesis stream.

          Example: "my-stream"
        • regionoptionalstring

          AWS region of the Kinesis stream.

          Example: "us-east-1"
        • useEnhancedFanOutoptionalboolean | null

          Use enhanced fan-out for the Kinesis stream.

        • iteratorTypeoptionalTRIM_HORIZONorLATESTorAT_TIMESTAMP

          Type of iterator to use when reading from the Kinesis stream. If AT_TIMESTAMP is used, the timestamp field must be provided.

        • timestampoptionalinteger | null

          UNIX timestamp to start reading from the Kinesis stream. Required if iteratorType is AT_TIMESTAMP.

          Example: 1615766400
        • authenticationoptionalIAM_ROLEorIAM_USER

          Authentication method to use with the Kinesis stream.

        • iamRoleoptionalstring | null

          IAM role to use for authentication. Required if IAM_ROLE is used.

          Example: "arn:aws:iam::123456789012:role/MyRole"
        • schemaRegistryoptionalClickPipeKinesisSchemaRegistryornull
          2 variants

          One of the following:

        • accessKeyoptionalMskIamUserornull
          2 variants

          One of the following:

        • protobufSchemaoptionalstring

          Base64-encoded .proto source or serialized FileDescriptorSet. Required with Protobuf format unless a schema registry is configured, and not supported with other formats.

          maxLength: 1048576, minLength: 1
          Example: "c3ludGF4ID0gInByb3RvMyI7IG1lc3NhZ2UgRXZlbnQge30="
      • null
    • pubsuboptionalClickPipePostPubSubSourceornull
      2 variants

      One of the following:

    • objectStorageoptionalClickPipePostObjectStorageSourceornull
      2 variants

      One of the following:

      • ClickPipePostObjectStorageSource
        View ClickPipePostObjectStorageSource schema →
        16 properties
        • typeoptionals3orgcsordospacesorazureblobstorageorcloudflarer2orovhobjectstorage

          Type of the ObjectStorage source.

        • formatoptionalJSONEachRoworJSONAsObjectorCSVorCSVWithNamesorTabSeparatedorTabSeparatedWithNames+2 more

          Format of the files.

        • urloptionalstring

          Provide a path to the file(s) you want to ingest. You can specify multiple files using bash-like wildcards. For more information, see the documentation on using wildcards in path: https://clickhouse.com/docs/en/integrations/clickpipes/object-storage#limitations

          Example: "https://datasets-documentation.s3.eu-west-3.amazonaws.com/http/**.ndjson.gz"
        • delimiteroptionalstring | null

          Delimiter used in the files.

          Example: ","
        • compressionoptionalnoneorgziporgzorbrotliorbrorxz+3 more

          Compression algorithm used for the files.

          Example: "auto"
        • isContinuousoptionalboolean | null

          If set to true, the pipe will continuously read new files from the source. If set to false, the pipe will read the files only once. New files have to be uploaded lexically order.

        • queueUrloptionalstring | null

          Queue URL for event-based continuous ingestion. For S3, provide an SQS queue URL. For GCS, provide a Pub/Sub subscription (e.g. projects/{project}/subscriptions/{name}). When provided, files are ingested based on event notifications rather than lexicographical order. Only applicable when isContinuous is true and authentication is not public.

          Example: "https://sqs.us-east-1.amazonaws.com/123456789012/MyQueue"
        • skipInitialLoadoptionalboolean | null

          If set to true, skips the initial load and only ingests files delivered by queue notifications. Only applicable when queueUrl is provided.

        • startAfteroptionalstring | null

          Skip all files up to and including this object key during the initial load. Cannot be provided when skipInitialLoad is true.

          Example: "events/2026-06-01/"
        • authenticationoptionalIAM_ROLEorIAM_USERorCONNECTION_STRINGorSERVICE_ACCOUNTorSERVICE_ACCOUNT_WORKLOAD_IDENTITY

          Authentication method. IAM_USER is for S3, GCS, and DigitalOcean Spaces. IAM_ROLE is for S3 only. SERVICE_ACCOUNT is for GCS only. For GCS, SERVICE_ACCOUNT_WORKLOAD_IDENTITY is in Private Preview. ClickPipes uses the GCP service account returned in gcpWorkloadIdentity.principal by the operation with operationId clickPipesServiceContextGet; grant it access to the source resources. CONNECTION_STRING is for Azure Blob Storage. PUBLIC uses no authentication.

        • iamRoleoptionalstring | null

          IAM role to be used with IAM role authentication. Read more in ClickPipes documentation: https://clickhouse.com/docs/en/integrations/clickpipes/object-storage#authentication

          Example: "arn:aws:iam::123456789012:role/MyRole"
        • connectionStringoptionalstring | null

          Connection string for Azure Blob Storage authentication. Required when authentication is CONNECTION_STRING.

          Example: "DefaultEndpointsProtocol=https;AccountName=myaccount;AccountKey=mykey;EndpointSuffix=core.windows.net"
        • pathoptionalstring | null

          Path to the file(s) within the Azure container. Used for Azure Blob Storage sources. You can specify multiple files using bash-like wildcards. For more information, see the documentation on using wildcards in path: https://clickhouse.com/docs/en/integrations/clickpipes/object-storage#limitations

          Example: "data/logs/*.json"
        • azureContainerNameoptionalstring | null

          Container name for Azure Blob Storage. Required when type is azureblobstorage.

          Example: "mycontainer"
        • accessKeyoptionalMskIamUserornull
          2 variants

          One of the following:

        • serviceAccountKeyoptionalstring | null

          Base64-encoded GCP service account JSON key. Required when authentication is SERVICE_ACCOUNT.

      • null

Response

JSON

200

Successful response

JSON
  • statusoptionalnumber

    HTTP status code.

    Example: 200
  • requestIdoptionalstring

    Unique id assigned to every request. UUIDv4

    format: uuid
  • resultoptionalobject
    2 properties
    • fieldsoptionalarray ofobject

      Inferred schema fields with their ClickHouse data types.

      3 properties
      • nameoptionalstring

        Name of the inferred field.

        Example: "user_id"
      • typeoptionalstring

        Inferred ClickHouse data type of the field.

        Example: "Int64"
      • optionaloptionalboolean | null

        Whether the field is optional (nullable) in the source.

    • metaoptionalClickPipeSchemaDiscoveryMetaornull
      2 variants

      One of the following:

Report an issue
Navigation