Skip to main content

Sinks

Console​

console

Print data to stdout.

Sample usage of console sink​

sinks:
  - name: console

Compass​

compass

Upload metadata to a given type in Compass. Request will be sent via HTTP to a given host.

Sample usage of compass sink​

sinks:
  - name: compass
    config:
      host: https://compass.com
      type: sample-compass-type
      mapping:
        new_fieldname: "json_field_name"
        id: "resource.urn"
        displayName: "resource.name"

File​

file

Sinks metadata to a file in json/yaml format as per the config defined.

sinks:
  name: file
  config:
    path: "./dir/sample.yaml"
    format: "yaml"

GCS​

Google Cloud Storage

Sinks json data to a file as ndjson format in Google Cloud Storage bucket

sinks:
  - name: gcs
    config:
      project_id: google-project-id
      url: gcs://bucket_name/target_folder
      object_prefix: github-users
      service_account_base64: <base64 encoded service account key>
      service_account_json:
        {
          "type": "service_account",
          "private_key_id": "xxxxxxx",
          "private_key": "xxxxxxx",
          "client_email": "xxxxxxx",
          "client_id": "xxxxxxx",
          "auth_uri": "https://accounts.google.com/o/oauth2/auth",
          "token_uri": "https://oauth2.googleapis.com/token",
          "auth_provider_x509_cert_url": "xxxxxxx",
          "client_x509_cert_url": "xxxxxxx",
        }

http​

http

Sinks metadata to a http destination as per the config defined.

sinks:
  name: http
  config:
    method: POST
    success_code: 200
    url: https://compass.com/v1beta1/asset
    headers:
      Header-1: value11,value12

Stencil​

stencil

Upload metadata of a given schema format in the existing namespace_id present in Stencil. Request will be sent via HTTP to a given host.

sinks:
  name: stencil
  config:
    host: https://stencil.com
    namespace_id: myNamespace
    schema_id: mySchema
    format: json
    send_format_header: false

Frontier​

frontier

Upsert users to frontier service running at a given 'host'. Request will be sent via GRPC.

sinks:
  name: frontier
  config:
    host: frontier.com
    headers:
      X-Frontier-Email: meteor@raystack.io
      X-Other-Header: value1, value2

Notes

Compass' Type requires certain fields to be sent, hence why mapping config is needed to map value from any of our metadata models to any field name when sending to Compass. Supports getting value from nested fields.