Dremio
Important Capabilities
Capability | Status | Notes |
---|---|---|
Asset Containers | ✅ | Enabled by default |
Data Profiling | ✅ | Optionally enabled via configuration |
Domains | ✅ | Supported via the domain config field |
Platform Instance | ✅ | Enabled by default |
Schema Metadata | ✅ | Enabled by default |
CLI based Ingestion
Install the Plugin
The dremio
source works out of the box with acryl-datahub
.
Starter Recipe
Check out the following recipe to get started with ingestion! See below for full configuration options.
For general pointers on writing and running a recipe, see our main recipe guide.
source:
type: dremio
config:
# Coordinates
hostname: localhost
port: 26257
# Credentials
authentication_method: password
username: user
password: pass
sink:
# sink configs
Config Details
- Options
- Schema
Note that a .
is used to denote nested fields in the YAML recipe.
Field | Description |
---|---|
authentication_method string | Is a Personal Access Token or a password used for authentication Default: password |
collect_pds boolean | Default: False |
collect_system_tables boolean | Ingest Dremio system tables in catalog Default: False |
data_product_pattern object | |
data_product_specs object | |
disable_certificate_verification boolean | Disable TLS certificate verification Default: False |
hostname string | Hostname or IP Address |
include_copy_lineage boolean | Default: True |
include_table_rename_lineage boolean | Default: True |
match_fully_qualified_names boolean | Whether schema_pattern is matched against fully qualified schema name <database>.<schema> . Default: False |
password string | Dremio REST API port Default: 9047 |
path_to_certificates string | Default: /vercel/path0/metadata-ingestion/venv/lib/python3.... |
platform_instance string | The instance of the platform that all assets produced by this recipe belong to. Default: |
port integer | REST API port Default: 9047 |
table_allow boolean | Default: True |
tls boolean | Whether the Dremio REST API port is encrypted Default: True |
username string | Dremio username |
env string | Environment to use in namespace when constructing URNs. Default: PROD |
schema_pattern AllowDenyPattern | Default: {'allow': ['.*'], 'deny': [], 'ignoreCase': True} |
schema_pattern.ignoreCase boolean | Whether to ignore case sensitivity during pattern matching. Default: True |
schema_pattern.allow array | List of regex patterns to include in ingestion Default: ['.*'] |
schema_pattern.allow.string string | |
schema_pattern.deny array | List of regex patterns to exclude from ingestion. Default: [] |
schema_pattern.deny.string string |
The JSONSchema for this configuration is inlined below.
{
"title": "DremioSourceConfig",
"type": "object",
"properties": {
"hostname": {
"title": "Hostname",
"description": "Hostname or IP Address",
"type": "string"
},
"port": {
"title": "Port",
"description": "REST API port",
"default": 9047,
"type": "integer"
},
"tls": {
"title": "Tls",
"description": "Whether the Dremio REST API port is encrypted",
"default": true,
"type": "boolean"
},
"disable_certificate_verification": {
"title": "Disable Certificate Verification",
"description": "Disable TLS certificate verification",
"default": false,
"type": "boolean"
},
"username": {
"title": "Username",
"description": "Dremio username",
"type": "string"
},
"authentication_method": {
"title": "Authentication Method",
"description": "Is a Personal Access Token or a password used for authentication",
"default": "password",
"type": "string"
},
"password": {
"title": "Password",
"description": "Dremio REST API port",
"default": "9047",
"type": "string"
},
"path_to_certificates": {
"title": "Path To Certificates",
"default": "/vercel/path0/metadata-ingestion/venv/lib/python3.10/site-packages/certifi/cacert.pem",
"type": "string"
},
"env": {
"title": "Env",
"description": "Environment to use in namespace when constructing URNs.",
"default": "PROD",
"type": "string"
},
"platform_instance": {
"title": "Platform Instance",
"description": "The instance of the platform that all assets produced by this recipe belong to.",
"default": "",
"type": "string"
},
"collect_pds": {
"title": "Collect Pds",
"default": false,
"type": "boolean"
},
"table_allow": {
"title": "Table Allow",
"default": true,
"type": "boolean"
},
"collect_system_tables": {
"title": "Collect System Tables",
"description": "Ingest Dremio system tables in catalog",
"default": false,
"type": "boolean"
},
"match_fully_qualified_names": {
"title": "Match Fully Qualified Names",
"description": "Whether `schema_pattern` is matched against fully qualified schema name `<database>.<schema>`.",
"default": false,
"type": "boolean"
},
"schema_pattern": {
"title": "Schema Pattern",
"default": {
"allow": [
".*"
],
"deny": [],
"ignoreCase": true
},
"allOf": [
{
"$ref": "#/definitions/AllowDenyPattern"
}
]
},
"data_product_pattern": {
"title": "Data Product Pattern",
"type": "object"
},
"data_product_specs": {
"title": "Data Product Specs",
"type": "object"
},
"include_table_rename_lineage": {
"title": "Include Table Rename Lineage",
"default": true,
"type": "boolean"
},
"include_copy_lineage": {
"title": "Include Copy Lineage",
"default": true,
"type": "boolean"
}
},
"additionalProperties": false,
"definitions": {
"AllowDenyPattern": {
"title": "AllowDenyPattern",
"description": "A class to store allow deny regexes",
"type": "object",
"properties": {
"allow": {
"title": "Allow",
"description": "List of regex patterns to include in ingestion",
"default": [
".*"
],
"type": "array",
"items": {
"type": "string"
}
},
"deny": {
"title": "Deny",
"description": "List of regex patterns to exclude from ingestion.",
"default": [],
"type": "array",
"items": {
"type": "string"
}
},
"ignoreCase": {
"title": "Ignorecase",
"description": "Whether to ignore case sensitivity during pattern matching.",
"default": true,
"type": "boolean"
}
},
"additionalProperties": false
}
}
}
Code Coordinates
- Class Name:
datahub.ingestion.source.dremio.dremio_source.DremioSource
- Browse on GitHub
Questions
If you've got any questions on configuring ingestion for Dremio, feel free to ping us on our Slack.
Is this page helpful?