Pipelines reference - Preparers
This section provides reference documentation for Pipelines Preparers. It includes information on the functions and views available in the aidb extension related to Preparers. See the Usage and Examples sections for more details.
Views
aidb.preparers
| Column | Type | Description |
|---|---|---|
| id | INTEGER | |
| name | TEXT | Name of the preparer. |
| operation | aidb.DataPreparationOperation | The kind of processing step that will be performed. |
| destination_schema | TEXT | Schema of the destination table where the output data will be stored. |
| destination_table | TEXT | Name of the destination table where the output data will be stored. |
| destination_key_column | TEXT | Column of the destination table that references the key in source data. |
| destination_data_column | TEXT | Column of the destination table where the output data will be stored. |
| options | JSONB | Configuration options for the data preparation operation. Utilizes the same API as the data preparation primitives. |
| source_type | TEXT | Type of source data the preparer is working with. Can be either 'Table' or 'Volume'. |
| source_schema | TEXT | Schema of the table with the source data the preparer will process. Only applicable for preparers of Table source type. |
| source_table | TEXT | Name of the table with the source data the preparer will process. Only applicable for preparers of Table source type. |
| source_data_column | TEXT | Column in the source table with the source data the preparer will process. Only applicable for preparers of Table source type. |
| source_key_column | TEXT | Name of the key column in the source table for reference with the output processed data. Only applicable for preparers of Table source type. |
| source_volume_name | TEXT | Name of the volume to use as a data source. Only applicable for preparers of Volume source type. |
Types
aidb.DataPreparationOperation
The aidb.DataPreparationOperation type is an enum that represents the different types of pre-processing steps that can be performed.
- ChunkText
- SummarizeText
- ParseHtml
- ParsePdf
Functions
aidb.create_preparer_for_table
Creates a preparer with a source data table.
Parameters
| Parameter | Type | Default | Description |
|---|---|---|---|
| name | TEXT | Required | Name of the preparer |
| operation | aidb.DataPreparationOperation | Required | Type of data preparation operation |
| source_table | TEXT | Required | Name of the source data table |
| source_data_column | TEXT | Required | Column in the source table containing the raw data |
| destination_table | TEXT | Required | Name of the destination table |
| destination_data_column | TEXT | Required | Column in the destination table for processed data |
| source_key_column | TEXT | 'id' | Column to use as key to reference the rows |
| destination_key_column | TEXT | 'id' | Key column in the destination table that references the source_key_column |
| options | JSONB | '{}'::JSONB | Configuration options for the data preparation operation. Utilizes the same API as the data preparation primitives. |
aidb.create_preparer_for_volume
Creates a preparer for a given PGFS volume.
Parameters
| Parameter | Type | Default | Description |
|---|---|---|---|
| name | TEXT | Required | Name of the preparer |
| operation | aidb.DataPreparationOperation | Required | Type of data preparation operation |
| source_volume_name | TEXT | Required | Name of the source volume containing the raw data |
| destination_table | TEXT | Required | Name of the destination table |
| destination_data_column | TEXT | Required | Column in the destination table for processed data |
| destination_key_column | TEXT | 'id' | Key column in the destination table that uniquely identifies the processed data |
| options | JSONB | '{}'::JSONB | Configuration options for the data preparation operation. Utilizes the same API as the data preparation primitives. |
aidb.bulk_data_preparation
Executes the configured data preparation operation on all data from the specified preparer’s source.
Parameters
| Parameter | Type | Description |
|---|---|---|
| preparer_name | TEXT | Name of the preparer. |
aidb.delete_preparer
Deletes the preparer's configuration.
Parameters
| Parameter | Type | Description |
|---|---|---|
| preparer_name | TEXT | Name of preparer to delete. |
Note
This does not delete the destination table or any data in it.
Could this page be better? Report a problem or suggest an addition!