Space GOAT Database Import

Space G.O.A.T (Global Object Aggregation Tool) generates and sends Searchcraft documents for ingestion given a database connection string and table name. It is a helpful option for quicking exporting from a primary data source into a Searchcraft index for situations where either a direction integration or a public feed is not available.

Prerequisites

Either PostgreSQL 16+ or MySQL 8.4+

Setup

Create a config.json file that follows this format

{
    "database_type": "postgres",                                // The type of database for ingestion. "postgres" | "mysql"
    "database_connection": "postgresql://connection_string",    // Connection string for the source database to be ingested.
    "database_table": "example_table",                          // Table name for the table to be ingested.
    "searchcraft_endpoint": "https://path/to/endpoint",         // Endpoint for Searchcraft ingestion. Configured via the dashboard.
    "index_name": "example_index_name",                         // Index for Searchcraft ingestion. Configured via the dashboard.
    "ingest_token": "example_ingest_token",                     // Searchcraft ingest token. Configured via the dashboard.
    "schema_mapping": [                                         // Mapping of source table columns to their Searchcraft counterparts.
        {
            "source_name": "postgress_id",  // Column name of the source data.
            "destination_name": "id",       // Destination name the source data should be mapped to.
            "data_type": "text",            // Data type to map to. Can be: "datetime" | "facet" | "text" | "f64" | "u64"
            "multi": false                  // Flag that dictates whether the data is ingested as an array or a single value.
        }
    ]
}

Usage

There are multiple commands available.

Display stats and analysis about the given database and index

./spacegoat analyze --config config.json

Run the ingestion process

`./spacegoat ingest --config config.json`

Remove any previously ingested documents from Searchcraft.

./spacegoat clear --config config.json