Sparkflows Copilot: Build Workflows, Pipelines, and Projects Using Natural Language

Skip repetitive manual work with Sparkflows Copilot. Describe what you want to build, and Copilot creates workflows, pipelines, and more.

Introducing Sparkflows Copilot

Sparkflows Copilot is an AI-powered assistant embedded into the Sparkflows platform. It enables prompt-based creation across multiple areas of the product, including workflows, pipelines, nodes, and projects.

Copilot focuses on accelerating setup and reducing manual work, while keeping users fully in control of what gets generated.

Configuring a Copilot

Sparkflows allows administrators to configure one or more Copilots by selecting a GenAI connection in the administration section. This setup determines which AI provider Copilot uses and how it is made available across the platform.

Once configured, Copilot can be used wherever natural language creation is supported in Sparkflows.

Generating Workflows from Natural Language

One of the core capabilities of Sparkflows Copilot is workflow generation.

Users can describe a task in plain language, such as reading a CSV file and printing the first few rows, and Copilot generates a Sparkflows workflow that reflects that request. More advanced prompts are also supported, including reading Parquet data, applying filters, selecting columns, and writing output datasets.

Rather than manually assembling nodes, users start with a generated workflow that can be reviewed and refined as needed.

ExamplePrompt:

Create a workflow that:

1. Reads a parquet file from this path “s3a://dp-nucleus-us-east-1-dev/Sparkflows-data/erics/270/”

2. Print the first 13 rows from Step 1 on a new branch

3. Filter rows where event_source = DirectGatewayRequest from Step 1

4. Select columns “record_uuid, event_source, payload” from Step 3

5. Save the output of Step 4 to this path "s3a://dp-nucleus-us-east-1-dev/Sparkflows-data/output" as a parquet file

Creating Pipelines Using Prompts

Sparkflows Copilot can also generate pipelines using natural language instructions.

When users describe how workflows should be connected or orchestrated, Copilot generates a pipeline that links the required workflows together. This allows users to define orchestration logic without manually wiring each pipeline component through the UI.

Pipeline generation follows the same prompt-driven approach as workflows, helping users move from intent to execution-ready pipelines more quickly.

Copilot Across Nodes: Prompt-Based Node Configuration

Copilot can be used at the node level to assist with configuring logic inside individual nodes.

When working with a Copilot-enabled node, users can describe the desired behaviour in natural language. Copilot then generates the corresponding logic or configuration for that node. This applies to nodes that require expressions, queries, or transformation logic, including SQL-based nodes.

This capability reduces manual effort when defining node behavior and helps users move faster while still allowing full review and control of the generated logic.Example Prompt : Provide select query for ID 1, 2, 3, 4 rows in fire_temp_table

Creating Entire Projects with Copilot

Beyond workflows and pipelines, Sparkflows Copilot can create entire projects from a natural language description.

Users can specify details such as the project name, description, category, group, and optionally provide a data dictionary to give Copilot additional context. Based on this input, Copilot creates a new project along with an initial structure and components.

This makes Copilot especially useful for project kickoffs and rapid prototyping.

Extending Copilot with MCP Connections

Sparkflows Copilot can also be integrated with external tools or clients using Model Context Protocol (MCP) connections.

With MCP, Copilot can receive natural language requests from external systems and perform actions such as creating workflows or modifying nodes without requiring direct manual interaction with the Sparkflows UI. These integrations are explicitly configured and controlled.

A Unified Prompt-Based Experience

Across workflows, pipelines, nodes, projects, and MCP integrations, Sparkflows Copilot follows a consistent model: users describe what they want to build, and Copilot generates the corresponding Sparkflows artifacts.

This prompt-based approach reduces setup time, minimizes repetitive configuration, and allows teams to focus more on outcomes rather than manual assembly.

Sparkflows Copilot brings natural language creation into the core of the data engineering workflow. By translating prompts into workflows, pipelines, node logic, and projects, Copilot helps teams move faster while maintaining transparency and control over what gets built.

It is a practical application of AI designed to streamline real-world data platform work, without disrupting existing processes.

Learn More