Skip to main content

Data Processor User Manual

Overview​

A Data Processor is used to read data from a source, transform it through a series of steps, and write the results to a new data source. It supports processing of both internal and external data sources.

Data Processors support pipelines of any size. However, depending on your subscription plan, limits may apply to the type and size of the data. See our pricing page for more information.

All data processor management—including creation, editing, deletion, and listing—is done via the Configuration Panel.

Building a Data Processing Pipeline​

Using the Pipeline Editor​

When creating or modifying a Data Processor, the final step involves editing the pipeline. The pipeline editor UI is divided into four key sections:

  • Left Panel: Lists all pipeline steps. You can add, remove, reorder, or edit each step.
  • Right Panel: Contains three sections:
    • Detailed Step Editor (Top-Right): Configure the logic of the selected step.
    • Input Preview: Displays incoming data for the current step.
    • Output Preview: Shows how data is transformed by the current step.

Pipeline Step Reference​

Each step in a data pipeline performs a specific transformation. Below is a reference of all supported step types.

Filter Step​

Filters records based on one or more conditions.

Each condition consists of:

  • Field
  • Operator (e.g., equals to, greater than)
  • Value

Conditions can be combined using logical AND or OR operators.

Supported Operators​

OperatorDescription
greater thanMatches if value > specified value
less thanMatches if value < specified value
greater/equal toMatches if value ≥ specified value
less/equal toMatches if value ≤ specified value
equal toMatches if values are equal
not equal toMatches if values are not equal
inChecks if value is in a comma-separated list (e.g., apple,pear,orange)
not inChecks if value is not in a comma-separated list
is numberMatches if value is a number (also excludes empty values)
is textMatches if value is text (also excludes empty values)

Filter Value Types​

  • Text: Shown as a grey box labeled ABC in the editor.
  • Number: Shown as a grey box labeled 123.
  • Comma-separated list: Used only with in and not in.

Change Type Step​

Converts field types. Supported types:

  • Text
  • Number

Select Step​

Keeps only selected fields; others are removed.

In the Detailed Step Editor, click Add Field to include fields.

Sort Step​

Sorts data based on selected fields and specified sort order (ascending or descending).

In the Detailed Step Editor, click Add Field to define sorting fields.

Pivot Step​

Performs aggregation by grouping data using:

  • Group By Fields
  • Aggregation Fields

Supported Aggregation Operators​

OperatorDescription
SumAdds numeric values
MaximumFinds the highest value
MinimumFinds the lowest value
AverageComputes the average of numeric values
JoinConcatenates text values

Map Step​

Transforms field values using expressions.

Each mapping consists of:

  • Target Field
  • Expression

Example: Price = Price + 10

Expressions may include:

  • Fields
  • Functions (begin with $)
  • Constants (text in double quotes, or numbers)

Use the + icon in the editor to add expressions easily.

Example Expressions​

$replace(first_name, " ", "_");
// Replaces spaces in `first_name` with underscores

$concat("$", $text($round(Price, 2)));
// Rounds `Price` to 2 decimals, converts to text, and prepends "$"

Supported Functions​

Text Functions

  • $length(TEXT_EXP)
  • $subtext(TEXT_EXP, START, [LENGTH])
  • $concat(TEXT_EXP1, TEXT2, [TEXT3])
  • $upper(TEXT_EXP)
  • $lower(TEXT_EXP)
  • $trim(TEXT_EXP)
  • $contains(TEXT_EXP, TARGET)
  • $replace(TEXT_EXP, SOURCE, TARGET)
  • $startswith(TEXT_EXP, TARGET)
  • $endswith(TEXT_EXP, TARGET)

Type Casting

  • $number(TEXT_EXP) – Casts to number
  • $text(NUMBER_EXP) – Casts to text

Number Functions

  • $abs(NUMBER_EXP)
  • $floor(NUMBER_EXP)
  • $ceil(NUMBER_EXP)
  • $round(NUMBER_EXP, [PRECISION])
  • $power(NUMBER_EXP, EXPONENT)
  • $sqrt(NUMBER_EXP)
  • $fromMillis(NUMBER_EXP) – Converts milliseconds to formatted timestamp

Rename Step​

Renames one or more fields.

Rebase Step​

For JSON (hierarchical) data only. Changes the root of the data to a specific path.

Example​

Input:

[
  { "customer": { "name": "John" }, "account_balance": 100 },
  { "customer": { "name": "Mary" }, "account_balance": 20 }
]

Rebased using path customer:

[{ "name": "John" }, { "name": "Mary" }]

Zip Step​

For JSON (hierarchical) data only. Combines two arrays into one array of objects, merging element by element.

Example​

Input:

{
  "open_price": [1, 2, 3],
  "close_price": [4, 5, 6]
}

Output:

[
  { "open_price": 1, "close_price": 4 },
  { "open_price": 2, "close_price": 5 },
  { "open_price": 3, "close_price": 6 }
]

Processing Big Data​

You can process data of any size using a pipeline. For sources under 128MB, processing is free. For larger data, you must enable Big Data Processing, which requires a premium plan. See our pricing page for details.

Key Differences with Big Data Processing​

  • Processing may take longer.
  • Outputs may not be available immediately.
  • Each large data processor is marked as type "Large".
  • You can monitor execution in the Run History, available in the Configuration Panel under Data Processors.

Data Processor Management​

Creating a Data Processor​

  1. Go to the Configuration Panel.
  2. Click Data Processors in the left menu.
  3. Click Add Processor in the top-right corner.

Modifying a Data Processor​

  1. Go to the Configuration Panel.
  2. Click Data Processors in the left menu.
  3. Locate the processor.
  4. Click the gear icon to modify.

Deleting a Data Processor​

  1. Go to the Configuration Panel.
  2. Click Data Processors in the left menu.
  3. Locate the processor.
  4. Click the cross icon to delete.

Report Bugs, Request New Features, and Win $50 Every Month

We value your precious feedback. Please contact us when you find a bug or would like to request a new feature. (In the main panel, click on the bob logo on the top left corner, then in the Dropdown menu click on “contact for bugs or new features”). Every month we will select an “opinion leader” and reward him/her with $50 in cash.