Skip to content

Script

Write Python code to clean or reshape data between steps.

When to use this

  • You need to reshape rows in a way the standard operation nodes (Join, Filter, Aggregate, Concatenate) cannot express
  • You want to derive new columns with custom logic

What you need

  • One or more upstream data sources (typically Import Data or another operation node) wired to the Script's inputs
  • Python code in the Python Code editor in the side panel
  • A name that says what the step does, such as Estimate missing freight costs, and a description of how it does it and which assumptions it makes. The code is the one part of a workflow a reviewer cannot read from the canvas, so Validate warns about a Script without either; the warning does not stop a run

How it works

Define a main function in the Python Code editor. It receives one argument per upstream input. Each input is a list of row dicts (one dict per row, keys are the column headers). Return a list of row dicts.

The node has one input for each parameter of main, named after it, so def main(orders, prices) gives it two inputs, orders and prices. A new Script starts with def main(input_data): and so has one input, input_data.

You can name the function def transform(data) instead of def main(input_data). Both work exactly the same way.

You can import the Python standard library, pandas, numpy and openpyxl. The versions are fixed, so the same code on the same rows gives the same result on every run. A Script cannot install packages or reach the network.

Each run gets a container of its own, and it is destroyed when the run ends. A Script can write files while it runs, but nothing it writes is there next time. Every run, including the next run of this same workflow, starts with a clean filesystem and sees only the data wired into its inputs. Saving a cleaned copy to reuse later does not work; a step that tried would read nothing.

Single input

Suppose an upstream Import Data loaded a CSV of supply records. Each row arrives as a dict. Keys are the CSV column headers and values are strings (cast with float(...) or int(...) as needed). Each row also carries a provenance key, a list naming the file and row it came from; it is left out of the example below. A row you return keeps it only if you copy it:

# input_data
[
    {
        "product": "widget",
        "location": "WH_A",
        "supply_quantity": "100",
        "co2_per_unit": "2.5",
    },
    {
        "product": "widget",
        "location": "WH_B",
        "supply_quantity": "200",
        "co2_per_unit": "1.8",
    },
    {
        "product": "gadget",
        "location": "WH_A",
        "supply_quantity": "50",
        "co2_per_unit": "3.2",
    },
    {
        "product": "gadget",
        "location": "WH_B",
        "supply_quantity": "150",
        "co2_per_unit": "2.1",
    },
]

This Script totals sourcing CO₂ per product:

def main(input_data):
    totals = {}
    for row in input_data:
        product = row["product"]
        qty = float(row["supply_quantity"])
        co2 = float(row["co2_per_unit"])
        totals[product] = totals.get(product, 0) + qty * co2
    return [
        {"product": product, "total_sourcing_co2": total}
        for product, total in totals.items()
    ]

The returned list is what downstream nodes see: one dict per row, with the keys you put in becoming the columns:

# return value, visible to downstream nodes
[
    {"product": "widget", "total_sourcing_co2": 610.0},
    {"product": "gadget", "total_sourcing_co2": 475.0},
]

Multiple inputs

A Script takes as many inputs as its function has parameters. Each input is a plain named parameter, and each parameter is one input socket on the node, named after it. Connect each upstream step to the socket it belongs to. The data arrives in that parameter no matter what order you draw the connections. The function still returns a single list of row dicts:

def main(supply_rows, demand_rows):
    supply = {r["location"]: float(r["total"]) for r in supply_rows}
    return [
        {**d, "supply_at_location": supply.get(d["location"], 0)} for d in demand_rows
    ]

A parameter after a bare *, as in def main(orders, *, prices), is an input like any other, with its own socket.

Every parameter needs a connection (a parameter with a default value is optional), and every connection must land on one of the sockets. If they do not line up, the workflow editor flags the step before you run, naming the sockets it expects: a parameter with nothing connected is Required input not connected.

When the code gives the node no inputs

The sockets are read off the code, so code they cannot be read from leaves the node without inputs. The draft still saves, and the workflow editor flags the step with Inputs not readable from code; Validate and Run refuse the workflow until the code is fixed. This happens when:

  • the code does not parse — the finding asks you to fix the syntax error first;
  • the code defines neither main nor transform at its top level;
  • the function takes *args, **kwargs or positional-only parameters (those before a /) — the finding asks for one named parameter per input port instead;
  • a parameter is named scmotif_context, which is reserved for the platform — the finding asks you to rename or remove it.

Alternatives

Use a dedicated node first. They need no code, are easier to read, and validate themselves:

  • Join — match rows by key columns (SQL-style joins)
  • Filter — keep rows matching a condition
  • Aggregate — group rows and compute totals
  • Concatenate — stack two same-shape datasets

For mapping raw data into a scenario, use a mapping node from the installed package. Its documentation describes the fields it accepts and the record type it produces.

Workflow wiring

A Script slots into the data pipeline anywhere you need custom logic on raw rows. It most often sits between Import Data and a mapping node, or between two operation nodes.

A Script's columns are whatever its code returns, so they are known once the workflow has run. Each run records the column names the Script produced, never the rows, and a mapping node after it lists those names and the run they came from. Change the Script or anything before it and the mapping node says the list may be out of date until the next run.

flowchart LR
    n1["Import Data (raw.csv)"] --> n2["Script (derive new columns)"]
    n2 --> n3["Save Data"]

For custom logic on assembled scenario data, choose a compatible node from the installed package. Script receives raw tabular rows; a scenario contains named tables whose structure and domain rules belong to the package.

Common mistakes

  • Collecting inputs with *args or **kwargs. The node cannot tell how many sockets that is, and Validate and Run refuse it. Name one parameter per input: def main(orders, prices).
  • Leaving the code unfinished. Code that does not parse, or has no main or transform, gives the node no inputs, and Validate and Run refuse the workflow until it is fixed.
  • Connecting fewer steps than the function has parameters. Every parameter without a default value needs a connection. Give the parameter a default to make that input optional.

Messages this node can show

This node has no messages of its own.

A run gives this node 10 minutes. A node that takes longer stops with Time limit reached.

Like every node, it can also stop on the checks before a run and the run's limits.