Skip to content

Script

Write Python code to clean or reshape data between steps.

When to use this

  • You need to reshape rows in a way the standard operation nodes (Join, Filter, Aggregate, Concatenate) cannot express
  • You want to derive new columns with custom logic

What you need

  • One or more upstream data sources (typically Import Data or another operation node) wired to the Script's inputs
  • Python code in the Python Code editor in the side panel

How it works

Define a main function in the Python Code editor. It receives one argument per upstream input. Each input is a list of row dicts (one dict per row, keys are the column headers). Return a list of row dicts.

You can name the function def transform(data) instead of def main(input_data). Both work exactly the same way.

pandas, numpy and openpyxl are installed alongside the standard library, pinned to the same releases SCMotif itself uses, so a node produces the same rows on every run. Nothing else is installed. A Script cannot install packages or reach the network.

Each run gets a container of its own, and it is destroyed when the run ends. A Script can write files while it runs, but nothing it writes is there next time. Every run, including the next run of this same workflow, starts with a clean filesystem and sees only the data wired into its inputs. Saving a cleaned copy to reuse later does not work; a step that tried would read nothing.

Single input

Suppose an upstream Import Data loaded a CSV of supply records. Each row arrives as a dict. Keys are the CSV column headers and values are strings (cast with float(...) or int(...) as needed):

# input_data
[
    {
        "product": "widget",
        "location": "WH_A",
        "supply_quantity": "100",
        "co2_per_unit": "2.5",
    },
    {
        "product": "widget",
        "location": "WH_B",
        "supply_quantity": "200",
        "co2_per_unit": "1.8",
    },
    {
        "product": "gadget",
        "location": "WH_A",
        "supply_quantity": "50",
        "co2_per_unit": "3.2",
    },
    {
        "product": "gadget",
        "location": "WH_B",
        "supply_quantity": "150",
        "co2_per_unit": "2.1",
    },
]

This Script totals sourcing CO₂ per product:

def main(input_data):
    totals = {}
    for row in input_data:
        product = row["product"]
        qty = float(row["supply_quantity"])
        co2 = float(row["co2_per_unit"])
        totals[product] = totals.get(product, 0) + qty * co2
    return [
        {"product": product, "total_sourcing_co2": total}
        for product, total in totals.items()
    ]

The returned list is what downstream nodes see: one dict per row, with the keys you put in becoming the columns:

# return value, visible to downstream nodes
[
    {"product": "widget", "total_sourcing_co2": 610.0},
    {"product": "gadget", "total_sourcing_co2": 475.0},
]

Multiple inputs

A Script accepts any number of inputs. Each parameter of the function is one input socket on the node, named after the parameter. Connect each upstream step to the socket it belongs to. The data arrives in that parameter no matter what order you draw the connections. The function still returns a single list of row dicts:

def main(supply_rows, demand_rows):
    supply = {r["location"]: float(r["total"]) for r in supply_rows}
    return [
        {**d, "supply_at_location": supply.get(d["location"], 0)} for d in demand_rows
    ]

Every parameter needs a connection (a parameter with a default value is optional), and every connection must land on one of the sockets. If they do not line up, the workflow editor flags the step with Inputs don't match the node's input ports before you run, naming the sockets it expects.

Alternatives

Use a dedicated node first. They need no code, are easier to read, and validate themselves:

  • Join — match rows by key columns (SQL-style joins)
  • Filter — keep rows matching a condition
  • Aggregate — group rows and compute totals
  • Concatenate — stack two same-shape datasets

For mapping raw data into a scenario, use a mapping node from the installed package. Its documentation describes the fields it accepts and the record type it produces.

Workflow wiring

A Script slots into the data pipeline anywhere you need custom logic on raw rows. It most often sits between Import Data and a mapping node, or between two operation nodes.

flowchart LR
    n1["Import Data (raw.csv)"] --> n2["Script (derive new columns)"]
    n2 --> n3["Save Data"]

For custom logic on assembled scenario data, choose a compatible node from the installed package. Script receives raw tabular rows; a scenario contains named tables whose structure and domain rules belong to the package.