Data Pipeline

Overview

When our ION Technology kicks in and starts a workflow, a cool but complex process begins. The data moves through different steps. At the core of this action is how the data gets transferred.

This happens mainly through something called the context object, using the ctx.response.data path.The key ctx.response.data is like a container that holds the data from one step before moving to the next. As each part does its job, the data inside changes.

It updates to show what just happened. We set it up this way to use memory better. This makes sure everything runs smoothly and keeps the data fresh with the latest updates.

Data pipeline diagram

However, there may arise scenarios where one may wish to retain the output from a particular node, to perhaps refer to it at a later juncture or utilize it in conjunction with other nodes as part of the overall workflow logic.

In such cases, it becomes paramount to have a means of preserving the response. This is where the utility of the variable-setter node shines through. By employing the variable-setter node, you are empowered to capture and securely store the response emanating from any node.

Check Variables: ctx.vars to better understand how to access this stored value on the workflow.

Once saved, this information remains accessible and at your disposal, ready to be retrieved and integrated into the steps that lie ahead, thereby greatly enhancing the flexibility and capability of your workflow management.