Skip to main content

Troubleshoot a single step in Data Pipelines

The debugging tool in Data Platform allows you to focus on individual steps within your data pipelines.

Re-running the entire pipeline can be time-consuming, especially for complex workflows. With the debugging tool, you can understand why a step failed, and run it again without executing the entire pipeline.

This feature is available only for custom pipelines. Feed importer pipelines and crawlers have their own error reporting mechanisms.

To use the debugging tool:

  • Go to Data Platform > Pipelines.
  • Click on a custom pipeline to see its individual steps.
  • Select the "Debug tool" view.

Pipeline templates

You'll find information about each step. The icon o the right tells you whether the step succeeded, failed, or is still running.

Re-run a single step​

In the debug view, you can test and debug an individual step without running the entire pipeline.

  • Open a pipeline and switch to the Debug Tool tab.
  • Click on the failed step you want to debug.
  • Use the available options:
    • Play icon - Re-run the step live. This executes the step as configured without affecting the rest of the pipeline.
    • Download icon - Download input and output files for further analysis. For example, in a JSON reader step, you can download the associated JSON file.

Pipeline templates

  • Review the input and output files and make corrections as needed.
  • Run the step again by clicking the play icon. Only this step of the pipeline will be executed. You will see the information on completion and time above the step.
  • After the run, you can view input and output files.

Editing pipelines is not allowed in debug mode. Switch to normal view to make changes.

When you re-run a step:

  • Files from earlier steps are used as input.
  • Actions in the step behave exactly as they would during a normal pipeline run.
  • The pipeline configuration remains unchanged.

Re-run conditions​

You can re-run a step even if:

  • The previous step does not have any files available.
  • Previous steps have not completed successfully.

In such cases, you’ll see a confirmation message explaining the potential limitations. You can proceed after acknowledging the message.

Track progress​

When a step is re-run, a progress bar or execution timestamp will appear.

You might need to refresh the page manually.

You can contact the Support team to request detailed logs and resolve complex issues for you.