Skip to main content

Error Handling Overview

Robust error handling is critical for building reliable data pipelines. NPipeline provides several mechanisms to manage errors that occur during data processing, allowing you to gracefully recover, retry operations, or isolate problematic data.

Why Error Handling Matters​

By default, if an unhandled exception occurs within a node during pipeline execution, the exception will propagate up the call stack, potentially halting the entire pipeline. While this behavior is suitable for critical errors that should stop processing immediately, it's often desirable to handle errors more selectively without bringing down the entire system.

Types of Errors in NPipeline​

Errors can generally be categorized by their source and impact:

  • Node-Specific Errors: Exceptions originating from logic within a specific ISourceNode, ITransformNode, or ISinkNode.
  • Data-Related Errors: Issues caused by the data itself (e.g., invalid format, missing values) that a node attempts to process.
  • Infrastructure Errors: Problems related to external dependencies like databases, APIs, or network connectivity.
  • Cancellation: While not strictly an "error," a CancellationToken can signal an intentional halt to processing, which nodes should handle gracefully.

Two Levels of Error Handling​

NPipeline distinguishes between two complementary levels of error handling:

1. Node-Level Error Handling​

Deals with errors that occur while processing an individual item within a specific node. You define what happens to that item:

  • Skip it and continue
  • Retry the operation
  • Redirect it to a dead-letter queue
  • Fail the entire pipeline

Use this when: Individual items fail during processing and you want to handle them without affecting other items.

2. Pipeline-Level Error Handling​

Deals with more severe errors that might affect an entire node's stream or the pipeline's execution flow:

  • Restart the failing node
  • Continue without the failing node
  • Fail the entire pipeline

Use this when: An entire node's stream fails (e.g., external service goes down) and you need to decide how the pipeline should recover.

Decision Tree: Choosing Your Approach​

Error Handling Strategies​

For Individual Item Failures (Node-Level)​

  • Retry: Transient errors (network issues, temporary resource constraints)
  • Skip: Non-critical errors or malformed data
  • Dead Letter: Problematic items for later analysis
  • Fail: When errors indicate critical system issues

For Entire Stream Failures (Pipeline-Level)​

  • Restart Node: When failures are transient and recoverable
  • Continue Without Node: When the node is non-critical to overall operation
  • Fail Pipeline: When errors indicate system-wide problems

Error Flow Visualization​

Quick Start​

For a quick introduction to error handling, see Getting Started with Resilience. This guide covers the mandatory prerequisites and basic patterns to get you started.

For detailed implementation of specific error handlers, see: