Process Modes

📄️ Process Modes - Overview

Data processing pipelines in Syntasa offer multiple process modes to efficiently manage data ingestion and updates. These modes are designed to handle different use cases, ensuring that data is either added, updated, or entirely replaced based on the required workflow. Choosing the right process mode depends on factors such as whether the dataset is partitioned, whether historical data should be retained, and how modifications should be managed.

📄️ Drop & Replace - Process Mode

The Drop & Replace mode is designed to completely overwrite an existing dataset with new incoming data during a job execution. This mode is useful when you want to ensure that the latest dataset reflects only the new data and does not retain any previous records.

📄️ Replace Date Range - Process Modes

In data science workflows, especially in time-partitioned datasets, managing data updates with precision is critical. Syntasa’s Replace Data Range process mode provides a reliable and controlled method to update output tables based on a specified execution date range.

📄️ Add New Only- Process Modes

The Add New Only process mode in Syntasa ensures that only new data partitions—those that do not already exist in the output table—are processed and added. Unlike other process modes, it does not overwrite or modify existing date partitions, making it an ideal choice when you only need to add new date partition without altering previously ingested records.

📄️ Add New & Replace Modified - Process Mode

The Add New & Replace Modified process mode in Syntasa is designed to efficiently handle data updates by incorporating new date partitions while ensuring that existing date partitions are updated if they have changed. This mode ensures that your dataset remains up to date by checking for both newly arriving data and modifications to previously processed records.

📄️ Process Mode — “Code Managed”

Within Syntasa, Process Modes define how the platform manages the lifecycle of output data generated by a process. In standard execution modes, Syntasa automatically handles operational behaviors such as partition cleanup, date-range filtering, incremental processing, table writes, and execution-state management. These built-in capabilities simplify common ETL and data engineering workflows by allowing developers to focus primarily on transformation logic rather than operational orchestration.

📄️ Handling Partitioned Data with “Code Managed” Process Mode

In a standard Syntasa workflow, the platform’s State Service automatically manages the lifecycle of partitioned tables. When you select a mode like Replace Date Range, the platform identifies which partitions exist in the target, drops them for the specified window, and writes new data.

📄️ Managing Non-Partitioned Data with “Code Managed” Process Mode

For non-partitioned tables, Syntasa’s standard process modes typically operate on an “all-or-nothing” basis. For example, the Drop & Replace mode deletes the entire table and recreates it with the current session’s data.

📄️ Contextual Guidance - Process Modes

As data orchestration workflows become more advanced, selecting the correct execution strategy is essential for maintaining data integrity, optimizing performance, and reducing unnecessary processing costs. To help users make more informed decisions, Syntasa 9.1 introduces Contextual Help for Process Modes.