Skip to main content

Architecture Overview: Cross-Project Deployment in GCP

As organizations scale their data operations, the traditional “single-project” cloud model often becomes a bottleneck for security, governance, and financial management. Syntasa addresses these challenges through a Cross-Project Deployment Architecture, which physically and logically separates the platform’s management functions from its data processing activities.

This architecture is defined by the isolation of the Control Plane and the Data Plane across distinct Google Cloud Platform (GCP) projects.

The Architectural Blueprint​

In a cross-project deployment, the Syntasa environment is divided into two primary domains.

The Control Plane (Management Project)​

The Control Plane serves as the command and control center of the Syntasa platform. It hosts the core application stack and orchestrates workflows across the environment.

Hosting​

A dedicated GCP project such as syntasa-mgmt-prod.

Core Components​

ComponentDescription
GKE ClusterHosts Syntasa microservices and platform services.
Syntasa UIUser interface for building and managing workflows.
Job OrchestratorHandles workflow scheduling and execution management.
Metadata DatabasesStores configurations, workflow definitions, and platform metadata.
Kafka Event BusCentral messaging layer for orchestration and status communication.

Primary Responsibilities​

  • User authentication and authorization
  • Workflow scheduling and orchestration
  • Monitoring and operational visibility
  • API and service management

The Data Plane (Compute & Storage Project)​

The Data Plane acts as the execution engine where workloads run and data assets reside.

Hosting​

A customer-owned or business-unit-specific GCP project such as data-science-production.

Core Components​

ComponentDescription
Dataproc ClustersExecute Spark jobs and compute-intensive workloads.
GCS BucketsStore staging data, temporary artifacts, and final outputs.
BigQuery Datasets (EventStore)Support querying, analytics, and event storage.
Pub/Sub TopicsEnable messaging and event-driven workflows.

Primary Responsibilities​

  • Large-scale data processing
  • Data storage and persistence
  • External system integrations
  • Runtime workload execution

Key Benefits of Isolation​

Enhanced Security and Reduced Blast Radius​

Separating the Control Plane and Data Plane creates a strong Zero-Trust security boundary.

Security Advantages​

  • Least Privilege Access:
    The Control Plane requires only scoped IAM permissions such as dataproc.editor within the Data Plane project rather than broad administrative access.
  • Data Sovereignty:
    Sensitive datasets remain fully isolated within the Data Plane project. Even if the Control Plane environment is compromised, project-level IAM boundaries continue protecting the underlying data assets.

Precise Cost Attribution and Billing​

Separating platform operations from compute and storage workloads simplifies cloud financial management.

Financial Benefits​

  • Direct Billing:
    Dataproc, BigQuery, and storage costs are billed directly to the Data Plane project, making departmental chargebacks and cost attribution significantly easier.
  • Quota Management:
    Heavy data-processing workloads in the Data Plane do not compete with platform services in the Control Plane for API quotas or infrastructure resource limits.

Regulatory Compliance​

For industries such as Finance and Healthcare, separating application management from data processing environments is often required for compliance.

Compliance Advantages​

  • Audit Integrity:
    GCP Cloud Audit Logs provide a clear separation between administrative actions in the Control Plane and data-access activities within the Data Plane.
  • Regional Compliance:
    Organizations can deploy the Control Plane in one region while maintaining the Data Plane in a different region to satisfy data residency and sovereignty regulations.

Technical Implementation Details​

Cross-Project Communication​

Syntasa securely connects the Control Plane and Data Plane using a high-performance integration architecture.

IAM Identity Federation​

The Syntasa Service Account located in the Control Plane project is granted IAM roles within the Data Plane project. This enables secure provisioning of resources and submission of workloads across projects.

Kafka via TCP Load Balancer​

To support real-time monitoring and status reporting, Syntasa exposes Kafka brokers through a Regional TCP Load Balancer (NLB).

This provides a stable cross-project virtual IP address that allows Dataproc nodes in the Data Plane to communicate back to the Control Plane, even across separate VPC networks.

Two-Tier Resolution Logic​

Syntasa provides a flexible routing model to determine which Data Plane should be used for a workload.

Platform Default​

A global GCP Project ID configured within the Infrastructure Settings acts as the default Data Plane destination for workloads and storage operations.

Runtime Override​

Individual Runtime configurations can specify their own GCP Project ID. This enables a single Syntasa deployment to manage multiple independent Data Planes simultaneously, such as:

  • Marketing workloads running in one project
  • Finance workloads running in another project

Summary​

The Cross-Project Deployment architecture provides the foundation for enterprise-grade data engineering within GCP environments.

By isolating the Control Plane and Data Plane, Syntasa delivers the security, scalability, governance, and financial transparency required by modern data-driven organizations while ensuring that data processing environments remain fully isolated and under customer control.