What is Data Flow Platform

What is Data Flow Platform

Data Flow Platform provides a SaaS-managed service using Kubernetes for moving data from various input data sources to target data destinations in-stream or bulk mode. This allows a user to use snapblocs UI to create a Data Flow Stack by configuring the data flow options, such as input source and target destination settings, and operational configuration for the target runtime environment. Kubernetes on Cloud platform (AWS, etc.) 
Also, the snapblocs UI allows a user to manage the lifecycle of the Data Flow Stack (start, update, terminate) and monitor the runtime behavior of the Stack through built-in observability features that measure how well data is moving and how internal states of the components are healthy.


​
Data Flow Platform includes the following software components:
  1. AWS EKS is used to provision stacks of Data Flow Platform using the customer's AWS account.
    Google GKE is used to provision stacks of Data Flow Platform using the customer's Google account.
    snapblocs provisions Data Flow Platforms following the well-architecture guides (i.e., AWS Well-Architected for AWS, Google Cloud Architecture Framework, etc.) for provisioning, configuring production-grade Kubernetes clusters and deploying workloads into the clusters. It provides benefits from patterns that have been used successfully for many customers in production environments. Also, snapblocs makes it easy to get started and easy to configure properly.
  2. Kubernetes is an open-source container-orchestration system for automating application deployment, scaling, and management. It is used to deploy selected Components.
  3. Kafka is used to building real-time data pipelines and streaming applications by integrating data from multiple sources and locations into a single, central Event Streaming Platform.
  4. Elastic is used to provide observability (monitoring, alerting, APM) for answering questions about what's happening inside the system just by observing the outside of the system.
  5. Grafana is used to build visualizations and analytics to query, visualize, explore metrics, and set alerts for quickly identifying system problems to minimize disruption to services.
  6. StreamSets Data Collector is a low-latency ingest infrastructure tool used to create continuous data ingest pipelines using a drag and drop UI within an integrated development environment (IDE). It is used to ingest source data in-stream or batch to other data platforms such as Data Lake, on-prem, or cloud datacenter.