AWS is one of the most comprehensive and broadly adopted cloud platforms providing on-demand cloud computing platforms. It offers over 175 fully-featured services from data centers globally.
Kubernetes is a system for automating the deployment, scaling, and management of containerized applications.
AWS EKS, a fully-managed Kubernetes service, is used on the Kubernetes platform for the AWS environment. EKS is a certified Kubernetes conformant, leveraging all benefits of open source Kubernetes tooling from the community. Integrate with AWS services for a scalable and highly available control plane that runs across multiple availability zones to eliminate a single point of failure.
snapBlocs provisions Data Platform on AWS using AWS Well-Architected guides for provisioning and configuring production-grade Kubernetes clusters and deploying workloads into the clusters. snapBlocs make it easy to get started and provides benefits from patterns that worked for many customers who are in production.
After creating a stack with an AWS and K8S for DaaS component, customize the AWS and K8S for DaaS components with the following settings.
Provider Key Name
Use the Provider Access Key to deploy a stack to a specific AWS account.
Provider Access Keys created at the account level are available for use by all projects and their stacks. In essence, they have shared keys.
Stacks can only use Provider Access Keys created at the project level within that project.
Stacks can use a key from their project or an account key.
Delete Provider Access Keys when not in use by stacks.
VPC
Amazon Virtual Private Cloud (Amazon VPC) is a logically isolated section of the AWS Cloud. Deploy a snapblocs platform with all required AWS resources as a single unit in a custom-defined virtual network.
Each AWS account comes with a default VPC and default Subnets in each AWS Region. A default VPC is ready to use without having to create and configure a custom VPC.
Use a default VPC suitable for getting started quickly or as a prototype for launching public instances such as a blog or simple website.
However, as the default VPC is not ideal for a secure production system, use a custom VPC.
Kubernetes API server Endpoint Access
Amazon EKS creates an endpoint for the managed Kubernetes API server used to communicate with the cluster (Kubernetes management tools such as kubectl). By default, this API server endpoint is public to the internet, and access to the API server is secured using a combination of AWS Identity and Access Management (IAM) and native Kubernetes Role-Based Access Control (RBAC). The private-only access option is currently not supported for now.
Kubernetes Configuration
A Kubernetes cluster consists of a set of worker machines, called nodes, that run containerized applications. The worker node(s) host the Pods that are the components of the application workload.
Enter the desired number of nodes to scale to for the stack.
Enter the minimum number of Dremio nodes that the node Auto Scaling group can scale into for the stack.
Enter the maximum number of nodes that the node Auto Scaling group can scale out to for the stack.
Min Size <= Desired Capacity <= Max Size
To enable secure access to the worker nodes, use the EC2 Key Pair.
Execute custom bootstrap scripts stored on S3 object when launching worker nodes. Customize Kubernetes worker nodes for installing required libraries, etc.
Dremio Node Group Configuration
In Kubernetes, node groups are a mechanism for creating pools of resources that can enforce scheduling requirements.
Dremio requires different types of resources (such as a larger AWS EC2 Instance Type) not needed for other components, DaaS Platform splits the node group into two node groups: Standard node group and Dremio node group.
A standard node group is for hosting all components except Dremio.
A Dremio node group is for hosting only Dremio components that require r5d.4xlarge (16 core, 128 GiB memory, and 2 x 300 GB NVMe SSD)
Enter the desired number of Dremio nodes to scale for the stack.
Enter the minimum number of Dremio nodes that the node Auto Scaling group can scale into for the stack.
Enter the maximum number of Dremio nodes that the node Auto Scaling group can scale out to for the stack.
Min Size <= Desired Capacity <= Max Size
Execute custom bootstrap scripts stored on S3 object when launching worker nodes. Customize Kubernetes worker nodes for installing required libraries, etc.
Select the AWS S3 Bucket option for the Dremio Distributed Storage Type on the production deployment. An S3 bucket is created during stack deployment if no S3 bucket exists for the Dremio distributed storage in the cloud environment.
Reference