How to configure an AWS and K8S for DaaS component

How to configure an AWS and K8S for DaaS component

AWS is one of the most comprehensive and broadly adopted cloud platforms providing on-demand cloud computing platforms. It offers over 175 fully-featured services from data centers globally.
Kubernetes is a system for automating the deployment, scaling, and management of containerized applications.
AWS EKS, a fully-managed Kubernetes service, is used on the Kubernetes platform for the AWS environment. EKS is a certified Kubernetes conformant, leveraging all benefits of open source Kubernetes tooling from the community. Integrate with AWS services for a scalable and highly available control plane that runs across multiple availability zones to eliminate a single point of failure.
snapBlocs provisions Data Platform on AWS using AWS Well-Architected guides for provisioning and configuring production-grade Kubernetes clusters and deploying workloads into the clusterssnapBlocs make it easy to get started and provides benefits from patterns that worked for many customers who are in production.
After creating a stack with an AWS and K8S for DaaS component, customize the AWS and K8S for DaaS components with the following settings.
Provider Key Name
  • Use the Provider Access Key to deploy a stack to a specific AWS account.
  • Provider Access Keys created at the account level are available for use by all projects and their stacks. In essence, they have shared keys.
  • Stacks can only use Provider Access Keys created at the project level within that project. 
  • Stacks can use a key from their project or an account key.
  • Delete Provider Access Keys when not in use by stacks.
  • See AWS Provider Access Method for details for creating Provider Access Keys.
VPC
  • Amazon Virtual Private Cloud (Amazon VPC) is a logically isolated section of the AWS Cloud. Deploy a snapblocs platform with all required AWS resources as a single unit in a custom-defined virtual network.
  • Each AWS account comes with a default VPC and default Subnets in each AWS Region. A default VPC is ready to use without having to create and configure a custom VPC.
  • Use a default VPC suitable for getting started quickly or as a prototype for launching public instances such as a blog or simple website. 
  • However, as the default VPC is not ideal for a secure production system, use a custom VPC.
Kubernetes API server Endpoint Access
  • Amazon EKS creates an endpoint for the managed Kubernetes API server used to communicate with the cluster (Kubernetes management tools such as kubectl). By default, this API server endpoint is public to the internet, and access to the API server is secured using a combination of AWS Identity and Access Management (IAM) and native Kubernetes Role-Based Access Control (RBAC). The private-only access option is currently not supported for now.
Kubernetes Configuration
  • A Kubernetes cluster consists of a set of worker machines, called nodes, that run containerized applications. The worker node(s) host the Pods that are the components of the application workload. 
  • Enter the desired number of nodes to scale to for the stack.
  • Enter the minimum number of Dremio nodes that the node Auto Scaling group can scale into for the stack.
  • Enter the maximum number of nodes that the node Auto Scaling group can scale out to for the stack.
  • Min Size <= Desired Capacity <= Max Size
  • Choose an instance type for the nodes. Before choosing an Arm instance type, make sure to review the considerations in Amazon EKS optimized Arm Amazon Linux AMIs.
  • To enable secure access to the worker nodes, use the EC2 Key Pair.
  • Execute custom bootstrap scripts stored on S3 object when launching worker nodes. Customize Kubernetes worker nodes for installing required libraries, etc.
Dremio Node Group Configuration
In Kubernetesnode groups are a mechanism for creating pools of resources that can enforce scheduling requirements. 
Dremio requires different types of resources (such as a larger AWS EC2 Instance Type) not needed for other components, DaaS Platform splits the node group into two node groups: Standard node group and Dremio node group.
A standard node group is for hosting all components except Dremio.
A Dremio node group is for hosting only Dremio components that require r5d.4xlarge (16 core, 128 GiB memory, and 2 x 300 GB NVMe SSD)
  • Enter the desired number of Dremio nodes to scale for the stack.
  • Enter the minimum number of Dremio nodes that the node Auto Scaling group can scale into for the stack.
  • Enter the maximum number of Dremio nodes that the node Auto Scaling group can scale out to for the stack.
  • Min Size <= Desired Capacity <= Max Size
  • Choose an instance type for the nodes. Before choosing an Arm instance type, make sure to review the considerations in Amazon EKS optimized Arm Amazon Linux AMIs.
  • Execute custom bootstrap scripts stored on S3 object when launching worker nodes. Customize Kubernetes worker nodes for installing required libraries, etc.
  • Select the AWS S3 Bucket option for the Dremio Distributed Storage Type on the production deployment. An S3 bucket is created during stack deployment if no S3 bucket exists for the Dremio distributed storage in the cloud environment.
Reference

1.1-JPE



    • Related Articles

    • How to configure an AWS and K8S component

      Amazon Web Services (AWS) AWS is one of the most comprehensive and broadly adopted Cloud platforms providing on-demand cloud computing platforms. It offers over 175 fully-featured services from data centers globally. Kubernetes is a system for ...
    • How to configure a Kafka component

      Kafka is a distributed streaming platform used to publish and subscribe to streams of records. Kafka gets used for fault-tolerant storage. Kafka replicates topic log partitions to multiple servers. Use it to stream data to other data platforms such ...
    • Configure Components

      Before proceeding, click here to configure a stack. Each component must be configured before deploying the stack. As the most of the components come with reasonable default values, Complete the following configuration settings: AWS and K8S component ...
    • How to add other IAM users to provide access to AWS EKS

      After deploying a stack, if someone tries to access AWS EKS using kubectl, (s)he will get the following authorization error:       "error: You must be logged in to the server (Unauthorized)." This happens when the Amazon EKS cluster is created by an ...
    • How to configure a Dremio component

      Dremio is a data lake engine that liberates big data with live, interactive queries directly on cloud data lake storage. Dremio delivers secure, self-service data access and lightning-fast queries directly on cloud providers AWS, GCP, or Azure. In ...