How to configure an Elastic Stack component

How to configure an Elastic Stack component

snapblocs use Elastic Observability for providing Observability of the running Data Platform. Observability of the Data Platform ensures that DevOps can easily detect undesirable behaviors (service downtime, errors, slow responses, etc.). And have actionable information to effectively pin down root cause (detailed event logs, granular resource usage metrics, and application performance monitoring). 
snapblocs use ELK StackMetricBeatFileBeat, and APM to bring logs, metrics, and APM traces together in an embedded Observability solution within the Data Platform so DevOps can monitor and react to events happening inside a running Data Platform environment. 
Choose either option to integrate Elastic with the Data Platform Stack.
  1. Create a new Elastic cluster within the Data Platform Stack.
  2. Integrate externally created Elastic cluster with the Data Platform Stack.
(Option 1) Create a new Elastic cluster within the Data Platform (Recommended option)
Choose this option to create a new Elastic cluster managed by the Data Platform Kubernetes infrastructure and seamlessly integrate with other Data Platform components without affecting external applications that are outside the Data Platform.
"ELK" is the acronym for three open source projects: Elasticsearch, Logstash, and Kibana. Elasticsearch is a search and analytics engine. Logstash is a server‑side data processing pipeline that ingests data from multiple sources simultaneously, transforms it, and then sends it to a "stash" like Elasticsearch. Kibana lets users visualize data with charts and graphs in Elasticsearch.
Elasticsearch is a highly scalable open-source full-text search and analytics engine. Store, search and analyze big volumes of data quickly and in near real-time. It is generally used as the underlying engine/technology that powers applications with complex search features and requirements.
Logstash is an open-source, server-side data processing pipeline that ingests data from a multitude of sources simultaneously, transforms it, and then sends it to your favorite "stash."
Kibana is an open-source data visualization dashboard for Elasticsearch. It provides visualization capabilities on stop of the content indexed on an Elasticsearch cluster. Users can create bar, line, scatter plots, or pie charts and maps on top of large volumes of data.
Filebeat is a lightweight shipper for forwarding and centralizing log data.
Metricbeat is a lightweight shipper that you can install on your servers to periodically collect metrics from the operating system and from services running on the server. 

Motivation for CPU requests and limits
Configure the CPU requests and limits of the Containers that run in the cluster, efficiently using the CPU resources available on the cluster nodes. By keeping a Pod CPU request low, it gives the Pod a good chance of being scheduled. Having a CPU limit that is greater than the CPU request, accomplishes two things:
  • The Pod can have bursts of activity, making use of CPU resources that happen to be available.
  • The amount of CPU resources a Pod can use during a burst of activity is limited to a reasonable amount.
If CPU limit is not specified for a Container, it can result in one of these situations:
  • The Container has no upper bound on the CPU resources it can use. The Container could use all of the CPU resources available on the node where it is running.
  • The Container runs in a namespace with a default CPU limit, and the Container is automatically assigned the default limit. Cluster administrators can use a LimitRange to specify a default value for the CPU limit.
ElasticSearch Settings
  • Secure access to the Kibana dashboard using User Name and Password. The original Kibana dashboard provided by Elastic is ordinarily accessible by anyone who can access the Kibana URL, opening up the security and maintenance issues due to unauthorized personnel accessing the Kibana dashboard. snapblocs adds basic authentication for the Kibana dashboard login using User Name and Password defined at the stack configuration.
  • Use Timeout in Seconds to control how long the Kubernetes controller waits for the successful creation of Elastic and its sub-components. If it times out, the Elastic component has failed. If it times out frequently, increase this value
  • Minimum Master (Eligible) Nodes. The master node is responsible for lightweight cluster-wide actions such as creating or deleting an index, tracking which nodes are part of the cluster, and deciding which shards to allocate to which nodes. Cluster health requires a stable master node. High availability (HA) clusters require at least three master-eligible nodes, at least two of which are not voting-only nodes. Such a cluster will be able to elect a master node even if one of the nodes fails.
(Option 2) Integrate externally created Elastic cluster with a Data Platform Stack
Choose this option for an external Elastic cluster within the Data Platform. Be aware that any changes and usages that occur during running Data Platform may impact external applications (system) that depend on the Elastic cluster.

What's Next


    • Related Articles

    • How to configure Elastic Stack+ Platform

      You can initiate configuring a new stack from a few different places: On the Home page, "Configure stack" button on the Stacks statistics block. On the Stacks page, the "Configure new stack" button on the top page On the Projects page, select Project ...
    • How to customize Elastic Stack+ Platform

      After configuring a new stack of Elastic Stack+ Platform by following this, you can customize the stack. Test / Proof of Concept (POC) Stack To create a simple test Elastic Stack+ Platform stack, set the following parameters. On AWS & K8S component: ...
    • How to configure new stack

      You must configure a stack configuration before deploying the stack to your cloud environment.  You can initiate configuring a new stack from a few different places: On the Home page, "Configure stack" button on the Stacks statistics block. On the ...
    • How to configure a Grafana component

      Grafana is open-source visualization and analytics software. Query, visualize and explore key metrics, set an alert to quickly identifying problems with the system to minimize disruption to services. snapblocs uses Grafana with Elastic Stack (ELK) to ...
    • How to configure a Kafka component

      Kafka is a distributed streaming platform used to publish and subscribe to streams of records. Kafka gets used for fault-tolerant storage. Kafka replicates topic log partitions to multiple servers. Use it to stream data to other data platforms such ...