ATOM Deployment Guide

ATOM deployment and operations requires hands-on experience installing Kubernetes clusters and deployment using Helm charts. This document assumes that you are familiar with Docker, containers, hypervisors, networking, and a good working knowledge of the operating systems.

Overview of ATOM Architecture

ATOM software is containerized and runs on a Kubernetes cluster. ATOM is provided as a self-contained installation package with all the required components:

ATOM Deployment

ATOM deployment requires software components to be deployed in a Kubernetes environment. Software will be distributed through a central repository.

Deployment scenarios

ATOM can be deployed in any of the following environments:

On-Prem Kubernetes
Google Cloud Platform (GCP)
Amazon web service (AWS)

ATOM can be deployed with all the components at a single location or some of the components distributed.

Local Deployment
Distributed Deployment

Local Deployment

Local deployment has all the ATOM software components deployed in a single Kubernetes cluster.

Distributed Deployment

Distributed deployment allows ATOM software components to be distributed across multiple logical or geographical locations. Distributed Deployment is applicable in the following scenarios:

Remote Agent – In some scenarios network equipment is distributed across different locations. ATOM Agent can be deployed close to the Network equipment for Security or performance reasons.
Geo-redundant HA – ATOM Components can be deployed across multiple Locations/Sites within the same region to provide Fault Tolerance against an entire Site/Location going down. More details in ATOM Multi Availability Zone based HA.

Target Infrastructure

ATOM Can be deployed On Premises, Cloud or a combination as summarized in the Table below.

Environment	Description	Use case	Prerequisites
Cloud (Amazon, GCP or Similar)	Typically for Staging & Production Deployments	Development Stage Production	Hardware Requirements
On Premises	Typically for Staging & Production Deployments Can be used for Multi user shared Development as well	Development Stage Production	On-Prem VMware ESXi, KVM Hardware Requirements
	ATOM Agent can be deployed	Development Stage Production
Cloud + On Premise	on-Premises while rest of ATOM can be deployed in the	On-Prem VMware ESXi, KVM Hardware Requirements
	Cloud

On-Prem VMware ESXi, KVM

For the Kubernetes cluster deployed on ESXi, KVM etc., make sure required Compute, Storage & Memory resources for VM nodes are allocated to have ATOM running on top of K8s cluster.

Anuta provides the OVA images for K8s Master and Worker nodes creation on ESXi, while the OVA’s can be converted to Qcow2 images to deploy K8s Master and Worker nodes on KVM.

Cloud (GCP / AWS)

As cloud deployments on GCP/AWS offer different variants of node-types, make sure the Node Type you selected matches the resources required for a Worker Node mentioned in Compute, Storage & Memory requirements(Separate Master Node not required in GCP/AWS).

For GCP deployment a e2-highmem-4 or custom-4-32768-ext Node type would be required and a r6i.xlarge Node type for AWS deployment.

Requirements

Before deploying ATOM in the kubernetes cluster, ensure that the following requirements are satisfied:

Hardware Requirements
Network Requirements
Kubernetes Cluster Requirements
Software Requirements

Compute, Storage & Memory

Note:

Overview

Type	Minimal 1 DC	Resilient 1 DC	Resilient 2 DC	Resilient 3 DC
Master Nodes Per DC	3	3	2 and 1*	1
Master Specs	4 vCPU, 32GB RAM, 300GB SSD	4 vCPU, 8GB RAM, 50GB SSD	4 vCPU, 8GB RAM, 50GB SSD	4 vCPU, 8GB RAM, 50GB SSD
Master Node shares workload	Yes	No	No	No
Worker Nodes Per DC	1	9	4	3

Worker Specs	4 vCPU, 32GB RAM, 300GB SSD	4 vCPU, 32GB RAM, 300GB SSD	4 vCPU, 32GB RAM, 300GB SSD	4 vCPU, 32GB RAM, 300GB SSD
Total Nodes	4	12	11	12
Total IP requirements	4+3(VIP)=7	12+3(VIP)=15	11+3(VIP)=14	12+3(VIP)=15

Refer IP Addressing Section for more details

*Refer 2 Sites Deployment section below

Minimal Setup

A Minimal setup that doesn’t support resiliency for ATOM components but has Kubernetes HA needs a cluster (3 masters and 1 worker node) based out of ESXi with recommendations listed below

Component

Requirements Description

K8s Master – 3 nodes

For each node storage reserved in ESXi = 300 GB (SSD)

CPU – 4 vCPU
Memory – 32GB

K8s Workers – 1 node

For each node storage reserved in ESXi = 300 GB (SSD)

CPU – 4 vCPU
Memory – 32GB

Total IP Address: 4 IPs + 3 VIPs = 7 IPs. Refer IP Addressing Section for more details. In this model K8s Master shares the workloads of ATOM components as well.

Resilient HA Setup

HA setup supporting resiliency with regards to one node or pod failures requires a Kubernetes cluster (3 masters and 9 worker nodes) based out of ESXi with the following details

Component

Requirements Description

K8s Master – 3 nodes

For each node storage reserved in ESXi = 50 GB (SSD)

CPU – 4 vCPU
Memory – 8GB

K8s Workers – 9 nodes

For each node storage reserved in ESXi = 300 GB (SSD)

CPU – 4 vCPU
Memory – 32GB

Total IP Address: 12 IPs + 3 VIPs = 15 IPs. Refer IP Addressing Section for more details. In this model K8s Master shares the workloads of ATOM components as well.

Multi-site Deployment (Remote ATOM agent)

For a Multi-site distributed deployment, where the ATOM agent is deployed remotely, a single ATOM agent (minimum) is deployed at each site in addition to the above setup choices. A Virtual Machine with below minimum spec is required at each site location(s):

Component

Requirements Description

1 Virtual Machine

Storage reserved in ESXi = 50 GB (SSD)

CPU – 4 vCPU
Memory – 8GB

Total IP Address: 1 IP. Refer IP Addressing Section for more details

ATOM Multi Availability Zone HA Deployment

ATOM supports deployment across multiple sites (aka Availability Zones) to support high availability in the event of a site failure provided these sites are connected over low latency links. This requires ATOM Components to be deployed across multiple sites or Availability Zones (AZs). Availability Zones are available when workloads are provisioned in a Cloud Service Provider. In this scenario, Kubernetes Cluster extends to multiple sites/Zones.

If ATOM is deployed in a single location, it is recommended that master nodes and worker nodes are at least deployed on 3 separate physical servers. In such cases, ATOM will continue to be available in the event of a single physical machine failure.

References:

Caveats:

Full Fault Tolerance against one Site failure requires ATOM deployment across 3 Locations/Sites.
In case only 2 Sites/Locations are available:
- Full Fault Tolerance against one Site failure is supported, however, due to the quorum requirements of some of the components like etcd, manual intervention may be needed if the site that has majority is down.
Multi-region deployment is not supported, ATOM clusters can only be deployed across multiple AZs within a region due to low latency requirements (<10 ms).
Some ATOM Components that support deployment across multiple Availability Zones or sites are sensitive to Latency. In such scenarios, there will be an impact on application performance or throughput

3 Sites Deployment:

For Each Site:

Component

Requirements Description

K8s Master – 1 nodes

For each node storage reserved in ESXi = 50 GB (SSD)

CPU – 4 vCPU
Memory – 8GB

K8s Workers – 3 nodes

For each node storage reserved in ESXi = 300 GB (SSD)

CPU – 4 vCPU
Memory – 32GB

Total IP Address across 3 sites: 12 IPs + 3 VIPs = 15 IPs. Refer IP Addressing Section for more details

2 Sites Deployment:

Site-1:

Component

Requirements Description

K8s Master – 2 nodes

For each node storage reserved in ESXi = 50 GB (SSD)

CPU – 4 vCPU
Memory – 8GB

K8s Workers – 4 nodes

For each node storage reserved in ESXi = 300 GB (SSD)

CPU – 4 vCPU
Memory – 32GB

Site-2:

Component

Requirements Description

K8s Master – 1* nodes

For each node storage reserved in ESXi = 50 GB (SSD)

CPU – 4 vCPU
Memory – 8GB

K8s Workers – 4 nodes

For each node storage reserved in ESXi = 300 GB (SSD)

CPU – 4 vCPU
Memory – 32GB

Total IP Address across 2 sites: 11 IPs + 3 VIPs = 14 IPs. Refer IP Addressing Section for more details

*In site-1 disaster case site-2 needs two master nodes to be deployed instead of one. Hence the required second master spec needs to be kept available ahead to handle disaster of site-1.

AWS Availability Zones

Refer to section Deploying New K8s Cluster for ATOM deployment in AWS which uses the Availability Zones(AZ) during deployment.

On Premises Across Data Centers / Locations

For on-premises deployment of a Multi Availability Zone Model across different sites, latency requirements have to be met.

Refer to section Deploying New Kubernetes Cluster for On Premises ATOM deployment which creates K8s cluster among Master and Worker Nodes across the ESXis/Locations/DataCenters having reachability.

Network Requirements

ATOM related Ports/Protocols

Each of the components of the ATOM application communicate with each other and external using the following ports and protocols.

Wherever applicable, Firewall rules need to be updated to allow the communication between external clients to ATOM or from ATOM software to network infrastructure or between ATOM software components.

End Points	Port	Communication protocol	Notes
Northbound communication [External clients, access to ATOM Portal, and other ATOM Mgmt Clients]
On-Prem Deployment of ATOM
ATOM Server (End user Application- NonSSO)	30443	HTTPS access	This will be the ATOM GUI page served via HAproxy.
Single Sign-On	443	HTTPS access	For Single Sign-On login over the VIP to access ATOM, Grafana, Kibana, Glowroot and Kafka-Manager
Minio	31311	HTTP access	ATOM FileServer/Minio Access
AWS Deployment of ATOM
Single Sign-On	443	HTTPS access	For Single Sign-On login over the VIP to access ATOM, Grafana, Kibana, Glowroot and Kafka-Manager
Minio	31311(NodePort)	HTTP access	ATOM FileServer/Minio Access

Inter-components communication [Applicable when ATOM agent and server components are deployed separately with possibly a firewall in between]
ATOM Server – ATOM Agent	7000	TCP/RSocket	Remote Agent communicates with Agent-proxy via agent-lb.
ATOM Agent – Application Performance Monitoring	8181	TCP/Grpc	Remote Agent drops performance metrics to central APM
Southbound communication with network devices from ATOM Agent [Applicable when a Firewall is present between ATOM agent and devices/NetworkElements]
ATOM Agent – Network Elements	80	Hypertext transfer protocol (HTTP) (IN/OUT)	OUT – Outbound IN – Inbound Different ports are used for various use cases in ATOM. Make sure PING reachability is also there.
443	Hypertext transfer protocol secure (HTTPS) (IN/OUT)
23	Telnet to network devices (TCP) (OUT)
21	FTP to network device (TCP) (IN)
22	SSH to network devices (TCP) (OUT)
161	SNMP to network devices (UDP) (OUT)
162	SNMP Trap Listening (Server) from network devices (UDP) (IN)
69	TFTP to network devices (UDP) (IN)
514	SYSLOG Listening (Server) port from network devices (UDP) (IN)
830	NETCONF to network devices(TCP) (OUT)
12455	Telemetry Server for TCP Communication (IN)
12456	Telemetry Server for UDP Communication (IN)
12454	Telemetry GRPC Server (IN)
2055	Netflow UDP Server (IN)

Please ensure that public access is available on all the nodes. If public access cannot be provided across the nodes then we need to consider an offline mode of installation of Atom Software by hosting Registry within your network.

Below are details of public domains which ATOM would access for pulling docker images and other binaries.

ATOM -> Required Public Access Details for Firewall if applicable.	Port/Protocol	Domain Name
443/https	registry-1.docker.io
443/https	quay.io
443/https	gcr.io
443/https	grafana.com
443/https	codeload.github.com
443/https	deb.debian.org
443/https	registry.opensource.zalan.do
443/https	ghcr.io

Kubernetes related Ports/Protocols(on-prem)

Below are Ports and Protocols which need to be allowed for Kubernetes cluster creation among VM nodes. These need to be allowed in Firewall if in between VMs there is a Firewall when VMs are spread across DCs etc..

Ports	Protocol	Notes
443	TCP	kubernetes API server(HA mode)
6443	TCP	kubernetes API server
2379-2380	TCP	etcd server client API
10250	TCP	Kubelet API
10251	TCP	Kube-scheduler
10252	TCP	Kube-controller-manager
10255	TCP	Kubelet
179	TCP	Calico CNI
9100	TCP	Prometheus
30000-32767	TCP	NodePort services

6783

TCP

Weaveport(deprecated)

Linstor related Ports/Protocols(on-prem)

ATOM uses linstor CSI driver as a storage provisioner on on-premises deployments. Below specified Ports and Protocols need to be allowed for Kubernetes among VM nodes related to Linstor. If the kubernetes cluster spreads across multiple DCs, these ports and protocols need to be open on the DC firewalls as well.

Protocol: TCP, Ports: 3366-3367, 3370, 3376-3377, 7000-8000

IP Addressing Requirements

One IP for each of the VM nodes.
For Minimal and HA Master setup, when 3 Masters are used, reserve one extra IP(virtual IP) belonging to the same subnet as other 3 Masters.
Two IPs(virtual IP) for application internal load-balancing related to api-server & agents.
IP addresses of all kubernetes nodes and virtual IPs should be in the same L2 segment.
Subnet 10.200.0.0/16 subnet is used internally in the ATOM kubernetes cluster for communication between microservices. If this subnet conflicts with any of the existing network device IPs then a different subnet of size /16 shall be chosen. Update the file wrapper.properties with the chosen subnet.

Kubernetes Cluster Requirements

ATOM Software needs to be installed on a dedicated kubernetes cluster and it can be deployed on the following Kubernetes Distributions:

Amazon EKS
Google GKE
Upstream Kubernetes (https://github.com/kubernetes/kubernetes). Anuta provides CentOS based OVAs/QCOW2 images. These are customized images that include all the required software components. ATOM kubernetes cluster can only be installed on nodes created by these images.

Any other Kubernetes Distribution or node OS distribution requires additional validation from Anuta and requires significant lead time depending on the distribution.

Anuta provides deployment artifacts such as OVA/QCOW2 images for master and worker nodes, scripts for creating the Kubernetes cluster and the container images required for deploying ATOM.

For creating a Kubernetes cluster, check if the following requirements are satisfied:

All the hardware requirements defined in the section, Hardware Requirements are met.

Anuta provided OVAs (Centos server with pre-installed minimal packages) are already imported into the vCenter template library.
Use Anuta provided QCOW2 images for deploying on KVM.
Static IPs are assigned to master and worker nodes

For bootstrapping the Kubernetes cluster, run the installation script. Installation script will need inputs like VM IPs, Gateway, Netmask, DNS Server, NTP Server details and will install all the required components such as

- Docker-ce
- Kubectl
- Helm

Once the Kubernetes cluster is formed, the ATOM deployment can be done subsequently. Refer to the section, “Procedure for Deploying ATOM”.

Deployment scripts and files

To simplify the deployment of Kubernetes clusters in your environment, the required scripts and files are organized into folders and are provided by Anuta Networks (in a zipped format).

Name of the file/folder	Description
ATOM	ATOM’s deployment files
node_setup.py	Helper Script to bootstrap the nodes and install the atom software.

ATOM Software Requirements

Before proceeding with the deployment of ATOM application, you must have the following software artifacts with you, obtained from Anuta Networks:

- Deployment Images
- Deployment scripts and files

Deployment Images

All the images required for deploying the components of ATOM will be pulled from the repositories, created in Quay (https://quay.io/repository/).

The images have been tagged with a specific name, in the format given below:

Deployment scripts and files

Deploying ATOM in the local setup involves deploying the components required to build the ATOM application using Helm charts. To simplify the deployment in your environment, the required scripts and files are organized into folders and are provided by Anuta (in a zipped format).

Name of the file/folder	Description
ATOM	ATOM’s deployment files
scripts	Check and install kubernetes, docker, helm, python packages

The key folder ATOM, contains Helm charts, templates and the deployment scripts which will be used for ATOM deployment. It has Helm charts like below

- databases — contains the deployment files of all databases – PolicyDB and kafka
- atom — contains multiple charts of individual microservice
- Infra — contains charts related to infra components such as web-proxy, logstash, glowroot etc.
- external-services — optional services to access external services like databases, kafka etc.
- grafana — contains the helm charts for Grafana monitoring tool
- persistence — contains the yaml files for creating persistent volumes
- tsdb-server and and tsdb-monitoring — contains the helm charts for tsdb
- minio — contains helm charts for minio/object storage
- sso — contains helm charts for sso objects
- metallb – contains helm charts for providing virtual load balancing service

Each of the above folders contains the following:

README.md – Detailed readme information
chart.yaml – Contains the information about the chart
values.yaml – Default configuration values for this chart
templates – A directory of templates containing the template, which when combined with values provided in the run-time generate a valid Kubernetes manifest file.

Security Apps on VM nodes before ATOM install

Users can install any security agents or clients on the VM nodes to meet their internal security compliance policies. Example – Trend Micro. Users have to make sure that these agents or clients shall not interfere with kubernetes processes and applications so that they are not modified when the ATOM is in running state. For information on ports that are used by Kubernetes and ATOM applications, please refer to section Networking Requirements.

Procedure for Deploying ATOM on-prem

ATOM applications can be deployed on new Kubernetes with help of Deployment scripts and files provided by Anuta.

New Kubernetes cluster

Verify that you have imported the shared Anuta OVA templates into your VMware vCenter.
Create master nodes and worker nodes – See section requirements for more details
Login credentials for these nodes will be atom/secret@123. For any python script executions use sudo for which password is again secret@123

NOTE: Do not login with root username into VMs

Run the node_setup.py which is present in the home directory using sudo privileges as shown below [Note:This script needs to be run on each node individually]:

Enter 1 (master) or 2(worker) depending on the type of node that you want to provision.

Choose among the following:

- Bootstrap Script: This script will initially help you set up basic Network Connectivity, Hostname configuration and NTP settings.
- Atom Installation: This script will be used to deploy k8s and bring up the atom software at a later stage. Complete steps 4-7 before invoking this.
Enter 1 to proceed with the bootstrap function and select the complete fresh setup by again choosing 1 as shown below:
Provide the following inputs as requested by the script:
- Interface Details to be provisioned along with relevant CIDR info.
- DNS Server Information
- NTP Server Information
- Hostname of the VM along with the hostname-ip to bind. Refer the screenshot below:

Network Configuration Details

NTP Server Configuration Details

Hostname Configuration Details

Once the bootstrap is complete proceed with the next steps. [Note: Hostname changes would be reflected on reboot only. Select yes to reboot if you wish to change the hostname]

Make sure Internet access is there from all the nodes.
After completion of the bootstrap process with VM reload, we are now ready to begin the atom installation process. Run the sudo python node_setup.py on Master Node for ATOM Installation.
Since it is a fresh install where the K8s cluster was also not created before, you can choose option1(Recommended) for Complete process of ZipDownload+K8s+ATOM Deployment (or) you can choose separately option 2(Only Download), option 3(only K8s) and option 5(ATOM).

In case K8s cluster is already setup, one can directly proceed with just Atom software download and installation by selecting appropriate choices of 2, 5 respectively as in the ATOM Deployment section.

To download the Atom Software Deployment zip provided by Anuta Networks Support team we can use any of the methods as seen below:

Wget: Utility for non-interactive download of files from the Web. User needs to enter the link as input and the files would be downloaded and extracted automatically.

SCP: Securely transferring computer files between a local host and a remote host based on the Secure Shell (SSH) protocol.
Manual: User can use any standard file transfer protocol to transfer the files on the home directory of the atom user.

After the ATOM deployment zip installation files are copied on the Master Node we can begin with the K8s deployment. Depending on whether we want a minimal or resilient setup provide the inputs as shown below:

Minimal Setup deployment

Choose 1 as Data center locations, select M for minimal size, provide VM IPs and Virtual IPs info.

Resilient Setup deployment

Zone or DC location

Choose 1 as Data center locations, select R for resilient size, provide VM IPs and Virtual IPs info. Virtual DCs(3 nos) are created to maintain resiliency in this scenario.

Choose 2 as Data center locations, select R for resilient size, provide VM IPs and Virtual IPs info. Input 2 master IPs for zone/DC 1 and 1 master IP for zone/DC 2. Provide 4 workers for each zone as shown below.

Zones or DC locations(Recommended)

Choose 3 as Data center locations, select R for resilient size, provide VM IPs and Virtual IPs info. Provide worker input as per zone/DC requirement.

Note: Provide DC input alongside as shown below

Above will create K8s cluster among Master and Worker Nodes spread across Esxi/Locations which have reachability.
On a different shell terminal to master node, you can as well verify the nodes cluster formation using the command “kubectl get nodes” and verify labels using the command “kubectl get nodes –show-labels”
If the deployment model is selected as offline then provide the Registry IP address and Project repo name as provided during the docker registry installation.
As now the Kubernetes cluster’s creation is done and it is ready for ATOM deployment. If option1 is chosen in step8 then, “ATOM Deployment” will happen next in the process or it can be invoked separately as well with option5.

ATOM Deployment

After ensuring that the prerequisites are taken care as described in the section, “Prerequisites for Deploying ATOM”, perform the following steps:

For Minimal or Resilient HA setup, ensure that K8s cluster is formed and the worker nodes are labelled properly as below using the command “kubectl get nodes

–show-labels”.

For Minimal setup and Resilient HA single DC setup. elasticsearch,broker,zookeeper,object_store,default_agent,grafana,distributed_db,agent1,sec urestore,northbound,thanos,monitoring_server,infra-tsdb

For2 DC/Zones:

Zone 1:

elasticsearch,broker,zookeeper,object_store,default_agent,grafana,distributed_db,agent1,sec urestore,northbound,thanos,monitoring_server,infra-tsdb,topology.kubernetes.io/zone=dc-1 Zone 2: elasticsearch,broker,zookeeper,object_store,default_agent,grafana,distributed_db,agent1,sec urestore,northbound,thanos,monitoring_server,infra-tsdb,topology.kubernetes.io/zone=dc-2

For Resilient-HA setup in 3 DC/Zones:

Zone 1: elasticsearch,broker,zookeeper,object_store,default_agent,grafana,distributed_db,agent1,sec urestore,northbound,thanos,monitoring_server,infra-tsdb,topology.kubernetes.io/zone=dc-1 Zone 2: elasticsearch,broker,zookeeper,object_store,default_agent,grafana,distributed_db,agent1,sec urestore,northbound,thanos,monitoring_server,infra-tsdb,topology.kubernetes.io/zone=dc-2 Zone 3: elasticsearch,broker,zookeeper,object_store,default_agent,grafana,distributed_db,agent1,sec urestore,northbound,thanos,monitoring_server,infra-tsdb,topology.kubernetes.io/zone=dc-3

To label a node use below command:

Note: Make sure you see label dc1, dc2 and dc3 appropriately based on the datacenter where it is present for Resilient HA setup. For scale Worker Nodes also the labelling approach remains the same as above.

To download the Atom Software Deployment zip(in case not done before) provided by Anuta Networks Support team, you can use any of the download methods described in the section New Kubernetes cluster step 9-11.

On the master node of the Kubernetes cluster, if option1 was chosen at step9 of New Kubernetes cluster or if option5 chosen to trigger ATOM deployment separately, then all of the ATOM application components/microservices will get deployed.

A successful ATOM deployment of the components using Helm will have sample output like below:

Folders creating done. PV creating done.

All module check is successful

deploying Linstor charts using piraeus-operator DB pods deployed

Helm chart haproxy got successfully deployed Helm chart keycloak skipped

Helm chart infra-kibana got successfully deployed Helm chart haproxy-gw got successfully deployed Helm chart dashboard got successfully deployed Helm chart oauth2 skipped

Helm chart lb got successfully deployed

Helm chart infra-grafana got successfully deployed

Helm chart infra-distributed-db-webconsole got successfully deployed Helm chart infra-logstash got successfully deployed

Helm chart broker got successfully deployed Helm chart zookeeper got successfully deployed

Helm chart infra-distributed-db-webagent got successfully deployed Helm chart infra-log-forwarder got successfully deployed

Helm chart elasticsearch-config got successfully deployed Helm chart schema-repo got successfully deployed

Helm chart infra-elasticsearch got successfully deployed Helm chart infra-distributed-db got successfully deployed returncode is 0

DB pods deployed

Helm chart infra-tsdb-monitoring got successfully deployed Helm chart minio got successfully deployed

Helm chart thanos got successfully deployed

Helm chart atom-workflow-engine got successfully deployed Helm chart atom-inventory-mgr got successfully deployed Helm chart atom-isim got successfully deployed

Helm chart kafka-operator got successfully deployed Helm chart atom-pnp-server got successfully deployed Helm chart atom-core got successfully deployed

Helm chart atom-qs got successfully deployed

Helm chart atom-agent-proxy got successfully deployed Helm chart atom-scheduler got successfully deployed Helm chart atom-sysmgr got successfully deployed Helm chart atom-agent got successfully deployed

Helm chart atom-telemetry-engine got successfully deployed Helm chart atom-ml got successfully deployed

Helm chart atom-frontend got successfully deployed Helm chart infra-glowroot got successfully deployed Helm chart burrow got successfully deployed

Helm chart jaeger-tracing got successfully deployed Helm chart kafka-control got successfully deployed Helm chart kafka-manager got successfully deployed Helm chart infra-web-proxy got successfully deployed Helm chart infra-tsdb got successfully deployed Helm chart modsecurity got successfully deployed Supplied atom as namespace

SSO URLS for application endpoints are: ATOM UI ==> https://172.16.18.20

KIBANA UI ==> https://172.16.18.20/kibana/ GRAFANA UI ==> https://172.16.18.20/grafana/ GLOWROOT UI ==> https://172.16.18.20/glowroot/

K8S UI ==> https://172.16.18.20/k8s/

KAFKA MANAGER UI ==> https://172.16.18.20/kafka-manager/ KEYCLOAK_URL ==> https://172.16.18.20/auth

TSDB_URL ==> https://172.16.18.20/prometheus THANOS_URL ==> https://172.16.18.20/thanos

TSDB_MONITORING_URL ==> https://172.16.18.20/prometheus-atom REMOTE AGENT URL ==> :7000

ZTP URL ==>

(‘atom_fqdn = ‘, ”)

sh /home/atom/atom-deployment/scripts/get_urls.sh atom 172.16.18.18

https://172.16.18.18:

Keycloak is active

Fetching token from admin-cli eyJhbGciOiJSUzI1NiIsInR5cCIgOiAiSldUIiwia2lkIiA6ICJPUmg4MWFSNUpOS21nelB6 aFNmSEVlYnQ5ekZRVVYyaVFwWG5hYmhLNERRIn0.eyJleHAiOjE2NDU1MDQ5NjYsImlhdCI6 MTY0NTUwNDkwNiwianRpIjoiZjQ2ZTBlZGYtM2FkMi00ODJmLTljYzktY2I2YzcyNzA4YTM1 IiwiaXNzIjoiaHR0cHM6Ly8xNzIuMTYuMTguMTgvYXV0aC9yZWFsbXMvbWFzdGVyIiwic3Vi IjoiNmUzYWEyZmItOGI1NS00ZjlmLTliMWQtZjFmZTBmZTdkZjBiIiwidHlwIjoiQmVhcmVy IiwiYXpwIjoiYWRtaW4tY2xpIiwic2Vzc2lvbl9zdGF0ZSI6IjkxMzliOTQzLWI5NjAtNGJl NS05MWYzLTI3ZGE1NmIzMzgyNiIsImFjciI6IjEiLCJzY29wZSI6ImVtYWlsIHByb2ZpbGUi LCJlbWFpbF92ZXJpZmllZCI6ZmFsc2UsInByZWZlcnJlZF91c2VybmFtZSI6ImFkbWluIn0. Shom2O7DkYkS9aI0MsdUitY7mSlDHUtSgsMiZiWwPNHvionKLFNVeE4ynhP8sl3k3KLZQ5UJ MbhORvKNorxLQqCLIZNZONhtFnxEY9OQLXQKkE29xORCPkpj1ooDISEU2Wj5quLkEpSh8BsP pN9bCcNeJKqabwbIBCdo8wGNFa8WrL5M34jNIMmKR-h2e6UrZMX9LpOpKY8B5z6w7kRQ3LwK f700etth24WMw4qlYkdYlk57OFoPcWa8PvcSA0_j52iva1Bv4vVE4EPfeR46bbhSillngBTS WA5ycuhyZPcwHJOpNE3GzkgCKeyygz9us7_BwYFLQ2cwS2Q13Qn-lQ

Endpoints reachability check in progress… Endpoints reachability check in progress… Endpoints reachability check in progress… Endpoints reachability check in progress… Endpoints reachability check in progress… Endpoints reachability check in progress… Endpoints reachability check in progress… Endpoints reachability check in progress… Endpoints reachability check in progress… Endpoints reachability check in progress… EndpointStatus

THANOS_URLREACHABLE

GLOWROOT UIREACHABLE

KEYCLOAK_URLREACHABLE

TSDB_MONITORING_URLREACHABLE

K8S UIREACHABLE

GRAFANA UIREACHABLE

KIBANA UIREACHABLE

ATOM UIREACHABLE

KAFKA MANAGER UINOT REACHABLE

TSDB_URLREACHABLE

After completing atom-deployment.Again run node_setup.py script and select 6th option and give master ip for checking basic functionality ,next select 7th option if you want to change passwords.
A summary of access URLs for various components deployed will be displayed after installation and if required can be obtained anytime by executing following command in scripts folder

The output will be similar to below

Docker registry for Offline deployment

ATOM can be deployed offline using the locally hosted docker registry. Docker images have to be pulled from a locally available registry to the respective nodes for atom deployment.

Verify that you have imported the shared Anuta docker-registry OVA template into your VMware vCenter.

The specs for docker registry VM will be 4CPU/32GB RAM/300GB SSD/1 NIC
Log into the VM using default creds atom/secret@123.
For bootstrapping the node with basic Interface, DNS and NTP configs run the node_setup.py which is present in the home directory using sudo privileges as described in the section New Kubernetes cluster
After completion of the bootstrap process we are now ready to begin the Docker registry installation process. Run node_setup.py script and select Docker registry installation by entering 2 when prompted for choice.
For a fresh install we can select “Complete Docker Registry Installation for offline Deployment” option by entering 1. If required we can perform each of the other steps in the exact order individually.In case of failure, the user can retry by giving appropriate options where the process had failed.

Provide the IP option using “1” or use hostname if they can be resolved. Give the project name which would serve the purpose of repo name.It needs to be provided at a later stage so do make note of it.

Default login for registry will be admin/admin (http:<registry-ip>)

Output of the above process may take time and would look as follows:

Redirecting to /bin/systemctl restart docker.service harbor/harbor.v2.2.1.tar.gz

harbor/prepare harbor/LICENSE harbor/install.sh harbor/common.sh harbor/harbor.yml.tmpl

prepare base dir is set to /home/atom/harbor

Unable to find image ‘goharbor/prepare:v2.2.1’ locally

docker: Error response from daemon: Get https://registry-1.docker.io/v2/: dial tcp: lookup registry-1.docker.io on 8.8.8.8:53: read udp

172.16.26.105:53734->8.8.8.8:53: i/o timeout.

See ‘docker run —help’.

[Step 0]: checking if docker is installed …

Note: docker version: 20.10.5

[Step 1]: checking docker-compose is installed …

Note: docker-compose version: 1.25.5 [Step 2]: loading Harbor images …

23e1126e5547: Loading layer [==================================================>]

34.51MB/34.51MB

0a791fa5d10a: Loading layer [==================================================>]

6.241MB/6.241MB

478208477097: Loading layer [==================================================>]

4.096kB/4.096kB

a31ccda4a655: Loading layer [==================================================>] 3.072kB/3.072kB

70f59ceb330c: Loading layer [==================================================>]

28.3MB/28.3MB

ef395db1a0f0: Loading layer [==================================================>] 11.38MB/11.38MB

fb2e075190ca: Loading layer [==================================================>] 40.5MB/40.5MB

Loaded image: goharbor/trivy-adapter-photon:v2.2.1

c3a4c23b7b9c: Loading layer [==================================================>] 8.075MB/8.075MB

00f54a3b0f73: Loading layer [==================================================>]

3.584kB/3.584kB

afc25040e33f: Loading layer [==================================================>] 2.56kB/2.56kB

edb7c59d9116: Loading layer [==================================================>] 61.03MB/61.03MB

e5405375a1be: Loading layer [==================================================>] 61.85MB/61.85MB

Loaded image: goharbor/harbor-jobservice:v2.2.1

ab7d4d8af822: Loading layer [==================================================>] 4.937MB/4.937MB

8eb4015cb760: Loading layer [==================================================>]

4.096kB/4.096kB

4be492c354d6: Loading layer [==================================================>]

3.072kB/3.072kB

ea3e1353d3dd: Loading layer [==================================================>] 18.99MB/18.99MB

20f1e7953be4: Loading layer [==================================================>]

19.81MB/19.81MB

Loaded image: goharbor/registry-photon:v2.2.1

e359335d9d06: Loading layer [==================================================>] 4.931MB/4.931MB

573c32deac46: Loading layer [==================================================>]

5.926MB/5.926MB

4462384e04f0: Loading layer [==================================================>]

14.86MB/14.86MB

93886c98b389: Loading layer [==================================================>]

27.36MB/27.36MB

481cc53e87f1: Loading layer [==================================================>]

22.02kB/22.02kB

34ddb9fc83e7: Loading layer [==================================================>]

14.86MB/14.86MB

Loaded image: goharbor/notary-server-photon:v2.2.1

f948e4c0caca: Loading layer [==================================================>] 6.783MB/6.783MB

ec1372991658: Loading layer [==================================================>] 9.097MB/9.097MB

1ef3f81e1f85: Loading layer [==================================================>]

1.691MB/1.691MB

Loaded image: goharbor/harbor-portal:v2.2.1

46d871958df2: Loading layer [==================================================>]

8.076MB/8.076MB

03e260326ab5: Loading layer [==================================================>]

3.584kB/3.584kB

6c53b42399ce: Loading layer [==================================================>]

2.56kB/2.56kB

f0859eadaaf8: Loading layer [==================================================>] 53.27MB/53.27MB

48e28227863e: Loading layer [==================================================>]

5.632kB/5.632kB

af9d6bf9cb83: Loading layer [==================================================>] 90.11kB/90.11kB

67d7d6940a94: Loading layer [==================================================>]

11.78kB/11.78kB

07662a79fbff: Loading layer [==================================================>]

54.2MB/54.2MB

6d0fecda12d9: Loading layer [==================================================>]

2.56kB/2.56kB

Loaded image: goharbor/harbor-core:v2.2.1

324f82f1e2f8: Loading layer [==================================================>]

35.95MB/35.95MB

e13d3998e590: Loading layer [==================================================>] 3.072kB/3.072kB

3735726c1403: Loading layer [==================================================>]

59.9kB/59.9kB

3da48fc3af0e: Loading layer [==================================================>]

61.95kB/61.95kB

Loaded image: goharbor/redis-photon:v2.2.1

6e7fceefe62a: Loading layer [==================================================>]

4.931MB/4.931MB

0148fb852b85: Loading layer [==================================================>]

5.926MB/5.926MB

fcfbd97f83cd: Loading layer [==================================================>] 13.33MB/13.33MB

9d99acddd376: Loading layer [==================================================>]

27.36MB/27.36MB

cb7528f98674: Loading layer [==================================================>] 22.02kB/22.02kB

816b6ef47521: Loading layer [==================================================>]

13.33MB/13.33MB

Loaded image: goharbor/notary-signer-photon:v2.2.1

ece94fe3fa7d: Loading layer [==================================================>] 4.936MB/4.936MB

361117114ba4: Loading layer [==================================================>]

62.71MB/62.71MB

8bcb062f0929: Loading layer [==================================================>]

3.072kB/3.072kB

4486548b56a1: Loading layer [==================================================>]

4.096kB/4.096kB

b3660e86e8c2: Loading layer [==================================================>] 63.53MB/63.53MB

Loaded image: goharbor/chartmuseum-photon:v2.2.1

ad64336d0e51: Loading layer [==================================================>] 77.49MB/77.49MB

3f760c535efc: Loading layer [==================================================>]

54.66MB/54.66MB

ce6390c67a6a: Loading layer [==================================================>] 2.56kB/2.56kB

e56ca8f2c586: Loading layer [==================================================>] 1.536kB/1.536kB

56b738911601: Loading layer [==================================================>]

18.43kB/18.43kB

14c3e8748a68: Loading layer [==================================================>]

4.067MB/4.067MB

5172b1fbd671: Loading layer [==================================================>]

278.5kB/278.5kB

Loaded image: goharbor/prepare:v2.2.1

5d79e0b031e3: Loading layer [==================================================>]

76.08MB/76.08MB

ae7c7f0e9c04: Loading layer [==================================================>] 3.584kB/3.584kB

85ec797b97cb: Loading layer [==================================================>]

3.072kB/3.072kB

0b1fe21c8422: Loading layer [==================================================>]

2.56kB/2.56kB

9dac10dcafad: Loading layer [==================================================>]

3.072kB/3.072kB

672cf3cb855c: Loading layer [==================================================>]

3.584kB/3.584kB

1fbe5ad20ece: Loading layer [==================================================>]

12.29kB/12.29kB

Loaded image: goharbor/harbor-log:v2.2.1

d4b2501bd60f: Loading layer [==================================================>] 8.076MB/8.076MB

b35ba3bc6760: Loading layer [==================================================>] 17.61MB/17.61MB

aad9bed872f0: Loading layer [==================================================>] 4.608kB/4.608kB

5233fbaf23ed: Loading layer [==================================================>]

18.43MB/18.43MB

Loaded image: goharbor/harbor-exporter:v2.2.1

e92fd18cc2cd: Loading layer [==================================================>] 6.783MB/6.783MB

Loaded image: goharbor/nginx-photon:v2.2.1

43cfa27fb7b9: Loading layer [==================================================>]

63.78MB/63.78MB

381a3761198d: Loading layer [==================================================>]

80.93MB/80.93MB

575a4ef00206: Loading layer [==================================================>]

6.144kB/6.144kB

d283c20b6814: Loading layer [==================================================>] 2.56kB/2.56kB

5ee933fe737a: Loading layer [==================================================>]

2.56kB/2.56kB

f666b92ffe52: Loading layer [==================================================>] 2.56kB/2.56kB

348980754dc5: Loading layer [==================================================>]

2.56kB/2.56kB

ad39d2f7b9b8: Loading layer [==================================================>] 11.26kB/11.26kB

Loaded image: goharbor/harbor-db:v2.2.1

dbebf4744f06: Loading layer [==================================================>] 4.937MB/4.937MB

b8e081520905: Loading layer [==================================================>] 4.096kB/4.096kB

442f06402474: Loading layer [==================================================>]

18.99MB/18.99MB

da1eb793d5c9: Loading layer [==================================================>] 3.072kB/3.072kB

2906b858cfe3: Loading layer [==================================================>]

25.32MB/25.32MB

795547d15c57: Loading layer [==================================================>]

45.14MB/45.14MB

Loaded image: goharbor/harbor-registryctl:v2.2.1

[Step 3]: preparing environment … [Step 4]: preparing harbor configs …

prepare base dir is set to /home/atom/harbor

WARNING:root:WARNING: HTTP protocol is insecure. Harbor will deprecate http protocol in the future. Please make sure to upgrade to https

Generated configuration file: /config/portal/nginx.conf Generated configuration file: /config/log/logrotate.conf Generated configuration file: /config/log/rsyslog_docker.conf Generated configuration file: /config/nginx/nginx.conf Generated configuration file: /config/core/env

Generated configuration file: /config/core/app.conf Generated configuration file: /config/registry/config.yml Generated configuration file: /config/registryctl/env Generated configuration file: /config/registryctl/config.yml Generated configuration file: /config/db/env

Generated configuration file: /config/jobservice/env Generated configuration file: /config/jobservice/config.yml

Generated and saved secret to file: /data/secret/keys/secretkey Successfully called func: create_root_cert

Generated configuration file: /compose_location/docker-compose.yml Clean up the input dir

[Step 5]: starting Harbor …

Creating network “harbor_harbor” with the default driver Creating harbor-log … done

Creating harbor-portal … done Creating redis… done Creating harbor-db… done Creating registryctl… done Creating registry… done Creating harbor-core… done Creating harbor-jobservice … done Creating nginx… done

✔ —-Harbor has been installed and started successfully.—–

WARNING! Using –password via the CLI is insecure. Use –password-stdin. WARNING! Your password will be stored unencrypted in /root/.docker/config.json. Configure a credential helper to remove this warning. See https://docs.docker.com/engine/reference/commandline/login/#credentials-store

Next we need to download the Atom deployment.zip and images.zip provided by Anuta Networks team on the docker registry. We can use any of the methods as shown below:

- Wget: utility for non-interactive download of files from the Web. User needs to enter the link as input and the files would be downloaded and extracted automatically.

- SCP: securely transferring computer files between a local host and a remote host based on the Secure Shell (SSH) protocol.
- Manual: User can use any standard file transfer protocol to transfer the files on the home directory of the atom user.

Note : Wget option may not work for offline deployment since we do not have public connectivity .

To download the images.zip provided by Anuta Networks team we can follow the similar procedure as stated above.Please note that the images.zip needs to be copied into the images folder in the home directory.]

[Troubleshooting: If the images.zip file is not found it is most likely that the folder must have been cleaned at the start of the node_setup script.In this case copy the images.zip again when prompted to do so.]

Give registry IP to be used and project name/repo name as provided earlier when setting up the registry.

You will observe the following script will execute and output as follows:

“sudo python docker-registry.py -r <IP/hostname of host> -p <Repository name given during installation> -t push -v <ATOM BUILD VERSION> ”.

Enter the Registry IP:172.16.26.5 Enter the Repo/Project name:release

INFO: Debug logs are sent to atom-registry.log

INFO: [‘docker-registry.py’, ‘-r’, ‘172.16.26.5’, ‘-p’, ‘release’, ‘-t’, ‘push’, ‘-v’, ‘11.3.0.0.48817’]

INFO: Push task is selected

INFO: tar -xf images/databases.tgz

INFO: docker load -qi images/databases/infra-broker:7.0.1.12242021.tar INFO: docker load -qi images/databases/kafka-operator:0.4.tar

INFO: docker load –qi

images/databases/infra-elasticsearch:7.10.2_with_alerting_reporting_plugi n.tar

INFO: docker load -qi images/databases/infra-filebeat:770.tar INFO: docker load -qi

images/databases/infra-distributed-db-webconsole:0.2.tar INFO: docker load -qi

images/databases/infra-kafka-prometheus-jmx-exporter:0.11.0.tar INFO: docker load -qi

images/databases/infra-distributed-db-webagent:0.4.tar

INFO: docker load -qi images/databases/postgres-operator:v1.6.3.tar INFO: docker load -qi images/databases/spilo-13:2.0-p7.tar

INFO: docker load -qi images/databases/infra-logstash:772_150321_v2.tar INFO: docker load -qi images/databases/infra-zookeeper:7.0.1.12242021.tar INFO: docker load –qi images/databases/elasticsearch-config:0.5.tar INFO: docker load -qi images/databases/infra-distributed-db:8.8.7.tar INFO: docker tag quay.io/release/infra-broker:7.0.1.12242021 172.16.26.5/release/infra-broker:7.0.1.12242021

INFO: docker tag quay.io/release/kafka-operator:0.4 172.16.26.5/release/kafka-operator:0.4

INFO: docker tag

quay.io/release/infra-elasticsearch:7.10.2_with_alerting_reporting_plugin 172.16.26.5/release/infra-elasticsearch:7.10.2_with_alerting_reporting_pl ugin

INFO: docker tag quay.io/release/infra-filebeat:770 172.16.26.5/release/infra-filebeat:770

INFO: docker tag quay.io/release/infra-distributed-db-webconsole:0.2 172.16.26.5/release/infra-distributed-db-webconsole:0.2

INFO: docker tag

quay.io/release/infra-kafka-prometheus-jmx-exporter:0.11.0 172.16.26.5/release/infra-kafka-prometheus-jmx-exporter:0.11.0 INFO: docker tag quay.io/release/infra-distributed-db-webagent:0.4 172.16.26.5/release/infra-distributed-db-webagent:0.4

INFO: docker tag registry.opensource.zalan.do/acid/postgres-operator:v1.6.3 172.16.26.5/release/postgres-operator:v1.6.3

INFO: docker tag registry.opensource.zalan.do/acid/spilo-13:2.0-p7 172.16.26.5/release/spilo-13:2.0-p7

INFO: docker tag quay.io/release/infra-logstash:772_150321_v2 172.16.26.5/release/infra-logstash:772_150321_v2

INFO: docker tag quay.io/release/infra-zookeeper:7.0.1.12242021 172.16.26.5/release/infra-zookeeper:7.0.1.12242021

INFO: docker tag quay.io/release/elasticsearch-config:0.5 172.16.26.5/release/elasticsearch-config:0.5

INFO: docker tag quay.io/release/infra-distributed-db:8.8.7 172.16.26.5/release/infra-distributed-db:8.8.7

INFO: docker push 172.16.26.5/release/infra-broker:7.0.1.12242021 INFO: docker push 172.16.26.5/release/kafka-operator:0.4

INFO: docker push

172.16.26.5/release/infra-elasticsearch:7.10.2_with_alerting_reporting_pl ugin

INFO: docker push 172.16.26.5/release/infra-filebeat:770

INFO: docker push 172.16.26.5/release/infra-distributed-db-webconsole:0.2 INFO: docker push

172.16.26.5/release/infra-kafka-prometheus-jmx-exporter:0.11.0

INFO: docker push 172.16.26.5/release/infra-distributed-db-webagent:0.4 INFO: docker push 172.16.26.5/release/postgres-operator:v1.6.3

INFO: docker push 172.16.26.5/release/spilo-13:2.0-p7

INFO: docker push 172.16.26.5/release/infra-logstash:772_150321_v2 INFO: docker push 172.16.26.5/release/infra-zookeeper:7.0.1.12242021 INFO: docker push 172.16.26.5/release/elasticsearch-config:0.5 INFO: docker push 172.16.26.5/release/infra-distributed-db:8.8.7 INFO: docker image prune -af

INFO: tar -xf images/linstor.tgz

INFO: docker load -qi images/linstor/csi-provisioner:v3.0.0.tar INFO: docker load -qi images/linstor/piraeus-ha-controller:v0.2.0.tar INFO: docker load -qi images/linstor/centos:8.tar

INFO: docker load -qi images/linstor/csi-attacher:v3.3.0.tar INFO: docker load -qi images/linstor/csi-snapshotter:v4.2.1.tar

INFO: docker load -qi images/linstor/kube-scheduler-amd64:v1.21.9.tar INFO: docker load -qi images/linstor/livenessprobe:v2.5.0.tar

INFO: docker load -qi images/linstor/drbd9-centos7:v9.1.4.tar INFO: docker load -qi images/linstor/piraeus-server:v1.17.0.tar INFO: docker load -qi images/linstor/drbd-reactor:v0.4.4.tar INFO: docker load -qi images/linstor/etcd:v3.4.15.tar

INFO: docker load -qi images/linstor/piraeus-csi:v0.17.0.tar INFO: docker load -qi images/linstor/piraeus-operator:v1.7.0.tar

INFO: docker load -qi images/linstor/csi-node-driver-registrar:v2.4.0.tar INFO: docker load -qi images/linstor/stork:2.6.5.tar

INFO: docker load -qi images/linstor/csi-resizer:v1.3.0.tar INFO: docker tag k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0 172.16.26.5/release/csi-provisioner:v3.0.0

INFO: docker tag quay.io/piraeusdatastore/piraeus-ha-controller:v0.2.0 172.16.26.5/release/piraeus-ha-controller:v0.2.0

INFO: docker tag quay.io/centos/centos:8 172.16.26.5/release/centos:8 INFO: docker tag k8s.gcr.io/sig-storage/csi-attacher:v3.3.0 172.16.26.5/release/csi-attacher:v3.3.0

INFO: docker tag k8s.gcr.io/sig-storage/csi-snapshotter:v4.2.1 172.16.26.5/release/csi-snapshotter:v4.2.1

INFO: docker tag k8s.gcr.io/kube-scheduler-amd64:v1.21.9 172.16.26.5/release/kube-scheduler-amd64:v1.21.9

INFO: docker tag k8s.gcr.io/sig-storage/livenessprobe:v2.5.0 172.16.26.5/release/livenessprobe:v2.5.0

INFO: docker tag quay.io/piraeusdatastore/drbd9-centos7:v9.1.4 172.16.26.5/release/drbd9-centos7:v9.1.4

INFO: docker tag quay.io/piraeusdatastore/piraeus-server:v1.17.0 172.16.26.5/release/piraeus-server:v1.17.0

INFO: docker tag quay.io/piraeusdatastore/drbd-reactor:v0.4.4 172.16.26.5/release/drbd-reactor:v0.4.4

INFO: docker tag gcr.io/etcd-development/etcd:v3.4.15 172.16.26.5/release/etcd:v3.4.15

INFO: docker tag quay.io/piraeusdatastore/piraeus-csi:v0.17.0 172.16.26.5/release/piraeus-csi:v0.17.0

INFO: docker tag quay.io/piraeusdatastore/piraeus-operator:v1.7.0 172.16.26.5/release/piraeus-operator:v1.7.0

INFO: docker tag k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.4.0 172.16.26.5/release/csi-node-driver-registrar:v2.4.0

INFO: docker tag quay.io/anuta/stork:2.6.5 172.16.26.5/release/stork:2.6.5

INFO: docker tag k8s.gcr.io/sig-storage/csi-resizer:v1.3.0 172.16.26.5/release/csi-resizer:v1.3.0

INFO: docker push 172.16.26.5/release/csi-provisioner:v3.0.0 INFO: docker push 172.16.26.5/release/piraeus-ha-controller:v0.2.0 INFO: docker push 172.16.26.5/release/centos:8

INFO: docker push 172.16.26.5/release/csi-attacher:v3.3.0 INFO: docker push 172.16.26.5/release/csi-snapshotter:v4.2.1

INFO: docker push 172.16.26.5/release/kube-scheduler-amd64:v1.21.9 INFO: docker push 172.16.26.5/release/livenessprobe:v2.5.0

INFO: docker push 172.16.26.5/release/drbd9-centos7:v9.1.4 INFO: docker push 172.16.26.5/release/piraeus-server:v1.17.0 INFO: docker push 172.16.26.5/release/drbd-reactor:v0.4.4 INFO: docker push 172.16.26.5/release/etcd:v3.4.15

INFO: docker push 172.16.26.5/release/piraeus-csi:v0.17.0 INFO: docker push 172.16.26.5/release/piraeus-operator:v1.7.0

INFO: docker push 172.16.26.5/release/csi-node-driver-registrar:v2.4.0 INFO: docker push 172.16.26.5/release/stork:2.6.5

INFO: docker push 172.16.26.5/release/csi-resizer:v1.3.0 INFO: docker image prune -af

INFO: tar -xf images/kubernetes.tgz

INFO: docker load -qi images/kubernetes/etcd:3.4.13-0.tar INFO: docker load -qi images/kubernetes/kube-proxy:v1.21.9.tar INFO: docker load -qi images/kubernetes/coredns:v1.8.0.tar

INFO: docker load -qi images/kubernetes/kube-apiserver:v1.21.9.tar INFO: docker load -qi images/kubernetes/kube-scheduler:v1.21.9.tar INFO: docker load -qi

images/kubernetes/kube-controller-manager:v1.21.9.tar

INFO: docker load -qi images/kubernetes/metrics-server:v0.4.2.tar INFO: docker load -qi images/kubernetes/eventrouter:v0.3.tar INFO: docker load -qi images/kubernetes/pause:3.4.1.tar

INFO: docker tag k8s.gcr.io/etcd:3.4.13-0 172.16.26.5/release/etcd:3.4.13-0

INFO: docker tag k8s.gcr.io/kube-proxy:v1.21.9 172.16.26.5/release/kube-proxy:v1.21.9

INFO: docker tag k8s.gcr.io/coredns/coredns:v1.8.0 172.16.26.5/release/coredns:v1.8.0

INFO: docker tag k8s.gcr.io/kube-apiserver:v1.21.9 172.16.26.5/release/kube-apiserver:v1.21.9

INFO: docker tag k8s.gcr.io/kube-scheduler:v1.21.9 172.16.26.5/release/kube-scheduler:v1.21.9

INFO: docker tag k8s.gcr.io/kube-controller-manager:v1.21.9 172.16.26.5/release/kube-controller-manager:v1.21.9

INFO: docker tag k8s.gcr.io/metrics-server/metrics-server:v0.4.2 172.16.26.5/release/metrics-server:v0.4.2

INFO: docker tag gcr.io/heptio-images/eventrouter:v0.3 172.16.26.5/release/eventrouter:v0.3

INFO: docker tag k8s.gcr.io/pause:3.4.1 172.16.26.5/release/pause:3.4.1 INFO: docker push 172.16.26.5/release/etcd:3.4.13-0

INFO: docker push 172.16.26.5/release/kube-proxy:v1.21.9 INFO: docker push 172.16.26.5/release/coredns:v1.8.0

INFO: docker push 172.16.26.5/release/kube-apiserver:v1.21.9 INFO: docker push 172.16.26.5/release/kube-scheduler:v1.21.9

INFO: docker push 172.16.26.5/release/kube-controller-manager:v1.21.9 INFO: docker push 172.16.26.5/release/metrics-server:v0.4.2

INFO: docker push 172.16.26.5/release/eventrouter:v0.3 INFO: docker push 172.16.26.5/release/pause:3.4.1 INFO: docker image prune -af

INFO: tar -xf images/standalone.tgz INFO: docker load -qi

images/standalone/thanos-receive-controller:latest.tar

INFO: docker load -qi images/standalone/dashboard:v2.0.0-rc5.tar INFO: docker load -qi images/standalone/modsecurity-spoa:v0.6.tar INFO: docker load -qi images/standalone/prometheus:v2.30.3.tar INFO: docker load -qi images/standalone/curator:5.7.6.tar

INFO: docker load -qi images/standalone/postgres_exporter:v0.8.0.tar INFO: docker load -qi

images/standalone/mc:RELEASE.2020-07-17T02-52-20Z.tar

INFO: docker load -qi images/standalone/jaeger-collector:1.21.0.tar INFO: docker load -qi images/standalone/haproxy-ingress:v0.11.tar INFO: docker load -qi images/standalone/thanos:v0.23.1.tar

INFO: docker load -qi images/standalone/rsyslog:latest.tar INFO: docker load -qi images/standalone/node-exporter:v1.1.2.tar

INFO: docker load -qi images/standalone/spark-dependencies:latest.tar INFO: docker load -qi images/standalone/pushgateway:v0.8.0.tar

INFO: docker load -qi images/standalone/busybox:1.28.tar

INFO: docker load -qi images/standalone/jaeger-agent:1.21.0.tar INFO: docker load -qi

images/standalone/jaeger-cassandra-schema:1.21.0.tar

INFO: docker load –qi images/standalone/kafka-manager:latest.tar INFO: docker load -qi images/standalone/example-hotrod:1.21.0.tar INFO: docker load -qi images/standalone/metrics-scraper:v1.0.3.tar INFO: docker load -qi images/standalone/jaeger-query:1.21.0.tar INFO: docker load -qi images/standalone/alertmanager:v0.21.0.tar INFO: docker load -qi images/standalone/helm-kubectl-jq:3.1.0.tar INFO: docker load -qi

images/standalone/minio:RELEASE.2020-07-27T18-37-02Z.tar

INFO: docker load -qi images/standalone/kube-state-metrics:v1.9.8.tar INFO: docker load -qi images/standalone/cp-schema-registry:7.0.1.tar INFO: docker load -qi images/standalone/kube-vip:v0.3.8.tar

INFO: docker load -qi images/standalone/burrow-exporter:latest.tar INFO: docker load -qi images/standalone/echoserver:1.3.tar

INFO: docker load -qi images/standalone/configmap-reload:v0.5.0.tar

INFO: docker tag quay.io/observatorium/thanos-receive-controller:latest 172.16.26.5/release/thanos-receive-controller:latest

INFO: docker tag quay.io/anuta/dashboard:v2.0.0-rc5 172.16.26.5/release/dashboard:v2.0.0-rc5

INFO: docker tag quay.io/jcmoraisjr/modsecurity-spoa:v0.6 172.16.26.5/release/modsecurity-spoa:v0.6

INFO: docker tag quay.io/prometheus/prometheus:v2.30.3 172.16.26.5/release/prometheus:v2.30.3

INFO: docker tag quay.io/anuta/curator:5.7.6 172.16.26.5/release/curator:5.7.6

INFO: docker tag quay.io/anuta/postgres_exporter:v0.8.0 172.16.26.5/release/postgres_exporter:v0.8.0

INFO: docker tag quay.io/anuta/mc:RELEASE.2020-07-17T02-52-20Z 172.16.26.5/release/mc:RELEASE.2020-07-17T02-52-20Z

INFO: docker tag quay.io/jaegertracing/jaeger-collector:1.21.0 172.16.26.5/release/jaeger-collector:1.21.0

INFO: docker tag quay.io/jcmoraisjr/haproxy-ingress:v0.11 172.16.26.5/release/haproxy-ingress:v0.11

INFO: docker tag quay.io/thanos/thanos:v0.23.1 172.16.26.5/release/thanos:v0.23.1

INFO: docker tag quay.io/anuta/rsyslog:latest 172.16.26.5/release/rsyslog:latest

INFO: docker tag quay.io/prometheus/node-exporter:v1.1.2 172.16.26.5/release/node-exporter:v1.1.2

INFO: docker tag quay.io/jaegertracing/spark-dependencies:latest 172.16.26.5/release/spark-dependencies:latest

INFO: docker tag quay.io/prometheus/pushgateway:v0.8.0 172.16.26.5/release/pushgateway:v0.8.0

INFO: docker tag quay.io/anuta/busybox:1.28 172.16.26.5/release/busybox:1.28

INFO: docker tag quay.io/jaegertracing/jaeger-agent:1.21.0 172.16.26.5/release/jaeger-agent:1.21.0

INFO: docker tag quay.io/jaegertracing/jaeger-cassandra-schema:1.21.0 172.16.26.5/release/jaeger-cassandra-schema:1.21.0

INFO: docker tag quay.io/anuta/kafka-manager:latest 172.16.26.5/release/kafka-manager:latest

INFO: docker tag quay.io/jaegertracing/example-hotrod:1.21.0 172.16.26.5/release/example-hotrod:1.21.0

INFO: docker tag quay.io/anuta/metrics-scraper:v1.0.3 172.16.26.5/release/metrics-scraper:v1.0.3

INFO: docker tag quay.io/jaegertracing/jaeger-query:1.21.0 172.16.26.5/release/jaeger-query:1.21.0

INFO: docker tag quay.io/prometheus/alertmanager:v0.21.0 172.16.26.5/release/alertmanager:v0.21.0

INFO: docker tag quay.io/anuta/helm-kubectl-jq:3.1.0 172.16.26.5/release/helm-kubectl-jq:3.1.0

INFO: docker tag quay.io/anuta/minio:RELEASE.2020-07-27T18-37-02Z 172.16.26.5/release/minio:RELEASE.2020-07-27T18-37-02Z

INFO: docker tag quay.io/coreos/kube-state-metrics:v1.9.8 172.16.26.5/release/kube-state-metrics:v1.9.8

INFO: docker tag quay.io/anuta/cp-schema-registry:7.0.1 172.16.26.5/release/cp-schema-registry:7.0.1

INFO: docker tag ghcr.io/kube-vip/kube-vip:v0.3.8 172.16.26.5/release/kube-vip:v0.3.8

INFO: docker tag quay.io/anuta/burrow-exporter:latest 172.16.26.5/release/burrow-exporter:latest

INFO: docker tag gcr.io/google_containers/echoserver:1.3 172.16.26.5/release/echoserver:1.3

INFO: docker tag quay.io/anuta/configmap-reload:v0.5.0 172.16.26.5/release/configmap-reload:v0.5.0

INFO: docker push 172.16.26.5/release/thanos-receive-controller:latest INFO: docker push 172.16.26.5/release/dashboard:v2.0.0-rc5

INFO: docker push 172.16.26.5/release/modsecurity-spoa:v0.6 INFO: docker push 172.16.26.5/release/prometheus:v2.30.3 INFO: docker push 172.16.26.5/release/curator:5.7.6

INFO: docker push 172.16.26.5/release/postgres_exporter:v0.8.0

INFO: docker push 172.16.26.5/release/mc:RELEASE.2020-07-17T02-52-20Z INFO: docker push 172.16.26.5/release/jaeger-collector:1.21.0

INFO: docker push 172.16.26.5/release/haproxy-ingress:v0.11 INFO: docker push 172.16.26.5/release/thanos:v0.23.1

INFO: docker push 172.16.26.5/release/rsyslog:latest INFO: docker push 172.16.26.5/release/node-exporter:v1.1.2

INFO: docker push 172.16.26.5/release/spark-dependencies:latest INFO: docker push 172.16.26.5/release/pushgateway:v0.8.0

INFO: docker push 172.16.26.5/release/busybox:1.28

INFO: docker push 172.16.26.5/release/jaeger-agent:1.21.0

INFO: docker push 172.16.26.5/release/jaeger-cassandra-schema:1.21.0 INFO: docker push 172.16.26.5/release/kafka-manager:latest

INFO: docker push 172.16.26.5/release/example-hotrod:1.21.0 INFO: docker push 172.16.26.5/release/metrics-scraper:v1.0.3 INFO: docker push 172.16.26.5/release/jaeger-query:1.21.0 INFO: docker push 172.16.26.5/release/alertmanager:v0.21.0 INFO: docker push 172.16.26.5/release/helm-kubectl-jq:3.1.0

INFO: docker push 172.16.26.5/release/minio:RELEASE.2020-07-27T18-37-02Z INFO: docker push 172.16.26.5/release/kube-state-metrics:v1.9.8

INFO: docker push 172.16.26.5/release/cp-schema-registry:7.0.1 INFO: docker push 172.16.26.5/release/kube-vip:v0.3.8

INFO: docker push 172.16.26.5/release/burrow-exporter:latest INFO: docker push 172.16.26.5/release/echoserver:1.3

INFO: docker push 172.16.26.5/release/configmap-reload:v0.5.0 INFO: docker image prune -af

INFO: tar -xf images/atom.tgz

INFO: docker load -qi images/atom/atom-python3:11.3.0.0.48817.tar INFO: docker load -qi images/atom/atom-agent-proxy:11.3.0.0.48817.tar INFO: docker load -qi images/atom/atom-pnp-server:11.3.0.0.48817.tar INFO: docker load -qi

images/atom/atom-telemetry-engine:11.3.0.0.48817.tar

INFO: docker load -qi images/atom/atom-ml:11.3.0.0.48817.tar INFO: docker load -qi images/atom/atom-python2:11.3.0.0.48817.tar

INFO: docker load -qi images/atom/atom-workflow-engine:11.3.0.0.48817.tar INFO: docker load -qi images/atom/atom-isim:11.3.0.0.48817.tar

INFO: docker load -qi images/atom/atom-inventory-mgr:11.3.0.0.48817.tar INFO: docker load -qi images/atom/atom-core:11.3.0.0.48817.tar

INFO: docker load -qi images/atom/atom-agent:11.3.0.0.48817.tar INFO: docker load -qi

images/atom/atom-telemetry-exporter:11.3.0.0.48817.tar

INFO: docker load -qi images/atom/atom-scheduler:11.3.0.0.48817.tar INFO: docker tag quay.io/release/atom-python3:11.3.0.0.48817 172.16.26.5/release/atom-python3:11.3.0.0.48817

INFO: docker tag quay.io/release/atom-agent-proxy:11.3.0.0.48817 172.16.26.5/release/atom-agent-proxy:11.3.0.0.48817

INFO: docker tag quay.io/release/atom-pnp-server:11.3.0.0.48817 172.16.26.5/release/atom-pnp-server:11.3.0.0.48817

INFO: docker tag quay.io/release/atom-telemetry-engine:11.3.0.0.48817 172.16.26.5/release/atom-telemetry-engine:11.3.0.0.48817

INFO: docker tag quay.io/release/atom-ml:11.3.0.0.48817 172.16.26.5/release/atom-ml:11.3.0.0.48817

INFO: docker tag quay.io/release/atom-python2:11.3.0.0.48817 172.16.26.5/release/atom-python2:11.3.0.0.48817

INFO: docker tag quay.io/release/atom-workflow-engine:11.3.0.0.48817 172.16.26.5/release/atom-workflow-engine:11.3.0.0.48817

INFO: docker tag quay.io/release/atom-isim:11.3.0.0.48817 172.16.26.5/release/atom-isim:11.3.0.0.48817

INFO: docker tag quay.io/release/atom-inventory-mgr:11.3.0.0.48817 172.16.26.5/release/atom-inventory-mgr:11.3.0.0.48817

INFO: docker tag quay.io/release/atom-core:11.3.0.0.48817 172.16.26.5/release/atom-core:11.3.0.0.48817

INFO: docker tag quay.io/atom-agent/atom-agent:11.3.0.0.48817 172.16.26.5/release/atom-agent:11.3.0.0.48817

INFO: docker tag quay.io/release/atom-telemetry-exporter:11.3.0.0.48817 172.16.26.5/release/atom-telemetry-exporter:11.3.0.0.48817

INFO: docker tag quay.io/release/atom-scheduler:11.3.0.0.48817 172.16.26.5/release/atom-scheduler:11.3.0.0.48817

INFO: docker push 172.16.26.5/release/atom-python3:11.3.0.0.48817 INFO: docker push 172.16.26.5/release/atom-agent-proxy:11.3.0.0.48817 INFO: docker push 172.16.26.5/release/atom-pnp-server:11.3.0.0.48817 INFO: docker push

172.16.26.5/release/atom-telemetry-engine:11.3.0.0.48817 INFO: docker push 172.16.26.5/release/atom-ml:11.3.0.0.48817

INFO: docker push 172.16.26.5/release/atom-python2:11.3.0.0.48817

INFO: docker push 172.16.26.5/release/atom-workflow-engine:11.3.0.0.48817 INFO: docker push 172.16.26.5/release/atom-isim:11.3.0.0.48817

INFO: docker push 172.16.26.5/release/atom-inventory-mgr:11.3.0.0.48817 INFO: docker push 172.16.26.5/release/atom-core:11.3.0.0.48817

INFO: docker push 172.16.26.5/release/atom-agent:11.3.0.0.48817 INFO: docker push

172.16.26.5/release/atom-telemetry-exporter:11.3.0.0.48817

INFO: docker push 172.16.26.5/release/atom-scheduler:11.3.0.0.48817 INFO: docker image prune -af

INFO: tar -xf images/calico.tgz

INFO: docker load -qi images/calico/pod2daemon-flexvol:v3.15.5.tar INFO: docker load -qi images/calico/cni:v3.15.5.tar

INFO: docker load -qi images/calico/node:v3.15.5.tar

INFO: docker load -qi images/calico/kube-controllers:v3.15.5.tar INFO: docker tag quay.io/calico/pod2daemon-flexvol:v3.15.5 172.16.26.5/release/pod2daemon-flexvol:v3.15.5

INFO: docker tag quay.io/calico/cni:v3.15.5 172.16.26.5/release/cni:v3.15.5

INFO: docker tag quay.io/calico/node:v3.15.5 172.16.26.5/release/node:v3.15.5

INFO: docker tag quay.io/calico/kube-controllers:v3.15.5 172.16.26.5/release/kube-controllers:v3.15.5

INFO: docker push 172.16.26.5/release/pod2daemon-flexvol:v3.15.5 INFO: docker push 172.16.26.5/release/cni:v3.15.5

INFO: docker push 172.16.26.5/release/node:v3.15.5

INFO: docker push 172.16.26.5/release/kube-controllers:v3.15.5 INFO: docker image prune -af

INFO: tar -xf images/calico.tgz

INFO: docker load -qi images/calico/pod2daemon-flexvol:v3.15.5.tar INFO: docker load -qi images/calico/cni:v3.15.5.tar

INFO: docker load -qi images/calico/node:v3.15.5.tar

INFO: docker load -qi images/calico/kube-controllers:v3.15.5.tar INFO: docker tag quay.io/calico/pod2daemon-flexvol:v3.15.5 172.16.26.5/release/pod2daemon-flexvol:v3.15.5

INFO: docker tag quay.io/calico/cni:v3.15.5 172.16.26.5/release/cni:v3.15.5

INFO: docker tag quay.io/calico/node:v3.15.5 172.16.26.5/release/node:v3.15.5

INFO: docker tag quay.io/calico/kube-controllers:v3.15.5 172.16.26.5/release/kube-controllers:v3.15.5

INFO: docker push 172.16.26.5/release/pod2daemon-flexvol:v3.15.5 INFO: docker push 172.16.26.5/release/cni:v3.15.5

INFO: docker push 172.16.26.5/release/node:v3.15.5

INFO: docker push 172.16.26.5/release/kube-controllers:v3.15.5 INFO: docker image prune -af

INFO: tar -xf images/infra.tgz

INFO: docker load -qi images/infra/infra-grafana:7.5.7.tar INFO: docker load -qi images/infra/infra-glowroot:0.9.tar

INFO: docker load -qi images/infra/infra-kafka-control:0.2.0.tar INFO: docker load -qi images/infra/infra-burrow:1.3.8.tar

INFO: docker load -qi images/infra/keycloak:13.1.0.tar INFO: docker load -qi images/infra/controller:v0.10.2.tar INFO: docker load -qi images/infra/speaker:v0.10.2.tar

INFO: docker load -qi images/infra/oauth2-proxy:v7.0.1-verifyjwt.tar INFO: docker load -qi images/infra/infra-kibana:710_opendistro.tar INFO: docker tag quay.io/release/infra-grafana:7.5.7 172.16.26.5/release/infra-grafana:7.5.7

INFO: docker tag quay.io/release/infra-glowroot:0.9 172.16.26.5/release/infra-glowroot:0.9

INFO: docker tag quay.io/release/infra-kafka-control:0.2.0 172.16.26.5/release/infra-kafka-control:0.2.0

INFO: docker tag quay.io/release/infra-burrow:1.3.8 172.16.26.5/release/infra-burrow:1.3.8

INFO: docker tag quay.io/release/keycloak:13.1.0 172.16.26.5/release/keycloak:13.1.0

INFO: docker tag quay.io/metallb/controller:v0.10.2 172.16.26.5/release/controller:v0.10.2

INFO: docker tag quay.io/metallb/speaker:v0.10.2 172.16.26.5/release/speaker:v0.10.2

INFO: docker tag quay.io/release/oauth2-proxy:v7.0.1-verifyjwt 172.16.26.5/release/oauth2-proxy:v7.0.1-verifyjwt

INFO: docker tag quay.io/release/infra-kibana:710_opendistro 172.16.26.5/release/infra-kibana:710_opendistro

INFO: docker push 172.16.26.5/release/infra-grafana:7.5.7 INFO: docker push 172.16.26.5/release/infra-glowroot:0.9

INFO: docker push 172.16.26.5/release/infra-kafka-control:0.2.0 INFO: docker push 172.16.26.5/release/infra-burrow:1.3.8

INFO: docker push 172.16.26.5/release/keycloak:13.1.0 INFO: docker push 172.16.26.5/release/controller:v0.10.2 INFO: docker push 172.16.26.5/release/speaker:v0.10.2

INFO: docker push 172.16.26.5/release/oauth2-proxy:v7.0.1-verifyjwt INFO: docker push 172.16.26.5/release/infra-kibana:710_opendistro INFO: docker image prune -af

Once the above script is executed, the docker registry has been installed and setup correctly. We can begin with k8s installation and Atom Installation.

Please follow the steps as stated in section New Kubernetes cluster to bootstrap master and worker nodes and setup the k8s cluster and install Atom.

Note: As it is an offline Installation we do not require internet connection on any of the nodes as long as the registry has been setup properly and NTP server is present to sync the time between all the nodes.

Please note after updating Node IP: Select yes option for offline installation and provide the registry ip and project/repo name when prompted.

Atom Installation would be complete and we can proceed by onboarding packages and devices on the platform.

ATOM Remote Agent Deployment

In the ATOM Distributed deployment model, Remote Agent is used to communicate, collect and monitor the networking devices in your infrastructure using standard protocols. Once the agent collects the data, it gets encrypted and sent to Anuta ATOM Server over an outgoing SSL Connection.

The ATOM Agent is an application that runs on a Linux server within your infrastructure as a docker container. ATOM agents have to be installed on each location of your device’s infrastructure.

For deployment of ATOM agent across various geographies perform the steps mentioned in the ATOM Remote Agent Deployment Guide [version 10.0]

Procedure of Deploying ATOM in GCP/GKE

ATOM can be deployed on Google Cloud Platform (GCP) Google Kubernetes Engine (GKE)

using the “Deployment scripts and files” provided by Anuta.

Prerequisites

An Ubuntu/CentOS machine that has access to the internet, so that the deployment scripts can be run. Below are some of the softwares to be installed on that machine.

Helm v3.5.4
- Installation procedure: https://helm.sh/docs/intro/install/
Gcloud SDK
- Installation procedure: https://cloud.google.com/sdk/docs/install
- Setup the gcloud SDK using “gcloud auth login” and “gcloud auth application-default login” if they are not set.
- Verification can be done using “gcloud container clusters list”
Kubectl installed v1.21
- Installation procedure:

https://kubernetes.io/docs/tasks/tools/install-kubectl/#install-using-nativ e-package-management

Docker installed v20.10 and above
- Installation procedure: https://docs.docker.com/engine/install/

Python2.7 pip package
- Install python-pip using “sudo apt-get install python-pip” or “sudo yum install python-pip” depending on the distro being used.
- Install the following packages using the below command
  - sudo pip install pyyaml==3.13
  - sudo pip install requests==2.20.0
  - sudo pip install setuptools==40.5.0
  - sudo pip install cryptography==2.3.1
  - sudo pip install pyJWT==1.6.4
  - sudo pip install cachetools==2.1.0
  - sudo pip install kubernetes==11.0.0

A site-to-site VPN setup between your datacenter and GCP created for ATOM to reach devices.
If the linux machine is created on the GCP, then confirm that the service account mapped has enough privileges to run as sudo. The permissions for the service account to have Kubernetes Engine, Compute Engine and Compute OS privileges to ensure cluster role creation is allowed during ATOM installation.

Deploying New K8s Cluster

After ensuring that the prerequisites are taken care, perform the following steps:

Minimal Setup:

❏ Click on the Create button which would open the option of cluster models

❏ Select the configure button of Standard kind

❏ Provide the name of the cluster and select the location of choice. Location to be set to Zonal and choice of the zone can be selected from the list available.

❏ Proceed to control plane version selection and select the Release channel radio button. Select the Regular channel from the list and ensure that the kubernetes version falls into v1.21

❏ At the left pane, select the default node pool, provide the name and number of nodes to 4(as per minimal deployment size)

❏ Proceed to the nodes section, select the node type as “Ubuntu with Docker(ubuntu)”. Set the size to “e2-highmem-4” under E2 series.

❏ Proceed to the Security tab under nodepool, and select the service account created for the same having enough privileges to host the compute instance.

❏ Proceed to the Metadata tab under nodepool, provide the following kubernetes labels

❏ Proceed to the Networking tab, select the network and node subnet as per the lab networking done.

❏ Select the Public or Private cluster depending on the choice. Provide the POD CIDR and Service CIDR if there are any which accordingly adds to the kubernetes pods.

❏ Set the “Enable Kubernetes Network Policy” by selecting the checkbox. Optionally select if other options are required.

❏ Optionally set if there are any options required at Security, Metadata and Features tab as required.

❏ Finally select Create at the bottom to create the kubernetes cluster on GKE.

Resilient-HA Setup:

❏ Click on the Create button which would open the option of cluster models

❏ Select the configure button of Standard kind

❏ Provide the name of the cluster and select the location of choice. Location to be set to Regional and choice of the zones(2 to 3 zones) can be selected from the list available.

❏ Proceed to control plane version selection and select the Release channel radio button. Select the Regular channel from the list and ensure that the kubernetes version falls into v1.21

❏ At the left pane, select the default node pool, provide the name and number of nodes to 4 so the total number of nodes is 8(as per resilient deployment size requirement). If the number of zones selected are 3 then provide the number of nodes to 3 per zone so the total number of nodes is 9.

Note: If node pools need to be separated across zones, create multiple node pool and select the specific node locations of the choice.

❏ Proceed to the nodes section, select the node type as “Ubuntu with Docker(ubuntu)”. Set the size to “e2-highmem-4” under E2 series.

❏ Proceed to the Security tab under nodepool, and select the service account created for the same having enough privileges to host the compute instance.

❏ Proceed to the Metadata tab under nodepool, provide the following kubernetes labels

Note: If multiple node pools are set, above node labels have to be provided for all the node pools.

❏ Proceed to the Networking tab, select the network and node subnet as per the lab networking done.

❏ Select the Public or Private cluster depending on the choice. Provide the POD CIDR and Service CIDR if there are any which accordingly adds to the kubernetes pods.

❏ Set the “Enable Kubernetes Network Policy” by selecting the checkbox. Optionally select if other options are required.

❏ Optionally set if there are any options required at Security, Metadata and Features tab as required.

❏ Finally select Create at the bottom to create the kubernetes cluster on GKE.

Deploying ATOM

After ensuring that the prerequisites described in the section “Prerequisites” are taken care of, perform the following steps:

Note: running sudo is required since deployment scripts run with sudo

Unzip the deployment-scripts folder, provided by Anuta, described in the section, “Deployment scripts and files”. Update wrapper.properties file accordingly.
- Cross verify if build number is set
- Cross verify if deployment_type is set to “gcloud”
- Set the public key to “enable” or “disable” depending on cluster type. When enabled, LoadBalancers created for ATOM would have public access over the internet.
- Update size to required value like “minimal” or “resilient”
- Set the zonal_resiliency to “enable” if the size is resilient to spread workloads across zones. For minimal size, this can be set to disable.
- Cross verify if image_pull is set to “quay”
- Cross verify if organization is set to “release”

Deploy ATOM by executing the following script

Executing above steps will complete the ATOM deployment.

After deployment is completed, the URL’s to access the ATOM application can be fetched by running

For local deployments, the services are accessible via nodePorts and for cloud deployments the services are accessible via LoadBalancers.

Procedure of Deploying ATOM in AWS

ATOM can be deployed on Amazon Web Services (AWS) Elastic Kubernetes Service (EKS)

using the “Deployment scripts and files” provided by Anuta.

Following diagram depicts ATOM deployment in terms of AWS resources.

Prerequisites

Deployment Machine – An Ubuntu machine running with v18.04 that has access to the internet, so that the deployment scripts can be run.

Spec:

cpu: 2 vcpu mem: 4GB storage: 50GB

Image: Ubuntu – 18.04.06 LTS (Bionic Beaver) OR

aws instance type: t2-medium

Community AMI: ubuntu-bionic-18.04-amd64-server-20211129 – ami-074251216af698218

SecurityGroup/FW Rules:

- Allow inbound ssh access to the ubuntu machine from DC.
- Allow all outbound to the internet.

Below software must be installed on that machine.

Helm v3.5.4

- wget -q https://get.helm.sh/helm-v3.5.4-linux-amd64.tar.gz
- tar -zxvf helm-v3.5.4-linux-amd64.tar.gz
- sudo mv linux-amd64/helm /usr/local/bin/
AWS CLI v2
- Installation procedure: https://docs.aws.amazon.com/cli/latest/userguide/install-cliv2-linux.html #cliv2-linux-install
EKSCTL 0.79.0
- Installation procedure: https://github.com/weaveworks/eksctl#installation
Kubectl installed v1.21 and above
- Installation procedure:

https://v1-21.docs.kubernetes.io/docs/tasks/tools/install-kubectl-linux/

python 2.7
- sudo apt install python
Kubernetes pip package v11.0.0
- Install python-pip using “sudo apt-get install python-pip”
- Upgrade the pip to latest using “sudo pip install pip —upgrade”
- Installed using “sudo pip install kubernetes==11.0.0”
Paramiko package v2.6.0
- Installed using “sudo pip install paramiko” Reboot the system “sudo reboot”

Connectivity between on-premises DC and AWS VPCs. Please refer to section “AWS

connectivity options” in Appendix for more information.

A /24 ip block as vpc cidr – ATOM deployment scripts use eksctl tool to create a kubernetes cluster on AWS. eksct by default creates a dedicated vpc and creates a eks cluster in the vpc. Please choose a vpc cidr to accommodate all the future needs considering the cluster expansion if needed.

Please refer to https://eksctl.io/usage/vpc-networking/ for more information about cidr planning.

ELBs
- All the ELBs are created by kubernetes service manifests.
- All ELBs are of type NLB and default internal. If you have a requirement to deploy “external” ELB, please modify values in wrapper.properties file – refer to section Deploying ATOM.
- ATOM by default uses self signed certificates. If you prefer to use the proper certificates, please refer to the section Custom SSL Certificate for ATOM
User with following IAM privileges are required
- IAM User with following permissions to create EKS cluster https://eksctl.io/usage/minimum-iam-policies/.,VPC full access policy.
- IAM User with s3 Full access policy. This is needed for storing device configuration in s3 using minio

- Create policy to provision volumes with type gp3.
  - From AWS console select Resources → IAM → Access Management → Policies

Click on create policy and select JSON and paste following json config.

- - Attach created policy to nodegroup arn used by cluster nodes.

From AWS console select Resources → IAM → Access Management → Roles. Find the nodegroup-arn (identified by cluster-name) used by nodes and click on that role. And select permissions and click on Attach policies and add the above policy to the role.

File server – An optional component needed only if server ZTP provisioning is used. Please refer to section File Server for ATOM ZTP in AWS.

Deploying New K8s Cluster

After ensuring that the prerequisites are taken care, perform the following steps:

❏ Login to the Ubuntu machine and follow below steps to configure aws cli aws configure

❏ Execute below command which does Kubernetes cluster addition with prefered node-type as r6i.xlarge

eksctl create cluster –name {cluster-name} –version 1.21 –region <region-name>

–node-type r6i.xlarge –without-nodegroup –node-volume-size 50

–node-private-networking –vpc-cidr <vpc-cidr> –ssh-access=true

–ssh-public-key={access-key} –vpc-nat-mode HighlyAvailable

Ex: eksctl create cluster –name aws-deploy-test –version 1.21 –region us-west-2

–node-type r6i.xlarge –without-nodegroup –node-volume-size 50

–node-private-networking –vpc-cidr 192.168.66.0/24 –ssh-access=true

–ssh-public-key=atom –vpc-nat-mode HighlyAvailable

NOTE: If ssh public key is missing then generate using ssh-keygen

If want to deploy on specific zones in a region “–zones=us-west-2a,us-west-2b”

❏ Delete the Amazon VPC CNI

kubectl delete ds aws-node -n kube-system

❏ Install Calico

Unzip the deployment-scripts folder, provided by Anuta, described in the section, “Deployment scripts and files”. From the atom-deployment folder, go into scripts using “cd scripts” and run below command.

kubectl apply -f calico.yaml

❏ Now we have a new VPC created for the EKS cluster. At this point please make sure the connectivity between the deployment machine and eks cluster VPC. Please refer to section “AWS connectivity options” in Appendix for more information.

❏ Create a nodegroup for worker nodes.

Based on type of setup being created, execute respective command

Minimal Setup:

eksctl create nodegroup –name {nodegroup-name} –nodes {total-number-of-nodes}

–nodes-min {minimum-number-of-nodes-in-availability-zone} –nodes-max

{maximum-number-of-nodes-availability-zone} –node-volume-size 50 —cluster

{cluster-name} –node-private-networking –node-type r6i.xlarge –ssh-access

–node-labels={labels} —managed

Example:

Resilient-HA Setup:

eksctl create nodegroup –name {nodegroup-name} –nodes {total-number-of-nodes}

–nodes-min {minimum-number-of-nodes-in-availability-zone} –nodes-max

{maximum-number-of-nodes-availability-zone} –node-volume-size 50 —cluster

{cluster-name} –node-private-networking

–node-labels=zookeeper=deploy,broker=deploy,grafana=deploy,elasticsearch=deploy,obj ect_store=deploy,distributed_db=deploy,default_agent=deploy,securestore=deploy,monit oring_server=deploy,infra-tsdb=deploy,thanos=deploy,agent1=deploy,northbound=deplo y –node-zones {availability-zone-name} –node-type r6i.xlarge –managed

For resilient setup labels for nodes will be assigned during creation of the nodegroup itself.
We recommend a minimum of 9 worker nodes; 3 nodes per nodegroup, 1 nodegroup per AZ, and 3 AZs.

Example:

eksctl create nodegroup –name aws-deploy-test-ng01 –nodes 3 –nodes-min 3

–nodes-max 3 –node-volume-size 50 –cluster aws-deploy-test

–node-private-networking

–node-labels=zookeeper=deploy,broker=deploy,grafana=deploy,elasticsearch=deploy,obje ct_store=deploy,distributed_db=deploy,default_agent=deploy,securestore=deploy,monitor ing_server=deploy,infra-tsdb=deploy,thanos=deploy,agent1=deploy,northbound=deploy

–node-type r6i.xlarge –node-zones us-west-2b —managed

eksctl create nodegroup –name aws-deploy-test-ng02 –nodes 3 –nodes-min 3

–nodes-max 3 –node-volume-size 50 –cluster aws-deploy-test

–node-private-networking

–node-labels=zookeeper=deploy,broker=deploy,grafana=deploy,elasticsearch=deploy,obje ct_store=deploy,distributed_db=deploy,default_agent=deploy,securestore=deploy,monitor ing_server=deploy,infra-tsdb=deploy,thanos=deploy,agent1=deploy,northbound=deploy

–node-type r6i.xlarge –node-zones us-west-2c —managed

eksctl create nodegroup –name aws-deploy-test-ng03 –nodes 3 –nodes-min 3

–nodes-max 3 –node-volume-size 50 –cluster aws-deploy-test

–node-private-networking

–node-labels=zookeeper=deploy,broker=deploy,grafana=deploy,elasticsearch=deploy,obje ct_store=deploy,distributed_db=deploy,default_agent=deploy,securestore=deploy,monitor ing_server=deploy,infra-tsdb=deploy,thanos=deploy,agent1=deploy,northbound=deploy

—node-type r6i.xlarge –node-zones us-west-2d —managed

NOTE: Kindly wait until the required number of nodes are in ready state. You can check the node status by running “kubectl get nodes”

❏ Create the IAM policy for gp3 volumes creation. Please refer to 5(c) in Prerequisites for respective steps.

❏ Update Kubeconfig

aws eks –region <us-west-2> update-kubeconfig –name {cluster-name}

Cross check once by running “kubectl get svc –all-namespaces” and observe if kubernetes services are running.

Deploying ATOM

After ensuring that the prerequisites described in the section “Prerequisites” are taken care of, perform the following steps:

“eksctl get cluster”

“aws eks update-kubeconfig –name {cluster-name} –region {region-name}”

Unzip the deployment-scripts folder, provided by Anuta, described in the section, “Deployment scripts and files”.
Update wrapper.properties file accordingly in [deployment] section.
- Cross verify if build number is set
- Cross verify if deployment_type is set to “aws”
- Update size to required value like “minimal” or “resilient”
- Cross verify if image_pull is set to “quay”
- Cross verify if organization is set to “release”
- Set the cluster_name in the [aws-eks] section.
Update wrapper.properties file to set ELB configuration in [aws-eks] section
- Choose ELB type – if public access to ATOM services is required, then set the following parameters “external”. “internal” is default.
  - elb_atom_ui_sso – ELB for providing ATOM UI over SSO
  - elb_atom_ui_direct – ELB for providing ATOM UI/rest API
  - elb_atom_agent – ELB for remote agents to connect to ATOM cluster
  - elb_atom_agent_apm – ELB for remote agents to send performance metrics to ATOM cluster
  - elb_atom_agent_incluster – ELB for devices to communicate with agent if the agent is running inside ATOM cluster
- set north_bound_source_ranges according to your requirement. By default it will allow all IPs.

eg: allowed_ip_range_atom_ui = “10.10.0.0/24,172.16.0.0/16”

- - allowed_ip_range_atom_ui_sso
  - allowed_ip_range_atom_ui_direct
  - allowed_ip_atom_agent
  - allowed_ip_range_atom_agent_apm
  - allowed_ip_range_atom_agent_incluster
Set the permissions to the log file by running. “sudo touch /var/log/atom.log && sudo chmod 777 /var/log/atom.log”
Modify accesskey,secretkey and service endpoint in ATOM/minio/values.yaml under s3gateway section.

s3gateway:

enabled: s3gateway replicas: 1

serviceEndpoint: “https://s3.{region}.amazonaws.com” accessKey: “XXXXXXXXXX”

secretKey: “YYYYYYYYYYY”

Deploy ATOM by executing the following script

Executing above steps will complete the ATOM deployment in AWS.

After deployment is completed, the URL’s to access the ATOM application can be fetched by running

ATOM services are accessible via LoadBalancers.

A Site-to-Site VPN would be needed between the ATOM server on AWS and a remote ATOM agent which has access to lab devices. Please refer to the section Distributed ATOM Agent Deployment for deploying a Remote ATOM Agent.

ATOM System Manager

ATOM provides deployment summary of all Components through System Manager. System Manager provides a high level view of all the components, number of instances, status & management URLs for some of the components.

Navigate to Administration> System Manager> Dashboard
To access components like Grafana, Kibana etc. select that component and you can see the Management URL at top right corner.

To Check the functionality of ATOM and its involved components, select the top level circle icon
To Check the ATOM Components status like liveness and number of instances per each ATOM components select the ATOM component as shown below.

Here on the right side we can see the number instances, refer below colour codings.

- Green: it represent the Activeness of the component
- Red: It represents the component is deployed and the required number of instances set as zero.
- Black: It represents the component is not deployed

To Check the ATOM Infra Components status like liveness and number of instances per each Infra components select the Infra component as shown below

To view the same data in a tabular form select the toggle button as shown below. Here also we can see the number instances per component, status and its management url if applicable.

Post Installation

AWS Specific

Security Group updates

Following ports are required to be included in the security groups for device monitoring. Select VPC under services and select Security Group under Security Section.

VPC – > Security → Security Group.

select a security group with the name eks-cluster-sg-{cluster-name}-xxxx and add following ports to that security group.

21(TCP) – FTP from device to ATOM.
69(UDP) – TFTP from device to ATOM
162(UDP) – Sending SNMP traps from device to ATOM.
514(UDP) – Syslog port on device.

the source of the above security group will depend on the device range.

Custom DNS name(CNAME) creation

Once ATOM gets deployed, UI can be accessed at the default DNS names given to the load balancers but those DNS names are not easy to remember and use. If you prefer to use a custom DNS name instead of the default DNS name, you can associate a custom DNS name(CNAME) for the load balancers.

Follow the below steps only if your domain is hosted in aws route53, otherwise follow the instructions from your dns provider to add CNAME.

Steps to add CNAME in route53:

Fetch the haproxy-gw load balancer.
- kubectl get svc -n atom | grep -i Load | grep haproxy-service-gw
Once haproxy LB value fetched, Select Route53 service in aws and select HostedZone
In Hosted Zones select one of the domain in which you want to create Record(CNAME type)
Click on Create Record
- In Record name provide the name from which you want to access ATOM UI.
- Select CNAME in Record type.
- In the value field , provide the haproxy-service-gw LB value.
- Click on Create records.

Once CNAME was added, to make it work changes are required in keycloak UI, atom-frontend and oauth2-proxy deployment.

Adding CNAME in keycloak Clients

Once CNAME was added in Route53, keycloak needs to be updated otherwise it would get InvalidURL when trying to access ATOM UI using CNAME.

steps to follow:

Fetch haproxy-service-gw LB value using following command
- kubectl get svc -n atom | grep -i Load | grep haproxy-service-gw
Access https://{LB}/auth in browser. and click on Administration Console.
Select the clients tab under the configure section and click on atom client ID.
In Settings, add CNAME value in Valid Redirect URLs and save it.

Deployment changes after adding CNAME

changes are required in atom-frontend and oauth2 deployment to make CNAME work.

Modify environment variables in oauth2-proxy deployment Steps:
- kubectl edit deployment oauth2-proxy -n atom
- Replace LB value with CNAME values in following environment variables, OAUTH2_PROXY_OIDC_ISSUER_URL and OAUTH2_PROXY_EXTRA_JWT_ISSUERS

Modify environment variables in atom-frontend deployment Steps:
- kubectl edit deployment atom-frontend -n atom
- Replace LB value with CNAME value in env variable KEYCLOAK_URL

Once deployment is successful, you can access the individual microservices running on different nodes of the Kubernetes cluster. Microservices can be accessed via System Manager Dashboard or using access details either via SSO or node-port. For detailed SSO information refer to section ATOM Single Sign-On

Name of the service	Description	How to access it?
Kubernetes Dashboard	Access for Kubernetes dashboard using SSO. Not available for AWS or GKE	https://<master-ip>/k8s/
ATOM	Local deployment: Login URL of ATOM UI & SSO based is via master IP Cloud deployment: Kgin URL of ATOM UI via LB of infra-web-proxy service & SSO is via LB of oauth2-proxy service	https://<master-ip>:30443 (for local users) https://<master-ip>/ (for SSO users) https://<FQDN> (for cloud users)
Grafana	Service which helps us in monitoring infrastructure health using heapster and time series database.	https://<master-ip>/grafana/ (for SSO users) https://<FQDN>/grafana (for cloud users)
Kibana	Service which helps us in log monitoring and analytics	https://<master-ip>/kibana/ (for SSO users) https://<FQDN>/kibana (for cloud users)

where master_ip is the IP address of the master node and its VIP ip in the case of HA masters setup.

To observe the IP addresses assigned to any of the microservices that could be deployed either on the master or the worker nodes, executing the following commands can help:

ATOM Single Sign-On (SSO)

ATOM Single Sign-On supports following Identity Providers (IdPs).

Keycloak – Keycloak is an open source identity provider and runs within the ATOM cluster.

For Log in with Atom, default user/password: admin/Secret@123 can be used. Additional users can be created by login to the atom authentication manager ui https://<master-ip>/auth.

Google – Please go to section Google Idp of this guide to configure the integration with Google SSO.

ATOM SSO support is enabled by default with the Keycloak as Identity Provider running locally in the ATOM cluster. Additional steps are needed to configure integration with different IdPs such as Google.

Default <master_ip> is configured during ATOM setup with self signed certificates. If a specific domain is desired, then users can provide a FQDN address for the master IP of K8s cluster and an SSL certificate associated with the same FQDN in PEM format.

Below are the steps listed for setting up Google SSO Integration with ATOM.

Google IdP

For Google based SSO logins, create callback urls in Google cloud platform.

- Login to Google cloud platform. In the project Dashboard center pane, choose APIs & Services tab.
- In the left navigation pane, choose “Credentials”

- Click the create credentials button. Select OAuth client ID.

- Select application type as web application

- Give name, add callback urls like below under Authorized redirect Urls and click on the create button.after that copy client id and secrets.

- Login to keycloak and update Identity provider details client id and client secret
- Now we can login to the atom application using google credentials.

ATOM System Alerts

Below are a list of ATOM System Alerts and scenarios when they can be generated, Actions which can be taken.

System Alert Name	Troubleshooting Steps
NodeHighMemory	Login to Grafana (https://<Master IP>/grafana/) Select Cluster Health dashboard Select the node which has a HighMemory alert and check which are the pods consuming more memory in that node.
NodeHighCPU	Login to Grafana (https://.<Master IP>/grafana/) Select Cluster Health dashboard Select the node which has HighCPU alert and check which are the pods consuming more CPU.

NodeHighDiskUsage	Login to Grafana. Select Cluster Monitoring Dashboard. Select the node which has High Disk usage and login to that node. Login to Master VM Find the IP of the node by following command. kubectl describe node <node-name> \| grep IP. you will get the IP of that node and then you can login to that node by using ssh. Check which folders are consuming Disk in /data folder by using du -h –max-depth=1 on that particular node. Run purge jobs in atom to cleanup the disk. If /data folder is consuming less then what shown in grafana then check the disk usage by following command df -h which will give full disk utilization. (/dev/mapper/centos-root is the filesystem we need to check)
NodeFault	This alert can be received for the following reasons. MemoryPressure: if pressure exists on the node memory (if the node memory is low) PIDPressure: if pressure exists on the processes (if there are too many processes on the node) DiskPressure: if pressure exists on the disk size (if the disk capacity is low) NetworkUnavailable: if the network for the node is not correctly configured. If you receive this Alert follow the steps below. Note the fault condition and node To get the list and status of nodes – kubectl get nodes. To check the fault condition type – kubectl describe <node-name>. In the output you will see the reason for the failure in conditions. Can be due to DiskPressure/Memory etc.
InstanceDown	This alert will come when the Node/VM is not reachable or down. If you receive this Alert, intimate to Admin.
PodNotReady	Pod can be in NotReady for a number of reasons. To get an overview of all the pods in ATOM execute kubectl get pods -n atom. To find out the reason why the specific pod was not ready execute kubectl describe pod <pod-name> -n atom Following can be the reasons for PodNotReady Taints on Node: If taints were added on node and pod spec doesn’t have tolerations w.r.t to taints on node then Pod will be in pending state.

	Insufficient Resources: If there are resource crunch in the cluster or required resources to deploy the pod was not available on the cluster then the pod will be in pending state. Node Selector: If the pod spec has nodeselector then for the pod to be eligible to run on a node, then node must have each of the indicated key-value pairs which is mentioned in nodeselector as labels. PV Claim: If scheduler doesn’t find node labels to deploy PV then pod will be in pending status with error “pod has unbound immediate PersistentVolumeClaims”. InitStuck: If a pod is stuck at init phase then subsequent init-containers are not ready. This could be because of the dependent pod being down. Ex: schema-repo dependent on broker.
ContainerNotUp	Below can be possible reasons for Container not being up. Crashloopbackoff ImagePullbackoff Application inside the container was not up. When this Alert is generated follow the steps below. Note the reason shown for the Container not up and pod-name Login to the Kibana (https://<Master IP>/kibana/) Filter by pod name Verify the logs and find if any error messages are shown.
ContainerTerminated	Containers can be killed for a number of reasons like OutofMemory (OOMKilled), Eviction, DiskPressure etc.. When this Alert is generated follow the steps below. Note the reason shown for the Container termination and pod-name Login to the Kibana (https://<Master IP>/kibana/) Examine kubernetes logs – use the query ‘pod-name : “eventrouter’ and select the appropriate time range, look for related logs. Additional filtering criteria like name of the pod can be used in the query. Look for errors or warnings in Pod logs – Filter by pod name. Ex- ‘“pod-name: “atom-core” and (error or warning or warn)’
ReplicasMismatch	This alert can be received when one or more pods are Not ready due to crashed application internally In pending state due to missing resource In failed container state When this Alert is generated follow the steps below Check the cause using kubectl logs <pod> -n atom and kubectl describe <pod> -n atom

ATOM System Alerts by default can be observed in ATOM > Assurance > Alerts. To enable those alerts to come into Slack as well, make sure to update the webhook url of the slack channel in config map of infra-tsdb-monitoring-alertmanager by following below steps:

kubectl edit configmap infra-tsdb-monitoring-alertmanager -n atom

Update your slack webhook url in slack_api_url option.
As intention is to have alerts observed in both slack and atom ui, make sure receiver field value is “default-receiver”
group_by: It is useful for alert deduplication and repeatability or stacking the alerts together.
- […] treats every label name and value as different, don’t change this unless you want different behavior.
  - For example: if you keep [device] as group_by attribute then each device alert will be notified only one irrespective of its type, severity etc..
Tune below timers based on your requirement, however default values are sufficient to get all the notifications.
- group_wait : How long to wait to buffer alerts of the same group before sending a notification initially. Usually, it will o to few minutes
- group_interval :How long to wait before sending an alert that has been added to a group for which there has already been a notification. Usually, it is 5 or more minutes
- repeat_interval : How long to wait before re-sending a given alert that has already been sent in a notification. Usually, it depends on the SLA’s to acknowledge and resolve the issues in your environment. Don’t keep less than 1 hour, as it chokes the system with too many duplicate notifications.

Note: Please do understand each option before changing from default to any other values as it impacts the throttling, alert deduplication.

Troubleshooting & FAQ

Following can be some issues seen during deployment.

Issue	Troubleshooting Steps
ATOM UI page is not reachable	Check if all the VM nodes in Esxi are in powered-on state Login to the Master and Check if K8s cluster shows all Nodes are in READY state Login to Master and Check if all the ATOM pods are in Status:Running state
ATOM UI page shows: 503 Service Unavailable	Login to Master and Check if all the ATOM pods are in Status:Running state. Check if all the Pods are showing READY 1/1, 2/2, 3/3 as applicable based on containers it holds.
ATOM deployment on KVM, where low CPU and I/O performance can impact	Cross check the CPU pinning if required and set the I/O mode to “native” in the node’s xml file
Overlapping IP address issue during ATOM deployment.	Calico CNI from Anuta defaults to 10.200.0.0/16 for the pods (containers). So one needs to cross check their lab networking before forming a kubernetes cluster.
Accessibility test between Remote Agent and ATOM Server over required nodePorts	To check the accessibility of databases running on ATOM Server from remote agent, one can run curl to one of the endpoints(nodePorts) like “curl -v https://<ATOM node IP>:<nodePort>” E.g: curl -v https://172.16.100.10:30081
Service Loadbalancers are in pending state	Check if metallb pods are running and active. If they are missing, then install the metallb helm chart. Prior to installation, set the VIP’s at values.yaml file

List of useful commands

Some of the commands that will be useful for debugging and troubleshooting.

Command	Description
helm create {package_name}	To create a directory
helm install {package_name} -n {name of app}	Deploy the application
helm ls -n atom	To check deployment status

Kubectl get deployments -n atom	To check deployments
Kubectl get pods -n atom	To check pod status
helm upgrade {releasename} {package-name} -n atom	deployment update with new changes.
helm history {package-name} -n atom	To view the history
helm uninstall {package_name} -n atom	To delete the package
helm rollback {package-name} version -n atom	Rollback the package

Cleanup of Deployment

By keeping the Kubernetes cluster, If ATOM Server deployment needs to be deleted for recreating it, then a proper cleanup needs to be done by following the below steps on the Kubernetes master.

Execute below cmd

helm uninstall `helm ls -n atom | awk ‘NR>1 {{print $1}}’` -n atom

From scripts folder of atom-deployment zip, execute ‘sh teardown-pv-pvc.sh’
From scripts folder of atom-deployment zip, execute ‘sh script_delete.sh’
Check if all the deployments got deleted or not by executing
- kubectl get deployments -n atom
- kubectl get statefulsets -n atom
- kubectl get pods -n atom
- helm ls -n atom
- kubectl get pv,pvc -n atom
Once all cleanup is done, execute ‘sudo python deploy_atom.py’ from Master node.

Guidance on KVM

Make sure you have the qcow images from Anuta or convert from OVA like below.

tar -xvf centos_1_21_300_linstor_0122.ova

# Convert the vmdk to qcow2

sudo qemu-img convert -f vmdk -O qcow2 centos_1_21_300_linstor_0122-disk1.vmdk centos_1_21_300_linstor_0122-disk1.qcow2

sudo qemu-img convert -f vmdk -O qcow2 centos_1_21_300_linstor_0122-disk2.vmdk centos_1_21_300_linstor_0122-disk2.qcow2

If you are working on a remote KVM machine without a GUI tool like vm manager, follow below steps

Dedicated Master node

virt-install –name “<VM_NAME>” –memory <RAM_IN_MB> –vcpus <CPU_COUNT> —disk

<FULL_PATH_OF_MASTER_QCOW2_IMAGE>,bus=virtio –network=<BRIDGE_NAME_AND_TYPE>

–vnc –import –nographics –os-type=linux –os-variant=centos7.0

ex:

virt-install –name “master” –memory 8192 –vcpus 4 —disk

/home/anuta/Downloads/master/centos_1_21_40.qcow2,bus=virtio

–network=bridge:virbr0,model=virtio –vnc –import –nographics

–os-type=linux –os-variant=centos7.0

For each Shared Master or Worker nodes with linstor disks

virt-install –name “<VM_NAME>” –memory <RAM_IN_MB> –vcpus <CPU_COUNT> —disk

<DISK1_QCOW2_IMAGE>,bus=virtio –disk <DISK2_QCOW2_IMAGE>,bus=virtio

–network=<BRIDGE_NAME_AND_TYPE> –vnc –import –nographics –os-type=linux

–os-variant=centos7.0

ex:

virt-install –name “worker1” –memory 32768 –vcpus 4 —disk

/home/anuta/Downloads/worker1/centos_1_21_300_linstor_1221-disk1.qcow2,bus=virt io –disk

/home/anuta/Downloads/worker1/centos_1_21_300_linstor_1221-disk2.qcow2,bus=virt io –network=bridge:virbr0,model=virtio –vnc –import –nographics

–os-type=linux –os-variant=centos7.0

In the case of Node having Multiple disks, make sure that

centos_1_21_300_linstor_1221-disk1.qcow2 is used for booting the VM. For this the disks need to be mapped appropriately to the correct name.

Boot disk always maps to centos_1_21_300_linstor_1221-disk1.qcow2 which maps to vda, while Data disk always maps to centos_1_21_300_linstor_1221-disk2.qcow2 which maps to vdb. If we still find that the VM does not boot appropriately a quick troubleshooting step would be to try and boot from the other disk.

Migration of Storage

Please follow the following steps to migrate Nodes from HDD to SSD or other suitable storage options.

STEP-1 ATOM prerequisites before Data store migration:

Put ATOM In maintenance mode.

Navigate to Administration > System Manager > Dashboard. Enable “Maintenance” option to put the system in maintenance mode.

Shutdown all nodes (VMs) that need to be migrated

STEP-2 ATOM prerequisites before Data store migration:

Migrate VM and change the Data Storage – for example, nodes running on esxi hosts can be migrated using vSphere.
Power on the Nodes

STEP-3 Post VM Migration Steps in ATOM:

Remove the maintenance mode.

Navigate to Administration > System Manager > Dashboard. Disable “Maintenance” option to clear the system from maintenance mode.

Following Example shows STEP-2 in a VMware based virtualization environment

- Right click on Node(VM) and select migrate option.

- Select the migration type.

- Select the storage type.
- Click on the Next and Finish button.

Steps to check logs in kibana

Open the kibana url , https://<master_ip>/kibana/
Create index pattern by going to Management/Index patterns/Create index pattern
Go to Discover and in the search box , search with pod name as shown below pod-name:”<pod-name>”, to check logs for specific pod

Some useful queries to get K8s events and K8s logs For atom core pod events

For K8s pod logs

Steps to check load distribution in kafka for config parser

It shows all partition details along with consumer and lag details

Logs for deployment failures

From the master node execute getlogsfrompod.sh shell script. The script is available in the scripts folder of atom-deployment zip.

This script creates a .tgz in /tmp folder.
Also collect /var/log/atom.log from master node.

Appendix

AWS Connectivity Options

Connectivity between DC and AWS:

AWS provides different options to connect the on-premises site to aws. We recommend the customer to follow the existing practice if there is one setup already. Here is the aws reference doc to consider different possible options. https://docs.aws.amazon.com/whitepapers/latest/aws-vpc-connectivity-options/network-to-am azon-vpc-connectivity-options.html

Connectivity between different VPCs:

If the deployment machine is in AWS, customers must establish the connection between the deployment machine and eks cluster. Since the deployment machine is recommended to be a different VPC than eks cluster, explicitly configuration steps are required to establish the connectivity between two VPC.

Using Transit GW:

Transit GW can be used in hub and spoke mode to attach both the VPCs to transit GW and add routes to respective route tables of private subnets in each VPC pointing to transit GW. Please refer to the below aws documentation about transit gateway configurations.

https://docs.aws.amazon.com/vpc/latest/tgw/tgw-vpc-attachments.html https://docs.aws.amazon.com/vpc/latest/tgw/tgw-route-tables.html

Custom SSL Certificate for ATOM

To apply a custom SSL certificate or a CA signed to the ATOM we need to follow the below steps.

In case of cloud:

Copy the certificate and private key to the jumphost

In case of on prem:

Login to the K8s master node as atom user
Copy the certificate and private key to the k8s master node

Execute following steps.

Delete existing secrets

kubectl delete secret -n atom atom-certificate

kubectl delete secret -n kube-system tls-secret(only for on-prem)

Create new secrets using the new certificate and private key.

kubectl create secret tls atom-certificate -n atom –cert=<certificate-filename>

–key=<private-key-filename> –dry-run -o yaml >cert-atom.yaml

kubectl create secret tls tls-secret -n kube-system –cert=<certificate-filename>

–key=<private-key-filename> –dry-run -o yaml >cert-dashboard.yaml

NOTE: Replace <certificate-filename> and <private-key-filename> with your files.

Apply the secrets

kubectl create -f cert-atom.yaml -f cert-dashboard.yaml

Restart the following pods using kubectl delete pod -n atom <pod name>
- infra-web-proxy (If certificate change is intended on direct ATOM UI)
- keycloak and oauth2_proxy (If certificate change is intended for SSO component)

File Server for ATOM ZTP

AWS EC2

Create CentOS 7 EC2 machine
- Goto AWS EC2 console
- Select Launch Instance, select CentOS 7 AMI from the community image
- Select the size as t2.medium
- Select the VPC and subnet that has access to the DHCP server of the required LAB(Target machine of PXE booting and ATOM). Public access can be selected based on access type and need.
- Next select the storage of about 100GB
- Next create or select the Security group which provides following access

- - SSH on TCP/22

- - TFTP on UDP/69

- - HTTP on TCP/80

- - ICMP ping(optional)
- Create or select the SSH keypair for accessing the VM
- Launch the instance
SSH using the private key and run the following command
- sudo yum install unzip –y
Copy the zip file shared by Anuta into the fileserver host and the extract the file using

unzip <FILENAME>.zip

Goto the unzipped folder and run sudo python pxe_environment.py -a <atom VIP>
Above command should bootstrap the node with required pxeboot files and default structure for the same. Note: The default configuration should be overwritten to use.
Verify the following files for required values
- Update the kickstart file for url and atom.tgz links under

/var/www/html/pxe/ks/centos7-ks.cfg and make sure that the URL has the connectivity for the target PXE machine.

- - If kickstart device is not default and requires specific NIC to be mentioned, update the –device=NIC_NAME for the network line
- Update the PXE config file under /var/lib/tftpboot/pxelinux.cfg/default
  - If kickstart device is not default and requires specific NIC to be mentioned, add ksdevice=NIC_NAME at the end of append line
- Goto the HTTP file server base path using cd /var/www/html/pxe
  - Extract the default atom.tgz using “sudo tar -xf atom.tgz”
  - Update the ATOM URL inside using “sudo vi atom/release/atom.properties”
  - Remove the old atom.tgz using “sudo rm -f atom.tgz”
  - Recreate the atom.tgz file using “sudo tar -cf atom.tgz atom”

ON-PREM VM

Create a CentOS 7 VM on the VMware or the KVM
- Provide the specs as 4cpu, 8GB ram and 100GB of storage
- Provide the static IP and add networking which has connectivity to below ports between ATOM and Fileserver

- - SSH on TCP/22

- - TFTP on UDP/69

- - HTTP on TCP/80
  - ICMP ping(optional)
SSH using the private key and run the following command
- sudo yum install unzip –y
Copy the zip file shared by Anuta into the fileserver host and the extract the file using

unzip <FILENAME>.zip

Goto the unzipped folder(pxeboot) using “cd pxeboot” and run sudo python pxe_environment.py -a <atom VIP>
Above command should bootstrap the node with required pxeboot files and default structure for the same. Note: The default configuration should be overwritten to use.
Verify the following files for required values
- Update the kickstart file for url and atom.tgz links under

/var/www/html/pxe/ks/centos7-ks.cfg and make sure that the URL has the connectivity for the target PXE machine.

- If kickstart device is not default and requires specific NIC to be mentioned, update the –device=NIC_NAME for the network line
- Update the PXE config file under /var/lib/tftpboot/pxelinux.cfg/default
  - If kickstart device is not default and requires specific NIC to be mentioned, add ksdevice=NIC_NAME at the end of append line
- Goto the HTTP file server base path using cd /var/www/html/pxe
  - Extract the default atom.tgz using “sudo tar -xf atom.tgz”
  - Update the ATOM URL inside using “sudo vi atom/release/atom.properties”
  - Remove the old atom.tgz using “sudo rm -f atom.tgz”
  - Recreate the atom.tgz file using “sudo tar -cf atom.tgz atom”

Table of Contents

Purpose of this document

Intended Audience

Overview of ATOM Architecture

ATOM Deployment

Deployment scenarios

Local Deployment

Distributed Deployment

Target Infrastructure

On-Prem VMware ESXi, KVM

Cloud (GCP / AWS)

Requirements

Compute, Storage & Memory

Note:

Overview

Minimal Setup

Resilient HA Setup

Multi-site Deployment (Remote ATOM agent)

ATOM Multi Availability Zone HA Deployment

References:

Caveats:

3 Sites Deployment:

For Each Site:

2 Sites Deployment:

Site-1:

Site-2:

AWS Availability Zones

On Premises Across Data Centers / Locations

Network Requirements

ATOM related Ports/Protocols

Kubernetes related Ports/Protocols(on-prem)

Linstor related Ports/Protocols(on-prem)

Protocol: TCP, Ports: 3366-3367, 3370, 3376-3377, 7000-8000

IP Addressing Requirements

Kubernetes Cluster Requirements

Deployment scripts and files

ATOM Software Requirements

Deployment Images

Deployment scripts and files

Security Apps on VM nodes before ATOM install

Procedure for Deploying ATOM on-prem

New Kubernetes cluster

Minimal Setup deployment

Resilient Setup deployment

Zone or DC location

Zones or DC locations(Recommended)

ATOM Deployment

–show-labels”.

Docker registry for Offline deployment

ATOM Remote Agent Deployment

Procedure of Deploying ATOM in GCP/GKE

Prerequisites

Deploying New K8s Cluster

Minimal Setup:

Resilient-HA Setup:

Deploying ATOM

Procedure of Deploying ATOM in AWS

Prerequisites

Deploying New K8s Cluster

Minimal Setup:

Resilient-HA Setup:

Deploying ATOM

ATOM System Manager

Post Installation

AWS Specific

Security Group updates

Custom DNS name(CNAME) creation

ATOM Single Sign-On (SSO)

Google IdP

ATOM System Alerts

Troubleshooting & FAQ

List of useful commands

Cleanup of Deployment

Guidance on KVM

Migration of Storage

STEP-1 ATOM prerequisites before Data store migration:

STEP-2 ATOM prerequisites before Data store migration:

STEP-3 Post VM Migration Steps in ATOM:

Steps to check logs in kibana