Insights Data Schemas
[Front page] [External data pipeline]
Insights Operator installed on a cluster sends its data regularly into Ingress Service. This service stores that data into S3 bucket for further processing. The retention policy of such data are two days.
1 (unofficial)
Archive containing data gathered by Insights Operaton from cluster.
Please look into https://github.com/openshift/insights-operator/blob/master/docs/gathered-data.md for up-to-date description.
This document is auto-generated by make gen-doc
collects the specified Custom Resource Definitions.
The following CRDs are gathered:
The CRD sizes above are in the raw (uncompressed) state.
Location in archive: config/crd/ Id in config: crds
collects anonymized CertificateSigningRequests. Collects CSRs which werent Verified, or when Now < ValidBefore or Now > ValidAfter
The Kubernetes api https://github.com/kubernetes/client-go/blob/master/kubernetes/typed/certificates/v1beta1/certificatesigningrequest.go#L78 Response see https://docs.openshift.com/container-platform/4.3/rest_api/index.html#certificatesigningrequestlist-v1beta1certificates
Location in archive: config/certificatesigningrequests/ Id in config: certificate_signing_requests
fetches the cluster Authentication - the Authentication with name cluster.
The Kubernetes api https://github.com/openshift/client-go/blob/master/config/clientset/versioned/typed/config/v1/authentication.go#L50 Response see https://docs.openshift.com/container-platform/4.3/rest_api/index.html#authentication-v1operator-openshift-io
Location in archive: config/authentication/ See: docs/insights-archive-sample/config/authentication Id in config: authentication
fetches the cluster FeatureGate - the FeatureGate with name cluster.
The Kubernetes api https://github.com/openshift/client-go/blob/master/config/clientset/versioned/typed/config/v1/featuregate.go#L50 Response see https://docs.openshift.com/container-platform/4.3/rest_api/index.html#featuregate-v1-config-openshift-io
Location in archive: config/featuregate/ See: docs/insights-archive-sample/config/featuregate Id in config: feature_gates
stores ClusterID from ClusterVersion version This method uses data already collected by Get ClusterVersion. In particular field .Spec.ClusterID The Kubernetes api https://github.com/openshift/client-go/blob/master/config/clientset/versioned/typed/config/v1/clusterversion.go#L50 Response see https://github.com/openshift/api/blob/master/config/v1/types_cluster_version.go#L38
Location in archive: config/id/ See: docs/insights-archive-sample/config/id Id in config: id
fetches the image pruner configuration
Location in archive: config/clusteroperator/imageregistry.operator.openshift.io/imagepruner/cluster.json Id in config: image_pruners
fetches the cluster Image Registry configuration
Location in archive: config/clusteroperator/imageregistry.operator.openshift.io/config/cluster.json Id in config: image_registries
fetches the cluster Infrastructure - the Infrastructure with name cluster.
The Kubernetes api https://github.com/openshift/client-go/blob/master/config/clientset/versioned/typed/config/v1/infrastructure.go#L50 Response see https://docs.openshift.com/container-platform/4.3/rest_api/index.html#infrastructure-v1-config-openshift-io
Location in archive: config/infrastructure/ See: docs/insights-archive-sample/config/infrastructure Id in config: infrastructures
fetches the cluster Ingress - the Ingress with name cluster.
The Kubernetes api https://github.com/openshift/client-go/blob/master/config/clientset/versioned/typed/config/v1/ingress.go#L50 Response see https://docs.openshift.com/container-platform/4.3/rest_api/index.html#ingress-v1-config-openshift-io
Location in archive: config/ingress/ See: docs/insights-archive-sample/config/ingress Id in config: ingress
fetches the cluster Network - the Network with name cluster.
The Kubernetes api https://github.com/openshift/client-go/blob/master/config/clientset/versioned/typed/config/v1/network.go#L50 Response see https://docs.openshift.com/container-platform/4.3/rest_api/index.html#network-v1-config-openshift-io
Location in archive: config/network/ See: docs/insights-archive-sample/config/network Id in config: networks
fetches the cluster OAuth - the OAuth with name cluster.
The Kubernetes api https://github.com/openshift/client-go/blob/master/config/clientset/versioned/typed/config/v1/oauth.go#L50 Response see https://docs.openshift.com/container-platform/4.3/rest_api/index.html#oauth-v1-config-openshift-io
Location in archive: config/oauth/ See: docs/insights-archive-sample/config/oauth Id in config: oauths
collects all ClusterOperators and their resources. It finds unhealthy Pods for unhealthy operators
The Kubernetes api https://github.com/openshift/client-go/blob/master/config/clientset/versioned/typed/config/v1/clusteroperator.go#L62 Response see https://docs.openshift.com/container-platform/4.3/rest_api/index.html#clusteroperatorlist-v1config-openshift-io
Location of operators in archive: config/clusteroperator/ See: docs/insights-archive-sample/config/clusteroperator Location of pods in archive: config/pod/ Id in config: operators
Output raw size: 245
[{“Name”:”config/clusteroperator/”,”Captured”:”0001-01-01T00:00:00Z”,”Fingerprint”:””,”Item”:{“metadata”:{“creationTimestamp”:null},”spec”:{},”status”:{“conditions”:[{“type”:”Degraded”,”status”:””,”lastTransitionTime”:null}],”extension”:null}}}]
fetches the cluster Proxy - the Proxy with name cluster.
The Kubernetes api https://github.com/openshift/client-go/blob/master/config/clientset/versioned/typed/config/v1/proxy.go#L30 Response see https://docs.openshift.com/container-platform/4.3/rest_api/index.html#proxy-v1-config-openshift-io
Location in archive: config/proxy/ See: docs/insights-archive-sample/config/proxy Id in config: proxies
fetches the ClusterVersion - the ClusterVersion with name version.
The Kubernetes api https://github.com/openshift/client-go/blob/master/config/clientset/versioned/typed/config/v1/clusterversion.go#L50 Response see https://docs.openshift.com/container-platform/4.3/rest_api/index.html#clusterversion-v1config-openshift-io
Location in archive: config/version/ See: docs/insights-archive-sample/config/version Id in config: version
fetches the ConfigMaps from namespace openshift-config.
Anonymization: If the content of ConfigMap contains a parseable PEM structure (like certificate) it removes the inside of PEM blocks. For ConfigMap of type BinaryData it is encoded as standard base64. In the archive under configmaps we store name of ConfigMap and then each ConfigMap Key. For example config/configmaps/CONFIGMAPNAME/CONFIGMAPKEY1
The Kubernetes api https://github.com/kubernetes/client-go/blob/master/kubernetes/typed/core/v1/configmap.go#L80 Response see https://docs.openshift.com/container-platform/4.3/rest_api/index.html#configmaplist-v1core
Location in archive: config/configmaps/ See: docs/insights-archive-sample/config/configmaps Id in config: config_maps
collects essential information about running containers. Specifically, the age of pods, the set of running images and the container names are collected.
Location in archive: config/running_containers.json Id in config: container_images
collects ContainerRuntimeConfig information
The Kubernetes api https://github.com/openshift/machine-config-operator/blob/master/pkg/apis/machineconfiguration.openshift.io/v1/types.go#L402 Response see https://docs.okd.io/latest/rest_api/machine_apis/containerruntimeconfig-machineconfiguration-openshift-io-v1.html
Location in archive: config/containerruntimeconfigs/ Id in config: container_runtime_configs
collects HostSubnet information
The Kubernetes api https://github.com/openshift/client-go/blob/master/network/clientset/versioned/typed/network/v1/hostsubnet.go Response see https://docs.openshift.com/container-platform/4.3/rest_api/index.html#hostsubnet-v1-network-openshift-io
Location in archive: config/hostsubnet/ Id in config: host_subnets
collects Top x InstallPlans from all openshift namespaces. Because InstallPlans have unique generated names, it groups them by namespace and the “template” for name generation from field generateName. It also collects Total number of all installplans and all non-unique installplans.
The Operators-Framework api https://github.com/operator-framework/api/blob/master/pkg/operators/v1alpha1/installplan_types.go#L26
Location in archive: config/installplans/ Id in config: install_plans
collects MachineConfigPool information
The Kubernetes api https://github.com/openshift/machine-config-operator/blob/master/pkg/apis/machineconfiguration.openshift.io/v1/types.go#L197 Response see https://docs.okd.io/latest/rest_api/machine_apis/machineconfigpool-machineconfiguration-openshift-io-v1.html
Location in archive: config/machineconfigpools/ Id in config: machine_config_pools
collects MachineSet information
The Kubernetes api https://github.com/openshift/machine-api-operator/blob/master/pkg/generated/clientset/versioned/typed/machine/v1beta1/machineset.go Response see https://docs.openshift.com/container-platform/4.3/rest_api/index.html#machineset-v1beta1-machine-openshift-io
Location in archive: machinesets/ Id in config: machine_sets
gathers cluster Federated Monitoring metrics.
The GET REST query to URL /federate Gathered metrics: etcd_object_counts cluster_installer namespace CPU and memory usage followed by at most 1000 lines of ALERTS metric
Location in archive: config/metrics/ See: docs/insights-archive-sample/config/metrics Id in config: metrics
Output raw size: 148
[{“Name”:”config/metrics”,”Captured”:”0001-01-01T00:00:00Z”,”Fingerprint”:””,”Item”:”SGVsbG8sIGNsaWVudAojIEFMRVJUUyAyLzEwMDAKSGVsbG8sIGNsaWVudAo=”}]
collects NetNamespaces networking information
The Kubernetes api https://github.com/openshift/client-go/blob/master/network/clientset/versioned/typed/network/v1/netnamespace.go Response is an array of netNamespaces. Netnamespace contains Name, EgressIPs and NetID attributes.
Location in archive: config/netnamespaces Id in config: netnamespaces
collects all Nodes.
The Kubernetes api https://github.com/kubernetes/client-go/blob/master/kubernetes/typed/core/v1/node.go#L78 Response see https://docs.openshift.com/container-platform/4.3/rest_api/index.html#nodelist-v1core
Location in archive: config/node/ Id in config: nodes
Output raw size: 491
[{“Name”:”config/node/”,”Captured”:”0001-01-01T00:00:00Z”,”Fingerprint”:””,”Item”:{“metadata”:{“creationTimestamp”:null},”spec”:{},”status”:{“conditions”:[{“type”:”Ready”,”status”:”False”,”lastHeartbeatTime”:null,”lastTransitionTime”:null}],”daemonEndpoints”:{“kubeletEndpoint”:{“Port”:0}},”nodeInfo”:{“machineID”:””,”systemUUID”:””,”bootID”:””,”kernelVersion”:””,”osImage”:””,”containerRuntimeVersion”:””,”kubeletVersion”:””,”kubeProxyVersion”:””,”operatingSystem”:””,”architecture”:””}}}}]
collects logs from openshift-apiserver-operator with following substrings:
The Kubernetes API https://github.com/kubernetes/client-go/blob/master/kubernetes/typed/core/v1/pod_expansion.go#L48 Response see https://docs.openshift.com/container-platform/4.6/rest_api/workloads_apis/pod-core-v1.html#apiv1namespacesnamespacepodsnamelog
Location in archive: config/pod/openshift-apiserver-operator/logs/{pod-name}/errors.log
collects logs from pods in openshift-sdn namespace with following substrings:
The Kubernetes API https://github.com/kubernetes/client-go/blob/master/kubernetes/typed/core/v1/pod_expansion.go#L48 Response see https://docs.openshift.com/container-platform/4.6/rest_api/workloads_apis/pod-core-v1.html#apiv1namespacesnamespacepodsnamelog
Location in archive: config/pod/openshift-sdn/logs/{pod-name}/errors.log
gathers the cluster’s PodDisruptionBudgets.
The Kubernetes api https://github.com/kubernetes/client-go/blob/v11.0.0/kubernetes/typed/policy/v1beta1/poddisruptionbudget.go#L80 Response see https://docs.okd.io/latest/rest_api/policy_apis/poddisruptionbudget-policy-v1beta1.html
Location in archive: config/pdbs/ See: docs/insights-archive-sample/config/pdbs Id in config: pdbs
collects ServiceAccount stats from kubernetes default and namespaces starting with openshift.
The Kubernetes api https://github.com/kubernetes/client-go/blob/master/kubernetes/typed/core/v1/serviceaccount.go#L83 Response see https://docs.openshift.com/container-platform/4.3/rest_api/index.html#serviceaccount-v1-core
Location of serviceaccounts in archive: config/serviceaccounts See: docs/insights-archive-sample/config/serviceaccounts Id in config: service_accounts
collects StatefulSet configs from default namespaces
The Kubernetes API https://github.com/kubernetes/api/blob/master/apps/v1/types.go Response see https://docs.openshift.com/container-platform/4.5/rest_api/workloads_apis/statefulset-apps-v1.html#statefulset-apps-v1
Location in archive: config/statefulsets/ Id in config: stateful_sets