Format of output files generated by Parquet Factory

Parquet Factory is a program that can read data from several data sources, aggregate the data received from them and generate a set of Parquet files with the aggregated data, storing them in a selected S3 or Ceph bucket. It is used to generate different data aggregations in the CCX Internal Data Pipeline, reading data from Kafka topics.

Schema version

1 (unofficial)

Description

Nine types of Parquet files (tables) are generated by Parquet Factory service:

cluster_info
rule_hits
failing_operator_conditions
alerts
gatherers_info
workload_containers
workload_info_images
available_updates
conditional_update_conditions
conditional_update_risks
archives

Format of these tables are described below.

Basic format

NOTE

byte array is used in Parquet files also to represent string (among other data structures). For more information please look at https://parquet.apache.org/documentation/latest/.

Format of `cluster_info` table

cluster_id (byte array) cluster ID represented as UUID
current_version (byte array) cluster version represented as string
platform (byte array) cluster platform specification
collected_at (timestamp) timestamp of report represented with millisecond precision
desired_version (byte array) desired cluster version represented as string
initial_version (byte array) initial cluster version represented as string
network_type (byte array) the type of network used by the cluster represented as string
channel (byte array) cluster’s upgrade channel
network_mtu (uint32) the network’s MTU size used by the cluster represented as uint32
network_kind (string) whether the network is a service or a cluster
network_size (string) the size of the network, extracted from the mask
network_host_prefix (uint8) the network host prefix
archive_path (byte array) path to the object stored in Ceph

`cluster_id` attribute

cluster_id uses its canonical textual representation: the 16 octets of a UUID are represented as 32 hexadecimal (base-16) digits, displayed in five groups separated by hyphens, in the form 8-4-4-4-12 for a total of 36 characters (32 hexadecimal characters and 4 hyphens). For more information please look at https://en.wikipedia.org/wiki/Universally_unique_identifier.

An example of UUID:

3ba9b042-b8b8-4714-98e9-17915c2eeb95

`current_version` attribute

Current cluster version represented as string, for example “4.5.24” or even “4.6.0-0.ci.test-2020-12-01-123456-ci-xx-yyyyyyy”

`platform` attribute

Cluster platform specification, for example “BareMetal”, “OpenStack”, “Azure” etc.

`collected_at` attribute

Timestamp of report represented with millisecond precision, for example “2021-02-08 00:22:19” (this is the display format, the precision is higher internally)

`desired_version` attribute

Desired cluster version represented as string, for example “4.5.24” or even “4.6.0-0.ci.test-2020-12-01-123456-ci-xx-yyyyyyy”

`initial_version` attribute

Initial cluster version represented as string, for example “4.5.24” or even “4.6.0-0.ci.test-2020-12-01-123456-ci-xx-yyyyyyy”

`network_type` attribute

This column represent the type of network used by the cluster. Some well-known values are “OpenshiftSDN”, “NCP” or “Kuryr”.

`channel` attribute

This column represent the cluster’s upgrade channel, for example “stable-4.7”.

`network_mtu` attribute

This column represent the network’s MTU size used by the cluster. It is expressed in bytes. For example, a MTU of size 1500 bytes is saved as 1500.

`network_kind` attribute

More than one per cluster_id. This is the type of networks the cluster has. It can be service or cluster and is gathered both from spec.serviceNetwork and spec.clusterNetwork.

`network_size` attribute

More than one per cluster_id. This is the size of each network in the cluster. It is extracted directly from the network mask. For example a “172.30.0.0/16” mask would have a “/16” size. The size is padded with zeros too “172.30.0.0/4” would have “/04”.

`network_host_prefix` attribute

More than one per cluster_id. This is the value of spec.clusterNetwork.hostPrefix. It is 0 if the network is of kind service.

`archive_path` attribute

This attribute contains path to object stored in Ceph. It must be real path with chunks splitted by slash character. The format of path is:

archives/compressed/{PREFIX}/{CLUSTER_ID}/{YEAR+MONTH}/{DAY}/{ID}.tar.gz |

An example of path:

archives/compressed/0a/0aaaaaaa-bbbb-cccc-dddd-ffffffffffff/202102/08/003201.tar.gz |

Format of `rule_hits` table

cluster_id (byte array) cluster ID represented as UUID
rule_id (byte array)
collected_at (timestamp) timestamp of report represented in milliseconds
archive_path (byte array) path to the object stored in Ceph

`cluster_id` attribute

An example of UUID:

3ba9b042-b8b8-4714-98e9-17915c2eeb95

`rule_id` attribute

It contains rule name and a key that are joined by

character.

Example of rule IDs:

pods_check_containers|POD_CONTAINER_ISSUE
operators_check|OPERATOR_ISSUE
pods_check|POD_ISSUE

`collected_at` attribute

Timestamp of report represented with millisecond precision, for example “2021-02-08 00:22:19” (this is the display format, the precision is higher internally)

`archive_path` attribute

This attribute contains path to object stored in Ceph. It must be real path with chunks splitted by slash character. The format of path is:

archives/compressed/{PREFIX}/{CLUSTER_ID}/{YEAR+MONTH}/{DAY}/{ID}.tar.gz |

An example of path:

archives/compressed/0a/0aaaaaaa-bbbb-cccc-dddd-ffffffffffff/202102/08/003201.tar.gz |

`cluster_id` attribute

An example of UUID:

3ba9b042-b8b8-4714-98e9-17915c2eeb95

`ebs_account` attribute

Account number, which is usually a positive integer represented as string, for example “123456”.

`email_domain` attribute

An e-mail domain, for example: “redhat.com”

`managed` (bool) attribute

Boolean flag containing info if the cluster is managed or not.

`support` attribute

Specifies the level of (customer) support for this cluster.

`collected_at` attribute

Timestamp of report represented with millisecond precision, for example “2021-02-08 00:22:19” (this is the display format, the precision is higher internally)

Format of `failing_operator_conditions` table

cluster_id (byte array) cluster ID represented as UUID
operator (byte array) specifies which is the reporting operator
condition_type (byte array) indicates the type of status reported by the operator
condition_status (bool) flag whether the reported operator is active or not
reason (byte array) specifies the main reason that the operator is reporting
raw_message (byte array) is the whole error/information message reported by the operator
archive_path (byte array) path to the object stored in Ceph
last_transition_time (timestamp) timestamp of the last reported transition to this status
collected_at (timestamp) timestamp of report represented with millisecond precision

`cluster_id` attribute

An example of UUID:

3ba9b042-b8b8-4714-98e9-17915c2eeb95

`operator` attribute

Name of the operator reporting its condition.

Some examples of operators that can be received:

authentication
kube-apiserver
monitoring
network

`condition_type` attribute

Type of the condition that is being reported by the operator. It will be a byte array from the following set of expected values:

Available
Degraded
Progressing
Upgradeable
Failing

`condition_status` attribute

Boolean flag that specifies the status of the given type reported by the operator.

`reason` attribute

A byte array containing a brief explanation about the operator condition status change.

`raw_message` attribute

A byte array containing the full description of the condition status change.

`archive_path` attribute

This attribute contains path to object stored in Ceph. It must be real path with chunks splitted by slash character. The format of path is:

archives/compressed/{PREFIX}/{CLUSTER_ID}/{YEAR+MONTH}/{DAY}/{ID}.tar.gz |

An example of path:

archives/compressed/0a/0aaaaaaa-bbbb-cccc-dddd-ffffffffffff/202102/08/003201.tar.gz |

`last_transition_time` attribute

Timestamp of the last transition to the reported status for the operator, represented with millisecond precision, for example “2021-02-08 00:22:19” (this is the display format, the precision is higher internally)

`collected_at` (timestamp) timestamp of report represented with millisecond precision

Timestamp of report represented with millisecond precision, for example “2021-02-08 00:22:19” (this is the display format, the precision is higher internally)

Format of `alerts` table

cluster_id (byte array) cluster ID represented as UUID
name (byte array) specifies the name of the reported alert
state (byte array) indicates the state of the alert (firing/pending/…)
severity (byte array) indicates the severeness of the alert
namespace (byte array) is the namespace the cluster belongs to, if any
job (byte array) is the name of the job during which the alert triggered, if any
labels (byte array) JSON containing the alert’s labels - necessary additional information
archive_path (byte array) path to the object stored in Ceph

`cluster_id` attribute

An example of UUID:

3ba9b042-b8b8-4714-98e9-17915c2eeb95

`name` attribute

Name of the alert being reported.

Some examples of alerts:

Watchdog
APIRemovedInNextReleaseInUse
KubePodCrashLooping
KubeAPIErrorBudgetBurn
ClusterOperatorDegraded

`state` attribute

Indicates whether the alert is pending or firing.

Examples:

pending
firing

`severity` attribute

Indicates how severe the alert is. Indicates the seriousness of the alert.

Examples:

warning
critical
alert
info

`namespace` attribute

This is the name of the namespace to which the cluster belongs to, if it belongs to any - the field may be empty. The value is extracted from the labels attribute.

Examples:

openshift-logging
openshift-kube-controller-manager
openshift-cluster-version

`job` attribute

Name of the job during which the alert triggered, if any. The field may be empty. The value is extracted from the labels attribute.

Examples:

kube-state-metrics
cluster-version-operator

`labels` attribute

A JSON encoded string/byte array containing necessary information for the alert about the cluster.

Examples:

{"prometheus":"openshift-monitoring/k8s","prometheus_replica":"prometheus-k8s-0"
{"group":"apiextensions.k8s.io","prometheus":"openshift-monitoring/k8s","prometheus_replica":"prometheus-k8s-0","resource":"customresourcedefinitions","version":"v1beta1"}

`archive_path` attribute

This attribute contains path to object stored in Ceph. It must be real path with chunks splitted by slash character. The format of path is:

archives/compressed/{PREFIX}/{CLUSTER_ID}/{YEAR+MONTH}/{DAY}/{ID}.tar.gz |

An example of path:

archives/compressed/0a/0aaaaaaa-bbbb-cccc-dddd-ffffffffffff/202102/08/003201.tar.gz |

Format of `gatherers_info` table

cluster_id (byte array) cluster ID represented as UUID
name (byte array) specifies the name of the reported gatherer
duration_in_ms (int) time in milliseconds that the gatherer was running
archive_path (byte array) path to the object stored in Ceph
collected_at (timestamp) timestamp of report represented with millisecond precision

`cluster_id` attribute

An example of UUID:

3ba9b042-b8b8-4714-98e9-17915c2eeb95

`name` attribute

The name indicates one of the gatherer modules that was used during the archive generation in the form of a Go function name (package.Function).

An example of function name:

clusterconfig.GatherPodDisruptionBudgets

`duration_in_ms` attribute

This attribute indicates the time (in milliseconds) that took to the Insights Operator to run this gathering function.

`archive_path` attribute

This attribute contains path to object stored in Ceph. It must be real path with chunks splitted by slash character. The format of path is:

archives/compressed/{PREFIX}/{CLUSTER_ID}/{YEAR+MONTH}/{DAY}/{ID}.tar.gz |

An example of path:

archives/compressed/0a/0aaaaaaa-bbbb-cccc-dddd-ffffffffffff/202102/08/003201.tar.gz |

`collected_at` attribute

Timestamp of report represented with millisecond precision, for example “2021-02-08 00:22:19” (this is the display format, the precision is higher internally)

Format of `workload_containers` table

cluster_id (byte array) cluster ID represented as UUID
shape_id (byte array) a shape element of the report data
namespace (byte array) is the namespace the cluster belongs to, if any
shape_instances (int64) represents each result from the report data
restart_always (boolean) the restart always setting of the container
image_id (byte array) the sha256 image running in the container
first_command (byte array) hash of the first command of the container
first_arg (byte array) hash of the first argument of the container
init_container (boolean) true if the container is an init container
archive_path (byte array) path to the object stored in Ceph
collected_at (timestamp) timestamp of report represented with millisecond precision

`cluster_id` attribute

An example of UUID:

3ba9b042-b8b8-4714-98e9-17915c2eeb95

`shape_id` attribute

a random UUID based on RFC 4122 and DCE 1.1 to uniquely identify the result from the report data

An example of UUID:

3ba9b042-b8b8-4714-98e9-17915c2eeb95

`namespace` attribute

This is the name of the namespace to which the cluster belongs to, if it belongs to any - the field may be empty. The value is extracted from the labels attribute.

Examples:

openshift-logging
openshift-kube-controller-manager
openshift-cluster-version

`shape_instances` attribute

an integer the represents the shapes no

`restart_always` attribute

true if restartPolicy option of the container is set to Always

`image_id` attribute

the sha256 hash of the image of the container

the format of the hash is:

sha256:b72024d660a552619ab9d28c6ff3d48e73e73e05de759838104c1f53db14f867

`first_command` attribute

the hash of the first command of the container.

An example of first_command:

Cl6kTzfbYztB

`first_arg` attribute

the hash of the first argument of the container.

An example of first_argument:

BuLBUMMJnyP_

`init_container` attribute

true if the container is an init container

`archive_path` attribute

This attribute contains path to object stored in Ceph. It must be real path with chunks splitted by slash character. The format of path is:

archives/compressed/{PREFIX}/{CLUSTER_ID}/{YEAR+MONTH}/{DAY}/{ID}.tar.gz |

An example of path:

archives/compressed/0a/0aaaaaaa-bbbb-cccc-dddd-ffffffffffff/202102/08/003201.tar.gz |

`collected_at` attribute

Timestamp of report represented with millisecond precision, for example “2021-02-08 00:22:19” (this is the display format, the precision is higher internally)

Format of `workload_image_layers` table

cluster_id (byte array) cluster ID represented as UUID
image_id (byte array) the sha256 image running in the container
layer_image_id (byte array) the sha256 of the layer of an image
layer_image_level (int64) the index of the layer
first_command (byte array) hash of the first command of the container
first_arg (byte array) hash of the first argument of the container
archive_path (byte array) path to the object stored in Ceph
collected_at (timestamp) timestamp of report represented with millisecond precision

`cluster_id` attribute

An example of UUID:

3ba9b042-b8b8-4714-98e9-17915c2eeb95

`image_id` attribute

the sha256 hash of the image of the container

the format of the hash is:

sha256:b72024d660a552619ab9d28c6ff3d48e73e73e05de759838104c1f53db14f867

`layer_image_id` attribute

the sha256 hash of the layer of the image

the format of the hash (same as image_id) is:

sha256:b72024d660a552619ab9d28c6ff3d48e73e73e05de759838104c1f53db14f867

`layer_image_level` attribute

Integer, represents the index of the image layer in the container

`first_command` attribute

the hash of the first command of the container.

An example of first_command:

Cl6kTzfbYztB

`first_arg` attribute

the hash of the first argument of the container.

An example of first_argument:

BuLBUMMJnyP_

`archive_path` attribute

This attribute contains path to object stored in Ceph. It must be real path with chunks splitted by slash character. The format of path is:

archives/compressed/{PREFIX}/{CLUSTER_ID}/{YEAR+MONTH}/{DAY}/{ID}.tar.gz |

An example of path:

archives/compressed/0a/0aaaaaaa-bbbb-cccc-dddd-ffffffffffff/202102/08/003201.tar.gz |

`collected_at` attribute

Timestamp of report represented with millisecond precision, for example “2021-02-08 00:22:19” (this is the display format, the precision is higher internally)

Format of `available_updates` table

cluster_id (byte array) cluster ID represented as UUID
current_version (byte array) cluster version represented as string
release (byte array) available update version
collected_at (timestamp) timestamp of report represented with millisecond precision
archive_path (byte array) path to the object stored in Ceph

`cluster_id` attribute

An example of UUID:

3ba9b042-b8b8-4714-98e9-17915c2eeb95

`current_version` attribute

Current cluster version represented as string, for example “4.5.24” or even “4.6.0-0.ci.test-2020-12-01-123456-ci-xx-yyyyyyy”

`release` attribute

Available update version, for example “4.10.0-fc.4”.

`collected_at` attribute

Timestamp of report represented with millisecond precision, for example “2021-02-08 00:22:19” (this is the display format, the precision is higher internally)

`archive_path` attribute

This attribute contains path to object stored in Ceph. It must be real path with chunks splitted by slash character. The format of path is:

archives/compressed/{PREFIX}/{CLUSTER_ID}/{YEAR+MONTH}/{DAY}/{ID}.tar.gz |

An example of path:

archives/compressed/0a/0aaaaaaa-bbbb-cccc-dddd-ffffffffffff/202102/08/003201.tar.gz |

Format of `conditional_update_conditions` table

cluster_id (byte array) cluster ID represented as UUID
current_version (byte array) cluster version represented as string
release (byte array) available update version
recommended (byte array) whether the conditional update was recommended
reason (byte array) reason for the update resolution
message (byte array) details for the update resolution
collected_at (timestamp) timestamp of report represented with millisecond precision
archive_path (byte array) path to the object stored in Ceph

`cluster_id` attribute

An example of UUID:

3ba9b042-b8b8-4714-98e9-17915c2eeb95

`current_version` attribute

Current cluster version represented as string, for example “4.5.24” or even “4.6.0-0.ci.test-2020-12-01-123456-ci-xx-yyyyyyy”

`release` attribute

Available update version, for example “4.10.0-fc.4”.

`recommended` attribute

Whether the conditional update was recommended, only “True”, “False” or “Unknown”.

`reason` attribute

reason for the update resolution, for example “AsExpected”.

`message` attribute

Details for the update resolution, for example “The update is recommended, because none of the conditional update risks apply to this cluster.”.

`collected_at` attribute

Timestamp of report represented with millisecond precision, for example “2021-02-08 00:22:19” (this is the display format, the precision is higher internally)

`archive_path` attribute

This attribute contains path to object stored in Ceph. It must be real path with chunks splitted by slash character. The format of path is:

archives/compressed/{PREFIX}/{CLUSTER_ID}/{YEAR+MONTH}/{DAY}/{ID}.tar.gz |

An example of path:

archives/compressed/0a/0aaaaaaa-bbbb-cccc-dddd-ffffffffffff/202102/08/003201.tar.gz |

Format of `conditional_update_risks` table

cluster_id (byte array) cluster ID represented as UUID
current_version (byte array) cluster version represented as string
release (byte array) available update version
risk (byte array) name of the risk for the conditional update
collected_at (timestamp) timestamp of report represented with millisecond precision
archive_path (byte array) path to the object stored in Ceph

`cluster_id` attribute

An example of UUID:

3ba9b042-b8b8-4714-98e9-17915c2eeb95

`current_version` attribute

Current cluster version represented as string, for example “4.5.24” or even “4.6.0-0.ci.test-2020-12-01-123456-ci-xx-yyyyyyy”

`release` attribute

Available update version, for example “4.10.0-fc.4”.

`risk` attribute

Name of the risk for the conditional update, for example “AlibabaStorageDriverDemo”.

`collected_at` attribute

Timestamp of report represented with millisecond precision, for example “2021-02-08 00:22:19” (this is the display format, the precision is higher internally)

`archive_path` attribute

This attribute contains path to object stored in Ceph. It must be real path with chunks splitted by slash character. The format of path is:

archives/compressed/{PREFIX}/{CLUSTER_ID}/{YEAR+MONTH}/{DAY}/{ID}.tar.gz |

An example of path:

archives/compressed/0a/0aaaaaaa-bbbb-cccc-dddd-ffffffffffff/202102/08/003201.tar.gz |

Format of `Archives` table

cluster_id (byte array) cluster ID represented as UUID
collected_at (timestamp) timestamp of report represented with millisecond precision
archive_path (byte array) path to the object stored in Ceph

`cluster_id` attribute

An example of UUID:

3ba9b042-b8b8-4714-98e9-17915c2eeb95

`collected_at` attribute

Timestamp of report represented with millisecond precision, for example “2021-02-08 00:22:19” (this is the display format, the precision is higher internally)

`archive_path` attribute

This attribute contains path to object stored in Ceph. It must be real path with chunks splitted by slash character. The format of path is:

archives/compressed/{PREFIX}/{CLUSTER_ID}/{YEAR+MONTH}/{DAY}/{ID}.tar.gz |

An example of path:

archives/compressed/0a/0aaaaaaa-bbbb-cccc-dddd-ffffffffffff/202102/08/003201.tar.gz |

Possible enhancements

Version (positive integer) should be included in the generated Parquet files into separate column.
For strings with fixed (and known) length, fixed_len_byte_array type should be used to speedup Parquet file generation and consuming.
Using dictionaries for cluster_id might not be optimal - more study needed.

Examples

Content of `cluster_info` table

| cluster_id                           | cluster_version | platform | collected_at        | desired_version | initial_version | network_type | channel       | network_mtu | network_kind | network_size | network_host_prefix | archive_path                                                                        |
| ------------------------------------ | --------------- | -------- | ------------------- | --------------- | --------------- | ------------ | ------------- | ----------- | ------------ | ------------ | ------------------- | ----------------------------------------------------------------------------------- |
| 1e1ec4a5-1234-4f6c-838d-4e1c19300787 | 4.6.16          | None     | 2021-03-18 11:20:21 | 4.6.16          | 4.6.16          | OpenshiftSDN | stable-4.6    | 296         | service      | /23          | 0                   | archives/compressed/1e/1e1ec4a5-1234-4f6c-838d-4e1c19300787/202103/18/112021.tar.gz |
| 5e6538de-1234-4740-abbe-93afdef885a7 | 4.7.0           | None     | 2021-03-18 11:54:37 | 4.7.0           | 4.7.0           | Kurir        | candidate-4.7 | 1492        | service      | /12          | 0                   | archives/compressed/5e/5e6538de-1234-4740-abbe-93afdef885a7/202103/18/115437.tar.gz |
| e9ae142e-1234-4c49-a937-80f57a7abb52 | 4.6.16          | None     | 2021-03-18 11:04:22 | 4.6.16          | 4.6.16          | NCP          | fast-4.6      | 1500        | service      | /08          | 0                   | archives/compressed/e9/e9ae142e-1234-4c49-a937-80f57a7abb52/202103/18/110422.tar.gz |
| d8757ed6-1234-4860-8065-983f838e581e | 4.6.17          | None     | 2021-03-18 11:13:50 | 4.6.17          | 4.6.17          | OpenshiftSDN | stable-4.6    | 17914       | service      | /32          | 0                   | archives/compressed/d8/d8757ed6-1234-4860-8065-983f838e581e/202103/18/111350.tar.gz |
| 735c7c4d-1234-438f-a494-225f664c72a4 | 4.6.16          | VSphere  | 2021-03-18 11:38:46 | 4.6.16          | 4.3.28          | OpenshiftSDN | stable-4.3    | 65535       | cluster      | /16          | 23                  | archives/compressed/73/735c7c4d-1234-438f-a494-225f664c72a4/202103/18/113846.tar.gz |

NOTE

cluster_id is mocked, these clusters are not real.

Content of `rule_hits` table

+--------------------------------------|-------------------------------------------|---------------------|-------------------------------------------------------------------------------------+
| cluster_id                              | rule_id                                     | collected_at          | archive_path                                                                           |
| --------------------------------------- | ------------------------------------------- | --------------------- | -------------------------------------------------------------------------------------- |
| 00000000-0000-0000-0000-000000000000    | pods_check_containers                       | POD_CONTAINER_ISSUE   | 2021-02-08 00:59:34                                                                    | archives/compressed/00/00000000-0000-0000-0000-000000000000/202102/08/005934.tar.gz |
| 11111111-1111-1111-1111-111111111111    | operators_check                             | OPERATOR_ISSUE        | 2021-02-08 00:59:34                                                                    | archives/compressed/11/11111111-1111-1111-1111-111111111111/202102/08/005934.tar.gz |
| 22222222-2222-2222-2222-222222222222    | pods_check                                  | POD_ISSUE             | 2021-02-08 00:59:34                                                                    | archives/compressed/22/22222222-2222-2222-2222-222222222222/202102/08/005934.tar.gz |
| +-------------------------------------- | ------------------------------------------- | --------------------- | -------------------------------------------------------------------------------------+ |

NOTE

cluster_id is mocked, these clusters are not real.

Content of `failing_operator_conditions` table

+--------------------------------------|----------------|----------------|------------------|------------------------------------|---------------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------|------------------------|---------------------+
| cluster_id                              | operator         | condition_type   | condition_status   | reason                               | raw_message                                                                                   | archive_path                                                                          | last_transition_time     | collected_at           |
| --------------------------------------- | ---------------- | ---------------- | ------------------ | ------------------------------------ | --------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------- | ------------------------ |
| 12345678-acc8-4439-9393-c12140c20033    | authentication   | Progressing      | True               | _OAuthServerDeploymentNotReady       | Progressing: not all deployment replicas are ready                                            | archives/compressed/12/12345678-acc8-4439-9393-c12140c20033/202104/28/074816.tar.gz   | 2021-04-28 07:27:34      | 2021-04-28 07:48:16    |
| 12345678-acc8-4439-9393-c12140c20033    | dns              | Progressing      | True               | Reconciling                          | At least 1 DNS DaemonSet is progressing.                                                      | archives/compressed/12/12345678-acc8-4439-9393-c12140c20033/202104/28/074816.tar.gz   | 2021-04-28 07:28:16      | 2021-04-28 07:48:16    |
| 12345678-acc8-4439-9393-c12140c20033    | kube-apiserver   | Progressing      | True               | NodeInstaller                        | NodeInstallerProgressing: 3 nodes are at revision 47; 0 nodes have achieved new revision 55   | archives/compressed/12/12345678-acc8-4439-9393-c12140c20033/202104/28/074816.tar.gz   | 2021-04-28 07:27:54      | 2021-04-28 07:48:16    |
| +-------------------------------------- | ---------------- | ---------------- | ------------------ | ------------------------------------ | --------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------- | ------------------------ | ---------------------+ |

NOTE

cluster_id is mocked, these clusters are not real.

Content of `alerts` table

+--------------------------------------|------------------------------------|---------|------------|---------------------------|--------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------+
| cluster_id                              | name                                 | state     | severity     | namespace                   | job                        | labels                                                                                                                                                                                                                                                                                                                                                                                       | archive_path                                                                           |
| --------------------------------------- | ------------------------------------ | --------- | ------------ | --------------------------- | -------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------- |
| 00335bc8-1234-5678-add6-b6ba003bad09    | Watchdog                             | firing    | none         |                             |                            | {"prometheus":"openshift-monitoring/k8s","prometheus_replica":"prometheus-k8s-1"}                                                                                                                                                                                                                                                                                                            | archives/compressed/00/00335bc8-1234-5678-add6-b6ba003bad09/202106/05/122338.tar.gz    |
| 00335bc8-1234-5678-add6-b6ba003bad09    | UpdateAvailable                      | firing    | info         | openshift-cluster-version   | cluster-version-operator   | {"channel":"stable-4.7","endpoint":"metrics","instance":"10.20.30.128:9099","job":"cluster-version-operator","namespace":"openshift-cluster-version","pod":"cluster-version-operator-55f5795c8f-sz2v7","prometheus":"openshift-monitoring/k8s","prometheus_replica":"prometheus-k8s-1","service":"cluster-version-operator","upstream":"\u003cdefault\u003e"}                                | archives/compressed/00/00335bc8-1234-5678-add6-b6ba003bad09/202106/05/122338.tar.gz    |
| 73ea4458-1234-5678-8d1d-476c565fa5ab    | UpdateAvailable                      | firing    | info         | openshift-cluster-version   | cluster-version-operator   | {"channel":"stable-4.6","endpoint":"metrics","instance":"10.20.3.4:9099","job":"cluster-version-operator","namespace":"openshift-cluster-version","pod":"cluster-version-operator-fd6fbbd78-5m99k","prometheus":"openshift-monitoring/k8s","prometheus_replica":"prometheus-k8s-0","service":"cluster-version-operator","upstream":"https://api.openshift.com/api/upgrades_info/v1/graph"}   | archives/compressed/73/73ea4458-1234-5678-8d1d-476c565fa5ab/202106/05/122348.tar.gz    |
| 73ea4458-1234-5678-8d1d-476c565fa5ab    | Watchdog                             | firing    | none         |                             |                            | {"prometheus":"openshift-monitoring/k8s","prometheus_replica":"prometheus-k8s-0"}                                                                                                                                                                                                                                                                                                            | archives/compressed/73/73ea4458-1234-5678-8d1d-476c565fa5ab/202106/05/122348.tar.gz    |
| 73ea4458-1234-5678-8d1d-476c565fa5ab    | AlertmanagerReceiversNotConfigured   | firing    | warning      |                             |                            | {"prometheus":"openshift-monitoring/k8s","prometheus_replica":"prometheus-k8s-0"}                                                                                                                                                                                                                                                                                                            | archives/compressed/73/73ea4458-1234-5678-8d1d-476c565fa5ab/202106/05/122348.tar.gz    |
| +-------------------------------------- | ------------------------------------ | --------- | ------------ | --------------------------- | -------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------+ |

NOTE

cluster_id is mocked, these clusters are not real. namespace and job is duplicated in labels for backwards compatibility

Content of `gathering_info` table

+--------------------------------------|--------------------------------------------|------------------|-------------------------------------------------------------------------------------|---------------------+
| cluster_id                              | name                                         | duration_in_ms     | archive_path                                                                          | collected_at           |
| --------------------------------------- | -------------------------------------------- | ------------------ | ------------------------------------------------------------------------------------- | ---------------------- |
| 12345678-acc8-4439-9393-c12140c20033    | clusterconfig.GatherPodDisruptionBudgets     | 26                 | archives/compressed/12/12345678-acc8-4439-9393-c12140c20033/202108/02/144656.tar.gz   | 2021-08-02 14:46:56    |
| 12345678-acc8-4439-9393-c12140c20033    | clusterconfig.GatherSAPVsystemIptablesLogs   | 27                 | archives/compressed/12/12345678-acc8-4439-9393-c12140c20033/202108/02/144656.tar.gz   | 2021-08-02 14:46:56    |
| 12345678-acc8-4439-9393-c12140c20033    | clusterconfig.GatherClusterImageRegistry     | 28                 | archives/compressed/12/12345678-acc8-4439-9393-c12140c20033/202108/02/144656.tar.gz   | 2021-08-02 14:46:56    |
| 12345678-acc8-4439-9393-c12140c20033    | clusterconfig.GatherSAPDatahubs              | 29                 | archives/compressed/12/12345678-acc8-4439-9393-c12140c20033/202108/02/144656.tar.gz   | 2021-08-02 14:46:56    |
| 12345678-acc8-4439-9393-c12140c20033    | clusterconfig.GatherContainerRuntimeConfig   | 30                 | archives/compressed/12/12345678-acc8-4439-9393-c12140c20033/202108/02/144656.tar.gz   | 2021-08-02 14:46:56    |
| +-------------------------------------- | -------------------------------------------- | ------------------ | ------------------------------------------------------------------------------------- | ---------------------+ |

NOTE

cluster_id is mocked, these clusters are not real.

Content of `workload_containers` table

+--------------------------------------|--------------------------------------|------------------|-------------------------------------------------------------------------------------|-----------------|----------------|-------------------------------------------------------------------------|---------------+--------------|-------------------------------------------------------------------------------------|---------------------+
| cluster_id                              | shape_id                               | duration_in_ms     | namespace                                                                             | shape_instances   | restart_always   | image_id                                                                  | first_command                  | first_arg                                                                             | archive_path                                                                        | collected_at        |
| --------------------------------------- | -------------------------------------- | ------------------ | ------------------------------------------------------------------------------------- | ----------------- | ---------------- | ------------------------------------------------------------------------- | ------------------------------ | ------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------- | ------------------- |
| 12345678-acc8-4439-9393-c12140c20033    | 18bf5b07-e337-4b8f-960d-cd5744325f07   | 26                 | archives/compressed/12/12345678-acc8-4439-9393-c12140c20033/202108/02/144656.tar.gz   | 1                 | true             | sha256:3de851e078cc8142486b565e56e41d173bcd1353fa4cadda961d171cf8c716e7   | n9CdwzVF                       | RNOaw_AuQeIY                                                                          | archives/compressed/12/12345678-acc8-4439-9393-c12140c20033/202108/02/144656.tar.gz | 2021-08-02 14:46:56 |
| 12345678-acc8-4439-9393-c12140c20033    | e3f3a567-5217-4195-a431-f88099044a17   | 27                 | archives/compressed/12/12345678-acc8-4439-9393-c12140c20033/202108/02/144656.tar.gz   | 1                 | true             | sha256:3de851e078cc8142486b565e56e41d173bcd1353fa4cadda961d171cf8c716e7   | n9CdwzVF                       | zO0oxtw_mcI5                                                                          | archives/compressed/12/12345678-acc8-4439-9393-c12140c20033/202108/02/144656.tar.gz | 2021-08-02 14:46:56 |
| 12345678-acc8-4439-9393-c12140c20033    | e0979dbf-46ae-4a3b-bb9e-ea6233b89364   | 28                 | archives/compressed/12/12345678-acc8-4439-9393-c12140c20033/202108/02/144656.tar.gz   | 1                 | true             | sha256:3de851e078cc8142486b565e56e41d173bcd1353fa4cadda961d171cf8c716e7   |                                |                                                                                       | archives/compressed/12/12345678-acc8-4439-9393-c12140c20033/202108/02/144656.tar.gz | 2021-08-02 14:46:56 |
| 12345678-acc8-4439-9393-c12140c20033    | c83ad82d-847a-406a-a9a1-b6bd37f7cfae   | 27                 | archives/compressed/12/12345678-acc8-4439-9393-c12140c20033/202108/02/144656.tar.gz   | 1                 | true             | sha256:3de851e078cc8142486b565e56e41d173bcd1353fa4cadda961d171cf8c716e7   |                                | 2RFIFohzNoYG                                                                          | archives/compressed/12/12345678-acc8-4439-9393-c12140c20033/202108/02/144656.tar.gz | 2021-08-02 14:46:56 |
| 12345678-acc8-4439-9393-c12140c20033    | e0979dbf-46ae-4a3b-bb9e-ea6233b89364   | 29                 | archives/compressed/12/12345678-acc8-4439-9393-c12140c20033/202108/02/144656.tar.gz   | 1                 | true             | sha256:3de851e078cc8142486b565e56e41d173bcd1353fa4cadda961d171cf8c716e7   |                                |                                                                                       | archives/compressed/12/12345678-acc8-4439-9393-c12140c20033/202108/02/144656.tar.gz | 2021-08-02 14:46:56 |
| 12345678-acc8-4439-9393-c12140c20033    | e0979dbf-46ae-4a3b-bb9e-ea6233b89364   | 30                 | archives/compressed/12/12345678-acc8-4439-9393-c12140c20033/202108/02/144656.tar.gz   | 1                 | true             | sha256:3de851e078cc8142486b565e56e41d173bcd1353fa4cadda961d171cf8c716e7   |                                |                                                                                       | archives/compressed/12/12345678-acc8-4439-9393-c12140c20033/202108/02/144656.tar.gz | 2021-08-02 14:46:56 |
| +-------------------------------------- | -------------------------------------- | ------------------ | ------------------------------------------------------------------------------------- | ----------------- | ---------------- | ------------------------------------------------------------------------- | ---------------+-------------- | ------------------------------------------------------------------------------------- | ---------------------+                                                              |

NOTE

cluster_id is mocked, these clusters are not real.

Content of `workload_image_layers` table

+--------------------------------------|-------------------------------------------------------------------------|-------------------------------------------------------------------------|-------------------|---------------|--------------|-------------------------------------------------------------------------------------|---------------------+
| cluster_id                              | image_id                                                                  | layer_image_id                                                            | layer_image_level   | first_command   | first_arg      | archive_path                                                                          | collected_at           |
| --------------------------------------- | ------------------------------------------------------------------------- | ------------------------------------------------------------------------- | ------------------- | --------------- | -------------- | ------------------------------------------------------------------------------------- | ---------------------- |
| 12345678-acc8-4439-9393-c12140c20033    | sha256:4321f197c82174adf28ade6643db4be78d72548d60dc45fc457e9038cde2f64a   | sha256:4321f197c82174adf28ade6643db4be78d72548d60dc45fc457e9038cde2f64a   | 0                   | n9CdwzVF        | RNOaw_AuQeIY   | archives/compressed/12/12345678-acc8-4439-9393-c12140c20033/202108/02/144656.tar.gz   | 2021-08-02 14:46:56    |
| 12345678-acc8-4439-9393-c12140c20033    | sha256:4321f197c82174adf28ade6643db4be78d72548d60dc45fc457e9038cde2f64a   | sha256:4321f197c82174adf28ade6643db4be78d72548d60dc45fc457e9038cde2f64b   | 1                   | n9CdwzVF        | zO0oxtw_mcI5   | archives/compressed/12/12345678-acc8-4439-9393-c12140c20033/202108/02/144656.tar.gz   | 2021-08-02 14:46:56    |
| 12345678-acc8-4439-9393-c12140c20033    | sha256:4321f197c82174adf28ade6643db4be78d72548d60dc45fc457e9038cde2f64a   | sha256:4321f197c82174adf28ade6643db4be78d72548d60dc45fc457e9038cde2f64c   | 2                   |                 |                | archives/compressed/12/12345678-acc8-4439-9393-c12140c20033/202108/02/144656.tar.gz   | 2021-08-02 14:46:56    |
| 12345678-acc8-4439-9393-c12140c20033    | sha256:4f32c1da6fc39bd6e32e8f03a9b99e32b1143357b0e1da4c2d8fee0e09dcd5a1   | sha256:4321f197c82174adf28ade6643db4be78d72548d60dc45fc457e9038cde2f64d   | 3                   |                 | 2RFIFohzNoYG   | archives/compressed/12/12345678-acc8-4439-9393-c12140c20033/202108/02/144656.tar.gz   | 2021-08-02 14:46:56    |
| 12345678-acc8-4439-9393-c12140c20033    | sha256:4f32c1da6fc39bd6e32e8f03a9b99e32b1143357b0e1da4c2d8fee0e09dcd5a1   | sha256:4321f197c82174adf28ade6643db4be78d72548d60dc45fc457e9038cde2f64e   | 4                   |                 |                | archives/compressed/12/12345678-acc8-4439-9393-c12140c20033/202108/02/144656.tar.gz   | 2021-08-02 14:46:56    |
| 12345678-acc8-4439-9393-c12140c20033    | sha256:4f32c1da6fc39bd6e32e8f03a9b99e32b1143357b0e1da4c2d8fee0e09dcd5a1   | sha256:4321f197c82174adf28ade6643db4be78d72548d60dc45fc457e9038cde2f640   | 5                   |                 |                | archives/compressed/12/12345678-acc8-4439-9393-c12140c20033/202108/02/144656.tar.gz   | 2021-08-02 14:46:56    |
| +-------------------------------------- | ------------------------------------------------------------------------- | ------------------------------------------------------------------------- | ------------------- | --------------- | -------------- | ------------------------------------------------------------------------------------- | ---------------------+ |

NOTE

cluster_id is mocked, these clusters are not real.

Content of `Archives` table

cluster_id	collected_at	archive_path
2c4bde1c-30fa-4f09-aaaa-36bb54ce3217	Fri Feb 10 2023 15:35:55 GMT+0100 (Central European Standard Time)	archives/compressed/2c/2c4bde1c-30fa-4f09-aaaa-36bb54ce3217/202302/10/143555.tar.gz
b731eff1-a5db-4b43-aaaa-b708813a15aa	Fri Feb 10 2023 15:35:57 GMT+0100 (Central European Standard Time)	archives/compressed/b7/b731eff1-a5db-4b43-aaaa-b708813a15aa/202302/10/143557.tar.gz
736b03a3-4d2c-4c8f-aaaa-b007283d58bb	Fri Feb 10 2023 15:36:03 GMT+0100 (Central European Standard Time)	archives/compressed/73/736b03a3-4d2c-4c8f-aaaa-b007283d58bb/202302/10/143603.tar.gz

NOTE

cluster_id is mocked, these clusters are not real.

Format of output files generated by Parquet Factory

Schema version

Description

Basic format

Format of cluster_info table

cluster_id attribute

current_version attribute

platform attribute

collected_at attribute

desired_version attribute

initial_version attribute

network_type attribute

channel attribute

network_mtu attribute

network_kind attribute

network_size attribute

network_host_prefix attribute

archive_path attribute

Format of rule_hits table

cluster_id attribute

rule_id attribute

collected_at attribute

archive_path attribute

cluster_id attribute

ebs_account attribute

email_domain attribute

managed (bool) attribute

support attribute

collected_at attribute

Format of failing_operator_conditions table

cluster_id attribute

operator attribute

condition_type attribute

condition_status attribute

reason attribute

raw_message attribute

archive_path attribute

last_transition_time attribute

collected_at (timestamp) timestamp of report represented with millisecond precision

Format of alerts table

cluster_id attribute

name attribute

state attribute

severity attribute

namespace attribute

job attribute

labels attribute

archive_path attribute

Format of gatherers_info table

cluster_id attribute

name attribute

duration_in_ms attribute

archive_path attribute

collected_at attribute

Format of workload_containers table

cluster_id attribute

shape_id attribute

namespace attribute

shape_instances attribute

restart_always attribute

image_id attribute

first_command attribute

first_arg attribute

init_container attribute

archive_path attribute

collected_at attribute

Format of workload_image_layers table

cluster_id attribute

image_id attribute

layer_image_id attribute

layer_image_level attribute

first_command attribute

first_arg attribute

archive_path attribute

collected_at attribute

Format of available_updates table

cluster_id attribute

current_version attribute

release attribute

collected_at attribute

Format of `cluster_info` table

`cluster_id` attribute

`current_version` attribute

`platform` attribute

`collected_at` attribute

`desired_version` attribute

`initial_version` attribute

`network_type` attribute

`channel` attribute

`network_mtu` attribute

`network_kind` attribute

`network_size` attribute

`network_host_prefix` attribute

`archive_path` attribute

Format of `rule_hits` table

`cluster_id` attribute

`rule_id` attribute

`collected_at` attribute

`archive_path` attribute

`cluster_id` attribute

`ebs_account` attribute

`email_domain` attribute

`managed` (bool) attribute

`support` attribute

`collected_at` attribute

Format of `failing_operator_conditions` table

`cluster_id` attribute

`operator` attribute

`condition_type` attribute

`condition_status` attribute

`reason` attribute

`raw_message` attribute

`archive_path` attribute

`last_transition_time` attribute

`collected_at` (timestamp) timestamp of report represented with millisecond precision

Format of `alerts` table

`cluster_id` attribute

`name` attribute

`state` attribute

`severity` attribute

`namespace` attribute

`job` attribute

`labels` attribute

`archive_path` attribute

Format of `gatherers_info` table

`cluster_id` attribute

`name` attribute

`duration_in_ms` attribute

`archive_path` attribute

`collected_at` attribute

Format of `workload_containers` table

`cluster_id` attribute

`shape_id` attribute

`namespace` attribute

`shape_instances` attribute

`restart_always` attribute

`image_id` attribute

`first_command` attribute

`first_arg` attribute

`init_container` attribute

`archive_path` attribute

`collected_at` attribute

Format of `workload_image_layers` table

`cluster_id` attribute

`image_id` attribute

`layer_image_id` attribute

`layer_image_level` attribute

`first_command` attribute

`first_arg` attribute

`archive_path` attribute

`collected_at` attribute

Format of `available_updates` table

`cluster_id` attribute

`current_version` attribute

`release` attribute

`collected_at` attribute

`archive_path` attribute

Format of `conditional_update_conditions` table

`cluster_id` attribute

`current_version` attribute