[Front page]

Insights Data Schemas

External data pipeline

Whole data flow

1. Event about new data from insights operator is consumed from Kafka. That event contains (among other things) URL to S3 Bucket
2. Insights operator data is read from S3 Bucket and Insights rules are applied to that data
3. Results (basically organization ID + account number + cluster name + insights results JSON) are stored back into Kafka, but into different topic
4. That results are consumed by Insights rules aggregator service that caches them
5. The service provides such data via REST API to other tools, like OpenShift Cluster Manager web UI, OpenShift console, etc.

Architecture diagram

Data format descriptions

1. Incoming messages in platform.upload.announce
2. Raw data stored in S3 bucket
3. Data produced by OCP rules engine
4. Messages consumed from ccx.ocp.results topic
5. OCP results written into RDS
6. Rule content produced by Content Service
7. Rule content consumed by Content Service