Data flow

The “end-to-end” data flow is described there (including Notification Writer service part):

  1. A customer cluster with Insights Operator installed sends new data containing info about the cluster into Ingress service
  2. The Ingress service consumes the data, writes them into an S3 Bucket, and produces a new message into a Kafka topic named platform.upload.announce.
  3. The CCX Data pipeline service consumes the message from the platform.upload.announce Kafka topic.
  4. That message represents an event that contains (among other things) an URL to S3 Bucket.
  5. Insights operator data is read from S3 Bucket and insights rules are applied to that data in ccx-data-pipeline service.
  6. Results (basically organization ID + cluster name + insights results JSON) are stored back into Kafka, but into different topic named ccx.ocp.results.
  7. That results are consumed by ccx-notification-writer service.
  8. ccx-notification-writer service stores insights results into AWS RDS database into new_reports table.
  9. Content of that table is consumed by ccx-notification-service periodically.
  10. Newest results from new_reports table is compared with results stored in reported table. The records used for the comparison depend on the configured cooldown time.
  11. If changes (new issues) has been found, notification message is sent into Kafka topic named platform.notifications.ingress. The expected format of the message can be found here.
  12. New issues is also sent to Service Log via REST API. To use the Service Log API, the ccx-notification-service uses the credentials stored in vault.
  13. The newest result is stored into reported table to be used in the next ccx-notification-service iteration.

Remarks

  1. Steps 1 to 5 are shared with the CCX Data pipeline
  2. Steps 7 and 8 are performed by ccx-notification-writer service.
  3. Steps 9 to 12 are performed by ccx-notification-service service.

Sequence diagram

sequence_diagram.png

Sequence diagram, full scale