All data ingestion is restored and up and running again. A postmortem of this incident is available upon request.
Posted Aug 30, 2023 - 07:26 UTC
Monitoring
Another internal service was suffering from this and wasn't ingesting data. This was resolved by restarting the service.
Posted Aug 30, 2023 - 07:20 UTC
Identified
Old certificate still exists in the cluster. We removed the old certificate after the ingress service uses the correct certificate (which is renewed automatically)
Posted Aug 30, 2023 - 03:20 UTC
Investigating
The certificate of all ingress HTTP traffic has expired.
Posted Aug 29, 2023 - 19:12 UTC
This incident affected: Ingress traffic (Lora ingress), Digital twin (Digital twin search, Digital twin API), UI (Webapp), and Infrastructure (Infrastructure | resources).