← Go back to owenthe.dev Status

InfluxDB causing degraded performance on multiple services

November 13, 2023 at 4:30 AM UTC

GitLab Server OctoCam Backend OctoCam Frontend OctoCam Pi-side

Resolved after 12h 30m November 13, 2023 at 5:00 PM UTC

Resolved - This issue has been resolved.

Monitoring - We have resolved an issue with an InfluxDB instance filling up the disk on an apartment server, in turn, causing issues for both the Nginx reverse proxy and OctoCam upload delays that started at about 11:00 PM EST on November 12.

During this outage, degraded performance was visible on these systems:

  • GitLab: Generally slow loading through the Nginx reverse proxy
  • Grafana: Extremely slow response times
  • OctoCam Frontend: Slower than normal image load times
  • OctoCam Backend: Upload image endpoint returning errors
  • OctoCam Pi-side: Increased image delays - up to 3 minutes

The bucket causing this issue has been deleted, and all systems are being monitored to ensure this issue has been fully resolved.

Last updated: November 21, 2023 at 7:28 PM UTC