Postmortem - At some point on the evening of March 10, we noticed that the apartment server had downgraded to a 10/100 link speed with the network switch its connected to. After a reboot which appeared to fix the issue and restore gigabit link speeds, we noticed GitLab was unaccessible at about 1:55 AM.
Following a networking configuration which configured a new bridge, changing DHCP reservations, changing the port the server was connected on, and replacing the physical cable, services were restored. We will continue to monitor if the server can hold a gigabit link speed.
Resolved (2:49 AM EDT) - GitLab is now accessible again, along with the OctoCam Backend. OctoCam has uploaded any backlogged images.
Resolving (2:45 AM EDT) - Working towards a solution to restore networking. The GitLab Server should be up within the next 5-10 minutes.
Identified (2:25 AM EDT) - There appears to be some networking issue with the apartment server - an IPv6 address is getting passed through to the GitLab server, but IPv4 is not getting passed through. Continuing to work towards a solution. Sporadic outages may occur on OctoCam Backend.
Investigating (1:55 AM EDT) - Investigating an outage with the GitLab Server at the moment.
Last updated: March 11, 2024 at 6:59 AM UTC