Resolved: call centers on both campuses were down

Incident Discovery Time: 07:44am on 11/01/2024 Time of Resolution: 02:30pm on 11/01/2024 Services Impacted:

Description of Impact

Bomgar was down, clients were unable to change the forwarding on their phones, and the following call centers were down:
CRC Service desk
BUMC IT Service Desk
CES (admissions, financial aid, registrar)
Dental CRC
Goldman Dental
Student Accounts
SHS
SHS Behavioral Medicine
SHS Insurance
University Service Center
HR
Student Employment

The main switchboard 3-2000 was also not accepting calls.

Incident Description and Resolution

IS&T teams have resolved the incident.

Additional Information

After troubleshooting with Cisco, a database replication issue was identified, with the node being unresponsive. Database replication services were restarted on that node, and database mismatch issues were found. A rebuild of the database was completed around 2:00pm and services were available again. If you continue to have issues, please contact the IT Help Center.

Previous Update

Incident Discovery Time: 07:44am on 11/01/2024 Services Impacted: Identity & Access Management (IAM)

Description of Impact

Call centers around campus are still down. It has also come to our attention that many clients are unable to change the forwarding on their phones at this time, as well.

Current Status

We are continuing to work on this issue and do not have an ETA for restoration at this time. Next Update: 03:00pm

Previous Update

Incident Discovery Time: 07:44am on 11/01/2024 Services Impacted: Identity & Access Management (IAM)

Description of Impact

Call centers around campus are down. These include:
CRC Service desk, (You can now reach the IT Help Center from a non-BU phone, or you can email ithelp@bu.edu)
BUMC IT Service Desk, (use:bumchelp@bu.edu)
CES (admissions, financial aid, registrar)
Dental CRC
Goldman Dental
Student Accounts, (use: studenta@bu.edu)
SHS, contact form
SHS Behavioral Medicine
SHS Insurance
University Service Center, contact us
HR
Student Employment

The main switchboard 3-2000 is also not accepting calls.

Current Status

IS&T teams continue to work with the vendor to solve this issue. Next Update: 01:30pm

Previous Update

Incident Discovery Time: 07:44am on 11/01/2024 Services Impacted: Identity & Access Management (IAM)

Description of Impact

Call centers around campus are down. These include:
CRC Service desk, (use: ithelp@bu.edu)
BUMC IT Service Desk, (use:bumchelp@bu.edu)
CES (admissions, financial aid, registrar)
Dental CRC
Goldman Dental
Student Accounts, (use: studenta@bu.edu)
SHS, contact form
SHS Behavioral Medicine
SHS Insurance
University Service Center, contact us
HR
Student Employment

The main switchboard 3-2000 is also not accepting calls.

Current Status

The IS&T team backed out a change that was made on Wednesday, 10/30, and are currently working with Cisco to investigate further. Next Update: 12:15pm

Previous Update

Incident Discovery Time: 07:44am on 11/01/2024 Services Impacted: Infrastructure

Description of Impact

The IT Help Center phones and other call centers around campus such as the Dental School are down. Bomgar is also down.

Current Status

IS&T teams believe this might be certificate related and are trying to remediate the situation now.

Additional Information

Please continue to email ithelp@bu.edu instead of calling the IT Help Center until this situation resolves. Next Update: 11:00am

Previous Update

Incident Discovery Time: 07:44am on 11/01/2024 Services Impacted: Infrastructure

Description of Impact

Cisco Finesse (the soft phone system that the IT Help Center uses) and Bomgar are two services that are reported unavailable this morning.

Current Status

IS&T teams have not yet identified the cause of the incident, but are investigating. Please email ithelp@bu.edu with any IT issues. Next Update: 10:00am