UPDATE: Recent Email Performance Issues

On Friday, July 8, ITS retired its legacy email routing system. This change should improve email delivery times to USC email accounts.

If you continue to experience email delivery problems, please contact the ITS Customer Support Center at 213-740-5555 to report the issue. 

Recent Email Performance Issues (Posted May 19)

Information Technology Services (ITS) has experienced recent email delivery issues that have resulted in significant delays, along with a number deleted email messages, sent between April 22 and May 12, 2016.

Delays in the delivery of email are not uncommon and may result from issues in both local systems and the systems of cloud service providers. ITS monitors email delays on a regular basis and notifies the USC community of email delays that are longer than normal.

The recent delay of USC email was caused by a complicated convergence of local and cloud-based issues, which are outlined below. Ultimately, the delay was caused by a backlog in ITS’s cloud-based antispam system, which did not correctly release messages from its email queue.

Resolving the email delivery issues is a top priority for ITS. In response to the recent delivery issues, ITS is expediting plans to retire our legacy email routing system.

If you continue to experience email delivery problems, please contact the ITS Customer Support Center at 213-740-5555 to report the issue. 

Chronology and Details of Email Delivery Issues Occurring between April 22 and May 16

  • Postfix, the university’s email routing system and a component of the legacy email infrastructure, has been experiencing degraded performance and deferred delivery of some email messages.
  • On May 5, Microsoft, our email provider, experienced a software bug that incorrectly labeled some legitimate email messages as suspicious and returned them to the queue of Proofpoint, our cloud-based antispam system. Microsoft corrected this bug on May 6.
  • Through a separate software bug, introduced by a vendor upgrade, Proofpoint did not attempt redelivery of the deferred email that was returned by Microsoft or Postfix. As a result of this bug, a large backlog of email was generated in the Proofpoint queue.
  • On May 9, at approximately 5 p.m., ITS released about 50,000 messages, which had been held in the Proofpoint queue for different periods of time and some of which may have been delayed by up to 7 days.
  • On May 10, Proofpoint applied an emergency fix to resolve its software bug. At that time, ITS believed the email delivery issue to be resolved.
  • In investigating feedback from some account holders about continuing email delivery issues, ITS discovered an error in a script that an ITS employee had created to remove undeliverable spam messages from the Postfix queue.  The scripting error led to some legitimate sent email being deleted from the antispam queue.
  • On May 16, ITS emailed all USC account holders whose sent messages had been erroneously deleted by the system. The ITS message contained the date, approximate time, and approximate size of the deleted messages.