On the 17th of June 2005, the IT Systems Team performed a server upgrade, replacing the old NFS server with a new one. Traditionally, the NFS server also plays the role of final-stage email delivery. The migration was performed between 7pm and 8pm, and there were (initially) no outstanding issues.
The old NFS server had a network interface with a public IP, and thus made all email deliveries from this (single) interface. While this had served us well for over a year, the IT Systems Team decided to change this design to attain a multi-point delivery system. BII is serviced by 3 ISPs and it made sense that we be able to explicitly choose which link to use. With a total of 9 application servers spread over 3 subnets, facing the Internet, we decided to place the mail delivery behind these app servers, such that any one of these app server could be the email delivery point. Thus, the new NFS server was deployed without any public IP. By controlling the default gateway, we could select any app server to NAT traffic onto the Internet. This meant that altering the email exit path would simply be a matter of changing the default gateway.
Over the weekend, no problems were encountered however the first reports of mail delivery failures came in on monday (20th June 2005). Initially only emails headed for a-star.edu.sg failed, however it later became clear that all email to the government domains run by the GEMS mail server were not being delivered. We soon discovered that it wasn't an issue of IP connectivity but rather of the configuration of the GEMS server itself. Quite some months ago, there was an inquiry on what IP address BII's mail server had. The purpose of this was to perform anti-spam configuration on GEMS. Since our email then had exactly one and only one exit point, we provided the single IP address.
Unfortunately, with our new design, our email exit point is no longer limited to a single IP address. In addition, since the old NFS server was decommissioned, its IP address was no longer in use. For this reason, all emails delivered to GEMS were rejected (treated as spam). Upon realizing this, we immediately reported our 3 IP ranges to our Network Manager, who then got in touch with the relevant individuals in charge of GEMS.
I was informed at about 9am on the 21st of June that we would need to furnish a report on the change of IP address(es) for the new mail server. The report would have to include an explaination on the security and authentication measures put in place to prevent misuse of our mail server. Fearing that this would be a lengthy process, we decided to make a (unplanned) configuration change such that all our outgoing emails will be delivered from the same IP source we previously submitted to GEMS. By 12 noon, after doing some cabling and config changes to the new server, emails were now being delivered from the original IP address, now used by the new server.
We now intend to submit the relevant documentation to A*Star, such that our outgoing emails can once again be dynamically routed as according to our intended architecture.