Assigning Multiple Gateways to Server with three NICs

So the problem doesn't seem to be THAT common apparently... But, none the less we have several servers all with three NIC cards (intentionally) and we're running Ubuntu 14.04 on each server.

Each NIC card is meant to connect to separate networks (management, tunnel and storage) and I want these separated vs lumped into one huge netmask because security is real  :)

The problem that we ran into here is that Linux while technically supports multiple gateways, you can't actually specify them in /etc/network/interfaces.

Let me say that again. YOU CANNOT SPECIFY MORE THAN ONE "GATEWAY" in your configs.

So for further explanation, you CANNOT do this:

auto eth0
iface eth0 inet static
       address 10.0.0.52
       netmask 255.255.255.0
       gateway 10.0.0.1

iface eth1 inet static
       address 10.0.1.52
       netmask 255.255.255.0
       gateway 10.0.1.1


This configuration WILL FAIL.  The gateway configuration is NOT unique to each interface, and can ONLY exist in this config file ONE TIME...  So, I know what you're thinking, how do I do this then?

Fortunately after many hours of toiling and writhing in pain, I came across the proper solution. In linux you will have to setup static routes to tell linux that the route for NIC eth1 and eth2 use a different next-hop.  The command that you can use to test is:

ip route add <nic IP> via <vLAN gateway IP> dev <interface name>

So you wind up with a final command like:

ip route add 10.0.1.31 via 10.0.1.1 dev eth1

But wait there's more!  After the server reboots this route will disappear so you need to add this to your configuration as a "post-up" command which essentially all post-up commands are commands that are run (you guessed it) after the interface signals it's "up".

Final /etc/network/interfaces config:

auto lo
iface lo inet loopbackauto eth0
iface eth0 inet static
       address 10.0.0.52
       netmask 255.255.255.0
       gateway 10.0.0.1
auto eth1
iface eth1 inet static
       address 10.0.1.52
       netmask 255.255.255.0
       post-up route add 10.0.1.52 via 10.0.1.1 dev eth1
auto eth2
iface eth2 inet static
       address 10.0.2.52
       netmask 255.255.255.0
       post-up route add 10.0.2.52 via 10.0.2.1 dev eth2

Lastly, you need to down and then up the NIC cards

# sudo reboot -n