multiple ips gateway conflict on ubuntu

[ryanrussell]

Hi,

Booting with one ip seems to work fine, but when I attempt to add a second IP I end up with an invalid /etc/network/interfaces file.

Background:

• I have a template that sometimes we just boot with a private IP, sometimes we boot with a private and public IP.
• I have two NIC's in the template, eth0 mapped to private, eth1 mapped to public.

I may missing something with multiple IPs on the routing config.

Here's a sample command:
ezmomi clone --template ubuntu-14.04-raw --hostname machine-name --cpus 2 --mem 2 --resource-pool 'pool' --ips 192.168.185.182 209.41.67.131

Relevant network config:
https://gist.github.com/ryanrussell/fb3065d8d6bae6f549c4

This is what I get for a /etc/network/interfaces file:
https://gist.github.com/ryanrussell/8afec84d64c33cb80f3e

I'm trying to automate a solution such as this:
https://www.thomas-krenn.com/en/wiki/Two_Default_Gateways_on_One_System#Solution

Here's a sample functional /etc/network/interfaces file:
https://gist.github.com/ryanrussell/c51688efee1a98e346a1

I can do this with chef post-boot if necessary, but everything about it feels hacky and wrong.

Is there a better way that I'm missing? Or a way to get a valid network interfaces file without manual intervention?

Thanks

[snobear]

Hi @ryanrussell -

I'm not super familiar with the Debian /etc/network/interfaces style config.

Couple questions...What happens when the new VM boots after provisioning? Any specific errors in the boot logs? Do eth0 and/or eth1 appear down after boot? Or are they appear up and its the routing thats screwy?

[ryanrussell]

Thanks for responding.. Here's a summary that's better than I could say it:

https://www.thomas-krenn.com/en/wiki/Two_Default_Gateways_on_One_System
You have built two or more network cards into one Linux system and each of these cards has its own default gateway. By default, you can only have one default gateway on a system. The case described would lead to asynchronous routing, whereby the router would reject the packets as appropriate.

On ubuntu, we can only have one default gateway. You have to build policy routes for additional nic traffic to be handled correctly.

There are no visual errors in the log per se, but you will experience seemingly random inabilities to route traffic correctly based on which default gateway gets used(especially when one is private ip space and doesn't publicly resolve). It's a bear to debug from the app layer down to network interfaces.

Happy to elaborate or talk more if it's helpful.