-
Notifications
You must be signed in to change notification settings - Fork 0
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
- Loading branch information
Showing
5 changed files
with
201 additions
and
14 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,34 @@ | ||
| apiVersion: "v1" | ||
| kind: "ConfigMap" | ||
| metadata: | ||
| name: "coredns" | ||
| namespace: "kube-system" | ||
| data: | ||
| Corefile: | | ||
| .:53 { | ||
| errors | ||
| health | ||
| ready | ||
| kubernetes cluster.local in-addr.arpa ip6.arpa { | ||
| pods insecure | ||
| fallthrough in-addr.arpa ip6.arpa | ||
| } | ||
| k8s_external k8s.askov.net | ||
| hosts /etc/coredns/NodeHosts { | ||
| ttl 60 | ||
| reload 15s | ||
| fallthrough | ||
| } | ||
| prometheus :9153 | ||
| forward . /etc/resolv.conf | ||
| cache 30 | ||
| loop | ||
| reload | ||
| loadbalance | ||
| import /etc/coredns/custom/*.override | ||
| } | ||
| import /etc/coredns/custom/*.server | ||
| NodeHosts: | | ||
| 192.168.4.4 k8smaster1.k8s.askov.net | ||
| 192.168.4.10 k8snode1.k8s.askov.net | ||
| 192.168.4.20 k8snode2.k8s.askov.net |
10 changes: 6 additions & 4 deletions
10
MetalLB/exposed_dns.yaml → CoreDNS/external_dns_service.yaml
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,16 +1,167 @@ | ||
| DNS server for LoadBalancer IPs | ||
| ====================================== | ||
|
|
||
| (Inspired by https://www.reddit.com/r/homelab/comments/ipsc4r/howto_k8s_metallb_and_external_dns_access_for/?rdt=40382) | ||
|
|
||
| Proposed setup: | ||
| This is both extremely difficult and surprisingly easy. The trick is to wrap you head around the weirdness of DNS. | ||
|
|
||
| 1. CoreDNS in Kubernetes | ||
| * It hosts a DNS server, and is updated as kubernetes change state | ||
| * MetalLB LoadBalancer external IP (192.168.32.x/24) for CoreDNS | ||
| 2. Domain Override in PFSense Dns Forwarder | ||
| * https://docs.netgate.com/pfsense/en/latest/services/dns/forwarder-overrides.html#domain-overrides | ||
| * Forward all *.k8s.askov.net requests to CoreDNS in Kubernetes | ||
| * CoreDNS will respond with the LoadBalancer IP of the service | ||
| The PFSense firewall is the normal DNS server for my LAN. I want it to resolve DNS requests like this | ||
| 1. Do I have a host override for the specific host? -> Use Host Override | ||
| 2. Do I have the specific host in DHCP? -> Use DHCP lease list | ||
| 3. Do I have a domain override for the specific domain? -> Forward to DNS server for this domain | ||
| 4. Forward to upstream (ISP) DNS servers | ||
|
|
||
| To get Kubernetes DNS working, we need to hook into step 3 in that list. To do this, we need to setup these 2 things: | ||
| 1. Set up PFSense DNS Domain override to point to the Kubernetes DNS server | ||
| 2. Set up CoreDNS in Kubernetes to respond to external requests | ||
|
|
||
| https://www.reddit.com/r/homelab/comments/ipsc4r/howto_k8s_metallb_and_external_dns_access_for/?rdt=40382 | ||
| Step 1 is easy. | ||
|  | ||
| Read more about it at https://docs.netgate.com/pfsense/en/latest/services/dns/forwarder-overrides.html#domain-overrides | ||
|
|
||
| Since Kubernetes already comes with [CoreDNS](https://coredns.io/), all we need to do is expose it as a service and enable the [k8s_external](https://coredns.io/plugins/k8s_external/) plugin | ||
|
|
||
| Exposing coredns as a service via the MetalLB loadbalancer just requires us to apply | ||
|
|
||
| ```yaml | ||
| apiVersion: v1 | ||
| kind: Service | ||
| metadata: | ||
| name: ext-dns | ||
| namespace: kube-system | ||
| annotations: | ||
| metallb.universe.tf/allow-shared-ip: "DNS" | ||
| metallb.universe.tf/loadBalancerIPs: 192.168.32.1 | ||
| spec: | ||
| type: LoadBalancer | ||
| ports: | ||
| - name: dns | ||
| port: 53 | ||
| targetPort: 53 | ||
| protocol: UDP | ||
| - name: dns-tcp | ||
| port: 53 | ||
| targetPort: 53 | ||
| protocol: TCP | ||
| selector: | ||
| k8s-app: kube-dns | ||
| ``` | ||
| Then we must reconfigure CoreDNS to use the `k8s_external` plugin. | ||
| In fact, all we need to do is add the line `k8s_external k8s.askov.net` to the config map `coredns` | ||
| (TODO this change might be lost on restart?) | ||
|
|
||
| The full configmap will then be | ||
| ```yaml | ||
| apiVersion: "v1" | ||
| kind: "ConfigMap" | ||
| metadata: | ||
| name: "coredns" | ||
| namespace: "kube-system" | ||
| data: | ||
| Corefile: | | ||
| .:53 { | ||
| errors | ||
| health | ||
| ready | ||
| kubernetes cluster.local in-addr.arpa ip6.arpa { | ||
| pods insecure | ||
| fallthrough in-addr.arpa ip6.arpa | ||
| } | ||
| k8s_external k8s.askov.net { | ||
| fallthrough | ||
| } | ||
| hosts /etc/coredns/NodeHosts { | ||
| ttl 60 | ||
| reload 15s | ||
| fallthrough | ||
| } | ||
| prometheus :9153 | ||
| forward . /etc/resolv.conf | ||
| cache 30 | ||
| loop | ||
| reload | ||
| loadbalance | ||
| import /etc/coredns/custom/*.override | ||
| } | ||
| import /etc/coredns/custom/*.server | ||
| NodeHosts: | | ||
| 192.168.4.4 k8smaster1.k8s.askov.net | ||
| 192.168.4.10 k8snode1.k8s.askov.net | ||
| 192.168.4.20 k8snode2.k8s.askov.net | ||
| ``` | ||
|
|
||
| The line `k8s_external k8s.askov.net` instructs CoreDNS to load the `k8s_external` plugin and become a DNS server for the given domain (`k8s.askov.net`), and thus respond to requests with the given Load Balancer IP. | ||
|
|
||
| With this change, we can immediately see that we can resolve the `test` service as both `test.default.svc.cluster.local` and `test.default.k8s.askov.net` | ||
|
|
||
| (note: You might have to restart the coredns pods (just delete them)) | ||
|
|
||
| ``` | ||
| [aabl@k8smaster1 ~]$ nslookup test.default.svc.cluster.local 192.168.32.1 | ||
| Server: 192.168.32.1 | ||
| Address: 192.168.32.1#53 | ||
| Name: test.default.svc.cluster.local | ||
| Address: 10.43.129.111 | ||
| [aabl@k8smaster1 ~]$ | ||
| [aabl@k8smaster1 ~]$ nslookup test.default.k8s.askov.net 192.168.32.1 | ||
| Server: 192.168.32.1 | ||
| Address: 192.168.32.1#53 | ||
| Name: test.default.k8s.askov.net | ||
| Address: 192.168.32.2 | ||
| [aabl@k8smaster1 ~]$ | ||
| ``` | ||
|
|
||
| When performing the lookup from my local workstation it also works | ||
| (here 192.168.2.1 is the pfsense firewall ip) | ||
| ``` | ||
| aabl@fedora:~$ nslookup test.default.k8s.askov.net 192.168.2.1 | ||
| Server: 192.168.2.1 | ||
| Address: 192.168.2.1#53 | ||
|
|
||
| Name: test.default.k8s.askov.net | ||
| Address: 192.168.32.2 | ||
|
|
||
| aabl@fedora:~$ | ||
| aabl@fedora:~$ nslookup test.default.k8s.askov.net 192.168.32.1 | ||
| Server: 192.168.32.1 | ||
| Address: 192.168.32.1#53 | ||
|
|
||
| Name: test.default.k8s.askov.net | ||
| Address: 192.168.32.2 | ||
|
|
||
| aabl@fedora:~$ | ||
| ``` | ||
| And we can thus access the service by it's DNS name | ||
| ``` | ||
| aabl@fedora:~$ curl test.default.k8s.askov.net | ||
| <!DOCTYPE html> | ||
| <html> | ||
| <head> | ||
| <title>Welcome to nginx!</title> | ||
| <style> | ||
| html { color-scheme: light dark; } | ||
| body { width: 35em; margin: 0 auto; | ||
| font-family: Tahoma, Verdana, Arial, sans-serif; } | ||
| </style> | ||
| </head> | ||
| <body> | ||
| <h1>Welcome to nginx!</h1> | ||
| <p>If you see this page, the nginx web server is successfully installed and | ||
| working. Further configuration is required.</p> | ||
|
|
||
| <p>For online documentation and support please refer to | ||
| <a href="http://nginx.org/">nginx.org</a>.<br/> | ||
| Commercial support is available at | ||
| <a href="http://nginx.com/">nginx.com</a>.</p> | ||
|
|
||
| <p><em>Thank you for using nginx.</em></p> | ||
| </body> | ||
| </html> | ||
| aabl@fedora:~$ | ||
| ``` |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters