Publish: first blog outage

content/posts/blogging/be-kind-choose-violence.md

@@ -1,6 +1,6 @@
 ---
 title: "Be Kind - Choose Violence"
-date: 2023-12-18T11:30:34-07:00
+date: 2023-12-18T05:30:00-07:00
 tags: ["Blogging", "Experiment"]
 series: "blogging"
 draft: false
@@ -36,7 +36,7 @@ Often we become beholden to practices that do nothing but slow progress and inno
 
 Repeat - and when you hit those blockers - IE code review or some other time commitment - find out how to next best pivot and continue down a new thread.
 
-Code stuck in review? Ping - be direct. No other options available? Execute on that next task with what you know now. Execute on next highest priority for your down. Know what it is you want to accomplish each day and execute. 
+Code stuck in review? Ping - be direct. No other options available? Execute on that next task with what you know now. Execute on next highest priority. Know what it is you want to accomplish each day and execute. 
 
 Finally instill this proactive behavior in others. Surround yourself with those who are Mission-Focused and you'll find that the progress you can have in a day and the ways you can challenge each-other are mind-boggling.  
 
@@ -45,7 +45,9 @@ Finally instill this proactive behavior in others. Surround yourself with those
 > ‘Be polite, be professional, but have a plan to kill everybody you meet.’
 - General Mattis
 
-Treat others with respect - Assume positive intent - Execute/Execute/Execute
+Treat others with respect - Remove your ego - Assume positive intent - Execute/Execute/Execute
+
+It's not one or the other - rather it is a pursuit of making myself better each day by engaging meaningfully with others and executing on the things I want to accomplish.
 
 ## Nerd Notes - New domain whodis
 

content/posts/homelab/outage-report-23.md

@@ -0,0 +1,42 @@
+---
+title: "First Blog Outage - Post Mortem"
+date: 2023-12-21T05:30:00-07:00
+tags: ["Homelab", "Experiment"]
+series: "homelab"
+draft: false
+---
+
+![](/images/coauthor.png)
+(Picture of my co-author and I writing this post)
+
+First off - this is meant to be a very light-hearted post. Something that I hope to impress on others is that Homelabs are a great opportunity to learn and grow your skills in new and relevant ways - but they come at the cost of being a labor of love. Something you maintain entirely in your free time and is best-effort to maintain and keep healthy.
+
+## The problem
+
+Shout out to Gavin for tossing me a message that something was wrong - I have no alerting on this setup and that should be a goal for future Brandt. 
+
+Nonetheless - attempting to hit my domain left me with a gateway error.
+
+## Troubleshoot 
+
+The error was obvious very quickly - as noted in the original blog post - the infrastructure that supports this blog is 2x Raspberry Pi 4's running K3s in a Highly Available 2x Server w/ External Datastore configuration.
+
+I switched my kubecontext to that cluster and was getting a connection error over Kubectl - so the problem looked to be with the cluster. SSH'd into one of the nodes and looked at `journalctl` - error was related to the external datastore/k3s having a problem with the certificates. 
+
+## Certificates
+
+I run my "production" homelab cluster on separate infrastructure and use the crunchydata pg-operator to deploy postgres clusters - this is what I use for the blog k3s cluster datastore. I currently have it configured with cert-manager and a self-signed issuer.
+
+Problem here is that I didn't modify some of my other letsencrypt certificate templates and the expiration was set for 90 days - well folks - we've been blogging for about that long now.
+
+So Cert-manager rolled my self-signed certificates which the postgres cluster was updated to use and then all hell broke loose.
+
+## Recovery 
+
+Recovery was actually pretty easy - bring the new ca/cert/key over to each node and restart the k3s service.... DONE!
+
+Everything came back online and healthy!
+
+## Next Steps
+
+Still a work in progress - winter vacation has a limited amount of coding time available - Long term plan may be to re-work my postgres self-signed certificates to be more static - otherwise move my blog to be orchestrated on the "production" homelab cluster.