ARGH-gust - State of Argh-dress Edition

Well, balls; I realised the other day that well my blog is supposed to document my struggles within my homelab, and how I'm going about doing things. Yet, for 6+ months my blog has focused on the progress with the tape library, which is part of the homelab, but is not the main focus of the thing. Thus, I either need to cut back on tape library progress, or increase the number of posts I'm doing. Likely I'll cut back on the library stuff, since I think I'm finally stable on it. 

In other news, I've been experimenting with AI, mostly having it write up scripts and things that seem tedious to me. I say "experimenting" because by the time that AI was introduced to my job as a requirement, it conflicted with the previous requirement of "No new development". In short I got pigeon-holed on a dead-end project, then got told "go use AI". Suffice it to say, I didn't use AI. So I've had to learn at home, with no guidance, and made up projects. Several of which are lame :) .

My first project started fairly simple. Given a list of directories containing photos from me and my partners cell phones, organise the photos into albums/folders, and give them useful names. 

That's where the simplicity ended, but not in the way I expected. I expected that grouping the photos would be the hard part, since computers can't "see" photos, it can't determine if a photo is related to another photo, like multiple pictures in a city garden. Turns out that the AI side stepped that by using the embed GPS coordinates and setting a time limit. Anything within 90 minutes*, and 0.5 km* are considered a group. Using that and the embedded created date time, the script gets to a good group with a leading date time as its name. E.G. 2026_06_01_Album_Name. 

The album name turned out to be the hard part. I had thought that I could just use a map API and get a location name, like, "oh, you're in the park, let's use the park name". In short, it's not that easy, the map API I'm using (OpenStreetMaps/Nominatim) has a bunch of different categories for different amenities, like "tourism", "amenity", "leisure", "shop", "historic", "natural", "landuse", "building", "man_made", "sport", and many others. This leads to a lot of false names, where the API lookup lands on a generic "City, State" fallback that's only supposed to happen when we end up taking pictures at some random sidewalk within the city limits. Along with this issue, the situation is exacerbated when the API returns non-helpful locations that don't describe where we actually are, I.E. "Parking lot" or "104 S Road". I've had to tweak the script about a dozen times in order to finally get it to grab a more descriptive name like I would expect. Though I still need to manually review the output, since I no longer trust that I'm not going to end up with a 100 albums of "YYYY_MM_DD_City_State".

 The logic that I've had to use to get the script to do this has become so complex that I decided to split it off of its original git repo into its own git repo, in order to keep up the changes, and be able to document the changes that are made by AI to the script. Along with documenting changes, I also have started using a Plan.md file, to keep the AI on track for the purpose of the script, and the previous decisions that I have made.

* I've had to tweak these values a couple of times to remove outlier images, and better "lock in" on a grouping of photos. To be honest, I'm not even sure these are the current values.

To move onto the second project that I attempted to use AI on was a simple Palword metric dashboard. 

I have setup a Palworld server(VM) for me and my friends, and its working well, except for 2 issues. First without logging in, its hard to tell if there's someone on the server, and whether or not I should reboot the server/VM Host for patching. Second, occasionally the game lags, its still unclear as to what the issue is, but my client drops from 60 FPS to about 45 FPS (guessing). Its not unplayable in that state, but its not fun.

So I wanted to get some metrics to see if it was a server issue or a client issue. Luckily Palworld has a whole page explaining how to get metrics from the server. So using that page and some constraints, I instructed AI to write me up a simple dashboard that I can use to check on the server, without fully logging into the server. Now at a minimum I can see if the server is available for patching, but hopefully it'll help me to see what the actual issue is.

The final project that I leveraged AI on is security. Specifically helping me setup an OpenBoa server to use with my homelab cluster. 

For those who are not knowledgeable of corporate K8S structuring, companies rarely hard code passwords into K8S manifests, as its a security issue for any one that touches the cluster. So they typically have a separate software to handle secrets, the ones I'm familiar with are Hashicorp's Vault, and Cyberark's Vault.
I wanted to do this with my homelab, not because I have too, but because I wanted to admin my cluster as if it were a production cluster. Like many other homelabers, I hard-coded my K8S secrets within the manifests themselves, a combination of ArgoCD applications, and "raw" kustomized manifests, which works great for my local lab, since its (probably) not a target for hackers out there, and its convenient to do, and change/update when needed. Yet, working with AI, it keeps "yelling" at me about the insecurity of this practice. I could tell it to ignore the problem, or I could use it to fix the problem.

To this end, I did some research and came across "OpenBao". It touts itself as a fork of the Hashicorp vault, and has similar commands/layout. This would mean that if I got good at OpenBao, then likely I would also be good at Hashicorp too.

With the product in mind, and the AI having access to the cluster, I had it write up a plan to migrate the hard-coded passwords into the new OpenBao vault, and then remove then from the git repo. Turns out, that was a bad idea. Mostly because the AI kept going off script to "fix" the cluster instead of focusing on the migration itself. So I felt that for every 1 password we got migrated, the AI would spend 30-45 minutes going down a rabbit hole that was ultimately frivolous**. After some tweaking I did manage to get the AI to focus on the task at hand, which significantly reduce the amount of churn the AI output. Eventually I was able to migrate almost all the passwords, there's a couple that I need to revisit due to infrastructure concerns, for now I am at least more secure then I was when I began this migration process.

** I have alot of "archived" apps (in an archive folder) that it kept trying to fix, and some apps that are transitioning to the archived folder and are no longer deployed, So of coarse the AI kept asking questions about bad configurations, and should we deploy these?

With that, I believe I have now caught you up as to what's happening with my homelab. I expect that there will be a couple of shorter posts coming quickly that better elaborate on the topics above, but we'll see as I may decide that its not worth my time to write those articles(Life's busy for me right now).