escapades in engineering

Ukodus: Building a Sudoku Galaxy in Rust and WebAssembly

Unknown — Thu, 19 Feb 2026 00:00:00 -0700

Ukodus: Building a Sudoku Galaxy in Rust and WebAssembly</h1>
Every Sudoku app on iOS is annoying. Ads after every puzzle. Subscriptions to unlock “hard” mode. Hints that just fill in the answer without teaching you anything. I got tired of it, so I did what any reasonable person would do: spent months building an entire Sudoku platform from scratch.</p>
The result is Ukodus</a> — a Rust-powered Sudoku engine that runs natively on iOS, compiles to WebAssembly for the browser, and includes a force-directed galaxy visualization where every puzzle ever played becomes a star. Because apparently I can’t just solve a problem without also building a cosmos around it.</p>

The Engine: 45 Techniques in Rust</h2>
The core of Ukodus is a Sudoku engine written in Rust. It implements 45 human-style solving techniques organized into 10 families:</p>

Singles</strong> — Hidden Single, Naked Single (the basics)</li>
Pairs & Triples</strong> — Naked Pair through Hidden Quad</li>
Intersections</strong> — Pointing Pairs, Box/Line Reduction</li>
Fish</strong> — X-Wing, Swordfish, Jellyfish, plus finned and mutant variants</li>
Wings</strong> — XY-Wing, XYZ-Wing, W-Wing, WXYZ-Wing</li>
Chains</strong> — X-Chain, 3D Medusa, AIC</li>
Rectangles</strong> — Unique Rectangles (6 types), Hidden Rectangle, Empty Rectangle</li>
ALS</strong> — Almost Locked Sets: ALS-XZ, ALS-XY-Wing, ALS Chain</li>
Forcing</strong> — Nishio, Cell/Region Forcing Chains, Dynamic Forcing Chains</li>
Other</strong> — Sue de Coq, Aligned Pair Exclusion, Death Blossom, BUG+1, Backtracking</li> </ul>
Each technique has a Sudoku Explainer (SE) difficulty rating. Hidden Single is 1.5 (beginner territory), Dynamic Forcing Chain is 9.3 (you need a PhD or a lot of patience), and Backtracking sits at 11.0 as the last resort. The engine uses these ratings to classify every puzzle into difficulty tiers: Beginner, Easy, Medium, Intermediate, Hard, Expert, Master, and Extreme.</p>
Why does this matter? Because most Sudoku apps rate puzzles by counting givens or using some vague internal metric. SE ratings map directly to the hardest technique you’d need to solve the puzzle. A puzzle rated 3.2 means you’ll need X-Wings. A 7.5 means ALS Chains or Nishio. You know exactly what you’re getting into.</p>
The engine also generates puzzles with guaranteed unique solutions, rates them on generation, and encodes them as 8-character short codes for sharing. The same short code works across web, iOS, and terminal.</p>
WASM for the Browser</h2>
The Rust engine compiles to WebAssembly via `wasm-pack</code>. The output is a 554KB .wasm</code> binary and a JS glue module that exposes the SudokuGame</code> class. The game renders to an HTML <canvas></code> — the Rust side owns all the drawing logic, which means the rendering is identical regardless of platform.</p>`
`Loading WASM in a SvelteKit app requires some care. You can’t let Vite try to bundle the WASM module at build time, so the loader uses a dynamic import with a @vite-ignore</code> pragma:</p>`
const </span>wasmJsPath </span>= '</span>/wasm/sudoku_wasm.js</span>'; </span>const </span>mod </span>= </span>await </span>import</span>(</span>/* @vite-ignore / </span>wasmJsPath</span>); </span>await </span>mod</span>.</span>default</span>({ </span> module_or_path: new URL('</span>/wasm/sudoku_wasm_bg.wasm</span>', window.location.</span>origin</span>) </span>}); </span></code></pre> The WASM files live in static/wasm/</code> and get served as plain static assets. No Vite WASM plugin, no special bundler config. The /play/</code> route sets ssr = false</code> so SvelteKit generates a minimal HTML shell that hydrates client-side — the WASM needs a browser environment with a canvas, so server-side rendering would just blow up.</p> The game loop is a standard requestAnimationFrame</code> cycle. Every frame calls game.tick()</code> on the WASM side, which handles input processing, animation, and canvas rendering. Keyboard events get forwarded from the Svelte component to the WASM engine via game.handle_key(event)</code>. The engine supports vim-style navigation (hjkl</code>), arrow keys, and WASD — because every good application should support at least three ways to move a cursor.</p>
The Galaxy</h2> Here’s where things got a little unhinged.</p> I wanted a way to visualize all the puzzles that had been played. A leaderboard felt boring. A list felt worse. Then I thought: what if every puzzle was a star, and puzzles with similar techniques formed constellations?</p> The Galaxy page uses D3’s forceSimulation</code> to create a force-directed graph. Each node is a puzzle, colored by difficulty tier (green for Beginner through near-black for Extreme). Node size scales with play count. Edges connect puzzles that share solving techniques, so similar puzzles cluster together.</p> The fun part is the convex hulls. D3 computes polygonHull</code> for each technique family and draws translucent overlays around them, so you can see the Fish cluster, the Wings cluster, the Chains cluster. It looks like a star map. Which is the point.</p>
simulation </span>= </span>d3 </span> .</span>forceSimulation</span><GalaxyNode>(</span>nodes</span>) </span> .</span>force</span>('</span>link</span>', </span>d3</span>.</span>forceLink</span><GalaxyNode, GalaxyEdge>(</span>edges</span>) </span> .</span>id</span>(</span>d </span>=> </span>d</span>.id).</span>distance</span>(</span>60</span>).</span>strength</span>(</span>0.3</span>)) </span> .</span>force</span>('</span>charge</span>', </span>d3</span>.</span>forceManyBody</span>().</span>strength</span>(-</span>80</span>)) </span> .</span>force</span>('</span>center</span>', </span>d3</span>.</span>forceCenter</span>(</span>width </span>/ </span>2</span>, </span>height </span>/ </span>2</span>)) </span> .</span>force</span>('</span>collide</span>', </span>d3</span>.</span>forceCollide</span><GalaxyNode>() </span> .</span>radius</span>(</span>d </span>=> </span>nodeRadius</span>(</span>d</span>) + </span>2</span>)) </span> .</span>alphaDecay</span>(</span>0.02</span>) </span> .</span>on</span>('</span>tick</span>', </span>ticked</span>); </span></code></pre> The Galaxy also has a live component. When someone completes a puzzle, a WebSocket message pushes the new node into the simulation in real-time. You can literally watch the galaxy grow. The WebSocket connects to /api/v1/ws/galaxy</code> through an Nginx proxy that keeps the connection alive for up to 24 hours:</p>
location /api/v1/ws/ { </span> proxy_pass $api_upstream; </span> proxy_http_version 1.1; </span> proxy_set_header Upgrade $http_upgrade; </span> proxy_set_header Connection "upgrade"; </span> proxy_read_timeout 86400s; </span>} </span></code></pre> There’s also a “secrets” system. By default, you only see 22 of the 45 techniques and 6 of the 10 families. The advanced families — Chains, ALS, Forcing, and Other — are hidden until you unlock them by completing harder puzzles. When you unlock secrets, the Galaxy reveals entire new constellations that were invisible before. It’s my favorite feature and probably the most unnecessary one.</p>
One Engine, Many Targets</h2>
The beauty of writing the core in Rust is that the same engine works everywhere:</p>

Browser</strong>: Compiled to WASM, loaded in SvelteKit, renders to canvas</li>
iOS</strong>: Compiled natively via Xcode, same Rust core, native UI shell</li>
Terminal</strong>: Rust binary with a TUI interface</li>
Shared codes</strong>: 8-character short codes and 81-character puzzle strings work across all platforms</li> </ul>
You can start a puzzle on iOS, share the code, and your friend can play the exact same puzzle in a browser. The engine deterministically generates the same puzzle from the same seed, so there’s no server round-trip needed to decode a shared puzzle.</p>
SvelteKit 5: Runes and Static Generation</h2>
The web frontend is SvelteKit 5 (Svelte 5.49) with TypeScript. The entire state management layer uses Svelte 5 runes — class-based stores with `$state</code>, $derived</code>, and $effect</code> in .svelte.ts</code> files:</p>`
class </span>PlayerStore </span>{ </span> </span>id </span>= </span>$state</span>(</span>''</span>); </span> </span>tag </span>= </span>$state</span>(</span>''</span>); </span> </span>secrets </span>= </span>$state</span>(</span>false</span>); </span> </span> </span>setTag</span>(</span>value</span>: </span>string</span>) </span>{ </span> </span>this</span>.</span>tag </span>= </span>value</span>; </span> </span>localStorage</span>.</span>setItem</span>(</span>PLAYER_TAG_KEY</span>, </span>value</span>); </span> } </span>} </span> </span>export const </span>playerStore </span>= new PlayerStore(); </span></code></pre> The app uses adapter-static</code> with prerender = true</code> and trailingSlash = 'always'</code>. Content pages (home, about, techniques, difficulty, privacy, how-to-play, app) get fully pre-rendered to static HTML at build time. Interactive pages (/play/</code> and /galaxy/</code>) set ssr = false</code> because they need browser APIs — canvas for the game, D3 DOM manipulation for the galaxy.</p> This hybrid approach means content pages load instantly as static HTML while the interactive pages get a lightweight shell that hydrates client-side. The build output is a directory of plain HTML, CSS, and JS files that any web server can serve. No Node.js runtime needed in production.</p> The previous frontend was 9 static HTML files with ~2,100 lines of inline JavaScript. The SvelteKit rewrite gave us proper component architecture, shared layouts (no more duplicated header and footer across 9 files), reactive state management, and PostHog analytics integration — all while maintaining the same zero-runtime deployment model.</p> Deployment: Kubernetes on a Homelab</h2> Yes, I’m running a Sudoku game on Kubernetes. I know</a>. The cluster runs on Talos Linux on Raspberry Pi 5s</a> connected via 2Gbps LACP-bonded networking</a> on a Turing Pi board. The dev environment is NixOS</a> because of course it is.</p> The frontend build is a multi-stage Docker image:</p> FROM</span> node:22-alpine </span>AS </span>build </span>WORKDIR </span>/app </span>COPY</span> frontend/package.json frontend/package-lock.json ./ </span>RUN </span>npm ci </span>COPY</span> frontend/ . </span>RUN </span>npm run build </span> </span>FROM</span> nginx:1-alpine </span>COPY</span> --from=</span>build</span> /app/build /usr/share/nginx/html </span></code></pre> Stage one builds the SvelteKit static site (including WASM assets). Stage two drops the output into Nginx Alpine. The final image is tiny — just Nginx serving static files.</p> The Nginx config handles the caching strategy:</p> Asset Type</th> Browser Cache</th> Edge Cache</th></tr></thead> /_app/</code> (hashed SvelteKit assets)</td> 1 year, immutable</td> 1 year</td></tr> /assets/</code> (images, icons)</td> 30 days, immutable</td> 30 days</td></tr> /wasm/</code></td> 1 hour, must-revalidate</td> 24 hours</td></tr> Pages</td> 60 seconds</td> 5 minutes, stale-while-revalidate</td></tr> /api/</code></td> no-store</td> no-store</td></tr> </tbody></table> Hashed assets get immutable caching because the hash changes on every build. WASM gets shorter caching because I might update the engine without changing the filename. API responses are never cached. Pages get short browser caches with longer edge caches and stale-while-revalidate</code> so users see content immediately while the cache refreshes in the background.</p> The API backend runs as a separate K8s service (api.ukodus.svc.cluster.local:3000</code>) that Nginx proxies to via Kubernetes DNS. Security headers (HSTS, X-Content-Type-Options, X-Frame-Options, Referrer-Policy) are set globally, and the WASM location block adds Cross-Origin-Embedder-Policy</code> and Cross-Origin-Opener-Policy</code> for SharedArrayBuffer</code> compatibility.</p> SvelteKit’s adapter-static</code> generates pre-compressed .br</code> and .gz</code> files at build time. Nginx serves these directly with gzip_static on</code>, so there’s zero CPU overhead for compression at request time.</p> What’s Next</h2> The engine still has room for more techniques. There are some fish variants and chain patterns I haven’t implemented yet. The iOS app needs feature parity with the web version’s galaxy view. And I keep thinking about adding a puzzle-of-the-day feature with global leaderboards.</p> But for now, it’s a Sudoku app that doesn’t have ads, doesn’t require a subscription, and teaches you actual solving techniques instead of just filling in answers. Which is all I wanted in the first place.</p> Play it</strong>: ukodus.now</a> Source</strong>: github.com/kcirtapfromspace/sudoku-core</a> iOS App</strong>: App Store</a></p> Building Talos Linux for Raspberry Pi 5 with Custom Kernel Unknown — Sun, 01 Feb 2026 00:00:00 -0700 Building Talos Linux for Raspberry Pi 5 with Custom Kernel</h1> Getting Talos Linux running on the Raspberry Pi 5 requires a custom kernel build. The stock Talos kernel doesn’t include the necessary drivers for RPi5’s RP1 PCIe controller and MACB ethernet. This post documents the complete build process and the issues I encountered along the way.</p> The Problem</h2> The RPi5 uses a new architecture with the RP1 southbridge chip connected via PCIe. This means:</p> Different ethernet driver (MACB + Broadcom BCM54213PE PHY)</li> Different device tree files (bcm2712 instead of bcm2711)</li> Network interface named end0</code> instead of eth0</code></li> Potential 4K vs 16K page size conflicts</li> </ul> Prerequisites</h2> Docker with buildx support</li> Local container registry (I used localhost:5001</code>)</li> ~50GB disk space for builds</li> ARM64 cross-compilation support</li> </ul> Repository Structure</h2> I created a monorepo with Talos and related projects as submodules:</p> talso-rpi5/ </span>├── checkouts/ </span>│ ├── pkgs/ # kernel builds </span>│ ├── talos/ # main OS </span>│ └── sbc-raspberrypi5/ # RPi5 overlay </span>├── _out/ # Build outputs </span>├── CLAUDE.md # Build documentation </span>└── AGENTS.md # Task delegation docs </span></code></pre> Step 1: Build the Custom Kernel</h2> The kernel needs specific configuration for RPi5:</p> # Clear Docker cache to ensure fresh build </span>docker</span> builder prune</span> -af </span> </span># Build kernel for arm64 </span>cd</span> checkouts/pkgs </span>gmake</span> kernel PLATFORM=linux/arm64 PUSH=true \ </span> REGISTRY=localhost:5001 USERNAME=wittenbude </span></code></pre> Critical Kernel Config Settings</h3> In checkouts/pkgs/kernel/build/config-arm64</code>:</p> # Use 4K pages (16K causes Talos mount API issues) </span>CONFIG_ARM64_4K_PAGES=y </span># CONFIG_ARM64_16K_PAGES is not set </span> </span># SD card drivers must be built-in (not modules) </span>CONFIG_MMC_SDHCI_PLTFM=y </span>CONFIG_MMC_SDHCI_BRCMSTB=y </span> </span># Ethernet support </span>CONFIG_MACB=y </span>CONFIG_PHYLIB=y </span>CONFIG_PHYLINK=y </span>CONFIG_BROADCOM_PHY=y </span></code></pre> The 4K vs 16K page dilemma:</p> 16K pages</strong>: Networking works, but Talos shadow bind mounts fail with EINVAL</li> 4K pages</strong>: Talos boots cleanly, networking requires matching kernel and DTB versions</li> </ul> Step 2: Build the Imager</h2> The imager creates the final bootable image. It needs to reference our custom kernel:</p> cd</span> checkouts/talos </span> </span># Get the kernel tag </span>KERNEL_TAG</span>=$</span>(</span>cd</span> ../pkgs </span>&& </span>git</span> describe</span> --tag --always --dirty</span>) </span> </span># Build imager with custom kernel </span>gmake</span> imager PLATFORM=linux/arm64 PUSH=true \ </span> REGISTRY=localhost:5001 USERNAME=wittenbude \ </span> PKG_KERNEL=localhost:5001/wittenbude/kernel:$</span>KERNEL_TAG </span></code></pre> Step 3: Build the RPi5 Overlay</h2> The overlay provides RPi5-specific device tree files and firmware:</p> cd</span> checkouts/sbc-raspberrypi5 </span> </span># Get kernel tag for PKGS reference </span>KERNEL_TAG</span>=$</span>(</span>cd</span> ../pkgs </span>&& </span>git</span> describe</span> --tag --always --dirty</span>) </span> </span># Build overlay with our custom kernel's DTBs </span>gmake</span> PLATFORM=linux/arm64 PUSH=true \ </span> REGISTRY=localhost:5001 USERNAME=wittenbude \ </span> PKGS_PREFIX=localhost:5001/wittenbude PKGS=$</span>KERNEL_TAG </span></code></pre> This ensures the DTB files match the kernel version - critical for RP1 initialization.</p> Step 4: Generate the Metal Image</h2> cd</span> checkouts/talos </span> </span>IMAGER_TAG</span>=$</span>(</span>git</span> describe</span> --tag --always --dirty</span>) </span>OVERLAY_TAG</span>=$</span>(</span>cd</span> ../sbc-raspberrypi5 </span>&& </span>git</span> describe</span> --tag --always --dirty</span>) </span> </span>docker</span> run</span> --rm -t --network</span>=host \ </span> -v</span> ./_out:/out</span> -v</span> /dev:/dev</span> --privileged </span>\ </span> localhost:5001/wittenbude/imager:$</span>IMAGER_TAG </span>\ </span> metal</span> --arch</span> arm64 \ </span> --board</span>=rpi_generic \ </span> --overlay-name</span>=rpi5 \ </span> --overlay-image</span>=localhost:5001/wittenbude/sbc-raspberrypi5:$</span>OVERLAY_TAG </span>\ </span> --extra-kernel-arg</span>="</span>console=tty1</span>" \ </span> --extra-kernel-arg</span>="</span>console=ttyAMA10,115200</span>" \ </span> --system-extension-image</span>=ghcr.io/siderolabs/iscsi-tools:v0.1.10 </span></code></pre> Output: _out/metal-arm64.raw.zst</code> (~88MB)</p> Step 5: Flash and Boot</h2> # Identify SD card </span>diskutil</span> list </span> </span># Unmount </span>diskutil</span> unmountDisk /dev/disk4 </span> </span># Flash (use rdisk for faster writes) </span>zstd -d -c</span> _out/metal-arm64.raw.zst | </span>sudo</span> dd of=/dev/rdisk4 bs=4m status=progress </span> </span># Eject </span>diskutil</span> eject /dev/disk4 </span></code></pre> Step 6: Apply Worker Configuration</h2> Once booted, the node enters maintenance mode. Apply the worker config:</p> # In maintenance mode (first boot) </span>talosctl</span> apply-config</span> --insecure --nodes </span><NODE_IP> --file worker.yaml </span> </span># After leaving maintenance mode </span>talosctl -n </span><NODE_IP> apply-config</span> --file</span> worker.yaml </span></code></pre> Critical: Enable Discovery</h3> The worker config must</strong> include cluster discovery settings, or the node won’t appear in cluster membership:</p> cluster</span>: </span> </span>id</span>: </span><CLUSTER_ID> </span> </span>secret</span>: </span><CLUSTER_SECRET> </span># Required for discovery auth </span> </span>controlPlane</span>: </span> </span>endpoint</span>: </span>https://<CONTROL_PLANE_IP>:6443 </span> </span>clusterName</span>: </span><CLUSTER_NAME> </span> </span>discovery</span>: </span> </span>enabled</span>: </span>true </span># THIS IS CRITICAL </span> </span>registries</span>: </span> </span>service</span>: </span> </span>endpoint</span>: </span>https://discovery.talos.dev/ </span> </span>kubernetes</span>: </span> </span>disabled</span>: </span>true </span></code></pre> Without discovery.enabled: true</code> and the cluster.secret</code>, the node boots and kubelet runs, but talosctl get members</code> won’t show it.</p> Network Interface</h2> RPi5 uses end0</code> for ethernet (not eth0</code>):</p> machine</span>: </span> </span>network</span>: </span> </span>interfaces</span>: </span> - </span>interface</span>: </span>end0 </span> </span>dhcp</span>: </span>true </span></code></pre> UART Debugging</h2> For boot issues, connect a UART adapter to the GPIO pins:</p> # Start logging session </span>screen -L -Logfile</span> /tmp/uart.log /dev/cu.usbserial-* 115200 </span></code></pre> Key boot messages to watch for:</p> BL31</code> - ARM Trusted Firmware starting</li> mmc0: new</code> - SD card detected</li> macb</code> - Ethernet driver loading</li> BCM54213PE</code> - PHY detected</li> machined</code> - Talos starting</li> </ul> Troubleshooting</h2> No Network Activity (NIC lights off)</h3> Check kernel/DTB version mismatch</li> Verify RP1 PCIe initialization in dmesg</li> Consider 16K page kernel if RP1 isn’t initializing</li> </ul> Node Not Appearing in Cluster</h3> Check talosctl get discoveryconfig</code> - must show discoveryEnabled: true</code></li> Verify cluster.secret</code> matches control plane</li> Check talosctl get members</code> from the node itself</li> </ul> Kubelet Certificate Errors</h3> Verify cluster.ca.crt</code> matches the control plane’s certificate</li> Extract correct CA: talosctl -n <CP_IP> get machineconfig -o yaml | grep -A1 "cluster:" | grep crt</code></li> </ul> Boot Loop at BL31</h3> Usually indicates 16K page kernel incompatibility</li> Rebuild with CONFIG_ARM64_4K_PAGES=y</code></li> </ul> Final Result</h2> After all this, I have a working RPi5 node in my Talos cluster:</p> $ kubectl get nodes -o wide </span>NAME STATUS ROLES VERSION INTERNAL-IP KERNEL-VERSION </span>talos-192-168-150-8 Ready <none> v1.35.0 192.168.150.8 6.12.67-talos </span>talos-ek0-5dx Ready control-plane v1.35.0 100.78.183.103 6.18.2-talos </span>talos-lwn-dba Ready <none> v1.35.0 100.95.115.98 6.18.2-talos </span></code></pre> The RPi5 is running the custom 6.12.67-talos kernel with 4K pages, working ethernet, and is fully participating in cluster workloads.</p> Repository</h2> The build configuration and documentation are available at:</p> Main repo:</strong> github.com/kcirtapfromspace/talos-rpi5</a></li> </ul> Submodule Forks (rpi5-support branch)</h3> pkgs</strong> (kernel config): github.com/kcirtapfromspace/pkgs</a></li> talos</strong> (OS changes): github.com/kcirtapfromspace/talos</a></li> sbc-raspberrypi5</strong> (overlay): github.com/kcirtapfromspace/sbc-raspberrypi5</a></li> </ul> References</h2> Talos Linux Documentation</a></li> siderolabs/sbc-raspberrypi5</a></li> siderolabs/pkgs</a></li> RPi5 RP1 Peripheral Datasheet</a></li> </ul> Matrix Homeserver with MAS, Keycloak & Tailscale Unknown — Sun, 25 Jan 2026 00:00:00 +0000 Matrix Homeserver with MAS, Keycloak & Tailscale</h1> A deep-dive into setting up a self-hosted Matrix homeserver with modern authentication, SSO via Keycloak, and QR code login support—all exposed securely through Tailscale Funnel.</p> The Goal</h2> Get a Matrix homeserver running on a Raspberry Pi that:</p> Is publicly accessible via Tailscale Funnel</li> Uses Matrix Authentication Service (MAS) for modern OIDC-based auth</li> Integrates with Keycloak for SSO</li> Supports QR code login from Element iOS/Android</li> Federates with other Matrix servers</li> </ul> Architecture Overview</h2> Internet </span> │ </span> ▼ </span>Tailscale Funnel (HTTPS :443) </span> │ </span> ▼ </span>┌─────────────────────────────────────────────────────────┐ </span>│ Raspberry Pi (thinkmeshmatrix.tail16ecc2.ts.net) │ </span>│ │ </span>│ ┌─────────────┐ ┌─────────────┐ │ </span>│ │ nginx-proxy │────▶│ Synapse │ │ </span>│ │ (:8090) │ │ (:8008) │ │ </span>│ └──────┬──────┘ └─────────────┘ │ </span>│ │ │ </span>│ ├────────────▶┌─────────────┐ │ </span>│ │ │ MAS │ │ </span>│ │ │ (:8080) │ │ </span>│ │ └──────┬──────┘ │ </span>│ │ │ │ </span>│ │ ┌──────▼──────┐ │ </span>│ │ │ mas-postgres │ │ </span>│ │ └─────────────┘ │ </span>│ │ │ </span>│ └────────────▶┌─────────────┐ │ </span>│ │ Keycloak │ │ </span>│ │ (:8080) │ │ </span>│ └─────────────┘ │ </span>└─────────────────────────────────────────────────────────┘ </span></code></pre> The Journey (and the Hiccups)</h2> 1. Initial Tailscale Setup</h3> Started by installing Tailscale directly on the Raspberry Pi running Synapse and enabling Funnel for public HTTPS access:</p> tailscale</span> up</span> --ssh </span>tailscale</span> funnel 8090 </span></code></pre> Changed the Pi’s hostname to thinkmeshmatrix</code> to reflect its purpose.</p> 2. Federation Woes</h3> Initial server was configured with server_name: matrix.local</code>—which obviously can’t federate since remote servers can’t verify it.</p> Hiccup #1</strong>: Empty federation_domain_whitelist: []</code> was blocking ALL outbound federation. The fix was to comment out that line entirely.</p> Hiccup #2</strong>: Matrix federation uses port 8448 by default, but Tailscale Funnel serves on 443. Fixed by adding serve_server_wellknown: true</code> to Synapse config, which tells remote servers to use port 443.</p> Had to reset the server with a new server_name</code>:</p> server_name</span>: "</span>thinkmeshmatrix.tail16ecc2.ts.net</span>" </span></code></pre> 3. Setting Up MAS (Matrix Authentication Service)</h3> MAS provides modern OIDC-based authentication for Matrix, enabling features like:</p> Delegated authentication to external providers</li> Fine-grained session management</li> QR code login (MSC4108)</li> </ul> Hiccup #3</strong>: MAS requires PostgreSQL, not SQLite. Deployed a postgres:16-alpine</code> container:</p> docker</span> run</span> -d --name</span> mas-postgres \ </span> --network</span> matrix-net \ </span> -e</span> POSTGRES_USER=mas \ </span> -e</span> POSTGRES_PASSWORD=maspassword \ </span> -e</span> POSTGRES_DB=mas \ </span> -v</span> mas-postgres-data:/var/lib/postgresql/data \ </span> --restart</span> unless-stopped \ </span> postgres:16-alpine </span></code></pre> 4. Nginx Reverse Proxy Routing</h3> The trickiest part was getting nginx to route requests correctly between Synapse, MAS, and Keycloak. Key insight: certain Matrix endpoints need to go to MAS, not Synapse:</p> # MAS endpoints </span>location /.well-known/openid-configuration { proxy_pass http://mas; } </span>location /oauth2/ { proxy_pass http://mas; } </span>location /authorize { proxy_pass http://mas; } </span>location /_matrix/client/v3/login { proxy_pass http://mas; } </span>location /complete-compat-sso/ { proxy_pass http://mas; } </span> </span># Keycloak endpoints </span>location /realms/ { proxy_pass http://keycloak; } </span> </span># Everything else to Synapse </span>location / { proxy_pass http://synapse; } </span></code></pre> Hiccup #4</strong>: Kept getting “No Such Resource” errors during SSO flow. Turned out several MAS endpoints were missing from nginx config: /complete-compat-sso/</code>, /link</code>, /consent</code>, /device</code>.</p> 5. Keycloak Integration</h3> Set up Keycloak as the upstream identity provider for MAS. This allows using Keycloak’s user management and potentially federating with other identity providers later.</p> Hiccup #5</strong>: MAS was redirecting users to http://keycloak:8080/...</code> (internal Docker hostname) instead of the public URL. Fixed by:</p> Adding Keycloak routes to nginx</li> Configuring Keycloak with KC_HOSTNAME</code>:</li> </ol> docker</span> run</span> -d --name</span> keycloak \ </span> -e</span> KC_HOSTNAME=thinkmeshmatrix.tail16ecc2.ts.net \ </span> -e</span> KC_HTTP_ENABLED=true \ </span> -e</span> KC_PROXY_HEADERS=xforwarded \ </span> ... </span></code></pre> Updating MAS config to use public URL for issuer</li> </ol> Hiccup #6</strong>: MAS cached the old Keycloak discovery document. Had to change the provider ID to force a refresh.</p> 6. User Provisioning Issues</h3> Hiccup #7</strong>: “Localpart not available” error when trying to register via SSO. The user existed in Synapse’s database from a failed previous attempt, but not in MAS. Had to manually clean up:</p> # Stop Synapse </span>docker</span> stop synapse </span> </span># Clean up orphaned user data </span>sudo</span> sqlite3 /home/pi/matrix-synapse/homeserver.db \ </span> "</span>DELETE FROM users WHERE name = '@username:server';</span>" </span>sudo</span> sqlite3 /home/pi/matrix-synapse/homeserver.db \ </span> "</span>DELETE FROM profiles WHERE full_user_id LIKE '%username%';</span>" </span> </span># Clean up MAS </span>docker</span> exec mas-postgres psql</span> -U</span> mas</span> -d</span> mas</span> -c </span>\ </span> "</span>DELETE FROM upstream_oauth_authorization_sessions; DELETE FROM upstream_oauth_links;</span>" </span> </span>docker</span> start synapse </span></code></pre> 7. Synapse Configuration Evolution</h3> Hiccup #8</strong>: Synapse 1.136+ deprecated experimental_features.msc3861</code> in favor of a stable matrix_authentication_service</code> config block:</p> # Old (deprecated) </span>experimental_features</span>: </span> </span>msc3861</span>: </span> </span>enabled</span>: </span>true </span> </span>issuer</span>: </span>https://... </span> </span># New (stable) </span>matrix_authentication_service</span>: </span> </span>enabled</span>: </span>true </span> </span>endpoint</span>: </span>http://mas:8080 </span> </span>secret</span>: "</span>shared-secret</span>" </span></code></pre> 8. QR Code Login (MSC4108)</h3> The final boss: getting QR code login working for Element iOS.</p> Hiccup #9</strong>: MSC4108 wasn’t being advertised. Had to add to Synapse config:</p> experimental_features</span>: </span> </span>msc4108_enabled</span>: </span>true </span></code></pre> Hiccup #10</strong>: The rendezvous endpoint was incorrectly routed to MAS instead of Synapse. Synapse has a built-in rendezvous server at /_synapse/client/rendezvous</code>. Removed the incorrect nginx route and let it fall through to Synapse.</p> Testing the rendezvous endpoint:</p> curl -X</span> POST "</span>https://server/_matrix/client/unstable/org.matrix.msc4108/rendezvous</span>" \ </span> -H </span>"</span>Content-Type: text/plain</span>"</span> -d </span>'</span>test</span>' </span># Returns: {"url":"https://server/_synapse/client/rendezvous/SESSION_ID"} </span></code></pre> Final Configuration</h2> Synapse (homeserver.yaml</code>)</h3> server_name</span>: "</span>thinkmeshmatrix.tail16ecc2.ts.net</span>" </span>public_baseurl</span>: "</span>https://thinkmeshmatrix.tail16ecc2.ts.net/</span>" </span> </span>serve_server_wellknown</span>: </span>true </span>allow_public_rooms_over_federation</span>: </span>true </span> </span>matrix_authentication_service</span>: </span> </span>enabled</span>: </span>true </span> </span>endpoint</span>: </span>http://mas:8080 </span> </span>secret</span>: "</span>shared-secret</span>" </span> </span>account_management_url</span>: "</span>https://thinkmeshmatrix.tail16ecc2.ts.net/account</span>" </span> </span>experimental_features</span>: </span> </span>msc4108_enabled</span>: </span>true </span></code></pre> MAS (config.yaml</code>)</h3> http</span>: </span> </span>public_base</span>: </span>https://thinkmeshmatrix.tail16ecc2.ts.net/ </span> </span>issuer</span>: </span>https://thinkmeshmatrix.tail16ecc2.ts.net/ </span> </span>matrix</span>: </span> </span>homeserver</span>: </span>thinkmeshmatrix.tail16ecc2.ts.net </span> </span>endpoint</span>: </span>http://synapse:8008 </span> </span>secret</span>: "</span>shared-secret</span>" </span> </span>upstream_oauth2</span>: </span> </span>providers</span>: </span> - </span>id</span>: </span>keycloak-provider </span> </span>issuer</span>: </span>https://thinkmeshmatrix.tail16ecc2.ts.net/realms/matrix </span> </span>client_id</span>: </span>mas </span> </span>client_secret</span>: "</span>keycloak-client-secret</span>" </span> </span>claims_imports</span>: </span> </span>localpart</span>: </span> </span>action</span>: </span>require </span> </span>template</span>: "</span>{{ user.preferred_username }}</span>" </span> </span>experimental</span>: </span> </span>qr_login</span>: </span> </span>enabled</span>: </span>true </span></code></pre> Data Persistence & Recovery</h2> All containers configured with:</p> --restart unless-stopped</code> for automatic recovery after power outages</li> Proper volume mounts for data persistence:</li> </ul> Service</th> Volume</th></tr></thead> Synapse</td> /home/pi/matrix-synapse</code> → /data</code></td></tr> MAS</td> /home/pi/mas</code> → /config</code></td></tr> MAS-Postgres</td> Docker volume mas-postgres-data</code></td></tr> Keycloak</td> /home/pi/keycloak-data</code> → /opt/keycloak/data</code></td></tr> nginx-proxy</td> /home/pi/nginx-proxy</code> → /etc/nginx/conf.d</code></td></tr> </tbody></table> Lessons Learned</h2> Docker networking is tricky</strong> - Internal hostnames (keycloak:8080</code>) work for server-to-server communication but not for browser redirects. Always use public URLs in OAuth flows.</p> </li> OIDC discovery caching</strong> - MAS caches discovery documents. When you change upstream provider URLs, you may need to force a refresh by changing provider IDs or restarting.</p> </li> Database cleanup matters</strong> - Failed registration attempts can leave orphaned records in Synapse’s database that block future attempts. Clean up both the users</code> and profiles</code> tables.</p> </li> nginx routing order</strong> - More specific routes must come before catch-all routes. The MSC4108 rendezvous endpoint needs to go to Synapse, not MAS.</p> </li> Synapse evolves fast</strong> - Configuration that works in one version may be deprecated in the next. The move from experimental_features.msc3861</code> to matrix_authentication_service</code> caught me off guard.</p> </li> </ol> Current Status</h2> ✅ Matrix homeserver running at thinkmeshmatrix.tail16ecc2.ts.net</code></li> ✅ Federation working with matrix.org and other servers</li> ✅ SSO via Keycloak operational</li> ✅ QR code login (MSC4108) functional</li> ✅ Automatic restart after power outages</li> ✅ Data persistence across container restarts</li> </ul> Resources</h2> Matrix Authentication Service Docs</a></li> Synapse Configuration Manual</a></li> MSC4108: QR Code Login</a></li> Tailscale Funnel</a></li> </ul> 2Gbps Link Aggregation on Turing Pi BMC with 802.3ad LACP Unknown — Sun, 11 Jan 2026 00:00:00 -0700 2Gbps Link Aggregation on Turing Pi BMC with 802.3ad LACP</h1> The Turing Pi 2.5 has two gigabit Ethernet ports (ge0 and ge1) on its BMC. By bonding these together with 802.3ad LACP, you can achieve true 2Gbps aggregate bandwidth for your cluster traffic. This post documents how I implemented this feature in custom firmware.</p> Why 2Gbps?</h2> In a homelab cluster, the BMC handles all network traffic for up to 4 compute nodes. With multiple nodes running workloads that generate significant network I/O, a single gigabit link can become a bottleneck. Link aggregation doubles the available bandwidth and provides redundancy.</p> Prerequisites</h2> Turing Pi 2.5 board</li> Custom firmware with bonding support (or build your own)</li> Switch with LACP support (I used a UniFi USW Pro Max 24 PoE)</li> Two Ethernet cables connected to both BMC ports</li> </ul> The Challenge</h2> Getting 802.3ad LACP working on the Turing Pi BMC required solving several issues:</p> 1. DSA Ports Share the Same MAC Address</h3> The ge0 and ge1 interfaces are DSA (Distributed Switch Architecture) ports on a Realtek switch chip. By default, they share the same MAC address, which breaks bonding since Linux requires unique MACs for slave interfaces.</p> 2. Hardcoded Bonding Mode</h3> The stock firmware’s S00dsa</code> init script loads the bonding module with a hardcoded mode:</p> modprobe</span> bonding mode=balance-alb miimon=100 </span></code></pre> This prevents changing to 802.3ad mode at runtime.</p> 3. Mode Changes Don’t Persist</h3> Even if you manually configure 802.3ad, the changes are lost on reboot because the module is loaded with balance-alb mode before the network configuration runs.</p> The Solution</h2> I created a proper bonding setup with these components:</p> Architecture</h3> Web Browser → BMC-UI (http://turingpi) → REST API → bmcd daemon </span> → network_config.rs → /etc/bonding.conf → S45bonding → bond0 </span></code></pre> Key Changes</h3> Fixed S00dsa</strong>: Removed the hardcoded mode from modprobe:</p> modprobe</span> bonding </span># No mode specified </span></code></pre> </li> Created S45bonding</strong>: A new init script that runs after network init and properly configures bonding:</p> Reads mode from /etc/bonding.conf</code></li> Sets unique MAC on ge1 before enslaving</li> Deletes and recreates bond0 with the correct mode</li> Adds bond0 to the br0 bridge</li> </ul> </li> REST API Integration</strong>: Added network_config.rs</code> module to bmcd for web UI control</p> </li> </ol> Configuration Files</h2> /etc/bonding.conf</h3> Contains the bonding mode (one of the supported modes):</p> active-backup </span></code></pre> The default is active-backup</code> for safety - it works without any switch configuration. Change to 802.3ad</code> for LACP once your switch is configured.</p> /etc/bonding.enabled</h3> Empty marker file that enables bonding when present.</p> S45bonding Init Script</h3> #!/bin/sh </span> </span>BONDING_CONF</span>="</span>/etc/bonding.conf</span>" </span>BONDING_ENABLED</span>="</span>/etc/bonding.enabled</span>" </span> </span>get_bonding_mode</span>() { </span> </span>if </span>[ </span>-f </span>"$</span>BONDING_CONF</span>" </span>]</span>; </span>then </span> </span>cat </span>"$</span>BONDING_CONF</span>" </span>2</span>>/dev/null | </span>head -1 </span>| </span>tr -d </span>'</span>[:space:]</span>' </span> </span>else </span> </span>echo </span>"</span>active-backup</span>" </span> </span>fi </span>} </span> </span>start_bonding</span>() { </span> </span>if </span>[ </span>! </span>-f </span>"$</span>BONDING_ENABLED</span>" </span>]</span>; </span>then </span> </span>return</span> 0 </span> </span>fi </span> </span> </span>MODE</span>=$</span>(</span>get_bonding_mode</span>) </span> </span> </span># Remove interfaces from bridge </span> </span>ip</span> link set ge0 nomaster </span>2</span>>/dev/null </span> </span>ip</span> link set ge1 nomaster </span>2</span>>/dev/null </span> </span>ip</span> link set ge0 down </span> </span>ip</span> link set ge1 down </span> </span> </span># Set unique MAC on ge1 (DSA ports share same MAC by default) </span> </span>BASE_MAC</span>=$</span>(</span>cat</span> /sys/class/net/ge0/address </span>2</span>></span>/dev/null) </span> </span>GE1_MAC</span>=$</span>(</span>echo </span>"$</span>BASE_MAC</span>" | </span>awk -F</span>: </span>'</span>{ </span> last = ("0x" $6) + 1; </span> printf "%s:%s:%s:%s:%s:%02x", $1, $2, $3, $4, $5, last </span> }</span>'</span>) </span> </span>ip</span> link set ge1 address "$</span>GE1_MAC</span>" </span> </span> </span># Delete existing bond and recreate with correct mode </span> </span>ip</span> link del bond0 </span>2</span>>/dev/null </span> </span>ip</span> link add bond0 type bond mode "$</span>MODE</span>" miimon 100 </span> </span> </span># For 802.3ad, set LACP rate to fast </span> </span>[ </span>"$</span>MODE</span>" = "</span>802.3ad</span>" </span>] </span>&& </span>echo</span> fast > /sys/class/net/bond0/bonding/lacp_rate </span> </span> </span># Enslave interfaces </span> </span>ip</span> link set ge0 master bond0 </span> </span>ip</span> link set ge1 master bond0 </span> </span>ip</span> link set ge0 up </span> </span>ip</span> link set ge1 up </span> </span>ip</span> link set bond0 up </span> </span>ip</span> link set bond0 master br0 </span>} </span></code></pre> Supported Bonding Modes</h2> Mode</th> Name</th> Switch Config</th> Description</th></tr></thead> 0</td> balance-rr</td> Aggregate</td> Round-robin</td></tr> 1</td> active-backup</td> None</td> Failover only</td></tr> 2</td> balance-xor</td> Aggregate</td> XOR hashing</td></tr> 3</td> broadcast</td> None</td> All slaves transmit</td></tr> 4</td> 802.3ad</td> LACP</td> IEEE LACP</td></tr> 5</td> balance-tlb</td> None</td> Transmit load balancing</td></tr> 6</td> balance-alb</td> None</td> Adaptive load balancing</td></tr> </tbody></table> Switch Configuration</h2> For 802.3ad LACP, your switch must be configured to expect LACP on those ports. On UniFi:</p> Go to Devices → Your Switch → Ports</li> Select the two ports connected to the Turing Pi</li> Create an Aggregate with LACP mode enabled</li> </ol> Verification</h2> Check the bond status:</p> cat</span> /proc/net/bonding/bond0 </span></code></pre> Expected output:</p> Ethernet Channel Bonding Driver: v6.8.12 </span> </span>Bonding Mode: IEEE 802.3ad Dynamic link aggregation </span>Transmit Hash Policy: layer2 (0) </span>MII Status: up </span>MII Polling Interval (ms): 100 </span> </span>802.3ad info </span>LACP active: on </span>LACP rate: fast </span>Active Aggregator Info: </span> Aggregator ID: 1 </span> Number of ports: 2 </span> Partner Mac Address: 9c:05:d6:63:f9:bb </span> </span>Slave Interface: ge0 </span>MII Status: up </span>Speed: 1000 Mbps </span>Duplex: full </span> </span>Slave Interface: ge1 </span>MII Status: up </span>Speed: 1000 Mbps </span>Duplex: full </span></code></pre> Key things to verify:</p> Bonding Mode: IEEE 802.3ad Dynamic link aggregation</code></li> Number of ports: 2</code></li> Partner Mac Address</code> shows your switch’s MAC (not 00:00:00:00:00:00)</li> Both ge0 and ge1 show MII Status: up</code> and Speed: 1000 Mbps</code></li> </ul> Web UI / API</h2> The bonding configuration is also accessible via:</p> Web UI</strong>: http://turingpi</code> → System > Network Settings</li> API</strong>: GET /api/network_config</code> returns current status</li> API</strong>: POST /api/network_config?enabled=1&mode=802.3ad&apply=1</code> to configure</li> </ul> Troubleshooting</h2> LACP Not Negotiating (Partner MAC is 00:00:00:00:00:00)</h3> Verify LACP is enabled on the switch ports</li> Check that both cables are connected</li> Wait 30+ seconds for negotiation</li> </ul> Network Unreachable After Mode Change</h3> If you change to 802.3ad but the switch isn’t configured for LACP:</p> Temporarily disable LACP on the switch</li> Access BMC and change mode to active-backup</code></li> Then configure switch for LACP and switch back to 802.3ad</code></li> </ol> Bonding Not Starting</h3> Verify /etc/bonding.enabled</code> exists</li> Check /etc/bonding.conf</code> contains a valid mode</li> Run /etc/init.d/S45bonding status</code> to check bond state</li> </ul> Flashing to Internal Storage</h2> To flash the custom firmware to internal storage without using an SD card:</p> Boot from SD card with working firmware</li> Attach UBI device: ubiattach -m 1 -d 0</code></li> Copy firmware: scp firmware.tpu root@turingpi:/tmp/</code></li> Write to UBI: ubiupdatevol /dev/ubi0_1 /tmp/firmware.tpu</code></li> Reboot without SD card</li> </ol> Links</h2> Pull Request</a></li> Linux Bonding Documentation</a></li> 802.3ad LACP Standard</a></li> </ul> Tips & Tricks of the trade: sanitize secrets from dirty commits Unknown — Wed, 15 Jan 2025 19:09:33 -0700 Tips & Tricks of the Trade: sanitize secrets from dirty commits</h1> Scan a repo for senstive keys or secretes. Did you accidently commit a api key.</p> Tools like gitleaks</a> or trufflehog</a> can be very effective aid in cleaning up dirty commits.</p> GitLeaks</h2> Lets cover gitleaks to remove an api key from history. Go install</p> brew install gitleaks </span></code></pre> Setup a config file .gitleaks.toml</code> in the root dir of the repo. This example excludes some rules from a git directory that is being used as a static asset store.</p> title = "Custom Gitleaks Configuration" </span> </span>[extend] </span> </span>useDefault = true </span> </span>tags = ["data_dir"] </span> </span>[[rules]] </span> </span>id = "generic-api-key" </span> </span>[[rules.allowlists]] </span> </span>paths = [ </span> </span>'''^data/raw-events-.\.(json|parquet)$''', </span>] </span> </span>[[rules]] </span> </span>id = "github-app-token" </span> </span>[[rules.allowlists]] </span> </span>paths = [ </span> </span>'''^data/raw-events-.\.(json|parquet)$''' </span>] </span></code></pre> Run the CLI command or better yet setup a pre-commit to always check.</p> ❯ gitleaks detect --source . --report-format json --report-path gitleaks-report.json </span> </span> ○ </span> │╲ </span> │ ○ </span> ○ ░ </span> ░ gitleaks </span> </span>2:24PM INF 373 commits scanned. </span>2:24PM INF scanned ~4469777469 bytes (4.47 GB) in 1m7.7s </span>2:24PM WRN leaks found: 6 </span></code></pre> Resolve the leaks</h2> Github has an excellent guide</a> using git filter-repo</code> . The tldr is you need to clone a fresh repo and use a file outside of that repo to replace text in side commits</p> ❯ git filter-repo --sensitive-data-removal --replace-text ../replacements.txt </span></code></pre> You can check the changes with grep</p> ❯ grep -c '^refs/pull/./head$' .git/filter-repo/changed-refs </span>11 </span> </span>❯ grep '^refs/pull/./head$' .git/filter-repo/changed-refs </span>refs/pull/123/head </span>refs/pull/37/head </span>refs/pull/372/head </span>refs/pull/379/head </span>refs/pull/42/head </span>refs/pull/433/head </span>refs/pull/48/head </span>refs/pull/57/head </span>refs/pull/72/head </span>refs/pull/73/head </span>refs/pull/76/head </span># {{title}} </span>{{date}} </span>Status: #Blog </span>Category: </span>Volume: </span>Difficulty: </span>KW: </span>## Should I Write This? </span>1. How does {{title}} fit into my deal? What offer or funnel is it promoting? </span> </span>2. What's the key takeaway, action or CTA? </span> </span>## Content Plan </span> </span>### Intro - What Makes It Interesting? </span>- Data Point or Fact </span>- Hook </span>- Anecdote </span>- Personal Story </span>- Examples </span>- Screenshots </span>- Links </span>- Quotes/Interview </span>- Imagery </span>- Embedded demo </span>- Real Life application </span>- Connection to something timely or well known </span> </span>### Main Content </span> </span>#### What's my point of view on {{title}}? Do I have a background in {{title}}? Do I want to exapnd on an old idea? </span> </span>#### What Questions do I get asked about this or would I ask about {{title}} </span> </span>#### Problems </span> </span>#### Solutions </span> </span>#### What do I want them to know, feel or do? </span>- Core Message </span>- CTA </span> </span> </span>## Blog </span># {{title}} </span>❯ git show refs/pull/37/head </span></code></pre> Next setup pre-commit to prevent this in the future.</p> Hello, World! Unknown — Thu, 09 Jan 2025 00:00:00 +0000 Welcome to the new blog! This site has been rebuilt from the ground up using Zola</a>, a blazing-fast static site generator written in Rust.</p> Why Zola?</h2> After using Hugo for a while, I wanted something that:</p> Stays fast</strong> - Zola is incredibly quick at building sites</li> Has simpler templating</strong> - Tera templates feel more intuitive</li> Is written in Rust</strong> - Because why not?</li> Works well with Obsidian</strong> - My notes become blog posts seamlessly</li> </ol> The New Publishing Workflow</h2> The content for this blog now lives in my Obsidian vault. When I push changes to the vault repository, a GitHub Action automatically triggers a rebuild of this site. Here’s how it works:</p> Obsidian Vault -> GitHub Push -> Blog Rebuild -> GitHub Pages </span></code></pre> No more fighting with Hugo’s markdown quirks. Just write in Obsidian, push, and publish.</p> What’s Next?</h2> I’ll be migrating some older content and writing new posts about:</p> Infrastructure and DevOps</li> Rust programming</li> Developer tooling</li> Random technical adventures</li> </ul> Stay tuned, and thanks for reading!</p> This post was written in Obsidian and published automatically via GitHub Actions.</em></p> performance testing with k6 Unknown — Thu, 06 Jun 2024 19:42:52 -0600 Performance testing with K6</p> wingbits Unknown — Thu, 06 Jun 2024 19:40:24 -0600 Flight tracking</p> caching-with-nodejs Unknown — Mon, 13 May 2024 16:18:43 -0600 file sync Unknown — Mon, 13 May 2024 11:35:10 -0600 Tips & Tricks of the Trade: file sync</h1> There is the ole trusty rsync</p> $</span> rsync</span> -avz ~</span>/compressed_file.tar.gz remoteuser@otherhost:/share/ </span> </span>$</span> ssh remoteuser@otherhost "</span>tar zxf /share/compressed_file.tar.gz -C /share/uncompressed/</span>" </span></code></pre> Transfer and Extract in One Step</p> rsync -avz -e ssh ~/compressed_file.tar.gz remoteuser@otherhost:/dev/stdout | ssh remoteuser@otherhost "tar zxf - -C /share/uncompressed/" </span> </span></code></pre> Use mbuffer</code></strong> when dealing with large data transfers, especially over unstable networks, where buffering can help ensure smoother and more reliable data flow.</p> $ mbuffer -s 1K -m 512 -i ~/compressed_file.tar.gz | ssh remoteuser@otherhost "tar zxf - -C /share/uncompressed/" </span></code></pre> mbuffer -s 1K -m 512 -i "$SOURCE_FILE"</code></strong> -s 1K</code>: Sets the buffer size to 1 Kilobyte.</li> -m 512</code>: Allocates 512 Megabytes of memory for buffering.</li> -i "$SOURCE_FILE"</code>: Specifies the input file to read from.</li> </ul> </li> |</code></strong>: Pipes the output of mbuffer</code> to the next command.</li> ssh remoteuser@otherhost "tar zxf - -C /share/uncompressed/"</code></strong> Connects to the remote host otherhost</code> with the user remoteuser</code>.</li> "tar zxf - -C /share/uncompressed/"</code>: Extracts the tarball received from standard input (-</code>) into the /share/uncompressed/</code> directory on the remote host.</li> </ul> </li> </ul> Use pv</code></strong> when you need a simple way to monitor the progress of data through a pipeline with minimal overhead.</p> $ pv ~/compressed_file.tar.gz | ssh admin@169.254.x.x "tar zxf - -C /share/uncompressed/" </span></code></pre> pv "$SOURCE_FILE"</code></strong> pv</code>: Monitors the progress of data through the pipe.</li> "$SOURCE_FILE"</code>: Specifies the file to transfer.</li> </ul> </li> |</code></strong>: Pipes the output of pv</code> to the next command.</li> ssh admin@169.254.x.x "tar zxf - -C /share/uncompressed/"</code></strong> Connects to the remote host with IP 169.254.x.x</code> using the user admin</code>.</li> "tar zxf - -C /share/uncompressed/"</code>: Extracts the tarball received from standard input (-</code>) into the /share/uncompressed/</code> directory on the remote host.</li> </ul> </li> </ul> raspberrypi-unifi-controller-ap Unknown — Sun, 11 Feb 2024 21:50:12 -0700 I picked up an AP from Unifi, but I already have a firewalla and some switches. I originally set up the AP manually with the Wi-Fi AP. If that’s all you need to do it will run, I couldn’t figure out how to get VLANs configured for specific. Well, you need to use the Unifi controller. You can get this with a cloud key or a dream machine, but it gives you the remote console. Turns out you can make your own Unifi controller. Let’s do that with one of the Raspberry Pi’s that are lying around.</p> A quick google later and big shout out to (emmet)[https://pimylifeup.com/rasberry-pi-unifi/] for getting the guide below guide together. Setup goes superfast, its like 15 minutes and spend more time looking for a nail as you now need to reset the AP so it can be adopted by the controller.</p> Take a peek at this guy, Evan he explains all the configurations and gives you a starting point on whether or not you should change a setting (Unifi advance wi-fi settings) [https://evanmccann.net/blog/2021/11/unifi-advanced-wi-fi-settings]</p> Configure the VLAN ID and map them each to a Wi-Fi network and associate them with your ap.</p> Then you just need to find all the Wi-Fi devices and rest their connection to your segmented networks. Congrats! you can now cut off internet access to all the IoT & music things that lay about your house.</p> # Installing the UniFi Controller on the Raspberry Pi </span> </span> </span>## Preparing your Raspberry Pi for the UniFi Controller </span> </span>```bash </span>sudo apt update </span>sudo apt upgrade </span>``` </span> </span>### Adding Entropy using rng-tools </span> </span>2. To improve the startup speed of the UniFi controller software on our Raspberry Pi, we need to install `rng-tools`. </span> </span>We will utilize this package to ensure the Raspberry Pi has enough entropy for the random number generation that the UniFi software uses. </span> </span>```bash </span>sudo apt install rng-tools </span>``` </span> </span>3. We now need to make a slight change to the rng-tools configuration. </span> </span>Begin editing the config file by running the following command. </span> </span>```bash </span>sudo nano /etc/default/rng-tools-debian </span>``` </span> </span>4. Within this file, find and uncomment the following line. </span> </span>Find </span> </span>```ini </span>#HRNGDEVICE=/dev/hwrng </span>``` </span> </span>Replace With </span> </span>```ini </span>HRNGDEVICE=/dev/hwrng </span>``` </span> </span>By uncommenting this line, we are adding to the amount of entropy (The amount of randomness) that the system has available. </span> </span>The Raspberry Pi features an integrated random number generator that we can utilize to increase the entropy pool. </span> </span>5. Once you have made the change, save the file by pressing CTRL + X, then Y, followed by ENTER. </span> </span>6. Finally, restart the `rng-tools` service by running the command below. </span> </span>```bash </span>sudo systemctl restart rng-tools </span>``` </span> </span>Once the service has finished restarting, it should now be safe to proceed to the next section of this guide. </span> </span>### Installing an Older Release of LibSSL </span> </span>7. Due to the version of MongoDB that we will be utilizing, we will need to install an older release of LibSSL to our Raspberry Pi. </span> </span>In particular, we will be installing LibSSL 1.0. You can download this old release by using the following command. </span> </span>```bash </span>wget http://ports.ubuntu.com/pool/main/o/openssl/libssl1.0.0_1.0.2g-1ubuntu4_arm64.deb -O libssl1.0.deb </span>``` </span> </span>8. Once the package has been downloaded, all we need to do to install it is to run the command below. </span> </span>```bash </span>sudo dpkg -i libssl1.0.deb </span>``` </span> </span>### Installing MongoDB to your Raspberry Pi for the UniFi Controller </span> </span>9. To use the UniFi Controller on your Raspberry Pi, we will need to install MongoDB. </span> </span>This is the database server that UniFi uses to store all of its data. As we can’t rely on the package repository, we will need to follow some additional steps. </span> </span>For this first step, we will download the latest available version of MongoDB 3.6 to our Pi. We are installing 3.6 as this is currently the only supported release for the UniFi Controller. </span> </span>```bash </span>wget https://repo.mongodb.org/apt/ubuntu/dists/xenial/mongodb-org/3.6/multiverse/binary-arm64/mongodb-org-server_3.6.22_arm64.deb -O mongodb.deb </span>``` </span> </span>10. Once the package is downloaded, install it by using the following command within the terminal. </span> </span>```bash </span>sudo dpkg -i mongodb.deb </span>``` </span> </span>11. Now that we have installed the MongoDB server, set it to start when your Raspberry Pi boots using the command below. </span> </span>```bash </span>sudo systemctl enable mongod </span>``` </span> </span>12. Finally, start MongoDB by running the command shown below in the terminal. </span> </span>This will start the server immediately, so we won’t have to wait till our device restarts. </span> </span>```bash </span>sudo systemctl start mongod </span>``` </span> </span>## Installing the UniFi Controller to the Raspberry Pi </span> </span>1. Our first task is to add the UniFi repository to our sources list. </span> </span>We can achieve this by running the command below. </span> </span>```bash </span>echo 'deb [arch=amd64 signed-by=/usr/share/keyrings/ubiquiti-archive-keyring.gpg] https://www.ui.com/downloads/unifi/debian stable ubiquiti' | sudo tee /etc/apt/sources.list.d/100-ubnt-unifi.list >/dev/null </span>``` </span> </span>You might notice that we are using “`amd64`” and not “`arm64`” or “`armhf`“. This is due to Ubiquiti not having their repository set up to mark “`arm64`” as compatible. However, it doesn’t hugely matter as, at the moment, it will still download files compatible with our Raspberry Pi. </span> </span>2. We now need to add the repositories GPG key by using the following command. </span> </span>```bash </span>curl https://dl.ui.com/unifi/unifi-repo.gpg | sudo tee /usr/share/keyrings/ubiquiti-archive-keyring.gpg >/dev/null </span>``` </span> </span>The GPG key is what helps tell the package manager it is downloading the correct package. </span> </span>3. As we made changes to the repositories, we need to now update the package list by running the command below. </span> </span>```bash </span>sudo apt update </span>``` </span> </span>4. Now finally, we can install version 17 of the OpenJDK runtime as well as the Unifi Controller software itself to our Raspberry Pi by running the following command. </span> </span>```bash </span>sudo apt install openjdk-17-jre-headless unifi </span>``` </span> </span>Installing UniFi through this method will automatically set up a service. This service will automatically start the UniFi software at boot. </span> </span>Additionally, we are installing version 17 of the Java runtime environment as it is currently the only version supported by the UniFi controller. </span> </span>## First Boot of the UniFi Controller on your Raspberry Pi </span> </span>In this section, we are going to walk you through the initial configuration steps of the UniFi software. </span> </span>1. First, retrieve the [local IP address for your Raspberry Pi](https://pimylifeup.com/raspberry-pi-ip-address/). </span> </span>If you have terminal access to your Pi, you can use the following command. </span> </span>```bash </span>hostname -I </span>``` </span> </span>2. With your Raspberry Pi’s IP address handy, go to the following web address in your favorite web browser. </span> </span>Ensure that you replace “`<IPADDRES>`” with the IP of your Raspberry Pi. </span> </span>``` </span>https://<IPADDRESS>:8443 </span>``` </span> </span></code></pre> Why does my internet suck? Smart Queues Unknown — Sat, 10 Feb 2024 16:39:55 -0700 Since the firewall has smart queue rules, it’s time to set them up. Firstly, let’s thank firewall people from getting around to adding domain based rules. Secondly, thumbs down on providers who don’t disclose their purposed based domains names.</p> First step, go find all the ports for the services you rely on. I vaguely recall that streaming services are all over UDP. Because nobody wants to do all the TCP handshakes. So when setting up my rules, i’ll just allocate the UDP connections to the HIGH Priority smart queue.</p> Second step, create the rules you want. For my setup, I had to go through and create up individual smart rules as to give priority to these endpoints or specific ports. I did manage to set up some network segmentation, so the rules are only scoped to the primary vlan that my phones, laptops, and tablets use. Wish firewalla managed a list or enabled a community defined list, so I wouldn’t have to hunt these all down.</p> Zoom</h3> (Zoom Firewall Ports)[https://support.zoom.com/hc/en/article?id=zm_kb&sysparm_article=KB0060548]</p> </th> </th> </th> </th></tr></thead> Protocol</strong></td> Ports</strong></td> Source</strong></td> Destination</strong></td></tr> TCP</td> 80,443</td> All Zoom clients</td> .zoom.us</td></tr> TCP</td> 443, 8801, 8802</td> All Zoom clients</td> </td></tr> UDP</td> 3478, 3479, 8801 - 8810</td> All Zoom clients</td> </td></tr> </tbody></table> Google Meet</h3> (Google Meet Firewall Ports)[https://support.google.com/a/answer/1279090?hl=en]</p> For audio and video, set up outbound UDP ports 3478 and 19302–19309. If you want to limit the number of Chrome WebRTC ports being used, use the ports specified at WebRTC UDP Ports</a>. </li> Or, you can limit those ports with your firewall.</li> </ul> </li> stream.meet.google.com</li> youtube.googleapis.com</li> www.youtube-nocookie.com</li> googlevideo.com</li> </li> </ul> Slack Huddles (aka Amazon Chime)</h3> Check that your network is set up to allow outbound traffic to UDP/22466. Otherwise, huddles will use TCP/443 for media transport (video and audio).</li> Allow outbound traffic to TCP/443. This is required for huddles to function, even if outbound traffic to UDP/22466 is allowed for media transport.</li> If you’d like, you can limit access to a specific IP range: 99.77.128.0/18. If your environment requires you to allow Slack’s required domains</a>, make sure you approve .m.chime.aws. We aren’t able to provide a list of static domains, and suggest allowing by wildcard to avoid any network disruptions.</li> </ul> ^ That’s because they don’t manage the service… https://cloud-native.slack.com/help/urls</p> [ </span> "</span>.chime.aws</span>", </span> "</span>a.slack-edge.com</span>", </span> "</span>a.slack-imgs.com</span>", </span> "</span>admin.slack.com</span>", </span> "</span>alpha.slack.com</span>", </span> "</span>api.slack.com</span>", </span> "</span>app.slack.com</span>", </span> "</span>assets.slack.com</span>", </span> "</span>avatars.slack-edge.com</span>", </span> "</span>b.slack-edge.com</span>", </span> "</span>b.slack-imgs.com</span>", </span> "</span>beta.slack.com</span>", </span> "</span>blog.slack.com</span>", </span> "</span>ca.slack-edge.com</span>", </span> "</span>cloud-native.slack.com</span>", </span> "</span>downloads.slack-edge.com</span>", </span> "</span>edgeapi.slack.com</span>", </span> "</span>email.slack.com</span>", </span> "</span>email2.slack.com</span>", </span> "</span>email3.slack.com</span>", </span> "</span>email4.slack.com</span>", </span> "</span>emoji.slack-edge.com</span>", </span> "</span>example.slack.com</span>", </span> "</span>files-edge.slack.com</span>", </span> "</span>files-origin.slack.com</span>", </span> "</span>files.slack.com</span>", </span> "</span>global-upload-edge.slack.com</span>", </span> "</span>go-beta.slack.com</span>", </span> "</span>go-debug.slack.com</span>", </span> "</span>go.slack.com</span>", </span> "</span>help.slack.com</span>", </span> "</span>hooks.slack.com</span>", </span> "</span>join.slack.com</span>", </span> "</span>my.slack.com</span>", </span> "</span>oauth2.slack.com</span>", </span> "</span>platform-tls-client.slack.com</span>", </span> "</span>platform.slack-edge.com</span>", </span> "</span>slack-email.slack.com</span>", </span> "</span>slack-files.com</span>", </span> "</span>slack-imgs.com</span>", </span> "</span>slack-infra-canvas.slack.com</span>", </span> "</span>slack-infra.slack.com</span>", </span> "</span>slack.com</span>", </span> "</span>slack.global.ssl.fastly.net</span>", </span> "</span>slackb.com</span>", </span> "</span>spellcheck.slack-edge.com</span>", </span> "</span>status.slack.com</span>", </span> "</span>try.slack.com</span>", </span> "</span>universal-upload-edge.slack.com</span>", </span> "</span>upload.slack.com</span>", </span> "</span>wss-backup.slack.com</span>", </span> "</span>wss-mobile.slack.com</span>", </span> "</span>wss-primary.slack.com</span>" </span>] </span></code></pre> Looks like they use chime under the hood for their “huddles”. So lets get the correct information for the</p> https://docs.aws.amazon.com/chime/latest/ag/network-config.html https://answers.chime.aws/articles/123/hosts-ports-and-protocols-needed-for-amazon-chime.html</p> Amazon Chime Meetings, Chat, and Business Calling</strong> uses 99.77.128.0/18 TCP/443 UDP/3478</li> </ul> MS Teams</h3> Of course this would be black hole of vague redirects to find the actual information. What a shitshow MS. Also, wish I could just set a domain fore these fools. https://learn.microsoft.com/en-us/microsoft-365/enterprise/urls-and-ip-address-ranges?view=o365-worldwide</p> Skype for Business Online and Microsoft Teams</h2> ID</th> Category</th> ER</th> Addresses</th> Ports</th></tr></thead> 11</td> Optimize Required</td> Yes</td> 13.107.64.0/18, 52.112.0.0/14, 52.122.0.0/15, 2603:1063::/38</code></td> UDP:</strong> 3478, 3479, 3480, 3481</td></tr> 12</td> Allow Required</td> Yes</td> .lync.com, .teams.microsoft.com, teams.microsoft.com</code> 13.107.64.0/18, 52.112.0.0/14, 52.122.0.0/15, 52.238.119.141/32, 52.244.160.207/32, 2603:1027::/48, 2603:1037::/48, 2603:1047::/48, 2603:1057::/48, 2603:1063::/38, 2620:1ec:6::/48, 2620:1ec:40::/42</code></td> TCP:</strong> 443, 80</td></tr> 16</td> Default Required</td> No</td> .keydelivery.mediaservices.windows.net, .streaming.mediaservices.windows.net, mlccdn.blob.core.windows.net</code></td> TCP:</strong> 443</td></tr> 17</td> Default Required</td> No</td> aka.ms</code></td> TCP:</strong> 443</td></tr> 18</td> Default Optional Notes:</strong> Federation with Skype and public IM connectivity: Contact picture retrieval</td> No</td> .users.storage.live.com</code></td> TCP:</strong> 443</td></tr> 19</td> Default Optional Notes:</strong> Applies only to those who deploy the Conference Room Systems</td> No</td> adl.windows.com</code></td> TCP:</strong> 443, 80</td></tr> 27</td> Default Required</td> No</td> .secure.skypeassets.com, mlccdnprod.azureedge.net</code></td> TCP:</strong> 443</td></tr> 127</td> Default Required</td> No</td> .skype.com</code></td> TCP:</strong> 443, 80</td></tr> 180</td> Default Required</td> No</td> compass-ssl.microsoft.com</code></td> TCP:</strong> 443</td></tr> </tbody></table> Apple</h3> https://support.apple.com/en-us/HT202944 https://support.apple.com/en-us/102036</p> I do a bunch of RDP to headless machines, so need to give those ports some priority.</p> </th> </th> </th> </th> </th> </th></tr></thead> 16384–16403</td> UDP</td> Real-Time Transport Protocol (RTP), Real-Time Control Protocol (RTCP)</td> —</td> connected, —</td> Messages (Audio RTP, RTCP; Video RTP, RTCP)</td></tr> 16384–16387</td> UDP</td> Real-Time Transport Protocol (RTP), Real-Time Control Protocol (RTCP)</td> —</td> connected, —</td> FaceTime, Game Center</td></tr> 16393–16402</td> UDP</td> Real-Time Transport Protocol (RTP), Real-Time Control Protocol (RTCP)</td> —</td> —</td> FaceTime, Game Center</td></tr> 16403–16472</td> UDP</td> Real-Time Transport Protocol (RTP), Real-Time Control Protocol (RTCP)</td> —</td> —</td> Game Center</td></tr> 5223</td> TCP</td> Apple Push Notification Service (APNS)</td> —</td> —</td> iCloud DAV Services (Contacts, Calendars, Bookmarks), Push Notifications</a>, FaceTime, iMessage, Game Center, Photo Stream</td></tr> 3478–3497</td> UDP</td> —</td> —</td> nat-stun-port - ipether232port</td> FaceTime, Game Center</td></tr> 3283</td> TCP/UDP</td> Apple Remote Desktop and Classroom</td> —</td> net-assistant, classroom</td> Apple Remote Desktop</td></tr> 5900</td> TCP</td> Remote Framebuffer</td> 6143</td> rfb</td> Apple Remote Desktop, Screen Sharing</td></tr> 5900</td> UDP</td> Remote Framebuffer, Real-Time Transport Protocol (RTP), Real-Time Control Protocol (RTCP)</td> —</td> —</td> Apple Remote Desktop, Screen Sharing</td></tr> 5901–5902</td> UDP</td> Real-Time Transport Protocol (RTP), Real-Time Control Protocol (RTCP)</td> —</td> —</td> Apple Remote Desktop, Screen Sharing</td></tr> </tbody></table> All the cool kids are running NixOS Unknown — Fri, 12 Jan 2024 00:22:11 -0700 All the cool kids are running NixOS</h1> My first intro to NixOS was a demo that an engineer ran through in my companies DevOps guild. My first thoughts are and still mainly consist of why are you not just dumping that into a container which also provides a deterministic, reproducible output. In general, I put NixOS on the back burner as I explored the funky town that is buildpacks novelty government cloud provides. I didn’t think much else of it for a couple of months.</p> Over in a homelab thread NixOS popped up again, this time in the context of using NixOS to manage Raspberry 5 configuration. I revisited some GitHub links but fell disappointed, just don’t think there is much traction there quite yet.</p> Then I was exposed to some humble nerd bragging, someone threw down the words:</p> “Very please with my dev machine these days. Apple Silicon Mac + nix-darwin + UTM(for Linux VMs). Nix flakes for dev environments/ reproducibility is so pleasant. Ghostty is already a solid terminal”</p> There was a lot that I needed to process with this declaration</p> Ghostty 👻 - that’s still closed beta by Mitchell Hashimoto</a> I’d love to try that out.</li> My Dev Machine/s have some sprawl that I would love to sort out. Some declarative steamroller would be nice ☠️</li> I’ll need to Google UTM</a> cause I’m still terrible at acronyms</li> Nix has popped up again, how far off could it be from Ansible</li> </ul> The quick follow answers:</p> Ghostty - yup still closed beta. Go watch the discord to get access beta Let’s fix developer sprawl with nix-darwin. Oh boy i’m a little confused to what’s happening with nix-darwin, flakes, home-manager I’ll figure out UTM when I get nix-darwin sorted.</p> Getting started with nix-darwin</a></h2> Baby Steps</h3> Get the TLDR and in-depth guide to get your feet wet with NixOS</p> nixos-and-flakes.thiscute.world</a></li> zero-to-nix.com</a></li> nix.dev - tutorials</a> Use Determinate Systems to install nix safely on MaxOS Nix Survival mode on MacOS</a></li> </ul> Search Packages: search.nixos.org</a></p> Useful takes on setup</h3> I perused a bunch of these trying to get a foundation on how various people interpreted using flakes, home-manager, or the likes.</p> yusef.napora.org - nixos-asahi</a></li> br0g.0brg.net - nix</a></li> krisztianfekete.org - nixos on-apple silicon with utm</a></li> srid - nixos-config</a></li> nixcademy.com - nix on mac</a></li> </ul> Get lost on what to do next</h3> Figure out how to install some packages, find some examples https://github.com/ryan4yin/nix-darwin-kickstarter/blob/main/rich-demo/flake.nix</p> Go get overwhelmed with Mitchell’s setup: https://github.com/mitchellh/nixos-config</p> Modified take on Mitchells: https://github.com/cor/nixos-config/tree/master</p> Guess I’ll setup cachix? I’ll figure out the meaning of this a little later https://app.cachix.org/cache/kcirtapfromspace-nixos-config#pull</p> Mitchell’s config</h4> Growing up I had typing class. Once or twice a week we would go to this dark room filled with computers, we got the joy of playing a speed typing game in silence for like an hour. This mild abuse to those of short attention spans feel like what i’m going through with Mitchells setup.</p> linux - well its been a decade since I ran a desktop</li> kitty/alacritty - new terminals i’ve been on iterm2 for a hot minute these days</li> fish - this seems fine, until copy/paste doen’t seem to work and your on page 12 trying to figure out if key bindings are all change</li> neovim - oh man what have I gotten into. I feel like all the simple things are now complicated again. At this point I’m afraid to ask. oh plugin hell, what are all these things - treesitter, lua, telescope, etc. great they’re installed now how do I create a branch</li> </ul> </li> </ul> </li> raycast - neat its another Command - spacebar</code> thing. I assume this is needed as nix doesn’t install applications into the Application dir.</li> tmux - so why do none Control - b + %</code> things work?</li> </ul> Cachix</h3> cachix.org</a></p> Guess I’ll setup cachix? I’ll figure out the meaning of this a little later. Though, this might another rabbit hole that is too much for me to think through completely. kcirtapfromspace-nixos-config</a></p> Personal Auth Token</h4> Login to cachix.org and figure out how to generate an auth token.</p> ❯</span> cachix authtoken <magical token here> </span>Written</span> to /Users/thinkstudio/.config/cachix/cachix.dhall </span></code></pre> After cachix is setup you’ll see the logs</p> copying</span> path '</span>/nix/store/7l8l8by558mf76vf9ngpg7lq0c8gwqby-source</span>' from '</span>https://cache.nixos.org</span>'... </span>[1</span> copied (147.6 MiB)</span>,</span> 24.8 MiB DL] evaluating derivation '</span>git+file:///Users/thinkstudio/.config/nix-darwin#darwinConfigurations.thinkstudio.system</span>' </span></code></pre> Virtual Machines with UTM</h2> https://mac.getutm.app/ Surprise UTM</a> is really just open source version of VirtualBox that works with Apple’s M1 ARM64 Architecture. As my work has led me deeper and deeper into the world of microservices and docker containers. I haven’t had a hypervisor in ages! I’ve led teams to containerize and use .devcontainers</a> to build an immutable env that can easily be shared across a team. I can see a setting where this is a requirement to contribute in a secure means. I like the abstraction from the host machine and adds a extra buffer of security. There is also the cattle not pets mentality, once the base configuration is established for the VM, these can be cloned to infinity and used as ephemeral or persistent dedicated local environment for any dev work.</p> Boot up</h3> Boot up can be annoying as `Display Output is not Active“</p> Trying to get the mouse to wake up the screen sometimes does nothing, or boot time is just that slow. I feel like on the mac studio with the allocation of 32GB of ram this should be snappy fast.</p> ISOs</h3> Go get them</p> nixos-iso</a></li> windows-iso</a> Via CrystalFetch</li> </ul> Hashed Password</h3> The nixos.nix</code> config houses a hashed password for the VM. This is a quick means to generate a compatible password if you do not have mkpasswd</code> available.</p> ❯</span> docker run</span> -it --rm</span> alpine sh</span> -c </span>'</span>printf "password" | mkpasswd -s -m md5</span>' </span></code></pre> example of user management with hashed passwords</a></p> Bootstrapping VM</h3> Mitchell has provided a Makefile filled with some convenient ssh commands that will help configure VMs. The order of operations:</p> Elevate to Root & Set Password</li> Check the ip with ifconfg</code></li> Run the `make vm/Bootstrap0</li> Run the `make vm/Bootstrap</li> Login with hashed password</li> Copy Secrets over to VM</li> </ul> Check git, gpg certs, Clone a Repo</h3> ❯</span> gpg</span> --list-secret-keys --keyid-format</span>=long </span>❯</span> eval "$</span>(</span>ssh-agent -s</span>)</span>" </span>fish:</span> Unsupported use of '</span>=</span>'. In fish, please use '</span>set SSH_AUTH_SOCK /tmp/ssh-XXXXXX8h5pip/agent.7808</span>'. </span> </span>❯</span> kcirtap@dev </span>~</span>> eval $(</span>ssh-agent -c</span>) </span>Agent</span> pid 7863 </span>❯</span> git clone git@github.com:kcirtapfromspace/nixos-config.git </span></code></pre> Install necessary packages</h3> Now for any project you have dedicated env which you can rip & replace.</p> nix-env -i </span><package> </span></code></pre> “Fun” little things</h3> Big fan of some of my ingrained muscle memory with mac keybindings</p> Command - L</code> yeah that will lock the screen Command - W</code> you wanted to close the VM right, Right Command - C/V</code> You’re going to want to press the CTRL button</p> Why does my internet suck? Unknown — Thu, 11 Jan 2024 01:18:48 -0700 Why does my Internet suck?</h1> Early my career, working from home was really a novelty, so I didn’t really care too much about my Internet. The TV works, I could play online matches of Halo. Life was great! Though, I do not miss the hours I spent idling in traffic, or my existential thoughts around the decades of politics that were involved for my manager to navigate and acquire an entire row of end cubes with a corner view of the flatirons.</p> Now I worry about how since then my work has transitioned to full-time views looking out into the world from the 1940s stucco box of bricks of which I call home. These days, existence has become more and more dependent on that little black box provided by Xfinity, for a convenient price of $15 dollars per month. It’s one after one a hodgepodge of calendar calls on a variety of vendors (FaceTime, Meet, Zoom, Hangout, Huddles or MSTeams) all with their own little annoyances. Yes, that list is rank ordered in terms of quality I experience. Considering my reality, it’s kind of wonder at the considerable effort I put into trying to ignore my internet woes. I have mainly gone by just accepting persistence quality issues as I sit on the call after call where my video drops out or how my audio sounds like DJ cutting records. I have largely just assumed that my issues stem from some combo of buggy products and Comcast being a soulless conglomerate who couldn’t care less of their users experience. I ultimately categorized these experiences with the tune of the Frank Sinatra song “That’s Life” like I do with so many other things.</p> While I feel that I have an unusually high tolerance for life’s little inconveniences, my partner’s tolerance is unusually thin for these things. I do hit breaking point, but it is more so related to hearing a barrage of exasperated grievances as with our terrible Wi-Fi. As of recent its been doubly triggering, because my years of perceived schlepping on the couch with my laptop isn’t actually work, she too is now alongside me on the couch schlepping remotely with her job & masters program. So the occasional grievance has morphed into a full-blown catastrophe. As I reflect that I now have trigger phrases like “Our Internet Sucks”, “The Wi-Fi is garbage”.</p> I originally started network changes small. Mainly to not pay rental fees. So, I at least ditched the rental Comcast gear. Reviewed cable modem and Wi-Fi router which had reasonable price and a healthy amount of reviews. In 2019, I picked up a TP-Link AC2300</a> and NETGEAR CM600</a> this was the era where we had the cheapest Comcast Internet that was available, and I was blissfully ignorant. This lasted until about early 2022 where I replaced the TP-Link with the NETGEAR R6700AXS</a>.</p> Around this time I also began to tinker more Raspberry Pi hobbies and in general our home devices started to grow(smart light, PiHole, AlgoTrading, & personal devices). In turn bumped up the bandwidth on Xfinity and bobs your uncle, right?</p> Not a chance! the complaints kept rolling, kicked off looking into getting fiber, but those leads didn’t go far. So in 2023 I maxed out the Xfinity bandwidth to Gig + speeds. Well, old modem wouldn’t handle speeds over a gig and regular usage didn’t come close to those speeds. So I picked up a new modem NETGEAR CM2000</a> to handle the additional capacity. I also like the concept that support for mesh might help deadspots. TP-Link AXE5400</a></p> The problem is my partner works at our dining table for the most part with a direct line of sight to the Wi-Fi router, which is probably only 12’ - 15’ away. So, I don’t think mesh will help her issues, but it might still solve my abysmal performance in my office. My desk in my office is probably only 15’ - 18’ feet away from the router, but there is a zigzag hallway, and a bathroom that block direct line of site. I really think stucco walls might have a metal lath that acts as a faraday cage, creating chaos for signals to reach that room.</p> The problem is the grievance bell is still ringing. At this point, our internet experience is like when bill & ted go to the future it’s all out of place. I’m streaming 4K, downloading docker images is a breeze, uploading is great, but it’s mainly the video conferencing, it’s still mainly shit.</p> Is it the Pi Hole? IDK its late 2023 figure i’ll drop the Pi Hole and pick up a Firewalla</a>. It will cover all the stuff I was doing on with the Pi Hole. DNS over HTTPS with a private Cloudflare endpoint, it has ad blocking and I can configure VPNs (though i’m not sold on this unless I was hosting my own and letting my traffic exit with AWS), it’s got VLANS, and can have HomeBridge</a> to manage all the IoT that i’ve picked up. Great! Was able to get all that running, but there still continues to be issues…</p> Well let’s figure out networking, all the backbone nics seems to be using 1GbE except for the Firewalla. There is some quirks and I think the agg to the Firewalla through the Wi-Fi modem is just not cutting it. Life at this point is more or less the same, except I’ve shelled out more money to have more or less the same experience that I was having back in 2019 (that’s not entirely true most internet experience is snappy. Except for Ad Heavy Sites such as Instagram (its really noticeable it’s a 50/50 crap shoot if content will load) The funny thing is if you switch to 5G it all loads quick. At this point my tiny home network looks something like this:</p> {{< image src=“static/sucky_network.png” caption=“Busted Home Network” alt=“Network Topology of my Home Network” width=“100%” >}}</p> Well, maybe the Ethernet devices don’t help the Wi-Fi-fi router. Created a new plan to pick up a switch and put that into the mix. Ended up fetching a QNAP sw-m408-4c</a> Pretty excited!! Eight 1GbE ports & four 10GbE. Now, I have enough ports to wire most my wired thing and hobbies. Now, I also get to negotiate true NBASE-T Ethernet 10 GbE with Mac Studio(😂 maybe one day my outbound network could absorb that). I also was able be able to configure dual 2.5GbE agg links to the Firewalla. This should be plenty enough to handle my network traffic.</p> {{< image src=“static/less_sucky_network.png” caption=“Less Busted Home Network” alt=“Network Topology of my Home Network” width=“100%” >}}</p> BufferBloat</h2> “Do your video or audio calls sometimes stutter? Does your web browsing slow down? Do video games lag? If so, bufferbloat may be to blame.</strong></p> What Is Bufferbloat? Bufferbloat is a software issue with networking equipment that causes spikes in your Internet connection’s latency when a device on the network uploads or downloads files.“</p> https://www.bufferbloat.net/projects/bloat/wiki/Tests_for_Bufferbloat/ https://bufferbloat-and-beyond.net/ This sounds like what i’m experiencing.</p> Reporting Results</h3> Mac Studio</h4> With the Mac Studio, I’m able to show that the network and general bandwidth is not an issue. Even when the agg link is two 1GbE connection, speeds are decent enough to get a Good rating from Cloudflare across the board. Bumping the agg links to the 2.5GbE(Firewalla nic’s max) I’m finally able to get use the full bandwidth from my ISP.</p> 10GbE -> 2GbE(Firewalla) -> Comcast 1M+GbE</h5> Cloudflare speed test</a> {{< image src=“static/speed_test_studio_2gbe.png” caption=“Cloud Flare Speed Test” alt=“Latency Report for my home 2GbE connection to Firewalla” width=“100%” >}}</p> 10GbE -> 5GbE(Firewalla) -> Comcast 1+GbE</h5> {{< image src=“static/speed_test_studio_5gbe.png” caption=“Cloud Flare Speed Test” alt=“Latency Report for my home using 5GbE connection to Firewalla” width=“100%” >}}</p> waveform bufferbloat test</a> {{< image src=“static/waveform_test_studio_5gbe.png” caption=“Waveform BufferBloat Test” alt=“BufferBloat Grading Repot my home using 5GbE connection to Firewalla” width=“100%” >}}</p> Flent</a> is a suite of tests we developed to diagnose bufferbloat and other connectivity problems. Because Flent has been tested to 40GigE, you can get a good feel for how the connection behaves while you tune your settings. In particular, Flent’s RRUL test</a> shows download and upload speeds and latency in one set of charts.</p> {{< image src=“static/flent_rrul_test_5gbe.png” caption=“Flent RRUL Test” alt=“Flent RRUL Graph of my home using 5GbE connection to Firewalla” width=“100%” >}}</p> Macbook</h4> With the macbook I’m 100% using Wi-Fi so I know there’s going to be some extra overhead over ethernet</p> Macbook Pro Wifi (5G) -> 1Gbe (Nic) -> 2GbE(Firewalla) -> Comcast 1+GbE</h5> waveform bufferbloat test</a> {{< image src=“static/waveform_test_macbook_wifi.png” caption=“Waveform BufferBloat Test” alt=“BufferBloat Grading Repot my home using wifi” width=“100%” >}}</p> {{< image src=“static/speed_test_studio_5gbe.png” caption=“Cloud Flare Speed Test” alt=“Latency Report for my home using WiFi” width=“100%” >}} Cloudflare speed test</a></p> The only consistency on WiFi is how inconsistent it is</h4> I ended up running back to back speed test test sitting on my couch while watching youtube through my TV(hardwired ethernet connection) & running the test from my macbook on the Wi-Fi. Again i’m only sitting about 6’ away from the router without any obstructions. This really shows the issue under a normal network load. I picked up that my Wi-Fi experience only worsens with moderate usage.</p> {{< image src=“static/speed_test_macbook_wifi_2.png” caption=“Cloud Flare Speed Test” alt=“Latency Report for my home using WiFi” width=“100%” >}}</p> {{< image src=“static/speed_test_macbook_wifi_3.png” caption=“Cloud Flare Speed Test” alt=“Latency Report for my home using WiFi” width=“100%” >}}</p> I continue to get warned about how bad my Wi-Fi is even google warns how bad my situation is.</p> {{< image src=“static/google_meet_terrible.png” caption=“Google Meet Showing My Connection Issues” alt=“Google Meet Latency Report for my home using WiFi” width=“100%” >}}</p> Next Steps</h2> Smart Queue with Firewalla</h3> I probably need to spend some time and energy to setup Smart Queue Features with the Firewalla. They recently release Cake</p> Dave Täht, co-founder of the Bufferbloat Project, commented: “The FQ-Codel (RFC8290) and the newer CAKE packet scheduling/AQM algorithms are nearly universal on servers and clients. They give the “little guy” – the small packets, the first packets in a new connection to anywhere – a boost until the flow achieves parity with other flows from other sources. DNS, gaming traffic, VoIP, videoconferencing, any new flow, to anywhere, get a small boost. That’s it. After that, all</em> network traffic gets treated equally. Big flows – from Netflix, Google, Comcast, your Mom, or to Timbuktu – all achieve parity</em>, with minimal delay and buffering, at the worldwide variety of real round trip times.“ 1</a></p> Firewalla has smart queue and with their latest release they have the option to use CAKE</p> CAKE (Common Applications Kept Enhanced) is a shaping-capable </span> queue discipline which uses both AQM and FQ. </span></code></pre> https://help.firewalla.com/hc/en-us/articles/360056976594-Firewalla-Feature-Smart-Queue#h_01H2TTZZ5B16YWQPWHDX8S5H7V</p> Network Segmentation</h3> Separate devices on to their own VLAN segments, Block IoT traffic to the internet</p> Anything Else I can think of</h3> Next steps could be to roll out any of the recommendations that BufferBloat article has on Wi-Fi hardware. Guess I pick something with openwrt</a> Really makes me think, if Ubiquity products also encounter these issues. I’ll also need to read more up on https://wiki.stoplagging.com/ as things like SQM only help to the ~350Mb connection</p> Tips & Tricks of the trade: video compression with ffmpeg Unknown — Thu, 26 Oct 2023 02:31:11 -0600 Tips & Tricks of the Trade: video compression with ffmpeg</h1> When you need to share a recording that isn’t ginormous I’m converting a video to GIF file with ffmpeg</code>:</p> ffmpeg </span>\ </span> -i</span> IMG_1288.MOV \ </span> -ss</span> 00:00:00.000 \ </span> -pix_fmt</span> rgb24 \ </span> -r</span> 10 \ </span> -s</span> 320x240 \ </span> -t</span> 00:00:10.000 \ </span> IMG_1288.gif </span></code></pre> #tips #video_compression #work</p> Building analytic NLP workflows with CI/CD, containers, & python on cloud.gov Unknown — Mon, 09 Oct 2023 18:39:10 -0600 Building analytic NLP workflows with CI/CD, containers, & python on cloud.gov</h1> I began this journey with familiar footing, providing reliable methods to enable data scientist to deliver insights on institutional data. The core fundamentals are not much different from delivering any other software product. There is a defined sandbox of system constraints on which shape the design, building, and operation of the overall system. We start with the fundamentals and build out with layers. To begin with we’re making sure the basics are followed like getting code and decisions documented in version control, making sure you and those are also operating with the same primitives to iterate and deliver value. From there we continue to layer on with understanding the prompt deliver against to explore is relatively simple; Explore how natural language processing can be used to evaluate data and identify duplicate values. Our data scientist whipped up a solution in a day or two and we’re done right? Not really, as underneath the initial experiment is the question of how to build a clear process or framework for the team to continuously deliver and compound our insights capabilities on the existing product.</p> Most of this work is net new, so there is some flexibility in how to pursue a solution. When I’m working on with greenfield technology products, I generally choose tooling that can be replicated locally and in the cloud. The current paradigm that I’m fond of is to pick a comfortable software language, bundle that up into a container image, apply some sort of version tags to that image, orchestrate getting that image to some sort of compute environment, and you’re good to go. From there, Developers can use the container image build locally, know what version is running in their various environments. From here, I explore the thought process and constraints in greater detail.</p> Check out the demo source code</a></p> The Constraints</h2> Just to reiterate what the to work through Build a greenfield mechanism to use NLP for processing entries with python scripts on cloud.gov infrastructure & the CI/CD tooling uses CircleCI.</p> Infrastructure</h3> Cloud.gov</a></h4> Let’s look into cloud.gov, why would teams build on this platform vs AWS gov cloud? It has two major benefits from the start yes first they handle all the infrastructure management, second they’re FedRAMP authorized. Teams inherently need less resources to build, deliver and manage the application lifecycle</p> Compliant from the start</a></h5> Cloud.gov offers a fast way for federal agencies to host and update websites, APIs, and other applications. Employees and contractors can focus on developing mission-critical applications, leaving server infrastructure management to us.</p> FedRAMP authorized</a></h5> Cloud.gov has a FedRAMP Joint Authorization Board (JAB) authorization, which means it complies with federal security requirements. When you build a system on cloud.gov, you leverage this compliance and reduce the amount of work you need to do.</p> Cloud.gov operates as a Platform as a Service (PaaS) abstraction on top of AWS, it enables developers to deliver applications efficiently without needing to worry about too much finagling with infrastructure. The good news is you don’t have to worry about most of the things SREs or Ops people build careers on worrying about such VPCs, subnets, ingress, egress, route table, firewalls, DNS, ec2, or ECS, EKS, access controls or whatnots. Cloud.gov of managed this by building their platform using Cloud Foundry.</p> Cloud Foundry</a></h4> Cloud Foundry has a CLI tool that enables developers to provision resources within an organization. There is also a Terraform provider that extends this functionality to have declarative Infrastructure as Code (IaC). A developer can easily add these to their CI/CD pipeline to automate the building, deploy, and maintenance processes of an application lifecycle hosted in cloud.gov. Operating with a proxy between the developer cloud.gov & AWS there are some quirks to navigate.</p> Limitations</h5> Some limitations with operating within a PaaS you kind of get what you get. For example, I built out a demo app using the SAP BTP trial platform</a> which also happens to operate on fundamentals provided by Cloud Foundry. SAP provides a free 90-day trial account, which was a great way to test some ideas. (By the way, incredible! No limits except for quotas, just build and use.)</p> There is usually a vendor provided library/marketplace of available service brokers</a> for use. For the SAP Trials account, there were 46 available for use, with the vast majority tailored as integrations into the SAP platform. For my needs, I use a more general purpose broker for Postgres there. With a couple of button clicks or a string of commands to the CLI, a fully managed Postgres database is deployed. An application can “bind” to a service to enable direct connectivity.</p> With a deployed application, logs are proxie’d through the service platform. This can be plagued with vague errors or terminology, but for the most part a developer can fetch the necessary logs for the deployed application, for service brokers not so much. Building any application with complex dependencies on service brokers could put you in a tricky spot trying to debug any problems not directly caused by the deployed application.</p> Cloud Foundry runs on the linux/amd64 x86 architecture. Building across platforms can be a cause for some headaches. I run a M1 Mac that uses darwin/arm64 which has produced numerous conflicts on compatibility when building things. For the past year or so I’ve run into some hiccup to architecture mismatches on some dependency, package, causing an unnecessary detour. The main thing that gets jammed up is that Cloud Foundry (the underlying open source project behind cloud.gov) uses buildpacks</a> in order to deploy applications.</p> I actually find builders and builders to be fascinating, I believe they will be the way forward, eliminating any need to meticulously craft a multi-stage container images. Which is exactly how I first approached tackling this demo.</p> Open Container Initiative (OCI)</h4> Open Container Initiative is a standard that was set in place by CNCF to ensure all container images follow the same protocol, this way, they can be used everywhere. I’m a big fan of using containers to encapsulate code and run in some sort of compute environment (pick your preference). After getting the NLP python script from the resident data scientist, my first step would be to bundle the code into a container. This is usually my first step when jumping into this greenfield initiative.</p> When it comes to building a container image the most familiar pattern is to build a Dockerfile there’s a ton of documentation out there, but in general use multi-stage builds to shrink the final product that will run. Also, if you can eliminate any means for shell access, do that too. Specific to building a python app check out pythonspeed.com</a> the guy knows a thing or two about containers and python and he crushes on guidance for building a custom python container image. I’ve somewhat arrived at my own personal preference for building images, largely influenced by his guidance. It boils down to using a base that updates and installs core dependencies for building the app, build the app, and then copy all the goodies to a final build image that is locked down from shell/ root access.</p> It generally looks something like this:</p> from</span> python:latest </span>as </span>build </span>run </span>apt-get install so many things </span>run </span>pip install -u pip wheel </span>#create a nice foundation like install wheels upgrade pip </span> </span>from build as package_installer </span>run </span>pip insatll all the things & delete all the cruft, maybe use a venv just so you know what you actually need </span> </span>from distroless:python as final </span>copy</span> --from build /venv /opt/venv </span>entrypoint [python app.py] </span></code></pre> [Inspect your container with Dive][dive]</p> Buildpacks</h4> This article provides a nice distinction between a Dockerfile and Buildpacks</a></p> TLDR; buildpacks provide an opinionated abstraction to build a container from sourcecode.</p> Instead of needing to meticulously handcraft the masterpiece that is your thousand line Dockerfile. You can pick a container builder like [paketo][paketo] leverage their python build pack to produce an equivalent image, what you were expecting with using distroless. Paketo advertises “Just bring your app and Paketo Buildpacks will detect what language your app is using, gather the required dependencies, and build it into an image.” There’s always some edge cases where something falls through the cracks (for me, it was trying to build PyTorch with CPU on amd64 architecture)</p> Cloud.gov also maintains and provided buildpacks, I can’t really speak towards the effiencey of provided buildpacks, but anything managed services or tooling we can leverage, reduces the complexity and overhead needed for long term support or maintenance.</p> CI/CD</h4> CircleCI</h5> Connecting repo</li> env secrets</li> git bot account</li> sbom</li> artifacts</li> publish to cloudfoundry</li> </ul> First contact</h2> My initial thought process was crank out a container image, use Cloud Foundry run-task command (which support docker images) and Bob’s your uncle right, clap our hands and were done. Of course, there would also need to be Static Code Analysis and container image scanning and potentially container runtime analysis to help alleviate the most concerns presented by any auditors. This could be paired with a Software Bill of Materials (SBOM) that generated on demand or with releases. Containers and Code can easily be version controlled with applying tags to your favorite container registry. This didn’t necessarily need to be a robust service (the current process handled entirely on a laptop) so I figured that the CI/CD or the native CRUD app could run the job on-demand or some cron. But of course there’s that saying no battle plan survives first contact…</p> After building the NLP container image, I mocked up an app to generate and populate fake data into Postgres database. This randomly generated a string, inserted into a table with a unique ID, and associated a unique user to that entry. Doing this allowed me to evaluate and see the NLP in action, identifying any duplicate records that that were created. Sweet! This design also more or less worked on the first or second try. Next, I tried to tie that into with CircleCI and deploy to Cloud Foundry. I immediately encountered disk quota issues. The container image being build by CircleCI was ~10G, at a loss as to why I dove in to see what the discrepency is. Of course CPU architecture is to blame Arm64 vs amd64, my local container image wound up somewhere between ~2G & 700Mb depending on how aggressive I got on cleaning up resources. Inspecting the image layers with [dive][dive] I was able to point fingers at biggest culprit for the added extra resources (Nvidia cuda for GPU processing) that are installed for amd64 versions of PyTorch. I now need to update the custom image to source the CPU version of PyTorch. Not an inherently difficult obstacle to overcome, patch my Dockerfile and carry on. But for some reason I continued to encounter various issues when deploying the docker image and determined I needed to understand in more depth as to what cloud.gov is doing with that docker image. It turns out it rebuilds the image with buildpacks.</p> Enter buildpack and a what is not a proverbial frustration and fascination with them.</p> {{< admonition type=note title=“When you deploy a docker image(OCI compliant) to cloud.gov it does not in fact deploy a docker image” open=false >}}</p> Runtime differences</a></h3> </blockquote> Pushing an application using a Docker image creates the same type of container in the same runtime as using a buildpack does. When you supply a Docker image for your application, Cloud Foundry</p> fetches the Docker image</li> uses the image layers to construct a base filesystem</li> uses the image metadata to determine the command to run, environment vars, user id, and port to expose (if any)</li> creates an app specification based on the steps above</li> passes the app specification on to diego (the multi-host container management system) to be run as a linux container.</li> </ol> </blockquote> No Docker components are involved in this process - your applications are run under the garden-runc</code> runtime (versus containerd</code> in Docker). Both garden-runc</code> and containerd</code> are layers built on top of the Open Container Initiative’s runc</code> package. They have significant overlap in the types of problems they solve and in many of the ways they try to solve them. For example, both garden-runc</code> and containerd</code>:</p> use cgroups to limit resource usage</li> use process namespaces to isolate processes</li> combine image layers into a single root filesystem</li> use user namespaces to prevent users with escalated privileges in containers from gaining escalated privileges on hosts (this is an available option on containerd</code> and is a default on garden-runc</code>)</li> </ul> </blockquote> Additionally, since containers are running in Cloud Foundry, most or all of the other components of the Docker ecosystem are are replaced with Cloud Foundry components, such as service discovery, process monitoring, virtual networking, routing, volumes, etc. This means most Docker-specific guidance, checklists, etc., will not be directly applicable for applications within Cloud Foundry, regardless of whether they’re pushed as Docker images or buildpack applications. {{< /admonition >}}</p> Using the buldkit</h3> The last statement in the documentation is what really caught me off guard. This creates the common scenario of works on my machine, logic that many teams can get into with drift between devices. I felt that this would break the any idempotency of delivering bundled in a container image and there was additional overhead of monitoring and validating container images, that is not present for the existing app that is using buildpacks. I began thinking it might be easier to deliver a web app with the cloud.gov build pack as thats what its designed to provide. Not having worked with buildpack I looked into what was needed. turns out not much. Really just need the code, a requirements.txt file and and entrypoint to the application (Procfile). I manually deployed this version of the application with the cloud.gov cli and ran into the same issues as before.</p> This is where I went down a couple rabbit holes until settling on a solution. The first was evaluating if I could correctly install PyTorch with poetry(thats a no) vs using the more common package manager pip. It turn out the way PyTorch manages their packages is just at odds. Poetry struggles with the nested explicit version that exclude the GPU packages and pip does the same with there being mismatched dependencies beween what gets installed and what needs to be used.</p> PyTorch vs spaCy</h3> This led me to look for alternatives to the PyTorch sentience-transformer. I came across spaCY which is a lightweight language model that can replace the functionality of what was originally being used in the NLP model. Once that was swapped I was able to successfully deploy the model well under the 2 GB container image I was building.</p> Building a flask demo</h3> I opted to show mock out a demo for this analysis so the team can evaluate usage. I began with a lightweight flask app that was easy to get started. I connected to the database and generated data. From the mock data I was able to query a column and output all the duplicates. Slapping on a super simple UI to add some templates, buttons, and some frontend javascript and of course set the background as NASA’s picture of the day. Run it from local and now there is a working demo of the data scientist NLP model assessing duplicates. How can this be useful beyond a demo? Make it an API</p> flask app</h4> lightweight simple to work locally</p> db connection</h4> create a reusable component to manage db connection</p> sample data</h4> control the outcomes</p> slap on simple UI</h4> add some templates, buttons, and some frontend javascript nasa picture of the day</p> working demo</h4> gunicorn delivers</p> Refactoring flask demo to fastAPI</h3> Drop the flask app and insert fastAPI, this required changing all the resposes to json. But you get the added benefit of getting a self documenting api page( same as what is seen in swagger/openapi).</p> add auth mechanism</h4> the hell that is a frontend auth token broker</h5> oauth2_scheme = OAuth2PasswordBearer(tokenUrl=“token”) HTTPBasic() encryption csrf_token frontend login vs api barer token</p> tagged based routes(or at least that how I think it works)</h4> seperation of auth routes from common routes. segmentation just sounded right in my brain</p> structure things as modules</h4> should have started with tests, but now we’ve got em</h4> middleware</h4> httpsredirect csrf_token</p> Database things</h5> Embracing first principles of agile work doesn’t need to be perfect, but get it out. Deliver the MVP first before having conversations about performance durability, and whatnot. Adding any greenfield services to existing products, needs to be eased into. There’s no need to rush, optimizing for a cashing queue until after first contact with the users.</p> Going to Prod</h5> Taking an application to production requires a lot of careful planning and testing. The application needs to be stable, secure, and able to handle the expected load. Here are top level considerations you should at least run through and think about:</p> Code Quality</strong></li> Performance</strong></li> Security</strong></li> Scalability</strong></li> Monitoring & Logging</strong></li> Disaster Recovery & Backup Strategy</strong></li> </ul> Remember that deploying to production is not the end of the development process but rather a new phase where you’ll need to monitor the system closely, gather user feedback, fix bugs, and continuously improve based on user needs and business goals.</p> Getting a demo [cloud.gov buildpacks]: https://cloud.gov/docs/getting-started/concepts/#buildpacks [cloudfoundry-community-buildpacks]: https://github.com/cloudfoundry-community/cf-docs-contrib/wiki/Buildpacks [dive]: https://github.com/wagoodman/dive [paketo]: https://paketo.io/</p> #blog #nlp #cloud_gov</p> Tips & Tricks: SSH Passthrough Unknown — Sat, 01 Jul 2023 00:00:00 -0600 SSH Passthrough</h1> Quick Reference Need to connect to a private host behind a jump host? Here are a couple of quick approaches:</p> 1. Use the -J</code> (ProxyJump) Option</h2> ssh -J</span> user@jump-host user@internal-host </span> </span></code></pre> This passes your SSH session through jump-host to reach internal-host without extra config files. 2. Utilize SSH Agent Forwarding</p> ssh -A user@jump-host</p> Now from jump-host:</h1> ssh user@internal-host</p> With -A, your SSH keys stay local, but are “forwarded” through jump-host to authenticate on internal-host.</p> dbt for Data Teams Unknown — Thu, 01 Jun 2023 00:00:00 -0600 Executive Summary</h1> The fast-paced, data-centric business environment necessitates agile, reliable, and effective data analytics solutions. This document provides a comprehensive overview advocating for the implementation of Data Build Tool (dbt™) as a strategic asset in your data stack, particularly when used in conjunction with AWS services like Redshift, Glue, and S3, and integrated into CI/CD pipelines through platforms like Github, Azure DevOps, or others.</p> dbt empowers data teams to create production-ready data pipelines with ease, while adhering to software engineering best practices. It serves as a catalyst in eliminating data silos, ensuring data integrity, fostering collaboration, and streamlining the deployment of data products. dbt’s compatibility with AWS services amplifies its utility. When used with a data warehouse like Amazon Redshift, dbt optimizes data transformations and analytics. In combination with AWS Glue, it provides a streamlined, end-to-end ETL process. Its flexibility with AWS S3 offers additional storage options, making it a versatile tool in a data architect’s toolkit. While the technology does present a learning curve and may have limitations for extremely large datasets, the benefits significantly outweigh the challenges. The document argues that adopting dbt is not just a technological decision but a business imperative for organizations striving to be data-driven in today’s competitive landscape. By integrating dbt into your data architecture, organizations can achieve faster, more reliable, and cost-effective data analytics, thereby accelerating their data capabilities.</p> Introduction</h1> While the company dbt Labs, Inc. does have multiple offerings through their SaaS product, for all purposes within in this document we are referencing the open-source tooling dbt Core, a binary file that can be installed on user or machine systems(via software package management tools like pip or homebrew). dbt Core ships with a command-line interface (CLI) for running your dbt project. The dbt CLI is free to use and available as an open source project(https://github.com/dbt-labs/dbt-core</a>).</p> Intended as a tool to help data engineers, data analysts, and data architects, this document advocates for the implementation of Data Build Tool (dbt™) alongside AWS services like Redshift and Glue. Implementing dbt can lead to the creation of agile, reliable, and effective data products. The modern business landscape requires organizations to move fast, make data-driven decisions, and be adaptive. Below is a framework that sets forth reasons why dbt should be a part of your data stack to ship trusted data products faster. </p> What is dbt™?</h2> https://www.getdbt.com/product/what-is-dbt</a> </p> dbt is a SQL-first transformation workflow that lets teams quickly and collaboratively deploy analytics code following software engineering best practices like modularity, portability, CI/CD, and documentation. Enabling anyone on the data team to safely contribute to production-grade data pipelines. </p> Analysts using dbt can transform their data by simply writing select statements, while dbt handles turning these statements into tables and views in a data warehouse. These select statements, or “models”, form a dbt project. Models frequently build on top of one another – dbt makes it easy to manage relationships between models, and visualize these relationships, as well as assure the quality of your transformations through testing.</p> Engineers and Analysts can deploy safely to dev environments testing out their assumptions and completing their assertions on the data. Git-enabled version control enables collaboration and a means to return to previous states, but also fosters an ecosystem in which data team members can easily get their code peer reviewed. Within the CI/CD pipeline teams are able to test every model prior to production, and share dynamically generated documentation with all data stakeholders. Teams can write modular data transformations in .sql or .py files – dbt handles the chore of dependency management that can easily be reused. </p> What dbt™ isn’t?</h2> dbt is not a data warehouse or a database itself, but rather a tool that can be used in conjunction with a data warehouse to make it easier to work with and manage data. Additionally, dbt is not a programming language, but it does use a programming-like syntax to specify how data should be transformed and loaded into a data warehouse. It is also not a visualization tool, although it can be used in conjunction with visualization tools like Tableau or Looker to help users understand and analyze their data</p> Why dbt™?</h2> Eliminate Data Silos</h3> Now your data teams can build models that connect with those built by the analytics team, each using the language they prefer. dbt supports modeling in SQL or Python, enabling a shared workspace for everyone that works on analytic code. </p> Reliability and Repeatability</h3> dbt’s “version control integration” means that all changes to data models are tracked. This is crucial in ensuring that past versions can be quickly restored, guaranteeing data integrity. With dbt data teams can build observability into transformation workflows with in-app scheduling, logging, and alerting. Protection policies on branches ensure data moves through governed processes including dev, stage, and prod environments generated by every CI run.</p> Self-documenting Code</h3> In dbt, metadata about data models can be embedded directly into the code. For example, you can include annotations within SQL files, which dbt will use to auto-generate documentation. This visibility is beneficial for cross-functional teams to understand data models. </p> Collaboration</h3> dbt’s integration with Git for version control fosters collaboration. Team members can work on parallel branches and easily merge changes, encouraging a more team-based approach. </p> Code Reviews</h3> For data engineering projects, code reviews ensure that the data transformations adhere to predefined quality standards, making the data more trustworthy. </p> Extensive Community Ecosystem</h3> Through dbt Hub(https://hub.getdbt.com/</a>) teams building with dbt can leverage community packages(from contributors like dbt Labs, Fivetran, Ay to refine the raw data in your warehouse. These community-available packages provide SQL macros that can be (re)used across dbt projects, from useful macros for performing data audits, to base models for Redshift System tables, or sophisticated privacy transformations that reduce reidentification risk but leave data usable for analysts. </p> Limitations</h2> While dbt offers powerful features, it has a learning curve and requires a cultural shift toward treating data as code. Furthermore, for very large datasets, dbt’s in-database processing may limit performance.</p> How dbt™ Fits with CI/CD</h2> Continuous Integration/Continuous Deployment (CI/CD) is a best practice in modern software engineering that involves automatically building, testing, and deploying code changes to production environments. In the context of data analytics, Data Build Tool (dbt) is an equivalent for data transformation, testing, and documentation. When dbt is integrated into a CI/CD pipeline, it enables automated data testing, version control, and deployment, thereby offering a comprehensive solution for agile data analytics. Azure DevOps serves as a robust platform to implement this integration, providing a variety of tools for code repository management, build and release pipelines, and more. Here’s how dbt fits seamlessly with CI/CD, particularly when using Azure DevOps.</p> Automated Testing</h3> Pre-Commit Hooks: Before a developer pushes code to the repository, pre-commit hooks can run dbt tests locally to ensure data models are correct.</p> </li> Pipeline Steps: As part of the CI/CD pipeline in Azure DevOps, automated tests can be configured to run every time there’s a code change. This ensures that the data models and transformations are correct before they are deployed.</p> </li> Quality Gates: dbt tests can act as quality gates in the pipeline. If a test fails, the pipeline can be configured to halt, preventing the deployment of faulty data models.</p> </li> </ul> Version Control</h3> dbt Project in Azure Repos: Your dbt project can be stored in Azure Repos, which allows it to benefit from version control, branching, and pull requests, just like any other codebase.</p> </li> Branch Policies: Azure DevOps allows you to set up branch policies, ensuring that dbt models can only be merged after passing specified criteria, like code reviews and successful test runs.</p> </li> Artifact Management: The built and tested dbt artifacts can be stored in Azure Artifacts, providing a historical record and facilitating rollbacks if necessary.</p> </li> </ul> Deployment Automation</h3> Parameterized Runs: Azure DevOps pipelines can be parameterized to deploy dbt models to various environments (e.g., dev, staging, prod) based on the branch being merged.</p> </li> Automated Rollbacks: If a dbt model fails in a higher environment like staging or production, Azure DevOps pipelines can be configured to automatically rollback to the previous stable version.</p> </li> Monitoring and Notifications: Azure DevOps provides monitoring tools and can be set to send notifications if the pipeline fails or if there are issues with the deployed dbt models.</p> </li> </ul> Collaboration and Access Control</h3> Role-Based Access: Azure DevOps supports role-based access control, allowing you to specify who can perform actions like triggering pipelines or merging dbt models.</p> </li> Collaboration Features: Azure Boards can be used for task tracking, linking your data tasks directly to your dbt development efforts.</p> </li> Documentation: dbt’s inherent documentation generation can be integrated into the CI/CD process, keeping all stakeholders informed about the current state of the data models.</p> </li> </ul> dbt and CI/CD together form an effective methodology for agile data transformation and analytics. By leveraging Azure DevOps as the platform for implementing this integration, organizations can automate data quality checks, maintain version control, and streamline the deployment process, resulting in faster, more reliable data analytics pipelines.</p> Complementing AWS Services</h2> Redshift: dbt and Redshift can optimize data transformations and analytics.</h3> Amazon Redshift is a fully managed data warehouse service, optimized for running complex queries and performing data analytics tasks. One of its key features is the ability to create materialized views, which pre-compute and store query results for faster retrieval. dbt can take advantage of Redshift’s features like materialized views to optimize data transformations. By utilizing these features, dbt models can be designed to run much faster, which is especially useful when dealing with large data sets or complex transformations. This can lead to reduced compute costs and faster time-to-insight for data analytics. Redshift uses a PostgreSQL-compatible SQL syntax, which is also the primary language for defining transformations in dbt. This compatibility means that you can seamlessly move SQL code between dbt and Redshift, thereby speeding up the development process. dbt’s built-in version control capabilities enable team members to collaborate on a single, unified model that leverages Redshift’s power. This ensures that changes are transparent, trackable, and reversible, adding a layer of reliability to your data processes.</p> Glue: Glue handles data ingestion, while dbt™ can focus on the transformation layer.</h3> AWS Glue is a fully managed extract, transform, and load (ETL) service that makes it easy for users to prepare and load data for analytics. AWS Glue can focus on data ingestion and schema management, preparing the raw data for subsequent transformations. Once the data is ingested into Redshift or another data warehouse, dbt can then take over to perform the transformations, tests, and data quality checks. This separation of concerns creates a modular data pipeline that is easier to manage, monitor, and scale. Glue’s Data Catalog feature acts as a centralized repository for metadata, allowing seamless transition into dbt. This improves discovery and enhances governance, as dbt can also document transformations, tests, and lineage. Glue and dbt complement each other by creating a more streamlined, end-to-end ETL process. Glue handles the initial steps, while dbt optimizes the data transformation layer, resulting in a more agile and efficient workflow.</p> S3: dbt™ can also read from and write to AWS S3 buckets, making it versatile in handling various data storage solutions.</h3> Amazon S3 (Simple Storage Service) is a widely used object storage service, suitable for storing large volumes of unstructured data. dbt’s capability to read from and write to S3 buckets adds an extra layer of versatility. This means you’re not confined to using a data warehouse for storage; you can also use S3 for raw or transformed data, giving you more options in how you architect your data solutions. Many organizations use S3 as a data lake to store raw data. dbt’s compatibility with S3 allows it to directly read this raw data, transform it, and either load it into a data warehouse like Redshift for analytical workloads or write it back to S3 in a more structured format. S3 provides a cost-effective storage solution, especially for long-term data retention. By using dbt to write transformation outputs back to S3, you can optimize costs for storage while maintaining high data quality and accessibility.</p> Conclusion</h2> In the fast-paced, data-centric business environment of today, the need for agile, reliable, and effective data analytics solutions cannot be overstated. The dbt emerges as a game-changer in this context, offering a robust framework that fosters collaboration, enhances data integrity, and expedites the deployment of data products. Its SQL-first transformation workflow, Git-enabled version control, and built-in CI/CD capabilities make it an indispensable tool for data professionals including data engineers, analysts, and architects.</p> dbt’s compatibility with AWS services further amplifies its efficacy. When paired with Amazon Redshift, dbt leverages advanced data warehousing capabilities to optimize transformations and analytics. Its synergy with AWS Glue simplifies the ETL process, creating a streamlined workflow from data ingestion to transformation. The ability to interact with AWS S3 offers additional storage flexibility, extending its utility across different storage paradigms.</p> Despite its learning curve and potential limitations for handling extremely large datasets, the advantages of implementing dbt, especially in conjunction with AWS services, far outweigh the challenges. With its focus on eliminating data silos, enforcing data reliability, enabling self-documenting code, and encouraging collaboration, dbt acts as a catalyst in the evolution of modern data stacks. When integrated into a CI/CD pipeline using platforms like Azure DevOps, it adds another layer of automation and governance, making your data operations not just agile but also resilient.</p> In summary, dbt should not be seen merely as an option but rather as a strategic asset for any organization serious about accelerating its data capabilities. Its power to transform data engineering practices, coupled with its seamless fit within the broader AWS ecosystem, makes it a compelling choice for modernizing your data stack. Adopting dbt is not just a technological decision; it’s a business imperative for organizations aiming to be data-driven in a competitive landscape.</p> Appendix</h1> Use cases/ Articles</h2> Build your data pipeline in your AWS modern data platform using AWS Lake Formation, AWS Glue, and dbt Core</h3> dbt</a> has established itself as one of the most popular tools in the modern data stack, and is aiming to bring analytics engineering to everyone. The dbt tool makes it easy to develop and implement complex data processing pipelines, with mostly SQL, and it provides developers with a simple interface to create, test, document, evolve, and deploy their workflows. For more information, see docs.getdbt.com</a>.</p> dbt primarily targets cloud data warehouses such as Amazon Redshift</a> or Snowflake. Now, you can use dbt against AWS data lakes, thanks to the following two services:</p> AWS Glue Interactive Sessions</a>, a serverless Apache Spark runtime environment managed by AWS Glue</a> with on-demand access and a 1-minute billing minimum</p> </li> AWS Lake Formation</a>, a service that makes it easy to quickly set up a secure data lake</p> </li> </ul> In this post, you’ll learn how to deploy a data pipeline in your modern data platform using the dbt-glue adapter built by the AWS Professional Services team in collaboration with dbtlabs.</p> With this new open-source, battle-tested dbt AWS Glue adapter, developers can now use dbt for their data lakes, paying for just the compute they need, with no need to shuffle data around. They still have access to everything that makes dbt great, including the local developer experience, documentation, tests, incremental data processing, Git integration, CI/CD, and more.</p> https://aws.amazon.com/blogs/big-data/build-your-data-pipeline-in-your-aws-modern-data-platform-using-aws-lake-formation-aws-glue-and-dbt-core/</a> </p> How SafetyCulture scales unpredictable dbt Cloud workloads in a cost-effective manner with Amazon Redshift</h3> SafetyCulture runs an Amazon Redshift provisioned cluster to support unpredictable and predictable workloads. A source of unpredictable workloads is dbt Cloud, which SafetyCulture uses to manage data transformations in the form of models. Whenever models are created or modified, a dbt Cloud CI job is triggered to test the models by materializing the models in Amazon Redshift. To balance the needs of unpredictable and predictable workloads, SafetyCulture used Amazon Redshift workload management (WLM) to flexibly manage workload priorities.</p> With plans for further growth in dbt Cloud workloads, SafetyCulture needed a solution that does the following:</p> Caters for unpredictable workloads in a cost-effective manner</p> Separates unpredictable workloads from predictable workloads to scale compute resources independently</p> Continues to allow models to be created and modified based on production data</p> https://aws.amazon.com/blogs/big-data/how-safetyculture-scales-unpredictable-dbt-cloud-workloads-in-a-cost-effective-manner-with-amazon-redshift/</a> </p> Manage data transformations with dbt in Amazon Redshift</h3> In this post, we demonstrate some features in dbt that help you manage data transformations in Amazon Redshift. We also provide the dbt CLI and Amazon Redshift workshop to get started using these features.</p> https://aws.amazon.com/blogs/big-data/manage-data-transformations-with-dbt-in-amazon-redshift/</a> </p> Automating deployment of Amazon Redshift ETL jobs with AWS CodeBuild, AWS Batch, and dbt™</h3> In this post, we show you how to automate the deployment of Amazon Redshift</a> ETL jobs using AWS Batch</a> and AWS CodeBuild</a>. AWS Batch allows you to run your data transformation jobs without having to install and manage batch computing software or server clusters. CodeBuild is a fully managed continuous integration service that builds your data transformation project into a Docker image run in AWS Batch. This deployment automation can help you shorten the time to value. These two services are also fully managed and incur fees only when run, which optimizes costs.</p> We also introduce a third-party tool for the ETL jobs: dbt™</a>, which enables data analysts and engineers to write data transformation queries in a modular manner without having to maintain the execution order manually. It compiles all code into raw SQL queries that run against your Amazon Redshift cluster to use existing computing resources. It also understands dependencies within your queries and runs them in the correct order. dbt™ code is a combination of SQL and Jinja (a templating language); therefore, you can express logic such as if statements, loops, filters, and macros in your queries. For more information, see dbt™ Documentation</a>.</p> https://aws.amazon.com/blogs/big-data/automating-deployment-of-amazon-redshift-etl-jobs-with-aws-codebuild-aws-batch-and-dbt/</a></p> Accelerating Data Teams with dbt Cloud & Snowflake</h3> Modern businesses need modern data strategies, built on platforms that support agility, growth and operational efficiency.</p> Snowflake is the Data Cloud, a future-proof solution that simplifies data pipelines, so you can focus on data and analytics instead of infrastructure management.</p> dbt is a transformation workflow that lets teams quickly and collaboratively deploy analytics code following software engineering best practices like modularity, portability, CI/CD, and documentation. Now anyone who knows SQL can build production-grade data pipelines. It transforms data in the warehouse, leveraging cloud data platforms like Snowflake.</p> In this Quickstart, you will follow a step-by-step guide to using dbt with Snowflake, and see some of the benefits this tandem brings. https://quickstarts.snowflake.com/guide/data_teams_with_dbt_cloud/#0</a> </p> dbt (Data Build Tool) Overview: What is dbt and What Can It Do for My Data Pipeline?</h3> There are many tools on the market to help your organization transform data and make it accessible for business users. One that we recommend and use often—dbt (data build tool) —focuses solely on making the process of transforming data simpler and faster. In this blog we will discuss what dbt is, how it can transform the way your organization curates its data for decision making, and how you can get started with using dbt (data build tool).</p> https://www.analytics8.com/blog/dbt-overview-what-is-dbt-and-what-can-it-do-for-my-data-pipeline/</a></p> Activating ownership with data contracts in dbt</h3> Data mesh, data contracts and shifting ownership to data producers has been presented as the solution. Data teams have largely brought into this promise but only a fraction can confidently say that they’ve seen the expected impact from their efforts. One of the reasons for this is that contracts introduce yet another technical and organizational burden for already stretched engineers.</p> With dbt 1.5 and the support for data contracts, data teams have the opportunity to roll out contracts themselves. While it doesn’t solve the full problem it provides a way to enforce quality at the intersection of teams that can be rolled out in days instead of months.</p> https://medium.com/@mikldd/activating-ownership-with-data-contracts-in-dbt-4f2de41c4657</a></p> #blog</p> Tips & Tricks: Super Delete S3 Objects Unknown — Thu, 01 Jun 2023 00:00:00 -0600 When you need to delete 10s of 1000’s of s3 objects</p> export </span>BUCKET</span>=</span>ecs-dbt-dev-staging-bucket </span>export </span>PREFIX</span>="</span>Connecticut/EmplId-Empl</span>" </span>aws</span> s3api list-objects-v2</span> --bucket </span>$</span>BUCKET --prefix </span>$</span>PREFIX --output</span> text</span> --query </span>\ </span>'</span>Contents[].[Key]</span>' | </span>grep -v -e </span>"</span>'</span>" | </span>tr </span>'</span>\n</span>' '</span>\0</span>' | </span>xargs -0 -P2 -n500</span> bash</span> -c </span>\ </span>'</span>aws s3api delete-objects --bucket $BUCKET --delete "Objects=[$(printf "{Key=%q}," "$@")], Quiet=true"</span>' _ </span></code></pre> It's still day 1... Unknown — Sun, 07 May 2023 21:39:46 -0600 Backstory</h2> How does one self manage their thoughts, time, aspirations, or memories? Recently, while I was Facetiming my brother I noticed that he had hand written notes on one of his hands. I had an instant flashback to high school where I often did the same thing for tracking phone numbers, assignments, or just drawing a face (though my brother is nearing his 30s). I realized i’ve been hacking through ways to track and tame my brain’s musings, upcoming tasks, and memories for years. I’ve gone through numerous phases such as photographing everything with a DSLR, carrying back pocket note cards and a space pen, there was the fountain pen era, post-it notes, journals, legal pads, whiteboards, todo lists, post-its, kanban boards. Year after year I usually find there is always something a bit lacking with me and my process.</p> First up is me, I’m a bit scatter brained often moving from one shiny object to the the next. I do really well when I have structure and I know all the i’ve done all my 7Ps (prior proper planning prevents piss poor performance). The end result is striving to do better and technology help with this, key point is tagging anything I can with Air Tags. Finding my keys no big deal, sunglasses on the other hand i’m probably on my third attempt scouring the house. What this actually means is that I’ve spent years building out processes that work for me and tossing out what doesn’t. Right now I feel like i’m in an aggressive Marie Kando phase.</p> Rule One: No Loosies</h3> Legal Pads, post-it notes, blank sheets of paper are a no go. I got myself into a bad habit of scribbling little notes(actually conceptually important trinkets of knowledge) in piles or heaps on my desk. My shiny object brain know has a weird spider sense on knowing where I remember some context that I had tracked there. That system had to go, all it did for me was build clutter in my workspace forcing me to dig around looking for that one legal pad I wrote my note on three months ago which now found itself acting as the wheel block so the Lego Bugatti doesn’t roll off the shelf.</p> Rule Two: Context Segmentation</h3> I’ve got three daily driver notebooks, one I use to scribble out a diagrams real quick, a recipe cookbook, and travel companion. I don’t think these need to go to the waste side. Keeping ideas or thoughts compartmentalized in journals is useful, you still get the tactile experience of putting pen to paper to capture in the moment. I like to call this napkin math, a means capture ephemeral ideas not fully formed, they might need to stew for little more time. Gone is the time for notebooks & journals to be scattered throughout my office, kitchen, and various bags. For the most part these are in place to act as a brain aid, not to be used for record keeping.</p> Rule Three: Consolidate the Digital Brain</h3> For a while, I found myself a groove with using OneNote and was pretty consistent user (albeit a very basic user). I found comfort in forming a ritualistic reliable means of keeping a daily activity log, what I was going to to next, planning my projects kept me really focused. However, changing jobs, a major data loss incident due to OneNotes weird syncing, their haphazard migration to cloud product, my migration from android to iOS, and lastly to various nit picks with the OneNote product, I gave up on aspirations OneNote being my source of truth for digital note taking.</p> Since then my note taking experience has been subpar, I’ve slowly fragmented across services like Apple notes, Google Keep, Evernote, Notion, OneNote, Bear, email, or for a while just google docs. I’ve spent five or so years trying to rebuild what I perceive as a good habit. This year I decided i’d attempt reset the foundations though, with a bit more of a constructive focus. I started out giving roam research</a> a shot it immediately didn’t take. I actually dusted off OneNote again (again its been like a half decade) and it was like reuniting with a familiar friend with improvements. It sync was better and for my initial purpose of tracking notes, logs, and what haves it was doing just fine. I was bringing back that structure that I was looking to have.</p> Around this time I was seeing similar conversations in my companies slack channels with whispers and conversations about what people are doing about notes. This one use Obsidian, or another on Notion, Roam and a few others also kept popping up. I wasn’t completely enamored with OneNote as I’ve been burned in the past. I’d figure I could install Obsidian and see what the fuss was about. I got off to a slow start, I didn’t dive too deep into the product and really only knew it supported markdown notes (my current preferred digital format). I let it sit on the shelf for a couple months.</p> Rule Four: Use Tooling You Like</h3> I’m fond of mocking up my understanding of how something works by drawing some lines accompanied with some notes. Don’t know why but if you give me some lines and a list I can sigh with relief that i’m going to be able to follow along, if you add some color i’m totally smitten. I’m particularly a fan of the robust offerings available to aid these efforts. Products Mermaid Diagrams, Plantuml, Mural, LucidCharts and others have all aided me at some point. Couple some of those tools with principles of C4 context diagrams. You now have a programatic means to convey logical understanding various participant personas.</p> People and Teams can programmatically create/generate diagrams reflect the logical structures of the applications, databases, or an entire code base. With CI/CD tooling updates the codebase can be reflected on every merge keeping these diagrams versioned throughout the application lifecycle. Though, when attempting to convey understanding to a wider audience that may not as in-tune with a codebase, newly learning ideas, or just might not be so technical. I’ve noticed that color coordinated and imaged based diagrams help to bridge the gap. By giving people something to attach to and recognize people may have an easier time to follow along. You can see this with content produced by ByteByteGo</a> or the team with Swirl AI. I personally gravitated to the content that Aurimas Griciūnas of Swirl AI</a>was producing. I absoutely needed to know what he was using to build his diagrams. I reach out to found he’s building things out about Excalidraw. I immediately started using it to mock out my understanding of complex services ideas.</p> Rule Five: Document and Share your experiences</h3> Really that it… Make some content good or bad and share that out.</p> Lightbulbs</h3> To catch things back up, I’m doodling in designated notebooks, I’m re-establishing good digital habits(mostly in OneNote), and at this point i’m newly mocking up Excalidraw diagrams which also coincides to writing content for internal engineering blog articles and presentations. During my Digital Kando phase former colleague of mine was a prolific sharer of tech tidbits, articles, repos, and tweets all related to software engineering or just cool things he found. At the company his shares gained somewhat of a internal cult following, with a dedicated slack channel. He had just implemented an RSS feed for all his shares(there’s a lot). I was curious how he was managing this firehose of information. So the secrets out…he’s just using Obsidian to take note of anything he finds cool then converts the notes to a website. See his site and rss feed: notes.billmill.org</a> Or check out the source code</a></p> This was largely beginning of my the light bulb moment/s. I needed to doubled down on Obsidian, ditch OneNote, and maybe I could just do the same but submit content to substack. This is were I went exploring the community plugins and the top download is local version Excalidraw with the ability community scripts to wrangle diagrams. Game over.. I’m making this happen: Obsidian notes based dev blog with diagrams, just need to sort out the little bits.</p> Dev Blog</h2> I have an on again off again relationship with dev blogs. I think this is my third iteration of hosting one, after a venture using umbraco, then ghost, and now Hugo(i’m not counting geocities). Let’s see if it last more than a month. 🤞</p> This time I at least have a scope: host dev blog using a static content generator, use CI/CD to provision IaC, do it serverless, live update content from Obsidian notes.</p> Static Generator</h3> So for for choosing a static website generator I went Hugo, it seemed popular and there a ton of themes with various support. I picked the LoveIt theme as it looked basic enough and had dark mode.</p> Hosting</h3> Netlifty hosting is certainly the easiest, and AWS Amplify was a quick 2 stepper to get working. That’s not quite what I wanted while it was easy. I was looking to surface the unseen leavers behind those services leavers being pulled or at least to some degree. So, there were a couple hiccups along my way. Like needing to setup an AWS org so I could use OIDC role logins (for myself and machine roles), going through the new AWS Identity Center SSO. Coordinating obsidian git sync was both straight forward to setup and awful, but I fee like I read into it too much, so had some churn there. Live updating on content pushes to GitHub was actually way easy, until i thought about immutable deploys. I still think my most churn hard parts were head banging moments on security headers being blocked and invalidating all the CDN content.</p> Look! Embedded diagram mocked up in my note taking app:</p> {{< image src=“static/the_easy_way.png” caption=“The Easy Way” alt=“Architecture Layout of Dev Blog using Obsidian Hugo and AWS” width=“100%” >}}</p> {{< image src=“static/the_irritating_way.png” caption=“The Irritating Way” alt=“Architecture Layout of Dev Blog using Obsidian Hugo and AWS” width=“100%” >}}</p> Live pushes from obsidian</h3> Obsidian it pretty good with its auto commits and backup to the remote origin GitHub.</p> {{< image src="static/obsidian_git.png" caption="Obsidian Source Control" alt="Obsidian Source Control" height="50" width="50" >}} {{< image src="static/unpublished_git.png" caption="Obsidian Auto Commit" alt="Obsidian Auto Commit" height="50" width="50" >}} </div> After coordinating the git sync between all my devices and GitHub. I setup a GitHub actions trigger to refresh the content on the fly. I originally started with using Repository Dispatch to manage the event which worked without issues.</p> name</span>: </span>Trigger Hugo Update on Push to Main </span> </span>on</span>: </span> </span>push</span>: </span> </span>branches</span>: </span> - </span>main </span> </span>paths</span>: </span> - '</span>the_archives/</span>' </span>jobs</span>: </span> </span>trigger_hugo_update</span>: </span> </span>runs-on</span>: </span>ubuntu-latest </span> </span> </span>steps</span>: </span> - </span>name</span>: </span>Commit empty message to target repo </span> </span>run</span>: </span>| </span> git config --global user.email "github-actions@example.com" </span> git config --global user.name "GitHub Actions" </span> git clone "https://${{ secrets.PAT }}@github.com/kcirtapfromspace/kcirtap.io.git" </span> cd kcirtap.io </span> git checkout main </span> git commit --allow-empty -m "docs: Obsidian Disbatch "ref": "${{ github.ref }}", sha: ${{ github.sha }}" </span> git push </span> </span># - name: Repository Dispatch </span> </span># uses: peter-evans/repository-dispatch@v2 </span> </span># with: </span> </span># token: ${{ secrets.PAT }} </span> </span># repository: kcirtapfromspace/kcirtap.io </span> </span># event-type: obsidian_push </span> </span># client-payload: '{"ref": "${{ github.ref }}", "sha": "${{ github.sha }}"}' </span></code></pre> I ended up using a custom git action to send empty commit to the website repos as it provides a more informative message within github actions. It also played a little nicer with incremental versioning action please-release. trigger_screenshot</p> {{< image src=“static/trigger_screenshot.png” caption=“Repository Disbatch vs Empty Commit” alt=“Repository Disbatch vs Empty Commit” width=“100%” >}}</p> Here is the sequence what I ended up putting together.</p> {{< image src=“static/sequence_diagram.png” caption=“Repository Disbatch vs Empty Commit” alt=“Sequence Diagram” width=“100%” >}}</p> immutable production clone</h4> I have a folder in my obsidian tree that I use to build the site content. New content applied to the directory triggers builds for blog site. Immutable deploys are handled by please-relase action. This action creates a PR for these PRs production steps are used to fetch the last empty commit with the correct content commit SHA. This commit is then used to sparse clone only the content directory into the build action. Content is injected into the Hugo build step to generate the static site. With the Release PR the site is published and the CDN cash is invalidated.</p> git</span> config</span> --global</span> user.email "</span>github-actions@example.com</span>" </span>git</span> config</span> --global</span> user.name "</span>GitHub Actions</span>" </span>git</span> clone</span> --filter</span>=blob:none</span> --no-checkout --sparse</span> git@github.com:kcirtapfromspace/obsidian.git obsidian </span>cd</span> obsidian </span>git</span> sparse-checkout init</span> --cone </span>git</span> fetch origin d13113fdc7b31b1d46bedd197a906ae90553c2a8 </span>git</span> checkout d13113fdc7b31b1d46bedd197a906ae90553c2a8 </span>git</span> sparse-checkout set the_archives </span>ls -lah </span>mv</span> the_archives ../src/content/ </span></code></pre> Initial nits</h3> I am a bit peeved that I need to use HTML to render the images correctly. Native markdown seems to mangle the image. There is a quirk with “ShortCode” that allows code comments to render the images inside fencing if using {{}}</code> The Hugo theme I picked doesn’t support svg rendering, so that another rabbit hole to dive down to navigate partial layout overrides.</p> < image src="static/the_easy_way.png" caption="The Easy Way" alt="Architecture Layout of Dev Blog using Obsidian Hugo and AWS"width="100%" > </span> </span>< image src="static/the_irritating_way.png" caption="The Irritating Way" alt="Architecture Layout of Dev Blog using Obsidian Hugo and AWS"width="100%" > </span></code></pre> WIP: the little bits</h2> Obsidian Note Taking Live git commit & sync(across devices via GitHub) Write Markdown Notes, Blogs, Musings Template Generator Excalidraw</p> Static site generator (needs markdown support) Deploy Wireframe site Terraform State s3 bucket dynamodb make sure your table names in terraform match up infra Route53 Static Site S3 bucket Cloud Front CDN origin story Ciphers fighting with ciphers and other whatnots Security Headers X-site scripting Edge Lambda (rewrite paths) WAF Certificate Needs san name to pass KMS encrypt the bucket, dynamodb table IAM Github Federated Idp Well known fingerprint Github Actions Assume Role ci/cd github actions terraform deploy sparse clone for hugo deploy uses deploy key to fetch private repo obsidian event push action uses pat token CloudFront Al release triggers are hard and finicky</p> Tiers of compliance Unknown — Mon, 01 May 2023 16:22:46 -0600 Regulatory Compliance in the Cloud</h1> Compliance is a critical aspect of any organization’s operations, particularly when it comes to data security and privacy. In the realm of cloud computing, compliance requirements can be complex and varied, depending on the industry and regulatory environment. This note provides an overview of the different tiers of compliance, including FISMA, FedRAMP, and HIPAA, and outlines the specific control areas that organizations must address to achieve compliance. By understanding these requirements, organizations can ensure that their cloud infrastructure meets the necessary standards for data security and privacy.</p> FISMA Compliance</h2> Control Area</th> FISMA Low</th> FISMA Moderate</th> FISMA High</th></tr></thead> Identity & Access Management</td> - Basic IAM policies - MFA for privileged users - Regular access reviews.</td> - More stringent IAM policies - MFA for all users - Role-based access control - Regular access audits - Temporary credentials for short-term access</td> - Enhanced user monitoring - Continuous access auditing - Additional access restrictions - Session duration limitations - Just-In-Time (JIT) access</td></tr> Data Encryption</td> - Encryption at rest (KMS, S3, EBS) - Encryption in transit (TLS)</td> - More stringent encryption key management - Automated key rotation - Dedicated KMS Customer Master Keys (CMKs)</td> - Enhanced encryption algorithms - Hardware security modules (HSM) integration - Stronger key management policies - Key access and usage logging</td></tr> Network Security</td> - Basic VPC setup - Security groups - Network ACLs</td> - Enhanced VPC isolation - WAF - Network traffic monitoring - Intrusion detection - VPN or Direct Connect for hybrid environments</td> - Advanced network protection - Anomaly detection - More comprehensive traffic analysis - Micro-segmentation of network resources - PrivateLink for service access</td></tr> Logging & Monitoring</td> - Basic CloudTrail and CloudWatch setup - Log storage and retention</td> - Extended logging (S3, Lambda, RDS, etc.) - More granular monitoring - Regular audits - AWS Config for resource tracking</td> - Real-time continuous monitoring - Advanced analytics (Amazon Elasticsearch, Kinesis) - Automated response capabilities (Lambda, Step Functions) - Centralized logging across accounts and regions</td></tr> Patch Management & Vulnerability Scanning</td> - Regular patching and updates - Basic vulnerability scanning</td> - Rigorous patch management - Regular vulnerability scanning (Amazon Inspector) - Remediation processes and tracking</td> - Continuous vulnerability scanning - More aggressive patch management processes - Integration with security information and event management (SIEM) tools</td></tr> Backup & Disaster Recovery</td> - Basic backup (S3, EBS, RDS) - Recovery processes</td> - More frequent backups - Enhanced recovery processes - Regular testing - Cross-region replication for backups</td> - High availability and redundancy - Multi-region deployment - Shorter recovery time objectives (RTO) and recovery point objectives (RPO) - Versioning and backup validation</td></tr> Incident Response</td> - Basic incident response plan - Notification and escalation processes</td> - Enhanced incident response plan - Regular testing and updates - Incident response team and training</td> - Advanced incident response capabilities - Automated response (Lambda, Step Functions) - Continuous improvement based on lessons learned - Integration with external threat intelligence sources</td></tr> Compliance Validation</td> - Regular audits and assessments - Compliance with FISMA Low requirements</td> - More comprehensive audits and assessments - Compliance with FISMA Moderate requirements - AWS Artifact for compliance documentation</td> - Rigorous audits and assessments - Continuous validation - Compliance with FISMA High requirements - Third-party assessments and certifications</td></tr> </tbody></table> FedRAMP Compliance</h2> Control Area</th> FedRAMP Low</th> FedRAMP Moderate</th> FedRAMP High</th></tr></thead> Identity & Access Management</td> - Basic IAM policies - MFA for privileged users - Regular access reviews.</td> - More stringent IAM policies - MFA for all users - Role-based access control - Regular access audits - Temporary credentials for short-term access</td> - Enhanced user monitoring - Continuous access auditing - Additional access restrictions - Session duration limitations - Just-In-Time (JIT) access</td></tr> Data Encryption</td> - Encryption at rest (KMS, S3, EBS) - Encryption in transit (TLS)</td> - More stringent encryption key management - Automated key rotation - Dedicated KMS Customer Master Keys (CMKs)</td> - Enhanced encryption algorithms - Hardware security modules (HSM) integration - Stronger key management policies - Key access and usage logging</td></tr> Network Security</td> - Basic VPC setup - Security groups - Network ACLs</td> - Enhanced VPC isolation - WAF - Network traffic monitoring - Intrusion detection - VPN or Direct Connect for hybrid environments</td> - Advanced network protection - Anomaly detection - More comprehensive traffic analysis - Micro-segmentation of network resources - PrivateLink for service access</td></tr> Logging & Monitoring</td> - Basic CloudTrail and CloudWatch setup - Log storage and retention</td> - Extended logging (S3, Lambda, RDS, etc.) - More granular monitoring - Regular audits - AWS Config for resource tracking</td> - Real-time continuous monitoring - Advanced analytics (Amazon Elasticsearch, Kinesis) - Automated response capabilities (Lambda, Step Functions) - Centralized logging across accounts and regions</td></tr> Patch Management & Vulnerability Scanning</td> - Regular patching and updates - Basic vulnerability scanning</td> - Rigorous patch management - Regular vulnerability scanning (Amazon Inspector) - Remediation processes and tracking</td> - Continuous vulnerability scanning - More aggressive patch management processes - Integration with security information and event management (SIEM) tools</td></tr> Backup & Disaster Recovery</td> - Basic backup (S3, EBS, RDS) - Recovery processes</td> - More frequent backups - Enhanced recovery processes - Regular testing - Cross-region replication for backups</td> - High availability and redundancy - Multi-region deployment - Shorter recovery time objectives (RTO) and recovery point objectives (RPO) - Versioning and backup validation</td></tr> Incident Response</td> - Basic incident response plan - Notification and escalation processes</td> - Enhanced incident response plan - Regular testing and updates - Incident response team and training</td> - Advanced incident response capabilities - Automated response (Lambda, Step Functions) - Continuous improvement based on lessons learned - Integration with external threat intelligence sources</td></tr> Compliance Validation</td> - Regular audits and assessments - Compliance with FedRAMP Low requirements</td> - More comprehensive audits and assessments - Compliance with FedRAMP Moderate requirements - AWS Artifact for compliance documentation</td> - Rigorous audits and assessments - Continuous validation - Compliance with FedRAMP High requirements - Third-party assessments and certifications</td></tr> </tbody></table> HIPAA Compliance</h2> When to NOT use Kubernetes Unknown — Fri, 17 Mar 2023 16:22:46 -0600 When to NOT use Kubernetes</h1> Have you ever wondered if using a Kubernetes for a small to medium scale application is worth the complexity and overhead? In many cases, the benefits might not outweigh the costs (financial or emotional) or the added complexity needed to maintain a Kubernetes platform, particularly when the applications require few resources and have low traffic volume. When does Kubernetes make sense for your application? Let’s take a look at some of the scenarios where Kubernetes may not be the best choice. What is the architecture of your application? Kubernetes is a natural fit for microservices, event-driven, and pipeline architectures, while may not be ideal for other architectures like n-tier, microkernel, service-oriented, or service-based architectures.</p> What is Kubernetes?</h2> {{< image src=“static/kubernetes.excalidraw.png” caption=“Kubernetes Architecture” alt=“Shows common baseline with k8s.” width=“100%” >}}</p> Getting back to the to some basics what is Kubernetes? Kubernetes is an open-source container orchestration platform used for automating deployments, scaling, and management of containerized applications. Kubernetes is particularly useful for large-scale applications with high traffic volume, and event-driven, microservice or pipeline architectures, where it provides powerful management, scaling, and deployment capabilities. Examples of such applications include e-commerce platforms, social media networks, and big data processing. In these scenarios, the benefits of Kubernetes often outweigh the complexity and overhead it introduces. Although, Kubernetes has numerous benefits and has proven to be a valuable tool for many developers and organizations, though it’s not always the best solution for every scenario. There are certain situations where it’s better to opt for a different container orchestration platform or not to use one at all.</p> Containers</h3> Containers offer a lightweight alternative to virtual machines, allowing you to package and run your applications in a standardized environment. Containers are portable, running on any infrastructure that supports them, including on-premises, cloud, or hybrid environments. Being more lightweight and efficient than virtual machines, containers enable running more applications on the same hardware. They are particularly well-suited for applications requiring consistent environments across development, testing, and production stages.</p> One major design choice to move from serverless to container orchestration platforms, services like Amazon ECS, AWS Fargate, and Kubernetes is code runtime will be heavily dependent on encapsulation within runtime containers. While Docker was once the de facto standard for containerization, alternative containerization tools like Containerd, Podman, Buildah, CRI-O have gained popularity, partly in response to Docker’s disruptive switch to an Oracle-like licensing model. These alternatives provide additional choices for organizations looking to adopt containerization in their software development and deployment processes.</p> Kubernetes Features</h3> Automated rollouts and rollbacks</li> Service discovery and load balancing</li> Storage orchestration</li> Self-healing</li> Secret and configuration management</li> Automatic bin packing</li> Batch execution</li> Horizontal scaling</li> IPv4/IPv6 dual-stack</li> Designed for extensibility</li> </ul> Now that we vaguely understand with a container is. Here you’ll see some of the features that Kubernetes promotes. Think on how these features could useful for your operation and maybe that will shed some light on why it’s gaining so much popularity.</p> Paired with a thriving community, an ever expanding suite of 3rd party integrations, and the ease in which anyone can get started( minikube, kind, k3s), really helps drive the adoption of K8s.</p> But it also happens that Kubernetes is particularly useful for large-scale applications with high traffic volume, that have streaming or batch operations and leverage microservices, event-driven, or pipeline architectures. where Kubernetes provides powerful and consistent foundation for management, scaling, and deployment capabilities.</p> Examples of such applications that leverage Kubernetes include e-commerce platforms, streaming neworks, social media networks, and big data processing. In these scenarios, the benefits of Kubernetes often outweigh the complexity and overhead it introduces.</p> Common Architectures with K8s</h3> Microservices</h4> Microservices architecture involves breaking down an application into a collection of loosely coupled services. Each service is a self-contained unit that can be deployed and scaled independently. This architecture is particularly suitable for Kubernetes, as it can manage the lifecycle, scaling, and deployment of these services efficiently, making the application more scalable, resilient, and easy to maintain.</p> In this e-commerce application example, multiple services are deployed within a Kubernetes cluster. The API Gateway routes requests to the corresponding service (Authentication, Product, Cart, or Order). Each service communicates with its respective database.</p> {{< image src=“static/microservices.excalidraw.png” caption=“microservices Architecture” alt=“Shows common Microservices Architecture.” width=“100%” >}}</p> graph TD </span> A[API Gateway] --> B[Authentication Service] </span> A --> C[Product Service] </span> A --> D[Cart Service] </span> A --> E[Order Service] </span> B --> F[User Database] </span> C --> G[Product Database] </span> D --> H[Cart Database] </span> E --> I[Order Database] </span> subgraph Kubernetes Cluster </span> B </span> C </span> D </span> E </span> F </span> G </span> H </span> I </span> end </span></code></pre> Event-Driven</h4> Event-driven architecture focuses on applications that are triggered by events that occur in the system. These applications react to events from various sources, such as user interactions or changes in data. Kubernetes is a good fit for event-driven applications when they are designed with microservices or when the application components need to be scaled independently. Kubernetes can manage the lifecycle of the application and its services, as well as scale the application to meet demand.</p> In this IoT application example, the IoT Device and User Application send events to the Event Bus, which routes the events to the appropriate services (Notification Service and Data Processing Service) within the Kubernetes cluster. These services handle notifications and data processing/storage.</p> {{< image src=“static/event-driven.excalidraw.png” caption=“microservices Architecture” alt=“Shows common Event-Driven Architecture.” width=“100%” >}}</p> Mermaid</summary> ```mermaid </span>%%{init: {'theme': 'pastel', "flowchart" : { "curve" : "basis" } } }%% </span>graph TD </span> A[IoT Device] --> B[Event Bus] </span> C[User Application] --> B </span> B --> D[Notification Service] </span> B --> E[Data Processing Service] </span> D --> F[User Database] </span> E --> G[Data Storage] </span> subgraph Kubernetes Cluster </span> D </span> E </span> F </span> G </span> end </span>``` </span></code></pre> </details> Pipeline</h4> Pipeline architecture involves breaking down an application into a series of stages, where each stage processes the data and passes it to the next one. This architecture is suitable for data-intensive applications or workflows. Kubernetes can be a good fit for managing pipeline applications when individual stages can be deployed as independent services, allowing for efficient management, scaling, and deployment.</p> In this data processing pipeline example, data is ingested, transformed, enriched, and stored within a Kubernetes cluster. Each stage in the pipeline is a separate service managed by Kubernetes.</p> {{< image src=“static/pipeline.excalidraw.png” caption=“microservices Architecture” alt=“Shows common Pipeline Architecture.” width=“100%” >}}</p> %%{init: {'theme': 'pastel', "flowchart" : { "curve" : "basis" } } }%% </span>graph LR </span> A[Data Source] --> B[Data Ingestion] </span> B --> C[Data Transformation] </span> C --> D[Data Enrichment] </span> D --> E[Data Storage] </span> subgraph Kubernetes Cluster </span> B </span> C </span> D </span> E </span> end </span></code></pre> Scenarios Where Kubernetes May Not Be the Best Choice</h2> Legacy Applications</h3> First up for hard situations where Kubernetes may not be the best fit is when dealing with legacy applications. Integrating older applications with new technology like Kubernetes can be challenging and may require significant effort and investment. Legacy applications are typically monolithic and have complex dependencies and architectures that can make them challenging to immediately containerize and integrate with modern technologies like ECS, Kubernetes, or Serverless.</p> Legacy applications can require significant effort and investment to retrofit legacy applications for containerization, which may not be cost-effective. Even if they are containerized, their monolithic nature and complex dependencies can make it challenging to integrate with Kubernetes. For instance, legacy applications may have baked-in ideas about scaling mechanisms, logging, storage, or cross-instance communication that do not align well with Kubernetes’ principles. In such cases, it might be more practical to use existing deployment methods that better suit the needs of these applications.</p> If necessary apply the Strangler Pattern (aka “Strangler Fig” or “Vine”) to modernize a legacy applications it can be a lengthy ordeal. As usually a facade is propped up and new elements are developed around or on top of the existing legacy system allowing both to be run in parallel before the legacy elements are eventually migrated/swapped out.</p> Small Teams</h3> Kubernetes can also be a poor fit for organizations with limited available resources. As the infrastructure, expertise, and resources required to run and maintain a Kubernetes cluster can be substantial.CNCF favors a reliance on an open-source ecosystem that encourages piecemeal design. With Kubernetes there is a plethora of third-party integrations, services, operators that all have varying opinions on how to achieve an outcome. Often, I have found myself taking detours to resolve hiccups of the functionality of the platform I took for granted(looking at your breaking changes Containerd). A small team may not be able to keep pace with the wack-o-mole process of fixing breaking changes that come up.</p> This ever expanding selection may be a less attractive option for organizations that want design for simplicity & not encourage flexibility. Managed Kubernetes services like Amazon EKS can help to reduce the expertise needed to set up and maintain a Kubernetes cluster. However, organizations still need to have a good understanding of Kubernetes concepts and best practices to effectively manage their applications, networking, security, and storage in the cluster.</p> Steep Learning Curve</h3> Moreover, the learning curve for Kubernetes can be steep, and it may not be the best choice for organizations with limited IT staff or developers. The platform requires significant expertise in areas such as networking, security, storage, and CI/CD to be effectively deployed and managed. This expertise can be challenging to acquire and may not be feasible due to budgets, staffing, or available domain knowledge. Additionally, Kubernetes is a rapidly evolving platform that introduces breaking changes with each new release, requiring dedicated resources to keep up with the latest developments in the space.</p> Regulatory and Cost Constraints</h3> Kubernetes may not be the ideal choice for organizations with stringent regulatory or cost constraints, such as financial institutions or healthcare organizations. The platform’s security features and network policies are inherently complex, necessitating substantial expertise for proper configuration and validation to meet compliance requirements. Organizations with strict regulations may find it difficult to grasp how services are secured, particularly when alternative services are available that rely on a cloud provider’s shared responsibility model to assume most of the responsibility for securing systems.</p> Moreover, the cost of operating a Kubernetes cluster can be considerable. For instance, consider an EKS cluster running a service continuously, with the container image updated monthly for security reasons. Using the AWS Cost Calculator, the EKS control plane costs $73 per month. In addition to this, clusters require nodes for computation. Running a single service for a month using Fargate would cost around $270.33, while running comparable nodes (m6g.xlarge) on EC2 instances provides more flexibility in cost control, with pricing options ranging from approximately $50 to $112 per month. These price scenarios result in annual costs ranging from ~$1,500 to around ~$5,000 per cluster, excluding expenses for VPC, DNS, databases, additional clusters, or logging and monitoring. For organizations with limited budgets or cost-consciousness, adopting a serverless architecture or running services on dedicated EC2 Spot Instances can be more cost-effective solutions, as they offer the ability to scale to zero when not in use.</p> Kubernetes may not be the ideal choice for organizations with stringent regulatory or cost constraints, such as financial institutions or healthcare organizations. The platform’s security features and network policies are inherently complex, necessitating substantial expertise for proper configuration and validation to meet compliance requirements. Kubernetes is not secure by default, so security practices are needed to keep everything secure. Paired with a large open-source ecosystem there may be a need to hand-roll container images create custom, secure, and optimized containers tailored to specific security posture. This could include services you might not have thought of that are bundled with Kubernetes. Organizations with strict regulations may find it difficult to grasp how services are secured, particularly when alternative services are available that rely on a cloud provider’s shared responsibility model to assume most of the responsibility for securing systems.</p> Moreover, the cost of operating a Kubernetes cluster can be considerable. For instance, consider an EKS cluster running a service continuously, with the container image updated monthly for security reasons. Using the AWS Cost Calculator, the EKS control plane costs $73 per month. In addition to this, clusters require nodes for computation. Running a single service for a month using Fargate would cost around $270.33, while running comparable nodes (m6g.xlarge) on EC2 instances provides more flexibility in cost control, with pricing options ranging from approximately $50 to $112 per month. These price scenarios result in annual costs ranging from ~$1,500 to around ~$5,000 per cluster, excluding expenses for VPC, DNS, databases, additional clusters, or logging and monitoring. For organizations with limited budgets or cost-consciousness, adopting a serverless architecture or running services on dedicated EC2 Spot Instances can be more cost-effective solutions, as they offer the ability to scale to zero when not in use. </p> By considering alternatives like AWS Lambda, Docker Compose, and AWS Fargate, developers and organizations can choose the most appropriate solution for their unique circumstances. Real-life examples of applications that have successfully adopted these alternatives demonstrate the importance of selecting the right container orchestration platform based on individual requirements. Ultimately, making an informed decision can lead to more efficient development, deployment, and management of containerized applications, while also optimizing costs and resources.</p> Software Architecture for the Cloud</a> by Dick Dowdell</p> Architecture Types that are not well suited for Kubernetes</h2> N-Tier</h3> N-tier architecture involves breaking down an application into a series of tiers, where each tier has a specific responsibility (e.g., presentation, business logic, data storage). While Kubernetes can be used to manage the lifecycle, scaling, and deployment of each tier, it is not inherently designed for n-tier architectures. However, it can still be a viable option for managing n-tier applications when the tiers are designed as independent services.</p> In this traditional n-tier web application example, the Web Server, Application Server, and Database Server are deployed on separate virtual machines (VMs) or physical servers. This monolithic architecture is less suitable for Kubernetes.</p> {{< image src=“static/n-tier.excalidraw.png” caption=“N-Tier Architecture” alt=“Shows common N-Tier Architecture.” width=“100%” >}}</p> %%{init: {'theme': 'pastel', "flowchart" : { "curve" : "basis" } } }%% </span>graph TD; </span> A[Web Browser] --> B[WAF] </span> B[WAF] --> C[Web Server] </span> C --> D[Application Server] </span> F --> E[Caching Server] </span> D --> E </span> D --> F[Database Server] </span> subgraph VMs or Physical Servers </span> B </span> C </span> D </span> E </span> F </span> end </span></code></pre> Microkernel</h3> Microkernel architecture is over 50 years old it involves building an application with a minimal core system and a set of plug-ins or modules that extend its functionality. This architecture is not inherently designed for Kubernetes, as the focus is on a minimal core rather than independent services. However, if the plug-ins or modules can be containerized and managed independently, Kubernetes could be used for managing their lifecycle, scaling, and deployment.</p> In this microkernel-based monitoring system example, the Core System communicates with multiple plugins (Reporting, Analytics, Monitoring). The tight coupling between the core and plugins makes this architecture less suitable for Kubernetes. Web Browsers like Firefox, Chrome, and Safari and IDEs like EclipseIDE or VScode are examples of microkernel-based applications where capabilities are added through plug-ins or extensions.</p> {{< image src=“static/microkernel.excalidraw.png” caption=“Microkernel Architecture” alt=“Shows common Microkernel Architecture.” width=“100%” >}}</p> %%{init: {'theme': 'pastel', "flowchart" : { "curve" : "basis" } } }%% </span>graph LR </span> A[Core System] --> B[Plugin 1: Reporting] </span> A --> C[Plugin 2: Analytics] </span> A --> D[Plugin 3: Monitoring] </span> subgraph VMs or Physical Servers </span> A </span> B </span> C </span> D </span> end </span></code></pre> Service-Oriented</h3> Service-oriented architecture involves building an application as a collection of loosely coupled services. These services can be deployed and scaled independently, making the application more scalable, resilient, and easy to maintain. Kubernetes is a good fit for service-oriented applications when they are designed with microservices or when the application components need to be scaled independently. Kubernetes can manage the lifecycle of the application and its services, as well as scale the application to meet demand.</p> In this diagram, the API Gateway routes requests to appropriate services. Each service is responsible for a specific domain within the e-commerce application. Services are connected through the Enterprise Service Bus (ESB), which handles communication and integration between them. Each service has its own dedicated database for managing its domain-specific data with CRUD (create,read,update, and delete)operations and is connected to the ESB.</p> {{< image src=“static/microkernel.excalidraw.png” caption=“Microkernel Architecture” alt=“Shows common Microkernel Architecture.” width=“100%” >}}</p> %%{init: {'theme': 'pastel', "flowchart" : { "curve" : "basis" } } }%% </span>graph LR </span> A[Web Frontend] --> B[API Gateway] </span> B --> C[Customer Management Service] </span> B --> D[Inventory Management Service] </span> B --> E[Order Processing Service] </span> B --> F[Shipping Service] </span> B --> G[Payment Service] </span> B --> H[Notification Service] </span> C -->|CRUD| I[Customer Database] </span> D -->|CRUD| J[Inventory Database] </span> E -->|CRUD| K[Orders Database] </span> F -->|CRUD| L[Shipping Database] </span> G -->|CRUD| M[Payment Database] </span> H -->|CRUD| N[Notification Database] </span> subgraph ESB[Enterprise Service Bus] </span> C </span> D </span> E </span> F </span> G </span> H </span> end </span></code></pre> Monolithic</h3> Monolithic architecture refers to an application design pattern where the entire application is built as a single, cohesive unit. All the functionalities, including UI, business logic, and data management, are bundled together into one codebase, and the application is deployed as a single unit. Monolithic applications are tightly coupled and can be more challenging to maintain, update, or scale.</p> {{< image src=“static/monolithic.excalidraw.png” caption=“Monolithic Architecture” alt=“Shows common Monolithic Architecture.” width=“100%” >}}</p> %%{init: {'theme': 'pastel', "flowchart" : { "curve" : "basis" } } }%% </span>graph LR </span> A[Web Browser] --> B </span> B["E-commerce Application<br>(Product Catalog, Shopping Cart,<br>Payment, User Management)"] </span> subgraph VMs or Physical Servers </span> B </span> end </span></code></pre> It may be worth considering alternatives like AWS Lambda, Docker Compose, and AWS Fargate, but its should be up to the digression of the developers and organizations to choose the most appropriate solution for their unique circumstances. Real-life examples of applications that have successfully adopted these alternatives demonstrate the importance of selecting the right container orchestration platform based on individual requirements. Ultimately, making an informed decision can lead to more efficient development, deployment, and management of containerized applications, while also optimizing costs and resources.</p> TLDR;</h2> Kubernetes is a powerful and versatile container orchestration platform that is particularly useful for large-scale applications with high traffic volume, where it provides powerful management, scaling, and deployment capabilities. However, for small-scale or simple applications, there may be more cost-effective and simpler solutions available.</li> When dealing with legacy applications, integrating with Kubernetes can be challenging and may require significant effort and investment. It may be more feasible to stick to existing deployment methods or to use a different container orchestration platform that better suits the needs of these applications.</li> Kubernetes requires significant expertise in areas such as networking, security, storage, and deployment to be effectively deployed and managed. This expertise can be challenging to acquire and may not be feasible due to budgets, staffing, or available domain knowledge.</li> Kubernetes may not be the best choice for organizations with strict regulatory requirements, where it can likely be easier to lean on a Cloud provider’s shared responsibility model to take on some of the operational responsibilities in securing systems.</li> </ul> Resources</h2> Software Architecture for the Cloud</a></li> do you need kubernetes?</a></li> kind</a></li> k3s</a></li> minikube</a></li> </ul> </iframe> Pi Hole Unknown — Tue, 27 Apr 2021 19:30:46 -0600 Pi Hole</h1> For a while I was running PiHole on a raspberry pi64. Installed docker, docker-compose, git Configured running raspberry pi with docker https://github.com/pi-hole/docker-pi-hole#readme</p> docker-compose.yaml</p> version: "3" </span> </span># More info at https://github.com/pi-hole/docker-pi-hole/ and https://docs.pi-hole.net/ </span>services: </span> pihole: </span> container_name: pihole </span> image: pihole/pihole:latest </span> # For DHCP it is recommended to remove these ports and instead add: network_mode: "host" </span> ports: </span> - "53:53/tcp" </span> - "53:53/udp" </span> - "67:67/udp" # Only required if you are using Pi-hole as your DHCP server </span> - "80:80/tcp" </span> environment: </span> TZ: 'America/Denver' </span> PIHOLE_DNS_: 127.0.0.1#5053 </span> TEMPERATUREUNIT: f </span> DNSSEC: 'true' </span> ADMIN_EMAIL: </span> </span> # WEBPASSWORD: 'set a secure password here or it will be random' </span> # Volumes store your data between container upgrades </span> volumes: </span> - './etc-pihole:/etc/pihole' </span> - './etc-dnsmasq.d:/etc/dnsmasq.d' </span> # https://github.com/pi-hole/docker-pi-hole#note-on-capabilities </span> cap_add: </span> - NET_ADMIN # Recommended but not required (DHCP needs NET_ADMIN) </span> restart: unless-stopped </span></code></pre> Configured piHole with DoH filtering DNS through Cloudflare (1.1.1.2) DNS cache - Pi-hole documentation</a></p> Update the Ad Lists based on recommendations Blocklist Collection ¦ Firebog</a></p> Add SSH for local hosts Securing DNS across all of my devices with Pi-Hole + DNS-over-HTTPS + 1.1.1.1</a></p>