Admiral v00049: Crash Diagnostics, Disk Encryption, and Cgroup2 Workloads

Admiral v00049 is a massive under-the-hood release focused on hardware-level security, deep crash diagnostics, and modernizing our workload isolation and image pull infrastructure.

Here is a breakdown of what has landed.

1. Deep Crash Diagnostics & System Recovery

When deploying to remote edge hardware, a silent kernel panic or a stuck process can mean a costly field visit. v00049 introduces aggressive new monitoring and logging to capture and recover from catastrophic states:

• Kernel Issue Capture: The host agent now actively captures and responds to a wider array of kernel issues, including Oops, soft lockups, hard lockups, panics, bad page states, and KASAN reports.
• Stuck Process Detection: Added an activity monitor to detect stuck processes, ensuring the supervisor can take action if the main runtime deadlocks.
• Crash Stack Dumps: The system will now dump its stack every ~5 minutes for debugging purposes.
• Historical Log Ingestion: On startup, Admiral now ingests the last block of the previous log, ensuring that crashes causing process exits or sudden reboots are properly recorded.
• Watchdog Retry Logic: Fixed hardware watchdog enablement to retry for up to 30 seconds, accommodating watchdogs that live in kernel modules that haven’t been loaded yet.
• Clock Reboot Fix: Fixed a weird issue where a stuck hardware clock could cause the system to reboot unexpectedly.

2. Security, Encryption & Hardware Metrics

We are laying the groundwork for enterprise-grade physical security and disk encryption across the fleet:

• Disk Encryption & Recovery: Implemented full disk encryption with remote recovery key support.
• TPM & Secure Boot: Initial implementations for TPM and Secure Boot are landing (full rollout coming soon).
• Image & Module Signing: Implemented dm-verity on images, kernel signing, module signing, and AppArmor enforcement. (Contact sales for details on enabling hardened modes).
• Hardware Metrics: Added disk SMART metrics, eMMC metrics, and NVME metrics to system telemetry (requires system >1.0.6).
• Expanded Hardware Detection: Added LPDDR4X detection and active reporting of OS versions and secure boot states.

3. Networking & Mesh Resiliency

We’ve overhauled our HTTP networking and Admiral Mesh cooldowns for maximum stability over degraded links:

• Unified Netclient Refactor: The Admiral netclient has been completely refactored into a drop-in replacement for all HTTP calls, bringing automatic retries, chunking, and QoS on the Linux link with backpressure monitoring. This now includes layer downloads.
• Mesh Cooldowns & Rebuilds: Refactored KCP cooldowns on the Admiral Mesh. Fixed a state where if the mesh client died, it would never rebuild, and fixed an integer overflow in the failure count logic.
• Link State Accuracy: Fixed an issue where pongs could arrive late and mark a dead link as active. Topology rotations are now allowed where no endpoints can be contacted but the link is up.
• State Preservation: Last contact times are now written to volatile memory to preserve state across service restarts.
• IPv6 Flap Fix: Fixed a bug where IPv6 rotations (which happen up to 30s) kept aggressively pushing specifications.

4. Storage, Image Management & Workloads

Workload isolation and image pulling have been massively streamlined and hardened:

• Image Pull & Heal Refactor: Refactored image pulling and caching. The system now automatically retries pulling layers with chunk resumption, and will automatically heal images if corruption is detected.
• Simplified Image Unpacking: Greatly simplified the image unpack process, and switched to using qgroups on images to get filesystem size instead of walking the tree.
• Cgroup2 Migration: Workloads have been moved to cgroup2 mounts instead of cgroup. The supervisor will now attempt to unwedge stuck cgroups automatically without collision, and clean up zombies where possible.
• Memory & Trimming: Fixed a bug where shared memory wasn’t being counted, causing OOM on the host. Implemented periodic disk trimming to extend flash life.
• Workload Fixes: Fixed port mapping on workloads, fixed bugs starting workloads with non-interactive states, and added a Volumes API for frontend integrations. A diagnostics log is now written directly from the workload manager.

5. Provisioning & Platform Polish

• WiFi & WPA3: Refactored WiFi scanning and connections. Admiral now manages wpa_supplicant directly, probing and supporting WPA3 networks.
• Identity & Installer Refactor: Refactored identity management (including wiping, resetting, and isolation states), installer steps, and the update staging and flow.
• Disk Selection & eMMC Life: Implemented new disk selection methods to always mount the disk where Admiral runs from. Removed discard=async to preserve eMMC life.
• Reboot & Offline Timings: Refactored rebooting behavior (requires system >1.0.6) and tweaked offline timings - nodes will now automatically reboot after 1hr from boot with no connectivity.
• Dynamic Feature Detection: Added dynamic detection for BPF and AppArmor support.