Skip Navigation

glizzyguzzler

@ glizzyguzzler @piefed.blahaj.zone

Posts
2
Comments
96
Joined
1 yr. ago

  • My Homelab Got Hacked - A Postmortem – Phunky Cafe

    Jump

  • The real takeaway is to run your containers ALL as read-only. There’s no way to run executables like this as read-only. They’d download to /tmp but not have executable access and womp womp.

    Many things aren’t read-only friendly, but so far I’ve found all containers can be beat into read-only mode! (Heimdall is one of the worst, shout out to it)

  • Podman: Issues with multiple users running containers simultaneously

    Jump

  • So the sudo echo does echo as sudo but doesn’t carry over? Makes sense damn I hate bash! Any way to keep the >>? Or do you need to tee? Cause the >> is pretty cool

  • Podman: Issues with multiple users running containers simultaneously

    Jump

  • The best way to run Podman is root with UserNS to dole out UID/GID protection. Running Podman as root allows you to share networks between containers while having the containers run under different users. If you go rootless, you'd need to run under one user to share the user's network space with all the containers you want.

    As for your issue, I can't really divine what the problem is from the errors. I avoid nginx because it's coded to not play well with user abstraction and changing the user with the files it wants to write to etc. Gotta write into a ton of random folders! So not sure exactly what is up. But with Podman root it is easy to run as root 0 internally and make nginx think it has all the control it could ever want.

    Try this setup (it is in Podman Quadlet format, apologies I don't know the compose versions). It runs the container as root 0 internally, externally it runs as some random UID/GID - secure! It uses Volume idmap to map the internal root 0 user to 1001 for write access to the Volume.

    Note that in Debian 13 symlinks are broken and won't work with idmap, just point to the original source. If you need symlinks, I have an alternate UserNS that maps internal user root 0 to external user 1001 directly. You'd drop the idmap in Volume then and use that. You lose some extra security - now the container is running as external user 1001 instead of some random UID/GID - but that's a pretty minor hit as long as your external user doesn't have access to tons of things. 

       
        
# Volumes to mount -> the @ is essential for saying "1001 is absolute and external" basically. 0 is internal. size of 1. You can map 1001 to 0 and 1002 to 1 with @1001-0-2, etc., etc., etc.  
Volume=/mnt/something:/etc/nginx/wants/to/write/here:rw,noexec,nosuid,nodev,Z,idmap=uids=@1001-0-1;gids=@1001-0-1  

# Run as user running the container  
UserNS=auto  
# [use this if req symlink b/c idmap does NOT work with symlinks] -> I tested and it is fixed in at least Podman v5.8.3, so Debian 14 will work with idmap and symlinks directly ! drop the idmap if using !  
# UserNS=auto:uidmapping=0:@1001:1,gidmapping=0:@1001:1  

# Security time  
NoNewPrivileges=true  
# https://man7.org/linux/man-pages/man7/capabilities.7.html  
DropCapability=all  
ReadOnly=true  
ReadOnlyTmpfs=True  
# These capabilities are needed for linuxserver's s6 "launcher" thing  
#AddCapability=CAP_CHOWN  
#AddCapability=CAP_DAC_OVERRIDE  
#AddCapability=CAP_FOWNER  
#AddCapability=CAP_SETGID  
#AddCapability=CAP_SETUID  
# Needs this if the container tries to bind below port 1024. I'm not sure if it is needed if it only tries to bind internally.  
#AddCapability=CAP_NET_BIND_SERVICE  

# TempFS for ReadOnly fixes I've used for nginx - may not be relevant for you. These are from getting Frigate running.  
PodmanArgs=--tmpfs /usr/local/nginx/conf:size=1M,rw,noexec,nosuid,nodev  
PodmanArgs=--tmpfs /usr/local/nginx/logs:size=40M,rw,noexec,nosuid,nodev  
PodmanArgs=--tmpfs /usr/local/nginx/client_body_temp:size=1M,rw,noexec,nosuid,nodev  
PodmanArgs=--tmpfs /usr/local/nginx/proxy_temp:size=1M,rw,noexec,nosuid,nodev  
PodmanArgs=--tmpfs /usr/local/nginx/fastcgi_temp:size=1M,rw,noexec,nosuid,nodev  
PodmanArgs=--tmpfs /usr/local/nginx/uwsgi_temp:size=1M,rw,noexec,nosuid,nodev  
PodmanArgs=--tmpfs /usr/local/nginx/scgi_temp:size=1M,rw,noexec,nosuid,nodev  
PodmanArgs=--tmpfs /etc/letsencrypt:size=1M,rw,noexec,nosuid,nodev

    Root Podman and UserNS=auto needs a containers user to pull uid/gid from. 

       
        
# Root Podman needs a `containers` "user" (not really a user, just a reserved uid/gid space)  
sudo echo "containers:2147483647:2147483648" >> /etc/subuid  
sudo echo "containers:2147483647:2147483648" >> /etc/subgid

    The documentation for Podman is critically lacking in the "hobbyist" space. Hope this helps.

    Edit: This approach works well because most Docker containers are built assuming they'll run as root 0. That's why Linuxserver uses the S6 overlay thing to jump from root 0 to something else. The container can be built to run as any user though, if you look at the Dockerfile for the container you're using, you'll see what user they're declaring it will run as (and likely what user owns all the files). Root 0 usually gets around that problem - unless they "cleverly" code it to try to prevent you from running the container as root 0 (I've run into this before! It was Heimdall from the Linuxserver people).

    Edit2: I've noticed you said no Volumes, so drop that. But you can still use the UserNS mapping to run it as root internally which should fix the internal permissions issues. The tmpfs stuff is if you declare ReadOnly for extra security - it's a great idea - but nginx is extra difficult in that regard. Disregard it while you get going. 

       
        
# Run as user running the container  
UserNS=auto  

# Security time  
NoNewPrivileges=true  
# https://man7.org/linux/man-pages/man7/capabilities.7.html  
DropCapability=all  
# These capabilities are needed for linuxserver's s6 "launcher" thing  
#AddCapability=CAP_CHOWN  
#AddCapability=CAP_DAC_OVERRIDE  
#AddCapability=CAP_FOWNER  
#AddCapability=CAP_SETGID  
#AddCapability=CAP_SETUID  
# Needs this if the container tries to bind below port 1024. I'm not sure if it is needed if it only tries to bind internally.  
#AddCapability=CAP_NET_BIND_SERVICE

    Edit3:Use sudo podman top nginx user huser group hgroup groups hgroups to see the internal user/host user (huser) mappings easily for debug. 

       
        
USER	HUSER		GROUP	HGROUP		GROUPS	HGROUPS  
root		2147485695	root		2147485695	105		105

    Here's an output from my frigate container. Internally (USER) it is root, externally (HUSER) it's some random UID. I've also mapped the internal group (GROUPS) 105 to the external group (HGROUPS) 105 so that it has render access.

  • I wrote a dice rolling script with 3d rendering, collisions and friction in bash because why not [edit: code available now]

    Jump

  • I wrote a dice rolling script with 3d rendering, collisions and friction in bash because why not [edit: code available now]

    Jump

  • Sign me up!

  • I wrote a dice rolling script with 3d rendering, collisions and friction in bash because why not [edit: code available now]

    Jump

  • It's insane how many features this has for being insane in of itself!

    Edit: that was running on anemic "1 core" (1 very shared core) VPS, bumping it up to 4, hooboy do those dice fly

  • I wrote a dice rolling script with 3d rendering, collisions and friction in bash because why not [edit: code available now]

    Jump

  • When I opened the code I immediately scrolled to the bottom to see how many lines, and hell yeah. I read through it and I'm horrified, but I'm even more horrified it works. This is truly an chievement

    Here it is running over SSH https://files.catbox.moe/qkwd4q.mp4 what more could I ask for, thank you for making this insane thing and sharing it!

  • What 32-bit hardware are you still selfhosting on?

    Jump

  • I’ve got a pi zero w doing shairport-sync https://github.com/mikebrady/shairport-sync , which makes the pi pretend to be an AirPlay receiver. I combine it with the cheapest DAC I can get (used to be Adafruit before they ditched theirs sadly, now it’s https://shop.pimoroni.com/products/audio-dac-shim-line-out ) and bam old speakers are airplay compatible. The Pi is on Trixie and everything.

    It’s fantastic that it can do the airplay2 and everything without skipping a beat. Great coding from the dev of it (Mike Brady)!

  • I wrote a dice rolling script with 3d rendering, collisions and friction in bash because why not [edit: code available now]

    Jump

  • Haha there is no shame in shell code, it’s already monstrous to start!

    I’ve written things in shell that would look sooo nice and concise in any other language, but it became a Cthulu-ass demigorgon so quick in shell! Have no fear, fear is the mind killer, the little death!

  • I wrote a dice rolling script with 3d rendering, collisions and friction in bash because why not [edit: code available now]

    Jump

  • Get this on the berg I wanna render my dice rolls over SSH stat

  • Deleted

    How do I handle my anger towards management

    Jump

  • Management people like that do deserve your ire. They’re shit people who make it hard for everyone else because they have an extreme authoritarian view of running the corpo and will do what they want at the expense of rationality.

    It’s extra stupid because if someone they need (very specific need for a pet project basically) or someone they know personally comes along - that person would get all accommodations in a heart beat.

    But your team means nothing, the project means nothing (unless it is the pet project of the dipshit who gets to make the decision). They’re a shit person and you can’t change that and it sucks.

    Sorry the inflexibility of dipshit management is hurting you. It happens and it’s their fault, they will cause damage to the corpo, but they’re also probably too weak to tank the whole thing. Just a callous inefficiency. And you can hate them, they deserve it. It’s so easy to see a way towards a solution - but if bozo thinks bozo can get away with burning you to preserve some ephemeral corporate mandate for “in person”, bozo will burn you. The corpo mandate may disappear in a few years, but all that dipshit can do is think short term. And it sucks for you, it suck’s for your coworkers, and again, I’m sorry.

    Be stronk, your labor is your power. You’ll have to leave with your power, and that sucks ass. But your partner will be happy, and that may be worth it to you - especially if this corpo has bozos like that in high places, you’d probably run into something stupid that would be a deal breaker at some point.

    Lastly, no allegiance to the corpo, but your coworkers can be good. They may advocate for you, they may reference for you. Don’t trust anyone or any promises from corpo, including your local manager on up. Ideally that’s not how it should work - talent is talent even if you have to be flexible to keep talent - but corpos are authoritarian by nature and lack guardrails so dipshits get hooked on capricious short-term behaviour.

    Be pissed, fuck em, don’t train your replacement & walk out before that, remember you’re worth keeping in a sane world, because you worth it

    Edit: lastly, don’t dwell on it. Leave the shitstain where it is at your old job, don’t bring it in your mind. That’s really hard because it’s literally one capricious decision causing the issue - but do it for your sake. That’s the cost of living in authoritarian structures, don’t let it eat at you. You worth it, you’ll find something else

  • Selfhosting Sunday! What's up?

    Jump

  • The docker compose file is great, clear once you get used to all its little sections. For volumes I only use what they call “bind mounts” which are where you have a folder on your system connected to the folder in the container. As opposed to the “docker volumes” which are internal to docker.

    And with those it’s super easy to just be like in the volume subsection:

    - /path/to/local/drive:/container/database:rw,noexec,nosuid,nodev,Z

    - /path/to/network/drive:/container/media:rw,noexec,nosuid,nodev,Z

    The :rw,noexec,nosuid,nodev,Z at the end is a great extra security thing that never causes problems. rw means read-write, you can switch it to ro for read-only if you’ve got something you want the container to only read from but not be able to modify. I use that for jellyfin’s media since I don’t want it doing anything but reading it. The noexec means don't let executables be run from the folder, never should happen so it just prevents a sick hack from being put in the folder and run. I forget what the others do but they’ve never been a bother. And Z means only one process can access the volume, you can switch it to lower case z to let multiple processes access the volume - and I’m not sure it does anything without SELinux going which I think only fedora does by default right now.

    Enjoy the info dump!

  • Selfhosting Sunday! What's up?

    Jump

  • Check out BookOrbit if you find Grimmory too hefty in the RAM reqs. It uses very little RAM for me. I think it also has import from Grimmory as well.

    As for the database Q, if you’re using docker/Podman it’ll be very easy to point the database folder (if included in the image, like Jellyfin does) to your local filesystem and the media to your NAS. Same idea if the container uses a Postgres container, just spread across two containers in that case.

  • Unlocking LUKS with NitroKey/Yubikey: FIDO2, HMAC-SHA1, or OpenPGP?

    Jump

  • Interested in what you divine, I’m switching from a USB drive with the key in it to one of those fancy things.

  • Hister: a private search engine [AIP]

    Jump

  • Oh damn this could replace the bookmakers. I have Linkding with the Linkding Injector extension, but this would be next level.

    You can view the saved text too, nice. Would it be possible to attach an HTML file from like single file for sites that have heavy image content as part of the “view” button? That’d completely replace Linkding/Karakeep/Linkwarden use cases for me

    Understandable if not, that’s ancillary to the text search focus

  • Selfhosting as a Linux newbie - Ubuntu Server LTS or Debian 13?

    Jump

  • Choose Debian. Real secret reason? Debian means you have to upgrade to new versions less. Ubuntu LTS lasts as long as Debian (5 years) but they crank out a new LTS every 2 years. If you have 3rd party sources, I’ve run into they only support the two latest LTSes - well you have one year left on your 5 year support… but do you really? And now you can put on your sad face because you get to do two LTS upgrades. All while Debian was over there on one version and everyone will target that one version and its previous version no problem.

    Debian just makes better sense for a server.

    (And Ubuntu is stupid with its “ooh extra patches just register your machine with our central authority” ok bud my Debian machine just says I have mail for some reason every time I boot it up, it doesn’t make it weird and gimmicky and based on a “free” deal that could change whenever they want to jerk you around)

  • Auth apps

    Jump

  • I use Authentik - works well - but I’m prepping to switch to Authelia for the config-based setup.

    I updated Authentik across 2 versions (they did 2 month-long supported tags, missed one; now they do 3 month-long supported tags) and it destroyed itself. Had to recover the DB from a backup (and then step through the version I tried to skip), and while I was doing that I was like “wait why does this have only a DB? Should just be a config file cause that’s all the depth I do with it” and lo that’s what Authelia is.

    Authentik is audited and Authelia has not been. Initially while I chose Authentik. But config-file robustness in the face of Authentik’s GUI-setup DB imploding swayed me not to care, it’d take so long and be so tedious to redo all my proxies and auths through the GUI. Plus I always forget what to click in the GUI when I come back to add some new program in 4 months.

  • Managing podman quadlets, users subids and such

    Jump

  • I added examples so you don't have to dig as far if you want to give it a try!

  • Managing podman quadlets, users subids and such

    Jump

  • I approach it the same as I did with Docker, one user per container.

    I started with rootless but networking within Podman is moot with multiple users per container. And one user for all containers to get networking has to lead to subUID clashes (and thus escape vectors) - unless someone can explain how not…

    But root Podman is just as secure anyway, and easier, so I just roll with UserNS=auto and use idmap= on the volumes to enable writing as the specified user for the container. And networking in Podman works because it’s one user space. By default UserNS=auto gives 1024 subUIDs to a container. I had to up that to the expected max of 65535 for Frigate to work (example bit I use with idmap, UserNS with the size and group mapping, and GroupAdd): 

       
        
Volume=/mnt/camraz:/media/frigate:rw,noexec,nosuid,nodev,Z,idmap=uids=@1111-0-1;gids=@1111-0-1#1020-1020-1  
UserNS=auto:size=65535,uidmapping=105:@105:1,gidmapping=105:@105:1  
GroupAdd=105

    So what's going on here is on the Volume= I have the camera folder mounted to where Frigate wants it, and I specify that it accesses that volume with idmap= and frigate's UID/GID 1111 and the camera group I made at GID 1020.

    The order for the idmap is @${hostUID}-${containerUID}-${num2map}. The @ symbol means that 1111 is an absolute host UID; drop that and 1111 is pulled from the containers subUID/subGID list. Most containers run as root (0) so you just map to that user.

    Side quest: For containers that have one of those mini-hypervisors like LinuxServers images and their "S6" thing, I worked around them starting with a root user but then requiring moving to a non-root user by specifying PUID=1 and PGID=1 then setting idmap=uids=@1999-0-2;gids=@1999-0-2 and reserving 1999 as username containerNameStartup and 2000 as containerName. This idmap= maps host 1999 to container 0 but maps two UID/GIDs consecutively, so it also maps host 2000 to container 1 keeping everything lined up perfectly - while LinuxServer images still don't get host root like they so desperately want.

    GroupAdd=105 adds in the group 105 to the container, which is notably not the host group 105. UserNS is needed to complete that. This just makes that group exist in the container.

    UserNS=auto:size=65535,gidmapping=105:@105:1 accomplishes the subUID/subGID size allocation to the expected default max of 65535 (because Frigate needs it) and the group connection from host 105 to container 105. Note that this syntax helpfully uses the opposite order as idmap=. It goes ${containerUID}:@${hostUID}:${num2map} and uses colons instead of dashes.

    Every other container is cool with the default 1024 (just UserNS=auto). The subUIDs are pulled from a non-existent user named containers that you need to enable for Podman root to work with UserNS, and it has like 2 million billion or something with their recommended setup, so it’s good on subUIDs/subGIDs.

    Command to set up root Podman with UserNS=auto: 

       
        
sudo echo "containers:2147483647:2147483648" >> /etc/subuid  
sudo echo "containers:2147483647:2147483648" >> /etc/subgid

    You can also have UserNS=auto and run the container as a specific host user directly instead of "nobody" with:

    UserNS=auto:uidmapping=0:@1111:1,gidmapping=0:@1111:1

    Same idea as how I mapped in the host group 105 above. Where the root user 0 is mapped to the host (via @) user 1111 so the container runs as the user. I don't use that because the idmap= works perfectly and then the container doesn't even run as the user that controls the files directly. But maybe there will be a use for that sometime(?).

    And I do have a fuckton of users; Debian once complained it ran out of numbers or something after like 20 users, so I just ran the first thing I found to make the UID limit some really big number, and I never thought about it again! No idea what it was..

    **Edit: ** I cleaned it up and added examples

    Also I mostly work with UIDs not users per-say, so I specify UID 1111 has access to Frigate files. I just make a user named frigate to make it a bit easier for me. When you do ls -l it says frigate instead of 1111; life is easier.

    Overall, it has been a lot of work to divine these things because the documentation and useful examples for Podman are confusing and non-existent. I think I've gotten most of this idmap= and UserNS= stuff working from reading GitHub issues that drop little tidbits. But after I get it working as I want it, it's solid, and I can now rinse-and-repeat in the future. No Docker daemon to fuck with or get hacked, and it's reliable since I just let systemd start these quadlets up.

    I'm not 100% done, but I'm close. Outstanding issue right now is that on server restart sometimes some Podman networks don't actually start up correctly and then containers that want them fail to start because of that (I have to restart the Podman network via systemctl to get them to exist in Podman). But eventually I'll find the stupid GitHub issue where someone mentions something relevant for that, and then I'll be golden!

  • Write-Up: My Multi-Purpose Travel Server

    Jump

  • Thanks for the write up, again! I def gotta up my NetworkManager game…

  • Selfhosted @lemmy.world
    glizzyguzzler @piefed.blahaj.zone

    Question WRT secure networking with Podman/Docker stack and a reverse proxy in a VM "DMZ"

  • Selfhosted @lemmy.world
    glizzyguzzler @piefed.blahaj.zone

    IPv6 & Opnsense & Not Exposing Machine-Specific IPv6s to Corpos