infophysics.org

My Pseudonym Setup

Posted on Jun 13, 2023

This is a short to-do list of things I want to do, to set up better my anon account:

  • Setup Email
  • Setup a small server to have a public IP to access my infra
  • Setup a VPN with wireguard
  • Harden the server and publish a blog
  • Host this blog in that small server

Let’s get to action. How long could this take me? It’s [07:09 AM].

  • Cloud Provider choice: I wanted to go for Digital Ocean, but a new email and same credit card triggered some antifraud system. Let’s try our second choice, a small service called Hetzner. Account created: [07:14 AM]. Need to go through “Account Verification”. Need to take picture of my document. Need to restart my phone because of a bug with the Camera in Graphene OS when you switch it off globally and try to use it in certain apps (not all of them). Probably a mishandling of the permissions. OK; this also triggered an alert. I guess I will create my pc with my existing Digital Ocean account, then repeat the procedure somewhere else.
  • Deploying a nixos “server” from the command line. A few years ago, I tried to create a digital ocean “droplet” entirely from the console. That was part of an experiment of increasing the ratio of information/noise I consume, therefore using less the web browser. From that experiment, navigating without JavaScript created a much leaner experience of browsing the web.
doctl compute droplet ls
doctl compute droplet create francis --tag-names newnix \
      --image ubuntu-22-04-x64 --region fra1 --size s-2vcpu-2gb \
      --user-data-file nixos-userdata-for-static-hosting \
      --enable-ipv6 --enable-monitoring --enable-private-networking --enable-backups
ID           Name       Public IPv4    Private IPv4    Public IPv6    Memory    VCPUs    Disk    Region    Image                     VPC UUID    Status    Tags      Features                            Volumes
429838682    francis                                                  2048      2        60      fra1      Ubuntu 22.04 (LTS) x64                new       newnix    backups,monitoring,droplet_agent

Why the name “francis”? “Pick a name that is gender neutral” contamination/overfitting of my brain.

set DROPLET_IP 123.34.110.180
set DROPLET_ID 429838682

Adding to my .ssh/config

Host francis
    Hostname 123.34.110.180
    User root
    IdentityFile /home/usr/.ssh/digitalocean

Ok, it worked, but I can’t access. Using firefox, entered the digital ocean “droplet console” to find what’s the problem.

ssh francis "cat /var/log/cloud-init-output.log"

Ok, let’s destroy it and try again.

doctl compute droplet delete $DROPLET_ID

So, selecting DROPLET_ID and DROPLET_IP if this takes too much will be a PITA. I’ll extract it automatically from doctl compute droplet list: I wonder how to create a function and keep it stored with fish. For the time being, let’s see if it’s really necessary.

set DROPLET_ID (doctl compute droplet ls | grep francis | awk '{ print $1 }')

Note the use of set. I’m trying out “fish”, it has some annoying differences with bash, but thats’ probably because I’m not used to it. It’s [07:44 AM], I knew this was not going to be a walk in the park with my ADHD.

Damn it! I need a way to test this nixos configuration file locally. It would have been a lot better than finding out about a syntax error 4 minutes ago:

error: syntax error, unexpected ';'
at /etc/nixos/host.nix:24:111:

  23|     openssh.authorizedKeys.keys = [
  24|       "ssh-ed25519 ... usr@mars-digitalocean";
    |                                              ^
  25|     ];

Ok, fixed, redeploying, this time it’s looking better. I’m using ssh to tail -f the installation of nixos-infect. [07:52 AM] Damn, I run into another bug related to this nixos configuration script. “services.openssh.port” does not exist. Cmon!! Python is not a nixos package anymore – python3 is the name now. Leaving this for search engine optimization

ace: warning: The option `boot.cleanTmpDir' defined in `/etc/nixos/configuration.nix' has been renamed to `boot.tmp.cleanOnBoot'.
ror:
     … while calling the 'derivationStrict' builtin

       at /builtin/derivation.nix:9:12: (source not available)

     … while evaluating derivation 'nixos-system-francis-23.05.861.d3bb401dcfc'
       whose name attribute is located at /nix/store/hsv0sv3ryif6k7zwqjlrngrdhwqdk9gz-nixos-23.05/nixos/pkgs/stdenv/generic/make-derivation.nix:303:7

     … while evaluating attribute 'activationScript' of derivation 'nixos-system-francis-23.05.861.d3bb401dcfc'

       at /nix/store/hsv0sv3ryif6k7zwqjlrngrdhwqdk9gz-nixos-23.05/nixos/nixos/modules/system/activation/top-level.nix:112:5:

        111|     installBootLoader = config.system.build.installBootLoader;
        112|     activationScript = config.system.activationScripts.script;
           |     ^
        113|     dryActivationScript = config.system.dryActivationScript;

     (stack trace truncated; use '--show-trace' to show the full trace)

     error: Package ‘python-2.7.18.6’ in /nix/store/hsv0sv3ryif6k7zwqjlrngrdhwqdk9gz-nixos-23.05/nixos/pkgs/development/interpreters/python/cpython/2.7/default.nix:330 is marked as insecure, refusing to evaluate.


     Known issues:
      - Python 2.7 has reached its end of life after 2020-01-01. See https://www.python.org/doc/sunset-python-2/.

     You can install it anyway by allowing this package, using the
     following methods:

     a) To temporarily allow all insecure packages, you can use an environment
        variable for a single invocation of the nix tools:

          $ export NIXPKGS_ALLOW_INSECURE=1

      Note: For `nix shell`, `nix build`, `nix develop` or any other Nix 2.4+
      (Flake) command, `--impure` must be passed in order to read this
      environment variable.

     b) for `nixos-rebuild` you can add ‘python-2.7.18.6’ to
        `nixpkgs.config.permittedInsecurePackages` in the configuration.nix,
        like so:

          {
            nixpkgs.config.permittedInsecurePackages = [
              "python-2.7.18.6"
            ];
          }

     c) For `nix-env`, `nix-build`, `nix-shell` or any other Nix command you can add
        ‘python-2.7.18.6’ to `permittedInsecurePackages` in
        ~/.config/nixpkgs/config.nix, like so:

          {
            permittedInsecurePackages = [
              "python-2.7.18.6"
            ];
          }

Now, again, waiting for the droplet to be created… drop ls (I assigned alias drop=doctl compute droplet because it was making me crazy). While I’m watching this output, I think of many ideas to ensure what it’s actually doing. Let’s take a look:

#cloud-config
write_files:
- path: /etc/nixos/host.nix
permissions: '0644'
content: |
  {pkgs, ...}:
  {
    environment.systemPackages = with pkgs; [
      vim wget tmux zsh htop python3 lynx git gnupg unzip
      nmap tcpdump zlib bc man-pages binutils nix
    ];
    system.stateVersion = "23.05";
    services.openssh.enable = true;
    services.openssh.settings = {
      "PermitRootLogin" = "prohibit-password";
    };
    networking.hostName = "francis";
    users.users.root = {
      openssh.authorizedKeys.keys = [];
    };
    users.users.usr = {
      isNormalUser = true;
      initialPassword = "changeme123";
      extraGroups = [ "wheel" ];
      openssh.authorizedKeys.keys = [
        "ssh-ed25519  usr@mars-digitalocean"
      ];
    };
  }
runcmd:
- curl https://raw.githubusercontent.com/elitak/nixos-infect/master/nixos-infect | PROVIDER=digitalocean NIXOS_IMPORT=./host.nix NIX_CHANNEL=nixos-23.05 bash 2>&1 | tee /tmp/infect.log

So, I can come up with a number of rabbit hole experimentation ideas:

  • Clone nixos-infect to another repo to be able to stop new updates automatically, preventing against supply chain attacks.
  • The main reason we’re using this script is to be able to automatically copy information about the network, disks, and rest of the digitalocean VPS infrastructure.

Now, I got the VPS started! Why did I wanted this box?

  • First, to serve my blog
  • Second, to act as a static IP that I can use to connect remotely

Ok, that would be better if we had more than one box. Maybe I can harden it later and run these two cases on different firejailed containers…

Ok, let’s run the blog. To do this, I need:

  • run hugo
  • copy files
  • run nginx
  • direct domain to box

First time, I won’t automate this. Second time, I’ll write it as a nix package and just deploy the new version.

  • change IP of infophysics.org to francis
  • wait for DNS propagation
  • open ports 80 and 443 in francis
  • find my old blog files
  • build
  • copy output to francis:/persist/www/infophysics.org
  • change francis nixos config to add nginx infophysics.org and acme certificates
  • build francis with “newnix”

I used the web UI to change the record, but I could have used doctl compute domain records ls infophysics.org, extract the IDs for ‘Type “A”’, and use doctl compute domain records update 335061532 --record-data 178.128.202.217 source

Cool! It’s [08:28 AM] and I see “123” served in anywhere. Let’s harden this. Let’s shut down remote access. Change the port of SSH. Allow only access from my current VPN IP. Create a wireguard connection. Add fail2ban.

First, a minor distraction: set up “francis” in my newnix setup. Newnix is based out of glissner’s dotfiles. Now, it’s [08:47 AM] and I’m running the first build on francis with newnix. Took me about 20 minutes, plus a distraction of creating a comic about a noob sysadmin like myself. “I run unstable in production” says my t-shirt. I never learned to deploy on OpenBSD.

I hope I don’t get locked out! This first update I am afraid of is locking myself out. The second one is not having configured something correctly, which, judging by this output…

updating GRUB 2 menu...
stopping the following units: acme-finished-infophysics.org.target, acme-fixperms.service, acme-infophysics.org.timer, audit.service, kmod-static-nodes.service, logrotate-checkconf.service, mount-pstore.service, network-setup.service, nginx.service, nscd.service, resolvconf.service, systemd-modules-load.service, systemd-oomd.service, systemd-oomd.socket, systemd-sysctl.service, systemd-timesyncd.service, systemd-udevd-control.socket, systemd-udevd-kernel.socket, systemd-udevd.service, systemd-update-done.service, systemd-zram-setup@zram0.service
NOT restarting the following changed units: getty@tty1.service, systemd-journal-flush.service, systemd-logind.service, systemd-random-seed.service, systemd-remount-fs.service, systemd-update-utmp.service, systemd-user-sessions.service, user-runtime-dir@0.service, user-runtime-dir@1000.service, user@0.service, user@1000.service
activating the configuration...
removing group ‘acme’
removing group ‘nginx’
removing group ‘systemd-timesync’
removing user ‘systemd-timesync’
removing user ‘acme’
removing user ‘nginx’
setting up /etc...
removing obsolete symlink ‘/etc/systemd/zram-generator.conf’...
removing obsolete symlink ‘/etc/systemd/timesyncd.conf’...
restarting systemd...
reloading user units for root...
reloading user units for usr...
setting up tmpfiles
reloading the following units: dbus.service, firewall.service, reload-systemd-vconsole-setup.service
restarting the following units: network-addresses-eth0.service, nix-daemon.service, sshd.service, systemd-journald.service
starting the following units: audit.service, kmod-static-nodes.service, logrotate-checkconf.service, mount-pstore.service, network-setup.service, nscd.service, resolvconf.service, systemd-modules-load.service, systemd-oomd.socket, systemd-sysctl.service, systemd-udevd-control.socket, systemd-udevd-kernel.socket, systemd-update-done.service
the following new units were started: clear-log.timer, home-manager-usr.service, nix-gc.timer, ntpd.service, pcscd.socket, run-credentials-systemd\x2dvconsole\x2dsetup.service.mount, systemd-vconsole-setup.service, time-sync.target

I think I don’t have users anymore! I left an active SSH connection, now let’s test the ssh connection out.

Ok, it worked! The new port is 42042. I have mistakenly activated password login. Now I’ll close all inbound except from my VPN’s IP

Now, I went through a rabbithole because I found the nix setup from a long time ago in a server I call ‘static’ (this is because I didn’t want to look for the drive with the backup for it). Now, I have the files in a local folder, uploaded them remotely, but given that I only allowed this IP to communicate, I will have no visitors for the moment. And that’s fine. I copied the blog to websites/infophysics.org. I wonder if I could build this as a nix package.

Now, let’s harden the box a little bit more. I’d like to set up access through wireguard on one port, and that’s it. Then, ports 80 and 443 probably need to be published. I wonder how secure can a linux + nginx box be. I’d like to setup firejail, and also I’d like to receive an instant message whenever there is a login attempt (failed or successful). Since whenever I am working, I could silence those notifications, but otherwise, I’d really want to know if someone is trying to access the box.

So, that brings me back to provisioning and “berlinwire”, an idea I had for wireguard connectivity. I think I still had it running last week. My goal with this was being able to SSH to any of them, but since I’ve been quite static here, I had no need for that. I have a lot of reticence to look for these files. Last time I lost a lot of time looking for them. Take a breath, and let’s go for it. It reminds me of many things in my TO-DO list and it triggers me:

  • Not correctly setup my mars machine with the newnix. The data is in the old nixos config.
  • All the clutter and bitrot of a folder in which I had an ambitious project that didn’t prosper much. Also, I am in a relatively lower state of content than I was when I started that project, so it was a painful lesson.
  • It’s not going to happen again! The blogging of this process is the key.

Idea for a comic. A transmetropolitan-style comment: success can hurt your ego. There’s a reason why successful people try to retreat from public life; it can be asphyxiating at points. But I do this for me. And you cute future fans. I hate your fanatism. It makes me try to be better.

Now, I can build my blog locally thanks to hugo. I placed it in the folder ~/projects/blog.

Anyways, back to the beginning. How are we doing in terms of goals? Note that it’s [09:50 AM], 3 hours later.

  • Setup Email (this was useless, I think so far)
  • Setup a small server
  • Setup a VPN with wireguard
  • Harden the server and publish a blog
  • Host this blog in that small server

Let’s follow the nixos guide for Wireguard to the letter.

umask 077
mkdir ~/wireguard-keys
wg genkey > ~/wireguard-keys/private
wg pubkey < ~/wireguard-keys/private > ~/wireguard-keys/public
  1. I had not wireguard-tools installed.
  2. I kept misconfigured the nix.{localdomain} proxy. I need to restore cache.nixos.org as the main substituter. - Hmm, this was working as intended. I’m sending a reboot, since it’s the first time the computer started, to see what’s going on and if it becomes healthier after a small kick. - shit, still not working. Setting [usr@francis:/persist/newnix]$ sudo nixos-rebuild switch --option substituters https://cache.nixos.org --flake '.#francis' as my default. - Yeah! It worked after I was done with the job of replacing defaults to false everywhere.
  3. C’mon! Wireguard! - Yes, but “wireguard-tools” was not in “sysadmin” - I know, I can nix-shell -p wireguard-tools to setup a bash for that. - OK, here I am. No success. Both have their interface up.
  4. Where is my packet Brian? - Let’s see: let’s try adding persistentKeepalive. Doesn’t seem to work. - Leaving a tab open with a ping to spot my glorious victory when I achieve that. - Oh! Maybe it’s my router not allowing strange ports? - While I watch helios being built, I feel that I’m having a cache problem. Can’t take that long to recreate a few things. It’s clearly failing on the network, DNS, or nix-cache proxy. - Let’s take a small detour to fix “phobos”’s adherence to newnix. It’s [10:43 AM]. I knew wireguard was going to be painful. - Wow, it looks like the port is blocked? - Oh, I forgot UDP packets. Now the glorious ping is working! [11:00 AM] - Successfully entering through ssh through the vpn!
  5. See? It wasn’t that bad and it was your fault anyways with the digitalocean UI and firewall.

Now, next!

  • Setup Email (this was useless, I think so far)
  • Setup a small server
  • Setup a VPN with wireguard
  • Host this blog in that small server
  • Harden the server and publish a blog

This is an infinite task. I said that my two main concerns were:

  • Reasonably setup fail2ban
  • Create a bot notification on my local matrix (now that I’m connected to helios, I could also connect to phobos)
  • Check if helios is actually acting as a firewall. I could add a route in francis to gateway “192.168.0.0/24” to 10.42.0.1 – and then I would have to configure the nat rule on helios to forward the packet back.
  • This seems like a pain on the ass, I tried quickly to change the NAT, but I stumbled upon my early optimization of trying to strictly define the upstream experience.
  • Shit, I messed my roomate connection. Will not try this again. [11:13 AM]

OK; seems like I have an interesting fail2ban setup. It would be great to add a wireguard rule, but it looks like that requires enabling “debug” logs on wireguard. I’ll leave that for later. [11:26 AM]

Took a lot of interesting advice from Solene’s blog (too bad that she’s so against crypto).

Now that my server is a little bit more hardened, let’s switch it on!

  • It looks like it didn’t correctly connected.
  • Now, let’s create another droplet to get it banned.
  • Listening on the console for:
journalctl -xefu fail2ban

So, it looks like it could improve. But anyways, 5 hours later, I now copy this to hugo server and have it all!

  • Setup Email (this was useless, I think so far)
  • Setup a small server
  • Setup a VPN with wireguard
  • Host this blog in that small server
  • Harden the server and publish a blog

But I want to create a different category for this, because it’s not “perennial” content, as the rest of the “words”. That’s why I didn’t put a date in the posts in words, but for my journal, I would like that.

Now, many things would require access to my other server in my local network, I’m going to expose that other server to the franciswire vpn, and change the local DNS that francis is using so that I can reach that git server.

I have the following things in my wishlist:

  • Add phobos to franciswire, connect back and forth
  • Change local DNS on franciswire to find phobos
  • Setup my blog as a nix flake, if it gets updated in the repo, update it.
  • Split up the content on the blog from the custom theme I created
  • Setting bot account
  • Add a “blog” category
  • Modify the code so that it works better
  • Whenever someone logs in to SSH, send me a matrix message

Other goals for further explorations:

  • Rename the project “decent.so” that I had into “social.infophysics.org”. Store a few invites for the first random users that get connected.
  • Start to talk about how can I use an AI assistant to help me through all this process.
  • Create an index of all my writings so that I can mix-and-match it at random and generate aleatory blogposts to put into my blog. The user gets a customized experience powered by an AI that personalizes all the pages for their taste, and can also chat with a virtual persona of mine.

Setting bot account

[nix-shell:~]$ matrix-commander --login password
Enter URL of your homeserver: [https://matrix.example.org] https://matrix.homeserver
Enter your user ID:  [@john:example.org]  or  [john] for @john:matrix.homeserver : @francisbot:homeserver
Please provide your Matrix account password.
Password:
Choose a name for this device: [matrix-commander] francis
Enter room ID for default room:  [!SomeRoomIdString:example.org]  or  [alias] for #alias:matrix.homeserver : #lobby:homeserver
2023-06-13 11:55:11,105:     INFO: matrix-commander: The provided login data is: homeserver='https://matrix.homeserver'
2023-06-13 11:55:11,105:     INFO: matrix-commander:                             user id='@francisbot:homeserver'
2023-06-13 11:55:11,105:     INFO: matrix-commander:                             device name='francis'
2023-06-13 11:55:11,105:     INFO: matrix-commander:                             room id='#lobby:homeserver'
The provided login data is: homeserver='https://matrix.homeserver'
                            user id='@francisbot:homeserver'
                            password=''
                            device name='francis'
                            room id='#lobby:homeserver'
Correct? (Yes or Ctrl-C to abort) Yes
2023-06-13 11:55:15,430:     INFO: matrix-commander: The persistent storage directory store was created for you.
2023-06-13 11:55:15,907:     INFO: matrix-commander: E233: Log in using method 'password' was successful. Credentials were stored in file 'credentials.json'. From now on you can run program 'matrix-commander.py' without log in, as an access token is stored in your credentials file. If you plan on having many credential files, consider moving them to directory '/home/usr/.config/matrix-commander'.

Ok, a fair amount of back and forth [03:15 PM] and now I have:

  • Add phobos to franciswire, connect back and forth
  • Change local DNS on franciswire to find phobos
  • Setup my blog as a nix package, if it gets updated in the repo, update it.
  • Split up the content on the blog from the custom theme I created
  • Setting bot account
  • Add a “blog” category
  • Modify the code so that it works better
  • Whenever someone logs in to SSH, send me a matrix message

Actually, not quite right. I still would like to create a new user for this and run that without privileges. This is my current code:

#! /usr/bin/env bash

lines=$(journalctl -u sshd --since '11 min ago' \
        | grep 'Accepted publickey' \
        | awk '{print $9 " login from " $11 ":" $13, "key:", $16 }')

for line in "$lines"; do
  matrix-commander -s /home/usr/.config/matrix-commander --room '#francis:homeserver' -m "$line"
done

In it, I make the assumption that “homeserver” will be resolved (the wireguard connection is up).

Now, I need to schedule this every minute with a nixos systemd service timer.

warning: the following units failed: send-francis-logins.service

× send-francis-logins.service - Send francis logins every minute
     Loaded: loaded (/etc/systemd/system/send-francis-logins.service; enabled; preset: enabled)
     Active: failed (Result: exit-code) since Tue 2023-06-13 13:23:45 UTC; 765ms ago
   Duration: 4ms
    Process: 73021 ExecStart=send-francis-activity > /tmp/log-francis (code=exited, status=203/EXEC)
   Main PID: 73021 (code=exited, status=203/EXEC)
         IP: 0B in, 0B out
        CPU: 1ms

Jun 13 13:23:45 francis systemd[1]: Started Send francis logins every minute.
Jun 13 13:23:45 francis (activity)[73021]: send-francis-logins.service: Failed to locate executable send-francis-activity: No such file or directory
Jun 13 13:23:45 francis (activity)[73021]: send-francis-logins.service: Failed at step EXEC spawning send-francis-activity: No such file or directory
Jun 13 13:23:45 francis systemd[1]: send-francis-logins.service: Main process exited, code=exited, status=203/EXEC
Jun 13 13:23:45 francis systemd[1]: send-francis-logins.service: Failed with result 'exit-code'.
warning: error(s) occurred while switching to the new configuration

It appears to be that the PATH doesn’t contains the files inside of bin/. This might be because systemd is not providing me with that, or because the nixos derivation doesn’t provide it, which seems more reasonable to me. I’d need to locate the derivation that builds that folder. It seems to be declared at modules/options.

ExecStart = ''${config.newnix.dir}/bin/send-francis-activity > /tmp/log-francis'';

Let’s try that! It seems to have improved, as now, it complains about the lack of bash:

Jun 13 13:31:51 francis systemd[1]: Started Send francis logins every minute.
Jun 13 13:31:51 francis send-francis-activity[74911]: /usr/bin/env: ‘bash’: No such file or directory

Ok! It started successfully… I think?

Jun 13 13:32:55 francis send-francis-activity[76798]: /nix/store/5w3awd23866ywjsrfhd6bmlrsb3hzvc0-source/bin/send-francis-activity: line 5: awk: command not found
Jun 13 13:32:55 francis send-francis-activity[76797]: grep: write error: Broken pipe
Jun 13 13:32:55 francis systemd[1]: send-francis-logins.service: Deactivated successfully.

Ok, YGTBKM, but let’s move forward.

Jun 13 13:37:46 francis systemd[1]: send-francis-logins.service: Consumed 24.238s CPU time, received 301.3K IP traffic, sent 658.6K IP traffic.

Wow! It worked! But it seems to use a lot of data, why? Also, I have a lot of issues trying to unsplit this message for matrix-commander.

Anyways, now we’re back, it looks like it’s working, but not the timer. From the wiki:

systemd.timers."hello-world" = {
  wantedBy = [ "timers.target" ];
    timerConfig = {
      OnBootSec = "5m";
      OnUnitActiveSec = "5m";
      Unit = "hello-world.service";
    };
};

This feels very good! Now, let’s see if it sends a message with this new login I just made.

It worked! Now, unfortunately I don’t have this on my phone. But I’ll look into doing that.

Missing in my task list:

  • Split up the content on the blog from the custom theme I created
  • Add a “journal” category
  • Modify the code so that it works better

Awesome! It feels very good to have the blog be in the repository and accessing it. I’m not going to split up the content from the theme, it seems unnecessary, but I will dedicate a lot of effort into throwing back some changes from my modifications.

Now, it’s [5:19 PM] and I’ll take a break, browsing my own blog from my phone and feel a little bit proud of having gone through all this.