can i connect it to other services like paperless, or would i need to manually import files?

Hister supports importing data from a few services, but paperless isn't supported yet. More details: https://hister.org/docs/import

Is it better to run it in docker or e.g. a LXC in proxmox? If i want to index files, it seems it needs the config.yml file

Docker is perfectly fine. Every settings option from the config file can specified using environment variables. The syntax is HISTER__[SECTION]__[OPTION]=[VALUE].

After just playing with it for a day, the disk usage is 150 MB

Probably most of the disk space is occupied by the Hister binary which contains all the N-grams required to identify ~30 languages. The index should be much smaller.

I figure I should use postgres instead of sqlite.

SQLite is more than enough for personal use, but if you prefer to use postgres, just specify the standard DSN formatted connection data to the server.database config option: https://hister.org/docs/configuration#database-backends

how to set that up, preferably with docker

Use the HISTER__SERVER__DATABASE="host=localhost user=hister password=hister dbname=hister port=5432" environment variable.

I am not familiar with pgvector

Hister automatically creates the database model and handles the migrations if required.