There probably is a clever way that you could do it, but clever ways are easy to overstep, misconfigure and can be unreliable long-term.

Congratulations, now your „password” (the 512-byte random key file) is stored as plaintext on your machine :)

With rate-limiting, non-trivial passwords are not viable to be brute-forced, so making them larger just doesn’t give you much.