If your API resides on a different origin than your front-end and you wish to send cookies to it, you will need to enable CORS on your server and send cookies with your requests by providing the option {credentials: "include"} to fetch.
Solution
I realized that I could just run the tRPC server and that also runs the client somehow — so no need for different ports and causing cors issues.