The react-query integration does a lot of heavy lifting for you, so we'd recommend using that. But you can take this in house and use the vanilla tRPC client with SWR if you like
I'm not really knowledgable on the SSR side, but I wouldn't think the @trpc/nextjs package offers you anything you can't quickly roll yourself once you take away react-query
You can use the raw @trpc/client@trpc/client and spend a bunch of times doing an swr or your own abstraction, but I wouldn't recommend it rather than using the ones we already provide which are battle tested