In our build system the next build creates a new client with a new number at the end (qb-<qb_agent_host_name>-xxxx(4 digit number)) so it never reuses. Do we have to always remove the old clients manually, or is there a way to have it use the same client?

How you are specifying p4 client name and root directory in p4 repository definition?

I checked a few of our build configs' p4 repo definition, and they all contain these:
Client Name Auto-generated safe name based on node and root directory
Root Directory not defined
we have a bunch of leftover clients like this qb-<qb_agent_host_name>-xxxx(4 digit number) generated each day and aren't accessed at all in the future

For each configuration, QB will create a different client, and they are differenticated with suffix numbers on same host. It is fine to delete these clients, just that QB will re-create them upon next run of the configuration.