Hello Dan,
There are a few options, I think the most basic is doing a periodic job https://docs.q2developer.com/tutorials/periodic/index.html. Do a paginated call for all users of like 1000 users at a time and do core calls to retrieve that data then store it in the db for that user. This still might be kinda expensive to do all those calls.
So if you have a cheaper way to pull all of them at once, you can create a batch file and have your periodic job pull from that.
Another option is you could create a ardent api extension https://docs.q2developer.com/tutorials/ardent/index.html that you just send one user at a time and you just run from your machine or some automated service you have built on your side. So that way you don't have to 100k core calls or however many users you have.
Hope this helps,
-Trey
To expand upon this idea, Dan, you likely have a cheaper way to call your core than going through the Q2 bridge connection. Writing that file to an SFTP server location means your periodic job could injest that and run updates on the Q2 side without having to make all the core calls. The Ardent API extension works, too, but then you end up making thousands of CaliperAPI calls instead.
— Brian Mego 3 years ago