To help with the real time, could maybe add a layer of hot caching or the user could submit a batch job. if one fails, it's not blowing up the request.