Yeah, agree that one could / should definitely put it through evals if it were going to be a product. There’s quite a bit to a good evals program, though.
I think the main implementation challenge here - as in most of these things - would be how you retrieve / traverse the distilled documents. I think it’s hard to disentangle your distillation method from your preferred retrieval method. Surfacing stuff that’s relevant to a query is pretty trivial, surfacing that in a way that takes account of complex cross-document references, less so. All old-school search skills, though - suddenly (or soon to be) in big demand, no doubt.
I think the main implementation challenge here - as in most of these things - would be how you retrieve / traverse the distilled documents. I think it’s hard to disentangle your distillation method from your preferred retrieval method. Surfacing stuff that’s relevant to a query is pretty trivial, surfacing that in a way that takes account of complex cross-document references, less so. All old-school search skills, though - suddenly (or soon to be) in big demand, no doubt.