I have a web frontend that shows to the enduser some values. These values are derived through complex algorithms.

I would like to be able to expose verbose calculation logs that are currently stored in a rotated file for support/research.

Now this log is currently not aggregated (e.g. Calculations for item A and B are in the same log) and we might need to look at historical calculations so no on demand logging.

The calculations are done frequently and I don’t think even a dedicated logging database would be feasible due to the number of logs.

Do I split them into aggregated logs (split by item) and then display them as is (either through shared filesystem or other) or is there a better way to do this?

We are using a Microsoft stack (.NET Core and Framework/Windows/SQL Server)