Repository navigation
Operation metrics in Delta history #312
Description
The community has often requested for programmatic access to the metrics of various operations (e.g., # number of rows updete/deleted/inserted in Merge) after they have completed. These metrics are visible in the SQL tab of the Spark UI, but there is no easy way to programmatically access to those values.
A simple idea is to record the operation metrics in history. We already record operation parameters in the history (e.g., delete query condition) in json format, so it is easy to extend it to store the operation metrics as well (in json).
Metadata
Metadata
Assignees
Labels
No labels
Activity
ekoifman commentedon Feb 6, 2020
👍
[-]Operation statistics in Delta history[/-][+]Operation metrics in Delta history[/+]rahulsmahadev commentedon Feb 10, 2020
Operation metrics for writes and merge - 54643ef
Update and delete - 9dd5424
streaming update - adc03b8
tdas commentedon Feb 24, 2020
Marking this as done. To be released with 0.6.0
Rename Flink connector artifact & folder (delta-io#312)