New ask Hacker News story: Ask HN: What metrics do you track when building agents?
Ask HN: What metrics do you track when building agents?
2 by rstagi | 0 comments on Hacker News.
The main metric I look at is inference cost, which is a pretty good indicator when something's off. However, a couple of days a go I discovered we were loading ~300 tools due to a bug, affecting both costs and accuracy. It was going on for some time and I didn't realize it by looking at the costs, cause it got hidden by other changes I made in the same period. Now I started tracking that and some other metrics, but it made me wonder what else I might be missing. So: what are you tracking? Any tips? Curious to hear
2 by rstagi | 0 comments on Hacker News.
The main metric I look at is inference cost, which is a pretty good indicator when something's off. However, a couple of days a go I discovered we were loading ~300 tools due to a bug, affecting both costs and accuracy. It was going on for some time and I didn't realize it by looking at the costs, cause it got hidden by other changes I made in the same period. Now I started tracking that and some other metrics, but it made me wonder what else I might be missing. So: what are you tracking? Any tips? Curious to hear
Comments
Post a Comment