Describe the bug
In the light of https://github.blog/news-insights/company-news/github-copilot-is-moving-to-usage-based-billing I observed the token usage and counts and got surprised of how much tokens are sometimes consumed for some relative simple tasks, especially for the input tokens.
I try to find out why but due to
its very hard to guess, I just found that often it seems the case that copilot is hitting a wall and then trying too hard to fix it itself instead of asking the user.
One example was where compilation failed because it used the wrong compile command. This resulted in an unfortunate and long cycle where it tried to use a complete other tool (e.g. javac instead of maven, writing an own python script instead of using a prepared shell script or java tool) wanted to search the whole maven local repo or even decompiling java classes to learn method signatures (where it could have downloaded the source would be probably more effective) then needed multiple iterations until if finally found something. This not only wastes tokens, but also lead to poor results (e.g. even if copilot was able to compile the actual CI build might fail because required dependencies where not added) and takes a lot of time. I noticed similar where copilot was not able to execute a test and then began to bring up a whole own infrastructure to execute the single unit test where it would probably be easier to just run the regular full test suite.
Instead of trying to hard in such case it would be extremely helpful if in such case copilot would just ask how to proceed with maybe an option to just try as hard as possible, skip the step or getting further assistance.
Affected version
No response
Steps to reproduce the behavior
No response
Expected behavior
No response
Additional context
No response
Describe the bug
In the light of https://github.blog/news-insights/company-news/github-copilot-is-moving-to-usage-based-billing I observed the token usage and counts and got surprised of how much tokens are sometimes consumed for some relative simple tasks, especially for the input tokens.
I try to find out why but due to
its very hard to guess, I just found that often it seems the case that copilot is hitting a wall and then trying too hard to fix it itself instead of asking the user.
One example was where compilation failed because it used the wrong compile command. This resulted in an unfortunate and long cycle where it tried to use a complete other tool (e.g.
javacinstead of maven, writing an own python script instead of using a prepared shell script or java tool) wanted to search the whole maven local repo or even decompiling java classes to learn method signatures (where it could have downloaded the source would be probably more effective) then needed multiple iterations until if finally found something. This not only wastes tokens, but also lead to poor results (e.g. even if copilot was able to compile the actual CI build might fail because required dependencies where not added) and takes a lot of time. I noticed similar where copilot was not able to execute a test and then began to bring up a whole own infrastructure to execute the single unit test where it would probably be easier to just run the regular full test suite.Instead of trying to hard in such case it would be extremely helpful if in such case copilot would just ask how to proceed with maybe an option to just try as hard as possible, skip the step or getting further assistance.
Affected version
No response
Steps to reproduce the behavior
No response
Expected behavior
No response
Additional context
No response