Repository navigation
Agent is suggests && in non-supporting versions of pwsh, awk in pwsh on Windows, etc. #6073
Description
I asked the agent to generate a README, then asked to commit.
The agent then tried to execute git add README.md && git commit -m "Add README file" in its own powershell terminal, which errors into
PS E:\Dropbox\code\pypeduct> git add README.md && git commit -m "Add README file"
In Zeile:1 Zeichen:19
- git add README.md && git commit -m "Add README file"
-
~~
Das Token "&&" ist in dieser Version kein gÞltiges Anweisungstrennzeichen.
+ CategoryInfo : ParserError: (:) [], ParentContainsErrorRecordException
+ FullyQualifiedErrorId : InvalidEndOfLine
Copilot Chat Extension Version: 0.25.2025030404
Version: 1.98.0-insider (user setup)
Commit: 21d13f4350b0651abfae76b98651d7c72551dff1
Date: 2025-03-04T06:32:40.978Z
Electron: 34.2.0
ElectronBuildId: 11044223
Chromium: 132.0.6834.196
Node.js: 20.18.2
V8: 13.2.152.36-electron.0
OS: Windows_NT x64 10.0.19045
Activity
Tyriar commentedon Mar 6, 2025
&&actually is supported by more recent versions of pwsh, so that could workaround this issue by updating.amogorkon commentedon Mar 6, 2025
Ok, I upgraded Powershell, which fixes this issue. It might be nice to catch this error and point the user in the right direction.
Tyriar commentedon Mar 6, 2025
Anselm Kiefner (@amogorkon) yeah I'm going to see if we can give a hint about the pwsh version to help here. Changing the model being used might improve how it's handling the error, it's a little surprising it's just being ignored.
16 remaining items
hediet commentedon Jul 3, 2025
Still happens:
But the log doesn't mention
&&anymore. Would be interesting to collect telemetry about how many tokes this burns. Maybe we can do something smarter (e.g. auto fixing&&to;)Frulfump commentedon Jul 3, 2025
Doesn't it make more sense to make it understand in which environment it's operating within, i.e if it knows it's using the legacy PowerShell (powershell.exe) would it still suggest using
&&which is perfectly fine to use in modern PowerShell (pwsh.exe)? I seem to have made Copilot use modern PowerShell so I don't run into this issue, not exactly sure how but I think I might have changed the default terminal profile.Tyriar commentedon Jul 3, 2025
Frulfump that's exactly what we do. LLMs are non-deterministic and don't like to listen sometimes. This is included only when the binary is
powershell.exe:Tyriar commentedon Jul 3, 2025
Henning Dieterichs (@hediet) I tried to reproduce this on sonnet 4 and got it to happen 1 out of 10 tries and then I gave up. I verified
&&isn't mentioned and the hint to use;is included. I'm going to make this out of scope for now as it's rare and the situation should improve over time as models do and it self heals by just running another one.When/if we add actual parsing of the shell command via tree-sitter or something this is something we could do reliably, otherwise we risk bricking the command.
hediet commentedon Jul 3, 2025
Would still be valuable to have telemetry to get insights of how many times this (and other situations) happen!
Maybe whenever a terminal command fails, with a 10% chance we can ask LLM to summarize the error into 1-5 words and send that via telemetry.
Tyriar commentedon Jul 3, 2025
Henning Dieterichs (@hediet) we have telemetry around the number of commands that fail. Getting the LLM to summarize an arbitrary string like that would be classified as PII
amogorkon commentedon Jul 3, 2025
Maybe tackle it from the other side: don't just try to prevent the issue but rather catch the exception and print
Suggestions how to recover: {self.recovery_actions}"(with recovery_actions being a json list of different possible solutions) so the agent can try to resolve the issue ideally without user intervention.Tyriar commentedon Jul 3, 2025
Anselm Kiefner (@amogorkon) it already does that which is why I think it's not such a big deal, especially since it rarely happens (at least for me):
Before I wouldn't be surprised if it wouldn't be able to figure it out, but the failure is enough at least in the screenshot above to nudge it to listen to the instructions about Windows PowerShell.
amogorkon commentedon Jul 3, 2025
Hmm. I'm tinkering with such "enriched" exceptions for my own projects and oftentimes it works as intented, but sometimes it seems the agent just ignores everything. Any idea what could be the reason and how to make it listen more aggressively? For instance, I found that it works best if the recovery suggestions are last in the output and are largely ignored if printed before the traceback. Does the agent combine temporaly correlated output of stdout and stderr?
Tyriar commentedon Jul 3, 2025
Anselm Kiefner (@amogorkon) based on my experience when there's a lot of context sometimes it just gets ignored, to what extent also depends on the model being used. Re-writing the command as Henning Dieterichs (@hediet) suggested is probably the right solution in the end, but since this is such a minor issue after the main problem was fixed it doesn't justify the investment for now.
There is a recency bias, but it is relatively close to the bottom in a user message.
It's all a single stream in the terminal, not like regular process launching. Any time I've seen it recently it rewrites it on next attempt fine.
hediet commentedon Jul 7, 2025
Another issue is that it often outputs bash commands in a powershell environment, which then often fails (especially for more complicated commands). I guess this is basically the same error - the LLM ignores the terminal environment.
Frulfump commentedon Jul 7, 2025
Yeah I've run into this as well and I think as you mention that it's the same kind of category of issues.
I've tried to mitigate it with rules like
But as the models aren't always following rules it fails sometimes but at least it happens less with the rules above.
Tyriar commentedon Jul 7, 2025
Moved the running of commands that it generally shouldn't be doing in pwsh to this issue: microsoft/vscode#254431