Netbeans cannot delete some text when editing TOML files #8174

Activity

1. niabot

   added

   kind:bugBug report or fixBug report or fix

   needs:triageRequires attention from one of the committersRequires attention from one of the committers

   on Jan 20, 2025
2. matthiasblaesing

   removed

   needs:triageRequires attention from one of the committersRequires attention from one of the committers

   on Jan 22, 2025
3. matthiasblaesing commented on Jan 22, 2025

   

   matthiasblaesing

   on Jan 22, 2025

   Contributor

   More actions

   Reproducible, @lkishalmi @vieiro could you please have a look?
4. lkishalmi

   self-assigned this

   on Jan 22, 2025
5. lkishalmi commented on Jan 22, 2025

   

   lkishalmi

   on Jan 22, 2025

   Contributor

   More actions

   Checking.
6. lkishalmi commented on Jan 22, 2025

   

   lkishalmi

   on Jan 22, 2025

   Contributor

   More actions

   Could not reproduce. However there is something weird in the stacktrace above.

   It has references of org.netbeans.modules.textmate.lexer.SyntaxHighlighting which should not be there.
   Is it possibe that some TextMate grammar get registered in that mime-path?
7. lkishalmi

   removed their assignment

   on Jan 22, 2025
8. vieiro

   changed the title [-]Netbeans hangs when editing TOML files[/-] [+]Netbeans cannot delete some text when editing TOML files[/+] on Jan 23, 2025
9. vieiro commented on Jan 23, 2025

   

   vieiro

   on Jan 23, 2025

   Contributor

   More actions

   Hi, I can see what @niabot says. In my case NetBeans does not "hang", but can't delete the last } in the given example.

   I'll try to take a look at it later on.

   For the record, here's the message I get in the IDE Log:

   SEVERE [global]
   java.lang.IllegalStateException: Lexer org.netbeans.modules.languages.toml.TomlLexer@419edac2
     returned null token but lexerInput.readLength()=2
     lexer-state: -1
     tokenStartOffset=85, readOffset=87, lookaheadOffset=88
     Chars: "\n\n" - these characters need to be tokenized.
   Fix the lexer to not return null token in this state.
   	at org.netbeans.lib.lexer.LexerInputOperation.checkLexerInputFinished(LexerInputOperation.java:435)
   	at org.netbeans.lib.lexer.LexerInputOperation.nextToken(LexerInputOperation.java:193)
   	at org.netbeans.lib.lexer.inc.TokenListUpdater.relex(TokenListUpdater.java:602)
   	at org.netbeans.lib.lexer.inc.TokenListUpdater.updateRegular(TokenListUpdater.java:255)
   	at org.netbeans.lib.lexer.inc.TokenHierarchyUpdate$UpdateItem.update(TokenHierarchyUpdate.java:325)
   	at org.netbeans.lib.lexer.inc.TokenHierarchyUpdate.processLevelInfos(TokenHierarchyUpdate.java:200)
   	at org.netbeans.lib.lexer.inc.TokenHierarchyUpdate.updateImpl(TokenHierarchyUpdate.java:171)
   	at org.netbeans.lib.lexer.inc.TokenHierarchyUpdate.update(TokenHierarchyUpdate.java:109)
   	at org.netbeans.lib.lexer.TokenHierarchyOperation.textModified(TokenHierarchyOperation.java:585)
   	at org.netbeans.spi.lexer.TokenHierarchyControl.textModified(TokenHierarchyControl.java:71)
   	at org.netbeans.lib.lexer.inc.DocumentInput.textModified(DocumentInput.java:128)
   	at org.netbeans.lib.lexer.inc.DocumentInput.removeUpdate(DocumentInput.java:121)
   	at org.netbeans.lib.editor.util.swing.PriorityDocumentListenerList.removeUpdate(PriorityDocumentListenerList.java:91)
   	at java.desktop/javax.swing.text.AbstractDocument.fireRemoveUpdate(AbstractDocument.java:285)
   	at org.netbeans.editor.BaseDocument.fireRemoveUpdate(BaseDocument.java:1656)
   	at org.netbeans.editor.BaseDocument.handleRemove(BaseDocument.java:1026)
   	at org.netbeans.editor.BaseDocument$FilterBypassImpl.remove(BaseDocument.java:2625)
   	at java.desktop/javax.swing.text.DocumentFilter.remove(DocumentFilter.java:84)
   	at org.openide.text.CloneableEditorSupport$DocFilter.remove(CloneableEditorSupport.java:2361)
   	at org.netbeans.editor.BaseDocument.remove(BaseDocument.java:939)
   	at org.netbeans.editor.BaseKit$DeleteCharAction$3.run(BaseKit.java:2184)
   	at org.netbeans.editor.GuardedDocument.runAtomicAsUser(GuardedDocument.java:333)
   	at org.netbeans.editor.BaseKit$DeleteCharAction.actionPerformed(BaseKit.java:2176)
   	at org.netbeans.modules.csl.core.CslEditorKit$GsfDeleteCharAction.actionPerformed(CslEditorKit.java:375)
   	at org.netbeans.editor.BaseAction.actionPerformed(BaseAction.java:324)
   	at java.desktop/javax.swing.SwingUtilities.notifyAction(SwingUtilities.java:1810)
   	at java.desktop/javax.swing.JComponent.processKeyBinding(JComponent.java:2956)
   	at java.desktop/javax.swing.JComponent.processKeyBindings(JComponent.java:3004)
   	at java.desktop/javax.swing.JComponent.processKeyEvent(JComponent.java:2918)
   	at java.desktop/java.awt.Component.processEvent(Component.java:6398)
   	at java.desktop/java.awt.Container.processEvent(Container.java:2266)
   	at java.desktop/java.awt.Component.dispatchEventImpl(Component.java:4996)
   	at java.desktop/java.awt.Container.dispatchEventImpl(Container.java:2324)
   	at java.desktop/java.awt.Component.dispatchEvent(Component.java:4828)
   	at java.desktop/java.awt.KeyboardFocusManager.redispatchEvent(KeyboardFocusManager.java:1952)
   	at java.desktop/java.awt.DefaultKeyboardFocusManager.dispatchKeyEvent(DefaultKeyboardFocusManager.java:883)
   	at java.desktop/java.awt.DefaultKeyboardFocusManager.preDispatchKeyEvent(DefaultKeyboardFocusManager.java:1146)
   	at java.desktop/java.awt.DefaultKeyboardFocusManager.typeAheadAssertions(DefaultKeyboardFocusManager.java:1020)
   	at java.desktop/java.awt.DefaultKeyboardFocusManager.dispatchEvent(DefaultKeyboardFocusManager.java:848)
   	at java.desktop/java.awt.Component.dispatchEventImpl(Component.java:4877)
   	at java.desktop/java.awt.Container.dispatchEventImpl(Container.java:2324)
   	at java.desktop/java.awt.Window.dispatchEventImpl(Window.java:2780)
   	at java.desktop/java.awt.Component.dispatchEvent(Component.java:4828)
   	at java.desktop/java.awt.EventQueue.dispatchEventImpl(EventQueue.java:775)
   	at java.desktop/java.awt.EventQueue$4.run(EventQueue.java:720)
   	at java.desktop/java.awt.EventQueue$4.run(EventQueue.java:714)
   	at java.base/java.security.AccessController.doPrivileged(AccessController.java:400)
   	at java.base/java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(ProtectionDomain.java:87)
   	at java.base/java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(ProtectionDomain.java:98)
   	at java.desktop/java.awt.EventQueue$5.run(EventQueue.java:747)
   	at java.desktop/java.awt.EventQueue$5.run(EventQueue.java:745)
   	at java.base/java.security.AccessController.doPrivileged(AccessController.java:400)
   	at java.base/java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(ProtectionDomain.java:87)
   	at java.desktop/java.awt.EventQueue.dispatchEvent(EventQueue.java:744)
   	at org.netbeans.core.TimableEventQueue.dispatchEvent(TimableEventQueue.java:136)
   [catch] at java.desktop/java.awt.EventDispatchThread.pumpOneEventForFilters(EventDispatchThread.java:203)
   	at java.desktop/java.awt.EventDispatchThread.pumpEventsForFilter(EventDispatchThread.java:124)
   	at java.desktop/java.awt.EventDispatchThread.pumpEventsForHierarchy(EventDispatchThread.java:113)
   	at java.desktop/java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:109)
   	at java.desktop/java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:101)
   	at java.desktop/java.awt.EventDispatchThread.run(EventDispatchThread.java:90)
   line 2:0 token recognition error at: '\n'
   line 3:0 token recognition error at: '\n'
10. vieiro commented on Jan 23, 2025

    

    vieiro

    on Jan 23, 2025

    Last edited by vieiro

    Contributor

    More actions

    I think this may happen in any other antlr4 lexers. The bridge hiccups when the lexer returns null for a token (due to a lexer/parsing error).

    This does not mean that the antlr4 grammar is wrong. The antlr4 [lexer] grammar works correctly for correct files, but can't handle a file with a missing }, for example. For antlr4 grammars to work correctly with the antlr editor bridge, the grammars must contemplate an "error" token and not just return null on incorrect files.
11. vieiro

    self-assigned this

    on Jan 23, 2025
12. mbien

    added

    Editor

    ANTLR

    on Jan 23, 2025
13. niabot commented on Jan 24, 2025

    

    niabot

    on Jan 24, 2025

    Author

    More actions

    @vieiro The simple example (deleting the last curly brace) did not "hang" Netbeans. But i also managed to get it stuck in an infinite loop by copying/cutting text. This was my workaround for not being able to type or delete a curly brace. This intial bug might have triggered another bug/effect down the line, but i have no stacktrace/exception of it.
14. 1 remaining item

    Load more

15. lkishalmi commented on Jan 24, 2025

    

    lkishalmi

    on Jan 24, 2025

    Contributor

    More actions

    Managed to reproduce. The issue is in the ANTLR Lexer for TOML

    To reproduce just have the following toml:

    a = {}

    then delete the last curly from right with <backspace>.

    The problem is that ANTLR sends an EOF token in this case, while still having some characters on the stream.

    I can try to patch the bridge to handle this cases better, though it would be the best to fix the TOML Lexer grammar.
16. lkishalmi commented on Jan 24, 2025

    

    lkishalmi

    on Jan 24, 2025

    Contributor

    More actions

    @vieiro I see this PR in your project. vieiro/toml-java#18

    That could help. Also each mode shall have an rule at the end that matches everything, maps to INVALID_TYPE and/or pops from the mode.
17. vieiro commented on Jan 24, 2025

    

    vieiro

    on Jan 24, 2025

    Contributor

    More actions

    Thanks @lkishalmi . I'll create a new PR that will help. Basically the lexer will have to return an "error" token whenever an invalid character is found. I'll explain during the weekend.
18. vieiro commented on Jan 24, 2025

    

    vieiro

    on Jan 24, 2025

    Contributor

    More actions

    That'd be a requirement for all antlr4 grammars that we currently use. The rust grammar does indeed detect these error cases too, IIRC.
19. vieiro

    added a commit that references this issue on Jan 25, 2025

    apache#8174: Bump ide/libs.tomljava

    9d13f06
20. mbien

    linked a pull request that will close this issue#8174: Bump ide/libs.tomljava #8192on Jan 25, 2025
21. vieiro commented on Jan 25, 2025

    

    vieiro

    on Jan 25, 2025

    Contributor

    More actions

    Well, there we go this change

    INLINE_TABLE_UNEXPECTED : . -> type(INVALID_VALUE), popMode;
    INLINE_TABLE_EOF: EOF -> type(INVALID_VALUE), popMode;

    We may want to update the antlr tutorial to explain that antlr4 lexers should contemplate error situation (such as this: a TOML inline table without an ending }) and return this "error token" (INVALID_VALUE in this case). That makes it easier for AntlrLexerBridge to map that token to an ERROR. Just throwing/capturing an exception in Antlr is not good enough for us.
22. vieiro commented on Jan 25, 2025

    

    vieiro

    on Jan 25, 2025

    Contributor

    More actions

    @vieiro The simple example (deleting the last curly brace) did not "hang" Netbeans. But i also managed to get it stuck in an infinite loop by copying/cutting text. This was my workaround for not being able to type or delete a curly brace. This intial bug might have triggered another bug/effect down the line, but i have no stacktrace/exception of it.

    Hi @niabot , the "hang" you are detecting may be due to your specific setup. As @lkishalmi explains you seem to have a texmate highlighing layer that may be interferring/blocking. Next time you can use jconsole or a jps + jstack combination to fetch an stack trace for us, and report again.
23. matthiasblaesing commented on Jan 25, 2025

    

    matthiasblaesing

    on Jan 25, 2025

    Contributor

    More actions

    @vieiro @lkishalmi I don't think your assessment is correct, that the user did something special. Have a look here:

    netbeans/ide/textmate.lexer/src/org/netbeans/modules/textmate/lexer/SyntaxHighlighting.java

    Lines 851 to 871 in ab20d9a

    	@MimeRegistration(service=HighlightsLayerFactory.class, mimeType="")
    	public static class FactoryImpl implements HighlightsLayerFactory {
    	@Override
    	public HighlightsLayer[] createLayers(Context ctx) {
    	Document doc = ctx.getDocument();
    	TokenHierarchy<Document> th = TokenHierarchy.get(doc);
    	if (th == null) {
    	return new HighlightsLayer[0];
    	}
    	//check the token hierarchy produces the TextmateTokens
    	return new HighlightsLayer[] {
    	HighlightsLayer.create(SyntaxHighlighting.class.getName(),
    	ZOrder.SYNTAX_RACK,
    	true,
    	new SyntaxHighlighting(doc))
    	};
    	}
    	}

    My reading of this is, that the textmate syntax hightlighter will always run (registered on the empty mimetype). These problem show up at all the locations where the tokensquence is iterated, for the reporter that was syntax highlighting.

    👍React with 👍1Reacted by Antonio Vieiro
24. vieiro commented on Jan 25, 2025

    

    vieiro

    on Jan 25, 2025

    Contributor

    More actions

    Yep, thanks @matthiasblaesing. We couldn't reproduce the hang with the stacktrace provides, if the fix does not solve the problem then we'll need to reopen with a stack trace.
25. vieiro

    added 2 commits that reference this issue on Feb 14, 2025

    apache#8174: Bump ide/libs.tomljava

    ...

    39f9ae4

    apache#8174: Bump ide/libs.tomljava

    ...

    f229a8e
26. vieiro

    added a commit that references this issue on Feb 27, 2025

    apache#8174: Bump ide/libs.tomljava

    ...

    652a76d
27. mbien

    added a commit that references this issue on Feb 27, 2025

    Merge pull request #8192 from vieiro/hotfix/8174

    ...

    f198805
28. mbien

    closed this as completedin #8192on Feb 27, 2025