Using the Java runtime, ThreadSanitizer flags many data races in the ANTLR runtime when each thread creates its own lexer + parser from the same generated grammar and parses concurrently.
Environment
- ANTLR 4.13.2
- JDK: OpenJDK 21 run with
-XX:+ThreadSanitizer
- Linux x86_64
Reproduce
Filter.g4:
grammar Filter;
prog : clause EOF ;
clause : or_clause ;
or_clause : and_clause (OR and_clause)* ;
and_clause : atom (AND atom)* ;
atom : ID OP value | LPAREN clause RPAREN ;
value : ID | INT | STRING ;
AND : 'AND' ; OR : 'OR' ;
OP : '=' | '!=' | '<' | '>' | '<=' | '>=' ;
LPAREN : '(' ; RPAREN : ')' ;
ID : [a-zA-Z_][a-zA-Z_0-9]* ;
INT : [0-9]+ ;
STRING : '"' (~["\r\n])* '"' ;
WS : [ \t\r\n]+ -> skip ;
RaceMain.java:
import org.antlr.v4.runtime.*;
public class RaceMain {
private static final String INPUT = "a = 1 AND b > 2 OR (c = 3 AND d != \"x\")";
public static void main(String[] args) throws Exception {
Thread[] ts = new Thread[8];
for (int i = 0; i < ts.length; i++) {
ts[i] = new Thread(() -> {
FilterLexer lexer = new FilterLexer(CharStreams.fromString(INPUT));
FilterParser parser = new FilterParser(new CommonTokenStream(lexer));
parser.prog();
});
ts[i].start();
}
for (Thread t : ts) t.join();
}
}
Run:
java -jar antlr-4.13.2-complete.jar -no-listener -no-visitor -o gen Filter.g4
javac -d build -cp antlr4-runtime-4.13.2.jar gen/*.java RaceMain.java
java -XX:+ThreadSanitizer -cp build:antlr4-runtime-4.13.2.jar RaceMain
Example race
Many races are reported. One example: a write in addDFAEdge racing against a read in getExistingTargetState on the same DFAState.edges slot.
WARNING: ThreadSanitizer: data race (pid=1900922)
Write of size 4 at 0x00021fa3f124 by thread T20 (mutexes: write M0):
#0 org.antlr.v4.runtime.atn.LexerATNSimulator.addDFAEdge(...)V LexerATNSimulator.java:656
#1 org.antlr.v4.runtime.atn.LexerATNSimulator.computeTargetState(...) LexerATNSimulator.java:283
#2 org.antlr.v4.runtime.atn.LexerATNSimulator.execATN(...)I LexerATNSimulator.java:204
#3 org.antlr.v4.runtime.atn.LexerATNSimulator.matchATN(...)I LexerATNSimulator.java:156
#4 org.antlr.v4.runtime.atn.LexerATNSimulator.match(...)I LexerATNSimulator.java:111
#5 org.antlr.v4.runtime.Lexer.nextToken() Lexer.java:141
#6 org.antlr.v4.runtime.BufferedTokenStream.fetch(I)I BufferedTokenStream.java:169
#7 org.antlr.v4.runtime.BufferedTokenStream.sync(I)Z BufferedTokenStream.java:152
#8 org.antlr.v4.runtime.BufferedTokenStream.setup()V BufferedTokenStream.java:254
#9 org.antlr.v4.runtime.BufferedTokenStream.lazyInit()V BufferedTokenStream.java:249
#10 org.antlr.v4.runtime.CommonTokenStream.LT(I) CommonTokenStream.java:92
#11 org.antlr.v4.runtime.Parser.enterRule(...)V Parser.java:628
#12 FilterParser.prog() FilterParser.java:106
#13 RaceMain.lambda$main$0()V RaceMain.java:12
#14 java.lang.Thread.runWith(...) Thread.java:1596
#15 java.lang.Thread.run()V Thread.java:1583
Previous read of size 4 at 0x00021fa3f124 by thread T18:
#0 org.antlr.v4.runtime.atn.LexerATNSimulator.getExistingTargetState(...) LexerATNSimulator.java:246
#1 org.antlr.v4.runtime.atn.LexerATNSimulator.execATN(...)I LexerATNSimulator.java:202
#2 org.antlr.v4.runtime.atn.LexerATNSimulator.matchATN(...)I LexerATNSimulator.java:156
#3 org.antlr.v4.runtime.atn.LexerATNSimulator.match(...)I LexerATNSimulator.java:111
#4 org.antlr.v4.runtime.Lexer.nextToken() Lexer.java:141
#5 org.antlr.v4.runtime.BufferedTokenStream.fetch(I)I BufferedTokenStream.java:169
#6 org.antlr.v4.runtime.BufferedTokenStream.sync(I)Z BufferedTokenStream.java:152
#7 org.antlr.v4.runtime.BufferedTokenStream.setup()V BufferedTokenStream.java:254
#8 org.antlr.v4.runtime.BufferedTokenStream.lazyInit()V BufferedTokenStream.java:249
#9 org.antlr.v4.runtime.CommonTokenStream.LT(I) CommonTokenStream.java:92
#10 org.antlr.v4.runtime.Parser.enterRule(...)V Parser.java:628
#11 FilterParser.prog() FilterParser.java:106
#12 RaceMain.lambda$main$0()V RaceMain.java:12
#13 java.lang.Thread.runWith(...) Thread.java:1596
#14 java.lang.Thread.run()V Thread.java:1583
Many similar reports come from addDFAState, computeTargetState, Interval.of, IntervalSet, and DefaultErrorStrategy.sync. Full
ThreadSanitizer log attached.
tsan.log
Question
Is using a separate lexer + parser instance per thread supposed to be thread-safe?
Using the Java runtime, ThreadSanitizer flags many data races in the ANTLR runtime when each thread creates its own lexer + parser from the same generated grammar and parses concurrently.
Environment
-XX:+ThreadSanitizerReproduce
Filter.g4:
RaceMain.java:
Run:
Example race
Many races are reported. One example: a write in addDFAEdge racing against a read in getExistingTargetState on the same DFAState.edges slot.
Many similar reports come from addDFAState, computeTargetState, Interval.of, IntervalSet, and DefaultErrorStrategy.sync. Full
ThreadSanitizer log attached.
tsan.log
Question
Is using a separate lexer + parser instance per thread supposed to be thread-safe?