More cases of Non-blocking parser reporting incorrect locations when fed with non-zero offset

[dnault]

Related to #531 -- it looks like there's still an offset calculation glitch when ByteArrayFeeder.feed() is called with a non-zero array offset.

Jackson version: 2.18.3

I've attached a slightly more comprehensive unit test. feedByteByByteFromOffsetZero() passes, but the others fail.

Thanks!

ByteArrayFeederOffsetTest.java.txt

[cowtowncoder]

For convenience, let's inline:

import java.io.IOException;
import java.util.ArrayList;
import java.util.List;

import org.junit.jupiter.api.DisplayName;
import org.junit.jupiter.api.Test;

import com.fasterxml.jackson.core.JsonFactory;
import com.fasterxml.jackson.core.JsonParser;
import com.fasterxml.jackson.core.JsonToken;
import com.fasterxml.jackson.core.async.ByteArrayFeeder;

import static java.nio.charset.StandardCharsets.UTF_8;
import static org.junit.jupiter.api.Assertions.assertEquals;

class ByteArrayFeederOffsetTest {
  private static final JsonFactory jsonFactory = new JsonFactory();
  private static final byte[] json = "[true]".getBytes(UTF_8);

  private record TokenAndByteOffsets(
    JsonToken token,
   long tokenByteOffset,
    long parserByteOffset
  ) {}

  /**
   * Returns the result of feeding the whole JSON document at once from array offset zero.
   */
  private static List<TokenAndByteOffsets> expected() throws IOException {
    try (JsonParser parser = jsonFactory.createNonBlockingByteArrayParser()) {
      ByteArrayFeeder feeder = (ByteArrayFeeder) parser.getNonBlockingInputFeeder();
     feeder.feedInput(json, 0, json.length);
      feeder.endOfInput();
      return readAvailableTokens(parser);
    }
  }

  @Test
  @DisplayName("Feed one byte at a time from array offset zero")
  void feedByteByByteFromOffsetZero() throws Exception {
    try (JsonParser parser = jsonFactory.createNonBlockingByteArrayParser()) {
      ByteArrayFeeder feeder = (ByteArrayFeeder) parser.getNonBlockingInputFeeder();
      List<TokenAndByteOffsets> actual = new ArrayList<>();

      for (int i = 0; i < json.length; i++) {
        byte[] temp = new byte[]{json[i]};
        feeder.feedInput(temp, 0, temp.length);
        if (i == json.length - 1) feeder.endOfInput();
        actual.addAll(readAvailableTokens(parser));
      }

     assertEquals(expected(), actual);
    }
  }

  @Test
  @DisplayName("Feed one byte at a time from non-zero array offset")
  void feedByteByByteFromNonZeroOffset() throws Exception {
    try (JsonParser parser = jsonFactory.createNonBlockingByteArrayParser()) {
      ByteArrayFeeder feeder = (ByteArrayFeeder) parser.getNonBlockingInputFeeder();
      List<TokenAndByteOffsets> actual = new ArrayList<>();

      for (int i = 0; i < json.length; i++) {
        feeder.feedInput(json, i, i + 1);
        if (i == json.length - 1) feeder.endOfInput();
        actual.addAll(readAvailableTokens(parser));
      }

      assertEquals(expected(), actual);
    }
  }

  @Test
  @DisplayName("Feed whole document at once from non-zero array offset")
  void feedWholeDocumentFromNonZeroOffset() throws Exception {
    // Same document, but preceded by a 0 byte.
    byte[] jsonOffsetByOne = new byte[json.length + 1];
    System.arraycopy(json, 0, jsonOffsetByOne, 1, json.length);

    try (JsonParser parser = jsonFactory.createNonBlockingByteArrayParser()) {
      ByteArrayFeeder feeder = (ByteArrayFeeder) parser.getNonBlockingInputFeeder();

      feeder.feedInput(jsonOffsetByOne, 1, jsonOffsetByOne.length);
      feeder.endOfInput();
      assertEquals(expected(), readAvailableTokens(parser));
    }
  }

  private static List<TokenAndByteOffsets> readAvailableTokens(JsonParser parser) throws IOException {
    List<TokenAndByteOffsets> result = new ArrayList<>();
    while (true) {
      JsonToken token = parser.nextToken();
      if (token == JsonToken.NOT_AVAILABLE || token == null) return result;
      result.add(new TokenAndByteOffsets(
        token,
        parser.currentTokenLocation().getByteOffset(),
        parser.currentLocation().getByteOffset())
      );
    }
  }
}

[cowtowncoder]

@dnault Thanks! I need to modify it a bit as records cannot be used 2.x tests (Java 8 baseline).

[cowtowncoder]

Added tests, should make it easier to verify fix whenever some are forthcoming :)

[cowtowncoder]

Fixed for 2.19.0 via #1415.