`DeserializationProblemHandler.handleUnexpectedToken()` no longer invoked for array-like types

[andpal]

Describe the bug
Before 2.12.x DeserializationProblemHandler::handleUnexpectedToken is invoked when trying to deserialize something with a structurally incompatible type, like deserializing a string from a START_OBJECT. In 2.12.x this no longer happens if the targeted type is an array-like type, such as an Iterable or a Collection. Instead the following exception is thrown: com.fasterxml.jackson.databind.exc.MismatchedInputException: Cannot construct instance of `java.util.ArrayList` (although at least one Creator exists): no String-argument constructor/factory method to deserialize from String value ('prop'), it appears that DeserializationProblemHandler::handleInstantiationProblem is invoked instead.

Version information
2.12.x
2.13.x

To Reproduce

pom.xml

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>
    <groupId>org.example</groupId>
    <artifactId>jackson-mve</artifactId>
    <version>1.0-SNAPSHOT</version>
    <packaging>jar</packaging>
    <properties>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
        <maven.compiler.source>1.8</maven.compiler.source>
        <maven.compiler.target>1.8</maven.compiler.target>

        <!-- This and 2.13.0 exhibit the issue -->
        <jackson.version>2.12.0</jackson.version>
        <!-- This is the last version where the test passes -->
<!--        <jackson.version>2.11.4</jackson.version>-->
    </properties>

    <dependencies>
        <dependency>
            <groupId>com.fasterxml.jackson.core</groupId>
            <artifactId>jackson-databind</artifactId>
            <version>${jackson.version}</version>
        </dependency>
        <dependency>
            <groupId>com.fasterxml.jackson.core</groupId>
            <artifactId>jackson-annotations</artifactId>
            <version>${jackson.version}</version>
        </dependency>

        <dependency>
            <groupId>org.junit.jupiter</groupId>
            <artifactId>junit-jupiter-engine</artifactId>
            <version>5.5.2</version>
            <scope>test</scope>
        </dependency>
    </dependencies>

    <build>
        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-surefire-plugin</artifactId>
                <version>3.0.0-M3</version>
            </plugin>
        </plugins>
    </build>
</project>

Test sample

package org.example.mve;

import static org.junit.jupiter.api.Assertions.assertThrows;

import java.io.IOException;
import java.util.ArrayList;
import java.util.Collection;
import java.util.Objects;

import org.junit.jupiter.api.Test;

import com.fasterxml.jackson.annotation.JsonCreator;
import com.fasterxml.jackson.annotation.JsonProperty;
import com.fasterxml.jackson.core.JsonParser;
import com.fasterxml.jackson.core.JsonToken;
import com.fasterxml.jackson.databind.DeserializationContext;
import com.fasterxml.jackson.databind.JavaType;
import com.fasterxml.jackson.databind.ObjectMapper;
import com.fasterxml.jackson.databind.deser.DeserializationProblemHandler;
import com.fasterxml.jackson.databind.exc.MismatchedInputException;
import com.fasterxml.jackson.databind.node.ObjectNode;

class ExceptionMappingProblemHandlerTest {

    private static final String PROP = "prop";

    private static final ObjectMapper MAPPER;

    static {
        MAPPER = new ObjectMapper();
        MAPPER.addHandler(new ExceptionMappingProblemHandler());
    }

    @Test
    void testHandleUnexpectedToken() {
        ObjectNode input = MAPPER.createObjectNode();
        input.set(PROP, MAPPER.createArrayNode());

        assertThrows(
            JsonStructuralMismatch.class,
            () -> MAPPER.treeToValue(input, StringProperty.class)
        );
    }

    @Test
    void testHandleUnexpectedTokenArray() {
        ObjectNode input = MAPPER.createObjectNode();
        input.put(PROP, "prop");

        assertThrows(
            JsonStructuralMismatch.class,
            () -> MAPPER.treeToValue(input, Array.class)
        );
    }

    static class Array {

        private final Collection<String> prop;

        private Array(Collection<String> prop) {
            this.prop = prop;
        }

        @JsonCreator
        static Array create(@JsonProperty(PROP) Iterable<String> prop) {
            ArrayList<String> list = new ArrayList<>();
            prop.forEach(list::add);
            return new Array(list);
        }

        @JsonProperty(PROP)
        public Iterable<String> getProp() {
            return prop;
        }
    }

    static class StringProperty {

        private final String prop;

        private StringProperty(String prop) {
            this.prop = Objects.requireNonNull(prop, "prop must not be null");
        }

        @JsonCreator
        static StringProperty create(@JsonProperty(PROP) String prop) {
            return new StringProperty(prop);
        }

        @JsonProperty(PROP)
        public String getProp() {
            return prop;
        }
    }

    public static final class ExceptionMappingProblemHandler extends DeserializationProblemHandler {

        @Override
        public Object handleUnexpectedToken(
            DeserializationContext ctx,
            Class<?> targetType,
            JsonToken token,
            JsonParser parser,
            String failureMsg
        ) throws IOException {
            throw new JsonStructuralMismatch(parser, "Some text here");
        }

        @Override
        public Object handleUnexpectedToken(
            DeserializationContext ctxt,
            JavaType targetType,
            JsonToken t,
            JsonParser parser,
            String failureMsg
        ) throws IOException {
            throw new JsonStructuralMismatch(parser, "Some text here");
        }

    }

    public static class JsonStructuralMismatch extends MismatchedInputException {

        JsonStructuralMismatch(JsonParser parser, String description) {
            super(parser, description);
        }
    }
}

Expected behavior
In the above test case, a JsonStructuralMismatch should be thrown for the array case as well.

[cowtowncoder]

(marking as 2.13 since fix, if any, unlikely to go in 2.12 since that branch will soon be closed).

One question: is it possible that a different method might be called on DeserializationProblemHandler instead?

[andpal]

Yes, it appears that handleInstantiationProblem is being called instead. The issue is that handleInstantiationProblem is called with Class<?> instClass which is the concrete type that Jackson tried to create instead of the one that was specified in treeToValue. Basically, what my problem handler implementation does is give specialized error message like Cannot create <sometype> from <sometoken> (<tokenToString>) to users of an API.

[cowtowncoder]

Hmmh. Tough call. I wish I had time to look into this but realistically speaking I don't.

I hope someone else with time can dig into this, to understand what the optimal failure cause would be.
Keeping in mind that just because List/array value is the target does not necessarily mean JSON input must be an array, esp. depending on settings.

[cowtowncoder]

Fixed. Similar issue likely affects Maps too.