[BUG] MultiSearchTemplate partial success response is broken

[VachaShah]

What is the bug?

A user reported: When a multisearch template request is sent to the client and if the request partially succeeds i.e. one or more templates in the request fails, instead of returning an error response for that particular item, the client fails with the below exception and discards all the results from that request.

org.opensearch.client.util.MissingRequiredPropertyException: Missing required property 'ErrorResponse.status'
        at __randomizedtesting.SeedInfo.seed([B7831F18B61D38F2:88DBA2B968F5C73E]:0)
        at app//org.opensearch.client.util.ApiTypeHelper.requireNonNull(ApiTypeHelper.java:89)
        at app//org.opensearch.client.opensearch._types.ErrorResponse.<init>(ErrorResponse.java:72)
        at app//org.opensearch.client.opensearch._types.ErrorResponse$Builder.build(ErrorResponse.java:156)
        at app//org.opensearch.client.opensearch._types.ErrorResponse$Builder.build(ErrorResponse.java:119)
        at app//org.opensearch.client.json.ObjectBuilderDeserializer.deserialize(ObjectBuilderDeserializer.java:98)
        at app//org.opensearch.client.json.DelegatingDeserializer$SameType.deserialize(DelegatingDeserializer.java:60)
        at app//org.opensearch.client.json.UnionDeserializer$SingleMemberHandler.deserialize(UnionDeserializer.java:91)
        at app//org.opensearch.client.json.UnionDeserializer.deserialize(UnionDeserializer.java:326)
        at app//org.opensearch.client.json.JsonpDeserializerBase$ArrayDeserializer.deserialize(JsonpDeserializerBase.java:343)
        at app//org.opensearch.client.json.JsonpDeserializerBase$ArrayDeserializer.deserialize(JsonpDeserializerBase.java:308)
        at app//org.opensearch.client.json.JsonpDeserializer.deserialize(JsonpDeserializer.java:87)
        at app//org.opensearch.client.json.ObjectDeserializer$FieldObjectDeserializer.deserialize(ObjectDeserializer.java:81)
        at app//org.opensearch.client.json.ObjectDeserializer.deserialize(ObjectDeserializer.java:185)
        at app//org.opensearch.client.json.ObjectDeserializer.deserialize(ObjectDeserializer.java:146)
        at app//org.opensearch.client.json.JsonpDeserializer.deserialize(JsonpDeserializer.java:87)
        at app//org.opensearch.client.json.ObjectBuilderDeserializer.deserialize(ObjectBuilderDeserializer.java:91)
        at app//org.opensearch.client.json.DelegatingDeserializer$SameType.deserialize(DelegatingDeserializer.java:55)
        at app//org.opensearch.client.transport.endpoints.EndpointWithResponseMapperAttr$1.deserialize(EndpointWithResponseMapperAttr.java:68)
        at app//org.opensearch.client.transport.rest_client.RestClientTransport.decodeResponse(RestClientTransport.java:322)
        at app//org.opensearch.client.transport.rest_client.RestClientTransport.getHighLevelResponse(RestClientTransport.java:288)
        at app//org.opensearch.client.transport.rest_client.RestClientTransport.performRequest(RestClientTransport.java:148)
        at app//org.opensearch.client.opensearch.OpenSearchClient.msearchTemplate(OpenSearchClient.java:1006)
        at app//org.opensearch.client.opensearch.OpenSearchClient.msearchTemplate(OpenSearchClient.java:1022)

How can one reproduce the bug?

Run msearch_template request where one or more templates would fail.

What is the expected behavior?

All results from the server should be returned in this scenario.

What is your host/environment?

Operating system, version.

Do you have any screenshots?

If applicable, add screenshots to help explain your problem.

Do you have any additional context?

After investigating the issue, this happens because for msearch request, when one or more searches fail, the server returns a status:

{
      "took" : 601,
      "timed_out" : false,
      "_shards" : {
        "total" : 1,
        "successful" : 1,
        "skipped" : 0,
        "failed" : 0
      },
      "hits" : {
        "total" : {
          "value" : 0,
          "relation" : "eq"
        },
        "max_score" : null,
        "hits" : [ ]
      },
      "status" : 200
    }
  ]
}

whereas for msearch_template it does not

{
  "took" : 2,
  "responses" : [
    {
      "took" : 2,
      "timed_out" : false,
      "_shards" : {
        "total" : 1,
        "successful" : 1,
        "skipped" : 0,
        "failed" : 0
      },
      "hits" : {
        "total" : {
          "value" : 2,
          "relation" : "eq"
        },
        "max_score" : 0.6785375,
        "hits" : [
          {
            "_index" : "employees",
            "_id" : "2",
            "_score" : 0.6785375,
            "_source" : {
              "id" : 2,
              "name" : "Othilia Cathel",
              "email" : "[email protected]",
              "gender" : "female",
              "ip_address" : "3.164.153.228",
              "date_of_birth" : "22/07/1987",
              "company" : "Edgepulse",
              "position" : "Structural Engineer",
              "experience" : 11,
              "country" : "China",
              "phrase" : "Grass-roots heuristic help-desk",
              "salary" : 193530
            }
          },
          {
            "_index" : "employees",
            "_id" : "4",
            "_score" : 0.62577873,
            "_source" : {
              "id" : 4,
              "name" : "Alan Thomas",
              "email" : "[email protected]",
              "gender" : "male",
              "ip_address" : "200.47.210.95",
              "date_of_birth" : "11/12/1985",
              "company" : "Yamaha",
              "position" : "Resources Manager",
              "experience" : 12,
              "country" : "China",
              "phrase" : "Emulation of roots heuristic coherent systems",
              "salary" : 300000
            }
          }
        ]
      },
      "aggregations" : {
        "test-aggs" : {
          "doc_count_error_upper_bound" : 0,
          "sum_other_doc_count" : 0,
          "buckets" : [
            {
              "key" : 193530,
              "doc_count" : 1
            },
            {
              "key" : 300000,
              "doc_count" : 1
            }
          ]
        }
      },
      "suggest" : {
        "test-suggest" : [
          {
            "text" : "heuristic",
            "offset" : 0,
            "length" : 9,
            "options" : [ ]
          }
        ]
      }
    },
    {
      "error" : {
        "root_cause" : [
          {
            "type" : "resource_not_found_exception",
            "reason" : "unable to find script [my-other-search-template] in cluster state"
          }
        ],
        "type" : "resource_not_found_exception",
        "reason" : "unable to find script [my-other-search-template] in cluster state"
      }
    }
  ]
}