final_pipeline appears to process twice when using the date match index processor

[edsyoo]

Elasticsearch Version

7.16.2

Installed Plugins

No response

Java Version

bundled

OS Version

Cloud

Problem Description

When using the final_pipeline option and the date match index in a pipeline to process data, the final_pipeline appears to run twice. This can be shown by using the append processor which adds two of the same values or a rename processor which shows a failure, but it will show the name of a field changed and shows a failure due to the old named field not being present.

Also referenced in a discuss topic to confirm that the example is valid.

Additional note: This seems to run fine on a 7.6.2 environment, so something between that and 7.16.2 may have caused this issue. #69727 or #75047 could be related.

Steps to Reproduce

Append method of showing the final_pipeline running twice:

PUT _ingest/pipeline/test-final-pipeline
{
  "processors": [
    {
      "append": {
        "field": "pipeline",
        "value": "completed"
      }
    }
  ]
}

PUT _template/test-template
{
  "index_patterns": [
    "test*"
  ],
  "settings": {
    "final_pipeline": "test-final-pipeline",
    "number_of_shards": 2
  }
}

POST /test-1/_doc
{
  "time-field": "2021-02-10T12:02:01.789Z",
  "CONTENT": "blah1"
}

PUT /_ingest/pipeline/routing-test-pipeline
{
  "description": "Time series DAY pipeline",
  "processors": [
     {
         "date_index_name": {
             "field": "time-field",
             "index_name_prefix": "test-",
             "date_rounding": "d",
             "index_name_format": "yyyy-MM-dd"
         }
     }
 ]
}

POST /test-1/_doc?pipeline=routing-test-pipeline
{
  "time-field": "2021-02-10T12:02:01.789Z",
  "CONTENT": "blah2"
}

Rename processor used to show a failure.

PUT _ingest/pipeline/test-final-pipeline
{
  "processors": [
    {
      "rename": {
        "field": "CONTENT",
        "target_field": "MALCONTENT"
      }
    }
  ]
}

PUT _template/test-template
{
  "index_patterns": [
    "test*"
  ],
  "settings": {
    "final_pipeline": "test-final-pipeline",
    "number_of_shards": 2
  }
}

POST /test-1/_doc
{
  "time-field": "2021-02-10T12:02:01.789Z",
  "CONTENT": "blah1"
}

PUT /_ingest/pipeline/routing-test-pipeline
{
  "description": "Time series DAY pipeline",
  "processors": [
     {
         "date_index_name": {
             "field": "time-field",
             "index_name_prefix": "test-",
             "date_rounding": "d",
             "index_name_format": "yyyy-MM-dd"
         }
     }
 ]
}

POST /test-1/_doc?pipeline=routing-test-pipeline
{
  "time-field": "2021-02-10T12:02:01.789Z",
  "CONTENT": "blah2"
}

Logs (if relevant)

No response

[elasticmachine]

Pinging @elastic/es-data-management (Team:Data Management)

[HiDAl]

Hello, I've tested this in the latest version of Elasticsearch and the issue is still occurring. Even more, I was able to recreating other scenarios where the issue is happening. Basically, every time you update the _index information for the current request, the final_pipeline will be executed twice.

This is another version, this time using a script processor:

PUT _ingest/pipeline/test-final-pipeline
{
    "processors": [
        {
            "set": {
                "field": "some-field",
                "value": "value"
            },
            "append": {
                "field": "pipeline",
                "value": [
                    "completed",
                    "{{{_index}}}"
                ]
            },
            "script": {
                "lang": "painless",
                "source": "ctx.CONTENT += ' >> ';" // mimic an append, just to validate the whole pipeline is executed twice and not only the final processor
            }
        }
    ]
}


PUT _template/test-template
{
  "index_patterns": [
    "test*"
  ],
  "settings": {
    "final_pipeline": "test-final-pipeline",
    "number_of_shards": 3
  }
}


PUT _ingest/pipeline/routing-test-pipeline
{
    "description": "Time series DAY pipeline",
    "processors": [
        {
            "date_index_name": {
                "field": "time-field",
                "index_name_prefix": "test-",
                "date_rounding": "d",
                "index_name_format": "yyyy-MM-dd"
            },
            "set": {
                "field": "where",
                "value": "from routing-test-pipeline"
            }
        }
    ]
}

PUT _ingest/pipeline/change-by-script
{
    "processors": [
        {
            "set": {
                "field": "where",
                "value": "from changed-by-script"
            },
            "script": {
                "lang": "painless",
                "source": "ctx._index = 'testing-index-final';"
            }
        }
    ]
}

The following document creation will execute the final_pipeline twice:

POST /test-1/_doc?pipeline=routing-test-pipeline
{
  "time-field": "2022-02-10T12:02:01.789Z",
  "CONTENT": "routing-pipeline"
}

POST /test-1/_doc?pipeline=change-by-script
{
    "time-field": "2022-02-10T12:02:01.789Z",
    "CONTENT": "change-by-æscript"
}

Right now I'm working on a bugfix for this issue.