How to merge unique values for a child array when merging objects

Question

I want to merge the .environments.default and .environments.dev objects together, so that dev values override any default values and default values fill in anything not defined in the dev object. If I use .environments.default * .environments.dev most of the objects are merged correctly, except for the array lists under the env and secrets objects. The entire dev object is used, but I want the array lists to be merged with unique values.

Is there anyway to get this result with jq? If it helps I could also try doing this in yq with the YAML file.

I would also really like to know if this is possible without hardcoding "env" or "secrets" anywhere in the code.

When I use .environments.default * .environments.dev on the below source...

Source:

{
"environments": {
    "default": {
      "config": {
        "security": {
          "user": 0,
          "group": 0
        },
        "resources": {
          "limits": {
            "memory": "256Mi",
            "cpu": "500m"
          }
        },
        "env": [
          {
            "name": "DEFAULT_ENV",
            "value": "DEFAULT_ENV_VALUE"
          },
          {
            "name": "BASE_URL",
            "value": "DEFAULT_BASE_URL"
          }
        ],
        "secrets": [
          {
            "secretID": "567",
            "mountPath": "/default/dir/to/567/path"
          },
          {
            "secretID": "123",
            "mountPath": "/default/dir/to/123/path"
          },
          {
            "secretUUID": "789",
            "mountPath": "/default/dir/to/789/path"
          }
        ]
      }
    },
    "dev": {
      "config": {
        "replicas": 1,
        "resources": {
          "limits": {
            "cpu": "5000m"
          }
        },
        "env": [
          {
            "name": "BASE_URL",
            "value": "DEV_BASE_URL"
          },
          {
            "name": "DEV_ENV_1",
            "value": "DEV_ENV_1"
          }
        ],
        "secrets": [
          {
            "secretID": "456",
            "mountPath": "/dev/dir/to/456/path"
          },
          {
            "secretID": "123",
            "mountPath": "/dev/dir/to/456/path"
          }
        ]
      }
    }
  }
}

I get...

Current output:

{
  "config": {
    "security": {
      "user": 0,
      "group": 0
    },
    "resources": {
      "limits": {
        "memory": "256Mi",
        "cpu": "5000m"
      }
    },
    "env": [
      {
        "name": "BASE_URL",
        "value": "DEV_BASE_URL"
      },
      {
        "name": "DEV_ENV_1",
        "value": "DEV_ENV_1"
      }
    ],
    "secrets": [
      {
        "secretID": "456",
        "mountPath": "/dev/dir/to/456/path"
      },
      {
        "secretID": "123",
        "mountPath": "/dev/dir/to/456/path"
      }
    ],
    "replicas": 1
  }
}

but I want...

Desired output:

{
  "config": {
    "security": {
      "user": 0,
      "group": 0
    },
    "resources": {
      "limits": {
        "memory": "256Mi",
        "cpu": "5000m"
      }
    },
    "env": [
      {
        "name": "BASE_URL",
        "value": "DEV_BASE_URL"
      },
      {
        "name": "DEV_ENV_1",
        "value": "DEV_ENV_1"
      },
      {
        "name": "DEFAULT_ENV",
        "value": "DEFAULT_ENV_VALUE"
      }
    ],
    "secrets": [
      {
        "secretID": "456",
        "mountPath": "/dev/dir/to/456/path"
      },
      {
        "secretID": "123",
        "mountPath": "/dev/dir/to/456/path"
      },
      {
        "secretID": "567",
        "mountPath": "/default/dir/to/567/path"
      },
      {
        "secretUUID": "789",
        "mountPath": "/default/dir/to/789/path"
      }
    ],
    "replicas": 1
  }
}

I don't understand your merging strategy. What determines that .environments.default.config.env[0] is kept in the output array .config.env (at pos 2), while .environments.default.config.env[1] is dropped. The only reason I can imagine is, that the latter's .name value already occurs in the target array's items. But as its .value value still differs, I cannot just check whole array items, yet I should keep it dynamic and not use field names. Then how? Same applies to .environments.default.config.secrets[1] where its .secretID already exists in the target, but .mountPath doesn't. — pmf
– pmf, Commented Feb 15, 2024 at 3:56
The env list should be unique by name and the secrets list should be unique by secretID. This is a deployment configuration so I want environment specific information. There will be a testing and prod environment too, that will need to be merged with the default configuration. — Chris F
– Chris F, Commented Feb 15, 2024 at 4:03
Understood, but you want a solution without hardcoding "env" or "secrets" anywhere in the code., so how else can I address .env.name and .secrets.secretID? Do you expect me to use just .name and .secretID, and query for them on any array item I can find? — pmf
– pmf, Commented Feb 15, 2024 at 4:05
I was wondering if it could be unique by the first object key in the array? The env and secrets objects are optional. So I wanted to have a way to do the merge without a big if statement. And also future proof it a little for objects that don’t exist yet. — Chris F
– Chris F, Commented Feb 15, 2024 at 4:07
Object fields don't have an order conveyed by the format, only array items do. The order you observe in a specific file is just its current representation. Try changing the order of any fields within an object, then compare the altered version to the original. JSON processors (including jq and yq) are expected to yield that they are still equal. — pmf
– pmf, Commented Feb 15, 2024 at 4:07

pmf · Accepted Answer · 2024-02-15 07:41:14Z

1

From the comments thread:

The env list should be unique by name and the secrets list should be unique by secretID. […] I was wondering if it could be unique by the first object key in the array? […] name always comes before value and secretID always comes before mountPath

Object fields don't have an order conveyed by the format, only array items do. The order you observe in a specific file is just its current representation.

Just for the fun of it, how would you do it anyway?

So, let the fun begin:

This captures and iterates over all items of non-empty arrays from the .default branch (saving the array path with paths(arrays | select(has(0))), each item's value with getpath(…)[], and the first item's key and value using to_entries[0]). The iteration's base document is the merge you already performed, but with these arrays removed from the .default branch, so the saved items can be individually tested and appended to it if they pass the condition that no other first key's value matches theirs: IN(.[][$v0.key]; $v0.value) | not.

.environments | [.default | paths(arrays | select(has(0))) as $p
  | getpath($p)[] as $v | {$p,$v,v0:($v | to_entries[0])}
] as $a | reduce $a[] as {$p,$v,$v0} ((.default | delpaths([$a[].p])) * .dev;
  setpath($p; getpath($p) | . + [select(IN(.[][$v0.key]; $v0.value) | not) | $v])
)

Demo

answered Feb 15, 2024 at 7:41

pmf

38.4k3 gold badges31 silver badges41 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

Chris F Over a year ago

This is great, thank you so much for posting this. I can't wait to play around with it in jqplay and learn some new functionality of jq.

Chris F Over a year ago

Hi @pmf, This works great, thank you very much! It took me a few hours, but I think I understand most of the logic. I could have never figured this out on my own, thanks again.

Chris F Over a year ago

I added a little more logic to the "test and append" section for cases where the array exist in the default branch, but not the dev branch. Let me know what you think.

.environments |  [.default | paths(arrays | select(has(0))) as $p   | getpath($p)[] as $v | {$p,$v,v0:($v | to_entries[0])} ] as $a |  reduce $a[] as {$p,$v,$v0} ((.default | delpaths([$a[].p])) * .dev;   if getpath($p) == null then       setpath($p; getpath($p) | . + [$v])   else       setpath($p; getpath($p) | . + [select(IN(.[][$v0.key]; $v0.value) | not) | $v])   end )

Chris F Over a year ago

Also, just out of curiosity, I don't see any difference in the final output whether the arrays are removed from the .default branch or not. Is that done just because there is no reason for those arrays to even be considered as part of the initial merge or can you think of a situation where leaving the arrays in the default branch might produce the wrong result?

Collectives™ on Stack Overflow

How to merge unique values for a child array when merging objects

1 Answer 1

4 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

4 Comments

Your Answer

Sign up or log in

Post as a guest

Related