How can I validate the azp claim?

You can leverage Policy-based Authorization to achieve this. In your case, a simple function policy should help check the azp claim.

By default, you need to use this policy in Authorize attributes on your controllers, pages, or endpoints. Since you might want this for all endpoints, you could add it as an authorization filter.

You can use directly the event OnTokenValidated in the JwtBearerEvents when setup the AddMicrosoftIdentityWebApi().

builder.Services.AddAuthentication(JwtBearerDefaults.AuthenticationScheme)
        .AddMicrosoftIdentityWebApi(jwtBearerOptions =>
    {
        configuration.Bind("AzureAdB2C", jwtBearerOptions);

        jwtBearerOptions.Events = new JwtBearerEvents()
        {
            OnTokenValidated = async ctx =>
            {
                if (((JsonWebToken)ctx.SecurityToken).Azp != "< The AZP value to check >")
                {
                    ctx.Fail("Invalid authorized party.");
                }

                await Task.CompletedTask;
            },
        };
    },
    microsoftIdentityOptions => { configuration.Bind("AzureAdB2C", microsoftIdentityOptions); });