PCM audio data strange values in NAudio

I want to make an application that reads out the db values from a microphone. For that I use NAudio to "record" a WaveIn stream, convert the Data to samples, get the max of each buffer, average the buffer maxima over 1 sec and convert those to dbfs. I use a WPF application on .NET 10 With 32 bit floating point audio everything works fine but when using 24bit PCM everything breaks. I only either get 0 (on audio off) or ~16777184 as soon as I get audio.

I also noticed that the buffer looks a bit weird:

160 234 255 160 234 255 128 233 255 128 233 255 64 232 255 64 232 255 32 231 255 32 231 255 0 230 255 0 230 255 224 228 255 224 228 255 192 227 255 192 227 255 160 226 255 160 226 255 160 225 255 160 225 255 128 224 255 128 224 255 128 223 255 128 223 255 160 222 255 160 222 255 160 221 255 160 221 255 192 220 255 192 220 255 192 219 255 192 219 255 0 219 255 0 219 255 32 218 255 32 218 255 96 217 255 96 217 255 128 216 255 128 216 255 224 215 255 224 215 255 32 215 255 32 215 255 128 214 255 128 214 255 192 213 255 192 213 255 64 213 255 64 213 255 160 212 255 160 212 255 32 212 255 32 212 255 160 211 255 160 211 255 32 211 255 32 211 255 192 210 255 192 210 255 96 210 255 96 210 255 0 210 255 0 210 255 192 209 255 192 209 255 128 209 255 128 209 255 64 209 255 64 209 255 32 209 255 32 209 255 224 208 255 224 208 255 192 208 255 192 208 255 192 208 255 192 208 255 192 208 255 192 208 255 192 208 255 192 208 255 192 208 255 192 208 255 224 208 255 224 208 255 0 209 255 0 

(first 256 bits)

The values seem to grow (and later wrap around) going down the buffer with volume having no effect. I would understand the LSB to do this but not the rest. This leads to the calculation for the max allways returning a high number since there is allways a triplet near 255 255 255. Sometimes I get a bit of stable data where the second and third bit of each triplet goes up and down according to the input volume. Sometimes its the whole buffer, sometimes its just a few dozen bits and I dont know what causes this.

Here is the code for displaying the buffer:

        private void OnDataAvailablePCM(object sender, WaveInEventArgs args)
        {         
            StringBuilder sb = new StringBuilder();

            foreach (byte b in args.Buffer[0..256])
            {
                sb.Append(b.ToString() + " ");
            }
            
            Application.Current.Dispatcher.BeginInvoke(DispatcherPriority.Normal, new Action(() => {
                (window as MainWindow).bufb.Text = sb.ToString();
            }));
        }

And here the one used to calculate the max (derived from the NAudio documentation but changed to 24 bit Integer):

        private void OnDataAvailablePCM(object sender, WaveInEventArgs args)
        {
            float max = 0;
            // interpret as 24 bit audio
            for (int index = 0; index < args.BytesRecorded; index += 3)
            {
                int sample = (args.Buffer[index + 2] << 16) |
                                        (args.Buffer[index + 1] << 8) |
                                        args.Buffer[index + 0];

                if (sample > max) max = sample;
            }

            stPeaks.Add(max);

            Application.Current.Dispatcher.BeginInvoke(DispatcherPriority.Normal, new Action(() => {
                (window as MainWindow).bufb.Text = max.ToString();
            })); 
        }

I really dont know whats causing this problem. In Windows and Voicemeeter everything is set to 24 bit 96kHz 2 channels. NAudio is set to 100ms buffer (changing has no effect) and I use new WaveFormat(96000,24,2) as the wave format.

I am trying to figure this out for hours now so any input is appreciated. TIA