Branch targets should be aligned, but otherwise there is no difference generally.
From the FU740-C000 manual (that's the SoC in the SiFive Unmatched)
3.2.5 Instruction Fetch Unit
The S7 instruction fetch unit is responsible for keeping the pipeline fed with instructions from memory. The instruction fetch unit delivers up to 8 bytes of instructions per clock cycle to support superscalar instruction execution. Fetches are always word-aligned and there is a one-cycle penalty for branching to a 32-bit instruction that is not word-aligned.
The S7 implements the standard Compressed (C) extension to the RISC‑V architecture, which allows for 16-bit RISC‑V instructions. As four 16-bit instructions can be fetched per cycle, the instruction fetch unit can be idle when executing programs comprised mostly of compressed 16-bit instructions. This reduces memory accesses and power consumption.
(...)
3.2.6 Branch Prediction
(...)
The BHT is a correlating predictor that supports long branch histories. The BTB has one-cycle latency, so that correctly predicted branches and direct jumps result in no penalty, provided the target is 8-byte aligned.
It is thus advisable that branch targets be aligned to 8 bytes or to at least align 32-bit instructions that are branch targets to 4 bytes. No penalty is mentioned for unaligned 32-bit instructions in an instruction stream, so there most likely is none.