If you're like me, you've run enough D&D 5th edition to notice that the whole "lowered hit point maximum" of most undead is at best a threat around level 1-3 but it quickly becomes just a tiny and rarely noticed detail with no measurable impact on the adventurers. At higher level gaming it's frankly a joke.
My thought is that this needs to be more threatening, and have a real impact on PCs. The idea is really simple: creatures that deal "hit point maximum" damage like undead should instead do hit die drain. The mechanic works like this:
In any case where the PC either has his or her hit point maximum lowered (saving throw potentially applicable) when the damage is dealt, instead apply the drain to hit dice. The PC loses 1 hit die for every 3 damage dealt that would ordinarily reduce the hit point maximum. Take 9 damage? Lose three hit dice instead.
If the undead turns the PC into a monster when hitting zero hit points (conditional to having a lowered hit point maximum) then the conversion instead happens when the PC hits zero HP and also has lost all hit dice due to drain.
You could even apply this as a general rule to all undead, in which any damage the undead deals also lowers hit dice at the same time. This could really make undead scary again, especially in numbers.
DMs could consider a few options....
CR Boost: Improved CRs for undead with this option would make sense.
Variable HD Loss: You could let the victim's class or size HD determine how many are lost by damage dealt. So a Warrior would lose 1 HD for ever 5 hit points dealt. I prefer the simpler method myself, but if I were to use this rule I'd round down on the HD for a slight edge to the undead.
Extreme Converters: Instead of the PC needing to hit zero HP and zero remaining HD, just let the conversion happen when the PC hits zero HD. This might be most suitable for really deadly horror-themed campaigns, or fantasy zombie apocalypse games.
Incorporeal Damage: instead of dealing HP damage have things like wraiths just deal HD damage directly; they never touch you physically, but your HD become the yardstick by which their sapping of life force can be measured.
Necrotic Resistance/Vulnerability: this is definitely damage of necrotic nature, so DMs could allow it to halve or double the damage used to determine HD loss. The easiest formula is to figure damage like normal first, then halve the HD loss result for resistance or double it for vulnerability.