preg_replace() remove all non-numeric characters EXCEPT trailing x

preg_replace() remove all non-numeric characters EXCEPT trailing x

I have the following code to remove all non-numeric characters:
$num = preg_replace('/\D/', '', $num);
I'd like to make it so that it removes all numeric-characters except any
trailing X (not case sensitive).
ex:
s34kr = 34
xX4rx = 4x
rs5t928X = 5928X