Looks like the config recommendation for native ping causes issue with sudo

Changing this

[Service]
CapabilityBoundingSet=CAP_NET_RAW
AmbientCapabilities=CAP_NET_RAW

to this

[Service]
AmbientCapabilities=CAP_NET_RAW

and it fixes the issues