I’m also using them for icons, and starting to use them more for graphics that I make when possible.
My only reservations is support for older browsers. From what I can find Internet Explorer 8 and back does not support SVG. Also some other older browsers. Hopefully this will work its self out with time.
Out of the box Bootstrap 4 do not supports Internet Explorer 8, so in general your future sites created with Blocs 3 will not render correctly on this version and older versions. Depending on the client in general, it is no problem any more to drop support for IE8
When using Foundation it has an option to redirect older versions of IE to an alternative page that you can build using old fashioned methods that basically predate responsive websites. It’s a good idea, but increasingly less relevant as time goes by, since the same users are probably seeing problems at most sites they visit now. It’s worth remembering that Microsoft dropped all support some time ago https://www.microsoft.com/en-gb/WindowsForBusiness/End-of-IE-support
I think you really need to know what sort of visitors your website typically receives and take a decision from there, since global statistics appear to be all over the place depending on where you search. Personally I am quite happy to use svg files where suitable.