Worth replacing amp or speakers if replacing head unit?

I have a 2006 gen II with JBL and NAV . I'm planning on buying new head unit to get Apple CarPlay. I'm the original owner, and have realized that we're going to keep this car for at least a few more years (it only has 110,000 miles and even if I bought a new car, one of my kids would want it - we all love it). Here's the question:

If I'm buying a new head unit and having it professionally installed, is it worth replacing either the JBL amp and/or the JBL speakers? How much would I have to spend to make it clearly sound better?

Even though you might (accurately) guess that I'm cheap because I'm still driving my 2006, I might splurge given that a new car would cost me a lot more in car tax, insurance, etc. I figure the new head unit will be something like $300-$600 (depending on wireless CarPlay or not). Would a really meaningful improvement in everything else be closer to an additional $500 (I'd probably do it), $1000 (I might do it as a splurge) or $1,500 (too much to splurge on an old car, especially for my 55 year old ears)?

Also, would I lose the center speaker?

 

Thanks, that's very helpful. I've since talked to an installer fwiw they agree with you - they said the JBL speakers are decent, they should sound even better with a new head unit, and because of the impedance issue, you'd pretty much have to replace the amp and all the speakers if you wanted to change anything, and to get bass as good as the JBL you'd probably have to add a subwoofer. They definitely weren't trying to upsell me - they were saying it wasn't worth it.

It's more expensive, but I'm going to have it installed instead of going with Crutchfield - there are a lot of good instructions (on here and from Crutchfield, from what I understand), but I'm a slow learner and have a tendency to break things when I take them apart. From what I've seen here, if I took my prius dash apart that much, it would scare me I'd never get it back the same.