Back to the OP, I just noticed that the Toyota USA video on DRCC would mislead you into believing what the OP tried. Watch the video where it talks about using the up/down to change the speed -- at about the 1:00 mark. It doesn't say how it works exactly, but it shows the speedometer staying in lockstep with the set-speed indicator for both up and down. If you didn't know better, you might think you could watch the speedometer -- as the OP did. Toyota USA How-To Videos | PriusChat
No disagreement here. I've done exactly what the OP did myself. But, to be fair, this is a problem on all cars with electronic versus mechanical cruise control.
Mechanical Cruise Control - that goes back a long long way - I never drove one with that. I first drove a car with Cruise Control 20 yrs ago, and it and all since have been electronic.