Five minute penalty?

When our Prius first starts up from cold the first 5 minutes or so the MPG is pretty poor. 25-30 MPG is typical for the first bar that shows up in the fuel efficiency display. That is I assume some combination of the ICE and the catalytic converter getting up to temperature, getting out of open loop, and so forth. Lower fuel economy shortly after startup is pretty typical for cars.

However, lately I have noticed that we sometimes get another "five minute penalty" after we make a short stop. And sometimes not. For instance, today we drove about 15 minutes, and then made a 10 minute stop where the car was turned off. When the car started back up the fuel efficiency display (eventually) showed another 5 minute penalty. We drove another 15 minutes, long enough for fuel economy to get back up go around 50 MPG, and made another stop. This one my wife went into the store without me for 10 minutes, and I sat in the car without shutting it off, leaving the transmission in P. When we drove away there was no 5 minute penalty, the fuel economy was again near 50, despite the ICE motor having been off for the entire 10 minutes, the same length as before. Drove 15 more minutes to yet another store. Shopped for 25 minutes (car off), and when we drove away, no 5 minute penalty, it was right back at 50 MPG from the get go.

Am I hallucinating here, or is there some logic in the car that sometimes triggers another longish warm up period under some conditions?

Note the weather was not at all extreme, it was around 70F for the whole trip, the sky was clear, the sun was out, and there was no wind to speak of. The first 10 minutes was partially on the highway, and all the rest was on suburban streets with light traffic at top speeds of 35 or 40 mph.