Perhaps we need berry farms and trips to the daycare. It's most likely due to that we're modeling our playing style after the games, and what do you usually do during downtime in the games? Breed? Grind? Create X-Berries? Go shopping? Hatch eggs?
In the anime, there was rarely ever any downtime that I can remember. Downtime was at the very beginning and very end of each episode and usually involved coming into the plot and then leaving it, so no real examples there. In the manga, again, it was coming into the plot and leaving it. Anything else was taken up by grinding, introspection, walks full of monologues, and cuts to other characters who were dealing with their own subplots. Again, no examples of downtime there.