Both! Persona 3 and 4 mastered a happy balance, in my opinion. See my post An Analysis on Persona 4.
I need an objective to make me want to play a game, even if it's nothing more than "An Italian plumber must travel through magic drain-pipes to save the mushroom princess from evil turtle/dragons"....(see, the plot of Mario sounds really interesting even though it's a simple game). "You can plant turnips or go fishing or gamble!" on the other hand...not so exciting to me.
From what it sounds like, you are trying to make a game with a bunch of mini-games that can appeal to different players. That's fine, but I think you need some kind of backstory behind the world that the player is in. Give it a name and a history. Develop some interesting characters for the player to interact with. The plot doesn't have to "drive" the game, per se, but it can make your world worth exploring.