How big your garden gym should be, really comes down to what you plan on doing with it. If you are looking for a space for a treadmill, spinning bike and some dumbbells, then a 2.5 or 3m x 3m will be perfectly fine. If you want to just lift weights and are looking at power rack approach, then the same holds true. If you want to marry cardio equipment and free weights, then a 5m x 3m would be a great choice.
One client, went for a multi-purpose garden gym, that was a total size of 5m by 3m. A portion of that space was partitioned for a shed, leaving 3.5m by 3m which accomodates space for floor exercises, a power rack with landmine attachment, dumbell rack and room for a cable column too. More than adequate for an advanced full body resistance workout.