If you're going to have an MC server that big you're likely also going to want to load balance with bungee cord and run multiple gameserver instances. Running your own mod will likely be what holds you back the most from big growth. Because each client that connects to your server will also have to mod their client to be in sync with the server. Its far easier to get people to try plugins because they work on a native spigot or bukkit server which allows users to keep vanilla clients and not get Forge and learn how to install mod packs; which also puts them at a higher risk for malware when they seek shady sites to obtain Forge.
With all these considerations, renting hosting space would be far less of an initial investment and significantly less of a risk for when you first get off the ground. In the future a cheap hosted solution could also serve as a beta test for your mod; that way you will have your production side and your testing side. If i were you only after you're off the ground and you're certain your marketing, website, and server can function and features can be relied on with stability and performance, then I would look into purchasing a dedicated machine.