With audio, it doesn't. Best to get out before you find your own point of diminishing returns. They kick in early and hard with audio gear.
The best advice is above. Spread the quality and spend less. Some 'high end' 70s and 80 gear on ebay can be just as good as today's gear. It may not be as small or blingy, but it will be a fraction of the cost.
Very good hifi can be had for relatively little money. The tech is 'old' and it doesn't evolve anything like PCs and processors. Arguably the iPod created the first big leap over CD, and that took nearly 30 years. Some would argue that's taken music quality backwards. But in reality, that's more often happening in the recording process.
Always remeber the weakest link thing, and dpending on your choice in music, it could be right there at the source. Very few bands left these days that really master a good recording, which will allow your hard earned to hear the difference.