The whole point of government is to ensure that your neighbors don't act like oricks. Obviously you don't want the neighbor to build something high that puts your property in permanent shadow, or otherwise ruin your property.
"The right to swing your fists ends where another man's nose begins" should be the motto to follow here. (And not "The right to swing your fists ends a statutory 3 meters away from any places that may contain noses")
If there is a specific noise/shadow or other detriment that is clearly caused to an adjoining property, then OK, the adjoining neighbor should be able to stop it (or sell the right to complain as an easement if they wish). If there isn't, and it just "looks ugly" and "isn't normal" then there is no meaningful detriment and they should be allowed to build
The problem is that there is a wide disagreement on what a 'clear detriment' to your neighbors property is. We all have different definitions.
If you think about it, zoning and land use laws are just the codifying of things that are detriments to your neighbors. Obviously, no one is going to agree with all of them, but they are what society has deemed worthy of forbidding.
OK, then I guess all I'm saying is I think my definitions are right and others' are wrong. My definition is:
Pretend there is a dark shroud covering away all boundaries of your property line. Whatever elements, whether noise, shadow, smell, smoke, etc enter into your property volume, you are allowed to complain about. Anything outside of that, you are not. How damned hard is that?
Having a view is something some people want and they're willing to pay for it. With your scheme, nobody can be sure they'll keep their view if the neighbor builds too high and blocks it.
I think almost all zoning laws actually meet your definition; there are very few zoning laws (that I know of) that specify what colors or shapes you can use on your property. They all deal with things that 'leak' outside your property.
Do you have any examples of zoning or land use laws that don't fit your definition of 'things we are allowed to regulate'?
That depends on in which culture and region you live. We have just built a new garage where we live. We have a quite lenient local government and great neighbours. The local government checked with the neighbours (10 of them, nobody objected), as we needed to replace a building that was too close to the border of the property (4 meters minimum distance) with something as close as what we replaced (1.5 meter), due to layout and topology of the property.
We needed to submit drawings of the garage, explain that it fit well with the main building, a distinct house from 1909, well known in the neighbourhood. Describe the colour to be used, including the colour on the garage door.
They come and check that the foundations get placed within four inches (10 cm) of what the building permit allows.
Do I mind? No. End result is good, took longer than expected due to heavy load on the city planning department.
In some villages in Switzerland you get fined if you don't have the right colour flowers on your balcony. (Or maybe the flowers showed up on your doorstep, with a mandatory bill, without having to ask.) :)
You might be surprised. There are certainly historic districts that can impose fairly explicit regulations on colors and shapes. One also reads about HOAs that cause trouble about insufficiently mowed lawns and the like although I don't know how common that is vs. "man bites dog" news stories.
San Francisco zoning laws actually just flat out discourage development. This is because San Francisco property owners like that their properties are becoming worth more so they vote that way.
In practice, having lived a couple houses from one (as a renter), it's more a theoretical problem than a practical one.
Given that the two condos that sold near me (one next door, and one two doors away) went in the 300k-350k range, it doesn't seem to be depressing the property values much either.
The complaint that's currently in vogue here in Seattle is that if someone builds high density housing in your neighborhood, then the people living there will want to park nearby, and suddenly the people living there lose their abundant and free street parking.