I kind of enjoy going into RSD and listening to them arguing about refs. The sides there are so eternally entrenched, and some posters there see literally every single post as form of this debate. Combine that with the "Jon Gabriel's greater internet fuckwad theory" and you have a recipe for debates that don't do much for anyone.
http://www.penny-arcade.com/comic/2004/03/19/The way I see it, self officiating makes ultimate attractive as a character building sport, lowers costs of participation, and makes the sport unique. On the downside, the concept is difficult to understand for younger players, makes it harder to become part of existing sports associations and makes the sport unique.
As long as observers are acceptable to the local athletic associations I think they fit the sport well and do a good enough job of minimizing the negatives without blunting the positives.