There are some problems with the unicorn. First, only you can see it. To everyone else it looks like, well, weird and peculiar, your flying. Then, remember, you cannot train your unicorn and if you ride only the unicorn, all the time, you’ll be hard to predict for everyone around. Being unpredictable constantly is the recipe for social seclusion. Being friends only with the invisible unicorn can be hard the older you get.
You also assume that up in the air no dangers await. Sometimes you’d need to flee like only the dark horse can or to command your victory only like the white horse can.
If you find the castle in the sky or the treasures at the end of the rainbow you’ll pat your back, alone, for choosing the unicorn, but alas, castles can’t be built in the sky and rainbows have no end :)