Having experienced different setups, it seems that collaboration is OK if everyone is at the same place. It's also OK is everyone is remote. Where it gets troublesome is when part of the team is collocated and part is remote. What you get is two unequal classes of employees which leads to all kinds of problems.
I work part time remotely, and we have other devs full time remote. I definitely feel like I'm "missing out" just being OOO part time, be it the hoc discussions or just going out for beers.
Of course, it only matters if you let it matter. Some people don't care or need that stuff. They worry about their own work and leave work at work, which I'm working on doing :)