The basic argument is "design a protocol (damn-it)".
I agree with this point, I've made it here and I don't think I'm the only one.
But everyone knows the Diaspora folks went the "product first, protocol later" route and a fair number of people agree this approach. So it's a disingenuous little to act "oh, they made a basic mistake so it just won't work." They went a route, at least, that lots of people think will work. It's not like they used hard coded eight-bit node identifiers, had each node multi-cast its information or some other design decision would guarantee things couldn't work.
Naively, on first glance, to me Facebook looked like just a nice interface on top of standard Internet functionalities. And there are standard protocols for all of these.
* Messengering we have
* Micro-blogging we have.
* Selectively sharing information is pretty simple. Give your "friends" a password to your website and let them connect via SSL or Oauth or something.
...
Nothing seemed that new. My first thought was, "why couldn't you just slam a UI/Application on top of this existing stuff and call it an open Facebook implementation"? Ah, but there is a funny thing I realized looking at the situation (and discussing it here). The unique selling point of Facebook and its open competitors boils down a "share and revoke" function. The umpteen Facebook users want to imagine they are "safe" when "privately" sharing their photos. They don't look too carefully into what "safety" or "privacy" involves and they probably don't want to look that carefully but they really want to have it. Facebook has flattered them into thinking they can just carry their real world intuitions into virtual space. Of course, this isn't quite "real world privacy" as a number of fired employees can atest.
Thus the "competing Facebook protocols" aim to provide something like "personal DRM". Without going into the details of the inherent problems, I'll just ask, "why would you want to?".
I would argue that rather than trying to simulate something that's impossible, that Facebook doesn't and can't give you, why not educate people. Allow secure sharing of information but start people learning about what makes sense to share or not. And sure, that's a big job but the Internet itself is kind of doing that.
Looking at the situation as a whole, Facebook is leveraging people's lazy, foggy and "intuitive" notions of privacy taken from the real world. Anything Facebook-like is going to "leak-like-a-sieve" and the more people learn about the Internet, the more they will realize this.
I don't agree that it makes sense to "design a protocol" before building the application. You can try to imagine what you want the protocol to look like at the end of the day, but you are likely to get it wrong if you don't have an application that's actually using it. It's important to keep your eyes open and try to avoid going down blind alleys, but there's not much value in building infrastructure before it's needed.
The questions raised in that post are good, and we intend to address them as they become relevant to what we have build. Discovery, namespacing and routing are areas where we are starting to feel tension on in the product, and will probably be tackled quite soon. The point about affecting interface design is good, but we look at it the other way. We want our interface design to drive the protocol design, so we want to make protocol choices as late as is reasonable.
Finally, it's fantastic that people are reading and forking the code. The more people out there trying things, the better.
I agree - it's a fantastic thing to be forking the code. The side effect is that with each fork whatever protocol Diaspora have chosen becomes more entrenched and difficult to change.
What I was trying to get at in the article was that app designers need to think about the protocol at some point since questions such as "are we relying on long polling or will the user refresh pages" start to play a question on the UI side too.
Also, if a product is marketed as being distributed, some thought as to how this will work and scale become important.
It's really good that Diaspora are thinking about what the user sees and programming from the user's POV. I spend a lot of time in the XMPP community and there is not enough of this focus. But a good programmer has to balance design with functionality and focus on many areas and some more thought on the protocol side would not go amiss.
We're very aware that we will be breaking interoperability with forks when we change the protocol, and there is a little bit of a worry that people won't keep their forks up to date and the protocol will fragment. I am optimistic that that won't happen, largely because the Diaspora product is the only part of the system actually visible to the outside world at the moment, and that gives us a pretty good way to draw a bright line between products that interoperate and products that don't.
I don't think that the UI polling/refresh question affects the inter-server protocol. Your seed is not your browser, and it's seed-seed communication that's the part of the protocol that matters in the long run.
Scaling is the pressure we're feeling most right now, and we're talking about how to deal with the issue of a distributed cluster with a coherent namespace. There are some interesting things about the Dynamo pattern that might be really useful to make this work. But without a running system with load and users and remote pods to connect to, it's very hard to figure out what you want.
I think that were at the point where the protocol needs something of a refactor, and I will probably make a post about it on the Diaspora-dev google group soon. Maybe we'll get some good feedback and design help!
As someone who has been working on this for a very long, my advice is to write the software to be protocol agnostic. Be able to switch out protocols easily, and support multiple protocols concurrently, because we still have no idea which protocol will win out. It may be something that hasn't even been built yet.
I agree, but this is harder than it sounds. Why? If this is your first time building such a system you don't even know what is protocol and what's not.
There's an old saying that you need to build three apps on your framework. That first app shows that your framework works for something. The second app shows that its more general than just the first app. The third app shows that you weren't lucky.
Curious, how was Facebook designed? Are there any post-mortem docs on it? I know that MySpace seemed really ad hoc. I wouldn't be surprised if Facebook was more ad hoc than we imagine it to have been
How do you measure "more ad hoc"? The only software that's not ad hoc are huge enterprise projects that fail most of the time.
If I had to guess I'd say that both MySpace and Facebook were completely ad hoc, but the difference was probably that Facebook kept their technical debt down so that they never accumulated an area of code that was too hairy to refactor or replace. MySpace on the other hand probably got complacent in their success and didn't worry too much as their architecture calcified into an unmaintainable mess. By the time they saw Facebook nipping at their heels their code base was past the point of no return, and the company culture made it impossible to hire the level of technical talent they would have needed to right the ship.
Of course I'm completely talking out my ass here, but that's the impression given by the respective product histories.
Who doesn’t imagine Facebook’s early design to be “ad hoc”? It was programmed as a spaghetti mess of PHP on Zuckerberg’s tiny laptop, and only worked because the user base was inherently limited to a few thousand people. Then the spaghetti mess was grown and added to over the next year or two, and the system scaled because each college’s network could be contained in a single server that only had to support a few thousand users (these servers used to use domain names like harvard.thefacebook.com or mit.thefacebook.com, IIRC). At some point they completely rewrote it with an architecture that would scale and not waste hardware, but it definitely didn’t start out that way.
Perhaps I misunderstand you, but "protocol agnostic" is a red flag to me. SOAP was designed to be protocol agnostic (there were specs for SOAP-over-SMTP and XMPP and so on and so on) and look how that turned out.
As the OP points out, protocol cannot be truly decoupled from architecture, especially when it comes to things like routing. There's a huge difference between a Twitter-like @user, an SMTP-like user@domain, a URL-based system like OpenID, etc.
There's always a trade off for just how agnostic you can be, so there's no way to support every possibility. But if you look at the different federated social web projects, there are clear similarities between them (ie, identity is in a username@domain format), so you can easily build around those similarities.
For instance, building around events ("posting a status update", "sending a friend request", "verifying an identity") and abstracting out those events into hooks is how Appleseed does it.
I have no misconceptions that this will hold up to every possible social protocol, past, present and future, but thinking about the protocol as an abstraction, and building the platform out from there, has already shown itself to be very useful.
I think Diaspora could do itself a huge favor by writing up a blog post explaining how the $200k was distributed. Everyone keeps harboring on the fact that they have that much money, but everyone forgets that they have quite a bit of liability with what they guaranteed on Kickstarter.
I think once that is done, people won't be so fast to suggest to them to hire a $100k/year engineer from Google or other things. Just my $0.02...
Yeah, they have to send out 3,296 t-shirts alone, and even if they get a great deal, and end up paying $5 each (incl. shipping & handling), that's $16,480 just in t-shirts.
Maybe the old software engineering maxim "build one and throw it away" is the best way to look at what Diaspora's done so far.
Yes, it would be great to have a well-thought-out protocol and secure software. But then again there's also huge value in having an implementation that people can play with. We hear all the time about how important it is to get your best guess at an minimum viable product into users' hands, and that's pretty much what Diaspora has done.
The article made the point about federation algo, and I think that's the most important: proper federation/decentralization IS the product. As of yet it is an unsolved problem in their camp (and many others). This isn't to take anything away from them, because its a Big Problem, but if they can't solve that, a slightly polished CRUD social networking app is just fluff.
The product is the protocol, not the app. That's the only potentially innovative thing they are doing. Sorry guys, aspects are a product made for privacy concerned software devs - normies don't care.
great comment, I had only one quibble or expansion on what you said:
> Sorry guys, aspects are a product made for privacy concerned software devs - normies don't care.
I'd argue that normals care about this too. Teens might want to keep their friend/school/personal stuff private from their family/friends-of-family/adults-they-know stuff. And adults often want to keep their "adult adult" stuff hidden or distinct from their family and/or coworkers. The whole "my mom wants to friend me on Facebook, ew!" or "my boss wants to friend me on Facebook, noooo!" factor.
Agreed 100%, but this feature alone isn't worth leaving facebook due to network effect. That may change, but if you have to boil it down to a black and white, it still leans far to the side of "not worth it to switch." Also, should this change, I find it highly unlikely facebook would find itself unable to pivot. Regardless of what you think of fb (I personally hate it) you really can't deny they have a world class engineering team that has a finger on the social pulse more than any other company right now.
I have tried building apps around these kinds of features, they don't really work (you end up building a new interface to email awfully quickly).
I think a huge requirement for success is that distributed social networking needs to be a framework, not simply an application (as Diaspora is now). Hopefully they're considering refactoring in that direction, and that definitely falls into point #1 (Coding before designing). You don't want to haphazardly code a framework, you need to design it on some level beforehand.
The killer feature that moves people over to distributed social networking is not distributed social networking itself. But third party developers building onto a distributed social networking framework is what will allow the next killer feature to be built.
Actually, no. Virtually all successful frameworks (Rails, Django, Tornado, MapReduce, etc.) grow out of an application or set of applications where common pieces are incrementally refactored out of the application and into the framework. When you start with the framework and design it upfront, you end up with an overdesigned mess.
Actually, I agree. My framework was built out of Appleseed the application, and was then re-factored into a framework, and most of the structure was based on what I had learned from building the app.
I don't think anyone should start by building a framework without any conception of what that framework is meant to do. But Diaspora already has an app of sorts, so they wouldn't exactly be doing that.
I share your frustrations about the quality of legacy Java web frameworks (though that Play one looks interesting, or would be if I had any interest in ever writing Java again). That said, I voted you down rather than voting you up to express my preference for HN having a high signal-to-smug ratio.
I voted you an aggregate of zero - +1 because I agree with you, -1 because I dislike people explaining their voting.
:)
I'm just teasing, no offense intended. I actually do agree with you and upvoted you - your comment just really reminded me of this quote from Steve Yegge (supposedly found on Reddit): "I upvoted you for the appropriate uses of 'its' and 'their' in the link title, but downvoted you because your link actually appeared on a little-known German social networking site several hours ago. I feel it is important that you understand that this is not a zero-vote of abstention, but rather a single upvote and a single downvote cancelling one another out."
A concretely realized framework isn't all that much more abstract than an application. Creating a framework doesn't mean you've created a well-defined interface, much less a well-defined protocol.
Moreover, a protocol is pretty much the only thing that has meaning over-a-wire. After all, you don't know what object you're talking to on the other side of a network connection so either you each have a well-defined protocol you're using or you don't.
I think the huge requirement for success is that is grows at grass roots level with completely non-technical people because of non-software reasons. I don't think you can compete with something like FB if your scope is "we need to develop better software" unless you have a whole other aspect of creative and analysis skills that this kind of development drive do not posses or engage.
I would say that both are important. The underlying architecture needs to be very flexible, and the user-facing interface must be familiar and intuitive to the lcd.
I'm struggling to think of examples where an open application has defined both a successful architecture/protocol and also provided a widely adopted user-interface.
For example, not many people used Tim Berner-Lee's original web browser but it did help define the standards that then went on to be picked up by Mosaic which was the first really popular web browser. However, by the point that Mosaic came out most of the features of the architecture of the web were in place and remain recognizable to this day.
Surely the goal is to provide common lower levels of a "social" protocol stack and allow as many and different user interface applications to layer on that common core?
Surely the goal is to provide common lower levels of a "social" protocol stack and allow as many and different user interface applications to layer on that common core?
That's one of the goals, but it's not the only goal. I think what you're bringing up is what makes social networking somewhat new terrain for open source. So far, open source has been able to dominate the server space because of superior tech implementations for an insanely low price point.
For social applications, that doesn't matter as much when you're trying to draw users. You need a usable UI, you need to learn all the lessons that the proprietary systems have learned over the years. It's not good enough to just have a better technology, you have to present it as well or better than the competition.
I think it's a new trajectory, because there's less incentive for walled gardens to open up than there was for Netscape to use http. So I think this is a necessary trail for open source to blaze through.
So in a lot of ways, there's very much a chicken-or-the-egg dilemma: Usable protocol stack that others can build on, or sizable enough user base that developers want to build on it. And the closest I've seen to open source dealing with these questions was actually, the gnu/linux desktop projects.
I think Mosaic actually used the CERN libwww library and I'm pretty sure that the CERN HTTP server was the standard at the time Mosaic was first released.
It would be interesting if there was an equivalent for social applications... a common server component, an application library and maybe a reference application but with plenty of scope for people to innovate in the UI without the need to continually reinvent the core communication components if they don't want to.
I think the problem is a bit higher level then that. This project feels like it lacks the necessary experienced leadership. What they are doing is Not Easy and this is being borne out by their product. Building apps isn't too hard, designing systems is.
Perhaps it was overly general and poorly worded, but I think the underlying point is clear and valid: building an open source distributed system intended to be used globally by diverse individuals and organizations is several orders of magnitude harder than building an app that lives within the doors of a single organization.
I want something that is also a platform. Maybe that's what you meant, but I want a decentralized alternative to Facebook Connect. I want to clone Google Apps and beat them at "+1" with an open data platform that anyone can build an app on top of. But then again, I have my own grand plans in my head. And on paper. And scrawled on a desk.
I have really bit my tongue since the beginning of diaspora - I think really they are simply victims of their own success. If they had a normal amount of publicity, or if they had gotten the 10 they asked for instead of 200 I think they'd be much better off. Much hungrier, scrappier. Less expectations to live up to. Way less people looking over their shoulder as they learn to code and manage a project. It's got to be a bitch to learn to dance on live TV as well.
I am not saying they are doing a bad job. Just that it's a very hard job to do. Most rock stars of today that built the next big thing had the luxury of not having to show anyone the code. And I'd say 90% of the time, that code is fucked, at least in parts. That's pretty recoverable - your product isn't the code it's the service or executable. Many opensource projects are similar - they can be hacky at first because no one even knows they exist - as they get traction they can fix things as they become issues.
But now with the whole internet breathing down their necks, celebrity status and a bunch of money burning a hole in their pockets they have no such luxuries at all.
I am inclined to agree though that they should really consider closing back up and rebooting both the design philosophy and the code base after folding a few key contributors into the core team.
Not that I'm right or that I know what I'm talking about, but just my opinion.
I think there's two issues with this, though. One, it's somewhat dangerous to have an incompatible fork so early on. Of course, the code is so immature that this may not be as big of a deal at this point.
The other is that I'm not sold on XMPP as the best protocol for distributed social networking. I think it can be overly verbose, and a simple protocol over HTTP/HTTPS makes more sense to me, and I've had good luck with that so far. That said, I'm not completely sold against XMPP, either.
Protocol and the architecture. Protocol-wise I have no problem with http - it's well understood and any RoR dev can pick it up an use it. And indeed you could even, with a lot of work, build a federated network off http(s).
But http is not a a federation architecture. XMPP provides this out the box and makes bootstrapping a secure federated social network much easier.
Although the software is protocol-agnostic, and hooks could be written to support XMPP, as well.
I think my biggest issue with XMPP (beyond it's verboseness) is that it requires a whole separate XMPP server. This is less of an issue for Diaspora, which already requires a series of services running, but for my project, Appleseed, the goal is to get everything running under a LAMP stack which can be run on any shared host.
Don't worry about the verboseness of XMPP - TLS with the deflate option drops that down to 10% of the original size. Or you can do what google did on Android and write your own binary protocol representation.
If I setup a new appleseed node and how do you really know it is me/example.com? I've yet to see that done well in http and would love to be proven wrong.
Right now, all requests require a callback with a token. user a@a.com sends a request to b@b.com with a generated token. b@b.com then sends a verification request back with that token to a@a.com, and a@a.com sends back true, and b@b.com completes the request.
Which, if I'm not mistaken, is very similar to how XMPP does it.
The protocol also tries and minimizes the amount of actions that a third party node can take. You only trust your own node to take actions on your behalf.
"My concern is that hype can also bite when you don't have something to ship or there isn't a big value-add over existing solutions."
This is critical. Diaspora could spend a lot of time in the architecture tower with nothing to show, and the early hype will bite them if they have nothing to show. I think they've done a good job making sure that doesn't happen.
Now that they've shown they can produce something, they need to slow down and make sure they are producing the right thing. When you are building a simple application, you can get away with design-by-coding; if you are trying to build a federated, global, social wonder, you are going to need to spend some time thinking about it.
Just to add to the chorus of voices: yes, Disapora team, please let your protocol spec define your application code, not the other way around. This one factor will make the most difference to the long term success of the project.
At this point, I don't think diaspora will work. The project has lost its hype, so it has no more fuel. Its heading towards the deadpool. It doesn't even look like it'll live on as an open source project.
The main issues with the diaspora project, besides buggy code, was that there was never a good reason to build it. People were not angry enough at Facebook to start learning how to run their own servers (which costs money) or pay diaspora to do it (which costs money)
This is one case where I think the number "Four" in the original title would have been better left in. Without the number, the title made me think this site might be a community bug tracker for Diaspora design mistakes. With the number, it would be clear that this is one person's many ideas.
Is it me or can't Diaspora just make it easier to run your own Seed server, even for playing around. Come on guys, stick in on a EC2 AMI and make it easier for us to play.
(Before you retort, yes, I am capable of spending the hour or so build the stack, but it's nice to have plug and play sometimes).
> If the original title begins with a number or number + gratuitous adjective, we'd appreciate it if you'd crop it. E.g. translate "10 Ways To Do X" to "How To Do X," and "14 Amazing Ys" to "Ys." Exception: when the number is meaningful, e.g. "The 5 Platonic Solids."
Because Diaspora is open-source, it attracts this sort of "look at how bad the code is" attention. I'm willing to bet that the first prototype of yahoo/google/facebook all had terrible code that may have lasted a long time. However, by not being open source, no one was able to critique them on their software design mistakes and instead had to focus on the product itself.
Since it is open source, Diaspora could probably use the writers of these critique pieces (bad design, security holes, etc.) to actually make a better product.
(Tangentially, why are there so many blog posts about Diaspora's code? Is the code actually unforkable? Or is it just because Diaspora painted itself into a corner promising a secure social network on a relatively quick timeframe and now everyone wants to prove them wrong?)
Since it is open source, Diaspora could probably use the writers of these critique pieces (bad design, security holes, etc.) to actually make a better product.
The writer of this critique, as well as myself, are both working on what we feel are better products, respectively.
I agree with this point, I've made it here and I don't think I'm the only one.
But everyone knows the Diaspora folks went the "product first, protocol later" route and a fair number of people agree this approach. So it's a disingenuous little to act "oh, they made a basic mistake so it just won't work." They went a route, at least, that lots of people think will work. It's not like they used hard coded eight-bit node identifiers, had each node multi-cast its information or some other design decision would guarantee things couldn't work.