Previous Entry Share Next Entry
Release 0.3.17
jducoeur wrote in querki_project
Another internally-focused release today -- important from an architecture point of view, but mostly not even slightly user-visible, and actually not very code-visible.

Today's main enhancement was the camel's nose in the tent of sharding. For the non-techies, this is all about the fact that you don't want to keep your entire application in one database -- indeed, as you become bigger and bigger, it becomes nearly impossible to do so. (At least, in more traditional databases like the MySQL that Querki is currently using.) Instead, you need to think in terms of "sharding" -- breaking your database into lots of smaller "shards", each of which is a more appropriate size.

Querki was designed for sharding from the outset, and I expect to start doing it in earnest next year. But we hit a need to do so now, to split the common System Tables -- the core concept like Users and Identities -- from the Space Tables that contain the user-level data. As Aaron and I worked through requirements, we realized that we needed different levels of security for these different tables, and to keep him happy about it, that meant separate databases and separate logins.

(One of Aaron's major responsibilities as CIO is keeping me honest on the IT side. He is paranoid about IT security the same way I am about application security; between us, we know the landscape reasonably well.)

So all database calls now declare whether they are working with System or User data, and things get routed appropriately. This has precisely no effect on anything you will see, except that the new object IDs are much larger than the older ones, because the Space data is now all in Shard 2. (The Shard where something is created affects its object ID.)

Much more user-relevant, but not visible to anybody but me yet (since nobody else is editing yet): I've begun to make Add Property not suck. This has been a thorn in my side for months: while adding a Property to a Thing is conceptually easy, and easier in Querki than in most systems, it is still a heck of a lot more work than it should be. (And is ugly and confusing, to boot.)

So the first half of that has been done. There is now an "Add Property" button; pressing it opens an inline dialog that lets you choose the property to add, and shows you the documentation on the selected property. It's all been AJAXified, so it works much more smoothly and gracefully than the previous approach, which required page roundtrips.

The next step is to embed Create New Property into the same workflow. Currently, Create Property is completely separate from creating and editing Things -- at the airy architectural level that makes sense, because a Property is internally more or less a Thing itself, and should kinda-sorta use the same conceptual UI.

In practice, though, that turns out to be dumb. You *use* Properties very differently from Things, and you *construct* them very differently. In practice, it's rare to use all the potential power in the Property-creation UI, and it's way too much hassle -- what you *want* is to be able to quickly add a new Property when you realize you need it, which is almost always when you want to add it to a Thing that you're editing.

So the new Create Property workflow is an alternative to Add Existing Property. It's going to be more "hard-coded" than I would prefer in many ways, but it will gradually become pretty context-sensitive, with the system offering you the logical options as you build the new property, instead of requiring you to understand all the subtle interactions of the meta-Properties. Hopefully this pass will get to the point where it makes sense to people other than me.

And in my "spare" time, I'm finally actually writing down the business plan. The Paul Graham essays have been very useful for getting my head together there, and raised some interesting questions; sitting down and running concrete numbers forward revealed some places where I'm going to need to be even more hard-headed about strategy than I'd previously realized. Suffice it to say, even 10%-per-week growth doesn't get me where I want to be soon enough. So be ready for me to start using the term "Inflationary Period" early next year (in the cosmological sense) -- I basically am going to need to kickstart things with a period of near-impossible growth for a little while. I have a pretty good sense of how to do that; we'll see if it works.

In general, though, the business plan is *not* going to be public -- that's why I pulled the link off of the wiki. I'm mostly being radically transparent with Querki, but there are aspects of where the company is going in the long run that I need to keep close to my chest. Sorry about that, but as I've thought it through, I've realized that I can't give the entire game away until I'm prepared to go toe-to-toe with much bigger powers. In a very real sense, the business plan is Querki's most important piece of IP. I'm sure that others could build this system -- what I don't think anybody else has properly thought through yet is *what* to build, and why, and how it all hangs together...

  • 1
Huh. I hadn't run across "sharding" before. But a little googling suggests that I'm not alone in guessing where it comes from:

Yeah, I certainly came across the term first in the MMO world. (Indeed, the first patent I ever wrote was my "user-specific sharding" concept, back in Trenza. Someday I'll find an opportunity to actually try that out.)

But nowadays it's crucial for building a scalable site; otherwise, your database server becomes a single-point-of-failure bottleneck. And for a system like Querki, the speed of light even becomes relevant: the very-long-term plan is geographically-distributed sharding, so that Spaces tend to "live" geographically closer to their owners, to keep ping times quicker.

Most startups don't bother with sharding until they are already getting large and start having problems. But this isn't the first time I've built a system designed for a million simultaneous users, and it's the sort of thing that is *much* easier if you build it in from the beginning. So the system was designed for sharding from the very beginning, and my plan is to introduce shards long, long before we actually need them, so that we just don't hit those problems...

  • 1

Log in

No account? Create an account