Previous Entry Share Next Entry
jducoeur wrote in querki_project
[This is the first in an occasional series on Querki's Architecture, aimed at the serious engineers in the audience. I'm gradually turning my notes into more coherent pages in the wiki; as it evolves, you can find this topic here, with all the Markdown nicely rendered.]

Okay, let's tackle the great architectural heresy right off the bat. Querki is, at least to start with, highly *stateful*. That is, the user isn't interacting directly with the database most of the time; instead, she is working primarily with an in-memory representation of the data.


Part of my response is, "Why not?" While "statelessness" has some real benefits, it has been turned a bit too much into a religion in recent years, and is often a knee-jerk requirement without careful thought. But let's take the question seriously. There are several interlocked reasons why I'm doing things this way.

There is no such thing as "stateless"

"Stateless" apps usually aren't anything of the sort. Nearly every app has *oodles* of state, ranging from user sessions to history to the information you're trying to serve.

What the shorthand of "stateless" usually means is, "stateless front end". Every web server is an equal peer, able to handle any request with aplomb. There's something very attractive to that, but it's misleading to call that "stateless". What it really means is that you've kicked the state can down the road to the database.

It's lovely when the database is well-suited to serving all of your state...

... but that's really not the case for Querki. Our data isn't tabular, like a conventional RDBMS wants, but it's also not a bunch of relatively simple, discrete documents like most of the NoSQL databases want. Instead, our data is *highly hierarchical* and *mapped*, cross-linked in ways that are poorly suited to any of the major databases.

Mind, that doesn't mean we can't represent that data: storage is fairly easy. But that brings us to the real rub:


I want Querki to be lightning-fast; indeed, I think it's necessary for commercial success, since users aren't going to put up with pokey performance in the modern Web. Doing all of the (fairly complex) queries that I anticipate for Querki is likely to be just plain *slow* on any of the databases I know, because they just aren't designed for it. Nothing's really designed to handle complex joins across heterogeneous data, and that's essentially what we're doing. The data will be fairly small, but still, I expect very slow response times -- prohibitively slow for some of the interactive features we have designed.

Interacting with the Data

Finally, there's the simple fact that Querki is trying a lot of radically different things, many of which map poorly to traditional SQL. There's a reason why I'm writing my own programming language for this system. Doing that with in-memory structures is pretty easy; trying to do it on-disk would be a lot more challenging, and I don't have time for those challenges quite yet.

Write-Through Cache

So instead, we're going to take most of the load off of the database engine, and just use it for storage. Everything will be contained in the DB, so that we have fast, easy and reliable recovery. But all the real interactions will be with an in-memory cache. Changes will go through that cache, and then be written out to disk. Queries *always* come through the cache: when you begin to work in a Space, we sweep that Space into memory, and work with the result.

Distributed State

We need to scale horizontally, of course. Suffice it to say, there will be one "master" copy of each Space, on some node; usually, this will be the node managing the session of the user who is working with it. Most interactions will be with immutable snapshots of the Space, which can be copied around the network as needed. This will be discussed in considerably more detail later in the Architecture docs.

Nothing's hard and fast

Mind, all of this is Querki v1, and I fully expect it to evolve. This architecture has many advantages, especially early on, but it does have one big cost: it takes a *lot* of RAM to run. I expect RAM to be the main expense in running Querki -- we're neither I/O nor CPU bound, we're memory-bound.

So in the long run, we'll probably revisit this decision, and move more out to the database. I fully expect that doing that well, with good enough performance, will require writing our own database, or putting significant extensions into one that is capable of it. (Eg, add an extension to hang off of Postgres hstore.) When we have a big enough company with some money, that will be a fun project. For now, though, with a one-man show for the programming, we'll go with something that works and doesn't distract me from the important stuff.

  • 1
In general, the decision sounds reasonable. I do think it is kicking the can down the road to future issues -- potential issues with limits in available RAM, when and how to drop things out of the in-memory cache, load balancing/client hand-off/distributed processing, etc. Those can all be dealt with when you get them.

Using a master DB (at least an ACID compliant/transactional DB) has an advantage of easy collision detection for in-flight data and assignment of responsibility; putting the system into RAM can have some issues here -- detecting asynchronous updates, presentation of the collision in a timely manner, client responsibility so that multiple parties aren't prompted to fix the collision, and the like. This is especially true for cases where the back-end is distributed across multiple servers, but it isn't the only case. Do you know have a plan as to how you are going to do cache change propagation and object ownership across the system without the transactional guarantees?


Given that doing the JOINs across heterogeneous data is a non-trivial processing cost, which is likely to be the long pole regardless of the back-end, what is your plan to address that? Thick(ish) clients that do some of the processing themselves? Predictive processing? Delayed processing and partial loads?


I'm also curious as to how you see the database being structured; In part this is because I don't know NoSQL DBs that well, but given the highly hierarchical data design, vis-a-vis NoSQL's key-value pair optimization, it isn't that straight-forward.

All good questions. I'm going to be running around this weekend, so I can't address them quite yet, but the short answer to the important contention questions is that Querki is going to be Actor-based from its tip to its toes, and that will be used to manage most forms of contention. I believe that the design accounts for all of the issues you've raised. (Albeit in some cases, a bit crudely at first -- eg, edit conflicts will simply bounce the later edit and ask the user to re-enter.)

More on this later -- the section on Actors is the next part of the Architecture document, so I might write that Monday...

Given that doing the JOINs across heterogeneous data is a non-trivial processing cost

Well, when I say "JOIN", I'm speaking very imprecisely -- that's just the nearest SQL match to the core concepts.

What's really going on is that Querki is an object-oriented, prototype-styled database -- I don't use the terms "object" and "class", but the ideas aren't too far off. The weird thing about it is that Properties are cross-cutting -- any Thing can declare that it uses any Property. This setup allows the data to be hierarchically structured most of the time, but with arbitrary exceptions when you need them, making it very easy to represent messy, real-world data.

Queries don't use SQL or anything remotely like it. Instead, I'm designing a fairly simple, functional, data-pipeline language (QL), which focuses on each step being a transformation across this cross-linked map of data. That turns out to work very well for many applications, and I believe will allow folks to do most of the "programming" with a relatively easy step-by-step wizard.

It should all be very easy and performant to do in memory, where each value is cross-linked to the Thing and Property that it comes from. But doing that in any conventional DB I know would be tricky to manage -- hence, I'd rather do that processing in-RAM until we have the resources to tackle persistence properly...

I'm also curious as to how you see the database being structured; In part this is because I don't know NoSQL DBs that well, but given the highly hierarchical data design, vis-a-vis NoSQL's key-value pair optimization, it isn't that straight-forward.

Yep -- another reason to do it in-memory to begin with. NoSQL matches the *storage* requirements decently well; while Querki isn't quite as unstructured as the NoSQL key/value bags, they can certainly represent the data. What they can't do is query across it fast enough.

Hence my plan, which is to load each "Space" (database, essentially) into memory when you start working with it, and do the processing directly in RAM. That's practical solely because any given Space is expected to be small. (And is capped at not-large: the current plan is that any given Space has a maximum of around 50k Things for the time being. That means that some kinds of apps aren't currently practical, but there are still tons of reasonable applications...)

  • 1

Log in

No account? Create an account