It's been quite a long time since I've posted anything here, but that doesn't mean the project's stopped moving -- quite the opposite, really. The complication has been that we've been running on two separate tracks for the past several months, and they are *finally* starting to come together.


The big news is that Querki is finally beginning to transition to Cassandra. From a technical perspective, this is by *far* the biggest change to Querki's architecture, one that we've been talking about for about three years and working on for the past six months. More technical details below the cut.
Collapse )
And now, on to the user-visible stuff...


Probably the most important enhancement from a user perspective is the addition of complexity modes.

The goal of Querki is to make it much, much easier to build and run your own little Spaces and Apps. But there's still a lot of complexity to it, especially to support the various power-Querki scenarios. We're always dealing with that tension between trying to keep the UI as simple as possible for the end user who just wants to participate, versus the experienced engineer who wants to build cool sites twenty times more quickly and easily than they can with traditional tools.

To deal with this, there is a new concept of which "mode" you want to run Querki in. There are three modes, initially defined like this (quoting the UI):
  • Participant: Appropriate for most people, who want to be able to participate in other peoples' Spaces and create Spaces based on Apps, but don't want to design their own Spaces from scratch. In Participant Mode, Querki keeps the complexity to a minimum while still giving you the tools you need to add and edit data, and participate in conversations.

  • Builder: For those who want to build Spaces that don't yet exist as Apps, tweak existing Apps to better suit their needs, or manage their Spaces in more detail. Builder Mode adds the Model Designer, so that you can define exactly the sort of data you need, as well as more powerful security tools.

  • Programmer: For the Querki power user. Programmer Mode adds all the bells and whistles, so that you can customize Spaces in more detail, write your own code in QL, design custom Types, and lots more.
I still don't love the name "Participant", but it's the best suggestion I've heard so far. Other suggestions welcomed. Also, note that the details of what is visible in which modes are still evolving, and this is just a first cut: observations and suggestions welcomed there as well.

Anyway, as of now everyone starts in Builder mode. Eventually, you'll choose when you're starting out, but we're not there yet. You can choose which mode you prefer from the login menu in the upper-right, and you can change it at any time -- it just takes a second. The differences are intentionally subtle; basically, it's just a matter of how many buttons and menu picks are available.

A very minor change, but one some folks might notice: object IDs are no longer simply sequential. You know those IDs that each object has, like ".3y28csa"? They used to grow simply monotonically, proceeding one at a time. (In base 36, but whatever.) This bottlenecked on the MySQL server, so the very first feature of the new Cassandra world involves each Querki application server having its own pool of them, and using them as needed. So they will mostly be larger than they used to be (the .3y2 series is now only being used for system-level objects like Users and Spaces), and they will jump around a lot. This is expected.

The Space and Page-loading experience has been slightly improved. Nothing dramatic, but we now provide some basic spinners. Most importantly, when loading a Space, you no longer spend a long time (it could be several seconds on mobile connections) staring at a blank white screen; you now quickly get *some* feedback that things are loading.

Notable Bugfixes

Photo Upload should now be more reliable: suffice it to say, the original implementation of Photo Upload wasn't terribly robust, and when we switched from a single machine to a cluster some months back it started becoming very erratic. This pipeline has now been heavily rewritten, and should now work reliably. Please tell me if you encounter problems with upload.

You can now dereference Properties through Tags: previously in QL, if you got to a Thing through a Tag, you couldn't immediately just refer to a Property on that Thing. Now that it's become clear that Tags should work just like Thing (Link) Properties, that seems silly -- I hit this bug myself and found it head-scratching that I couldn't do this. So now you can.

A slash at the end of a Space's URL is no longer required: several people got confused when they tried to go to a Querki Space and got an orange error page instead -- the problem was that they had forgotten the slash at the end of the Space's URL. This now works as expected.

Tabbing no longer gets messed up when you add a List element: previously, adding a new element resulted in all sorts of havoc in tabbing through the page. It now makes more sense. Note one other change that went in at the same time: the delete button is now intentionally omitted from the tab order. I had found it too easy to accidentally delete a list element by tabbing to the wrong place and hitting Enter. This is no longer possible. (This may require further examination from an accessibility POV -- we'll look into it in that context.)

You can now sort on Properties of Model Types: the _sort() function has always allowed you to sort a List of Things, but it hadn't allowed you to sort a list of complex values. This wasn't so much a bug as a feature that had never been implemented. That function has now been refactored to work properly here -- you should be able to sort a list of Model Values as expected.

Sorting of text is now properly case-insensitive: while fixing the previous, I discovered that in many cases Text fields were being sorted case-sensitive. As a rule, Querki always sorts case-insensitively, so I've fixed that.

That's it for now, but expect major releases to start coming a lot more frequently, now that we have our Cassandra cluster up and running for real, so I can start making some enhancements I've been planning for years now...

Release 2.0.0 -- OPEN BETA

As of a couple of minutes ago, release 2.0.0 of Querki is now live. This isn't much different from a features POV (nearly all the new fixes and enhancements relate to system monitoring), but hugely different from a business one, because Beta is now officially declared Open.

The old invitation pathway is still available -- I expect it's still the way people will usually join Spaces, and therefore how many will join Querki. But the old concept of a "pending invitee" -- the waiting list before you are allowed to create your own Spaces -- is now abolished. You can now go to the Querki homepage, click on the Sign Up button there, and fill in your info. The system will send you a validation email; click on the link in that, and you'll be a full member, able to create your own Spaces.

As always, comments and questions are highly encouraged. And I think we've got a fair ways to go before I can say with a straight face that the system is really easy to use. But we're now at the "don't be shy" stage -- if you know folks who you think would find Querki useful, please point them at it...

The reason for Querki, summarized

I was just reminded that, while I posted this to my personal journal, I didn't post it here. This is the most concise explanation for *why* I'm building Querki:

Small communities typically don't have a lot of data in the grand scheme of things -- but the data they do have is often quite complex, and they care about it passionately. That's where Querki comes in.

Or to put it more simply: any time you're tempted to use a spreadsheet for non-numeric data, you probably should be using Querki instead. As my friend Matt recently put it, "Querki manages the hidden complexity of the everyday stuff".

Release 1.4: Security Revamp

Major release today, entirely focused on one subject: Querki's security system has gotten a major rewrite.

tl;dr: There is now a "Security for" item in the Actions menu, which provides a halfway-decent UI for managing the Security for this Space, Model or Thing. And the notion of security "inheritance" has been massively rewritten, to make more sense.

On to the gory details...

First, a reminder of how security works in Querki. There are actually two security models, which are closely related but intended for different situations. The "coarse-grained" system is intended for most use cases, and is built around the notion of "Roles". The system has several pre-canned Roles, which carry particular permissions across the Space. Each one has all the permissions of the ones before it, and adds a bit:
  • Basic Member: can read stuff in this Space, and that's about it.

  • Commentator: can comment on Things.

  • Contributor: can create and edit Instances.

  • Editor: can design Models, and (once the notion of Moderation is in place) will be able to moderate Conversations and approve outside contributions.

  • Manager: not yet implemented, but will be able to do *nearly* everything the Owner of the Space can do.
And while it's not listed officially, the Owner of the Space can do anything to it -- it's not possible to lock out the Owner. (There is also a nascent concept of Custom Roles, but we're barely started on that.)

This coarse-grained approach is by far the easier way to go, and hasn't changed in this revamp. It's controlled from the Sharing page for the Space, which lets you say what Role to give someone when you are inviting them, and to change their Role later. I recommend that most people, most of the time, just use this Role-based system: it's easy to understand and suffices most of the time.

Today's changes concern the lower-level and older "fine-grained" system. This is actually a richer security model than most application frameworks provide out of the box: you can control exactly who has which permissions across the Space, for a particular Model, or for individual Things. This lets power users tweak and tune the edge cases to their liking: making *this* page Public in an otherwise Private Space, or locking down a few Things to only be accessible to the Editors. (Note that you can consider it an "add-on" to the Role-based approach -- even when doing fine-grained work, I recommend viewing that as tweaks to the Roles.)

The fine-grained system is very powerful, but it's also traditionally *very* hard to use -- you needed to add particular Advanced Properties by hand in the Advanced Editor, and fill them in with the correct values. Even I found it to be a pain in the tuchus.

Worse, I made some design mistakes early on in how "inheritance" worked in this system. Specifically, I was wrestling with the problem that you needed to be able to restrict who could design a given Model, while allowing more people to create and edit its Instances. Similarly, I needed a way to declare Space-wide who could by default edit the Instances of this Space, separately from being able to edit the Space itself. So I added what proved in retrospect to be a horrible hack: the Who Can Edit Children Permission. That almost made sense, until I began to realize that all the same issues pertained to all the other permissions as well.

So I've thrown out Who Can Edit Children entirely -- it's now deprecated, and likely to be deleted once all Spaces have evolved to the new approach. We now have a concept of the "Instance Permissions object" on each Space and Model. The one on the Space contains the Permissions that are, by default, applied to all Things in the Space; the one on a Model contains the default Permissions for that Model's Instances.

Just as importantly, though, that detail is now hidden from you, replaced by a specialized Security Page, which you can get to from the Actions menu. This lists the Permissions that are relevant to the Thing you're looking at, with straightforward radio buttons to let you choose what you want. If you're looking at the Space, it has a secondary tab marked Instances, which lets you set the default Space-wide Permissions for all Things. If you're looking at a Model, it also has an Instances tab, which lets you set the default Permissions for all Instances of this Model.

There are five options that may be available for any given Permission:
  • Public: if checked, everyone (even Anonymous viewers from the Web) can do this. Public is only available for "read-like" Permissions for the time being.

  • Members: if checked, all Members of the Space can do this.

  • Owner: if checked, then only the Owner can do this.

  • Custom: if checked, this shows a Tag Set input box, that lets you specify exactly the Persons and Roles that can do this. This is *really* getting down into the weeds, but is how you get access to the full power of the system when you need it.

  • Inherit: when looking at a Model's or Instance's Security page, this is the default, and simply means that it should inherit the value from "above" -- the Model inherits from the Space's Instances tab, and the Instance inherits from the Model, or if that's marked Inherit as well, it picks it up from the Space's Instances tab.
I'm not going to claim that this approach is perfect: there's still a lot going on here. But I believe it's an enormous improvement: it gives you quick and easy access to exactly the options that are legal for the Thing you're looking at, and the common options are simply a click. And the change to the actual security model means that Spaces and Models now have their own Permissions, entirely separate from those of the Instances that inherit from them.

Existing Spaces should be automatically "evolved" to use this approach when you next load them: the Instance Permissions objects will be automatically created, and the Permissions currently on the Spaces and Models will be copied as appropriate. (Note that existing Who Can Edit Children values will become Who Can Edit on the new Instance Permissions object.) I've tested this pretty heavily, but *please* check it out and tell me ASAP if you find problems. This is a big enough rewrite that bugs wouldn't be surprising, and security is one of the areas where bugs are generally not tolerated.

One significant change, although I don't believe it affects anyone but me yet: previously, if you marked the Space as not publicly readable, that meant that it was *hidden* from non-Members -- if a non-Member tried to access that Space at all, the system would deny that it even existed. I've decided that that's a poor way to implement Hidden, especially because it prevented you from opening just one or two pages in a Space where the root wasn't Publicly readable. So that's been removed; we'll eventually add a first-class "Hidden" flag for this purpose.

Related to that: the Who Can Read Permission is a bit "soft" when it comes to the Space itself. For Models and Instances, Who Can Read is quite absolute: if you don't have Who Can Read on a Thing, it doesn't exist in your view of the world. Literally -- it gets snipped out of your view of the Space, which means that we're pretty confident about that. We can't do that for the Space, though -- it would be hugely destabilizing -- so instead it currently just hides the Default View with an error message. This undoubtedly has edge cases that need fixing, and in general this will need improvement.

And related to that: the Security UI currently allows you to mark a Model as non-Readable, while its Instances are Readable. Don't do that. Like I said, if something's non-Readable, it doesn't exist at all -- so if you hide the Model, odds are good that you're going to get unexpected behavior. Again, this is an edge case that probably needs more thought and improvement.

The sharp-eyed may notice that there is a new Who Can Design Permission, separate from Who Can Edit. This was driven by the realization that designing Models is fundamentally different from Creating and Editing Instances, and will often have different permissions. Think of Who Can Design as being the combination of Create and Edit, specifically for Models.

Also, note that the Sharing Page has been broken into several tabs -- that page was getting pretty long anyway, and I needed to learn how to use Bootstrap Tabs.

Finally, here's the Rosetta Stone to understanding permissions, if you really want to grok the system exactly. When we are looking up whether you are allowed to do perform an action on a Thing, we ask, in order:
  • Does this Thing explicitly set this permission? If so, follow what that says. If it says "Inherit"...

  • Does its Model's Instance Permissions have it set? If so, do that. If that says "Inherit"...

  • Do you have a Role that gives you this Permission? Is so, allow it. Otherwise...

  • Do what the Space's Instance Permissions say.
It's not a trivial system, but I believe it fits the Querki approach to the world well, and now that the new UI is in place, it should be at least adequately usable...

Something to keep an eye out for

A general request for folks to keep a watchful eye open: I spent the morning in a worrying firedrill, because the Issues Space had gotten into a Weird State. The behavior strongly suggests that we somehow wound up with two copies of that Space alive at the same time, which *should* be impossible. It's a matter of real concern -- while Querki is pretty resistant to data corruption (and so far, the problem doesn't seem to cause data loss, just immense confusion), the notion that there can only be one copy of a Space alive at a time is one of the system's key invariants, and we have to be absolutely intolerant of violations of that.

The problem is, I have absolutely no idea how to recreate this situation, since I have no idea how it happened, or even exactly what "it" is; my attempts to repro it have simply failed. So I'm asking you to keep an eye out for it.

The behavior will probably show up as a Space having short-term "memory" problems. Most typical is that you create a Thing, and then find that it's not there. More precisely, if you reload several times, you find it *sometimes* there and *sometimes* not, inconsistently. (Which is why I believe the problem has to do with duplicate copies of the Space, where some requests are routing to the copy that created the Thing, and some to the other.) This may manifest in the middle of creating a Thing -- you'll see an error saying something like "_edit didn't receive a Thing". (Which is because the Client made a request to create the Thing, succeeded, but then the request to *edit* that Thing went to the other copy.) It also results in a *lot* of inconsistent errors when trying to edit this Thing (since sometimes it is trying to save the change to a Space that doesn't know this Thing exists).

I've put in some logging spewage that *may* help understand the problem. So if you see this happening, please tell me ASAP, so I can go look into the logs and see if they say anything helpful. I want to get this problem diagnosed and fixed as soon as possible -- I consider data integrity to be the very highest priority for Querki, and any problem that threatens that even slightly is automatically critical.

Sorry about the inconvenience. We tested as much as we could before switching to the new cluster, but it was inevitable that some bugs would slip through the cracks.

Speaking of which, a bug that you're less likely to hit but should be aware of: displaying a *very* large page is currently likely to fail. This one is completely understood -- the rendered page is simply too large to send around the network, it turns out. The problem will only happen if the rendered QText for the page is over 128k, and the only pages I know that evince this problem are the ones in the Issues Space that try to display *all* of the Issues. (So don't try those pages right at the moment.) I'm working on the problem, but it may be a couple of weeks before I can push out a good fix for it -- doing this right is going to require some protocol changes, which will be a bit of a headache, but it's worth taking the time to solve this properly and permanently.

Release 1.3.15

Today's release is terribly important for a reason that is mostly not user-visible: Querki is now running on Amazon Web Services.

Since the original prototype release at the beginning of 2013, Querki has been running on a single server, running in a local colo. That's worked decently well for us -- indeed, it's been comforting to see how well the system works in a pretty antiquated server -- but it's never made any sense for a real production environment. In particular, it's been the reason why I've been keeping the beta locked down: while we were using only a fraction of that server's resources, I haven't been sure how many users we were able to support, and we had no good way to scale beyond whatever that limit was. So if we grew too fast, we would be screwed.

Now, we're running on a three-node virtual cluster. That's essentially invisible to you, but crucial for scalability. Since these are virtual machines (and fairly small ones to start), it's relatively quick and easy for us to "grow" them as needed, and to add more machines to the cluster. Querki was designed for this from the beginning, so in principle we should be able to easily grow the system as the usage grows.

We've done a fair amount of testing in this new configuration, but there's always the possibility that we've missed something. So if you find anything that's now broken, please bring it to my attention ASAP: the goal here is that Querki's new home shouldn't matter to you, except in that the system should be more reliable, faster and more scalable going forward.

On to the more user-visible stuff, which is mostly UX-centric:

Enhancements and Features

_QLButtons and _QLLinks now have a "thinking" icon: I've gradually been evolving a best practice, that big and complex pages (such as the main Period Games Rules page, which can take a while to display) should show their sections behind _QLLink or _QLButton. But even those sections can sometimes take a couple of seconds, and a couple of seconds is a very long time. So these buttons now display a spinner icon while they are "thinking", to provide some basic feedback.

All external links now open in a new tab/window: this basically brings Querki in line with the way most modern systems work. "External" means all links outside of this Space, including links to other Spaces. Note that this applies to both QText-style links and explicit <a> tags. I believe the new behavior should be as expected, but please give a yell if you find weirdness. (Also, we're now more consistent that all external links are marked "nofollow", to prevent link-spam and keep Google happy with us.)

_refs can now be used without a Property: historically, you can use _tagRefs either restricted to a specific Property or not; if not, it shows *all* tag references to this. However, _refs required a specific Property. It has now been brought in line with _tagRefs, at least to a limited degree: if used without a Property, it will produce all "hard" (Thing-type) references to this Thing through Properties that are Restricted To this Thing's Model. More importantly...

Added the _allRefs function: I've come to the conclusion that the distinction between _refs and _tagRefs is mostly artificial from your point of view -- usually, you want both. So while I'm leaving both of them in place, I've added a new _allRefs function, which combines exactly the results of both of those. I recommend using _allRefs in place of _refs and _tagRefs unless you specifically want to limit your results.

The default Default View now includes a "Things That Use" button: one of the goals of the system is that the most-needed functions should be at your fingertips, without needing to write QL. I've found that the single most common need for QL is to find out what points to the current Thing. So that's now built in -- if you haven't created a Default View, you'll find a new button in place that, when pressed, displays the _allRefs for this Thing. This makes it easier to navigate your Space without writing any QL.

If a Comment and everything below it is deleted, simply don't show it: we currently lean towards showing "Comment Deleted", for the same reason many other systems do so -- if you completely delete a comment in the middle of a conversation, it can deeply mess up context, and even produce socially-disastrous misleading results. That makes sense, but it meant that, if you post a comment in error and then delete it, you get an annoying tombstone that you can't get rid of. So we now examine the situation: if there is nothing *below* the deleted comment, we don't even display it any more.


QL can now cope with empty List Literals: I found that, if you tried to display a completely-empty Location, with none of its fields filled in, you would get an Unexpected Error. The underlying cause turned out to be that, if you tried to show a List Literal and all of its components came out empty, it would crash the QL processor. This now simply produces empty, as expected.

Fixed the menu bar at ~900px: alexx_kay pointed out that, on his tablet, the menu bar at the top was displaying as two lines, and obscuring the stuff below that. This turned out to be a Bootstrap styling issue: if the screen width was between 750 and 992px, it would have this misbehavior. I've hacked around this by disabling Bootstrap's fixed width for that bar. It's not a perfect solution (you can still get the two-line problem if the bar gets completely filled up, and I don't love the current styling), but it's an improvement for now.

Related to that: the menu in the upper-right no longer says "Logged in as", simply to save pixel width. It just shows your login name now; that brings us in line with many online systems, so I think it should be okay.

Release 1.3.14

A smaller and more routine release this time, but with a few nice details...

Enhancements and Features

The most significant addition is the new Location Type. This is another high-level semantic type available to use as you see fit. (I needed it for the Carolingian Site Book, but it's been in the plans for years.) For now, this simply is a structure that lets you put in street addresses, but with the added benefit that, if you fill in the address completely, the display automatically includes a link to the Google Map for that Location. (This comes from the new _mapLink function.)

Eventually, Locations will be enhanced with the ability to grab "here" with one click from your phone, much like you can take a Photo when you're using Querki from a phone. And we'll likely add a mechanism to embed the Google Map for a Location directly in-page, instead of it being an out-of-page link. But for now, this is a nice useful start.


Spaces now use the Advanced Editor: I've found, over the past year or so, that I almost never want the Instance Editor when I'm editing the Space itself. So that's been removed -- if you click the Edit button from the Space's Root Page, you'll go to the Advanced Editor.

_QLButtons pass the lexical context properly: there was a brief regression where the _createHere function stopped working when inside of a _QLButton. This was because, when I enhanced _QLButton to accept *any* Type (not just Things), it stopped automatically using the passed Thing as the lexical context. This has now been enhanced and rewritten so that _QLButton and its ilk capture the *actual* lexical context that they are defined in, and pass that through to the call when you press the button. (This not only fixes the bug, it's more correct than what we were previously doing.)

The new Login dialog now works better: when the dialog appears, the keyboard focus moves to the Email/Handle field. And if you press Enter from the Password field, it does the login -- you no longer need to press the Login button itself. By and large, this makes the dialog feel more natural and correct.

Local functions in QL

As mentioned yesterday, the current release of Querki includes a concept of "local functions". That's worth a bit of discussion unto itself. (This is only interesting for experienced programmers.)

QL is a pure functional programming language, stripped down to the bare bones -- to date, it's been arguably simpler than Scheme. That's intentional: simplicity is the name of the game here, because I want non-programmers to be able to do useful things in it, with the barest minimum of syntax.

But it's also intended to provide enough capability so that power users (that is, professional programmers) can quickly create fairly complex custom displays that fit their needs. And I've been starting to chafe at factoring problems when I try to do that.

When I try to build something really interesting (such as the new Documentation by Category page), I would wind up with duplicate "code". Or at least, duplicate text expressions, which are basically code. One of my mottos of programming is, "The Root of all Evil is Duplicate Data" -- duplication tends to make things much more difficult to maintain, and leads to errors. Indeed, that was one of the origins of Querki -- trying to do things in traditional wikis wound up requiring a lot of duplication, and that was causing me problems.

That's the motivation behind Local Functions. We've had a formal concept of Functions for a long time, but they're pretty heavyweight: you need to create a Function Property, and fill that in with the desired code. (Or alternatively, create a Thing and set its _apply Property to the desired code.) Either way, that winds up creating a *global* function -- useful for many cases, but it feels like an awful lot of work when you just want to be able to do the same thing several times in one expression.

So I've added local functions, which are a lot like functions in a typical pure-functional programming language. Let's look in more detail at the example I mentioned yesterday (whose full text can be found in the unit tests):
[[Album._instances -> _groupBy(Artists) -> +$byArtist
     _def $From First Album($artist, $field) = $byArtist -> _groupGet($artist) -> _first -> $field
     $From First Album(Blackmores Night, Link Name)
     $From First Album(Eurythmics, Artists -> Genres)
This is based on top of a small CD Space that is defined in the Querki test suite -- several Artists and Albums, and Genres that they fit into. Let's deconstruct this code, and understand what's going on.

The first line groups all of the Albums by Artist, and assigns that grouping to $byArtist:
Album._instances -> _groupBy(Artists) -> +$byArtist
Remember that +$byArtist is the new syntax to assign the received value to $byArtist.

The second line is the interesting bit, a function definition:
_def $From First Album($artist, $field) = $byArtist -> _groupGet($artist) -> _first -> $field
_def is something Querki hasn't had before, an actual keyword. I thought about other syntaxes, but decided that they were too clever and idiosyncratic to live -- let's just make a function definition look like a function definition.

The _def keyword must be followed by the name of the function; as with all local names, this must start with $. Note that, like all Querki names, it may have spaces in it. I'm still of mixed minds whether I like that style for function names: you're certainly welcome to use camel-case or underscores or dashes if you prefer, but Querki allows spaces as a general rule, and there was no reason to change that here.

The function name may (optionally) be followed by a parameter list. If you omit the parameter list, the function will still have access to the calling context, as well as to any bound names that are in scope. The parameters, again, must start with $. There is no concept of optional parameters yet, nor of calling parameters by name, but both might come in time.

Then comes = , to signify the function definition. This syntax may look odd to Java programmers, but it's common among functional languages, and gets the point across. Indeed, it's kind of literally true, in ways that we'll discuss below. Note that the equals sign must be surrounded by spaces.

Finally, we get the meat of the function. This is specifically a phrase -- a one-liner -- because that's what you most often want. I believe you can put in a full multi-line expression by surrounding it with parentheses, but that hasn't been tested yet. This phrase starts out by receiving the context that was passed into the function call, and may use any names that are in scope, as well as the parameters.

So in this case, we're operating entirely on the parameters, which should be an Artist, and a "field" -- really, in this case, an expression that we're going to apply to the first album for that artist. We go back to the $byArtist groupings that we computed in the first line; get the Albums for the specified $artist using _groupGet(); grab the first Album in the list; and apply $field to that.

The last two lines are function calls:
     $From First Album(Blackmores Night, Link Name)
     $From First Album(Eurythmics, Artists -> Genres)
The first one produces the name of the first album by Blackmore's Night; the second takes the first album by the Eurythmics, looks up its Artists (which is, of course, the Eurythmics), and produces their Genres. In practice, this winds up printing "Fires at Midnight\n[Rock](Rock)".

Hopefully that's mostly pretty intuitive, but I'd like to call your attention to one important nuance. Keep in mind that QL is, essentially, a macro language; that shows up in the way parameters are handled.

Think about how parameters work in a typical programming language. When you call a function with some parameters, those parameters are evaluated *first*, in the *calling* context; the results are then passed in to the function. You're probably so used to that that you don't even think about it most of the time.

That is *not* how parameters work in QL, intentionally. Instead, the expression itself is passed as the parameter. On the last line of the above example, we don't start by computing Artists -> Genres locally and pass the result into $From First Album -- instead, we pass the expression Artists -> Genres as the $field parameter, and we evaluate that expression when the parameter gets used.

The result is that you can kind of think of QL local functions as if they were macros. (They aren't actually, but the effect is similar.) That last line can be thought of as literally saying:
$byArtist -> _groupGet(Eurythmics) -> _first -> Artists -> Genres
This approach has its advantages and disadvantages. I find that it does occasionally bite me (and it's going to add special challenges to the long-term plan to make QL strongly-typed), but most of the time it's deliciously powerful. It allows you to easily do things that many languages find quite challenging -- even my much-loved Scala would have to resort to nuisance-to-define macros in order to do some of this stuff. And I'm finding that, in principle, I *usually* need that power to do what I need in Querki: the most common use for parameters is passing a Property or Expression to be evaluated in the right place.

One final note: as mentioned above, functions don't require parameters, and receive the context at the point of call. Here's another unit test, that illustrates this:
[[Album._instances -> _groupBy(Artists) -> +$byArtist
   _def $Show Artist = _first -> Link Name
   $byArtist -> _groupGet(Blackmores Night) -> $Show Artist
This variant calls the simpler $Show Artist function, which just receives the list of Albums passed into it. Note that the body of the function can be a text literal; this is how you can use the same text expression in several places.

I hope you find this useful. Local functions are really only aimed at the 1% of Querki users who are trying to do really serious stuff in it, but if you fall into that category, I believe they'll make your life easier...


Lots of major, visible changes this time around, at various levels. Some of this is pretty important to advanced Querki users. Most of it is about "usability" at one level or another; that's going to be an ongoing theme for the time being.

Enhancements and Features

First, and in some ways most usefully, there is new Documentation by Category page. I've been coming to the conclusion that *I* can't find things in the docs any more -- there's just too much there -- which means they're getting to be useless for everybody else. So I've created a new page that shows the high-level functional categories, and lets you dive in and see what's in each of those. This lets you, say, just look at the functions for Photos, and ignore the Logic functions.

Internally, we now have a categorization scheme for all of the Functions, Properties, Things and Types, and I've done a first pass at that. This is not the last word there: I strongly encourage comments along the lines of, "I expected to find _foo in the category Bars and Bazzes". At the moment each item is in only one category, but the system should allow items to be listed in multiple categories (it's a Tag Set under the hood, of course), so it should be decently flexible.

Also, you'll find that that page is far more interactive than the old docs -- everything happens directly in that page. I found that I've been getting annoyed by all the page flips and Back buttons required in looking things up in the old reference pages, so the new one does everything in-page, using _QLLink very deeply. Indeed, the hardcore Querki hackers may want to take a look at the source for that page, which involves what may be the most complex and sophisticated QL expression written to date. You'll have a lot of WTF moments looking at it, all of which should be explained below: many of this week's changes were driven by my desire to make this page work correctly, without compromises.

BTW, this page is a fine illustration of how to use _QLButton/_QLLink to make a very complex page highly responsive. Overall, this contains all the documentation -- it's a huge amount of data. But the page only initially shows an outline, which you can dive down into, so each level is nicely quick. I strongly recommend this pattern, and will be thinking about ways to make it easier to code.

Querki now has local function definitions: this is only interesting to power users, but it is *huge* for power users, one of the most important enhancements this year. Enough so that I'm going to give it its own post tomorrow, describing it in detail and going through the implications. But to whet your whistle, this means you can now say things like (from one of the unit tests):
[[Album._instances -> _groupBy(Artists) -> +$byArtist
  _def $From First Album($artist, $field) = $byArtist -> _groupGet($artist) -> _first -> $field
  $From First Album(Blackmores Night, Link Name)
  $From First Album(Eurythmics, Artists -> Genres)
Between this and the local value bindings introduced last time, it's finally possible to build *good*, properly-factored, relatively maintainable, complex expressions in QL.

The _QL[Button | Link | Input] functions can now receive most Types, not just Things: mindways hit this recently, and I've encountered it once or twice myself -- _QLButton is turning out to be one of the most powerful functions in Querki for producing really nice UI, but it has so far required you to pass a Thing into it, even when that made no sense. This turned out to be a blocker for the Documentation by Category page, which needed to be able to pass Tags into _QLLink.

So I've finally bitten the bullet, and made this work more correctly. Mind, it still can't handle absolutely everything -- in particular, if you pass it a List, it'll only use the first value. And it will only work for the "real" Types, which have a defined serialization scheme. But in principle it should work with all of those, which should cover most use cases.

Added the _uniqueHtmlId function: this is mainly useful for _QLButton patterns, where you often want to create a pair of a _QLButton and a div to show the results in, linked by the div's ID. If the button is based on a Thing, you could use its OID to create the div's ID, but otherwise it was a real problem. So this function deals very specifically with that: it simply generates a random ID that you can use for this purpose. (Obviously, you can use it for other purposes as well, but this is the only use case I currently know of.)

Four-space indent in QText is no longer interpreted as "code": one of the problems that has been plaguing the serious users has been that, if you indent your QText four spaces or more, it automatically goes into "code" mode, showing everything as fixed-width and ignoring the rest of the QText. This holdover from Markdown has been making it hard to build interesting pages. So I've tracked down the code that makes that happen, and disabled it. So this misfeature is now officially removed from QText -- if you need the "code" look and feel, surround it with triple-backticks, like this:
This is my code

Introduced the _groupGet() function: _groupBy() has been turning out to be one of Querki's more important and useful functions, and I use it a lot. But it has had a weakness: the only way to use the groups found is to iterate over them, in whatever order they happen to be in. That's often fine, but sometimes (as in the Documentation by Category page), you know what the expected keys are, and you want to pick the values out based on the keys.

Hence, _groupGet(). This receives the results from _groupBy, and takes one parameter, a key. It produces all of the elements that have that key. Thus, you can use _groupBy to split some Things up, and then use _groupGet to display them in the order you choose.

Login is now in-place: one of the more common annoyances, I've found, is that I go to a page, only to realize that I'm not logged in, but when I choose "Log in", it takes me to a separate screen so I lose my place. This is now reworked to catch up with the 2010s: login now happens in-page, so your workflow is preserved. (There is a known bug, that you have to press the "Log in" button instead of just hitting Enter to log in; I'll deal with that soon.)

_QLButton and _QLLink now have open/close affordances: thanks to metahacker for recently reminding me about the importance of affordances. This reminded me about the fact that *I* may be comfortable with the behavior of _QLButton and _QLLink -- in particular, the fact that clicking them once opens something up, and clicking again closes that -- but it's entirely opaque to the naive user. So I've added simple up/down arrows to make this a bit more obvious. It's not especially *pretty*, but it should make things clearer, and serves as a first pass fix.

This icon won't be shown if you are using the "append" or "replace" flags, or with _QLInput. You can manually suppress it in _QLButton/_QLLink by using the new "noIcon" parameter.

We are now using Bootstrap Modals instead of jQuery UI Dialogs: or in human-speak, the dialog boxes now look different. I have mixed feelings about whether the Bootstrap look-and-feel is better or worse than the jQuery UI one, but it *works* better, and lets me fix some long-standing bugs in the dialog boxes.

Note: I've encountered some edge cases with the new dialogs, where after using a dialog, you turn out to have lost your scrollbars. At this point, I understand the bug, and I *think* I've fixed it everywhere, but if you encounter this problem, please tell me ASAP.

The default Default View of Models and Things is now slightly clearer: not a dramatic change, but it now omits a few internal fields that you shouldn't ever care about, which were adding bulk with no benefit. And when displaying a Model, it separates the Instance and Model Properties, the same way as you see in the Model Designer.

Added the _usingSpace() function: this is a pretty obscure function -- I'm not sure anybody outside of system will care -- but it's available if you need it. This takes a Space as its parameter: either the current Space or one of its Apps. It alters the context so that that Space will be used for the rest of this QL phrase. Basically, it allows you to ignore everything in the "child" Space, and only use the contents of the App.


You now get a proper error if you try to go to a Space you can't read: there was a subtle bug, such that if you tried to navigate to Space thingy, with URL https://www.querki.net/u/justin/thingy/#!thingy, and you weren't allowed to read that Space, you would get redirected to https://www.querki.net/#!thingy, and the client would hang at "Loading...". This one was actually a pain, relating to the way the stuff after the "#" is handled in the browser, but it should now properly redirect you to Index with an error.

_tagRefs now finds all references, not just the ones in ordinary Things: not something that the average user will hit, but as part of building the new Documentation by Category, I added Tags to all the Functions -- and discovered that _tagRefs wasn't finding them! So this now works as expected -- _tagRefs finds *everything* with the specified Tags, not just Things.

TextTypes can now be considered as ParsedTextType: that's generally meaningless to you, but the upshot is that many function parameters now work better than they used to. In particular, parameters that have previous only worked with text literals (such as the label parameter for _QLButton) now accept Properties as well.