• kdb+/q Origins
• q Crash Course
  • The q Interpreter
  • Comments
  • Atoms
  • Arithmetic Operators
  • Variables
  • Lists
  • List Operations
  • Functions
  • Example: Fibonacci Sequence
  • Example: Poker Hand Generator
  • Dictionaries
  • Tables
  • Example: Calculating P&L
• Less Code is Better Code

This summer, while working in New York, I was forced to learn a rather unusual programming language called q.

Q is a small but shockingly powerful programming language, with a small yet unusually expressive syntax. The q programming style is like none other, taking code compactness to an extreme—q programmers pride themselves in writing short code, where extraneous whitespace is shunned, variable names are short, and operators are heavily overloaded, with many symbols representing 4-5 functions depending on context.

I found the process of learning q to be equal parts painful and fascinating. I hope that from this article, you too can get a flavor of the experience I endured this summer.

kdb+/q Origins

Q is a functional programming language originally inspired by APL, an array-based language infamous for its heavy use of nonstandard characters. The q language was originally developed by Arthur Whitney, a computer scientist who, among many things, is known for his extremely compact programming style.

To illustrate this point, the following C code snippet is a fully functional interpreter for another programming language called J. As the legend goes, this code was written by Arthur Whitney in a single afternoon, together with a friend whom he was visiting at a farm.

typedef char C;typedef long I;
typedef struct a{I t,r,d[3],p[2];}*A;
#define P printf
#define R return
#define V1(f) A f(w)A w;
#define V2(f) A f(a,w)A a,w;
#define DO(n,x) {I i=0,_n=(n);for(;i<_n;++i){x;}}
I *ma(n){R(I*)malloc(n*4);}mv(d,s,n)I *d,*s;{DO(n,d[i]=s[i]);}
tr(r,d)I *d;{I z=1;DO(r,z=z*d[i]);R z;}
A ga(t,r,d)I *d;{A z=(A)ma(5+tr(r,d));z->t=t,z->r=r,mv(z->d,d,r);
 R z;}
V1(iota){I n=*w->p;A z=ga(0,1,&n);DO(n,z->p[i]=i);R z;}
V2(plus){I r=w->r,*d=w->d,n=tr(r,d);A z=ga(0,r,d);
 DO(n,z->p[i]=a->p[i]+w->p[i]);R z;}
V2(from){I r=w->r-1,*d=w->d+1,n=tr(r,d);
 A z=ga(w->t,r,d);mv(z->p,w->p+(n**a->p),n);R z;}
V1(box){A z=ga(1,0,0);*z->p=(I)w;R z;}
V2(cat){I an=tr(a->r,a->d),wn=tr(w->r,w->d),n=an+wn;
 A z=ga(w->t,1,&n);mv(z->p,a->p,an);mv(z->p+an,w->p,wn);R z;}
V2(find){}
V2(rsh){I r=a->r?*a->d:1,n=tr(r,a->p),wn=tr(w->r,w->d);
 A z=ga(w->t,r,a->p);mv(z->p,w->p,wn=n>wn?wn:n);
 if(n-=wn)mv(z->p+wn,z->p,n);R z;}
V1(sha){A z=ga(0,1,&w->r);mv(z->p,w->d,w->r);R z;}
V1(id){R w;}V1(size){A z=ga(0,0,0);*z->p=w->r?*w->d:1;R z;}
pi(i){P("%d ",i);}nl(){P("\n");}
pr(w)A w;{I r=w->r,*d=w->d,n=tr(r,d);DO(r,pi(d[i]));nl();
 if(w->t)DO(n,P("< ");pr(w->p[i]))else DO(n,pi(w->p[i]));nl();}
 
C vt[]="+{~<#,";
A(*vd[])()={0,plus,from,find,0,rsh,cat},
 (*vm[])()={0,id,size,iota,box,sha,0};
I st[26]; qp(a){R  a>='a'&&a<='z';}qv(a){R a<'a';}
A ex(e)I *e;{I a=*e;
 if(qp(a)){if(e[1]=='=')R st[a-'a']=ex(e+2);a= st[ a-'a'];}
 R qv(a)?(*vm[a])(ex(e+1)):e[1]?(*vd[e[1]])(a,ex(e+2)):(A)a;}
noun(c){A z;if(c<'0'||c>'9')R 0;z=ga(0,0,0);*z->p=c-'0';R z;}
verb(c){I i=0;for(;vt[i];)if(vt[i++]==c)R i;R 0;}
I *wd(s)C *s;{I a,n=strlen(s),*e=ma(n+1);C c;
 DO(n,e[i]=(a=noun(c=s[i]))?a:(a=verb(c))?a:c);e[n]=0;R e;}
 
main(){C s[99];while(gets(s))pr(ex(wd(s)));}

(Source: An Implementation of J)

Arthur cofounded KX Systems in 1993 to sell kdb+, a column-based relational database system. kdb+ is very effective at storing and analyzing time series data, and uses q as the main language to manipulate data. Together, the kdb+/q tech stack is used for a variety of financial data processing purposes by several banking and financial institutions, such as Morgan Stanley and Citadel.

q Crash Course

The rest of this article will showcase the q language, emphasising the quirky features unique to it.

This is not intended to be a full q tutorial; it's meant to give you a glimpse of what working with this language looks like. For a more comprehensive overview, I suggest you read Q for Mortals.

The q Interpreter

Q is a fully interpreted language. Like other interpreted languages such as python or javascript, you can run q code either by running the interpreter on a source file, or directly typing code into a REPL. In this article, I will show q code entered into the q interpreter along with the evaluated result, as follows:

q)42
42
q)1+2+3+4
10

Comments

Anything following a / is considered a line comment and is ignored.

q)"Hello, world!"   / I'm just a comment, don't mind me.
"Hello, world!"

Atoms

Like all good programming languages, q has basic data types like integers, floats, bools, and chars. These singular values are called "atoms".

q)3        / Integers
3
q)78.90    / Floats
78.9
q)"q"      / Chars
"q"
q)0b       / Bools (0b is false, 1b is true)
0b

We've also got symbols, a common primitive in many functional languages. You can create a symbol with a backtick followed by some characters.

q)`bob123
`bob123

There are also a bunch of other datatypes for date/time, short/long sized ints, etc which you can read more about q datatypes here if you're interested.

Arithmetic Operators

Q has basic arithmetic operators for addition, subtraction, multiplication, division, etc for its numerical data types.

q)1+1
2
q)3.1-4.5
-1.4
q)60.2*70.3
420000
q)3%2    / Division uses `%` since `/` is used for comments
1.5

We also have comparison operators for (in)equalities.

q)123=123
1b
q)6>7
0b
q)42>=42
1b

There are many other built-in operators in q ("many" be an understatement…), some of which you will see later in this article.

Variables

Variables are assigned using :.

q)mynum:3
q)yournum:4
q)mynum*2
6
q)mynum+yournum
7

Lists

In q, you can make a list out of most atom types by jamming them next to each other on the same line.

q)1 2 3 4 5 6 7
1 2 3 4 5 6 7
q)3.0 4.0 5.5 6.5
3 4 5.5 6.5
q)`good`bad`ugly
`good`bad`ugly

A string is a list of chars.

q)"some chars"
"some chars"

A list of bools kind of looks like a binary string if you squint hard enough.

q)10010b
10010b

Lists can be indexed with square brackets.

q)a:10010b
q)a[0]
1b
q)b:`i`am`learning`q
q)b[3]
`q

You can also make a list of multiple types using parentheses, with items separated by semicolons.

q)l:(1; 2.5; "hello"; `sup)
q)l[2]
"hello"

List Operations

Many q operations and built-in functions that work with atoms automatically extend themselves to apply to lists as well.

For example, to add 10 to a list of numbers, you can simply type the following:

q)1 2 3 4 5+10
11 12 13 14 15

It also works with two lists, as long as they're the same length.

q)1 2 3 4 5+10 20 30 40 50
11 22 33 44 55

This ability is part of why q excels at expressing complex data processing tasks. Under the hood, these list operations use SIMD CPU instructions for blazing fast performance™.

q)a:1 2 3 4 5
q)b:3 3 3 4 4
q)a*a         / Square each number in the list
1 4 9 16 25
q)a=b         / Compare two lists for equality
00110b
q)(a+b)%2     / Find the average of each pair of numbers
2 2.5 3 4 4.5
q)abs a-b     / Return the absolute difference between two lists
2 1 0 0 1

Functions

This is where the syntax gets spicy!

Functions are declared with a pair of curly braces, with the parameter list enclosed within a pair of square brackets. They can be called using square brackets.

q)f:{[num] num*2}
q)f[5]
10
q)g:{[a;b;c] a+b+c}
q)g[1;2;3]
6
q)h:{[foo;bar] foo+bar}
q)h[1 1 1;2 3 4]    / Anything can be passed as arguments, for example lists
3 4 5

Steps of a function are separated with ;, with the final expression being the returned result.

q)h:{[a;b] c:a+2; d:b*2; c+d}
q)h[1;1]
5

Like in many functional languages, the brackets are actually optional. You can simply call a function by typing the function followed by its arguments.

q)f:{[x] x*2}
q)f[2]
4
q)f 2      / Does the same thing!
4

Note that in q, operator precedence is always right-to-left. Supposedly, this is to simplify the code for the parser of the q interpreter.

q)4-3*2+1    / Equivalent to 4-(3*(2+1))
-5
q)f:{[x] x*2}
q)f 1+2      / Equivalent to f[1+2]
6
q)f f 2      / Equivalent to f[f[2]]
8

For functions with 3 parameters or less, you can even omit the parameter list! The variables x, y, and z are implicitly the first, second, and third parameters as necessary.

q)f:{x+y}
q)f[20;30]
50

Example: Fibonacci Sequence

Let's try using q to solve a real-world problem! Along the way, we'll learn some of q's many powerful built-in functions.

The Fibonacci sequence is a sequence of numbers, recursively defined as follows:

• $f_1 = 1$
• $f_2 = 1$
• $f_n = f_{n-1} + f_{n-2}$ (for $n>2$)

Let's write some code to generate a list of the first $n$ Fibonacci numbers in q using a recursive approach. We'll want to make a function that takes some head of the Fibonacci sequence and appends the next number in the Fibonacci sequence to the end.

In q, the take operator # will allow us to take the first $n$ elements of a list. If the number is negative, it takes the last $n$ elements instead. The sum function returns the sum of the numbers in a list. Using these functions, we can take any head of the Fibonacci sequence and get the next number in sequence.

q)-2#1 1 2 3 5
3 5
q)sum -2#1 1 2 3
8

Combining this with the join operator , we can append the value to the end of the list to get the next step of our sequence.

q)arr:1 1 2 3 5
q)arr,sum -2#arr
1 1 2 3 5 8

Extracting this into a function, we can now iteratively generate the Fibonacci sequence.

q)step:{x,sum -2#x}
q)step[1 1]
1 1 2
q)step[step[1 1]]
1 1 2 3
q)step[step[step[1 1]]]
1 1 2 3 5

Unfortunately, it's rather cumbersome to have to repeatedly apply this function. Luckily, we can use the built-in do accumulator operator / to repeatedly apply a function to an argument a specified number of times (don't ask me how the parser differentiates this from comments, I have no clue).

q)step:{x,sum -2#x}
q)fib:{step/[x; 1 1]}
q)fib 5
1 1 2 3 5 8 13
q)fib 10
1 1 2 3 5 8 13 21 34 55 89 144

Fun fact: the ratio of consecutive values in the Fibonacci sequence provides increasingly accurate approximations of the golden ratio $\varphi=1.618033\dots$, using the ratios function, we can see this for ourselves.

q)fib 10
1 1 2 3 5 8 13 21 34 55 89 144
q)ratios fib 10
1 1 2 1.5 1.666667 1.6 1.625 1.615385 1.619048 1.617647 1.618182 1.617978

The find operator ? allows us to quickly identify the index of a value in the Fibonacci sequence.

q)fib 10
1 1 2 3 5 8 13 21 34 55 89 144
q)(fib 10)?5
4
q)(fib 10)?34
8

Suppose we wanted to pretty print the sequence as a string, we can use the string function to cast each integer into a string, then use the sv (scalar-from-vector) function to join the strings together using a separator.

q)string fib 10
,"1"
,"1"
,"2"
,"3"
,"5"
,"8"
"13"
"21"
"34"
"55"
"89"
"144"
q)", " sv string fib 10
"1, 1, 2, 3, 5, 8, 13, 21, 34, 55, 89, 144"

Example: Poker Hand Generator

Let's create a program that can deal a 5 card poker hand.

A standard deck of playing cards has 13 card values (Ace, 2-10, Jack, Queen, King) and 4 card suits (Spades, Hearts, Clubs, Diamonds). We'll create a deck of cards in q, using two character strings to represent a card with a suit and value.

The each operator is a handy built-in function that takes an existing function and applies it to each element of a list, in many functional languages this is called a "map" higher-order function.

We can use this operator to apply a function to each value in an array. For example, we can join each card value with a suit, e.g. "S" for spades, to generate all the spades cards in a deck of cards.

q){x,"S"} each "A23456789TJQK"
"AS"
"2S"
"3S"
"4S"
"5S"
"6S"
"7S"
"8S"
"9S"
"TS"
"JS"
"QS"
"KS"

Let's abstract away the card suit into a second parameter. Since we're calling a two-parameter function with only one argument, the function becomes partially applied. We can then complete the partially applied function by calling it with a value.

q){x,y} each "A23456789TJQK"
{x,y}'["A23456789TJQK"]
q)({x,y} each "A23456789TJQK")["S"]
"AS"
"2S"
"3S"
"4S"
"5S"
"6S"
"7S"
"8S"
"9S"
"TS"
"JS"
"QS"
"KS"

Using the each map, we can call this partially applied function with each card suit to generate the full deck of cards.

q)({x,y} each "A23456789TJQK") each "SHCD"
"AS" "2S" "3S" "4S" "5S" "6S" "7S" "8S" "9S" "TS" "JS" "QS" "KS"
"AH" "2H" "3H" "4H" "5H" "6H" "7H" "8H" "9H" "TH" "JH" "QH" "KH"
"AC" "2C" "3C" "4C" "5C" "6C" "7C" "8C" "9C" "TC" "JC" "QC" "KC"
"AD" "2D" "3D" "4D" "5D" "6D" "7D" "8D" "9D" "TD" "JD" "QD" "KD"

We have one slight problem, in that our output is actually a nested list of lists. Using the raze function, we can flatten this into a single list. Let's save this list into a variable called deck.

q)deck:raze ({x,y} each "A23456789TJQK") each "SHCD"
q)deck
"AS"
"2S"
"3S"
"4S"
"5S"
"6S"
"7S"
"8S"
"9S"
"TS"
"JS"
"QS"
"KS"
"AH"
"2H"
"3H"
"4H"
..

There are a few things we can do to make this expression shorter. Firstly, we can replace {x,y} with the literal (,) function. Secondly, we can use the single quote ', which actually does the same thing as the each map but with fewer characters.

q)deck:raze (,)'["A23456789TJQK"]'["SHCD"]

To draw a poker hand, we just need to pick 5 cards from the deck at random. Fortunately, there is an overload of ? called deal (legally distinct from the ? find operator) that can pick $n$ random entries from an array. Note that we must pass a negative number to pick the entries without replacement.

q)-5?deck
"2D"
"QH"
"AH"
"5C"
"5H"

Thus, we can compactly write a function that deals a hand of poker as follows.

q)deal:{-5?raze (,)'["A23456789TJQK"]'["SHCD"]}
q)deal[]    / Note that this function has 0 parameters
"KS"
"2D"
"JS"
"2S"
"AC"
q)deal[]
"TS"
"AC"
"5D"
"QC"
"8H"

It is left as an exercise to the reader to determine the winning hand amongst a set of dealt poker hands in q.

We can again use the sv (scalar-from-vector) function to combine the cards into one string.

q)" " sv deal[]    / Using sv to form one string
"JH 8D 3D 2H AH"

Incidentally, there's an inverse function called vs (vector-from-scalar) that can split a string into a list of strings on a single character.

q)" " vs "JH 8D 3D 2H AH"
"JH"
"8D"
"3D"
"2H"
"AH"

Dictionaries

Dictionaries are a way to create a map data structure in q, they're formed by placing a ! between two lists of the same length.

q)d:`a`b`c!1 2 3
q)d
a| 1
b| 2
c| 3

Indexing is done with square brackets.

q)d[`a]
1
q)d[`b`c]    / You can also index with a list to return multiple values simultaneously
2 3

A dictionary can be updated by setting the value for a key.

q)d[`a]:42
q)d
a| 42
b| 2
c| 3

A dictionary can have anything as its values, including more lists!

q)`name`age`gpa!(`billy`bob`joe; 16 18 21; 85 76 63)
name| billy bob joe
age | 16    18  21
gpa | 85    76  63

Tables

Tables are the bread and butter of the kdb+ database, allowing you to process column-based datasets natively within the q language.

A table consists of a list of named columns, followed by rows of data conforming to the column shape. The values in each column must all have the same datatype.

One way of creating a table is to apply the flip operation to a dictionary with lists as values, as seen in the previous section.

q)flip `name`age`gpa!(`billy`bob`joe; 16 18 21; 85 76 63)
name  age gpa
-------------
billy 16  85
bob   18  76
joe   21  63

There's also built-in notation for tables.

q)([] name:`billy`bob`joe; age:16 18 21; gpa:85 76 63)
name  age gpa
-------------
billy 16  85
bob   18  76
joe   21  63

Tables can be queried with a syntax similar to SQL.

q)tbl:([] name:`billy`bob`joe; age:16 18 21; gpa:85 76 63)
q)select name, age from tbl
name  age
---------
billy 16
bob   18
joe   21
q)select from tbl where gpa>70
name  age gpa
-------------
billy 16  85
bob   18  76
q)delete from tbl where name=`billy
name age gpa
------------
bob  18  76
joe  21  63

You can also index a table to get the $i$th row. The resulting row is a dictionary, mapping the column name to its value.

q)tbl:([] name:`billy`bob`joe; age:16 18 21; gpa:85 76 63)
q)tbl
name  age gpa
-------------
billy 16  85
bob   18  76
joe   21  63
q)tbl[0]
name| `billy
age | 16
gpa | 85
q)tbl[1]
name| `bob
age | 18
gpa | 76

Example: Calculating P&L

Suppose we have a list of buy and sell orders on the stock market and want to calculate for each stock, your current position (how many shares you own) and P&L (how much profit/loss you made from trading that stock, not including the value of current shares owned).

We can easily generate a random list of trades in q using built in operators, we will make use of yet another overload of ? called roll, where n?m will generate a list of n random integers in the range [0,m), given integers m and n.

Using this, we generate a mock table of 20 trades. Each row of the table indicates how many shares of a stock you bought, and at what price. We'll use the convention that a negative quantity indicates how many stock shares you sold instead.

q)trades:([] sym:20?`AAPL`TSLA`NVDA`MSFT; qty:(20?100)-50; price:20?1000)
q)trades
sym  qty price
--------------
AAPL 43  481
AAPL 14  564
AAPL -18 578
NVDA 41  920
NVDA -36 240
TSLA 43  444
...

The position for any given stock is simply the sum of the stock quantity (sum qty), and the P&L is negative of the sum of financial volume, where financial volume is price times quantity for each trade (neg sum qty * price). We will aggregate our calculations by the stock symbol (by sym) so that we get our P&L result for each stock.

The full calculation can be completed in a single line with a syntax very similar to SQL.

q)select position: sum qty, pnl: neg sum qty * price by sym from trades
sym | position pnl
----| ---------------
AAPL| 72       -29487
MSFT| -44      24200
NVDA| 53       -65419
TSLA| -1       4006

The resulting table tells us our position and P&L per stock. Note that in this case, a negative position would be considered a short position.

The resulting object is a keyed table, which is like a table except that it has a unique primary key column. I won't go too into detail about this type of table, just know that indexing a keyed table using the primary key returns a dictionary corresponding to that row of the table.

q)pos:select position: sum qty, pnl: neg sum qty * price by sym from trades
q)pos[`AAPL]
position| 72
pnl     | -29487
q)pos[`MSFT]
position| -44
pnl     | 24200

Less Code is Better Code

From my experience with q, I learned two valuable lessons about programming language design.

First, is the inherent benefit of short code. Shorter code takes up less screen space, which means more time spent reading and writing code, and less spent scrolling and hopping between files. q takes this to the extreme, where extraneous whitespace and empty lines are usually collapsed to use the minimum number of characters necessary.

Shorter code also means fewer bugs and a lower maintenance burden. I was told that the core of the q language interpreter is a mere 1000 lines of C, which you can imagine is much easier to maintain and debug compared to a project with orders of magnitude more lines of code.

Second, is the power in choosing the right abstractions. The q language is built on a solid foundation of core datatypes (lists, dictionaries, and tables), along with a carefully chosen set of built-in functions, which allow the language to excel at its role as a language used to process and analyze large datasets. I'm often amazed that q's parser is able to work sensibly at all, as the q language carefully threads the needle around dozens of symbols, each having several meanings depending on context.

I'm not claiming that every language should strive to emulate the extreme style that q promotes, nor do I think q is a perfect language without flaws. Nonetheless, I do believe the strengths of a short coding style and well-chosen abstractions of q are valuable, something which I feel other languages could stand to learn from.

As Mark Twain famously once wrote:

If I had more time, I would have written less code.