Data Structures

Types

Each type can be represented in two ways: a numeric reference value and a character reference value.
e.g., the long type can be represented as -7 or j.
To examine the type, use the type function. But when you read the output, only read the numeric not the character:

q) type 12345          // returns -7h

In the example above, the output -7h indicates that the type of 12345 is long (represented by -7), while the 'h' suffix is for -7 itself you can safely ignore.

Nulls and infinities

Nulls are created by padding the type with 0N, and infinities are created by padding the type with 0W.
For example, null longs and long infinities are 0Nj and 0Wj respectively.
A generic long without any type can be represented as ::. It's usually used as placeholder in functions.

Casting

Three ways to do casting with $.

            q) `float$42    // use symbol name. symbol name for hour is not hour but hh
            q) "f"$42       // use character name
            q) 9h$42.       // use numeric name
        

Can also use their functional forms.
Special attention needed for symbols and strings:

• Only strings can be directly casted into symbols. Other data types must be converted to strings before casting to symbols.
• When you cast a string, you have to use character name, and you have to use an upper case character. e.g. "F"$"34.789"
• You can cast multiple strings into different types at once. e.g.

  
                      q) "SJF*"$"JPM";"100";"4.5";"test string"
                      `JPM
                      100
                      4.5
                      "test string"
                  

Checking equality

Three ways to check equality:

1. = and its counterparty <> don't do type checking. They only compare values.
2. ~ this is exact match: not only checks the value but the type as well. It treats a list as a single entity.
   However, ~ does not check attributes on lists. So asc[1 2 3 4]~1 2 3 4 returns true.
3. like this is used for string comparison.

List

Concatenating lists

Easiest way is to use ,.
enlist is essentially the same as joining an empty list using , for single item lists.
But if you use enlist on a 1D list, it'll create a 2D list, whereas , still gives you a 1D list.

            q) enlist 4             // same as (),4
            q) enlist 1 2 3         // this returns a 2D list: (1; 2; 3), type 0h
            q) (),1 2 3             // this returns a 1D list: 1 2 3, type 7h
        

Generating lists

• roll/deal operator ?
  It generates random numbers. The first parameter is the number of random numbers you want, and the second parameter is the range of the random numbers, which can also be a list.
  
  • When you pass a positive number, it's called roll. It samples with replacement.
  • When you pass a negative number, it's called deal. It samples without replacement.

  
                      q) 10?100      // roll: generates 10 random numbers with replacement   
                      q) 10?`yes`no  // roll: generates 10 random symbols from a given list
                      q) -10?100     // deal: generates 10 random numbers without replacement
                      q) -300?100    // error: cannot draw 300 items from 100 integers without replacement
                  

• til: this is similar to range() in python, inclusive of 0 but exclusive of the end value.

  q) til 5     // returns 0 1 2 3 4

• retrieve a list from a table: tbl`colname
• generate a list from another list
  • take: take the first n items. n can be negative.
    If you take more than the length of the list, it circles back to the beginning of the list.

    q) 3#til 10     // returns 0 1 2

  • drop: drops the first n items. n can be negative.
    If you drop more than the length of the list, it returns an empty list.

    q) 3_til 10     // returns 3 4 5 6 7 8 9

  • sublist: a more flexible version that allows you to take not from the beginning of the list, but from any index.
    The syntax is start_index how_many_items sublist L

    q) 2 4 sublist til 9  // returns 2 3 4 5; take 4 from index 2

• cut: this cuts a list into a list of sublists of the same length specified by the first parameter.

  q) 4 cut til 12   // returns (0 1 2 3; 4 5 6 7; 8 9 10 11)

• enlist: can either create a 1D list from a single element, or a 2D list from a 1D list

Indexing lists

1D lists

Indexing can be done with or withouth the square brackets [].
If index out of bound, it'll return null of the same data type as the first element in the list.
Pay attention though, in KDB DO NOT use negative index. It doesn't work like python. It'll return null instead of returning the element from the end of the list.
If you want to achieve similar effect, do this

q) L[-2 + count L]

If you want to index a list of indices instead of a single element, you can do this:

q) @[L; 1 2]

2D lists

Indexing a 2D list is also similar to python.
You can either do M[i;j] or M[i][j]. If you want a row or a column, it's similar to python slicing.

            q) M: (1 2 3; 4 5 6; 7 8 9)
            q) M[1;3]     // returns 6; row 1, column 3
            q) M[1]       // returns 4 5 6; row 1
            q) M[;1]      // returns 2 5 8; column 1
            q) M[1;]      // returns 4 5 6; row 1
        

Searching in a list

Two ways to find in a list: find ? and where. To find in a list we use the same operator ?. It is called find here not roll, which we used to generate a random list.
The key difference is the order of the parameters. For find, the list needs to be the first parameter; but for roll, the list needs to be the second parameter.
Another way is to use where. Different from find, we only return the first occurence, where will return all occurences.
It's also possible to pass a list to be searched.

            q) L:10?til 100     // this is roll, generating a list
            // suppose L is 43 7 56 7 89 5 34 5 23 45
            q) 5?L              // this is find, finding the first index of 5, which is 5
            q) where L=5        // this is where, finding all indices of 5, which are 5 and 7
        

Modifying lists

The vanilla way to do this is to use the index to assign a new value, like L[2]:99.
But if you want to modify a list at multiple indices at once, it's easier to use the @ operator.
Syntax: @[list; indices; function; args to function] where function can be assign :, or add + ,etc. Only the first two parameters are required.
When you pass just the list name as the first parameter, e.g., L, this creates a copy of the list. But when you pass the symbol name with a backtick, e.g., `L, this modify the original list in place.
Examples:

1. simple index: @[sampleFares;(2*til 5)]. This returns the 0th, 2th, 4th, 6th, and 8th element.
2. assign values at indices: @[sampleFares;(2*til 5);:;99f]. This assigns value 99f to the 0th, 2th, 4th, 6th, and 8th element.
3. negating values at all indices: @[sampleFares; ::; neg]

For 2D lists, you need to use . instead of @.
Syntax: .[matrix; index at depth; function; args to function]
Again both @ and . can be used as operators rather than in functional form as we do above. If you use them as operators, the list has to be the second argument.

Handling nulls in a list

One way to handle nulls is to use where, but even easier, you can use fill ^ and fills keywords.
Fill need to take a given value to fill the nulls, while fills will use the last non-null value to fill the nulls.

            // Fill ^ examples
            q) L: 1 2 0N 0N 3 8 0N
            q) 0^L          // returns 1 2 0 0 3 8 0; fill nulls with 0
            q) avg(L)^L     // returns 1 2 3.5 3.5 3 8 3.5; note that avg(L) is 3.5 not 2, nulls are not counted
            q) L2^L         // fill L with another list L2; L2 must be the same length as L
            // Fills example
            q) fills L      // returns 1 2 2 2 3 8 8; fill nulls with the last non-null value
        

Handy list functions

• except: remove items from a list
• in: check if an item is in a list
• inter: find intersection of two lists. No guarantee that there is no duplicate.
• union: find union of two lists
• distinct: find distinct elements in a list

Strings

Strings are lists of characters in KDB. So anything that applies to lists also applies to strings.
Besides what applies to lists, there are some string-specific functions.

Parsing and composing strings

Two functions can be used for this purpose: sv string from vector, and vs vector from string.

            q) vs[","; "hello,world"]           // split the string at ,
            "hello"
            "world"
            q) sv["|"; ("hello"; "world")]      // put chunks together and glue using |
            "hello|world"
        

A performance hack is that, if you need to perform the same function on a list of strings, better concatenate them into a long string, perform the function, then split them.

Search and replace in strings

If you only want to search, can use ss.
If you want to search and replace, can use ssr. The first parameter is the string to be searched, the second parameter is the pattern to be replaced, which can be regex, and the third parameter is the replacement, which can be a function.
But pay attention, when you use regex, the wildcard * won't work in either ss or ssr. You need to use ? instead.

q) ssr["atorb"; "t?r"; upper]        // returns aTORb; regex matches tor, then convert them to upper TOR

Common string operations

• trimming strings: three options ltrim, rtrim, and trim
• padding strings: using positive values to pad to the right, and negative values to pad to the left.

  q) 10$"example"      // pad 3 spaces to the right
  q) -10$"example"     // pad 10 spaces to the left

  If you pass a value that is less than the length of the string, it extracts a substring.
• uppercase and lowercase: just use upper and lower
• remove non alphanumeric strings:

  q) myStr except .Q.nA

  Here .Q.nA is a predefined list of all non-alphanumeric characters.

Dictionaries

Dictionaries

Dictionaries are key-value pairs made of two lists of equal length.

            q) d: `a`b`c!1 2 3
            q) d
            a | 1
            b | 2
            c | 3

            // lookup values
            q) d[`a]          // returns 1
            q) d `a           // same, not in functional form

            // lookup keys
            q) d?1            // returns `a

            // modify dictionaries
            q) d[`a]: 10      // assign new values
            q) d[`e`f]: 4 5   // add new items
            q) `a`b _ d       // remove items. Note that there must be a whitespace before the _

            // other handy stuff
            q) `e`f # d       // extract a subset of items
            q) d,d2           // upsert d2 into d
            q) d^d2           // upsert d2 into d, but if the value is null in d2, it'll still use the value in d
            q) `s#d           // sort d by key
        

Column dictionaries

If the values of a dictionary are lists, then it is called a column dictionary.

            q) cd: `a`b`c!(1 2 3; 4 5 6; 7 8 9)
            q) cd
            a | 1 2 3
            b | 4 5 6
            c | 7 8 9
        

Tables are flipped column dictionaries.
Keyed table and dictionary share the same type 99h. A keyed table is NOT a table, but a dictionary. More details about tables can be found in the Q SQL section.