Bigquery nested array

In BigQuery, an array is an ordered list consisting of zero or more values of the same data type. You can construct arrays of simple data types, such as INT64, and complex data types, such as STRUCTs. The current exception to this is the ARRAY data type because arrays of arrays are not supported. To learn more about the ARRAY data type, including NULL handling, see Array type.

With BigQuery, you can construct array literals, build arrays from subqueries using the ARRAY function, and aggregate values into an array using the ARRAY_AGG function.

You can combine arrays using functions like ARRAY_CONCAT(), and convert arrays to strings using ARRAY_TO_STRING().

Constructing arrays

Using array literals

You can build an array literal in BigQuery using brackets ([ and ]). Each element in an array is separated by a comma.

You can also create arrays from any expressions that have compatible types. For example:

Notice that the second example contains three expressions: one that returns an INT64, one that returns a FLOAT64, and one that declares a literal. This expression works because all three expressions share FLOAT64 as a supertype.

To declare a specific data type for an array, use angle brackets (< and >). For example:

Arrays of most data types, such as INT64 or STRING, don't require that you declare them first.

You can write an empty array of a specific type using ARRAY<type>[]. You can also write an untyped empty array using [], in which case BigQuery attempts to infer the array type from the surrounding context. If BigQuery cannot infer a type, the default type ARRAY<INT64> is used.

Using generated values

You can also construct an ARRAY with generated values.

Generating arrays of integers

GENERATE_ARRAY generates an array of values from a starting and ending value and a step value. For example, the following query generates an array that contains all of the odd integers from 11 to 33, inclusive:

You can also generate an array of values in descending order by giving a negative step value:

Generating arrays of dates

GENERATE_DATE_ARRAY generates an array of DATEs from a starting and ending DATE and a step INTERVAL.

You can generate a set of DATE values using GENERATE_DATE_ARRAY. For example, this query returns the current DATE and the following DATEs at 1 WEEK intervals up to and including a later DATE:

Accessing array elements

Consider the following table, sequences:

This table contains the column some_numbers of the ARRAY data type. To access elements from the arrays in this column, you must specify which type of indexing you want to use: either OFFSET, for zero-based indexes, or ORDINAL, for one-based indexes.

Finding lengths

The ARRAY_LENGTH() function returns the length of an array.

Converting elements in an array to rows in a table

To convert an ARRAY into a set of rows, also known as "flattening," use the UNNEST operator. UNNEST takes an ARRAY and returns a table with a single row for each element in the ARRAY.

Because UNNEST destroys the order of the ARRAY elements, you may wish to restore order to the table. To do so, use the optional WITH OFFSET clause to return an additional column with the offset for each array element, then use the ORDER BY clause to order the rows by their offset.

Example

To flatten an entire column of ARRAYs while preserving the values of the other columns in each row, use a correlated cross join to join the table containing the ARRAY column to the UNNEST output of that ARRAY column.

With a correlated join, the UNNEST operator references the ARRAY typed column from each row in the source table, which appears previously in the FROM clause. For each row N in the source table, UNNEST flattens the ARRAY from row N into a set of rows containing the ARRAY elements, and then the cross join joins this new set of rows with the single row N from the source table.

Examples

The following example uses UNNEST to return a row for each element in the array column. Because of the CROSS JOIN, the id column contains the id values for the row in sequences that contains each number.

Note that for correlated cross joins the UNNEST operator is optional and the CROSS JOIN can be expressed as a comma-join. Using this shorthand notation, the above example becomes:

Querying nested arrays

If a table contains an ARRAY of STRUCTs, you can flatten the ARRAY to query the fields of the STRUCT. You can also flatten ARRAY type fields of STRUCT values.

Querying STRUCT elements in an ARRAY

The following example uses UNNEST with CROSS JOIN to flatten an ARRAY of STRUCTs.

You can find specific information from repeated fields. For example, the following query returns the fastest racer in an 800M race.

This example does not involve flattening an array, but does represent a common way to get information from a repeated field.

Example

Querying ARRAY-type fields in a STRUCT

You can also get information from nested repeated fields. For example, the following statement returns the runner who had the fastest lap in an 800M race.

Notice that the preceding query uses the comma operator (,) to perform an implicit CROSS JOIN. It is equivalent to the following example, which uses an explicit CROSS JOIN.

Flattening arrays with a CROSS JOIN excludes rows that have empty or NULL arrays. If you want to include these rows, use a LEFT JOIN.

Creating arrays from subqueries

A common task when working with arrays is turning a subquery result into an array. In BigQuery, you can accomplish this using the ARRAY() function.

For example, consider the following operation on the sequences table:

This example starts with a table named sequences. This table contains a column, some_numbers, of type ARRAY<INT64>.

The query itself contains a subquery. This subquery selects each row in the some_numbers column and uses UNNEST to return the array as a set of rows. Next, it multiplies each value by two, and then recombines the rows back into an array using the ARRAY() operator.

Filtering arrays

The following example uses a WHERE clause in the ARRAY() operator's subquery to filter the returned rows.

Note: In the following examples, the resulting rows are not ordered.

Notice that the third row contains an empty array, because the elements in the corresponding original row ([5, 10]) did not meet the filter requirement of x < 5.

You can also filter arrays by using SELECT DISTINCT to return only unique elements within an array.

You can also filter rows of arrays by using the IN keyword. This keyword filters rows containing arrays by determining if a specific value matches an element in the array.

Notice again that the third row contains an empty array, because the array in the corresponding original row ([5, 10]) did not contain 2.

Scanning arrays

To check if an array contains a specific value, use the IN operator with UNNEST. To check if an array contains a value matching a condition, use the EXISTS operator with UNNEST.

Scanning for specific values

To scan an array for a specific value, use the IN operator with UNNEST.

Example

The following example returns true if the array contains the number 2.

To return the rows of a table where the array column contains a specific value, filter the results of IN UNNEST using the WHERE clause.

Example

The following example returns the id value for the rows where the array column contains the value 2.

Scanning for values that satisfy a condition

To scan an array for values that match a condition, use UNNEST to return a table of the elements in the array, use WHERE to filter the resulting table in a subquery, and use EXISTS to check if the filtered table contains any rows.

Example

The following example returns the id value for the rows where the array column contains values greater than 5.

Scanning for STRUCT field values that satisfy a condition

To search an array of STRUCTs for a field whose value matches a condition, use UNNEST to return a table with a column for each STRUCT field, then filter non-matching rows from the table using WHERE EXISTS.

Example

The following example returns the rows where the array column contains a STRUCT whose field b has a value greater than 3.

Arrays and aggregation

With BigQuery, you can aggregate values into an array using ARRAY_AGG().

The array returned by ARRAY_AGG() is in an arbitrary order, since the order in which the function concatenates values is not guaranteed. To order the array elements, use ORDER BY. For example:

You can also apply aggregate functions such as SUM() to the elements in an array. For example, the following query returns the sum of array elements for each row of the sequences table.

BigQuery also supports an aggregate function, ARRAY_CONCAT_AGG(), which concatenates the elements of an array column across rows.

Note: The array returned by ARRAY_CONCAT_AGG() is non-deterministic, since the order in which the function concatenates values is not guaranteed.

Converting arrays to strings

The ARRAY_TO_STRING() function allows you to convert an ARRAY<STRING> to a single STRING value or an ARRAY<BYTES> to a single BYTES value where the resulting value is the ordered concatenation of the array elements.

The second argument is the separator that the function will insert between inputs to produce the output; this second argument must be of the same type as the elements of the first argument.

Example:

The optional third argument takes the place of NULL values in the input array.

• If you omit this argument, then the function ignores NULL array elements.

• If you provide an empty string, the function inserts a separator for NULL array elements.

Example:

Combining arrays

In some cases, you might want to combine multiple arrays into a single array. You can accomplish this using the ARRAY_CONCAT() function.

Zipping arrays

Given two arrays of equal size, you can merge them into a single array consisting of pairs of elements from input arrays, taken from their corresponding positions. This operation is sometimes called zipping.

You can zip arrays with UNNEST and WITH OFFSET. In this example, each value pair is stored as a STRUCT in an array.

You can use input arrays of different lengths as long as the first array is equal to or less than the length of the second array. The zipped array will be the length of the shortest input array.

To get a zipped array that includes all the elements even when the input arrays are different lengths, change LEAST to GREATEST. Elements of either array that have no associated element in the other array will be paired with NULL.

Building arrays of arrays

BigQuery does not support building arrays of arrays directly. Instead, you must create an array of structs, with each struct containing a field of type ARRAY. To illustrate this, consider the following points table:

Now, let's say you wanted to create an array consisting of each point in the points table. To accomplish this, wrap the array returned from each row in a STRUCT, as shown below.

How do you make a nested column in BigQuery?

What is array AGG in BigQuery?

How do you create an array in BigQuery?

How do you query nested objects in BigQuery?