Columns and expressions

Shape order lines, calculate per-unit values, and export identifiers with deliberate types.

Step 1 of 6 · Learn

Select and rename

select chooses output fields and alias names a column without changing the original table. Each order_items row is one order line. line_total covers every unit on that line; it is not an individual unit price. The example renames it to amount.

Lesson reference: PySpark and Scala

Select and rename

PySpark example

from pyspark.sql.functions import col, round

result = order_items.select(col("order_item_id"), col("line_total").alias("amount")).orderBy("order_item_id")
result.show()

Scala example

import org.apache.spark.sql.functions._

val result = order_items.select(col("order_item_id"), col("line_total").alias("amount")).orderBy("order_item_id")
result.show(false)

Calculate a unit value

Divide each line_total by its quantity to obtain unit_value. withColumn adds the derived expression; round controls its displayed precision. Every quantity here is positive. Real datasets need a policy for missing or zero quantities before division.

PySpark example

from pyspark.sql.functions import col, round

result = order_items.withColumn(
    "unit_value", round(col("line_total") / col("quantity"), 1)
).select("order_item_id", "unit_value").orderBy("order_item_id")
result.show()

Scala example

import org.apache.spark.sql.functions._

val result = order_items.withColumn(
    "unit_value", round(col("line_total") / col("quantity"), 1)
).select("order_item_id", "unit_value").orderBy("order_item_id")
result.show(false)

Cast an identifier

cast("string") exports an identifier as text. The example exports product_id as reference alongside quantity. An order line has several identifiers with different meanings; use order_item_id when the exported reference must identify the line itself. Sort before projecting away the sort key.

PySpark example

from pyspark.sql.functions import col, round

result = order_items.orderBy("order_item_id").select(
    col("product_id").cast("string").alias("reference"), col("quantity")
)
result.show()

Scala example

import org.apache.spark.sql.functions._

val result = order_items.orderBy("order_item_id").select(
    col("product_id").cast("string").alias("reference"), col("quantity")
)
result.show(false)

example.pyPySpark

from pyspark.sql.functions import col, round result = order_items.select(col("order_item_id"), col("line_total").alias("amount")).orderBy("order_item_id") result.show()

1. Read order_itemsOne row per order line.

2. Select and renameApply the displayed expression without changing the source DataFrame.

3. Compare the resultNine order lines remain nine rows; selecting columns does not aggregate them.

Source order_items9 rows

order_item_id	order_id	product_id	quantity	line_total
1	1001	201	1	59.5
2	1001	202	1	30
3	1002	203	2	99
4	1002	206	1	50
5	1003	204	1	35
6	1004	203	3	149.97
7	1004	205	1	70.02
8	1005	203	1	49.99
9	1006	201	2	120

Select and rename

Result9 rows

order_item_id	amount
1	59.5
2	30
3	99
4	50
5	35
6	149.97
7	70.02
8	49.99
9	120

Nine order lines remain nine rows; selecting columns does not aggregate them.

The example is loaded in the editor. Run it as written, then try a small change.

Runs on the Spark backend. First startup may take a moment.

Run your code to see the result.