Module 11: Writing DataWeave Transformations

  • Home /
  • Duras /
  • Module 11: Writing DataWeave transformations
Table of Contents

About

1.

Objectives

  • Write DataWeave expressions for basic XML, JSON, and Java transformations
  • Write DataWeave transformations for complex data structures with repeated elements
  • Define and use global and local variables and functions
  • Use DataWeave functions
  • Coerce and format strings, numbers, and dates
  • Define and use custom data types
  • Call Mule flows from DataWeave expressions
  • Store DataWeave scripts in external files

Notes

Intro

Goal

  • You have been using DataWeave throughout class

    • To write inline expressions to dyanically set the value of properties in event processors
    • To transform data - this was mostly generated by the graphical drag-and-drop editor so far

  • In this module, you

    • Learn to write DataWeave transformations from scratch
    • Get familiar with the language so you can write more complicated transformations that are not possible with the drag-and-drop GUI

  • At the end of this module, you should be able to

    • Write DataWeave expressions for basic XML, JSON, and Java transformations
    • Write DataWeave transformations for complex data structures with repeated elements
    • Define and use global and local variables
    • Define and use DataWeave functions
    • Coerce and format strings, numbers and dates
    • Define and use custom data types
    • Call Mule flows from DataWeave expressions
    • Store DataWeave scripts in external files

Create transformations with the Transform Message component

Topic video

  • Creating transformations with the Transform Message component

    • To now, you have used the Transform Message component to

      • Create transformations using the visual editor
      • Define metadata for the input and output payload
      • Write basic transformation expressions

    • But…

      • Where does the code go?
      • Can you save the code externally and reuse it?
      • What happens when you create sample data for live preview?
      • Does the target of a transformation have to be the payload?

  • Where is the DataWeave code?


    Sample Data for Live Preview

    • By default, the expression is placed in the Mule configuration file

    • Sample data used for live preview is stored in src/test/resources


      Sample Data for Live Preview

  • Exporting DataWeave code to a DWL file


    Exporting DataWeave Code to a DWL file

    • In the Transform Message component, click the Edit current target button and set source code to file

      • Transform is saved in a DWL file
      • DWL files are stored in src/main/resources

  • Reusing DataWeave code

    • A single script can be stored in an external DWL file and referenced


      Reusing DataWeave code

      • In the Transform Message component using the resource attribute
      • In an element that has an expression property (like the Choice router) using the ${<:filename>} syntax
      • You can also create modules (libraries) of reusable DataWeave functions

  • Creating multiple transformations

    • You can also create multiple transformations with one Transform Message component


      Creating Multiple Transformations

  • Walkthrough 6-1: Create transformations with the Transform Message component

    • Create a new flow that receives POST requests of JSON flight objects
    • Customise propagated metadata
    • Add sample data and use live preview
    • Create a second transformation that stores the output in a variable
    • Save a DataWeave script in an external file
    • Review DataWeave script errors

Writing DataWeave transformation expressions

Topic video

  • The DataWeave expression is a data model for the output

    • It is not dependent upon the types of the input and output, just their structures
    • It’s against this model that the transform executes

    • The data model of the produced output can consist of three different types of data

      • Objects: Represented as collection of key value pairs
      • Arrays: Represented as a sequence of comma separated values
      • Simple literals

  • Example DataWeave transformation expression


    Example DataWeave Transformation Expression

  • The output directive

    • Specifies the mime type (format) that the script outputs


      Output Directive

  • Two types of DataWeave errors

    • Scripting errors

      • A problem with the syntax

    • Formatting errors

      • A problem with how the transformation from one format to another is written
      • For example, a script to output XML that does not specify a data structure with a single root node

    • If you get an error, transform the input to application/dw

      • If the transformation is successful, then the error is likely a formatting error

  • Including comments in DataWeave expressions

    • Comments that use Java-like syntax can be used

      // My single-line comment here
/** My multi-line
comment here. */

Extras

  • DataWeave Explained | Lightboard Series

    • DataWeave weaves between different formats
    • It’s about defining structure
    • The output engine that you pick based on the output format will dictate what that turns into
    • DataWeave is not about lines of codes, it’s an expression and has to evaluate or roll up to a value e.g. object, string, array, number
    • It’s like JSON with functions overlaid
    • JSON has very limited syntax

    • DataWeave parts

      • Header: Specifies what output engine to fire up or what output format to use, using an output keyword
      • Delimeters —
      • Body: is like a return section in Java where you are only allowed to return some sort of structure e.g. string, object etc

Transforming basic data structures

Topic video

  • Writing expressions for JSON or Java input and output

    • The data model can consist of three different types of data: objects, arrays, simple literals


      Writing expressions for JSON or Java input and output

  • Writing expressions for XML output

    • XML can only have one top-level value and that value must be an object with one property

      • This will throw an exception


        Throw DataWeave Exception

      • This will work


        Working DataWeave Expression

    • Specifying attributes for XML output

      • Use @(attName: attValue) to create an attribute


        XML Attribute

  • Writing expressions for XML input

    • By default, only XML elements and not attributes are created as JSON fields or Java object properties

    • Use @ to reference attributes


      Expression for XML Input

  • Walkthrough 6-2: Transform basic JSON, Java, and XML data structures

    • Write scripts to transform the JSON payload to various JSON and Java structures
    • Write scripts to transform the JSON payload to various XML structures

Transforming complex data structures with arrays

Topic video

  • Working with collections

    • Use the map function to apply a transformation to each element in an array

      • The input array can be JSON or Java

      • Returns an array of elements


        Working with Collections

  • The transformation function (or lambda)

    • Inside the transformation function

      • $$ refers to the index (or key)

      • $ refers to the value


        Transformation Function or Lambda

  • Use explicit arguments in lambdas for more readable code


    Explicity Arguments in Lambdas

  • Walkthrough 6-3: Transform complex data structures with arrays

    • Create a new flow that receives POST requests of a JSON array of flight objects
    • Transform a JSON array of objects to DataWeave, JSON, and Java

Extras

  • DataWeave Map Function | Lightboard Series

    • The map function takes in a collection or an array and transforms it into another array
    • A lambda function (nameless function) can also be used
    • It maps a structure defined by the user for each element within a collection, and outputs the structure

    • Structure

      payload map {
  firstName: $.fName,
  lastName: $.lName
}

// Equivalent to
payload map (object, index) -> {
  firstName: object.fName,
  lastName: object.lName
}

Transforming complex XML data structures

Topic video

  • Writing expressions for XML output


    Writing Expressions for XML Output

    • When mapping array elements (JSON or Java) to XML, wrap the map function in {(…)}

      • {} are defining the object
      • () are transforming each element in the array as a key/ value pair

  • Writing expressions for XML output (cont)


    Writing Expressions for XML Output (cont)

  • Writing expressions for XML output input

    • Use .* selector to reference repeated elements


      Writing Expressions for XML Input

  • Beware of tools that “prettify” responses

    • In some tools (like Postman), make sure you look at the raw response and not the pretty response


      Beware of Prettified Responses

    • This is because they tools only display the last node
  • Java doesn’t allow repeating properties of the same name like JSON and DW, so only the last input element is displayed because in Java property names must be unique

  • Walkthrough 6-4: Transform to and from XML with repeated elements

    • Transform a JSON array of object to XML
    • Replace sample data associated with a transformation
    • Transform XML with repeated elements to different data types

Defining and using variables and functions

Topic video

  • Defining and using global variables

    • Use the var directive in the header

    • Assign it a constant or a lambda expression

      • DataWeave is a functional programming language where variables behave just like functions

    • Global variables can be referenced anywhere in the body


      Defining and Using Global Variables

  • Defining and using variables in a syntax similar to traditional functions

    • DataWeave includes an alternate syntax to access lambda expressions assigned to a variable as function

      • May be more clear or easier to read for some people


        DataWeave Alternate Syntax

  • Defining and using local variables

    • Use the do keyword with the syntax do {<variable declaration header> --- <body>}

    • Local variables can only be referenced from within the scope of the expression where they are initialised


      Defining and Using Local Variables

  • Walkthrough 6-5: Define and use variables and functions

    • Define and use a global constant
    • Define and use a global variable that is equal to a lambda expression
    • Define and use a lambda expression assigned to a variable as a function
    • Define and use a local variable

Formatting and coercing data

Topic video

  • Changing data type is called coercion

  • Using the as operator for type coercion

    price: payload.price as Number
price: $.price as Number {class: "java.lang.Double"}

  • Defined types include

    • Any (the top-level type)
    • Scalar Data types: Null, String, Number, Boolean
    • Complex Data types: Object, Array, Range
    • Date, Time, LocalTime, DateTime, LocalDatetime, TimeZone, Period
    • More…

  • Using format patterns

    • Use the metadata format schema property to format numbers and dates

      price as Number as String {format: "###.00"},
someDate as DateTime {format: "yyyyMMddHHmm"}

  • Walkthrough 6-6: Coerce and format strings, numbers, and dates

    • Coerce data types
    • Format strings, numbers and dates

Using custom data types

Topic video

  • Defining and using custom data types

    • Use the type header directive

      • Recommended to lead with an uppercase letter, without special characters

        output application/json
type Ourdateformat = DateTime {format: "yyyyMMddHHmm"}
---
someDate: payload.departureDate as Ourdateformat

  • Transforming objects to POJOs

    • Specify inline customer:payload.User as Object {class: "my.company.User"}

    • Or define a custom data type to use

      type User = Object {class: "my.company.User"}
---
customer:payload.User as User

  • Walkthrough 6-7: Define and use custom data types

    • Define and use custom data types
    • Transform objects to POJOs

Using DataWeave functions

Topic video

  • Review: Using DataWeave functions

    • Functions are packaged in modules
    • Functions in the Core module are imported automatically into DataWeave scripts

    • For functions with 2 parameters, an alternate syntax can be used

      #[contains(payload, "max")]
#[payload contains "max"]

  • Calling functions that have a lambda expression as a parameter

    • The alternate notation can make calling functions that have a lambda expression as a parameter easier to read

      sizeOf(filter(payload, (value) -> value.age > 30))
sizeOf(payload filter $.age > 30)

  • When using a series of functions, the first function in the chain is executed first: filter then orderBy then groupBy

    flights filter ($.availableSeats > 30) orderBy $.price groupBy $.toAirport

Using DataWeave functions not in the Core module

  • Functions in all other modules must be imported

  • Import a function in a module

    output application/xml
import dasherize from dw::core::Strings
---
n: upper(dasherize(payload.firstname ++ payload.lastname))

Import a module

output application/xml
import dw::core::Strings
---
n: upper(Strings::dasherize(payload.firstname ++ payload.lastname))

Walkthrough 6-8: Use DataWeave functions

  • Use functions in the Core module that are imported automatically
  • Replace data values using pattern matching
  • Order data, remove duplicate data, and filter data
  • Use a function in another module that you must explicitly import into a script to use
  • Dasherize data

Accessing flow results in DataWeave

Topic video

  • Accessing flow results from DataWeave scripts

    • There are 2 primary methods for accessing a flow’s resultant payload in DataWeave

      • Use the lookup function

        • {a: lookup("myFlow", {b: "Hello"}) }
        • Deprecated in recent versions of Mule Runtime

      • Use a Flow Reference and variable

        • Use the target attribute of the Flow Reference to place the flow result in a variable
        • Reference that variable from within the DataWeave script
        • This is the recommended method: it avoids unexpected behaviour due to DataWeave engine thread dynamics

      • In addition to flows, Java methods can be invoked

        • Import the Java class using java!


          Invoke Java Methods in DataWeave

  • Calling flows from a DataWeave expression

    • Use the lookup function {a: lookup("myFlow",{b:"Hello"}) }
    • The first argument is the name of the flow to be called - cannot call subflows
    • The second arugment is the payload to send to the flow as a map
    • Whatever payload the flow returns is what the expression returns

  • Walkthrough 6-9: Look up data by calling a flow

    • Call a flow from a DataWeave expression

Walkthrough stops

Extras

  • Walkthrough video
  • Topic slides

Summary

DataWeave code can be inline, in a DWL file, or in a module of functions

The data model for a transformation can consist of three different types of data: objects, arrays and simple literals

Many formats can be used as input and output including JSON, Java, XML, CSV, Excel, Flat Fil, Cobol Copybook, Fixed Width, and more

The DataWeave application/dw format can be used to test expressions to ensure there are not scriptiong errors

Use the map function to apply a transformation function (a lambda) to each item in an array

A lambda is an anonymous function not bound to an identifier

When wrapping array elements (JSON or Java) to XML, wrap the map operation in {(…)}

DataWeave is a functional programming language where variables behave just like functions

Define global variables in the header with var

  • Assign them a constant or a lambda expression
  • Use fun directive to access lambdas assigned to variables as traditional functions

Define local variables in the body with do()

  • The scope of the do statement defines the boundaries of local variables

Functions are packaged in modules

  • Functions in the Core module are imported automatically into DataWeave scripts
  • Use the import header directive to import functions in all modules

Functions with 2 parameters can be called with 2 different syntax

Use the metadata format schema property to format numbers (#’s) and dates

Use the type header directive to specify custom data types

Transform objects to POJOs using: as Object {class: “com.myPOJO”}

Use lookup() to get data by calling other flows

Test your knowledge

Question 1

Question 2 What is the correct way to define the DateYearFirst custom data type?

  • DateYearFirst = Date{format: "yyyy/MM/dd"}

Question 3 What is the correct way to format the decimal 20.3844 as a string to two decimal places?

  • ~20.3844 as String{format: “.0#”}

Question 4 What DataWeave 2.0 type can be used as input to a map operation?

  • Array

Question 5

    • payload filter $.price < 500 orderBy $.price groupBy $.toAirport

Anki

References

  1. . “Module 11: Writing DataWeave transformations”. Available at: . (Accessed: [2025-03-09 Sun 21:11]). ↩︎

Random Posts