Lithium: elevating ETL with ephemeral and self-hosted pipelines

At Atlassian, there are many situations we need to move large amounts of data between different systems – validating and transforming them in-flight. What is unique about Atlassian’s use cases is that several require data pipelines to be entirely provisioned at run time.

Normally, we’d consider frameworks such as Apache Flink to move and transform data at scale – effectively through a streaming ETL (Extract, Transform, and Load). ETLs have been around for decades – we’ve all worked with or benefitted from them.