Octopai Has Announced A New Data Lineage Platform

Israeli data discovery company Octopai has announced a new data lineage platform, Data Lineage XD, which takes care of and automatically discovers and records data transformations and flows within customer data areas. “XD” stands for Interdimensional, and the name isn’t just a marketing gimmick. The Octopai system can map what the company calls cross-system lineage, intra-system lineage, and end-to-end column lineage.

Octopai executives briefed jsspu on the data lineage XD and demo aspects of the platform. They explain that cross-system lineage maps the flow of data across systems, from initial ingestion and extraction, transformation and loading (ETL) to reporting and analysis. It provides color-coded network diagrams in interactive visualizations to illustrate flows and dependencies. The image above provides an example.

Internal system lineage maps transformations of data columns in ETL processes, reports, or database objects. To achieve this, Octopai not only has connectors to data sources and destinations, but also to various ETL platforms, business intelligence (BI) backends and self-service BI visualization tools. Instead of treating everything as a database, Octopai’s connectors have a contextual understanding of how to read metadata and code in SQL scripts and stored procedures. It can also read “code” (detailed transformations and dependencies) in ETL and BI assets.

End-to-end column lineage details data column-specific lineage between systems. It is particularly relevant to regulatory compliance, impact analysis, and root cause analysis. Because the connectors provided by Octopai are very component-specific, the column lineage that Octopai can generate is very granular and detailed.

It supports the following platforms:

• Support for eEnterprise and cloud data warehouse platforms such as Teradata and Snowflake;
• ETL platforms such as IBM Data Stage and Information;
• Enterprise BI platforms such as IBM Cognos and SAP BusinessObjects;
• Self-service BI tools, such as human-composed pictures or scenarios, and five different Oracle products, ranging from databases and data warehouses to ETL and BI;
• For fans of the Microsoft stack, Octopai supports SQL Server (relational), Analysis Services Multidimensional and Tabular Schema (BI), Integration Services (ETL), SQL Server Reporting Services and Power BI.
• On the Microsoft Cloud, Octopai supports Azure SQL Database, Azure Synapse Analytics (SQL Pool), Azure Analytics Services and Azure Data Factory.

Its use cases:

Applications of this technology fall into several categories. First, the platform is a great way to help teams keep track of what they build and use it to resolve errors in the data pipeline more quickly. However, consulting firms/system integrators and internal development organizations responsible for supporting, enhancing, and/or migrating systems that they did not build themselves can also take advantage of this platform.

This ability to analyze, visualize, and help teams learn how to build systems can also work very well in M&A scenarios, where one IT organization has to be responsible for the assets of another IT organization.

Data Lineage XD provides support for several more platforms than listed above, and more will be released in the future. In addition to the core data lineage facility, the data Lineage XD provides a business glossary, which is provided by its data discovery capabilities and aided by XD’s detailed knowledge of a particular platform.

Introduction of Octopai

Octopai, like Gudu SQLFlow, is one of the best Data lineage tools or software available in the market today.

The Octopai company was created by a group of BI professionals who were clearly involved in the implementation of the project and understand the difficulties of building or being responsible for such a system. For organizations with complex enterprise BI/analytics systems that need to be controlled, the Octopai platform is worth a look.