Thursday, June 15, 2017

DataWorks Summit 2017 - Verizon Finance Data Lake

Verizon has stood up a finance data lake to be a shared Enterprise Data Repository and enable self-service data discovery. This combines data from multiple SAP systems into a single, easy to access data repository. Benefits included:

  • Simplifies access to ERP data
  • Reduces data replication
  • Enables sharing of data (eliminating point to point integration)
  • Drive data archiving strategy
  • Rationalize data sources
  • Central set of business rules
  • Common reporting and analytic tools
  • Unified location to apply Data Governance and Security
The Architecture has data flow up from the source systems and into the data lake (~1,000 tables). The data movement is a combination of direct file imports, batch ingestion with Sqoop and incremental ingestion. The incremental ingestion was done by CDC and partition swaps originally but was eventually replaced with Attunity Replicate to simplify the maintenance and speed up processing.
Governance and security is applied, the sensitive data is either tokenized or not allowed into the cluster. The data is processed through the ETL layer, consisting mostly of Hive processes. The Logical model is the semantic layer provided by the reporting tools. 



There are areas to improve.

  • Data Validations are executed with manual scripts after the fact.
  • Data Governance is limited and needs review as more users are given access to the data
  • There is no managed data catalog at this time
  • Very few curated and maintained data sets
  • Self-service data preparation tools are being reviewed
  • Capabilities are limited to small number of uses

14 comments:

  1. The death of a beloved pet often devastates the person with whom it spent its final days. The desolation tends to increase with every pleasant year of life you shared together.


    EntertainmentBee
    TheMoviesBio
    PetRefine
    ThePetsAbout

    ReplyDelete
  2. Good. I am really impressed with your writing talents and also with the layout on your weblog. Appreciate, Is this a paid subject matter or did you customize it yourself? Either way keep up the nice quality writing, it is rare to peer a nice weblog like this one nowadays. Thank you, check also virtual edge and subject line for a thank you email

    ReplyDelete
  3. Directory For PlacesMarch 16, 2023 at 4:51 AM

    Directory for Places also offers a range of resources for cultural travelers, with a dedicated section that highlights places that are particularly suited for cultural experiences. https://bit.ly/directory-for-places Whether you're interested in museums, art galleries, historical landmarks, or cultural events, you can find it on the platform.
    A business guide provides information and guidance on various aspects of business management and ownership.

    ReplyDelete

  4. This way you can get a lot of information. Keep researching and reviewing.

    ReplyDelete