CMSI 486: Welcome to Week 10

This Week's Class Agenda

What's Going On This Week

1. Project SCRUMS
2. Introduction to Graph Databases
3. Introduction to Neo4J
4. Some interesting comparisons

Introduction to Graph Databases

Just what *IS* a graph database?

BIG DATA… … just keeps getting bigger.
Year by year, more and more data is logged into databases for all kinds of reasons. Businesses have come to rely on applications that can interpret data, and make connections between both related and unrelated data items, to improve the profit margins and sales, and to cut down on the overhead of having people do the extensive analysis of the data. Thus, those applications must be able to accommodate HUGE amounts of data, and also must be able to detect patterns, quickly provide relevent facts [both independent facts and connections], and output information that can be used to improve business value.

Although the relational model for database engines has been the gold standard for many years, its limits are now being reached, as we've seen on previous pages during the course of the semester. It's ironic, but relational databases are actually quite poor at handling actual relationships between data points. The table model and inflexibility of pre-defined schemas mean it is difficult to add new connections between data or to make different kinds of connections within the database.

So, graph databases to the rescue! But what exactly *IS* one of these blasted things?

As we saw two weeks ago in the NoSQL discussion, graph databases use the concept of a computer science graph to store data and relationships. Each data item is a node, and nodes are connected by edges. Both nodes and edges can have attributes associated with them. The model we will use, Neo4j, uses a very specific type of structure known as the labeled property graph, which is one of the most generic and versatile graph models. It provides four different building blocks for structuring data:

1. NODES are used to store entity information. like books, cars, students, buildings, and so on. This is the basis of where the actual data items are stored.
2. RELATIONSHIPS or edges are used to explicitly connect the nodes to one another and provide a way to structure the nodes. You can think of edges as sort of a pre-defined and pre-executed join operation, but one that is set up for several possible levels of granularity. For example, since edges always have a type, a direction, a start node, and an end node, they can be self-referencing to provide easy ways of storing relationships like a person is a daughter and a mother and an aunt and a sister. Such things are much more difficult in the relational model.
3. PROPERTIES is the term that is applied to what we previously know as attributes. They are stored as name/value pairs, sort of like we've seen in the document store databases like our old friend mondo. However, since edges [relationships] can have properties, too, this opens up a whole new class of storage, which can be used to further qualify the relationship between nodes, and which can be used to help with queries and graph traversals.
4. Finally, LABELS are a way to create subgraphs, which can be a big help when trying to index the graph and to provide meta-information about the nodes so that there doesn't need to be a node containing definitions for nodes or type properties.

get pic from learning book p. 74

What are some use cases for them?
[Excerpted from The top 5 use cases of graph databases from neo4j.com]

• Case #1: Fraud Detection
  Banks and insurance companies lose billions of dollars every year to fraud. Traditional methods of fraud detection fail to minimize these losses since they perform discrete analyses that are susceptible to false positives and negatives. Knowing this, increasingly sophisticated fraudsters develop a variety of ways to exploit the weaknesses of discrete analysis.
  Graph databases offer new methods of uncovering fraud rings and other complex scams with a high level of accuracy through advanced contextual link analysis, and they are capable of stopping advanced fraud scenarios in real time.
  

  Why Use a Graph Database for Fraud Detection?

  While no fraud prevention measures are perfect, significant improvements occur when you look beyond individual data points to the connections that link them. Understanding the connections between data, and deriving meaning from these links, doesn’t necessarily mean gathering new data. You can draw significant insights from your existing data simply by reframing the problem in a new way: as a graph. Unlike most other ways of looking at data, graphs are designed to express relatedness. Graph databases uncover patterns that are di cult to detect using traditional representations such as tables. An increasing number of companies use graph databases to solve a variety of connected data problems, including fraud detection.
  

  Example: E-commerce Fraud

  As our lives become increasingly digital, a growing number of financial transactions are conducted online. Fraudsters have adapted quickly to this trend and have devised clever ways to defraud online payment systems. While this type of activity can and does involve criminal rings, even a single well-informed fraudster can create a large number of synthetic identities and can carry out sizeable schemes. Consider an online transaction with the following identifiers: user ID, IP address, geo-location, a tracking cookie and a credit card number. Typically, the relationships between these identifiers should be (almost) one-to-one. Some variations naturally account for shared machines, families sharing a single credit card number, individuals using multiple computers and the like. However, as soon as the relationships between these variables exceed a reasonable number, fraud should be considered as a strong possibility. The more interconnections exist amongst identifiers, the greater the cause for concern. Large and tightly-knit graphs are very strong indicators that fraud is taking place.
  By putting checks into place and associating them with the appropriate event triggers, such schemes can be uncovered before they are able to infliict significant damage. Triggers can include events such as logging in, placing an order or registering a credit card – any of which can cause the transaction to be evaluated against the fraud graph. Fan-out might be skipped, but complex graphs can be flagged as a possible instance of fraud.
  

  Conclusion

  Graph databases are the ideal enabler for effcient and manageable fraud detection solutions. From fraud rings and collusive groups, to educated criminals operating on their own, graph databases uncover a variety of important fraud patterns – and all in real time.
  
  
• Use Case #2: Real-Time Recommendation Engines
  Whether your enterprise operates in the retail, social, services or media sectors, offering your users highly targeted, real-time recommendations is essential to maximizing customer value and staying competitive. Unlike other business data, recommendations must be inductive and contextual in order to be considered relevant by your end consumers.
  With a graph database, you’re able to capture a customer’s browsing behavior and demographics and combine those with their buying history to instantly analyze their current choices and then immediately provide relevant recommendations – all before a potential customer clicks to a competitor’s website.
  

  Why Use a Graph Database to Power Real-Time Recommendation Engines?

  The key technology in enabling real-time recommendations is the graph database. Graph databases also out-class other database technology for connecting masses of buyer and product data (or connected data in general). Making effective real-time recommendations depends on a database that understands the relationships between entities, as well as the quality and strength of those connections. Only a graph database efficiently tracks these relationships according to user purchase, interactions, and reviews to give you the most meaningful insight into customer needs and product trends. Graph-powered recommendation engines can take two major approaches: identifying resources of interest to individuals; or identifying individuals likely to be interested in a given resource. With either approach, graph databases make the necessary correlations and connections to serve up the most relevant results for the individual or resource in question.
  

  Example: Walmart and eBay

  Retail industry leader Walmart has sales of more than $460 billion and employs 2.2 million associates worldwide, serving more than 245 million customers weekly through its 11,000 stores in 27 countries and e-commerce websites in 10 countries. Their development team has decided to use a graph database to serve up real-time product recommendations by using information about what users prefer. Walmart Software Developer Marcos Wada states that a graph database helps us to understand our online shoppers' behavior and the relationship between our customers and products, providing a perfect tool for real-time product recommendations. E-commerce giant eBay has also found success using a graph-powered suggestion engine, in this case, for a sophisticated real-time courier/package routing solution. Senior Developer Volker Pacher at eBay says his team found a graph database to be literally thousands of times faster than our prior MySQL solution, with queries that require 10-100 times less code. Today, our graph database provides eBay with functionality that was previously impossible.
  

  Conclusion

  Storing and querying recommendation data using a graph database allows your application to provide real-time results rather than precalculated, stale data. As consumer expectations increase – and their patience decreases – providing these sorts of relevant, real-time suggestions will become a greater competitive advantage than ever before. Real-time recommendation engines provide a key differentiating capability for enterprises in retail, logistics, recruitment, media, sentiment analysis, search and knowledge management.
  
  
• Use Case #3: Master Data Management
  Master data is the lifeblood of your enterprise, including data such as: Users; Customers; Products; Accounts; Partners; Sites; and Business units. Many business applications use master data and its often held in many different places, with lots of overlap and redundancy, in different formats, and with varying degrees of quality and means of access. Master data management (MDM) is the practice of identifying, cleaning, storing, and – most importantly – governing this data. MDM best practices vary along the spectrum of merging all master data into a single location to managing data assets for easy access from a single service or application. In both cases (or any hybrid solution), enterprise data architects need a data model that provides for ad hoc, variable and exceptional structures as business requirements change. That sort of rapidly evolving model fits best with a graph database.
  

  Why Use a Graph Database for Master Data Management Solutions?

  Because master data is highly connected and shared, poorly built MDM systems cost business agility in a way that ripples throughout your enterprise. Most legacy MDM systems rely on a relational database which isn't optimized for traversing relationships or rapid responsiveness. These data connections and relationships in your master datasets are essential to competitive advantage as business analytics evolve. The good news is that graph databases are ideal for modeling, storing and querying the hierarchies, metadata and connections in your master data. With graph databases, your master data is much easier to model, costing you fewer resources (modelers, architects, DBAs and developers) than building a relational solution. In addition, with a graph database, you don’t have to migrate all of your master data into a single location. Graph relationships easily connect your siloed data between CRM systems, inventory systems, accounting and point-of-sale systems to provide a consistent vision of your enterprise data.
  

  Example: Employee Hierarchy Data

  In your master data, a hierarchy is any structure where nodes have other nodes above and below them, possibly with multiple branches. One example of a master data hierarchy is employee reporting and supervisory structures. A small hierarchy is easy enough to model and maintain in a relational database. But as soon as we model a much larger set of employees, both querying and maintaining the data gets more expensive. For example, if an employee gets a promotion, every relationship must be reset for every hierarchy in which the employee participates. Of course, such pure hierarchies rarely exist in the real world. Employees often report to a multiple people, and sometimes reporting relationships exist only for transitional reasons (such as job shadowing or coverage). In fact, most business hierarchies are actually networks lled with real-life complexities and many kinds of r elationships. Traditional hierarchies need to be reimagined as networks that are easier and more flexible to model with a graph database as business needs change. While the example discussed has to do with employee reporting relationships, the same principle of master data networks applies to product listings, document relationships and sales or customer data.
  

  Conclusion

  The best data-driven business decisions aren’t based on stale information silos. Instead, you need real-time master data with information about data relationships. Graph databases are built from the ground up to support data relationships. With more efficient modeling and querying, organizing your master data in a graph yields relevant answers faster and with more exibility than ever before.
  
  
• Use Case #4: Network and IT Operations
  By their nature, networks are graphs. Graph databases are, therefore, an excellent fit for modeling, storing and querying network and IT operational data no matter which side of the rewall your business is on – whether it’s a communications network or a data center.
  Today, graph databases are being successfully employed in the areas of telecommunications, network management, impact analysis, cloud platform management and data center and IT asset management. In all of these domains, graph databases store configuration information to alert operators in real time to potential shared failure modes in the infrastructure and to reduce problem analysis and resolution times from hours to seconds.
  

  Why Use a Graph Database for Network and IT Operations?

  As with master data, a graph database is used to bring together information from disparate inventory systems, providing a single view of the network and its consumers – from the smallest network element all the way to the applications, services and customers who use them. A graph representation of a network enables IT managers to catalog assets, visualize their deployment and identify the dependencies between the two. The graph’s connected structure enables network managers to conduct sophisticated impact analyses, answering questions like:
  
  • Which parts of the network – which applications, services, virtual machines, physical machines, data centers, routers, switches and ber – do particular customers depend on? (Top-down analysis)
  • Conversely, which applications and services, and ultimately, customers in the network will be affected if a particular network element – such as a router or switch – fails? (Bottom-up analysis)
  • Is there redundancy throughout the network for the most important customers?
  
  A graph database representation of the network can also be used to enrich operational intelligence based on event correlations. Whenever an event correlation engine (such as a Complex Event Processor) infers a complex event from a stream of low-level network events, it assesses the impact of that event against the graph model and triggers any necessary compensating or mitigating actions.
  

  Example: A Large European Telecom Provider

  To showcase the use of a graph database in the IT and network operations sector, here is an excerpt from an interview with a software consultant who helped implement a graph database solution for one of Europe's largest telecommunication providers.
  This telecom provider had a very large complex network with many silos and processes – including network management information spread across more than thirty systems. The large number of data sources was in part due to network complexity, and in part due to di erent business units, as well as organic growth through mergers and acquisitions. These different sources also created a very non-linear fabric that had to be modeled and understood from various dimensions. Prior to using a graph database, they had different network layers stored in different systems – for instance, one system might be dedicated to cell towers, another to ber cables and another devoted to information about consumers or enterprise customers. One of their business challenges was around maintenance and ensuring redundancy – they needed to know if they took a device down for maintenance, exactly who might be impacted and what the penalties might be, as well as what alternate routes might better mitigate the impact. [Implementing a graph database solution] was almost a dream business case because you could measure the benefit of the project as the telecommunications provider began to manage production-level changes that impacted its many actual customers. After implementation of the graph database model and the impact analysis queries, it was easy to extend the application to support single point of failure (SPOF) detection thanks to the flexibility of the graph model. Also, by providing an effectively unified cross-domain view, experts from different silos could work together for the first time and agree on a common domain terminology.

  Conclusion

  Discovering, capturing and making sense of complex interdependencies is central to e ectively running Network and IT operations are a critical part of running an enterprise. Whether it’s optimizing a network or application infrastructure or providing more e cient security-related access – these problems involve a complex set of physical and human interdependencies that are a challenge to manage. The relationships between network and infrastructure elements are rarely linear or purely hierarchical. Graph databases are designed to store that interconnected data, making it easy to translate network and IT data into actionable insights.
  
  
• Use Case #5: Identity & Access Management
  Identity and access management (IAM) solutions store information about parties (e.g., administrators, business units, end-users) and resources (e.g., files, shares, network devices, products, agreements), along with the rules governing access to those resources. IAM solutions apply these rules to determine who can or can’t access or manipulate a resource. Traditionally, identity and access management has been implemented either by using directory services or by building a custom solution inside an application’s backend. Hierarchical directory structures, however, can’t cope with the complex dependency structures found in multi-party distributed supply chains. Custom solutions that use non-graph databases to store identity and access data become slow and unresponsive as their datasets grow in size.

  Why Use a Graph Database for Storing Identity and Access Data?

  A graph database can store complex, densely connected access control structures spanning billions of parties and resources. Its richly and variably structured data model supports both hierarchical and non-hierarchical structures, while its extensible property model allows for capturing rich metadata regarding every element in the system. With a query engine that can traverse millions of relationships per second, graph database access lookups over large, complex structures execute in milliseconds not minutes or hours. As with network and IT operations, a graph database access control solution allows for both top-down and bottom-up queries:
  
  • Which resources – company structures, products, services, agreements and end users – can a particular administrator manage? (Top-down)
  • Given a particular resource, who can modify its access settings? (Bottom-up)
  • Which resource can an end-user access?
  Access control and authorization solutions powered by graph databases are particularly applicable in the areas of content management, federated authorization services, social networking preferences and software as a service (SaaS) offerings, where they realize minutes-to-milliseconds increases in performance over their relational database predecessors.
  

  Example: Telenor Norway

  Telenor Norway is an international communications services company. For several years, it has offered its largest business customers the ability to self-service their accounts. Using a browser-based application, administrators within each of these customer organizations can add and remove services on behalf of their employees.
  To ensure users and administrators see and change only those parts of the organization and the services they are entitled to manage, the application employs a complex identity and access management system which assigns privileges to millions of users across tens of millions of product and service instances. Due to performance and responsiveness issues, Telenor decided to replace its existing IAM system with a graph database solution. Their original system used a relational database, which used recursive JOINs to model complex organizational structures and product hierarchies. Because of the join-intensive model, their most important queries were unacceptably slow. In contrast, once they implemented a graph database solution, Telenor realized the performance, scalability and adaptiveness necessary for handling their identity and access management needs, reducing queries that once took many minutes to milliseconds.
  

  Conclusion

  For an enterprise organization, managing multiple changing roles, groups, products and authorizations is an increasingly complex task. Relational databases simply aren’t up to the task of managing your identity and access needs as queries are far too slow and unresponsive. Using a graph database, you seamlessly track all of your identity and access relationships real-time results, connecting your data along intuitive relationships. With an interconnected view of your data, you have better insights and controls than ever before.

Why are graph databases so cool?

blog info

Why are graph databases not so cool?

It is an interesting phenomenon that typing graph databases suck into your favorite search engine does NOT return any links that actually address that situation. What does this mean? You can find dissenting opinions on just about ANYTHING by typing the appropriate phrase into google… so what's up with that?

Why are graph databases not more popular than they are?

One reason is the fact that most users still have a relational mindset which means they are used to the RDBMS model and are slow to change to something that is new and requires deeper understanding and training. In addition, many management chains are hesitant to invest in a technology with which they are not familiar, so until these stakeholders understand the graph database model, there will be little investment in graph DBMS applications. A second reason is that there is not a completely standard language for programming them, like there is SQL for the relational model. this means that implementers must learn to use the appropriate API for whatever graph database they have decided to use. A third reason is that humans are largely linear in their thinking, and adapt most readily to the idea of a relational model because it is easier to understand than a graphical one. The linear relationships seem more natural, so it is foreign to people to have to think in terms of real relationships, a.k.a. graphs. Yet another reason is the idea that a single database is one size fits all, which [although it is rarely the case in the real world any longer] makes implementers more reticent to include graph databases alongside the RDBMS in a system. Finally, there is the problem of visual tools, which oracle and other have implemented to help DBA's with DBMS management tasks. Again, there are few standard tools for graph databases for viewing data in a graphical manner.

Introduction to Neo4j

Neo shares a number of core concepts with relational engines. For one thing, it is performs all the four main operations of any database, i.e., CRUD. Second, Neo is optimised for use with On Line Transaction Processing [OLTP] engines, so it is automatically optimized for transaction processing [and concurrency, of course]. Thus, it upholds the principles of transactional integrity and operational availability.

Neo is different from relational databases [which indreasingly becomes a misnomer as we discover what real relational databases are all about] in that the relationships are the primary focus, not the data items and their attributes, so the number of joins and foreign keys [and associated complexity] drops to zero. This focus also allows modeling that more closely matches the problem domain, since it is more life-like in its approach to the abstractions/models in most all domains.

What is Cypher?

Cypher is the query language that Neo4j uses. Cypher is a declarative language that looks at what to query instead of how to query. It is designed to make things easy for the user and the DBA, so it is relatively human-friendly. This helps users come up to speed quickly, and also facilitates spending less time building and optimizing the queries, which in SQL can occupy a significant amount of development time.

Here is a link to the Neo4j Cypher quick reference card so we can take a look at the language.

Some interesting comparisons

What other graph databases exist?

What is the difference – isn't a graph database a graph database?

Let's go take a look…

SCRUM: Project teams

Arteaga, Peters, & West Bahar & Garcia Bennett & Mikol Boyac, Li, & Lopez Braekman, Persily, & Wroblewski Crowther, Rajavasireddy, & Simmons Derham & Ochsner Filler & Zafiris Jay & Nguyen & Prochnow Kern & Lizarda & Moini Martin Flora & Namba & Santander Nguyen & Ruiz & Tollive Patterson & Richardson Raymundo & Swaminarayan Kang & Patterson Wiggins