Data Modeling with Neo4j: “School Immunization in California” CSV to Graph

1 state, over 9 million children, and 42,981 rows of CSV immunization data. After many rough drafts, I was finally able to land on an efficient and aesthetically pleasing way to map out the immunization data of children in California (found and downloaded online from the California Department of

Education*).In this post our goal is to walk through the data modeling process to show how this CSV data can be connected meaningfully with Neo4j. What makes this data so interesting is its varying degrees of location, three distinct grade levels, and a dense record of immunization numbers and percentages-all spanning over two separate school years.

After successfully mapping the data, I could then easily explore it, answering questions such as: Where in California has the lowest amount of children vaccinated?, Are less parents vaccinating their children in 2015 compared to 2014?, and Which age group is more up to date on its vaccinations?. Furthermore, I was able to clearly visualize the data in small and large quantities using the neo4j graph.

* The public school dataset can be found at http://www.cde.ca.gov/ds/si/ds/pubschls.asp . The private school dataset can be found at http://www.cde.ca.gov/ds/si/ps/ .

Preparing the Files

Before anything else, we have to modify the CSVs by:

• Changing the column names: capitalize only the first letter of each word and leave the rest of the word in lower case.
• Replacing # with Number, % with Percent, 1 with One, + with Plus, and taking out any dashes and parentheses. More simply, spell out the symbols and numbers found in the headers.

Mapping the Data: Determining the Relationships Between Locations and Schools

Before getting into the code, I took pen to paper by simply drawing out the nodes and their relationships to 

Read More......

Data Modeling with Neo4j: “Chemicals in Cosmetics” Step-by-Step Process

Take this unique dataset in a CSV format and transform it into a graph using Neo4j. Using the Neo4j model, we can compact the vast number of relationships and properties within the Chemicals in Cosmetics dataset, creating more meaningful and easily applicable data.

Meet the Data

Since 2005, all Californian cosmetic companies are required to provide the information of any cosmetic product that contains chemical(s) that cause or are suspected to cause cancer, developmental birth defects, or harm to the reproductive system. This list, of the cosmetics and chemicals in question, is openly provided on the California government website*. Even more intriguing, are the numerous properties about the products and chemicals, such as important dates and times, whether the product is still being sold, whether the chemical is still being used, and much more. The interconnectedness of this data illustrates the power of the property graph model and its ability to succinctly store information by prioritizing both the nodes and the relationships.

Below is a list of the more ambiguous/allusive headers found in the CSV. Going through and understanding the headers is a vital step for developing an accurate graph model.Headers & Descriptions:1. CDPHId (#): CA Dpt. of Public Health identification number for product. May occur more than once.
2. CSFId (#): CDPH identification number for CSF.
3. CSF: Color, scent, and/or flavor. Not all products have specific colors, scents, or flavors.
4. CompanyId (#): CDPH internal identification number for company.
5. PrimaryCategoryId (#): CDPH identification number for category.
6. PrimaryCategory: Type of product (13 primary categories).
7. SubCategoryId (#): CDPH internal identification number for subcategory.
8. SubCategory: Type of product within one of the 13 primary categories.
9. CASId (#): CDPH identification number for

Read More......

Neo4j for Your Modern Data Architecture

We recently sat down with Neo Technology, the company behind Neo4j, the world’s leading graph database, to talk more about our role as a trusted Neo4j solution partner and to dive deeper into our Neo4j Enterprise offerings.

Talk to me about GraphGrid. What’s your story?

So to understand GraphGrid, let’s dive into a little back story: We co-founded AtomRain nearly seven years ago with the vision to create an elite engineering team capable of providing enterprises worldwide with real business value by solving their most complex business challenges. As we figured out what that looked like practically, we found ourselves moving deeper down the technology stack into the services and data layer where we handled all the heavy lifting necessary to integrate data sources and provide the functionality, performance and scale needed to deliver powerful enterprise service APIs.

In early 2012, we had our first exposure to Neo4j and experienced first hand the potential of

Read More......

MySQL to Neo4J

You’ve probably heard that an effective way to take move data from an existing relational database to graph is using LOAD CSV. But what exactly does the process of converting all or part of the database tables from MySQL to Neo4j using LOAD CSV involve start to finish? We’ll be using Mysql5 Northwind database as our example. There is a Neo4j tutorial that has a similar explanation using Postgres and discusses the graph modeling aspects as well. So definitely good to read through that. Here we’ll focus on MySQL and the CSV export in preparation for the Neo4j import.

the

First we’ll install and connect to the MySQL database:

1 2	$ brew install mysql $ mysql.server restart

*Note: We’re skipping all MySQL server security because for this demonstration its simply an intermediary to get the data we need for the Neo4j LOAD CSV process.

Now using freely available MySQL Workbench or Sequel Pro connect to your localhost MySQL server. You should be able to do this directly on 127.0.0.1 without any username or password because we skipped the normal process of securing the server.

Import the Northwind.MySQL5.sql that you downloaded above. If you’re using Sequel Pro, you do this by choosing File -> Import… -> browse to your download and select Northwind.MySQL5.sql
When the import is finished you’ll see all the tables available for export to Neo4j. The specific tables we are interested in for our Neo4j graph model are Categories, Customers, Order Details, Orders, Products and Suppliers.

Export each table with right + click and selecting Export -> As csv file.
Customize the CSV file with settings that import smoothly into Neo4j (most should be selected by default):
1. NULL fields should export as a blank because it’s more efficient validate an actual existence or IS NULL check rather than actually creating the property with the literal string value “NULL” as value.
2. Escape values such as quotes with \ so quotes in the middle of the field do not break the

Read More......

Modeling Time Series Data with Neo4j

I’ve been receiving many questions recently at trainings and meetups regarding how to effectively model time series data with use cases ranging from hour level precision to microsecond level “How granular do I really need to make this to efficiently work with and expose the time-based data being analyzed?” and “Do I need to generate all time nodes down to the desired precision level?” The balance that needs to be considered is the initialization and maintainability of all the time series nodes versus the dynamic creation as time series events require their existence and the impact the missing nodes may or may not have when querying time series events by various date and time ranges.

precision. In assessing the various approaches possible, I landed on a tree structure as the model that best fit the problem. The two key questions I found myself asking as I went through the process of building the time tree to connect the time series events were,

Originating the Time Tree

I ultimately decided that it would be most effective to create the hour, minute and second level nodes only when needed to connect an event into a day. So I expanded on the work done by Mark Needham in his post Neo4j: Cypher – Creating a time tree down to the day. The main modeling change in this step was to use a single CONTAINS relationship going from the higher tree toward the lower tree level to simplify movement up and down the entire tree through the use of depth-based pattern matching. Additionally I concluded that Read More......

Graph Advantage: Personal Gamification

User attention spans online are quite short today with all the options vying for their eyeballs. Gasification is an approach that surfaced as being an effective way to engage with users in a more engaging and interesting manner. Gamification in a non-generic, personal manner with the right incentives for each individual requires a very complete and connected understanding of that user.

Defining Gamification

Gamification involves integrating gaming mechanics into marketing and user interaction strategies. It’s a method that drives consumer engagement and participation since it produces positive behavior via incentives and rewards. As a matter of fact, this form of “gaming” can drive recurring user engagement and positive perception, as long as the approach is based on the right incentives.

Benefits of Gamification with Neo4j

Gamification is becoming more widely seen in today’s user strategy and is on the mind of those driving product and marketing decisions. Getting gamification right can transform the way businesses deal with their customers. Businesses are developing these game mechanics for their target users by means of rankings and customizations to get them to work for customers. The mechanics apply to situations that help promote customer loyalty, motivate buyers to continue making purchases, and provide attractive incentives to maintain their interest.

Involving not only the customer and your products, but also introducing rewards, challenges, and purpose as part of the interaction introduces a completely new and dynamic set of complexity with a very real-time and responsive nature to it. All these components working together in real-time can engage the user in higher and continued levels of participation.

When it comes to querying connected data in real-time, there is no database better than Neo4j. As a native graph database, Neo4j is designed for graph traversals in constant time, which means many data types can 

Read More......

Graph Advantage: User Personalization

User personalization is intended to tailor each individual’s experience to them and really provide a more human element to the interaction. Providing this aspect of feeling known and understood rather than just being a generic set of eyes can go a long way towards more fulfilling and continued engagement with your users.

Digital retail is an major space where personalization surfaces because it’s challenging to find the right balance and approach for the interaction. By concentrating on the online retail journey of your customer and making it a personalized experience, you and your customers both benefit from the more personal and meaningful interaction.

Personalization through Digitizing Retail Knowledge

Many strategies for personalization involve long-running offline batch processes that take a considerable amount of time to complete in order to consider changes to what the system understands about me as an individual user. This delay in response is a major barrier to a personalized engagement with your user. The amount of data to be considered in total is quite astounding for the large established retail chains. However even with all this data there is still the possibility to provide a real-time user personalization experience for your customers. The information that is relevant for enhancing the online experience of each individual customer is quite small, although complex from a data connectedness perspective.

Customers today have become quite willing to share personal details in exchange for improved shopping offers and experience. By using these details in a meaningful way you’ll be demonstrating an understanding of your customers and showing how you can utilize such knowledge to enhance their experience, they’ll be more likely to continue to lean into the experience by providing even more insights into their preferences.

A lot of online retail websites offer a plethora of navigation trees, but people have been trained to make use 

Read More......