Visualizing large data sets with wijmo enterprise webinar

Editor's Notes

• #3: Welcome everyone, thank you all for attending the Visualizing Large Data Sets with Wijmo Enterprise webcast. This session will be recorded and put on our website wijmo.com
• #4: My Name is Ross Dederer. I work in developer relations here at Grapecity, specifically for the Wijmo product. Before I joined Developer Relations, I worked on the support team for one of our reporting solutions so I have a background in reporting as well . I am joined by Chris Bannon the Product Manager for Wijmo. Chris will be joining us today to answer any questions you guys have during the presentation. Chris gave a webcast on Inside Wijmo5, earlier in the month, I encourage you to check it out if you haven’t already. His video like this one will be available on Wijmo.com
• #5: So this webcast will be about 45 minutes long, during that time, I have some slides and some demos online to share with you all. Whatever time is left at the end I will reserve for discussion or Q&A. To begin, let me outline what we are going to be covering << this afternoon>>. First I want to introduce Wijmo Enterprise and give a brief history of how the product got where it is today. This is mainly for those who have never seen Wijmo before or maybe you have but its been a few months since you looked at it and you aren’t familiar with some of our new offerings. As always. Check the website regularly for updates and additional content like this webcast. After I have introduced Wijmo Enterprise, I want to spent some time talking about where web development is today and what areas I have seen developers focusing on and some of the challenges they are to facing. In my discussions, I have found that it comes down to the fact that data is getting larger and more complex, performance is playing a more important role, and the adaptation of the ‘mobile web’ which is creating this responsive design buzz around the web world.. In order to introduce FlexGrid and FlexChart, I am going to need to introduce everyone to our CollectionView which is our powerful MVVM data layer. Forthly, I want to talk about how we can visually optimize a chart for displaying large complex data. Wijmo Enterprise handles this with our FlexChart control. Lastly, I want to show you how you can optimize our FlexGrid control to display data in a meaningful way that is user friendly. For each of these controls, I will introduce it, spend a few slides talking about some relevant features when it comes to creating performant, responsive, data visualization controls, and then show how you can use these features to greatly enhance the behavior and usability of your data driven web-application powered by Wijmo Enterprise.
• #6: So What is Wijmo Enterprise? Let me introduce the flagship product in WE, Wijmo5.
• #7: Collection of JavaScript User Interface controls. When we talk about performance, loading as little as possible at every step is key and Wijmo 5 is great because it has zero dependencies unlike w3. Grapecity and ComponentOne have been focusing on Data Grids and Charts for over 25 years across almost all of the platforms. Included in the offering of Wijmo5 is the FlexGrid if you have used it before. A few years ago Microsoft was including it as a control in visual studio in the VB days. Focus on Performance from day1, focusing on small, lightweight, extensible controls, which I will be demoing. So what makes wijmo5 so unique and what is the biggest difference between Wijmo3 and Wijmo5? Well, it’s the way you interact with the controls. Lets discuss it. DEMO: www.wijmo.com
• #8: In 2011 ECMAScript5 was release. Talking about ECMASCRIPT is the topic for an entirely different talk but let me point out the major differences and the most important features that it provides us. Go over the code on the screen Advantages of Controls -> Re usability etc
• #9: Wijmo 5 is modular. The core is required, everything else is optional. Modules are small (less than 100k minified). The core contains general-purpose classes such as Event and CollectionView and globalization. FlexGrid is our data grid, with the same object model as the XAML version. FlexChart is our chart component, with support for multiple data types. Input contains controls such as ComboBox, InputDate, InputTime, InputNumber, etc. All controls are MVVM-compatible with object modes similar to their XAML counterparts. All this makes creating HTML5 applications as easy as creating Silverlight applications. And it also makes porting Silverlight applications much easier. Now that we have an understanding of Wijmo’s aritecture, lets talk about web development today. www.Wijmo.com
• #11: Before we can begin talking about how to visualize data is important to understand a few things about data. In most web applications, the data object boils down to two different types Quantitative vs Qualitative data: Quantitative data - Numerical data that shows the who, what, when and where. Qualitative data - Non-numerical data that demonstrates the why or how. For those of us who have used Google Analytics, it provides a lot of qualitative data such has who has come to your site, how they got there, and what they did when they did get there. What it doesn’t tell you is why? Well, we can use things like bounce rate and other things to answer those questions but what challenges in general does representing data have: Displaying meaningful results as fast as possible Dealing with outliers Changing the interactivity based on device As well as some other optimization such as grouping ---- So what do we do with the data once we get it? We need to visualize it somehow
• #12: Data Visualization is defined as the presentation of data in a graphical format. This is great because humans are NOT the best at looking at RAW data and seeing the “bigger picture”, no pun intended. When I say RAW I am referring to a JSON string or something like that. We visualize this generally using Grids and Charts Visualizing data isn’t always the complete solution, interacting with the data is equally as important as visualizing itI will introduce some features of the controls that really allow to make our data very interactive. . If you want to take your application to the next level, you can add some analytics to your grids and charts as well, which I will show. This is an importance concept because you can put the power in the hands of the end user and don’t need a developer to hand every use case. SO WE have talked a bit about data, and visualizing it. Lets talk about performance.
• #13: In the web, performance is a feature of a great application. We all know and understand how important performance is. We all can agree that the speed at which your application performs is never fast enough.. Performance and bounce rate have a lot in common. Bouncing rate is an important factor that every webmaster should take into account. Bounce Rate is the percentage of visitors to a particular website who navigate away from the site after viewing only one page. Basically, in terms of UI design, if someone looks at your homepage on a mobile device and leaves right away, you probably need to do some work on your mobile layout and design, which I will be getting into more with Grids. So why is performance so important. Lets talk about some research by Greg Linden, who currently works for google and is a self described internet relic, and has over 32 years experience . Greg did a presentation on Big Data and shared his significance of site speed in terms of revenue. In his work at amazon, Linden talked about how a 100 milisecond delay could lead to revenues of the online company dropping by 1%. Last year alone Amazons estimated reveunue was 74.5billion so that’s about 750 million dollars a year. Bing did some testing to see what would happen if their sites responded 1.5 seconds slower. They noticed a 5% drop in User Satisfaction, Revenue per user, and number of clicks. They also tested it at 500ms and noticed a 1% drop. A huge indicator of how performant a website is is not only with how long it takes to load data but MORE importantly is how it takes your end users to access that data. Of course, how long it takes for users to find relevant data depends greatly on the layout of the web application, so lets now talk about Screen Sizes and layouts before we talk about how to make these controls responsive. Source: https://developers.google.com/web/fundamentals/performance/ Case Studies http://sixrevisions.com/web-development/why-website-speed-is-important/
• #14: Lets take a look at Web Development today with regards to device screen sizes. Today, we are generally dealing with mobile, tablet, and desktop web applications. Due to the growing number of mobile devices out there, the old idea of have two separate websites for mobile and desktop is getting too expensive and difficult to maintain. Many Web App Developers are choosing to write one application that responsive. That is, that looks and feels natural no matter that size or input method from the user. This is due to the idea that in a few years there will potentially be many more challenges for website layouts. Responsive design can prove itself a significant advantage in the market in terms of multi-device functionality, making future fixes easy, and appealing to a much widest audience of users Today, designers are using layout configurations to move div elements around the page but I want to take this concept and add an additional layer to it, and that is making the controls themselves responsive and not just their height and width. Lets take a look at two examples that demonstrate what I am explaining.
• #15: Responsive layout is not enough and he is why it is not enough
• #19: You get CollectionView with Wijmo 5. The wijmo 5 data binding interface is the CollectionView. It is a collection of several classes that separate the visual aspect of the controls into a data layer. To start using the collectionView start by declaring it and passing a regular array as a data source. iCollectionView: Enables collections to have the functionalities of current record management, custom sorting, filtering, and grouping. It provides a powerful MVVM model. Its kind of hard to describe this In words so lets just look at an online demo DEMO http://demos.componentone.com/wijmo/5/PureJS/CollectionViewIntro/CollectionViewIntro/
• #20: FlexChart- included in Wijmo 5, many extensibility points, including Financial Charrts
• #21: The FlexChart is included in Wijmo5, This is the base class for our charts. It contains one flexible API that is defined by a chart type at either the control level or the series level if you are working with composite charts. In the next few slides we will discuss how you can utilize the features such as stacking series on top of one another to address one or all of our main concerns about programming in the web space today. The Chart can be dynamic as well. As I said earlier, the flexchart is the base class and one of the key features Wijmo Enterprise offers in its extensibility points. Well in our 2015v2 release we released a few extensibility points for the chart and I would like show one of those next, its called the Financial Chart DEMO http://demos.componentone.com/wijmo/5/PureJS/FlexChartIntro/FlexChartIntro/
• #22: The Finance chart is very industry specific but since its an extension of the FlexChart, its very easy to pick up and learn. It is included in our Wijmo Enterprise offering. LIST CHART TYPES IMHO, the hardest part is understanding the differences between all of the different types. Luckily. We have plenty of blogs on the site, from entry level 101 too more advanced concepts. For todays discussion we will keep it on the 101 to mid tier side. I mentioned we added a few extensibility points, well another one is the Analytics module which I will discuss in depth in a few slides. Other Features Markers Range Selector Trend Lines
• #23: Stacking series can be very useful when display a wide range of information. When data is stacked like it is in the picture above, it is much easier for the human eye to understand the differences between the two. This is important because a good graphical display is pleasing to the eye and therefore can have the effect of impressing the viewer and encourage them to read more of a page or site. Speaking of keeping people on your site once they get there, its important that they can find their information very quickly, as we discussed before. So, If a mobile user visits your website, we all understand that screen space cannot be wasted. How do we fix this? Perhaps excluding data points or series is not an option. By using the idea of Composite charts which we have pictured to below, you can load multiple charts for a desktop user in the view port but as the screen sizes scale down, you can begin transfer charts into series and loading all series into one chart. You could use this in a layout shifter and actually combine elements at the control level using Wijmo APIs.
• #24: So here is a graphical display of the concept that I explained on the last slide. Note that we removed the chart title and added some legends. Since we are talking about screen space lets talk about Outliers in Datasets.
• #25: Lets imagine a scenario where we are plotting large datasets that may have outliers. Lets take a basic bubble chart and improve it by changing the axes scales. Lets assume we are starting with a dataset containing population, and per-capita income for about 200 countries. Once we have our dataset, we can easily calculate GDP and add a datapoint. So once we have our dataset, lets add a flexchart to it, we start by adding a dataset and the chart type, the name of the property for the x axis. So if we run this we should see the following chart. So it looks like the chart rendered correctly but is it very clear?. Well, as we can see there are some outliers here that make the dataset almost un readable. The only thing you can truly gather from this data set is that there’s a few countries that are way outside the norm. Also, The chart would be clearer if it showed GDP in billions and population in millions. So no, the chart isn’t very readable. So how do we make this chart more readable. Well, as we talked about before. The solution is going to be found in the Wijmo base class under globalization. Basically, this involves changing the format property for the axes. Wijmo’s Globalization class supports scale modifiers in format strings.
• #26: We can see our X Axis is much more readable now. We wont do anything with the Y Axis at the moment as we want to convert that to a logarithmic scale. Why? Well, we still have outliers however, ( US CHINA AND INDIA ). Many types of data sets have this type of pattern including economic and demographic data where the data tends to cluster around the average and outliers then force the bulk of the data to bunch. There are two simple ways to address this so lets discuss
• #27: We can set Min and Max properties to exclude outliers, which would mean that the cluster gets spread over a larger portion of the chart thus making it more readable. Two problems in our example: You would be leaving out the US, and that isn’t cool, but more generally, we would be deliberately excluding the largest GDP and population countries. Using this algorithm would exclude the initial cluster but that would end up revealing more clusters and we would keep sacrificing accuracy for clarity. A better approach in this case is to change the axes to a logarithmic scale. With logarithmic axes, the distance between tick marks is a multiple of the log base, rather than a constant value. Because of this, logarithmic axes naturally compress the scale as values grow towards the maximum. To enable logarithmic scale in the FlexChart, we set the logBase property on the Y axis to a value such as 2, 5, or 10. Once we do this, each tick mark represents a distance 2, 5, or 10 times larger than the previous one. Lets take a look at how this will render
• #29: So we have discussed some cool properties and features of the FlexChart that you as a developer can implement to really maximize the value and screen space of your charts. One of the nice things about the chart, is that when it comes analyzing data, you can enable certain features that lets your users take control. For example, your end users can do analysis, add annotations, and drill down through data all without needed a programmer. So lets talk about who we can use some of the modules for analysis and visualizing qualitative data.
• #30: In my experience in charting, trendlines tend to come up in discussions a lot, probably for a good reason. So its important that we spend some time talking about it. The Analytics module for FlexChart is made available via Wijmo.chart.analytics Trend Lines are used to represent trends in data to examine the problems of prediction. Currently the FlexChart supports all of the trendlines that Microsoft Excel Supports This is very powerful because it tries to normalize your dataset. It can also provide your end users will a really quick way to analyze a data set. Live Demo http://demos.componentone.com/wijmo/5/Angular/FlexChartAnalytics/FlexChartAnalytics/#/trendline In addition to adding additional series, we can add an annotations layer.
• #31: So this is basically the idea of taking other data sets and adding another interactive layer on top of the chart surface that help with you data analysis. This idea is best demoed Live Demo http://demos.componentone.com/wijmo/5/Angular/FlexChartAnnotation/FlexChartAnnotation/ http://demos.componentone.com/wijmo/5/Angular/EditableAnnotationLayer/EditableAnnotationLayer/
• #32: By adding additional layers or elements to our page we were able to add more complexity to your chart. Lets now shift into displaying less data but in a more meaningful way. Like Grouping in Grids, charts have this concept of Drilling down, which allows you to present data in digestible chucks. This is widely used in reporting. We can take this a step future and add toolips and breadcrumbs to really help give the user context. Live Demo http://demos.componentone.com/wijmo/5/Angular/FlexChartGroup/FlexChartGroup/
• #33: Live Demo – Weather Chart - http://demos.componentone.com/wijmo/5/PureJS/WeatherChart/WeatherChart/ Live Demo – Stock Chart http://demos.componentone.com/wijmo/5/Angular/StockChart/StockChart/
• #34: Included in Wijmo is our Flagship control, the FlexGrid.
• #35: Grids and Charts are a key part to data visualization. Charts are great for taking a large amount of data points and ‘truncating’ all of those data points down to a single 400x400px chart. Grids allow us to present the entire dataset to the user and have them do operations on that dataset. Our end users should be able to sort, edit possibly, filter data, very easily Wijmo Enterprise utilizes some tricks that will greatly increase your application performance scrolling The key to all grids and a performant application is a solid data layer. We have already covered the collectionVIew Demo: http://demos.componentone.com/wijmo/5/PureJS/FlexGridIntro/FlexGridIntro/
• #36: Grouping data or Grouped Data is a statistical term used in data analysis. The theory is that you take a Raw dataset and create a separate table that consists of the frequencies of any given item. In our examples we will be grouping data based on some field in our collectionview and instead of displaying the frequency, we will simply display the underlying data. Not only is grouping important from a data viz standpoint but also from a responsive design angle. FlexGrid provides a few ways of doing this: The group panel control allows you to add a drag and drop grouping UI to any FlexGrid Control. This is included in an additional module so all we need to do is include wijmo.grid.grouppanel.js, add an element to hold the grouping ui and create the control when the page loads. The other option is to do grouping via a dialog which would be more mobile friendly Live Demo http://demos.componentone.com/wijmo/5/Angular/GroupPanel/GroupPanel/ http://demos.componentone.com/wijmo/5/Angular/SortingGroupingUI/SortingGroupingUI/
• #38: Once we have a data layer in place, the first thing we do is visualize the entire dataset in a grid. Once we do that, we realize that we need to scroll around to see all of the information. By default, all scrolling in the FlexGrid is virtualized. Only the DOM elements needed to render the cells visible in the Viewport are created. Of course we buffer it a bit so there it appears smooth. Live Demo SHOWCASE MOBILE SCROLLING http://demos.componentone.com/wijmo/5/Angular/Benchmark/Benchmark/#/flexgrid
• #39: Searching - Searching and filtering a similar. When you do a search, you obviously know what you are looking for. The trick is to have one single input box that searches the entire dataset Filtering can be important if you are looking for something in particular but your not sure exactly what. For example, you know something happened over a given date range Lets look at the three ways you can filter data on the FlexGrid. The image on the top is the excel-like dialog. This is opened when a user clicks on a icon in the header row of a column and you have your traditional Excel Like filter box with combo box options The picture below is a input box directly in the column headers in additional to a combo box. This might not help the desktop users out to much but this layout is great when working on larger touch only devices. The user can click in the input box and get a keyboard open or select an option from the combo box. This is easier because you don’t need to try and touch that little icon in the column header but its worth noting that you can increase the size of the hit box. This second method seems more fluid though on mobile. The third option is just a text box that filters that data on a predefined field. Not you can have a combox box above it to select when field you want to filter on and then have a text box that will actually fitler some data. In additional to filtering and freezing and following the same concepts of data grouping lets talk about the DetailRow extension. http://demos.componentone.com/wijmo/5/Angular/Explorer/Explorer/#/grid/intro Live Demo http://demos.componentone.com/wijmo/5/Angular/FlexGridFilter/FlexGridFilter/ http://demos.componentone.com/wijmo/5/Angular/HeaderFilters/HeaderFilters/ http://demos.componentone.com/wijmo/5/Angular/FlexGridIntro/FlexGridIntro/
• #40: The detail rows are implemented using the FlexGridDetailProvider class. The FlexGridDetailProvider class has a createDetailCell function that creates HTML elements to show the details for any row. The content of the detail cells is completely customizable. In addition to the createDetailCell function, the FlexGridDetailProvider class has other properties that allow you to customize when the detail rows are displayed, their dimensions, etc. Very similar concept to grouping its just the data can behave in a different manor
• #41: Redo this screen shot
• #42: http://demos.componentone.com/wijmo/5/Angular/RowDetails/RowDetails/