If you like the idea of using D3 for data visualization but you want something a bit higher level, check out Observable Plot[1], a library by the D3 team that adds a lot of conventions and conveniences on top of D3, specifically for building plots, charts and graphs.
And even beyond that, there's Observable Framework[2], a static site generator for building visualization dashboards (that supports Plot, D3 and lots of other libraries).
I like d3 and have built a lot of things in it but every time I go back to it after not using it for a few years, I can never remember much, the docs just confuse me more, and I feel like I’m back at square one.
Whereas you could show me some partial differential equations to solve that I haven’t touched in 10 years and that somehow comes back quicker.
But I also think the d3 reference docs are absolutely horrible too.
I have used D3 extensively to the point of building integrations for AngularJS and Angular for it. I have always found it to be exceptionally unintuitive even if it is quite powerful. It is easy to get wrong, hard to debug and hard to grok.
Not saying this is an easy problem to solve either. If I needed to generate some very specific data driven graphics it would probably still be my go to.
Yes same, and I'm curious of what mental modal it uses as it does not register with me, who otherwise has no problem programming.
As far as I remember it is inspired by the Grammar of Graphics, as is ggplot2 in the R ecosystem: https://link.springer.com/book/10.1007/0-387-28695-0
For me it also always had this moment when it „clicked“ just to disappear after not using it for a couple of months.
Sounds like it's based on a good model for the problem space, but a model that's alien to other day-to-day ones.
Which is an interesting problem!
Is it better to optimally model ones problem space?
Or suboptimally model it, but create a model that's closer to developer/user expectations?
That is exactly it. A big part of the mental model is understanding the concept of tidy data from Hadley Wickham too.
I could not understand some of the design choices until I understood tidy data and not being much into R, I never ran across the idea previously.
Mike Bostock is so brilliant and has put in such an immense amount of work into d3 that if you even start trying to build your own data visualization javascript library you will end up leveraging d3 at some point as to not reinvent the wheel.
Used d3 on two different react projects. Charts weren’t the main feature but quite big still. When the dust settled I got a feeling React+SVG had made d3 obsolete - there’s some value in d3 zoom and interaction code, but their intricate system of data sources and auto updating views felt out of time. Maybe it was cutting edge stuff during jquery years, but modern FE frameworks do it better nowadays. Without the need to learn yet another api.
Again, I’m not an expert on d3, just someone who spent ~12-18months maintaining couple visualizations (among other things)
So happy to read this in a few different comments. I took a data-viz class in college and just felt stupid because d3 didn't click for me.
Which other library would you recommend?
I haven’t found anything else that comes close and that’s why I haze myself over and over.
What has worked best for me is to disregard d3 attr. Attr made sense a long time ago but now if using a framework it is just so much easier to work with SVG elements directly instead of so many attr just for svg attributes.
You still end up finding a million ways to use the d3 library even if you never use attr at all.
hereas you could show me some partial differential equations to solve that I haven’t touched in 10 years and that somehow comes back quicker.
The HCI (human calculation interface) for calculus has been lovingly polished for many, many generations. A quick read over Newtons original texts will quickly reveal that wasn't always so.
It seems unfair to hold an API to the standard, really.
Happens to me as well, with the exception of the equation thing.
I feel it is that way much less than other plotting libs.
Maybe 80% of what I do is applications, 10% systems and 10% data analysis. Often I will spend two weeks making Jupyter notebooks and then not look at it for two months.
I find most plotting libs have a lot of arbitrary things to remember, they usually have several nano DSLs (strings that get interpreted in ways more complex than atoi.). On top of that most plotting libs have something wrong from my point of view in various areas like deciding the range of the axes or handling huge numbers of points or something.
D3 makes you do a lot yourself but the interfaces it provides to do that are conceptually straightforward. Instead of using someone else’s bloated and buggy general purpose code though an APi that feels like assembling a ship in a bottle you can just do it right with D3.js.
I agree and I think that’s why I end up re-learning d3 over and over.
But having to do it yourself is a lot of work so you have to make an executive decision between shoehorning some plotting library that can’t do a chart feature that honestly should be supported versus building an entire chart from scratch.
I have had the same experience. As a tech lead these days I insist that nobody uses libraries that require a high level of domain knowledge. Basically code that is incomprehensible even for the person that wrote it should be avoided as hard as possible. D3 is a prime example of such a library.
I only did one bigger project with D3 and had the same experience! I was thinking about the same thing before I read your post. Not really sure why, but when I was doing it regularly it felt smooth, but now I can't remember anything.
I also have written a lot d3, between versions 2 and 7, and the refactoring that has happened meant a lot of examples online that were hard to comprehend were even harder to update.
I feel like its more stable now though. Something clicks for me since ive started writing it in more imperative style with svelte+d3 rather than d3 alone. The generated elements are easier for me to reason about, rather than otherwise relying on inspecting the generated elements with dev-tools after the generation.
This site was helpful to me, to combine d3 and svelte: https://svelte.recipes/
Even if you never end up using it professionally, I think it's worth it to learn D3. The core point that the library makes is that your projection of the data (say, a 1080p screen, 1920x1080) is always different from the dimensions of the data itself (say, grading from 0-100). At some point you need to decide how to scale the data (0-100) so that it draws correctly according to the projection (1920x1080). You can do the math yourself, or you can use a library like D3 to do the math for you.
Even if you don't draw data on the web, you'll probably eventually draw data somewhere, and you'll be glad to look for an equivalent in that ecosystem.
That's a good way of framing the core concept.
D3 was never intuitive for me. Maybe the problem is that it tries to do too much, at too many different level of abstraction? If it's just about projecting data, that's easy enough to understand. But I guess projection is also closely tied to axis and other pieces.
I found resources like Scott Murray's "Interactive Data Visualization for the Web" to be invaluable; the D3 docs are too dense/reference-focused for me, but his book introduced the concepts in a slower, more tutorial-like fashion.
One of the first things I ever programmed in PHP, of my own creation was a binomial distribution graph png generator.
Pretty sure I had to scale the rectangles I was drawing for the graph according to the dimensions of the png I was making.
It was of course very trivial, and nothing like the amazing things D3 can do. But it was a neat experience, making something of my own for very first time that produced an image on the computer.
The real problem with D3 is that, as a library, it doesn't really do visualizations. It gives you functions and exmaples to compose your own visualizations, whose key requirement is an expert-level understanding of JavaScript (particularly closures and functional programming). What mbostock says is the core "abstraction" of D3, the select and select diffing, is only an affordance for writing reentrant "paint()", with the flourish that your paint statements are expressed as builder statements, which themselves are scoped to select and can be broken up for various effects. Especially in animated examples, D3 does suprisingly little for you, as it's up to you to write the outer loop and track state over time. Nor does D3 give you any abstraction for encapsulating a visualization, as a function or otherwise. This is left as an exercise for the reader. All of this comes as a big surprise, or shock, to the new D3 user. It would be like adopting a new database and finding only a library for file IO and BTree indices and query optimizers, that you need to wire up yourself.
Well, D3 is not a set of tools for charts (alike matplotlib, ggplot or so), but a low-level library to create your data visualization from scratch. Depending on your goal, it may or may not be what you need.
Yes, and I think that's the correct expectation to set. I don't even disagree with its design choices - I like the composability of pure functions as much as the next guy.
Check out visx. It should really be called D4: Declarative Data Driven Documents. It swaps out d3-select (the DOM manipulation module of D3) in favor of React. That this is possible and works great is a testament to the great design of D3.
Can anyone recommend a good plotting library for Python, that is not matplotlib?
Vega/Vega-lite has a python interface called Altair.
I haven't used Altair but I wanted something higher level than D3 (which I've used but it feels fairly low level) so I went through the learning curve for Vega.js, and it could handle every usecase that I tried, some of which were pretty complex.
Seaborn isn’t too bad. It’s basically a convenience wrapper over top of matplotlib.
https://opensource.com/article/20/4/plot-data-python gives some common options. What kind of plots are you trying to achieve? Interactive? Jupyter notebooks? Reporting? SVG or HTML output? You might also like to look at things like https://evidence.dev
I've been in love with D3.js and have used it as the key technology for dozens of data visualization projects, both commercial and personal. However, its rendering syntax now feels dated, reminiscent of the jQuery era. While I still use various auxiliary functions (e.g., for ease or scaling), I now create actual components in a modern framework (in my case, Vue 3 and TypeScript).
Despite this shift, I am grateful to D3.js for the beautiful years it gave me and for teaching me various good practices in data visualization. The defaults and demos by Mike Bostock have been both an inspiration and an example of good standards, aligned with Tufte's minimalism.
I am very interested in this, do you have any examples for vue3 and typescript + d3 auxiliary functions? Thanks!
Well, see my current work in progress: https://github.com/stared/balfolk-viz/blob/main/src/componen... and D3 part in https://github.com/stared/balfolk-viz/blob/main/src/scripts/....
Learn D3 then (assuming you're doing something chart-like as most of us are) move to something like VisX or some non-React equivalent and never look back.
I view D3 and jQuery as being somewhat similar and both are very prone to spaghetti. Something like VisX adds a lot of structure over the D3 you're using to keep things sane.
I’ve been using D3 on and off for quite some time. I’m just starting now with react and visx definitely looks very promising as it bridges the gap nicely. Do you find it limiting though (like you have to fight the tool) to achieve something more custom vs. using plain D3?
VisX starts with low-level primitive components that match very closely to D3 primitives. It then builds some chart primitives on top of those and then builds some charts on top of that. You can build pretty much anything using the lowest-level primitives and still have a better developer experience than D3 IMO.
If you don't need any real customizability (and know you won't need it in the future), I'd stick with something with something less customizable, but faster to setup. If you know you need custom, take the 10-20% more time to setup and be able to change in the future as that'll be faster than porting your charts over.
I love the visualizations created but something about D3 syntax has just never clicked for me. It seems overly convoluted, verbose and from an old era. We've thankfully moved to a more declarative way of doing things.
D3 is from jQuery era so there are a lot of methods for manually manipulating the DOM like d3.select() and then selection.attr() etc. but those patterns tend to lead to components that are lengthy, hard to read and hard to maintain. Arguably a better approach is to group each D3 module based on whether it "modifies DOM" or "doesn't modify DOM". Then use React/Svelte to declaratively render the DOM and only use the D3 modules from the latter group for their math utilities.
This exists, it’s called visx, and it’s great.
D3 has an unnecessarily non-intuitive API design, and I think that kills it for a lot of people.
For example, why must a data join be performed like `selection.data(array).join(element-type)` instead of something like `selection.join(element-type, array)`. Looking from a pure API-design perspective, not getting bogged down in the technical details.
Lazy eval method chaining is a natural fit for applying+branching large amounts of HTML/CSS state updates but I feel that the designers took it a bit too far with the core D3 operations.
I don't think it was for the technical fit or performance reasons, but more a philosophy about everything starts with data, and graphics are just visualizations anchored to the data points (or a functionally derived property of the data points).
That also means the d3-* libraries compose really well, since the data is the common binding, and not some conceptual class or custom element.
D3 is inspired by the Grammar of Graphics by Wilkinson.
Another tool also inspired by GG is aptly named ggplot2 and for the R language. It offers a much more terse and functional inspired interface that also works well with an uncommon tool named Rush, R one-liners for the Shell, for quickly creating graphics from CSVs piped into the tool.
For a split second I hoped this one would be about Diablo 3.
I've heard great things about Vega [1], which sits on top of D3. It's a dependency of OpenSearch Dashboards, allowing users to create custom dashboards on log and observability data [2]. The vega library might alleviate some of the concerns others are expressing about the learnability of D3.
[1] https://vega.github.io/vega/docs/ [2] https://opensearch.org/docs/latest/dashboards/visualize/vega...
I keep searching for entirely not-web based diagramming/charting solutions. That can be driven via CLI or using some library, and output SVG or PNG.
All of these things usually depend on browser functionality or use some janky as fuck SSR solution using a headless browser to generate them server side.
Of particular note, I’d also like to be able to programmatically generate Venn diagrams. It seems there is, bizarrely, a complete dearth of anything in this space. A couple of JavaScript chart libraries support them, but generally speaking there seems to be hardly any support for them.
gnuplot seems good - just no Venn diagram support.
I just picked up D3 + React for a VS Code webview project. Pretty easy to pick up but I find jumping in with examples very black box (as someone who doesn't have a ton of web experience). That plus going from JS to TS had some quirks that I didn't realize when in JS land.
I could really use a "start from the basics" tutorial.
I will say Github Copilot has been excellent at helping me with random TS errors when porting my graphs from JS to TS. It's nice how typed D3 can be when you finally opt into it.
Related:
D3 in Depth - https://news.ycombinator.com/item?id=16844250 - April 2018 (52 comments)
D3 is a terrific toolkit. The fact that it is a low-level toolkit means you have to write more code, but on the other hand, you have more power. One of the negatives of D3 is that D3 charts are not responsive by default. Whether the chart fills the screen, or fills a small area in a dashboard, the grid lines are spaced the same and fonts are the same size. You can add responsiveness manually. It would be nice if a future version was responsive out of the box.
This was the only resource that made D3 make sense for me. The author is responsive and helpful too.
D3 is the first visualization toolkit that I really liked.
I really like the force directed graphs in the d3 examples but I've found that they often settle in poor configurations as the nodes and edges grow. I wonder if I need a 3d visualization at higher scale? Maybe someone else has experience with these.
Here's a recent example I built: https://alexsci.com/rss-blogroll-network/
Plot and Framework are locked into the Observable ecosystem, which has its own learning curve. Learning D3 offers the best flexibility and control.
This is like saying that a computer without an operating system offers the best flexibility, or that a pile of sand is more flexible than an assembled computer.
D3 is not an empty computer. It's a high level library built in a high level language that runs in a browser. You are very very very far away from "metal". Being able to do something in a few clicks or little typing is nice until you need to do something more complicated. Sometimes it's worth actually learning how something basically works. Sometimes it isn't, it's everybody's choice.
So D3 is not what I’d consider a high-level library. The point of it is to be able to do all kinds of stuff not supported by your average charting solution’s out of the box histograms and whatnot. It’s a great way to make more bespoke visualizations.
Of course sometimes you just want to make a damn histogram without reinventing the wheel which is where some of the niceties discussed above come in.
D3 runs in the browser and it manipulates DOM nodes, a very high level abstraction. You can accomplish something with it using SVG, which is again a high level abstraction. You can give it data and it will spit out numbers that you can use to draw something on the screen. You can bind that data to elements and it will keep track of what's entering and what's exiting. You can apply fancy transitions with a line of code! You can make it change an appearance of e.g. a graph by changing the interpolation method. And so on.
Sure you have to write code, but it does a lot of things in the background that would take you s lot longer to do yourself. High level is not just "here's a library on top of D3 that you just give data to and you're done, but then you have to cry for days because you need to do something it didn't think of - 80% of use cases". When I used it I didn't get it, but then I took some time, a couple days, to actually understand it and I got a lot more productive. It's a great feeling when instead of thinking in workarounds for a missing configuration option you can think in solutions for the actual task. A lot quicker and more rewarding.
I didn't do much computer graphics in my life, some mandatory university classes, but I'd say doing it on the simplest beginner level it was a much different experience.
Fair enough. I meant specifically in the context of data visualization libraries, having done a fair bit of that sort of work in my own career. It’s harder to make a basic chart in d3, compared to many out of the box solutions, but it’s possible to do pretty much whatever you want, thanks to (what I consider) the low-ish level of abstraction it provides. I.e. DOM manipulation vs direct specification of standard chart attributes.
I've used D3 in nodejs to generate a png. Is it really that different than using something like Python or Java to do the same?
I might have confused your point, but if your goal is to build charts for data analysis, or exploratory data analysis, then D3 is considered low-level for the purpose of getting to the analysis fast.
D3 is low level for doing what it does. Visualizing data.
It’s hard to think of a lower level abstraction for that purpose.
In what way are these free open-source tools “locked-in” to Observable? Observable Plot is a vanilla JavaScript library that is released under the ISC license and can be used with any style of web development (e.g., React, Svelte, whatever). And likewise Observable Framework is released under the ISC license and you’re able to self-host projects anywhere and develop locally?
Can you add the library as a html link.
Anything else would be unwise to invest time in learning.
Yes? Examples here: https://observablehq.com/plot/getting-started
Mike, thank you for D3. It’s had a huge impact on me personally.
D3 for me is feature complete. Has been that way for me for years (maybe since v3).
Never had a problem, never felt the need to upgrade. Still have a bunch of stuff that just works.
Thank you for your effort.
Plot isn't locked into the Observable ecosystem at all - it's a JavaScript library that you can use independently of everything else there.
I'll ignore the part where you're wrong about the lock-in, but to the rest of what you said...
Yes? I guess I didn't think it needed to be said so explicitly, but if you are opting into Plot and Framework, you are accepting convention over configuration as they say.
And for some situations, that is what you want.
If you value flexibility and control above all else, then I agree they may not be for you.
But if you value other things, like speed of iteration, or just getting some good data visualizations built without a deep investment in learning underlying concepts, then Plot and/or Framework have definite advantages over doing it all with D3.
Yeah, I can say from experience of D3 at least, it's very far from locked in. In fact I would ask what library you're thinking of could be less.
The community also has helped with some bizarre integrations I was trying to do, into libraries like highcharts for strange reasons etc.
These days I kinda want the opposite. Most of the time I know exactly what I want my output to look like. But frameworks inherently provide a large bag of presets and if you want something slightly different from the preset it's a pain in the ass.
For many types of plots I find it's actually kinda easier to just manipulate SVG data directly. It's not that hard!
I'd like to see a library/framework that focuses on making it easier to manipulate SVG or canvas output. It should be trivial to make tweaks. But the higher level a framework goes the harder it is to step off the golden path.
Why does d3 not work for you if you just want to manipulate the svg directly? You can use it to generate or update arbitrary xml as far as I know.
The higher level in this case is observable plot, not d3.
Yeah I hear you, if you know exactly what you want, Plot is probably not for you..
But I've found lots of value in using Plot to explore data, and quickly throw something together and look at it, and decide if it's a good way to present the data or not.
Although I will say that you can "drop down" to the SVG pretty easily with Plot. Lots of examples even on their site where they start with Plot marks and then heavily customize the SVG directly.
Have you tried p5js?
As much as I hate to even ask for but is there a Matlab like drag and drop interface for web visualization.
I try to avoid graphs with html5 canvas like the plague maybe a new way to design them would be nice.
Sounds like you want Tableau. It even comes with the same price tag as Matlab!
Does the website feel janky to anyone else ? I have routinely found content-rich webpages to be janky.
edit: nvm, turned on hardware acceleration. Smoother, but still jankier than I'd like on a Ryzen 9.
Observable Plot is amazing. It is very flexible and intuitive to use. I now prefer it over Plotly.js which I used to use before.
The docs are pretty OK, the only thing missing is a usable API reference section, so discoverability is not great.