Im surprised to see military traffic over the US. They tend to fly with ADSB out turned off and have an agreement with FAA to be able to do that. Also surprised to see gliders who generally don’t broadcast ADSB out. In the US, we are required to have ADSB out within most controlled airspace and within the mode C veil of a major airport (within 30nm and up to 10,000 MSL). So most GA planes have it but remote areas have planes without ADSB out.
https://adsb.lol provides this data licensed under ODbL, this site violates the attribution and share alike clauses
You can find the attribution and the link to the details at the bottom right corner of the main page. It links to the documentation: https://github.com/ClickHouse/adsb.exposed, which provides the full details. Additionally, you can read the license here: https://github.com/adsblol/globe_history_2023/blob/main/LICE...
https://opendatacommons.org/licenses/odbl/1-0/
4.3.a. is clear:
a. Example notice. The following text will satisfy notice under Section 4.3:
Contains information from DATABASE NAME, which is made available here under the Open Database License (ODbL).
At the time if writing, the attribution does not make it clear where to get the data and what the terms of the license of the data are.
Only a (data: adsb.lol) which does not even comply with copyright attribution (Which funnily enough you are complying with for OpenStreetMap, while only using that for display, while creating your derivative database of the globe_history_2023 and globe_history_2024 database...)
I find it super frustrating to see well funded tech companies boldly abuse open data resources and assume they can just get away with it. I guess that's the theme of 2024. But to make it clear: you are in violation of the license the data is provided under, and as one of the copyright holders I object to that.
Could you please clarify the details?
I took extra care to provide all the needed attributions and credits and I believe it is complete and sufficient. If I missed something, please describe it at https://github.com/ClickHouse/adsb.exposed/issues, and I will correct it.
please read the license and its requirements. It is YOUR responsibility to comply with those requirements in order to be allowed to use the data. It is ridiculous that a well funded company asks a volunteer community to tell them how to comply with the license - after they have violated the license and used it for a marketing stunt.
You might be frustrated and that's fine, personally I don't know what's missing either but you simply saying "YPU need to figure it out" isn't helpful at all.
Well, here's the language in the license that apparently is too hard for you to find:
4.3 Notice for using output (Contents). Creating and Using a Produced Work does not require the notice in Section 4.2. However, if you Publicly Use a Produced Work, You must include a notice associated with the Produced Work reasonably calculated to make any Person that uses, views, accesses, interacts with, or is otherwise exposed to the Produced Work aware that Content was obtained from the Database, Derivative Database, or the Database as part of a Collective Database, and that it is available under this License.
I can't read it for you, but I can summarize it for you. You are required to make sure that someone who uses the product (i.e., Clickhouse's marketing stunt thingy) becomes aware of the license and origin of the underlying data. And not by digging into some GitHub repo, but right there, on the page.
I don't think anyone is questioning the correctness of what you are saying, just the aggressive tone and assumption of malice over mistake. Maybe take this as an opportunity to educate licensees rather than ridicule them.
Their banner on the bottom links to adsb.lol and adsbexchange.com, is your specific concern that they don’t have the odbl license called out? I personally didn’t have much trouble figuring out where the data for this project came from based on their banner alone, and thought it was honorable of them to publish their process for obtaining the data.
I have no horse in this race, but am really confused by this aggressive reaction to what I perceive to be a good-faith use of this data. Is this the prelude to some scheme by which you plan to extract money from ClickHouse? The grievance in these replies is genuinely unclear to me.
Licenses get abused for one of two reasons, more or less: 1) the license is unenforceable; 2) no one enforces the license.
In some cases, this is only one reason.
This is some copyleft troll-level pedantry [0]. They clearly made a good effort to comply with the license. Additionally, ADSB.lol requires contributors to license their contributions under CC0. Databases that don't have some sort of creative work involved in their compilation aren't copyrightable, so it's very dubious that anyone could enforce any sort of restrictive license over ADSB.lol's database as a whole when its individual contributions are CC0.
[0]: https://pluralistic.net/2022/01/24/a-bug-in-early-creative-c...
Not at all. It is the polite request to please correctly attribute the data that a company is using for their marketing stunts.
I love the fact that techbros being called out for violating other people's intellectual property immediately revert to "TROLL!" or "this is too hard!" instead of actually engaging with the question at hand. And btw - until you actually create the software stack to collect the data and run an aggregator, don't condescend on people who do the hard work that you seem to feel free to copy in violation of said license.
No one said complying was too hard. This is one of the coolest things anyone's done with your data, they made a good faith effort to comply, and you've done nothing but act like a jerk and project your issues all over this thread. If I ran an ADS-B aggregator that was based on the same software as a half-dozen others, which hadn't seen any significant innovation in years, sitting on tons of data with absolutely amazing potential, I might consider praising the person who just revolutionized the world of ADS-B analysis & visualization, and possibly begging them to help me. And "P.S., could you throw in [specific attribution text]."
Hi, thanks for sharing! Unfortunately besides empty map I don't see any visualization. Browser dev console is showing some CORS errors.
EDIT: Nevermind, clickhouse.com was on disconnect's ad blocking list which I used for DNS blocking.
We bought clickhouse.com from the previous owner 2.5 years ago, who used it for some ad network - it was in some block lists, but I hope we managed to clean up most of them.
Still blocked in Disconnect list due to “Malvertising”
I'm not sure that list should be used any more. I tracked down a pihole issue a while ago to find this "The list you referenced was used in our legacy products. It is not maintained, has not been updated, and is not actively distributed us."
https://www.reddit.com/r/pihole/comments/wtizpa/deprecation_... (ETA: https://github.com/disconnectme/disconnect-tracking-protecti...)
Interesting, thanks for the heads up, just had it removed.
It seems to be all working on my end though.
fully empty map on iOS safari for me.
This is beautiful! Definitely worth looking at the examples in the Github repo:
https://github.com/ClickHouse/adsb.exposed/
I particularly like the example of helicopters following the river Thames in London:
https://github.com/ClickHouse/adsb.exposed/?tab=readme-ov-fi...
Impressive. Though, for one of the examples, set in an area with active military conflict, I wondered whether the information is new and useful to anyone involved.
Nah. This data is all publicly available already and easily accessible for any motivated nation state. Who, if they're in the area, would also be able to use their own receivers and radars in any case.
Military planes that show up on ADS-B do so for either of the two reasons (or both):
a) to make civilian air traffic control life easier
b) as (strategic) signalling - this applies for example to NATO tankers and electronic intelligence platforms flying over the Eastern Flank.
Remember, you want your opponent to know how you can potentially hurt them if things go bad, that's the basis of deterrence. This is why for example certain aspects of nuclear weapon systems are completely out in the public.
Yes, though there's also another reason:
c) Safety, being open about the flightpaths of tankers and intelligence-gathering aircraft flying in international airspace near those areas reduces the ability of Russia to claim they weren't informed or that the aircraft strayed into restricted airspace. That means a lower probability of incidents, though it's still not completely safe as demonstrated in 2022: https://www.bbc.com/news/uk-66798508
Is it possible to export a report of traffic by a selected region?
Yes, it is possible to connect to the database directly and run an arbitrary query. Example:
$ clickhouse-client --user website --host kvzqttvc2n.eu-west-1.aws.clickhouse-staging.com
clickhouse-cloud :) SELECT t, desc, count() AS c FROM planes_mercator_sample100 GROUP BY ALL ORDER BY c DESC LIMIT 10
┌─t────┬─desc────────────────────────────┬────────c─┐
1. │ B738 │ BOEING 737-800 │ 51530781 │
2. │ A320 │ AIRBUS A-320 │ 37196762 │
3. │ C172 │ CESSNA 172 Skyhawk │ 20049393 │
4. │ A321 │ AIRBUS A-321 │ 19983151 │
5. │ A20N │ AIRBUS A-320neo │ 14938832 │
6. │ B38M │ BOEING 737 MAX 8 │ 14200826 │
7. │ B737 │ BOEING 737-700 │ 13929403 │
8. │ A319 │ AIRBUS A-319 │ 13906164 │
9. │ E75L │ EMBRAER ERJ-170-200 (long wing) │ 12006441 │
10. │ A21N │ AIRBUS A-321neo │ 10965047 │
└──────┴─────────────────────────────────┴──────────┘
To obtain an SQL query for a particular region, you can open the browser dev tools (F12), switch to Network, and copy a particular request that is made when you select an area with the rectangle selection tool.
Ah very nice, thank you kind internet friend.
The one glider path over the LAX area is interesting.
What was going on there?
I think what you see is CIVIL AIR PATROL and CIVIL AIR PATROL INC
Types: GLID (SGS 2-33A)
Flights: N7589, N2037T
Registration: N7589, N2037T
When you select the dottet square button in the lower left and select a rectangle the planes within this rectangle are listed
Very interesting project!
Oh, thanks. I somehow missed that UI.
The glider I was thinking about is actually N914SF, a Pipistrel Sinus. That's a motorized glider, which makes a lot more sense flying though Class D airspace, right above LAX.
Sometimes I feel lucky that human's electromagnetic spectrum perception is very limited to the so called "visible spectrum".
Imaging alien species that evolved to perceive a wider electromagnetic spectrum, the earth must look like a disco ball when their spaceship approaches.
Much like we're lucky to have eyes that happen to see the sun's strongest frequencies?
This is really cool when you dig into how much fidelity there is here. Also, a fantastic marketing campaign for Clickhouse!
As for the dataset - is this continually updated or how "fresh" is it at any given moment?
There are two data sources: adsb.lol and adsbexchange. The first is updated each day from https://github.com/adsblol/globe_history_2024. The second should be updated each month (they provide only sample data for the first day of each month), but I still have to put it in cron.
The update scripts are also open-source, published here: https://github.com/ClickHouse/adsb.exposed/blob/main/prepare...
How do it get ADS-B data from over the oceans? I guess it isn't satellite (which I now is used for actual tracking) since there is obviously a lot less lines over the oceans (but lines nonetheless).
There are satellite ADS-B receivers and some of that data makes its way to public aggregators. https://www.icao.int/SAM/Documents/2014-SAMIG13/02%20NavCana...
Wow, this is beautiful!
I predict this will get a hug of death $soon
I keep the dashboard open :) So far ok with ~2000 QPS.
The image loading/rendering is fascinating, beginning grainy and then smoothing with time. Can you elaborate on what’s happening?
The readme contains a lot of the implementation details: "We use three different tables with different levels of detail: planes_mercator contains 100% of the data, planes_mercator_sample10 contains 10% of the data, and planes_mercator_sample100 contains 1% of the data. The loading starts with a 1% sample to provide instant response even while rendering the whole world. After loading the first level of detail, it continues to the next level of 10%, and then it continues with 100% of the data. This gives a nice effect of progressive loading."
This is fantastic!
I'm evidently the only person who has no idea what in the world this is or where the data comes from other than clearly being some flight/vehicle data
HN has such a unique group of people
https://en.wikipedia.org/wiki/Automatic_Dependent_Surveillan...
The "strange hole near Mexico City" example in the Github repo's README is another volcano.
This is fantastic! Is the airplanes.live related to ADS-B exchange or is this an unrelated effort?
This is arrestingly beautiful. I kind of want to get a print or two of some of these. Even the color choices are magnificent
Wow. I have no idea what I'm looking at but it's pretty. And I do recognise the extended centrelines of local runways, as well as the typical instrument approach tracks into them!
Edit: Having read the readme, I have a better idea of what I'm looking at. Really impressive technically as well.
It's like writing webgl shaders for ADS-B. In SQL.
Incredible work, and I hope this kicks off a lot of innovation in the world of aircraft traffic analysis & visualization, which I think has been kind of stuck in a rut for a while.
Super cool! Visually dazzling.
If you want to build something similar on a raspberry pi, here is a tutorial: https://questdb.io/blog/create-flight-radar-raspberry-pi-que...
tiles by aggregation on a pixel level
Not sure I'm understanding this.
Is a "tile" then 1x1 pixels in size? If so, does the server maintain a pixel-addressable cache of the 1x1 tiles?
The detail is amazing, you can see down the the individual tiedown rows at many small airports.
@Author: the link has huge pictures shown small, pre-shrinking them would be good. There's 85MB for 36 images, a lot of bandwidth wasted.
Very pretty! As it happens, I too have started ingesting ADS-B Data into ClickHouse recently, but have nothing nearly as beautiful as this.
I'm hitting the airplanes.live API every 10 seconds using ClickHouse's URL table function and storing in a MergeTree: https://github.com/JosephRedfern/airhoover/blob/main/airhoov.... Would love to use refreshable materialised views for this, but at the moment there's no append functionality (refresh only), so have to use Python to to trigger the query.
There's an open instance here: https://airhoover.joesstuff.co.uk/play?user=default#U0VMRUNU.... Only 1.5 days or so of data, I truncated before setting up tiered storage (local disk + backblaze b2).
Cool being able to e.g. get a break-down of aircraft type by operator (https://airhoover.joesstuff.co.uk/play?user=default#U0VMRUNU...) or who (supposedly) flies their planes fastest (https://airhoover.joesstuff.co.uk/play?user=default#U0VMRUNU...)
Fighters often don't have ADS-B, but there are lots of other military aircraft that routinely use it (transports, etc.). Go to https://globe.adsbexchange.com and press U to show only military.
I'd read about small retro reflectors they bolt on when not in combat so the stealth fighters will show on domestic radars
Edit: they're called Luneburg Lenses
My town, north of Dayton, OH was recently overflown by a group of fighters (I believe F-18's). I was surprised that they didn't appear on the ADS-B tracker site I ran inside and looked at. I guess it makes sense.
I'm just a couple of miles from DAY and see a lot of traffic every day. It would be interesting to know how military aircraft like that coordinate with civilian air traffic control.
(I like to listen to Dayton approach while watching an ADS-B tracker site. I enjoy seeing the traffic fly over my neighborhood. I find it oddly amusing to look up at a plane I just heard getting clearance to land knowing that I just heard the voice of somebody up there thru my speakers. I don't know why it's so pleasing...)
I tracked, briefly, a F15E flying past yesterday.
I'm surprised about this, I thought it's pretty easy to add ADS-B to modern IFF module and they can just switch between different modes.
Just speaking from experience, in Colorado no military has it on except for the trainers. Having a chinook pass under you and be totally invisible on the iPad is a weird feeling.
For general transport/rebasing, it seems like it would just make life easier on air traffic controllers, and probably also other traffic in the air (I think TCAS uses ADS-B?). By way of analogy, the military might have permission to drive around at night with their lights off, but it'd make me more comfortable if they did that only when it was operationally useful (i.e. I assume B-2s taking off from Missouri to go bombing have transponders off the whole way.)
One of the "tells" for military aircraft flying without ADSB is when you see refueling tankers (KC-135) doing loops for periods of time, and no other military aircraft around. Those tankers are refueling something, they don't just fly around for grins.
Military aircraft operators in the USA are pretty cautions about non-ADS-B Out or non-transponder operations. Air Force bases etc. will have active MACA (Mid-Air Collision Avoidance) programs, with aircraft and controllers working to reduce conflict potential with civilian traffic. Many bases will have MACA information on their web sites and staff to contact, and they are typically very responsive. Ironically the ones I have worked with were trying to encourage more civilian GA aircraft to adopt transponders, and to utilize the flight following services of their ATC/RAPCON.
TCAS uses active transponder interrogation from the TCAS unit interrogating a threat target's Mode-C or Mode-S transponder. TCAS only uses ADS-B In in an indirect way, in large part to acquire traffic in the area that is not yet a threat, and reduce it's RF congestion caused by excessive interrogation, especially of legacy Mode-C targets. A TCAS II system will fly you right into say a UAT out equipped aircraft without issuing an RA (resolution advisory) if that threat aircraft has no transponder or an inop transponder.
TCAS II only issuing an RA based on active interrogation of a threat aircraft's transponder is a kind of safety feature given the potential spoofing of ADS-B Out data.
Sometimes they forget to turn it off:
https://theaviationist.com/2023/11/22/usaf-ac-130j-iraq/
That article states that it's unlikely that they forgot, and much more likely that it was deliberate.
Interesting, when it happened at the time the theory was they forgot.
If you look at https://globe.adsbexchange.com, and filter for military/interesting (the U button at the top of the screen), you'll see that at any given time there are a LOT of military flights over the US. Most are transport, refueling tankers and what seem to be pilots-in-training (mostly in Texas and Florida, some in Colorado), but there are plenty of helicopters and smaller Lear-type jets. When the president or vice president is flying somewhere there are typically one or more E-3 AWACS planes in the air to provide radar coverage.
There are certainly military planes that fly without ADSB, but for flights where secrecy doesn't matter, they seem to fly with it on. I've seen all manner of planes with ADSB, from U2 spy planes, F-15, F-16, A-10, the occasional B-52, and more.
Cool. Whenever I see military traffic here in Colorado they don’t show up on adsb except for the DA20 trainers in the pattern south of Springs.
There's plenty of H60s and H47s out of Lewis McChord on ADSB, not to mention P8's, C17's, etc...if you're on flight following you'll get advisories or even just watching "the fish finder" you'll see them.
Those Reaper drones show up on FlightRadar24 and ADSBExchange while patrolling over the Black Sea monitoring Crimea and beyond.
I started playing around with ADS-B and military aircraft are all over the place, at least on the east coast of the USA.
In fact, this Seahawk just flew over: https://globe.airplanes.live/?icao=ae6904
The ADS-B Out situation with gliders in the USA is complex. Many high-performance modern gliders used for racing and cross country flying are equipped with Mode-S transponders with 1090ES Out (either 14 CFR 91.227 ADS-B Out compliant or TABS/TSO-C199 ADS-B Out compliant) as well as the glider specific FLARM traffic awareness system that broadcasts and receives proprietary low-power position data. FLARM systems, at least as used in the USA also typically receive 1090ES In so they drive the integrated traffic display and warning systems in these modern glider cockpits. For historical and technical reasons gliders use a FLARM modified NMEA serial data protocols for traffic data in the cockpit instead of what is found in GA aircraft ADS-B traffic display systems.
Gliders potentially face very different risk scenarios and their owners will hopefully equip for what is the most significant risks to them and others. e.g. gliders often fly close to each other, especially when thermalling together, if flying with other gliders in remote areas then the FLARM system optimized to handle glider on glider threats is optimal (where GA focused ADS-B produces far to many false alerts), if flying near lots of GA aircraft ADS-B Out (and In via FLARM) is likely optimal, if flying near airliners, fast jets and tactical military aircraft then transponders alone even without 1090ES Out may be most critical item for their SSR, TCAS and IFF compatibility. Ideally owners do equip with all three... Mode-S, 1090ES Out, and FLARM.
The challenge may be more where there are low-cost/low-value gliders, maybe trainers and glider club owned gliders especially those located in busy traffic areas. You would hope those owners long ago got the message they should be equipping with some forms of supplementary traffic broadcast/awareness systems.
UAT adoption in gliders is nearly non-existent as FLARM systems only receive directly on 1090ES In and you don't want to rely on ADS-R coverage in remote or mountainous areas.
TABS/TSO-C199 is an easier route to an approved installation of ADS-B Out systems in type-certified gliders. TSO-C199 was developed by industry and the FAA following the 2006 mid-air collision between a glider and Hawker business jet near Minden NV. Experimental category gliders will typically have 14 CFR 91.227 compliant installations done under the same "meets performance requirements" clauses as many experimental power aircraft.