Monday, 20 February 2017

Coders just rescued NASA’s Earth science data

Credit: Jamie Lyons/Wired

Signs are that Donald Trump hasn’t revised his attitude on climate change,1 & he’s stacked his cabinet with deniers & those hostile to the environment.2

With this in mind, a group of coders took action to make NASA’s climate data safe, & track future deletions

Wired – “Diehard coders just rescued NASA’s Earth science data”

1 Australian Broadcasting Corporation: “Badlands National Park deletes tweets on climate change statistics, following Donald Trump gag order”,

2 The Atlantic: “The Donald Trump Cabinet Tracker”,
/ coders rescued NASA’s Earth science data coders took action to make NASA’s climate data safe UC Berkeley’s campus 200 adults Doe Library rescue federal climate data groups more than 20 cities believe Trump administration might want to disappear this data down a memory hole hackers scientists students collecting save outside government servers groups like DataRefuge Environmental Data and Governance Initiative organized Berkeley hackathon collect data from NASA’s earth sciences programs Department of Energy doing more than archiving diehard coders building robust systems monitor ongoing changes to government websites keeping track of what’s been removed learn exactly when the pruning began Tag It Bag It data collection is methodical group immediately sets web crawlers easily-copied government pages sending text Internet Archive digital library hundreds billions snapshots webpages tag more data-intensive projects pages lots links databases interactive graphics for the other group called baggers coders write custom scripts scrape complicated data sets sprawling patched together federal websites systems written piecemeal over 30 years no coherent philosophy providing data websites Daniel Roesler chief technology officer UtilityAPI volunteer guides Berkeley bagger group coder who goes by Tek download multi-satellite precipitation data NASA’s Goddard Space Flight Center starting August access Goddard Earth Science Data required login totally legal digging around site DataRefuge prohibits hacking Tek found buried link old FTP server clicked started downloading data all of 2016 some 2015 non-coders hit dead-ends too 404 Page not found errors NASA’s Earth Observing System website empty databases Global Change Data Center reports archive NASA’s atmospheric CO2 datasets real problem when or why data disappeared backed it up scientists understand it better DataRefuge EDGI monitoring changes and deletions building software that can do it automatically Future Farming most advanced software builders tools they’ll need filters separate mundane updates major shake-ups explored blockchain-like systems build auditable ledgers of alterations engineers call version control how do you know if something has changed How do you know if you have the latest How do you keep track of the old stuff writing code volunteers signed build tools DataRefuge EDGI organizers movement vast decentralized network from all 50 states and Canada volunteers code tracking software archive a little bit every day group had collectively loaded 8,404 NASA DOE webpages Internet Archive covering the entirety of NASA’s earth science efforts built backdoors download 25 gigabytes from 101 public datasets scripts larger datasets finished running mood was somber still so much work to do climate change data just the tip of the iceberg Eric Kansa anthropologist manages archaeological data archiving non-profit group Open Context huge number of other datasets being threatened with cultural historical sociological information National Parks Service huge data portal contains park visitation stats GIS boundaries inventories of species computer ran scripts pull out a list everything in the portal working his way through each quirky dataset /