The CAVE - a system for managing a clubs survey data
Posted: 15 Jul 2022 12:11
Hi all
This is an example of how I setup a system for my caving club to archive and manage its cave survey data. Go to "The CAVE" here: https://suss.caves.org.au/cave/ The CAVE stands for "Cave Archive and Versioning Experiment". It is basically a simple static website front end to a Mercurial repository. Total lines of code is about 1200.
All of our raw data consists of scanned survey sheets and that data is entered as Survex format files. These are checked into a users local Mercurial repository clone of that cave area. The user then pushes up the changes to our Mercurial repo at suss.caves.org.au. They use EasyMercurial (https://easyhg.org) as it has a very simple and relatively intuitive interface. Each night a cron jobs runs some bash and python programs. They check out the repo into a clean working directory, process the contents of the working directory into static HTML pages, and finally copy those pages to the website directory.
One of the important features of this system is that each directory of the cave data contains a LibreOffice spreadsheet called "metadata.ods". Those contain the metadata for every file such as filename, description, authors, copyright owner and license. That's the data that is displayed in the tables in the web pages. If users don't enter any one of the mandatory metadata fields then they can still push to the repo but the web page will display a warning that there are metadata errors that need to be attended to.
Of course the other important feature is that the data is in a distributed version control system so there will be multiple copies around, and the data is not in any proprietary archive system. If this centralised repo is no longer maintained or breaks cavers will have local copies.
If I may say so, I think a particularly nice feature is the logo
It's an SVG image and it took more effort to get working nicely than to code the main application. The logo shows a laser representing a Disto. The faint background numbers represent station from & to, bearing, elevation and distance. That second line of data is the actual elevation of the laser! Reload the page and the data line and the angle of the laser will change.
At present I'm converting the code to work with Python 3 instead of Python 2 and I will put up the code on https://sourcehut.org with an Open Source license in the next month or so.
Mike Lake
This is an example of how I setup a system for my caving club to archive and manage its cave survey data. Go to "The CAVE" here: https://suss.caves.org.au/cave/ The CAVE stands for "Cave Archive and Versioning Experiment". It is basically a simple static website front end to a Mercurial repository. Total lines of code is about 1200.
All of our raw data consists of scanned survey sheets and that data is entered as Survex format files. These are checked into a users local Mercurial repository clone of that cave area. The user then pushes up the changes to our Mercurial repo at suss.caves.org.au. They use EasyMercurial (https://easyhg.org) as it has a very simple and relatively intuitive interface. Each night a cron jobs runs some bash and python programs. They check out the repo into a clean working directory, process the contents of the working directory into static HTML pages, and finally copy those pages to the website directory.
One of the important features of this system is that each directory of the cave data contains a LibreOffice spreadsheet called "metadata.ods". Those contain the metadata for every file such as filename, description, authors, copyright owner and license. That's the data that is displayed in the tables in the web pages. If users don't enter any one of the mandatory metadata fields then they can still push to the repo but the web page will display a warning that there are metadata errors that need to be attended to.
Of course the other important feature is that the data is in a distributed version control system so there will be multiple copies around, and the data is not in any proprietary archive system. If this centralised repo is no longer maintained or breaks cavers will have local copies.
If I may say so, I think a particularly nice feature is the logo
It's an SVG image and it took more effort to get working nicely than to code the main application. The logo shows a laser representing a Disto. The faint background numbers represent station from & to, bearing, elevation and distance. That second line of data is the actual elevation of the laser! Reload the page and the data line and the angle of the laser will change. At present I'm converting the code to work with Python 3 instead of Python 2 and I will put up the code on https://sourcehut.org with an Open Source license in the next month or so.
Mike Lake