Habitat maps

Missing data

As mentioned last week, I'm working towards doing a regional habitat map based on global data, to see if we can utilise Tessera's global reach to make up for local data-deficiencies. But in doing this I ended up doing a review what data we were using from GBIF, and found a couple of interesting points.

The major one was that despite GBIF containing species occurrence data from many different sources, including Cornell's eBird repository which is often cited in the space I'm working and indeed is used by the Lumbierres et al habitat crosswalk work the major work I'm building upon, the way I was filtering GBIF meant that the eBird data was being dropped. This is quite major as it turns out 83% of occurrences in Madagascar where we have major data-deficiency issues are in eBird.

Digging in, this happened because the eBird data doesn't contain any GPS accuracy information, and we specify that we want occurrences with high degrees of GPS accuracy given Tessera is built at 10m per pixel. eBird has it's own classification for location accuracy, and not only does that not map to the fields I was filtering on in GBIF, the values it does use are not in GBIF at all. So I had to request access to the eBird research database. This good 4 days to get approval, and so has blocked things somewhat - I don't see the point going up to a global run before I've worked in the eBird data given how significant it is.

Whilst waiting I did do some other analysis of the data I had, looking at how GPS accuracy itself impacted how much data I've got. This is interesting in part because the habitat maps I'm generating are often used at significantly lower resolution than Tessera's 10m per pixel and so might be able to benefit from more vague data, and in part due to a Part II Project I'm supervising (more on that below). Looking at the four regions I've been generating maps for this is how many occurrences we get based on our current filter including occurrences from 2020-2030:

Note the 'All records' will include eBird data, so things will change here. What's interesting to me is that going from 10m to 100m has much less of a jump that I'd expected, it picking up at 1km. the other thing that it was interesting to note was that although we add more records, it doesn't necessarily equate to more location coverage:

That's not to say the data isn't useful, but it's also not the magic bullet one might assume it would be in the data-sparse areas. I'll be interesting to see how this map changes once we add the eBird data.

I did some other cuts of the data too: rather than just trying to move to a global set of labels contributing to the map of a region, I took a look at how that would work with an expanding buffer, to try keep the labels somewhat local (e.g., 500km, 1000km, 2000km). Mostly this doesn't look good for my normal two test areas, UK and Madagascar, as they're surrounded by water, particularly Madagascar. Going by latitude band instead does help, but mostly I suspect because for Madagascar that pulls in northern Australia, where you once again get into people have more free time and mobile phone access to contribute to things like iNaturalist. Nothing conclusive as a result, but it's interesting to start wondering about these things, as pure global runs will be quite expensive in compute.

More robust analysis of maps

Whilst I was in eBird data limbo, I invested time in improving how I validate our map runs. Like with any iterative research project, when I started I did something minimal so that I had some way of getting a sense of how well I was doing without spending a lot of time on it when we didn't know if the project has legs. However, I also knew that as we grew what we were doing wasn't good enough: in Madagascar and Minas Gerais we were outscoring the related work despite being poorer visually, driven by the scarcity of occurrences somewhat, but also by poor method on my behalf.

My high-school maths teacher opened up the final year with the statement that she didn't like probability maths, and so we'd be skipping that module, and whilst I agree with their reasoning, it has meant I'm always slightly on the back foot with statistics and analysis, and so I had to play catch up as I upgraded the evaluation for what is building a statistical product. I know understand k-fold cross validation (also random-cross validation), and more about spatial cross-validation (which I shared a bunch of papers Aneesh Naik had pointed me at last week - the opening of this paper makes for a good review of why you'd want either method).

By the end of the week I implemented the k-fold random-CV, and it started to highlight the weaknesses of my approach in data-deprived areas against the other metrics, which is good, as the results it gave before were just clearly biased on the samples I was using. One particularly silly flaw was that as the pipeline evolved to add more filtering of points, the separation of training data was happening downstream of those, which is clearly bad practice also. Amusingly though, fixing this caused all validation scores to drop - because my filtering of GBIF data was actually cleaning up errors in the original data - so all methods performed better if you do that. Still, I was benefitting more because this influenced my training data - so it's good to get all this fixed.

The spatial-CV I'm currently still waiting on to run for the regions of interest. The challenge with spatial-CV is that the point density, despite the thinning work I did last week, is still very un-uniform across regions, and so I can't just do a straight regular grid for this, and I'm evaluation the CAST R package to automatically generate cells with equal occurrence density.

Wood pigeons are a worse case scenario

As a fun point to wrap all this up for these weeknotes, I did discover that, in the UK at least, the Wood Pigeon is empirically (using our new new random-CV testing) the enemy of my software :)

So what's going on here? Well, if you look at the IUCN Red List habitat data for the Wood Pigeon, it lists many kinds of artificial habitat, but not urban. However, in the UK I think I've mostly only ever seen them in urban environments - we had a pair that nested in our back garden for many years. From this I'd argue that they're misclassified. This is then compounded by most occurrences being recorded by people in urban areas. Our pipeline then throws all that data away, as it tries to clean up the data, and isn't something the other methods do, so they score much higher than we do. It's a sort of perfect storm of our various datasources.

One thing that helps Lumbierres et al here is that they allow multiple mappings from the Copernicus Land Cover Map they use as their base layer to IUCN habitat classes, and indeed map "built-up" to both Urban and Rural Garden. In theory, this is wrong based on the definition of rural garden being a small farmsted according to the IUCN guidelines, but it does reflect the data, and so I think leads to a better habitat map that captures the IUCN habitat classes as used rather than as defined. This is something I'm keen to capture, and now I realise my OSM filtering stages might need to be slightly more forgiving, as I've based them on following the IUCN habitat class definitions quite closely.

Part II Students

I mentioned last week that I have one final year undergrad project to supervise this coming academic year, but now it looks like I'll have a second one! The first one is looking at improving dynamic resource adaptation in my declarative geospatial library Yirgacheffe, and the second is working with the aforementioned GPS accuracy issues, a topic suggested by Aneesh when the student in question expressed an interest in doing something related to Tessera, and it suits me as it's a topic clearly close to what I'm working on.

For readers not in Cambridge: the undergrad course here is three years: part I-A (one A - aka first year), part I-B (one B - aka second year), and part II (two - aka third year). As someone who was never a student here this still causes me to pause to think every time I bump into this. My understanding is that it's a historical quirk that has never been updated, and I just felt that given I'm going to be saying "part II projects" a lot in the coming months I should just get this out the way.

This coming week

• Finish of the spatial-CV integration
• Review the impact of the eBird data on data-deprived areas like Madagascar and Minas Gerais before we move on to global labels.
• Make sure the part-II project students are unblocked.

Every year, the second Thursday of October is International RSE Day!

Today, we celebrate our work and contributions to research, and we share widely our passion for our job. Many in the Nordics and all over the world are celebrating in different ways, from the RSE community in Helsinki meeting for a community lunch, to blog posts, to simple hashtags. Some of us are celebrating simply by enjoying a good cup of coffee together in our weekly coffee break taking place every Thursday at 09:00 CE(S)T / 10:00 EE(S)T: everybody is welcome!

Here is to us! Happy International RSE Day!

Habitat maps

My plan was to do a semi-global run this week, but to get there I ended up with a stack of small steps towards that which instead took up the time.

OSM downloads to Overpass

Up to now I've been accessing OpenStreetMap (OSM) data by doing region specific downloads from Geofabrik. Obviously I could just scale this up and download ALL TEH REGIONS!, but instead I turned to another option. Colleague Anil Madhavapeddy has been setting up a local OSM mirror with Overpass server so we can query the Overpass API to our hearts content without hitting limits on the public version and/or annoying people.

The main interesting thing here for me was I'd not considered that the country specific downloads actually miss out overlapping/enclosing polygons that are not entirely within the download area. For my use case, where I'm trying to find very local things (forests, parks, etc.) this won't make a huge difference, particularly as I've mostly been focussed on island nations (UK, Madagascar), but if I had done this for say Austria, then there's a chance I'd have lost items that crossed the border. With access to a full global dataset I get those.

Spatial density reduction

One concern I've had for a while is about data density being non-uniform across the globe. That is to say, in a single year of GBIF observations I get more species occurrence data in the UK than I do in Madagascar in ten years. This is also true in OSM data, whereby the UK has high coverage and Madagascar has very little. There's quite a lot written in academic literature about the perils of such data biases, and so before scaling to a global run I wanted to look into this. It also relates to performance, as my current implementation of the map generation is O(n^2), so thinning out the data has some other benefits.

To this end I read a bunch of papers around this problem, including some recommended by colleague Aneesh Naik who's been looking into similar things for his own habitat map and Species Distribution Modelling (SDM) work. Some papers what I read (or at least skimmed):

• The Effects of Sampling Bias and Model Complexity on the Predictive Performance of MaxEnt Species Distribution Models by Syfert, Smith, Coomes, 2013
• Correcting sample selection bias in maximum entropy density estimation by Dudík, Schapire, and Phillips, 2005
• Sample selection bias and presence-only distribution models: implications for background and pseudo-absence data by Phillips, Dudík, Elith, Graham, Lehmann, Leathwick, and Ferrier, 2009
• Environmental filters reduce the effects of sampling bias and improve predictions of ecological niche models by Varela, Anderson, García-Valdés, and Fernández-González, 2014
• Spatial filtering to reduce sampling bias can improve the performance of ecological niche models by Boria, Olson, Goodman, and Anderson, 2014

The tl;dr here is that the papers covered three approaches,

• Synthesising data statistically to fill in the gaps
• Using environmental data to help thin out samples with the same context
• Using sample density based on location to thin data

For this application, where we're already using the Tessera foundation model to fill in gaps, I feel that's already doing some level of statistical gap filling, I don't need to do it twice. The environmental approach from Varela et al 2014 I quite liked, but I don't currently have the data to drive that. So in the end I went with plain old geospatial filtering based on Boria et al 2014. That's not to say that I've picked the best one, just that I don't see this as the biggest risk factor in the success or failure of this work, so I felt just picking one and moving on was a good start. The main thing I got from the papers was that which one performs best is subject to the application (Varela et al found their application was worse for just plain geographic filtering rather than environment filtering), so to do this properly you need to actually try them and work out what's good for your application - which is another reason to start simple and iterate.

However, I'm not just doing a naive transplant of the Boria et al method. Whilst I am dividing occurrences into an even sized grid, and then thinning per grid cell, I'm trying to be a bit more smart than that:

1. Within each cell I bucket by habitat of the occurrence, and keep one sample of each habitat, so we don't lose habitat resolution this way.
2. To thin down the values within a cell I use the Tessera embedding value to pick the one with the average smallest distance to all other embeddings

That last step could definitely do with improvement, but again, not the biggest risk in my pipeline right now, so it gets added to the todo list to revisit if I ever get a global map that I'm somewhat happy with. The idea is we're picking the most representative value, but I think if there were more hours in the day I'd actually take a histogram of all samples, throw away outliers (as incorrect input - i.e., bad data in GBIF), and take the embedding values that are in the histogram peaks and keep those (as there might be young and old trees for example).

So, definitely more work to be done, but I had bigger problems to de-risk (see next section for short term, and longer term it's can I get something end-to-end that isn't hot garbage before I improve on it incrementally).

However, one last note on this is that having ran this on both UK and Madagascar with 100 metre cells, I actually saw samples drop off in Madagascar more than in the UK, despite the UK having two orders of magnitude more data! This I believe is because in Madagascar you're seeing most occurrences happen in tourist hot spots, so you're just getting repeated samples from the same location. Basically the data is less well distributed in Madagascar than it is in the UK, which makes total sense (to me). So this is a win, but not quite the win I'd naively hoped it would be at the start, and I still have a disproportionately larger set of samples in the UK than in Madagascar.

The thinning has had an impact on the habitat map of the UK, in a bemusing way: more bogland around Lewis, Sutherland and Caithness. Looking at the map this seems okay - though I'd need to track down a local expert to comment on it^W^W^W^W^W^W^W^Wgo there on my motorbike and check, you know, for science.

Dealing with performance issues vs my own confidence

One recurring challenge in this work has been the variable performance of accessing the Tessera data via source.coop using the geotessera Python package. For those not familiar with the context here: Tessera is a global machine learning model that operates at a resolution of 10 metres per pixel and covering many years, all of which means there is a lot of storage required, more than we have at our disposal in our research group, despite having some quite significant storage servers. The solution to this is that the full global storage set is stored on S3, and we access it that way. Between the University's fast network links, source.coop using a CDN, and local caching, in theory accessing Tessera like this should be close to what we'd get from one of our own local storage servers.

The problem I've faced is that seems to be true sometimes, but more often than not I get significant fall off in performance to the point where at times it's just not worth trying. The issue seems to coincide with US working hours, and so I try to get my work in early on in the day, but when a run takes a couple of days to process it doesn't really matter when I start it. I hit this again last week, and I just decided enough was enough I have to solve this or there's no point trying to do a global run when doing a UK run will vary between an hour and seven hours depending on when I start it.

I've reported this in a bug report previously, but truth be told I've felt very uncomfortable about this because the code I'm running when I hit the issue is generated via Claude Code. I wrote a month or so back about how on this project I was actually trying to use Claude Code to see if I could find a way to work with it that I felt comfortable with. The key seemed to me to be not giving the overall pipeline over to the code generation tool, but rather I'm still setting the overall structure of the pipeline I'm building, and using it to generate specific individual transform stages, so that I can still have a strong mental model of what is going on.

However, that's been thrown into question by this constant performance challenge, as I don't really know what's going on because I didn't write the code. Anil, quite rightly, asked me to file a bug on the issue so that it could be looked at, but I've felt very uncomfortable doing so on the say-so of a coding agent. For all I know it could just be holding the tooling wrong, even though I have given it full access to the Geotessera source code and the "skill" for using it. Ultimately, if I'm going to waste people's time, I want to know I've put in some time first, and no matter how much telemetry I got Claude Code to generate I still felt I didn't really know what the source of the problem was.

So on Friday I just spent the day learning what exactly is going on under the hood with the Geotessera stack, and the end result is that I've actually realised I don't need the Python package at all, I don't need to use the CDN and am going directly to the S3 store, and this combined with being able to optimise my code based on the storage model I see that is being used at source.coop, I've actually got the initial problem stage down from taking a worst case 7 hours to just ten minutes.

Under the hood, the source.coop storage is based on Zarr a chunked n-dimensional array storage format. There is also an index held in a Parquet file.

The first win I realised was that just opening the Parquet index file on S3 takes two minutes using Pandas, the standard numerical data frame library on Python. So one suspicion I have on the performance problem, is that accessing the index repeatedly is happening slowing things down. Thankfully I realised two things:

1. I can use the aws cli tools just to clone the entire index locally. It's only 135MB, which is peanuts on a pipeline like ours. Once cloned access is instantaneous.
2. The way the raw data in the Zarr files is stored means I don't need the index anyway :)

The data within the Zarr files is stored based on a hierarchy of structure. First the data is split by UTM zone, which allows each different area of the globe to be stored with much less distortion than if it was all stored in a single map projection. The data is then also split up by year, important if you're doing temporal analysis, though I'm not currently doing that. Then finally within that the data is broken up into a set of tiles that are themselves stored as a series of 256x256 pixel chunks. For completeness there is a icechunk index wrapping the zarr store, which provides some enhanced metadata. However, for my use case that's not important, other than I use it to find the zarr store from the source.coop published icechunk index - but once I've done that I'm using zarr directly.

The main problem I was having was with the occurrence refinement stage, where I look up the Tessera embedding value for hundreds of thousands of points, which was by far the main pain point. Because now I have access to the underlying chunk structure in zarr, I rewrote this to take the occurrences, process them to go from lat,lng coordinates to UTM Zone, and then chunk index within that zone, and sort based on the zone/chunk column, and so now I only fetch chunks once, and I have no need for the index to do this, as it's all simple math to figure it out.

So now I've actually managed to eliminate the Geotessera library from my project, and by going direct I'm (currently) seeing a significant performance improvement. Now, just because it was good on a Friday afternoon, doesn't mean I've won. But now I have my own direct version, I plan on trying to run both versions periodically during the coming week to work out if it's the underlying S3 access that's problematic, or something further up the stack.

There's two meta points here of interest, beyond just trying to escape the performance issues I was seeing.

Firstly, whilst as a software "engineer" I always feel good about removing a dependency on a third party library, I don't see that as a win here. Geotessera isn't aimed at me anyway: it's aimed at ecologists who shouldn't be expected to go understand CDNs, S3, geospatial data formats like zarr, and so on. They need to be spending their time working out how we're going to save the planet not how to restructure their dataframes to align with on disk formats. So it's not good enough to say "it works for me" and move on. Once I have more data I need to get that to the Geotessera team so that they can work in improving this for the people who matter more.

Secondly, it's a good example of how my brain works ultimately. My use of Claude Code, which I will continue to use for this project, is predicated on its focussed application so that I can still have a mental model of what's going on at all times, as that's how I like to work. At some point you do have to let go: is it when you use a third party library? is it when you call a function on the operating system? is it when you trust the floating point extension of the CPU? There is always some level of abstraction at which point have to trust things work. But when they go wrong, I've generally relied upon being able to quickly understand why: oh, it's in that library I'm using, I'll just read the code or write a unit test or talk to a person that knows better. With coding agents I feel very exposed when things go wrong, as I don't have any real understanding of where things are going wrong, and because I've been distant to the actual code that is exercising the bug, there's a higher activation energy required to start understanding it.

I'm not expecting this observation to be a revelation to anyone, but the point of trying to do a coding agent first development workflow on this project is to try teach myself where and when this workflow works for me and where it doesn't. Ultimately I think I enjoy programming, and low level programming, because I like understanding the details. But I also like being a manager on a project as I like being able to build out further than I could as an individual, and coding agents somewhat give you that advantage, except when things go wrong there is no other detail-orientated person that you're managing to dig into it and understand why. I personally feel very uncomfortable telling people they have a problem based on the say so of a coding agent, so I guess I need to mentally budget that although coding agents help speed up some aspects, there will be a cost to be paid at the inevitable break down points. My method of targeted application of coding agents in this pipeline helps constrain that, but doesn't remove it.

But I feel for projects like Yirgacheffe, which I build other pipelines on top of and the point is it's doing the hard math for me, I feel glad that these are still hand written where I can understand it all (or at least quickly page it back in). A lot of my confidence in building pipelines fast on top of Yirgacheffe comes from having a high level of belief in what I've built there.

Part II Project student

On the topic of Yirgacheffe, it looks like I'll have a final year student to supervise that is keen to look at some of the resource management issues in Yirgacheffe, which is exciting news. There's a whole bunch of untapped work here that I've wanted to do for ages and never had time, so having someone start to dig at these is going to be fun.

Urban environment exhibit

Continuing from last week's look at the Fire & Ice exhibition as art overlaps the politics of geospatial work, this week I went to see the Wide Angle View exhibition at the Royal Institute of British Architects (RIBA) in Liverpool. This exhibition covered photography from the Manplan series of magazines they published in 1969/1970. These were special editions of their Architectural Review magazine that focussed not just on the buildings, but on the interaction between people (despite the sexist title) and buildings. The photography was amazing, and well printed. Rather than use architectural photography experts they had gone to regarded street photographers in large to capture how people lived in this built environment, making for some quite powerful images.

I think what was interesting to me was I felt a definite disconnect between what it was implied we were meant to feel upon seeing these images in the year they were published, versus what I felt looking at them today. A lot of the "better life through modern building", aka brutalism, vs the old brick factories being seen as a rotten core to be replaced as soon as possible now feels very misplaced. There was a photo of some miners in a locker room and a text blurb about how they had these legacy dead-end, no-hope jobs with the implication that this futurism was going to save them, but then I feel this is the work-style that deliveroo and uber eats riders now have inherited - jobs below the API as Adrian McEwan pointed out as we considered the photograph. I had to look up this reference:

There’s a model of the modern economy that separates jobs into two kinds: those above and below the API. Algorithms are replacing middle management, and if you don’t have a job telling computers what to do, sooner or later your job will consist of doing what computers tell you.

I guess the old API was just energy, in which case it's the same argument back then and now. Technically Claude Code didn't tell me to do the debugging detail for it, but if you squint enough you can start to see that framing coming.

The point here with yet another art exhibition review in a supposedly tech orientated set of weeknotes is that once again, technology and now it is "applied to people" is political, and we technologists always need to try keep that in mind. It was true twenty years ago when we failed to see it and built the closeable internet, and so it's something I need to keep trying to remind myself of now.

This coming week

• Back to the global run efforts now that performance issues are algorithmic and not ghosts in the machine.
• Wait to hear back from Ali if she's happy with the final round of LIFE maps I generated the other week, and if so get those up on Zenodo and Anil's new LIFE explainer site
• FOSS4G:UK is next week! In my head I had another week to prepare, so I really need to nail down my talk, particularly as I've had to beg them to let me go off-piste and give a demo rather than submitting a PDF of slides ahead of time. They've made it clear there is no sympathy for anything going wrong, so I really do need to get this nailed down.

Last two weeks

Copenhagen

I missed my weeknotes last week due to being away in Copenhagen for a few days. Other than adding to the prevailing narrative that Copenhagen is a nice place, a few specific notes for here.

Firstly, I wrote last year about learning that the Baltic Sea is effectively brackish water, and so I was pleased to spot this salinity map in the local aquarium, Den Blå Planet:

This is a topic that is once again close to mind as I realise that from space you can't really tell water salinity, making the results I'm getting with coastal waters and lakes in my habitat map work somewhat challenging.

Next up I saw this wonderful landscape in the Fire & Ice exhibition of works by Kalaallit artists at the Copenhagen Contemporary art centre:

It is a rendition of the town of Maniitsoq in Kalaallit Nunaa/Greenland, by Emilia Sølvsten rendered in sugar cones on a sugar base, linking the colonial past of the country with the product that is also heavily tied to colonial history. As someone who builds maps, I found the landscape itself a wonderful artefact, something that the Copenhagen Contemporary does well with it's series of very large halls allowing for art to be built at a scale. At the same time linking the cartography to europe's past colonialism is an interesting inversion of maps being used by those trying to take control rather than those who have been on the other side of things. As someone who builds maps I try to be aware of the way we've traditionally drawn and rendered the world in 2D has been driven by those who wish to assert a global, western view, rather than one that will speak at a local scale, which goes beyond projection, but to even concepts like how we colour water on maps. And whilst today most people making maps aren't trying to make such assertions of power, it's just built into the defaults in all the tools we use, and what feels like no-choice is still reinforcing that legacy. There's more on the topic in this article by Emily Jacobi, including some nice physical maps made in Kalaallit Nunaa.

Finally, on the topic of maps, I was very taken with not just the local public transport infrastructure (which worked amazingly well), but also the map:

Superficially it is just another riff on a common theme for public transport maps, but I like the rainbow swoop through the city centre, and in particular I love the typeface they use:

It seems a nicely humanised version of the original Macintosh system font Chicago to my eyes. As luck would have it, there was an exhibit at the Danish Design Museum about the work of Kontrapunkt, the design agency who made the custom typeface used.

Such a lovely font!

LIFE catchup

As I mentioned last time that I was having to do an update to the LIFE pipeline, and that dragged on. I managed to avoid restructuring the pipeline to avoid some of the slowness of using snakemake to do the dependancy tracking on a per species level. When I started the pipeline it was on a home-grown custom pipeline tool, where this was faster, but snakemake does struggle with the fact that in the LIFE pipeline we use 34K species to generate around 250K AOHs and then another 250K per-species extinction risk maps before reducing them again to the final output. I did a little work towards making the AOH calculator support building AOHs from a single GPKG/GeoJSON of many species rather just one species per input, but my pending trip to Copenhagen and my pending deadline to get some LIFE data to colleagues made me chicken out of a re-write of all that code just now.

In the end I managed to get some performance wins by discoverring that by default gdalwarp is very memory conservative, and if you tell it to use more memory using the -wm flag then it gets significantly faster. Our big servers have over a terabyte of memory (bought in the pre-AI bubble days - so expensive, an order of magnitude less expensive than today), and so telling the gdalwarp stages in the pipeline to use 20% of available RAM got my reprojection times from from near 20 minutes to a couple of minutes. Given this happens quite often in the early stages of the pipeline, that's a significant saving. I suspect I'll need to update Yirgacheffe to do this too, as if you re-project your map data at any point it is just calling gdal.Warp.

Habitat maps

On the habitat map I had some predictably disappointing results as I moved my method from the data-rich UK (lots of species occurrences on GBIF and lots of land types/usage in OpenStreetMap) to areas that are more interesting from a biodiversity point of view like Madagascar where data coverage is very poor. I knew that the results would be bad, but it's always still a bit sad when you're proven correct on that. However, it was never the original plan to build per-country maps like I have been doing, I've just been doing that because of limited data access. Thankfully whilst I was away on holiday Tessera v1.1 global coverage is now available. Tessera v1.0 has been available globally for a while, but that had poor coastal masks, which is not suitable for what I do as a lot of biodiversity is found along the coast lines, even for terrestrial vertebrates. In theory Tessera embeddings are globally comparable, so in theory I should be able to find other regions that have data that have similar habitats to Madagascar and that'll help improve the quality of those data-deficient regions.

Oh, and if you're interested in the details of my habitat map work, then I gave a group talk on it last week, which you can see here:

This week

LIFE

Ali pointed out an error in one of the scripts in the LIFE pipeline at the end, so having just completed a LIFE run I need to fix that small error and re-run the tail section of the pipeline.

Habitat maps

I had a good meeting with Sadiq Jaffer and Anil Madhavapeddy planning out some Tessera/Habitat map bits. I need to do an initial global run next, and then I'm going to seek assistance from Sadiq in some of the Tessera touch points, as I'm a bit of a Tessera newbie and I'm going to need help making this run efficiently/effectively as I go from country scale to global scale.

It’s all the rage to either ban anything touching content generators (“AI”) or go all in on it. Neither are quite satisfying to me, and seem to miss the point. Below is an attempt at the general policy I would prefer, if someone were contributing to one of my projects.

Once something is machine generated, it is hard for a human to get understanding (and control) of it again (it just becomes too verbose and not flowing with a human’s thoughts). Thus, accepting a single large machine-generated contribution can affect all future work on that project. This is one of the many reasons to reject such contributions.

The way I see it,

• When something is machine-generated, it’s hard for a human to understand and modify it. Thus, machine-generated content shouldn’t be submitted to human-made projects by outsiders (the project owner decides to make that transition).

• If a project is machine-generated, then I don’t need you to run the generation machine for me. I can do it myself, and I can control and review it better that way.

The below is not to say that I think machine-generated content should necessarily be accepted at all or even is good. There are plenty of other reasons to reject it. Still, many reasons miss the point, so here is my point.

Still, a content generator machine can be useful, but where to draw the line? I currently (2026) would say the following for my projects:

Generating content should not replace human thought. A good way to judge this is the amount of content out compared to the amount of content in. If something would take you N time units to make without “AI”, and using whatever “AI” you want takes about N time units to make it with, I’ll consider this essentially a human contribution.

Machine-generating a contribution to save a lot of human thought can sometimes be useful, but submitting it for review by another human isn’t (they should review the generation process instead). Thus, generated (or partly generated) contributions should be declared so that they can be given an appropriate type of review. I will have a low rejection threshold since you didn’t spend much time on it and if you can do it in [few time units], I can do it in [few time units] too and can probably get the results I want faster by adjusting the input prompt myself, rather than reviewing it and making suggestions back to you.

Yes, this does make an asymmetry: I can use more of the content generator on my projects than you can. Likewise, I won’t use when I send stuff to you. That’s how it works. Unfortunately, this does make a bit of a “rich get richer” system and affects how fast others can develop their skills, and that needs a solution.

You can stop reading here. If you want some specific examples:

• Good: Using translation or assistive technologies, even spelling or grammar checkers, for your own stuff you contribute to me is fine: it helps you do human work more efficiently, or do a trivial processing task faster.

  • If it significantly changes your words, it’s much less so.

• Bad: running the same as above on my content and contributing the changes. I can do that myself just as easily.

• Good: using a content generator to figure out what to do or how to do something, then doing most of the work yourself (or starting with a very early prototype).

• Questionable: Using a content generator to paraphrase, rewrite, or change style is a borderline case. It outputs about as much as went in, but for what purpose? I would want to know what you think; don’t undersell yourself. I would also do this myself if needed. If I wanted it in my style from a content generator, I would run the generator myself to get my style.

• Uncertain: using a content generator to come up with some initial concept then expanding it out. For example, I might use it to come up with an idea for a roleplaying scenario, and then expand it out myself. I am bad at coming up with scenarios, but I think it is the expansion and refinement that is hard. I might be OK with the final product (machine idea + human development) contributed, but it’s heavily dependent on the situation.

• Bad: Using a content generator to write anything significant is a “no”. Most of this was in the content generator and gives me no value. Just refine your bullet points and let me ask the content generator if I need an expansion. Even if you fix it up, I can’t tell what you did vs. what has no value. Maybe instead use that to fix up your original prompt and just send me that?

  • Use a generator to write a cover letter for a job? Not going to want to read it.

  • Use a content generator to write a report? I’m just going to skim it and not give much attention. Just tell me the prompt.

  • Use a content generator to curate a section of background info, and delimiting it saying “This background was machine generated based on my curation and revision and think it’s useful?” Not sure if it’s fully good, but at least I can live with it and won’t hold it against you if it’s clear.

• Bad: using a content generator to fix some code or add a feature. Describe the feature and I can prompt and review myself (yes, might want to try the prompt yourself to make sure it works).

• Bad: you use a content generator to make some decision for yourself. This may be fine for you, but don’t pass it off to someone else as “your” decision.

Last week

Liverpool Makefest 2026

Liverpool Makefest 2026 happened on Sunday and seemed to go well. This year rather than being an exhibitor I was on the organising team, as the event was effectively rebooted. After many hiccups along the way, it all seemed to come together in the end with 30 makers setting up stall and a few hundred people coming to see them. It was touch and go at times, but it looks like we pulled it off. It was definitely a team effort: so particular thanks to Jackie, Neil, Dan, and Mike, along with all the volunteers from the DoES Liverpool community who ensured it went smoothly on the day, and the makers, some of which travelled quite a long way, who came and made the day so full of interesting things to see, hear, and touch.

Habitat mapping

The main focus last week was migrating from version 1.1 of Tessera to the beta version 2 embeddings. In addition to the data layer changes, this also required a reworking of my inference code as the geotessera Python package's API changed quite significantly (at least, based on how I was using it) as I migrated from the numpy storage system in v0.9 to the zarr based backend. The perils of innovating on multiple fronts!

In the UK this seemed to not have a huge impact on my results, which is to be expected as I was relatively data rich in the UK: it would be interesting to see how I can scale back my data usage in the UK and still have good results perhaps, or as Anil Madhavapeddy suggested to use the lower resolution version of Tessera v2. Basically for Europe we can probably generate quite good habitat maps using much less compute and/or data than I'm currently doing.

However, the flip side to that is that my very first run on Madagascar was mostly rubbish, due to a combined lack of occurrence data and Open Street Map data in that country. In the UK I found I got a reasonable map with just 50K occurrences going in, but Madagascar is over twice the size of the UK and has a much more varied set of habitats, and I only have around 5K occurrences, and those are very much clustered in popular points (mostly urban areas and forests), leaving a lot of the country under represented, and this clearly wrong in the final map. In small areas though it does a nice job, such as spotting these irrigation channels amongst the fields around Andapa.

habitat mapsatellite view

At the top level this is quite good, with the dark purple being arable and the light purple being urban. The greens are mostly subtropical moist forest with a smattering of subtropical montane forest, and then the blue is mostly wetlands, though technically this should be artificial drainage - though water is awkward in these maps without other data. But overall the matching is one of the few reasonably good bits we resolved, but that is one small bright star in an otherwise dark sky of incorrectness (yes, yes, I'm not a writer :).

As with Brazil I'll review the data to see if there's any obvious pipeline deficiencies, particularly with respect to regional variances in how OSM data is structured (I wrote about this before for Brazil), but ultimately I do think we just are data deficient for my approach in that region. In theory, that shouldn't matter, as Tessera is a global model, and pixels are comparable around the world, so if I have similar habitats in other countries that are more rich in occurrence data (perhaps in the southern parts of Africa, Indonesia, etc.) then that will work just as well. The challenge there is that the v2 embeddings for Tessera are not yet global, and so I'll need to see what I can do there. I still have confidence in the method I'm using, it's just how soon can I get the data to try and de-risk its applicability in areas like this.

In the mean time if this is going to be blocked on more Tessera v2 coverage, then I might spend a bit of time shoring up my ideas around using Tessera to help suggest OSM edits.

LIFE

I was asked to make a custom LIFE layer for the LIFE team, which I failed to hit the deadline for. Thankfully it wasn't for a major part of the paper, and so it'll probably be worked into the review phase - it's just an attempt to make a more accurate version of an estimate they already have some data for. Still, never fun to send people the "you're not going to get this in time" email. The main reason it failed was that having not run the LIFE data for half year, I hit the usual environment atrophy that plagues modern software engineering. My GDAL had updated since I last ran it, Yirgacheffe had changed and I wanted to update that, and some of the code in LIFE itself for this run was also out of date (the scenario in question not being one we run regularly). In theory an hour or two's work, but then I need to double check everything before doing the large aggregated run and it all adds up.

In theory I'd dockerise it all, and so the code could be frozen in time, but I use MLX to run this on my Apple Silicon using its GPUs, and that doesn't work with Docker. The main learning probably is that whilst I can do that for development, I should probably dockerise it and check it runs on our large Linux machines before I park a project.

This week

This is a short week as I'm on PTO for the back half of the week. Priorities are:

• Fix some of the smaller outstanding issues on my Tessera switch - I have some UTM tearing I need to fix in the final rasters
• Get this LIFE layer done so I don't need to repeat this panic when the paper reviews come back in

Interesting links

• I missed the previous Alpaca Conference, a celebration of generative art held in Sheffield in 2025, but the CFP the next one in May 2027 has gone live. I think I'll submit a workshop on generative art in a functional programming style using Claudius/OCaml. By then the HTML version will hopefully be usable so people can actually make things they can share.

I’ve been interested in why organizations work (or seem to not work, despite everyone in them being good people). Another book cited the book “The Unaccountability machine” (wikipedia, HelMet libraries) as being related to this topic, so I checked it out. In the end, the book did provide some thoughts on this, mainly through the lens of complex webs of decision making and accountability sinks. The first half was most relevant to me; the second half ventured into economics and society (which was still interesting, but not my main focus).

Summary

In modern times, the word “cybernetics” makes one think of human/machine combinations or maybe computer technology in general, but the original definition (1940s and onward) was about decision making, feedback, and control of systems (machines, organizations, economies, etc.). It’s important to keep this in mind, otherwise the topic and review below can easily get confused with unrelated technology matters. Cybernetics and the book are about information flows.

Systems don’t have motivations. They are made up of people who have motivations and outside influences. How can a system made of good people have bad outcomes? How can it be avoided? These are the kinds of questions I wanted to answer.

There are several main lessons I got from the book:

• The number of states a system can be in vs the amount possibly gradations of control input. This is known as “variety”. An example from my university:

  The university defines there are four categories of data (levels of confidentiality), and that should dictate how the data is handled. However, a university is a complex thing with very different {operations, teaching, research} divisions, and four categories to control data (input variety) just isn’t enough to express the wide variety of actual data (and there isn’t enough variety of other processes or exceptions). This leads to people doing whatever they want (ignoring the rules since they don’t seem applicable), or never-ending discussions about how the system needs to be tweaked to make it work.

  The idea that the internal complexity of a system has to match the external complexity it faces is the Law of Requisite Complexity, and once I realized this, I started seeing almost every dysfunctional thing of our university being caused by its violation.

• Cyberneticists have broken down organizations into five systems needed for their control (Viable system model) (operations, regulation, optimization or integration, intelligence, and philosophy or identity, see more in Chapter 5’s summary below). My own team is too small to have these as defined roles, but thinking of them helps me realize some things we need to make sure get done.

  Still, the broader idea of “systems have higher systems watching over them” is relatively general, and we should make sure for each thing we do, someone has the bigger picture and can intervene if things are going outside of the expected parameters, and make sure the parameters are adjusted. “Our people are smart and independent” doesn’t replace the people watching the bigger picture.

• Any complex organization isn’t one mind, or many individuals, but many different parts (people) communicating in a complex network. This can easily allow decisions to be made and propagated elsewhere in a way that doesn’t quite make sense, with little feedback or accountability to the place the decisions were made. This makes the accountability sinks (which was apparently defined here, though I would have thought it was older).

  A complex network of nodes sending signals to each other, sometimes having a result that is undesired? Sounds like “artificial intelligence” these days, with hallucinations and wrong answers, doesn’t it? Just like with “AI”, there needs to be a feedback system to keep an eye on things.

The second half of the book is focused on economics and society (because the author is somewhat of an economist) and less relevant to my work, but I found that it was quite useful to me anyway. Overall, I found some good points that I could apply to my own non-economics life.

Overall thoughts

I liked the book and was happy to read something that combined something in my comfort zone (science-ish description of cybernetics) with something outside it (economics). I’ve learned some buzzwords that I can use in my job to push for changes, though I predict it’s an uphill battle to get anything actually done.

As my team gets more complex, we need to encapsulate different parts and have their internal variety and management, while having a more limited variety that reaches higher levels for the systematic decision making.

Chapter-by-chapter analysis

Below is a summary of each chapter. I state what the book says (with some of my opinions or take-away messages mixed in), but I am not qualified to judge if the book is correct or not.

Part 1: The nature of the crisis

(0) Introduction

The industrialization of society makes larger and more complex organizations, and they need to be controlled. The theory of these control structures is “cybernetics” (the original definition).

I liked this intro, because it related to my thing about industrialization (of finance, control, etc) being the more relevant problem in our society than just capitalism that most people complain about. The idea is that when things get too big, they somehow makes more than their share of problems.

(1) Something’s up

Organizations sometimes do things that just make no sense, for example, the banking crisis of 2008. No person wanted this, yet somehow it happened. There were even people who could predict it, but no one would listen to them because the system was on autopilot and not taking feedback.

Accountability sinks are ways that decisions can be made such that no individual person is accountable. A decision may come out of the organization, but all the individuals of the organization can say “I don’t agree with this but it’s not me who decided” - so where does it come from? Who takes accountability? These are quite common when you start looking for them, when everyone can pass off the responsibility for a decision/rule to someone else (often laundering it through various policies or laws).

I will choose to not mention one example the book gives involving animals, since it’s just too tragic. Another example given is academic publishing. Professional publishers have managed to insert themselves into the academic system and extract huge amounts of money, and most people agree there are serious problems with relying almost exclusively on citations for academic career purposes. Yet, still, it stays, because it’s now part of the system and the politics of making any other decision about careers would be too much.

There are two types of decision-makers, kings (who are humans and to whom decisions can be appealed), and priests (who speak the word of god and thus you can’t appeal to them). Too often, a decision may be made by a “king” but gets conveyed by “priests”, and there is no way to send feedback that something is wrong. This can cause unrest.

All the above isn’t just an accident. It’s a trade-off, discretion (to do the right thing) vs following the rules. What’s the balance? This is the next chapter.

Overall, this chapter said what I wanted to hear about organizations, bad choices, and and unaccountability and left me wanting an analysis.

(2) Stafford Beer

This chapter is about the historical context.

Stafford Beer was one of the founders of management cybernetics - as discussed above, how decisions get made. Beer looked at systems as composed of black boxes (something with defined inputs and outputs, but the internals can’t be examined): different components, and each component had some behavior. You couldn’t/didn’t look into the boxes. Of course, you could recursively look into them if you wanted, or package multiple black boxes into another black box. Still, at some level, one has to draw black boxes around some parts of the system and analyze the parts outside of that.

Management cyberneticists looked at systems as black boxes and that interact with each other, to discover various theories of control. Thinking in terms of black boxes gives the saying “the purpose of a system is what it does”. This is important to keep in mind so that one doesn’t blame the individuals for what the system does - you have to look at the connections and control structures.

In the end, systems/organizations are controlled by meetings between the different levels and components. This often happens by resource bargains between different components, so that each part can focus on its internals with a defined interface to the outside.

I understand the concept of the black boxes, it took some time to fully internalize the recursive nature of them. I’m still not quite clear where one should decide to draw the black boxes (what level and what size), but I guess that’s part of the art of the field. You should probably draw them at the places that are most useful for your current problem. In the end, this question doesn’t matter too much for my understanding.

(3) Aliens among us

So, a system is made up of black boxes once you go deep enough. These are all connected and sending signals to each other. It becomes a… neural network? There are clear metaphors to “artificial intelligence” here, where the system takes on a mind of its own, but each individual part can’t have accountability. This also relates to the Chinese room (translation) thought experiment.

This applies to all kinds of things, from social media algorithms to corporate management. Yet, we know the organization itself is not sentient. So, the organization itself isn’t accountable, but neither are the independent parts (because they are just playing their role).

The more connections a system has, the harder it is to understand. However hard you look, you will probably miss something.

If you are interacting with one of these systems, you can be given the policies/documents in response to a query about why a decision was made a certain way. But that never explains why the policies are made a certain way. The accountability for them has been sunk into the network itself, and if people only try to hold individuals accountable, everyone can escape accountability.

So, in the end, you see the same thing over and over again. Comments about an organization that say they admire all the people inside, but despise the overall organization.

This part of organizations being a computing network really made sense to me and started to build my understanding of why it is so hard to change broken policies (and certainly matches with my “most people I know are well-intentioned and good at their jobs”). When we need something to change, we can’t just blame individuals. We should definitely start thinking of organizations as their connections, not only as their individuals. It doesn’t give a solution yet, though.

Intermission: Computing ponds and rabbit hole

This was about the idea of a pond of decaying matter sending signals to each network and becoming sentient or otherwise a network of intelligence. Or other ways of making systems of “dumb” objects sending signals to each other to form a larger whole that appears capable of purposeful action.

As a person in computing, this all flowed quite naturally and makes sense in the concept of neural nets/artificial intelligence.

Part 2: Pathologies of the system

(4) How to psychoanalyze a non-human intelligence

This starts off with an explanation of why “cybernetics” these days refers to computers: because people got too good at building transistors and computing devices. It took all the attention from the decision making part of the field.

This chapter talks about how systems can be controlled and analyzed. The main consideration is what kind of information controls it, and this gets to the concept of variety of information. Variety, in my own words, somehow relates to the number of states a system can have, or its inherent complexity. It can also refer to the number of possible outside stimuli or control settings a system has. Variety of internal states (or complexity) needs to somehow be aligned with variety of control.

Example: Let’s take a room’s thermostat. It has one input, the temperature, and has one output, amount of heat applied. The variety of control matches the variety of internal states of the system. If we wanted to control temperature and humidity, but only have heating control, it is quite possible that we can’t react to all possible states of the room and bring it to the desired state.

When you have a complex system consisting of lots of black boxes, there are many different states you can be in. What happens if you can’t measure enough information about the state, or have enough control knobs to affect it? You have a problem. The Law of Requisite Variety says you need as much control variety as you have variety of internal states.

This is a general argument and note that it doesn’t require knowing exactly what the measurements and controls are. You can reason that a system has lots of internal variety but not that many ways of control, and know you have a problem - even without knowing exactly what the states or controls are.

This chapter resonated with me and I like the general theory of variety. It provides me a powerful tool to say “you are over-simplifying the policies you are making” - though I imagine it will be hard to get the right people to understand this, if they don’t want to understand what is wrong with their policy from the policy itself.

(5) Cybernetics without diagrams

This tries to get into the deep theory of cybernetics without being too technical. It says this is hard, and perhaps it’s not so interesting. However, one part was quite interesting:

A system that maintains its state needs different parts to be a viable system that is capable of maintaining control over itself. There are different standard functions needed to stay in homeostasis with its environment. The cyberneticists thought that such a system needs to have:

• System 1: operations. What the system actually tries to do for the real world. It does the main thing (say, run a computing cluster), but left alone it could go out of control and over- or under-build what is needed or not coordinate maintenance and use.

• System 2: regulatory. This is the basic regulation of operations to make sure that operations stays synced. In our computer cluster example, it may coordinate hardware upgrades, software installations, and maintenance to make sure the system works. This makes the basic resource bargains for how stuff works. This is about preventing clashes and managing conflicts (things everyone thinks are necessary).

• System 3: optimization or integration. This directs different components towards a main purpose and allocates things between different System 1 operations. This is about the overall mission (compared to System 2). This is the first real “management” level and what some people complain about being unnecessary. In our cluster example, this might include allocating resources from hardware upgrades to support and vice versa depending on overall needs of the users.

Systems 1 and 2 absorb most of the variety of customers, but sometimes customers will need something beyond what they can provide. In this case, System 3 needs to step in and adjust the mission by escalating to the level above. Too much System 3 involvement in System 1 (rules with no exceptions) can result in breakdown, since System 3 can’t process all the variety coming in to System 1. This is micromanagement and takes time away from what System 3 should be doing. The other system, of leaving everything to System 1, can result in no planning. The book makes a point that there can be an informal System 3 by System 1 managers; this is OK as long as they make sure it gets done - this is the value of informal meetings.

• System 4: intelligence. If System 3 manages what is happening now, System 4 is planning for the future by looking outwards to the world (not just the current state but predicting what is coming). In our cluster example, this might be looking at emerging technologies that change the way clusters are run.

• System 5: philosophy or identity. In some sense this balances changes identified by System 4 with the need to keep unit running from System 3 (you can’t change too fast or you lose your identity, and then maybe you cease to exist). It balances the broad organizational units. System 5 decides what the system does, and thus its purpose. In our cluster example, this might involve thinking of the role of clusters as cloud computing and AI become more common.

These systems are recursive throughout the organization. The whole organization has its five systems, then each sub-unit will have its five systems, and so on all the way down.

It doesn’t say these have to be different people or units, just that they need to happen. I do sort of wonder how rigorous the justification is for separating all of them (the book didn’t go into detail about how they were decided).

These systems also have their own amount of variety of signals going in and out. They variety in control signals to each system to be matched to the complexity of the task they are performing.

If they don’t have enough variety to handle an emergency, they need to be able to “declare an emergency” and call a higher level in order to bring the resources needed to solve a problem. The book uses the example of a red handle in a train that means you have to do an emergency stop and notify the control center. All schedules are now messed up and you need a higher level of the organization to fix the problem. (This doesn’t just mean a higher System number; it means the higher recursive “viable system” level with all its boxes).

While I wouldn’t go building these into my teams, it is a good checklist I would use to make sure our team is handling everything that needs to be done. It also provides some good abstraction levels for dividing up tasks, once a team gets large enough.

Intermission: decerebrate cats

This tells a metaphor about how you can cut the link in a brain between the cerebellum and the rest of the brain. The living being still seems alive (walking, eating, grooming itself, etc.). But by any measure, it doesn’t think or have purposeful actions anymore. It can keep living and functioning as long as the environment is what it is used to, but once the environment changes, it can’t adapt. This was apparently part of real experiments done.

Stafford Beer talked about decerebrate organizations which similarly have lost the ability to adjust to the world. It will keep surviving until a shock comes up, and then goes into crisis. If it can’t re-establish the higher-level Systems, it will simply collapse. It warns that if someone sees their organization decerebrate-ing itself, start updating your CV and looking for new jobs.

The book gives the example of universities as typical decerebrate organizations, and this is when I started getting sad. It’s exactly what I have been seeing with some parts of my university, in particular about challenges challenges with legal compliance processes or data classification. The world is more complex than it was before and it’s trying to apply old processes to new things, but no one can take a step back and say “wait, we need to adjust how we see ourselves to fit the current state of the world.” It might be more about feedback rather than the System 5, but still I got feelings from reading this

Part 3: The blind spots

Now, the book makes a clear shift and starts connecting to economics and society. This was interesting to me, but not the main thing I intending to learn about. You could say this part attempts to explain how “late-stage capitalism” works (and has become such a buzzword for everything that’s wrong in the world).

If you don’t care about the society-wide aspects, capitalism, and economics, you can scan the rest of the summary. Even the book’s conclusion focuses on economics and society and was not exactly relevant to my work.

(6) Economics and how it got that way

First, what is an economist? The book says that economics is the science of optimizing scarce resources (and since everything is scarce, economists get to play the rulers of society by making models to say the best way to run everything in the world). It does this by trying to express everything in terms of money.

Like many fields, economists have “physics-envy” and have tried to make models and predictions. However, economists can often forget that these are just models and many assumptions have gone into them, and the assumptions can pre-determine the conclusions. You can also forget that the model is just a model, and the real world can do things outside what you predict (this is especially the case since humans are complex) - it calls this the Ricardian Vice.

This is obvious to physicists but worth making clear: just because something is proven in your nice model with the complexity stripped out doesn’t mean it is true in the real world.

One of the dominant models is that an economy is so complex it’s impossible for any normal control methods to handle it - you can’t bring enough variety to control all the components. Thus, the assumption is that a free market composed of many different parts making their own contracts and looking after their own needs serves a computing fabric that can work to control the economy.

Everything can combine into a “the market knows best” (if you make the right assumptions), with the market serving as a computing fabric for making complex decisions - and is the seed of unaccountability.

“Market as computing fabric” matches with mainstream western thought, and I don’t really object to that. Later the book explains what is wrong beyond “capitalism bad”.

(7) If you’re so rich, why aren’t you smart?

This starts by talking about how economics is a theoretical and not practical science. Economics is not accounting and most economists couldn’t deal with real businesses, and the book claims this is because economists want to be seen as their own special profession above accountants, managers, and similar.

Accounts and accounting are the basic ways to measure a business, and they serve two purposes. One is to control the business, another is to tell outsiders how the business is doing. Accounting is difficult in any big enough business, and while there are standard practices, there are some choices made. The same set of numbers aren’t necessarily good for all purposes (mandatory reporting for public companies vs internal bookkeeping for how it’s going).

Financial reporting also by its nature reduces variety, and makes fewer signals than there are internal states of the company.

So, what signals does one use to manage a company? It’s easy to hire management consultants and they work to optimize the short-term public financial results at the cost of long-term operational value.

We need to be careful that our reports match with what we need to be measuring.

Intermission: Meanwhile, in Chile

Stafford Beer went to Chile during its socialist era with the idea of applying management cybernetics to a command economy that was nationalizing industries (and thus had to manage them). They had computers and were trying to get enough information all in one place (the futuristic-looking “operations room”) to manage it. While it sounded like an interesting idea, applying the law of requisite variety makes one think even this wasn’t enough.

In the end, the US supported a coup (as usual) before it could go far enough to get results (or not).

Part 4: What happens next?

Now we get fully into the economics and politics of the modern world.

(8) Enter Friedman

This section first taught me about Milton Friedman, a name I had heard often but didn’t know the story of. Intentionally or not, he popularized the “neoliberalism” chain of thought which, more or less, said that a corporation can’t know what is best for society, so the best it can do is try to maximize its profits and let the market (the “computing fabric”) work out the rest.

There’s a distinction between the managerial class (those working to manage the companies, they may care about it) and the investing class (or capitalist class) which only cares about profits. The basic idea of neoliberalism was that, if the management class tries to do something for the benefit of society, it’s spending someone else’s money and lacks accountability (taking on the role of a government).

This whole “maximizing shareholder value” concept (combined with “legal personhood” and “limited liability”) are purely artificial. It’s simply a choice that could have been made another way, or not at all. And we’ll see this invented philosophy causes many downstream effects.

The “leveraged buyout” is when private equity buys a company that is under-performing to replace the management or otherwise force it to be more efficient. This was seen as the ultimate threat for any managers that didn’t sufficiently try to maximize their companies’ profits, to “keep companies in line”, so to say. Once bought-out, the companies were given their own debts and forced to cut back anything else in order to repay those debts, on threat of going bankrupt. All together, this replaces any signals of “care about society” with “ignore everything else and maximize your profits in the short term”, which in the end leads to everything falling apart long-term.

This chapter is the thesis of the book, what is wrong with “late-stage capitalism” and all that. In summary, the financial industry is so focused on optimizing profits (one signal) that everything else is falling apart. This matches what I have seen.

(9) The morbid symptoms

This starts off summarizing the book (this is my paraphrasing):

The basic problem is that systems/companies need to be able to re-organize themselves as the world changes. However, due to the financialization of everything, all the other signals needed for society/companies to react to the world can’t be processed, because “make profits” and “pay off debt” override it. This leads to society’s polycrisis.

Stafford Beer’s philosophy was to pay attention to information crossing boundaries. As companies get sliced up, managers’ ranks thinned, and functions outsourced, you lose the capacity to convey this information.

Some theorists had, about 100 years ago, thought the function of the investing class was to take risks and provide what the working class needed, while insulating the working class from the ups and downs of the business cycle. Now that is reversed, where people think the system is the investing class works to maximize its profits, using the working class in the process. The rise of things like populism is the “red handle” that things are not going OK.

The chapter ends by saying this is the nature of the current polycrisis. The system is growing and getting more complex. It has been assumed that the free market will take care of things, but the signals of the computing fabric has been subverted and now all it can care about is short-term profit maximization. All the middle layers of communication are slowly being gotten rid of because it’s unnecessary bloat, but really they were the ones carrying the messages. No one protests “red tape (of bureaucracy) holds us together”.

**I agreed with the what is here, but then again I don’t have the knowledge to know if I should agree or not. Still, roughly makes sense from what else I have read about problems with society.**qq

(10) What is to be done?

“If you can hear the problem, you can hear the solution.”

As for the problem of private equity and leveraged buy-outs, one proposed solution is that limited liability should be moderated. Limited liability serves as an accountability sink to whoever buys a company. The author thinks this might be enough.

Broadly, what the author thinks is needed is more viability and less maximization of single goals. Right now, financialization has made one goal take full precedence over everything else, and the very limited variety of these signals means so much gets left out of the decision-making process, at least at the highest levels.

Going back to the artificial intelligence metaphor, we have made an artificial intelligence, told it to maximize profit, and it’s going wild. Every system set up as a maximizer needs another system watching over it.

The book hypothesizes that things like social media provides more ways for more signals to be sent, but how will all these be managed? Can they be? Will AI (another maximizing system) be assigned to do it? It’s just another system that can be blamed. I read this as idle thoughts rather than a fleshed-out proposal (and personally I would be quite hesitant to go down this path, but it’s an interesting thought).

Overall I didn’t quite think there was a great conclusion for the book, but as the first quote says, maybe that’s not the point. This is hard, and if we can see the problem, maybe we can work towards it. I’m hesitant to for any solution that involves social media sending signals, but it is an interesting thought to come back to. Anyway, I don’t need a conclusion or recipe for what to do about our economy, I’m happy with the lessons I learned along the way.

Last week

Local developments

This week I attempted to make a more nuanced version of my habitat map than I have done to date, and I'm glad I did, as it turned up some hidden problems with my data-processing that were obscured in the previous simpler versions. And better yet, both problems were evident a few minutes walk from my front door (possibly even visible from my office window if I leaned out in a way that wouldn't be approved by health and safety officials at the University).

Before we get into details, some context. Until now, I've been generating my new habitat maps using broad habitat categories, referred to as "level 1" categories from the IUCN Habitat Classification Scheme. At level 1 you are just deciding if each pixel in the map is "forest" or "shrubland" or "wetland" and so forth. In total there's 18 level 1 categories. Each category is then broken down into more detail, the "level 2" categories, so you get "tropical forest dry", "tropical forest moist", "boreal forest", etc. I'd thus far avoided working at level 2, partly because the applications that this map will be used for have thus far worked at level 1, and partly because I wasn't confident that I'd have enough data in certain parts of the world to give me the necessary level of detail. I've discussed in the last couple of weeknotes the struggle I've been having with generating my map for the Brazilian states of Minas Gerais and Espírito Santo due to low occurrence counts in GBIF and lack of detail in Open Street Map (OSM) zoning.

Operating at level 1 is defensible, however, it'd be much more interesting to operate at level 2. Whilst the main inspirations for my own work comes from the Lumbierres et al 2019 paper which worked at level 1, the other habitat map paper that is of comparable note is the Jung et al 2020 paper which is generated at level 2, although is perhaps the more simplistic modelwise.

Anyway, for fun I decided to try process the UK, where I know we have high levels of species occurrence data and OSM coverage, at level 2 and see what happened. At first glance, the map looked great: you could still see the same overall trends as the level 1 map, just now where we had "forest" before we had mostly "forest temperate", what was "shrubland" was mostly "shrubland temperate" and so forth.

But all was not quite what I expected, and so I dug in and was amused to discover that the problems were ones that were observable from where I live.

The Great Cormorant and its subtropical forests

The first oddity I noted in the new map was there was a small but non-zero amount of subtropical moist forest in the UK. Now, I know from discussions between my colleagues Vinicius Tonetti and Aneesh Naik that there is some small patches of rainforest in the UK, and indeed Aneesh had tried using Tessera to map them, so I wondered if perhaps that's what was showing up. But when I plotted the locations of the pixels, it was clear that this wasn't what I found, I had a more general misclassification problem:

The root cause turned out to be a slightly silly gotcha of my own doing. I looked at which species in the UK would be seeding my map generation with subtropical moist forests, and it turns out that this was the main culprit, as taken on walk around West Kirby Marine Lake, not five minutes from my desk, a couple of months ago:

This is the Great Cormorant, a common bird around UK coastlines, and one we often see when out for a walk along the coast. If we look at the habitat preferences of the species as per the IUCN Red List data, we can see that it mostly likes different types of wetland and marine habitats, which aligns with what I see when I go for a walk. But we also see it likes a single type of forest: subtropical moist forest. Now, as a computer scientist I'm not an expert on this, but I'm fairly sure that's not what it's finding in the UK.

If we look at the range map for the Great Cormorant, that is to say the areas on the planet where a given species might exist according to the experts, we can see it's divided up into several different colours:

Each colour is what's called a "season": green, found along the coast, is resident (always present), the orange inland is non-breeding, and then the yellow up on the northern Scottish islands is breeding. You can kinda of think of resident as being an area where the species is present in both breeding and non-breeding seasons. This reflects the bird is migratory somewhat.

Similarly, if we look at the habitat preferences of the Great Cormorant from the IUCN Red List, we can see that its preferences are also tied to season. And in particular, its preference for subtropical moist forests is only in the breeding season.

The bug in my code was that when fetching species data from the IUCN Red List I was merging all the different seasons into one single range map and one habitat preference list, whereas the data is more subtle than that: certain habitats are only preferred in certain seasons. When I started this project I just copied my existing species data-fetching script from one of the application pipelines, where this subtlety wasn't important, but here it very much is. In my habitat map generation code we were seeing Great Cormorants in the UK in inland areas OSM marks as forest, and so we were noting that this area must be a forest the species likes, so adding all these occurrences of subtropical moist forest. However, if I actually treat the seasonality properly, then we know that the inland range of the Great Cormorant is only for its non-breeding season and it's not interested in forests then.

Now, I think that in general it's unlikely that the Hebrides have much in the way of subtropical moist forest either, despite the data supporting that technically. But in that case we have to rely on general statistics when we make the map removing those points, as we don't have other data to say it's not likely. I'll come on to this sort of filtering in the next bug.

The important thing to note here, other than the foibles of copying and pasting code from other projects, is that this was a problem in the level 1 map too, just it was hidden as we converted all the different types of forest down to the single "forest" label making it impossible to see. So the takeaway here is that even if the end of this project is a level 1 map, I should be working at level 2 as much as possible to highlight things like this.

The Marine Lake that wasn't

The other bug that showed up when looking into this one relates to where I was stood when I took the above photo. I was stood on the wall of the West Kirby Marine Lake, which is an artificial lake used for boating and (if you're so inclined) swimming:

This is formed by an area of what I assume was beach that has been surrounded by a wall that is just below the highest tide level, which means it gets filled up with sea water periodically, and thus is a nice safe place for people to learn to sail etc.:

You can see the wall, which has a nice wide path on top of it for walking (it's about 2.5km around), in this photo of another bird we often see: the Little Egret:

In particular, in that photo you can see the water level is almost overflowing, which indicates a recent high tide (or tide plus storm) was enough to top the lake up.

You may be wondering what the point is. Well, when I looked at my level 2 habitat map I couldn't help but notice that the marine lake clearly stood out:

The dark blue to the edges is "Marine Neritic - Pelagic", which is basically the sea, then we the light blue as we come in land that is "Marine intertidal - mud shoreline", and you can see the intertidal zone in the above photo clearly enough. But the reason the West Kirby Marine Lake stands out is because it has been predicted as "Wetlands (inland) - Permanent freshwater lakes". There's several things wrong here, but the most important one for this discussion is the "freshwater" bit - I just spent ages telling you it's filled by the sea, so it is most assuredly not freshwater (though, as I've never been brave enough to swim in it, you might want to question my confidence here :).

So why do we make it freshwater? Well, it's because Open Street Map listed it as such. Until I corrected it, the West Kirby Marine Lake OSM entry was in OSM as natural=water, water=lake, salt=no. Thus when I use OSM data to refine the list of habitat preferences for species occurrences in the lake I've restricted them to freshwater wetlands, forcing those pixels to that level 1 category. It's basically poor hygiene on the input data.

Whilst I can fix this individual instance, it's similar to the issue I flagged last week in that just manual fixes like this don't scale. Ultimately it flags a broader problem I have which I don't deal with in my pipeline currently: incorrect input data. Both in terms of occurrences are perhaps mistagged as the wrong species, or the GPS coordinates are off slightly, or incorrect labels in OSM like this one. On the whole both GBIF and OSM are wonderful resources, but in any data set there will be some errors, and so the question is how to weed those out.

Thankfully, Tessera, the foundational machine learning model that I'm using to generate these maps, should be able to help here. If based on occurrences and OSM data I decide that a set of my training pixels are all "forest" say, but I think a few will be wrong due to occurrence data incorrectness or OSM filtering incorrectness, then I should be able to look at the statistical spread of the values in the Tessera embedding space, and spot outliers. That is to say, if for each pixel I look at the values in the 128 dimensions per pixel, I can see how well clustered the values are, and remove any that are outliers on the grounds that they are likely errors. This would need some testing to check I'm not throwing away good data, but in general I think this is probably the correct approach.

My colleagues Anil Madhavapeddy and Sadiq Jaffer have been doing some work around clustering in the embedding space using Uniform Manifold Approximation and Projection for Dimension Reduction, which seems to show clustering performance quite well, so I'll need to see if I can crib notes from their work.

Radical cartography

I picked up this book last year and finally got around to starting to read it:

It feels particularly pertinent now that I'm building my own maps. As has been in the news this week with the UN moving to promote people to use the Equal Earth Projection, the lines we draw on maps are inherently political. In my line of work this is something I worry about a little, but probably not nearly enough. We make maps saying on some given metric where it is good or bad practice to do farming or logging or whatever, usually in the tropical countries where there is the most biodiversity to be impacted, but we do it from a place that is quite removed from that context. Historically it's generally been a bad thing when UK institutions try to dictate to the rest of the world how to run their affairs: that's a context in which our work needs to be considered. Hopefully the science we do is made open to question and scrutable so that people can pick it apart and see if what we claim holds up for themselves - this is one more reason that open science is so important. That hopefully enables it to be integrated with other information and insights we lack from not being of those places about which we attempt to derive results.

This is why, in the habitat map work, I've been keen to seek out experts who have on the ground experience who can comment on the maps I produce and how well they reflect the ground based on their knowledge rather than a mix of my assumptions and what I can encode in Python scripts.

I'm still early on in Radical Cartography, but already some of it has felt directly relevant:

"...These are conventions that render the world unnaturally solid and well ordered. Cartography therefore becomes radical when it embraces the inherent uncertainty, multiplicity, and subjectivity of both our data and the world itself."

This relates to the idea that the data is not certain, and thus the output isn't certain. We're starting to see in the habitat map domain the idea of probability or certainty coming in to what before were presented as absolute: this pixel is forest, that pixel is desert - there is no room in the TIFF format for ambiguity. But actually you can do this: when working on integrating farming data into the Jung et al habitat map last year our group decided that the best approach was to add it as a probability per pixel when converting what was in the original map, given we didn't know exactly where the better farming data should be (it being at 10km per pixel over our 1.8km per pixel map). Talking to James Ball and Aneesh about their work they're building confidence layers to go along side their habitat maps to try make it clear that we don't have perfect knowledge.

Anil has also pointed me at The Power of Projections: How Maps Reflect Global Politics and History by Arthur Jay Klinghoffer, so that's on the list next.

Claudius for the web

I did a little more on the web version of Claudius, and I think I've mostly got the API to where I like it, but I still need to get the bootstrap story sorted in my head, so I'm hoping to go over that with Patrick Ferris over virtual coffee this coming week.

Geoguessr

I instigated an initial work lunchtime Geoguesser. It was a small start (in the end due to time pressure it was just me and Sadiq), but we had fun, and as ever I learned a little from it.

Related to my earlier point: it's a tiny thing, but I've enjoyed using Geoguesser as a way to learn more about the world that I study from afar. For example, I've been studying Minas Gerais and Espírito Santo in Brazil, and I can explore them in Geoguesser, or at least the bits Google has visited. Whilst that is far from making me an on the ground expert, I find it useful to at least stop these areas from just being a set of abstract pixels on a top down map. When we do all this work the interesting places of the world are the parts with humans in it and they have their lives and pressures and it's just a tiny reminder that the work we do isn't just some abstract math problem, it's a reflection of human existence either in or indirectly impacting these areas.

Coincidentally it was the world championships this weekend, and the final match was really quite something.

This week

• I was meant to start on a LIFE update required for a paper submission by the LIFE team, but I failed to understand what it was that was being asked of me. I now have clarification, so hopefully I can get that moving so it's done before I vanish to Denmark in ten days time.
• I want to try tidy up my OSM categorisation code, which has been expanded on as ideas come, but I suspect could do with a bit of rationalisation. What I really want is a meta description that I can compile to both a diagram and be used in Python code to do the actual filtering of occurrence/habitat data, but I fear that is a yak I don't have time to shave (am actually grateful I don't have time for that).
• Get back to habitat mapping.

Useful/interesting links

• Whilst looking for a colour scheme that is colour-blind friendly for the range maps, I found the Okabe-Ito colour set.
• On the back of last week's chat about how to suggest edits for OSM, Adrian McEwan pointed me at MapComplete where I could publish topics that others can contribute to. He also pointed out there are quarterly project proposals in the OSM community, which could be another place to consider this.

Last week

Habitat mapping

After a few weeks of incremental improvement, I finally hit a stage where "progress" lead to regression in the results of my attempt to build up a habitat map. In part this comes from trying to infer certain things from Open Street Map data, and how different OSM communities approach things. For instance, in the UK any town will have a residential area enclosing polygon, but in practice that isn't the case in Brazil. If you happen to sample a point in a town that isn't an actual building then you are most likely just to get that as an unclassified area. In theory we could do some proximity inference, but for now I'm more interested in what is and isn't actually stated in the OSM data.

I did find a close proxy, which in Brazil is using administrative zones at level 10. The admin zones in OSM are interesting, as each country has different levels of administrative zone based on how that country manages sub-regions. For Brazil there is one that is down to the suburb/neighbourhood level, which isn't used in all towns and villages, but does seem to be somewhat used, so I thought I'd give that a go.

At the same time there's a lot of nature reserves in the parts of Brazil I'm looking at, and whilst they don't say what kind of nature they're preserving, I wondered if we could use those to rule out anthropogenic (human made) change. Turns out this isn't the case - most of Anglesey in Wales is actually under a preserve and includes towns. But I tried it anyway, but only where there wasn't some existing polygon explicitly stating town.

The net result of these is somewhat of a regression. I don't think they're particularly bad rules, more that in Brazil we're just data starved, and so any rule that happens to tweak the data one way is having exaggerated impact.

Sticking with OSM, I've made a habit of when I spot pasture now I see if it's correctly recorded in OSM, as my suspicion was that more than often it isn't properly recorded. This added an extra layer of fun to my trip to Yorkshire (see below) last weekend as I tried to remember where the fields I spotted with sheep and horses in happened to be :)

In general I see two things happening frequently in OSM in the UK:

• Pasture is misclassified as landuse=farmland. To my reading of the OSM wiki, farmland should be arable, whereas pasture should be meadow with a subtype of pasture. I've seen this both around Cambridge and on the Wirral; for an example, these are horse paddocks: https://www.openstreetmap.org/way/324028977
• There are walls/fences marked, but there is no actual polygon for the field at all, which I saw quite a lot around the area of Yorkshire we were in. I drove past all these fields with horses and sheep in them today: https://www.openstreetmap.org/#map=17/53.806429/-1.959402

I have corrected examples of 1, e.g., https://www.openstreetmap.org/way/188906871 where it was farmland but I know it's had sheep in it for several years at least - but that manual fixing by just one person doesn't scale.

This I think means two problems for my current approach on using OSM to refine occurrence data:

1. We under-represent pasture as there's just less of it
2. We probably over-represent arable as a result of pasture being labelled as farmland

In the short term I think I might need to treat OSM farmland as either, with a bias to arable if a species likes both. But in the long term I do wonder if we can use Tessera to help fix this. If we tag examples where we very much know what is arable and pasture in OSM (solid truths), then we should be able to use tessera to then work out at least the misclassified fields and create a list of suggested edits for review.

I spoke to some people better connected with the OSM community at the event I was attending, and they suggested if I had that I could try to engage the UK OSM community for one of their semi-regular themed events, so I feel that might be a good side project for us to run in the group, perhaps if we can find a student with some time.

Finally on this I had a good long conversation with Vinicius Tonetti, a collaborator who is very familiar with the Brazillian landscape, and he answered a lot of naive questions from me on what to expect on the ground in the regions of Minas Gerais and Espírito Santo. I have to work hard to keep a sense of scale for Brazil: it's tempting to think of these sub-regions of Brazil being like counties in England, or even like the individual states of Scotland, Wales, etc. but Minas Gerais alone is 2.5 times the size of the UK, and so I can't just assume a level of uniformity that my mind defaults to for that sort of sub-national unit.

Anyway, I was very grateful for for Vinicius's patience as he educated me about the makeup of the area, and it made me feel that at least in Espírito Santo, whilst we have some obvious problems (roads being marked as arable in my maps), and it's clearly not up to the standard we have in the UK, it's not terrible, and perhaps actually doing better than other habitat maps in broad strokes.

Claudius for the web

I went to Open Source Hardware Camp 2026, a small gathering of the UK open hardware community. Set in the wonderful Hebden Bridge, it's a slightly odd event where I think many of us go to meet people there more than to see the talks or workshops. Not that there weren't good talks: particular highlights being Jo Hinchliffe talking about what's new in FreeCAD and Stuart Childs talking about boding bare, unpackaged chips to PCBs in Shenzhen. But I think there was a sense, between the all male speaker list and with one of the community stalwalts passing earlier this year and generally it feeling like an aging community of where might it be going.

In the social gathering the day before the conference I felt we did find a couple of threads that might be pulled upon. Quite a few community members are involved in various computer preservation projects, from those volunteering at The National Museum of Computing to build a working EDSAC, and Kevin Murrell who gave a talk on emulated PDP's with physical interfaces. There was a common thread though that these not just be objects of worship, but things people can engage with. This brought back to mind that in the last OSHCamp in 2024 there was some members of the sizecoding retro demo-scene at the sister event the day before, who are people making new and exciting demo effects using tools like TIC-80 or old 8-bit computers - seeking the challenge of doing more with less, which is the essence of these historic machines. I feel there's a potential connection there if only we could find a way to bridge it. What would that crew get out of the blinken lights on these old computers?

To that end, in quite moments over the weekend I dusted off Claudius, the retro-style graphics library I made for OCaml and did something Anil Madhavapeddy had suggested a while back, and started porting it to run in a browser, using the Js_of_ocaml OCaml bytecode to Javascript compiler. Progress was quite swift, such that after just a day, and having never used JSOO before I was able to get an unmodified example from Claudius running in a browser, like so:

Claudius normally uses SDL for graphics, via the Tsld OCaml bindings, but I'd always assumed one day there would be different backends, so I've always made sure that the SDL code was relatively well contained. Claudius maintains an entire rendering pipeline to a virtual framebuffer, so once I'd tidied up the few SDL abstractions that had been missed, porting the run-loop from Claudius to target an HTML canvas element instead was relatively simple. That above example I just took the tunnel example tick function and copied and pasted it into the same file my new lashed up run-loop and it Just Worked™.

Why? Well, mostly it makes it much easier to share the output. In the past I've done this by running it, capturing a video of it, converting that video to a format suitable for the web, and putting it on my website. With this you should just be able to post the actual live demo to the web. And because of how Cladius runs, you could then do other things like present different variations and have web page controls change say the palette or let you parameterise the constants in the code to let people explore the math behind the demos more easily. It doesn't directly link the two worlds I outlined above, but it felt like a little contribution I could make to encourage more of that sort of behaviour.

There's probably a couple of days work before I can upstream all these changes to Claudius proper. I've refactored Claudius now to have the SDL backend be an optional import, and I have started to make the JSOO alternative backend. Keyboard input is in progress, and mouse input shouldn't be too hard once that is working. I think the main thing I want to think about is the abstraction for just working with it: the old SDL based method of code, compile, run I don't think quite makes sense for the browser - where are you going to run it? Should we generate a template holding page for the canvas element required so you can easily get started? Can we do that with a simple HTTP server too? But that looks quite different I think from saying you want to embed it within another page. I suspect a couple of cups of tea will sort it all out :)

Multithreading

Whilst on our way to Yorkshire we stopped off at Lancashire at the Queen Street Mill, a former cotton mill that still has a working steam engine connected to looms. I can highly recommend the guided tour they run, where we were shown the end-to-end process by which cotton was processed and turned into cloth, including a lot of working machines.

What interested me is how much parallelism is in play in a factory like this, which makes me feel there is a good parallel programming guide somewhere in this factory. There's some simple, obvious parallelism, such as making the back beam, which is the width wise set of parallel threads that will be woven together to be cloth. These rolls were made on machines that literally showed the multithreading at play:

They also had even wider machines of the same kind when they needed to make wider cloth, showing some simple scaling in parallelism:

There was also some less obvious but still relevant stories of parallelism too. The cotton for the shuttle that passes through the parallel threads from the back beam is on a much smaller reel, as the shuttle needs to be small to be able to pass through the loom back and forth. But the supplied reels didn't necessarily match the size of whatever you were using, so the would need to be rewound for a given mill's set of looms. In this mill they actually unwound these small input reels into larger conical bobbins with enough material from dozens of these smaller reels, which was a slow process, but they did this because they had another machine that could then automatically unwind this larger bobbin onto many final shuttle reels without the need for human intervention, a sort of job queue system perhaps (which was also running in parallel):

The point is that the raw inputs sometimes needed to go through many transforms, some of which appear slower than you'd want, but that is to enable some greater level of parallelism downstream. This is a common challenge in data-science, where the ideal path to throughput might not make sense at every single stage along the way.

And so far this has been all to get one machine running: in the main hall of the factory originally would have sat 900 or so looms (there are now around 300 IIRC), and a single operator would be managing four or more looms, trying to keep them fed (instruction scheduling on a CPU with multiple units :). All 900 machines were driven by a single steam-engine that has shafts and gears and belts overhead.

They also, for some of the fancier machines, had punch-card programming for the patterns:

Anyway, I highly recommend if you're around Burnley that you stop off and take the guided tour. They fire up the steam engine in full once a month, but if you miss that, like us, they have an electric motor drive a section of the factory so you can still get to see the machines in motion.

This week

• Wrapping up the minimal Claudius JSOO backend before that falls out my head again
• Writing up my paper outline for the habitat work to help ensure everyone knows what I'm planning
• Move to the new geotessera xarr code for my work as it is clear that what I'm currently doing with offline tiles isn't the future for how tessera is to be used.

Last week

Tessera group meeting

On Monday there was a small Tessera group show-and-tell in the lab for one of the funders, where I presented some of the habitat map work I've been doing with Tessera. The collective set of talks and demos spanned from how the Tessera sausage is made through to more higher level application of the model from people like myself and the ecologists. It's interesting to me that I was sitting on the ecology side of the fence for this one. In my career, whilst I feel I've shown I can do deep down and nerdy when required, I've always been more driven by the application of technology to have some impact rather than technology for its own sake, and that was very evident in where I sided in the in this show and tell. Not that I was the only CS person straddling that line, but I think it's good to know where one sits, particularly as this probably marks me as not a particularly good academic, at least within my nominal domain that I hold a degree and the one that pays my salary.

Indeed, this is what caused me to leave first academia and then commercial research a couple of decades ago, as it felt a bit disconnected from immediate impact. Not that I don't think research is important, far from it, but I personally get more joy and motivation from knowing that the work I do will enable others in some more obvious (to me) way. But that's why I've enjoyed being in academia so much this time around, sitting as I do between disciplines: I get to enable longer term research with that immediate feedback of enabling the ecologists I work with.

Many, many years ago I was part of the NESTA Crucible program, which took 30 early career researchers and aimed to try forester interdisciplinary collaboration. Whilst it was a lot of fun mixing with others from a wide range of disciplines, we all agreed at the time it was going to be a very hard program for them to assess impact wise. Here I am these days living the Crucible dream when that program is long since shuttered and I suspect anyone interested in it has moved on. But if anyone from those days does stumble over this, know that it was much appreciated :)

FOSS4G UK talk accepted

I submitted a talk on Yirgacheffe to FOSS4G:UK 2026 - Free and Open Source Software for Geospatial, UK for those wondering - which has been accepted. It's in Leeds in October, so I look forward to heading over for that and meeting up with the community there.

Data mapping

Last week I alluded to the challenges of limited data as I moved the focus of the habitat map generation from the UK to more biodivesity-wise interesting regions like Madagascar and Brazil. I've been forced to consider this a bit more as just pointing my pipeline as it was at the Brazilian state of Minas Gerais generated a fairly garbage map.

As mentioned last week, I'm dealing with limited data on two fronts: firstly from GBIF where I get species occurrence records, and secondly from OpenStreeMap data where I extract information about natural records and land usage. I showed a heat map last week of how biased GBIF clearly is to the places like Europe, North America, Japan, and Australia, but since I found a nice summary from 2021 of the biased distribution of GBIF records in Sampling Biases Shape Our View of the Natural World by Hughes et al. Although the paper is now five years old, the heat maps from GBIF today still closely match the diagrams and issues in the paper, so I feel it's still quite a relevant read. For my current particular line of investigation I found two particularly interesting observations:

• The sampling is biased by elevation: with both mountains and marine trenches being particularly data poor. Even in the UK I have observed this, with a lack of sample data on mountain tops leading to my current UK habitat maps showing the tops of the Black Cullin mountain ranges as urban :) I have just one sample from that area in GBIF using my current filters, and we don't have particularly high mountains in the UK.
• The sampling is biased by accessibility, with most data being close to cities and within some close approximation to roads. This certainly matches again my observation of data in the UK, where I can see very clear biases for nature reserves, footpaths, etc.

There's a bunch of other observations in there which one probably would have guessed but it's nice to have quantified, such as the dominance of birds in the data: 87% of all records in 2021, 100 bird species account for 38.4% of all animal records, 10 species account for 11% of all records, and 2% are just mallards and starlings combined.

The elevation thing is particularly interesting too, as anyone who as seen tree-lines on mountains will know that elevation does define habitat, so lacking signals at different elevations is genuinely problematic. The hope here though is in Tessera: in theory there is enough nuance encoded in the model that even a few samples on mountain tops that aren't physically near bit are habitatually (sorry) near, then the distance in the model will be small and we'll get appropriate matches.

All this comes around from the fact that for the UK I just used a single year of observations and got reasonable results in my habitat map generation method (Black Cullins aside), but in Minas Gerais, even after opening up my data to plus or minus five years I still was generating poor maps. I think this is also driven by poor OSM data on the ground there. But I have noticed that the neighbouring state of Espírito Santo does have much better coverage (Minas Gerais is the larger region here, and the smaller one Espírito Santo):

My hope again is that if I can get better data in Espírito Santo then I can find enough points that are similar to Minas Gerais to help in the embedding space.

It's also worth noting that this problem isn't unique to us: Lumbierres et al is similarly going to be impacted by this issue. However, as ever in this domain something is better than nothing, and understanding the limitations is just as important as the artifact itself.

Misc

I had intermittent power outage on Friday which took me off main duties, so I finally migrated my other website over to my newer EIO/Htmlit based stack that powers this one. Whilst I quite liked the Dream framework that powered the old one, it did seem to hang once a week and need restarting, whereas the new one based on EIO has been solid.

The migration did make me realise just how much div spam there are in my pages in a vague attempt to make them reactive (both to screen size and content). It's quite horrible under the hood. I do kinda wish I could just go back to doing explicit pixel coordinate layouts: perhaps I just put all the content in hidden divs and manually lay them out in javascript based on the current viewing area... thankfully I dislike web development enough that I'll leave that yak unshaven :)

This week

• More data gap plugging
• More habitat map experiments
• I need to send some data to people, which is still an annoying process for things I can't publish for everyone
• Open Source Hardware Camp 2026

Last week

Both tardy and sparse weeknotes. Sparse as I had my second shingles vaccine last week which took me out for a day, but better a day feeling unwell now than a potentially unlimited amount of unwellness in the future.

Puntcon

My weeknotes are a little tardy because I was down in Cambridge briefly. On Sunday it was Puntcon, the somewhat annual get together of technologists, artists, and social scientist types (and anyone else who is interested). As ever, a fun time chatting to interesting people, and one idea that I need to find time to try prototype before I forget.

Habitat maps

I've continued to push along on the habitat map work. The UK maps are reasonably well refined now, but the UK is the beginner level/easy mode for this sort of thing: the UK is incredibly data rich, and so I need to start looking at other parts of the world now to see if my method is going to work in other areas or be useless due to a lack of data. To be clear, I'm not finished in the UK, but I feel I've de-risked the UK part significantly, so now I want to de-risk it going global. Ultimately, whilst it's nice it's showing promise in the UK, for the applications where I want to apply this work the UK isn't that interesting.

You can see the aforementioned data disparity if you look at this heat map of GBIF occurrences:

Similarly for OpenStreetMap data - as much as I adore OSM, you do see the availability of data fall of once you get outside populated areas. Even within Europe, when I was looking at the area around Kullberg, Sweden, to make a Minecraft map I found none of the buildings in that area were in OSM and I had to instead use the national buildings database.

As a concrete example of that data deficit and going back to GBIF, if I take occurrences of terrestrial vertebrate species in the UK I have around 80K points for 2024, but for Minas Gerais, one of the states of Brazil, which is 2.5 times the size of the UK, I have just 600 points for 2024.

The other challenge with operating outside the UK is it's harder for me to personally eye-ball the results and know if they're comically bad or not. Thankfully I've had some people come forward that can help me: Vinicius Tonetti has just joined the group I worked with on the LIFE metric, and he has experience with the Minas Gerais region (which is why I'm looking at it), and Anil connected me with Miranda Lam at Bangor University who is working with ground truths in Madagascar. I should say reconnected, as Miranda was working on the Tropical Moist Forest based project that I first started on when I joined the University.

I'm also being nudged to move to v2 of Tessera, which is in beta, over the v1.1 I'm currently using. Seems this is getting to the point where for early projects like mine it's good enough, and by the time I'm doing things seriously it'll be closed to production.

This week

• Tessera group show-and-tell meeting in Cambridge
• Meeting up with lots of other peopel whilst I'm in Cambridge
• Pressing on with the habitat map work outside the UK

Last Week

Tessera Habitat Mapping

I'm continuing to push forward mostly on the habitat mapping work, with occasional side quests into tooling bits I need to support that (changes to Yirgacheffe and Geotessera, more on that below). It's starting to take shape, in that the maps I'm making have some recognisable features, but I'm still at the phase where I'm learning about the data and having to question initial assumptions: the maps are in that awkward phase where they look good at first glance but are questionable when you dig in.

For instance, here's a view on Anglesey, over on the north east of Wales:

You can see a good mix of urban (light pink), arable land (dark pink), shrubland, and forest. It picks out the Newborough Forest with it's tall pine trees, and even the race track on the island. However, it misclassifies Mynydd Parys, an old copper mine that you can explore, as urban rather than rocky - a common mistake if I look around mountain ranges otherwhere. Still, good progress since last week.

Rather than talk about the map, I thought I'd share some of the process of how I'm building the pipeline to build the map. This is in part motivated by discussion in the Nordic-RSE Zulip, and in part as I've been trying something new, which has worked well for me so I want to record that. The context here is that I have an idea how we can improve on the habitat maps we use for the other pipelines I've worked on, and so I'm implementing this, aided by colleagues how have more experience on both general ecology and in building habitat maps for other domains. The focus is how can I quickly prove that this idea has legs or not, and now can I keep track of that, and keep my collaborators informed, both of whom are remote (including remote to each other).

The coding

Whilst in general I write all of my production code myself, for this project I've used Claude Code to help build a prototype pipeline quickly, but done so in a way that the output mimics my own code and overall architecture so that I can keep a mental model of what the code is going. For me I'm not trying to replace writing code as much as just doing it faster so I can more quickly work out if this project is going to succeed or fail at a science methodological level. The challenge with any abdication of code writing though is keeping a mental model of what's going on so that when the code doesn't do what you expect you can dig in to understand why: is it a bug in the code? is it the data didn't conform to expectations? is it the scientific method is flawed and the approach needs rethinking? To be able to ask and answer those questions I need to always have a working model of the overall pipeline structure in my head. I feel I've achieved this through a couple of practices in this project, most of which I apply to my own code writing anyway.

The first is that my data-science pipelines follow the tried and testing UNIX philosophy of many small pieces loosely coupled with queryable results at each stage. Each Python script in the pipeline typically will only do one transform within the pipeline: one to fetch data from repository, one to filter the data as we need for this experiment, another to combine the filtered data with some other source, etc. These small scripts then join together, Voltron style, to form the larger pipeline. As mentioned, I do this anyway when writing pipelines by hand, as I think it's a key part of keeping scientific pipelines flexible: the very nature of science is to question and challenge and try other ideas, so by having a pipeline made of lots of small parts it's easier to swap one stage for another, and if my ecology colleagues want to understand just one bit of the pipeline so they can play around, then similarly keeping each script small hopefully makes it more accessible.

So whilst smaller scripts is a good start to maintaining accessibility to the code, I need to be able to quickly read through the code and understand it. So the second trick has been establishing a baseline for code style and structure. I remember many years ago I had to read a bunch of low level code within WebKit, the browser backend behind Apple's Safari browser, and I was amazed at how easy to read it was given I was coming in cold, it was written in a language I haven't used that heavily, and it was quite gnarly code in places. This was made possible as WebKit (at least back then) had quite a well adhered to set of coding guidelines. That experience has shaped how I write my code for the data-science pipelines I write, as I hope that others will be able to take them and work with them after I move on to other projects.

How does that work with Claude Code? Well, like with many projects, there are aspects of this project that are quite similar to projects I've written in the past: I'll need to get some data from the IUCN Red List and some species occurrence data from GBIF, etc. And so I seeded the project with those scripts and asked Claude Code to modify them keeping within the existing code style. I then asked it for new scripts to follow the layout I have, which is quite boiler plate heavy: PEP8 import order, using argparse for all inputs, doing that in a main function before calling the actual "science logic" in another function (which means pipeline parts can be combined in a library by others, which I've seen people do), it all has type hinting, and must pass automatic format and typing checks. Because of this guidance I find the code generated by Claude Code easier to quickly read and edit than if it was just implementing it using whatever style was the default behaviour.

Thirdly, I take a very iterative approach to building the pipeline out. So for each update of the pipeline I change one or two things, ensure those work as I'd wanted, run the data, assess the impact, write up the status (more on this later) and then move on. So I use Claude Code to nudge forward at each stage, one step at a time, meaning again I understand what has changed as I go.

And finally, I do go and edit the code when I think it's either quicker for me to do so than convert my intent into a prompt for Claude Code, or I just want to understand this particular bit in detail. I'll even write or rewrite whole sections to test and idea, see that it has legs, and then use Claude Code to do the more tedious house keeping around it. This way I'm still involved and so my mental model is kept up to date.

This approach has meant I'm moving faster than if I'd written everything by hand and was keeping it to a high quality bar readability wise, but slower I guess than if I was more hands off - but for me the key is maintaining my own mental model of what the pipeline is doing at all times.

Orchestration

The pipeline is glued together with Snakemake. I don't particularly like Snakemake, but I don't have a better alternative, and it is at least generally popular in the data-science community. I do this because some glue is needed to take all these scripts I've generated and run the end-to-end experiment, particularly for my ecologist colleagues who, although they may understand the overal scientific method the pipeline implements, they don't have the same detailed mental model of the implementation as I do.

This glue often lags behind code in the rush to test new ideas, particularly if one doesn't like the glue tooling, but in this case I use Claude Code to generate it (and have it formatted using standard tooling for readability), and importantly I have Claude use the Snakemake script to execute the pipeline. This way I can ensure that at all times the code and the glue logic are in sync.

Repeated evaluation

Having built a couple of large data-science pipelines for ecologists I've learned that I need to be very meticulous in recording intermediary results so I can explain to said scientists what decisions are being made within the pipeline, not just provide them with the final result. They, understandably, want to not just see the answer, but when something doesn't match their expectations they want to know why without having to come to me.

A technique that sounds perhaps excessive at first but has proven useful time and time again is to have each of the scripts generate a spreadsheet alongside whatever data artefact it would normally create - something I learned from Chess Ridley when working with her to build my implementation of her STAR pipeline. In each spreadsheet will be both a set of summary statistics and a record of any filtering decisions. For example, in the STAR pipeline that Chess wrote she generated a spreadsheet that had a row every possible species that could be considered, and then a column for each filtering decision (is it extinct already, does it have any range data, etc.), so you could see how far a given species made it before it fell out. It's a lot of data, but when you have two runs of the pipeline and one isn't doing what you expect, you can immediatly review these spreadsheets to spot where the data differs before you start looking at the code, and it sped up debugging my implementation of Chess's STAR pipeline so much faster. Since then other projects I've worked on do this, and again the ecologists now ask for that spreadsheet before asking me about the code, which is a win for all involved.

For this pipeline I've turned this telemetry up to eleven, not just doing it where I think it might be useful, but I make it a default for every script. This way I can readily compare the impact of any change we make to the pipeline, and I can do that at the point of the change, not just in the final aggregated result. Useful for when Anil asked me why adding marine species had helped with habitat classification in the Cairngorms: I have an audit that shows we added this many species, and more coastal species like rocky places, and so the rocks within the Cairngorms had more data to match. Even just seeing the overall impact of habitat changes from that change was easier to explain than looking at the final raster:

The other benefit of this telemetry, which I'd not originally considered when I started this, is that because I used Claude Code to add code to generate them, it is aware they exist, and without prompting now when I use Claude Code to do a run of the pipeline after a change it will compare the spreadsheets between the runs and highlight significant changes in any trends.

Communicating progress

As I mentioned before, my colleagues on this project are remote, and whilst we have a chat channel we can use to discuss things, I wanted a way to ensure that every decision I make is recorded somewhere so they can keep on top of what I have and haven't done. I'm acutely aware I'm a computer scientist working on ecology things, and so there's lots I don't know: by making my process transparent I hope to head off getting too far down a wrong path.

To solve this, at least for my side, I actually took inspiration from taking the minutes for the DoES commuinity meeting. For each iteration I write a short set of "minutes", using a todo list to review action items, and having sections for important points I want others to know. These all go into a single PROGRESS.md in the code, and so the latest code matches the progress report. For each pipeline iteration I have the following sections:

• Approach: A sentence or two about what I changed methodologically from the previous run, with perhaps a note about the expected impact.
• Inputs: Layout out where the data is coming from, in particular if I've added or removed data since the last run.
• Results: Here I discuss what has changed, using the data from the spreadsheets I generated to outline if the impact did anything.
• Actions: Here I list any outstanding ideas or suggestions from the team in markdown todo format. So any time I or my collaborators have an idea, it goes into this list, and so they can see what I've acted on and what I haven't.

It's early days, but if nothing else this document is useful for me. The notion others might read it gives me motivation to write it, but like these weeknotes the primary audience is future me.

Summary

None of the above is rocket science (it's somewhere between computer science and zoology, totally different), but for me this has been an interesting new mode of working, seeing how I can use a tool like Claude Code to help me derisk/fail-faster, and how I can do so whilst keeping both myself and my colleagues in the loop. It'll be interesting to see (for me at least) whether my ability to keep the whole pipeline in my head whilst being somewhat detached from every line of code continues to hold as the project scales up.

Yirgacheffe 2.1

Last week I mentioned a bit of work I did on Yirgacheffe based on the needs of the habitat map project, and I did a few more bits this week to help with assembling single maps from tiles that are in multiple different projections.

One of the more interesting bits (to me) was implementing the take operator, which takes a geospatial layer and an array of values, and then the pixels in the source layer are used to index the array, and the value found is in the resultant pixel - basically it's a look-up table operator. As with most operators, it is modelled on the Numpy operator of the same name. This same operator also exists in MLX, which is a Python library made for Apple that is Numpy compatible but will use Metal GPU acceleration when ran on Apple hardware. Where it gets interesting though is that MLX and Numpy have slightly different semantics of what happens when your input layer has indexes that exceed the array, but I want Yirgacheffe to perform the same on both CPU and GPU.

In the end I had to add code to the GPU path, slowing it down, so that I could get consistent behaviour. My hand was slightly forced on this as the MLX out-of-bounds indexing behaviour is described as undefined, so I can't rely on what it does today being what it does in the future. It does raise the question of why do I support two different backends in Yirgacheffe (Numpy for CPU and MLX for GPU) when in theory MLX has a CPU backend also. Initially it was because MLX was quite young when I started using it, but I get the sense it's more mature and certainly Apple keep updating it. But at the same time, if I was told one of the two projects had stopped development and I had to guess which one, I'd say MLX every time, as I just don't trust Apple not to get bored and do something else - they are very proactive at deprecating and removing APIs.

So for what feels like an uncommon operator, I think I made the right choice, but it is interesting to occasionally reconsider why one spends so much effort on decisions made several years ago.

This week

• Looks like I might be giving a demo of the habitat mapping work next week in Cambridge, so I'll try and come up with a nice landing point for this work by the end of the week - not that I'll be finished, but making sure I have a story to tell if I'm presenting it.
• Blogging: I need to write up my Nordic-RSE 2026 conference report still
• Geotessera changes: I had to fix a couple of things in Geotessera for my particular use case last week, and now I need to work out if they're worth upstreaming given that there's a big rewrite under way to move the pipeline from Numpy arrays to xarray.

Last Week

Words not come good today - I basically blew all my writing energy on another post that is not yet published (explained below), and so this one will be quite short (other than the later sections, which I wrote as I went along last week).

Habitat mapping

I've done some more on the habitat mapping front. I implemented a iterative label refinement thing, an implementation of an approach similar to IPAL. This took a lot longer than I expected, mostly as working with GeoTessera is a lot slower than I assumed based on the data volumes I was using - my script ended up taking three hours just to process 50K points. I need to dig into this this week, as I don't know why it was slow, so could just be me being silly. But given how much experimentation I need to do, solving this is kinda key to making any progress. I did get the script to run though, and so now I can try refine the parameters to get better results vs the arbitrary thresholds I initially specified.

It did cause me to look into the data, which was kinda of interesting. I got 1,009,424 occurrences from GBIF for 2024 in the UK for 182 species I was interested in. This fell just to 58,017 when I filtered for samples within the species range! Now, to be clear, I've not taken all of the range polygons for each species, as I've ignored those labelled as "passage" - but I need to try sense check this, as it feels odd.

Of those 50K that remained, there's some super obvious biasing going on. If we look at the area around where I live:

There's clear clustering in the points that remain. If we look at the bits I've highlighted:

1. This is RSPB Burton Mere - so somewhat obvious we get more sightings around nature reserves like this.
2. The Leeds and Liverpool canal, a popular walking destination I assume?
3. I'm surprised Formby forest isn't more popular, as it's a noted location for Red Squirrels.
4. Conversely, I was surprised the town of Southport is so popular! I had a look, and it's mostly common birds, so I assume just there's a local active birding community there.
5. A bit of woodlands - seems a common clustering point across the UK.
6. I somehow forgot to add 6 :)
7. Chester Zoo! Thankfully I filter out by species range, so there are no giraffes in here, but there's a lot of common UK species still tagged here.

The point is that the GBIF data, in the UK at least, is very unevenly concentrated, which means I have to also allow/adjust for that.

The final things I did was read through From Pixels to Patches: Pooling Strategies for Earth Embeddings and a similar discussion thread on our group Zulip, as Tessera beingat 10m is too fine scale for this work really: both the habitat maps I want to produce are for projects running at 1km or so, and the occurrence data is unlikely to be that good for 10m (which might be a part of why I've got so few occurrences).

Foundation model explainer

I took another go this morning at writing my "what is Tessera"/"what is a geospatial foundation model" explainer, and once again ran out of energy 2/3rds the way through. If nothing else though it's proving useful for refining my own understand of what it is - which ultimately is one of the main benefits of doing any form of teaching.

Yirgacheffe tweaks

I've started accruing some small updates to Yirgacheffe, which at some point I'll bundle into a minor release. To help make interop with Geotessera a bit better I've added support for using the Affine package for map projections. Yirgacheffe grew out of trying to wrap GDAL, so where necessary has been using its regular Python tuple for storing map offset and pixel scale, but GeoTessera is Rasterio based, and rasterio uses the Affine package with a different value ordering to GDAL. I've now updated Yirgacheffe to both accept the Affine objects for map projection and offsets, and also return them back when requested.

I also needed to process some polygon data for Ali, whereby I had to inspect a geopackage to get the metadata to let me pull apart the 200 plus project polygons into individual layers. I've had a ticket on Yirgacheffe to prove access to the data fields in geopackages and geojson for ages, so I finally just fixed that.

In both cases though this just reminds me of my plan for Yirgacheffe 3.0, which is to support using multiband data (so multiband GeoTIFFs or GeoPackages with many polygons in them) in operators that will then apply operators to each layer individually, allowing us to express parallelism in a declarative fashion. However, I'm not yet contemplating tackling that!

PAT testing

I went to a workshop at DoES Liverpool ran by Mike Gorman where I learned about how to Portable Appliance Testing - aka PAT, which is more commonly referred to as "PAT testing", living in that special language hell alongside "SCSI interfaces". I think we're all familiar with PAT being a thing required in most workplaces, so I thought it'd be fun to learn how to do it. It's not actually required by law in the UK that electrical goods are PAT tested[1], but it is required that workplaces provide a safe environment, and it's one part of demonstrating that which is why it's so pervasive; similarly insurance providers often require it, again as a standard way to check people are taking things seriously. Interestingly, equipment under a year old doesn't need testing as its assumed to have shipped in good condition.

Anyway, it was fun learning about the various classes of equipment that need testing, and about the (roughly) two parts of the test: doing a visual inspection to check for damage, check fuses are correct, and so forth, and then using the magic testing box (we were using a Kewtech EZYPAT) to check the appliance. Although it's automated quite a bit, you still are engaged in the process with many devices: for example I had to test a extension reel, and the acceptable resistance on an extension reel is a function of the cable length and the diameter of the cores within the cable, so some bookwork is required, as the magic box doesn't know that. If you can remember power equals current times voltage, then you have all the high school physics you need for this.

This Week

• Looking into my Geotessera performance - is it an issue or is it unrealistic expectation?
• Experiment with my label refinement process
• Look to bring in other datasets to augment using specialists to refine generalists.

1. (PATed? I think we should adapt that: we know the server is happy as it's been patted :)

   ↩︎︎

I hope we had a low threshold for sharing anything in chat, as long as the sharer puts a bit of effort into a summary. In the CodeRefinery chat, we’ve thought about how to build more interaction in our chat (even though, compared to some orgs, we think we have quite a bit. Perhaps that’s the power of Zulip). My proposal is “news channels”. The goal is like a blog or news aggregator, with the similar goal of educating ourselves and getting to know each other better.

I like seeing a wide variety of links, but time is limited. Not everyone can read every link posted, so I hope that people put their human brain into a summary (suitable for the readers) that is educational even if read alone. Not clickbait, and not “read the link for more”.

The basic idea is that we people want to blog more, but that’s a lot of work. It’s lower threshold to declare some chat channels to be “blog-like” and encourage even off-topic things to go there. Hopefully it leads to more blogging, too.

For example, I propose the CodeRefinery chat declare these as news channels:

• #general: News channel for anything remotely related to scientific computing.

• #random: News channel for anything else.

“News channel” does not mean that channel is for only news posts, or for things that could be blog post. It means “for things you might want to tell others about”.

This post is my personal opinion, but will be updated with the practices of the organizations I work with (CodeRefinery, Aalto Scientific Computing). You are welcome to copy and modify this. This should not be used to tell people they are doing something wrong, but only to encourage new interaction. You don’t have to do what is here.

What is clickbait?

Philosophy: Clickbait (according to wikipedia) “is designed to attract attention and to entice users to follow (“click”) that link” [note: normal clickbait is often misleading or sensationalized, and that’s not a problem in our chats.]. My main consideration is you need to click through to know the actual story.

To me, anti-clickbait means putting the main message as early as possible so that one can stop reading as early as they want and still get something:

• Read the first sentence, get a 1-sentence understanding of the topic.

• Read the first paragraph, get a 1-paragraph understanding of the topic.

• Read the first section, get a 1-section understanding of the topic.

• Apply this recursively at every level.

This requires more thought in writing, but is a service for the reader. I think we should strive more towards this [note: Does this post meet these criteria? I usually try to, but I don’t always manage.]. It’s related to Bottom-line Up Front communication, from which I got some of these ideas.

Basics

This is my proposal:

• News channels are non-work channels, and you don’t have to be subscribed. Don’t expect all your colleagues to be subscribed. If you have an announcement for a team, make sure you mention it in a work channel, too.

• You can post anything within the scope, for example:

  • Link to other content you have found interesting.

  • Link to you blog posts or things you have created.

  • Pictures you have taken.

  • Original ideas that might become blog posts later.

  • Things you have learned (doesn’t have to have a link), watched, etc.

  • A question or thought you have (including something you might blog about later).

• Put some human effort into describing it (see below). Please no bare or low-context links. Yes, people can read things themselves, but there’s too much. Our networks helps filter summaries.

• There’s not a limit (yet) on the amount you can post, but in general, the more you post, the more human effort you should put into describing things. People can independently subscribe to wherever you get material from if they want everything from there.

• Follow-up discussion on any content is encouraged, but not required. Don’t get discouraged if no one reacts. It might mean your description is good enough. Do react to interesting things to encourage more of that type of thing.

You are especially encouraged to post anything new you create, no matter how small it is. This is how we learn of each other’s skills.

How-to

Further thoughts, adapt as is appropriate:

• Create a new topic for each new thing. Don’t worry about making too many topics, but you can always reply with more stuff for relevant old topics if it’s relevant. Topics are the way someone quickly filters through it, and remember they will “Mark all as read” once they have browsed enough. People may look through their Zulipchat inbox at topics and decide what to click or not, so make it easy for someone to know if it’s relevant to their interests.

• You can use a detailed topic description, for example:

  • “TIL: uv for python package installation”

  • “OpenAI has acquired Astral (the company behind uv)”

  • “venv-apptainer: new script for containing virtual environments”

• Include a description - not just a bare link or wall of text. You should put some human effort into the description, even if only one sentence. If you can do this, we will be happy with whatever it is. The description should be enough for someone to learn some bare basics if they don’t read further (and know if they should read more).

  • At minimum, an initial description that stands alone and provides some education. Not just clickbait, but a real description. Can be anywhere from a sentence or two, to a paragraph. People have a limited time and attention, so write for that.

  • You are welcome to go as far in depth as you would like after that… long info is OK.

  • Especially consider giving your own thoughts on the matter, if it’s by someone else. Do you agree with it? Disagree? Have further ideas? These kind of reviews make links far more interesting, because many people don’t have the full context to understand things.

• Even if you wrote something elsewhere and you are linking to it, give a summary in the chat.

A reminder: Do not use this blog post to tell people they are doing something wrong. Don’t think the barrier is too high. Don’t assume this post is the best practices. We want you to do your best, and read what others do, and everyone to continually improve.

The benefits?

Does this seem like a lot of work? Well, yeah, but it’s the trade-off to engagement. In the age of generated content, we want human effort. Look at it this way: thinking about, writing, and reading these summaries is good general communication practice with people outside your field.

Insights from the International RSE Survey 2026

This post shows results from the 5th international RSE survey that ran in early 2026. Over 950 Research Software Engineers from 55 countries shared insights into their career and interests.

The full dataset is available for download at https://github.com/softwaresaved/RSE_survey_longitudinal/. An interactive dashboard with all international RSE survey data is available at https://rse-survey.soton.ac.uk/superset/dashboard/RSE_survey/. We have also compiled a Quarto book where you can review the findings for each survey question: https://nordic-rse.org/rse-survey-2026/.

The analysis below is based on 62 submitted responses from the Nordic countries. Percentages are calculated from the respondents who answered each question, so the number of responses may vary slightly between questions.

Most submitted responses came from Finland (32.3%), Sweden (30.6%), and Norway (16.1%), followed by Estonia (12.9%), Denmark (6.5%), and Iceland (1.6%).

The sample spans early-career and mid-career respondents. The most common age groups were 35 to 44 years (44.3%) and 25 to 34 years (32.8%), followed by 45 to 54 years (18.0%). In the analysis below, we use three age groups: under 35, 35-45, and 45+.

Interdisciplinary RSE community: Broad but concentrated in STEM

Nordic RSEs work across many disciplines, but the survey responses are strongly concentrated in STEM fields.

Computer Science was the most common work area, selected by 71.0% of respondents. This was followed by Biological Sciences (37.1%), Physical Sciences (32.3%), and Medicine/Health Sciences (27.4%).

Computer Science was also the most common disciplinary background, selected by 38.7%. Physical Sciences followed at 22.6%, Biological Sciences at 9.7%, and Mathematical Sciences at 8.1%.

Most respondents had advanced academic qualifications: 54.8% held a PhD and 40.3% held a Master's degree.

RSE role and software experience

Software development is central to the work of most respondents. 93.5% write software for academic research as part of their job, and 67.2% consider themselves professional software developers.

Many also have substantial experience: 35.5% reported 15+ years of software development experience, followed by 10 years (17.7%) and 5 years (11.3%).

Most respondents are not primarily in leadership roles. Only 8.2% said that most of their role consists of leading a group of software developers or RSEs.

The software they develop is often used beyond an individual researcher. 75.9% reported writing software for more than two users.

Employment and job titles

Most respondents reported relatively stable employment conditions. 74.2% were in permanent employment, 22.6% were on fixed-term contracts, and 95.2% worked full time.

Among respondents under 35, permanent and fixed-term positions were more evenly balanced: 55.0% permanent and 45.0% fixed-term. This may reflect the fact that many younger respondents are in PhD or postdoctoral positions, which are usually fixed-term.

Job titles varied widely. Respondents reported their job titles in free text, which we grouped into broader categories. The most common was Software Engineer or Developer (37.5%), followed by Researcher (17.5%), Technical/Professional Support Role (12.5%), and Faculty/Professor (7.5%).

In addition, 30.9% said they are known within their group by a different job title than their official one. This highlights that RSE work is often present even when the formal title does not say "Research Software Engineer".

Most respondents work in universities (79.0%). Universities were also the most common background for respondents' previous role (51.6%), followed by private companies (22.6%).

Time allocation: protecting time for software and research

Software development took the largest share of working time, at 44.7% on average. This was followed by research (19.9%), other activities (15.6%), teaching (14.9%), and management (14.4%).

When asked how they would prefer to spend their time, respondents wanted more time for software development and research, and less time for management and other activities. The largest desired reduction was in management time, at -6.6 percentage points.

There were some differences by age group. Respondents aged 35-45 and 45+ wanted to reduce teaching time slightly, while respondents under 35 wanted to increase it. This may reflect differences in career stage and responsibilities.

Overall, the results suggest that RSEs see software development and research contribution as the core of their work, while management and other organisational tasks can reduce time available for these activities.

Technical toolbox: Python, GitHub, HPC and collaboration

Python is by far the most commonly used programming language among respondents, selected by 95.1%.

Git was reported as a version control tool by all respondents, and GitHub was selected by 93.4% as a collaboration platform for software development. Other frequently selected collaboration tools included Git (70.5%), GitLab (52.5%), and Slack (50.8%).

HPC was the most frequently selected deployment environment, at 58.6%. This was followed by laptops or desktops (48.3%), stand-alone servers (39.7%), and cloud platforms and GPU (31.0%).

For day-to-day development, 73.3% primarily use GNU/Linux, followed by macOS (15.0%) and Windows (10.0%).

There are also some age-group differences in programming language selections. Among respondents aged 45+, the most frequently selected languages after Python were JavaScript at 42.9%, C++ at 28.6%, and SQL at 28.6%. Among those aged 35-45, R was selected by 48.1%, JavaScript by 44.4%, and SQL by 37%. Among respondents under 35, JavaScript was selected by 30%, while C++ and Rust were each selected by 25%.

Generative AI in RSE work

Generative AI tools are already used regularly by many respondents. 42.6% reported daily use in RSE-related tasks, and 19.7% reported weekly use. 16.4% reported never using generative AI in their RSE work.

The most common use case was writing and debugging code, selected by 82.0%. Designing software architecture was the least common use case, at 20.0%. This suggests that generative AI is currently used more for practical coding tasks than for higher-level software design.

Among tools used at work, ChatGPT was selected by 63.3%, followed by Claude (40.8%) and Gemini (26.5%).

Looking ahead, 59.3% expected generative AI not to significantly change demand for RSE roles over the next five years. 22.2% expected demand to increase, while 18.5% expected it to decrease.

Most respondents expected generative AI to increase their own productivity (66.1%). Views on code quality and job satisfaction were more mixed. 44.6% expected code quality to stay the same, while 36.8% expected job satisfaction to decrease.

When asked what skills are needed to use generative AI effectively, common themes included critical thinking, scepticism and output validation; solid software engineering skills; prompt engineering and generative AI literacy; judgment about when and how to use AI; code comprehension; and testing and quality assurance.

Software quality, testing and software citation

Developer testing was the most common testing approach: 78.7% selected "developers do their own testing". Automated testing with continuous integration was selected by 52.5%, while 31.1% selected "no formal testing but user feedback".

Software citation appears to be common among those who answered the citation questions. 82.1% said they always reference software directly and 66.7% said they always reference a published paper describing the software.

Persistent identifiers are less common. 80% of respondents who answered this question said they never generate a DOI or other persistent identifier for their software.

Open-science practices appear widespread: 88.7% reported having an ORCID ID, and 94.9% of those who answered the licensing question said they always license their software with an open-source licence.

Job satisfaction and labour market demand

Respondents reported high levels of satisfaction. 87.1% were satisfied with their current job position, while 4.8% were unsatisfied. Career satisfaction was also high: 80.3% were satisfied and 9.8% were unsatisfied.

Many respondents also saw their expertise as relevant beyond their current workplace. 69.0% agreed that their experience is in demand on the labour market, compared with 15.5% who disagreed. This confidence appears across age groups, and is especially strong among respondents aged 45+, where 83.3% agreed that their experience is in demand. Among those under 35, 73.7% agreed, and among those aged 35-45, 57.7% agreed while 19.2% disagreed.

Recognition of research contribution was stronger from supervisors than from institutions. 81.4% agreed that their contribution is recognised by their supervisor, compared with 61.0% who agreed that it is recognised by their institution.

Career progression and promotion uncertainty

Although many respondents are satisfied and feel that their skills are in demand, career progression within current roles appears less clear.

Overall, 48.3% disagreed that they are likely to gain a promotion within their current group, while 20.7% agreed. The pattern was stronger among older respondents: among those aged 45+, 58.3% disagreed and only 8.3% agreed.

Promotion processes also appear unclear. 51.8% disagreed that the process for gaining a promotion is clear and understandable, compared with 28.6% who agreed.

Younger respondents were generally more positive about career opportunities. Among those under 35, 60.0% agreed that there are many opportunities within their chosen career plan. Among respondents aged 45+, 72.7% disagreed.

This age pattern should be interpreted carefully. For younger respondents, especially those in PhD positions, the next career step may be more clearly defined. For mid-career and senior RSEs, progression may be less standardised, especially where institutions do not have formal RSE career paths.

At the same time, 43.9% agreed that their next position is likely to be an RSE or Research Software Developer role. This suggests that many still see themselves continuing in the field, even when promotion routes are unclear.

Skills development and training needs

Most respondents learned the skills they need for RSE work through practice. 83.6% selected learning on the job, and 65.6% selected self-taught learning. A university course or module was selected by 50.8%, and online courses by 32.8%.

The most common future skill interests were AI-assisted development and agents" (22.5%), HPC, parallel and GPU computing (22.5%), communication, teaching and writing (20.0%), DevOps, CI/CD and pipelines (12.5%), and software testing, quality assirance and verification (12.5%).

Many respondents are also involved in teaching or training. 47.5% reported providing training one to two times per year, and 50.9% teach or support credit-bearing modules or courses.

Time for personal skills development varies. 38.3% said they always have some time for their own skills development, while 28.3% said they have time only in weeks with a low workload.

Networks, community and local support structures

Many respondents have peer networks beyond their closest colleagues. 62.0% reported having such a network, including 24.1% who had to build it from scratch.

A majority, 55.9%, reported being part of a dedicated research software group within their organisation. This suggests that Nordic RSEs already have local structures for support and collaboration.

44.1% are members of an RSE association. Among respondents who answered the follow-up question about interest in joining an RSE organisation, 78.6% said they would be interested.

The main things respondents hoped to get from such an organisation were networking (85.1%), research software standards (72.3%), training (57.4%), and research collaborations (53.2%). Priorities varied by age: respondents under 35 emphasised job opportunities (66.7%), while those aged 45+ placed more weight on research collaborations (63.6%).

Funding sources show a mix of institutional and project-based support. Institutional support was selected by 72.9% and grants by 67.8%. Access to dedicated RSE funding appears limited: only 16.7% said they have access to RSE-type grants in their institution or country, while 40.0% said no and 43.3% did not know.

Publications, dissemination and conferences

51.6% of respondents have published a paper about their software, for example in journals such as JOSS or SoftwareX.

When software contributes to a paper, acknowledgement practices vary. Being named as a co-author is relatively common: 25.8% said this always happens and 54.8% said it happens sometimes. Being named as main author is much less common: 2.9% said this always happens, while 64.7% said it never happens.

47.4% have presented software work at a conference or workshop. Commonly mentioned areas included digital humanities, HPC and e-infrastructure, chemistry and materials science, astronomy and physics, and RSE community conferences.

Project workload and resilience

RSE workloads often include several projects. 80.3% of respondents are currently involved in one to four software projects, and 39.7% work on projects that typically include a project manager.

Project resilience appears to be a concern. 58.9% said their current project would stall if just one knowledgeable developer left. Only, 16.9% work on projects that typically have a plan for how to cope with developers leaving.

What respondents want from a potential future Nordic-RSE institute

Respondents were also asked how a potential future Nordic-RSE institute could support community engagement and knowledge transfer. The most common answer was organising conferences, workshops, and similar events, selected by 86.7%.

When asked what tasks a Nordic-RSE institute should take on, respondents most often selected advocacy for research software towards politicians and decision makers (71.7%), community engagement and training (70.0%), and promotion of RSE career paths (70.0%).

These priorities fit well with the broader survey findings. Respondents are skilled, often satisfied, and feel that their expertise is in demand. At the same time, they want clearer career structures and more opportunities to learn, connect, and be represented.

Please share your thoughts or comments!

We would love to hear what you think. Please take a moment and share your thoughts about the survey and the analysis in our collaborative document: https://hackmd.io/@nordic-rse/rse-survey-2026.

What's next

These findings will directly shape how we, together with you at Nordic-RSE develop our activities and priorities going forward. We'll follow up with a separate post on the concrete steps we're taking in response. We will also link to resources on where to find training, networking and career that already exist for RSEs in the Nordics today. If you would like to get involved, come to one of our coffee breaks or biweekly meetings: https://nordic-rse.org/join/