jonathan@researchcomputingteams.org
Last issue (#187) we talked about challenging times ahead. Since then, even in the UK things aren’t going great, but in the US…
There’s been a bit of a delay in followup; as some of you know, I’ve been talking individually with leads and managers, especially colleagues in the US about all the stuff going on, and offering help when I can (and at least an understanding ear when I can’t).
With the turmoil of direct and indirect cuts to health research (of questionable legality), cuts to NSF staff, graduate programs are shrinking, summer research undergraduate programs are being shrunk dramatically… and now swingeing cuts to NASA, an agency so broadly popular at home and abroad that it does a decent line selling merch.
And yes, there have been some successes in court rulings, but people who are determined to slash science budgets aren’t going to stop after a foiled attempt or two.
While all the advice in #187 is still relevant, I’ve had a number of conversations with US colleagues about more urgent near-term actions leads and managers can take. I’m going to share the distilled advice here. Not all of it will be relevant to everyone’s situation, but a lot of the moves you’d want to make are in the same direction.
Broadly, the advice falls into the following categories;
Your boss, or your grand-boss, or your great-grand-boss, is hearing about institutional scenario planning, potential pivots, and triaging priorities, probably in a pretty chaotic fashion.
They are not going to be allowed to loop you directly into that, or even directly share any of it. Everything’s too messy and half-formed, there’s probably comments being made that shouldn’t be circulated, and there are unlikely worst-case scenarios being thought about. It would be wildly irresponsible to sending random managers those half-baked probably-not-going-to-need-it-anyway scenarios.
But if you’re going to be a helpful partner to your boss/grand-boss/etc, you need to know some of the priorities right now so you can move things in whatever direction you can. Is the priority finding new funding lines, bringing in external business? Are there particular areas of research (health & life sciences, social sciences) that are going to be hurt most, and is the desire to try to cushion the blow there with resources from elsewhere? Is it time to start cutting costs? [LJD: yes, it is]. What’s the best role you can play right now?
Talk to whoever you can, indicate your willingness to help, and try to get some sense of the directions and priorities. Those will shift, and rapidly. But the more and more frequent information you can get, the more specifically you and your team can be prepared.
A lot of the suggestions last issue were about this, and they remain good and useful activities; but if you haven’t (say) started a program of collecting and writing up testimonials/success stories already, this may be too late to start, as people’s minds are going to be focussed elsewhere.
And I can’t say this enough - run these past your manager to make sure you’re not missing anything.
This one is tough, because the way our budgets work there’s generally fairly modest amounts of discretionary spend - most of or funding comes to spend a specific way for a specific effort.
But in the likely situation of cuts coming, having already cushioned the blow a little bit and maybe having a little free cash can be very helpful.
To the extent that there are things you can cut current or future spending on, now’s the time. Your institution probably already has some travel freeze and possibly other freezes on things like training spending. If there’s licenses or cloud spend or support costs you could cut, even if it would cause inconvenience, these are things to start cutting.
The one that hurts here - and it’s the only one that really moves any needles in our teams, given our cost structures - is hiring. If your institution hasn’t already frozen hiring, and you do have a req open, well, it’s almost certainly time to close it. Yes, even if you’ve put a lot of work into it, yes even if you’ve been doing a lot of interviewing, yes, even if you’ve been waiting for years to be able to hire this position. This bites, because there are a lot of good people out on the market right now. But if drastic cuts do come, it’s generally last in first out, and hiring someone just to lay them off in a couple months is a terrible, irresponsible thing to do to someone.
Here especially, run everything by your manager before doing anything that would be hard to undo, as there may be local context specific to your situation that changes the advice above. Most of it is going to be pretty close to universal, though; they are standard “batten down the hatches” moves when a storm seems to be coming.
There are a lot of partnering efforts that can be done in the longer term which are extremely valuable - I have been meaning to call out University of Michigan partnering w/ Los Alamos on AI research - and I do think these efforts can be useful for a lot of our teams.
But that’s not an answer for today.
Right now we need extant funding lines that we and our researchers can tap into. Some of this might mean making sure alternate calls for funding proposals are sent to our researchers and users; it might mean following up with our best clients and maybe the second tier of clients to see what they’re considering applying for.
It would be very useful to be seen to be making an effort contributing to grantwriting efforts from these other funding sources. Go out of your way to volunteer to write text, generate some relevant paragraphs of your teams capability in the format that any such proposals need to be in, and share broadly.
If your team has ever done fee-for-service work with private sector or external clients, move mountains to see if you can to start bringing in a bit more business there - maybe training, maybe direct services, anything in your portfolio. Start with clients you’ve already worked with, and ask for referrals if they don’t have anything for you. Any trend upwards in revenue from these sources would be almost as meaningful as the actual amount of money involved, if it seems like the trend could continue. Similarly if you can do anything to help some of your clients land some private sector contract work.
And you’re going to have to toot your own horn here, and have your boss and so on do the same. Yes, that’s uncomfortable for us, and well, too bad. In times when a lot is going on, simply doing something good and worthwhile isn’t enough - you have to be seen to do it for it to have the needed impact.
Share your successes, however minor, in the form of useful information and help. “We’ve had good success helping researchers X, Y, Z submit to this alternate funding stream - we’re happy to show others how”; “Here’s how we’re re-connecting with our industry clients and succeeding in bringing in some additional revenue that way - does anyone have any other approaches that have worked for them? Happy to put together a document where we can all share what’s working”.
Some of our teams will suffer cuts. Others won’t. We don’t necessarily know which one we’ll be, so we have to be prepared for the worst.
Part of the preparation involves having a short clear presentation about the work we’re doing, how tied it into other parts of the ecosystem, how costs have already cut, and (as above) how it is an essential, highly leveraged, cost effective way to do the things the institution needs.
The goal of that work is to justify the existence of the team.
Even if the justification for the team as a whole is accepted, people are going to pouring over books looking for anything that can be cut. Again, in our line of work, the big expenses are equipment that’s already been bought, and people’s salaries. That means individuals are going to be looked at for cuts.
As a manager or lead, it’s our responsibility to have ready information about each of our team members to make the case for them. This is sort of the same justification above for the team, but at an individual level:
I really hope you don’t find yourself going through your team roster with someone who is seriously looking for cuts, but if you do the best thing you could have done for them is to have put together the strongest possible case, in the language of the immediate priorities of the organization, for each of them. The time to do that is now, when you don’t and hopefully won’t need it. Because if you wait until you do need it, it will be too late to do a good job.
Again, your boss will be able to give good feedback with important local context that I don’t have.
This is a stressful time for your researcher clients, and for your team members. Being in contact with the researchers and helping with you can is good and important work, but our primary responsibility is for our team members.
There’s not much we can tell them - or anyone - about the future. But we can be there for them to hear their concerns, we can reassure them that we’re doing our best and that we’re looking out for the team, and we can direct them into the areas where the institution most needs them.
If you’re not already doing one-on-ones with team members, this is the time to start, so they have a regular check-in with you. Don’t promise anything you can’t guarantee, but be a steady presence, a calm voice, and a ready ear.
Focus efforts where the institution most needs them focussed right now, have the team (and ideally individuals on your team) seen to be valuable and contributing during this challenging time, be there for your team members, and that’s about all anyone can ask of you.
If you need to ask some advice, or even just to vent, or have taken some action you think others would benefit hearing from and want to share, you can email me at any time (jonathan@researchcomputingteams.org), or hit reply to this email, or schedule a short call with me . I know this is a tough time and I’m happy to talk.
And now the roundup (all on happier topics!)
Over at Manager, Ph.D., I talked **last week **about influencing people, and how being right is just table stakes.
The roundup covered posts on:
The Anthropic Economic Index - Anthropic
I know I’ve brought this point up in the past - but whether we as technologists use LLMs in our work is not a super interesting question. The bigger question is, as people who support research, how are we adjusting to the situation where the researchers and trainees are adopting these tools pretty enthusiastically.
Anthropic recently released a blog post, paper, and dataset on the high-level tasks Claude is being used for:
One can question the task and categories - they follow O*NET which I’m sure is a perfectly sensible industrial classification but generate odd results for our purposes (bioinformatics is the top result under “office and administrative”; as with technical writing under “arts & media”).
But people, including presumably our clients, are using at least Claude for software development, data analysis, and scientific research quite a bit.
And this can be great! Maybe this lifts some of the pressure first-tier tech support (“what does this error message mean? Why won’t this run? How do I do X again?”). But sometimes it will replace an easy issue with a complicated one if the fix isn’t right.
It will also change how people interact with the software we produce, and the systems we set up. And that changes some of our incentives:
How are you seeing these tools being used, and how are you responding? Let me know - just hit reply, or email me at jonathan@researchcomputingteams.org
(One final note; apparently “general management” tasks are underrepresented in the Claude dataset, which disappoints me. One of the tasks that this sort of tool can absolutely and uncontroversially help us with is practicing tricky conversations which often hold us back as managers and leads, or be kind of a “rubber duck debugging” (but the duck talks back!) about interpersonal / interdepartmental situations.)
USRSE’25 Call for Submissions - 6-8 October, Philadelphia PA USA, Papers due 2 May
I stopped collecting conference information for the newsletter, but I’m happy to include a blurb for a worthwhile conference if asked! In this case it’s the US-RSE’s annual conference:
Whether you’re a data scientist, digital humanist, scientific programmer, software developer, or research software user, US-RSE is where people at the intersection of code and research come together. The USRSE’25 conference is your chance to connect with peers, mentors, and experts in the fast-growing world of research software. Don’t just take our word for it—100% of last year’s post-conference survey respondents said they would return and recommend the conference to others.
Information is available at the conference website; you can volunteer as a reviewer by filling out this form before April 1st.
Keygen is now Fair Source - Zeke Gabrielse, Keygen
I’m fascinated by “fair source” and other “source-available” licences, that potentially open sustainability options for research software by making the source available but reserving some rights (such as running it as a commercial service, for SSPL, or distributing a licence key for some of the fair source licenses). I suspect people would absolutely hate this initially, but I like the idea of having the transparency of source available (which I do think is important for science) while still making the tool saleable in some sense.
In this article Gabrielse, the author of software licensing tool Keygen, talks about the reasons for choosing the Fair Code License for his product.
The Sources of Researcher Variation in Economics - Huntington-Klein et al, SSRN 5152665
Same data, different analysts: variation in effect sizes due to analytical decisions in ecology and evolutionary biology - Gould et al., BMC Biology
Even faced with the same data, ecologists sometimes come to opposite conclusions - Cathleen O’Grady, Science
There’s been a flurry of “many analyst” projects since the first one, in 2018, in Psychology. They’re fascinating!
Weirdly, especially given how intrinsically interesting they are, how they connect with our work in particular, and the importance of the questions they’re asking for science in general, I’ve seen little to no discussion of them in the research software or research data science communities. I think this is a mistake.
In many of these papers, we see huge differences in results from practitioners in the field, including seeing a strong positive effect, strong negative effect, and no statistically significant effect. That’s unsettling!
As O’Grady points out, it’s not necessarily shocking, but some of the explanations are actually worrying in their own right:
The first question asked how the growth of blue tit chicks is influenced by competition with siblings in the nest. […]. Yet expertise plays an important role in winnowing down the choice of analyses, he says. Scientists already familiar with blue tits, for instance, would have a better idea about what analyses to run.
Given the data set in question and the analysis, this sounds dangerously close to “people most likely to know the expected answer get closer to the expected answer”, which is actually not super reassuring!
And here’s a point which strikes right to the heart of the software side of our business: in the economics one, which sees similar variations:
Across researcher demographics, occupation, and professional experience, there was no strong relationship between researcher background and either the level of the effect estimate they reported […]. **The only relevant difference we found is that the minority of researchers who used the R programming language were more likely to report outlier estimates than researchers who used Stata.
The reason inferred in the ensuing discussion on the econ internets around this is that Stata’s widely commented on rigid defaults and economics focus provide guardrails here, which R lacks being a more general purpose framework with many contributed open-source packages, where you can do anything you want multiple ways.
So - is that freedom bad, maybe? As tools and infrastructure providers, to what extent do we owe researchers enforced good defaults and a clear “best-practices” path for standard analyses (which, let’s be honest, is most of the analyses done)? Or at least mechanisms for user communities to define and share those for themselves? Or are our tools strictly caveat emptor even if users get tripped up by bad defaults? Because that makes it easier to do different things?
I don’t know the answer to these questions, but they’re relevant, important, and fascinating, and I think they merit more discussions in our communities than I see them get.
And some other questions or discussion points immediately leap to mind:
The Ultra-Scale Playbook: Training LLMs on GPU Clusters - Tazi, Mom, Zhao, Nguyen, Mekkouri, Werra, and Wolf, HuggingFace
A lot of research computing teams are wrestling with helping researchers train models on multiple GPUs or multiple nodes, when both the staff and researchers are new to it. There’s a lot of technology and it’s been moving very fast!
This extremely comprehensive resource by Tazi et al goes very deeply into not just the approaches but they problems they aim to address. Maybe most remarkably, it’s illustrated by a data set of 4000 performance results varying all the different parameters and methodologies they’re talking about.
The material is very dense, and many readers will benefit from following links when given to other discussions of the topic - but it’s the best all-in-one overview of larger-scale training I’ve seen so far.
I’m a little late to this, but just noticing the SimOps framework and initial certification.
I think there’s both entirely practical reasons to want increasing professionalism and standardization around best practices for running simulation workloads the way we’ve been seeing with MLOps, and reasons to want to visibly be doing something in this space to try to shed the (unfair) image we have of being the more old fashioned, less sophisticated cousins of our colleagues in cloud and AI.
I like the framing and the general idea, but haven’t had the time to go through any of the materials. Does any of it seem useful/good? (Even if it’s obvious, it can be useful and good to have this stuff promulgated somewhere as best practices and standards).
Glen Lockwood wrote a short bit about some of his work with SimPy for discrete event simulations for modelling, e.g., HPC system reliability.
For your colleagues or users who have now got a grasp on the basics of the terminal, job control might be a good first intermediate step.
Kind of late to this, but the Google office suite (motto: “Eh, good enough, I guess”) of docs, slides, and drawings finally supports markdown. You have to turn it on (per document?) in Tools > Preferences, after which you can “copy as Markdown”, “paste as markdown”, import/export markdown…
Speaking of markdown, here’s using Mermaid to show traces in markdown/github/gitlab/etc.
Fun project, VeriNum, looking at formally verified numerical methods/primitives - cool that it’s being done, but also a demonstration of how hard it is for even simple things like ‘parallel dot product’.
Quick overview of loaders in the context of trying a different glibc. On a separate note, I’ve been looking for ages into material that goes super deep into the whole linker and loader subsystem, LMK if you have a favourite reference.
Fun computational discussion of an extension to Fermat’s Last Theorem, Beal’s Conjecture.
The UX of lego interface panels.
The Ross Ibarra lab’s very generous repository of successful grant, fellowship, and job applications, with links to some of the very very few other similar efforts out there.
A minecraft server written in COBOL.
Conway’s game of life in a manpage with groff.
Turning R scripts and Google sheets into mobile apps or mobile-friendly webapps, respectively.
And that’s it for another week. If any of the above was interesting or helpful, feel free to share it wherever you think it’d be useful! And let me know what you thought, or if you have anything you’d like to share about the newsletter or stewarding and leading our teams. Just email me, or reply to this newsletter if you get it in your inbox.
Have a great weekend, and good luck in the coming week with your research computing team,
Jonathan
Research computing - the intertwined streams of software development, systems, data management and analysis - is much more than technology. It’s teams, it’s communities, it’s product management - it’s people. It’s also one of the most important ways we can be supporting science, scholarship, and R&D today.
So research computing teams are too important to research to be managed poorly. But no one teaches us how to be effective managers and leaders in academia. We have an advantage, though - working in research collaborations have taught us the advanced management skills, but not the basics.
This newsletter focusses on providing new and experienced research computing and data managers the tools they need to be good managers without the stress, and to help their teams achieve great results and grow their careers. All original material shared in this newsletter is licensed under CC BY-NC 4.0. Others’ material referred to in the newsletter are copyright the respective owners.
This week’s new-listing highlights are below in the email edition; the full listing of 381 jobs is, as ever, available on the job board.
