Does an increase in Anthropic's benchmark index accurately represent an equivalent rate of capability increase in well specified tasks?

As a SWE I've yet to find a reliable benchmark to compare different SWE capabilities between models. This year OpenAI stopped using SWE-Bench-Verified and a few months later ditched SWE-Bench-Pro. A large portion of the questions were broken or had been trained on.

I hesitated to comment as I've no background of Animal welfare campaigning and only learnt about most of these projects this summer myself. Given the downvoting and lack of responses though my reply is probably better than nothing and might encourage someone with more knowledge on this to improve upon my response.

The people and orgs you mention have the same aims as yourself, to reduce or eliminate the suffering factory farmed animals face and no doubt are putting a lot of their time and energy in to trying to accomplish this. I get that this is a repost intended for a different audience but I think some editing to make this more of a debate than a call to resistance would have been more fruitful. 

Your analysis of who is behind them seems accurate to me and you're posting in the right place to have the people and orgs you mention see this post but you haven't really engaged with their ideas, made a case for why they're bad, and in some cases may be misrepresenting these initiatives.

I had a couple conversations with Every's Executive Director a few years ago in the wake of one of their other donation matches that a bunch of EAs participated in, and, based on my understanding developed then, my guess is that someone at Every.org just had the thought "How about we do an animal welfare donation match this giving season. A lot of people from the effective altruism community have participated in our past donation matches and I bet many of them would participate in this one, making it a successful way to attract more donors to our platform and get more people donating."

Are you aware of whether the match was met last time? For tax reasons and because these aren't my usual charities this donation wouldn't be optimal for me but if this target is unlikely to be met it would be worth my time to research the charities and loss of tax return income in order to consider this.

"It's nice that you are wanting to donate your money, including to these nonprofits that are very cost-effective, but I think it would be better if you donated your money to this other nonprofit instead that will use the money to do even more good. You could also donate to both if you want and that'd be completely fine, but in either case I am going to respectfully decline your match offer in order to allocate my limited donation budget to where I think it will do the most good."

It may be better to reframe this as: "this is great, please do more of this could you consider charity x and y for a future round." rather than a "I'm not donating this time please consider charity x and y instead"

I think the greater impact is to encourage them towards any EA charities rather than try to get them to replace one EA charity with our own personal preferences. I'm worried that them getting dozens of conflicting messages telling them to use different charities could even encourage them away from EA charities.
 

Note: If you think that one of the animal nonprofits is only, say, 50% as cost-effective as the best giving opportunity you can identify, I'd discourage you from giving to that animal nonprofit in this match despite the naive expected value calculation (3*50%=150%) suggesting that donating to the animal nonprofit with the match will lead to more good than donating to the best giving opportunity without a match. One reason I'd give for this is that I don't think that we should let large donors incentivize us to allocate our giving budgets to less cost-effective giving opportunities, since that seems like a bad incentiblve.

The opposite could be argued, that additional participation from small donors or publicity (such as this post) around the match scheme might encourage the large donor to allocate more of their giving budget to EA funds in future.

If you donate significant amounts through Gift Aid, it's worth making sure you're claiming your higher-rate relief, either by filing a Self Assessment or by asking HMRC to adjust your tax code.

How do you call up HMRC to adjust your tax code? Do you need to know how much you will make / donate in advance to do this?

I've been filling out self assessments each year to do this and it's been getting more difficult each year. 2 years ago the online repayments stopped going through and I needed to call up their helpline. last year the helpline couldn't help me and I needed to write a letter to HMRC.

It seems like there would be a lot of benefits to having public impact assessments. 

It verifies it's taken place, assures applicants a given talent programme is worth their time, gives talent programme orgs visibility on who's most effective and why, can be seen by other potential donors, and would allow the impact assessment process to be scrutinised and improved upon.

Maybe cG can't do this as making how they make donation decisions might corrupt their process. In this case the independent analysis fulfills a different purpose rather than being better or worse.

I think informing rejected candidates of how many people applied is another quick-win. The only time I've got figures on applicants is when I've got in to a fellowship or role and never when I've been rejected.
So far I've seen:
* Spar Projects with 1/50 applicants and 6/75 applicants accepted. (I've never got in to SPAR but have worked in an office with a SPAR mentor).
* Two fellowships where 20/140 and 15/170 were accepted respectively.
These were all unpaid positions (bar the last one which covered living expenses), I can only imagine applying for paid roles would be even more competitive.

I think having rejected candidates know how competitive these roles/courses/fellowships are would likely reduce the drop-out from the AIS talent pool so would be a win-win for both the applicants and the orgs.

The text is a bit hard to read when you zoom in on the images linked in this post.
The google drive link was high enough res, mostly commenting to make your work more accessible to people scanning the article.
 

Python & Fellowship Tests
Repost from blog (I'd appreciate any feedback)

This Autumn has been a bit of a mix of continuing with Arena, being sick, and applying for half a dozen fellowships and courses.

Claude and Astra's fellowships both required code signal's industry coding assessments. I did a week of preparation for both of them before taking the test but did a lot worse than I thought I would. Being relatively new to Python and not being able to code at high speed without making mistakes was my downfall with these. I'm generally not great at thinking fast be it coding or chess.

I did enjoy practising Python. It's very valuable given most the career switches I'm considering use it. It was refreshing to see this kind of test instead of algorithm and data structure tests, which I feel involve revising something you're not going to need for the job. I created some mock practice tests using LLMs that were excellent preparation for the real deal. I've also picked up some Anki flashcards for the most common Python functions, which should come in handy for next time.

I also did tests that involved writing essays about papers for Cooperative AI Research fellowship, MARS, LASR, Deepmind, ERA, and the 2nd stage of Astra. With the exception of the ERA test (really interesting paper on LLM coding capabilities), I didn't enjoy these. But they were probably valuable for fit testing; if I'm not enjoying reading papers and writing about them, AI safety research probably isn't going to be my cup of tea.

I've decided, for the time being, to stop applying for new fellowships. Joining a fellowship would be a great fit test in itself, but it's quite a large time commitment, especially when you include the x number of applications you'll need before getting accepted to one. I have limited time to do fit tests this year, so my priority now that I'm getting a sense that pure research isn't my thing, is to focus on finishing ARENA, start to explore career paths other than AI safety, and contribute to more open-source projects. I need to be careful not to use up all my time doing applications rather than getting my hands dirty with projects that will give a much better indication of my fit for a given career path.