Listen to resources from the AI Safety Fundamentals: Alignment course! https://aisafetyfundamentals.com/alignment
…
continue reading
The podcast of the Green Revolution. Join us for insightful conversations with experts and pioneers who are shaping the future of our planet.
…
continue reading
Carry the Two pulls back the curtain to reveal the mathematical and statistical gears that turn the world. We’re the show for people who enjoy discovering hidden elements that impact our lives in the most unexpected ways, and math is certainly one of those! We are a curiosity-driven podcast that looks to find unique perspectives from the fields of mathematics and statistics. We use stories to convey how mathematical research drives the world around us, with each episode tackling a different ...
…
continue reading
This is a narrative travel podcast about a solo female backpacker who interviews strangers she meets while backpacking. Stories of adventure traveling like National Geographic, interview style like Fresh Air, and diverse/alternative storytelling like This American Life and Snap Judgement.
…
continue reading
This podcast features conversations featuring world-class innovators, investors and policy experts that took place at the annual SOSV Climate Tech Summit. The event is hosted by SOSV, a deep tech VC writing its first checks via its startup programs HAX (hard tech) and IndieBio (life sciences). SOSV has over 1,000 startups in portfolio and manages over $1.5 billion. Podcast Producer: Ben Joffe Podcast Summary: Written by gpt-4-turbo, edited by Ben Joffe Intro Voice: Cloned voice of Ben Joffe ...
…
continue reading
This paper explains Anthropic’s constitutional AI approach, which is largely an extension on RLHF but with AIs replacing human demonstrators and human evaluators. Everything in this paper is relevant to this week's learning objectives, and we recommend you read it in its entirety. It summarises limitations with conventional RLHF, explains the const…
…
continue reading
![Artwork](/static/images/128pixel.png)
1
Constitutional AI Harmlessness from AI Feedback
1:01:49
1:01:49
Відтворити пізніше
Відтворити пізніше
Списки
Подобається
Подобається
1:01:49
This paper explains Anthropic’s constitutional AI approach, which is largely an extension on RLHF but with AIs replacing human demonstrators and human evaluators. Everything in this paper is relevant to this week's learning objectives, and we recommend you read it in its entirety. It summarises limitations with conventional RLHF, explains the const…
…
continue reading
This more technical article explains the motivations for a system like RLHF, and adds additional concrete details as to how the RLHF approach is applied to neural networks. While reading, consider which parts of the technical implementation correspond to the 'values coach' and 'coherence coach' from the previous video. A podcast by BlueDot Impact. …
…
continue reading
Join us as James chats with Charlie Cibrario Assereto, lecturer at Columbia University to learn all things climate policy related.James Atfield
…
continue reading
![Artwork](/static/images/128pixel.png)
1
Intro to Brain-Like-AGI Safety
1:02:10
1:02:10
Відтворити пізніше
Відтворити пізніше
Списки
Подобається
Подобається
1:02:10
(Sections 3.1-3.4, 6.1-6.2, and 7.1-7.5) Suppose we someday build an Artificial General Intelligence algorithm using similar principles of learning and cognition as the human brain. How would we use such an algorithm safely? I will argue that this is an open technical problem, and my goal in this post series is to bring readers with no prior knowle…
…
continue reading
This post is about language model scaling laws, specifically the laws derived in the DeepMind paper that introduced Chinchilla. The paper came out a few months ago, and has been discussed a lot, but some of its implications deserve more explicit notice in my opinion. In particular: Data, not size, is the currently active constraint on language mode…
…
continue reading
We show that the double descent phenomenon occurs in CNNs, ResNets, and transformers: performance first improves, then gets worse, and then improves again with increasing model size, data size, or training time. This effect is often avoided through careful regularization. While this behavior appears to be fairly universal, we don’t yet fully unders…
…
continue reading
![Artwork](/static/images/128pixel.png)
1
Eliciting Latent Knowledge
1:00:27
1:00:27
Відтворити пізніше
Відтворити пізніше
Списки
Подобається
Подобається
1:00:27
In this post, we’ll present ARC’s approach to an open problem we think is central to aligning powerful machine learning (ML) systems: Suppose we train a model to predict what the future will look like according to cameras and other sensors. We then use planning algorithms to find a sequence of actions that lead to predicted futures that look good t…
…
continue reading
This post tries to explain a simplified version of Paul Christiano’s mechanism introduced here, (referred to there as ‘Learning the Prior’) and explain why a mechanism like this potentially addresses some of the safety problems with naïve approaches. First we’ll go through a simple example in a familiar domain, then explain the problems with the ex…
…
continue reading
It would be very convenient if the individual neurons of artificial neural networks corresponded to cleanly interpretable features of the input. For example, in an “ideal” ImageNet classifier, each neuron would fire only in the presence of a specific visual feature, such as the color red, a left-facing curve, or a dog snout. Empirically, in models …
…
continue reading
Abstract: Existing techniques for training language models can be misaligned with the truth: if we train models with imitation learning, they may reproduce errors that humans make; if we train them to generate text that humans rate highly, they may output errors that human evaluators can't detect. We propose circumventing this issue by directly fin…
…
continue reading
The field of reinforcement learning (RL) is facing increasingly challenging domains with combinatorial complexity. For an RL agent to address these challenges, it is essential that it can plan effectively. Prior work has typically utilized an explicit model of the environment, combined with a specific planning algorithm (such as tree search). More …
…
continue reading
Gradient hacking is a hypothesized phenomenon where: A model has knowledge about possible training trajectories which isn’t being used by its training algorithms when choosing updates (such as knowledge about non-local features of its loss landscape which aren’t taken into account by local optimization algorithms). The model uses that knowledge to …
…
continue reading
This paper presents a technique to scan neural network based AI models to determine if they are trojaned. Pre-trained AI models may contain back-doors that are injected through training or by transforming inner neuron weights. These trojaned models operate normally when regular inputs are provided, and mis-classify to a specific output label when t…
…
continue reading
Chain-of-thought prompting has demonstrated remarkable performance on various natural language reasoning tasks. However, it tends to perform poorly on tasks which requires solving problems harder than the exemplars shown in the prompts. To overcome this challenge of easy-to-hard generalization, we propose a novel prompting strategy, least-to-most p…
…
continue reading
Right now I’m working on finding a good objective to optimize with ML, rather than trying to make sure our models are robustly optimizing that objective. (This is roughly “outer alignment.”) That’s pretty vague, and it’s not obvious whether “find a good objective” is a meaningful goal rather than being inherently confused or sweeping key distinctio…
…
continue reading
Previously, I argued that emergent phenomena in machine learning mean that we can’t rely on current trends to predict what the future of ML will be like. In this post, I will argue that despite this, empirical findings often do generalize very far, including across “phase transitions” caused by emergent behavior. This might seem like a contradictio…
…
continue reading
Using hard multiple-choice reading comprehension questions as a testbed, we assess whether presenting humans with arguments for two competing answer options, where one is correct and the other is incorrect, allows human judges to perform more accurately, even when one of the arguments is unreliable and deceptive. If this is helpful, we may be able …
…
continue reading
This article explains key drivers of AI progress, explains how compute is calculated, as well as looks at how the amount of compute used to train AI models has increased significantly in recent years. Original text: https://epochai.org/blog/compute-trends Author(s): Jaime Sevilla, Lennart Heim, Anson Ho, Tamay Besiroglu, Marius Hobbhahn, Pablo Vill…
…
continue reading
Alternative title: “When should you assume that what could go wrong, will go wrong?” Thanks to Mary Phuong and Ryan Greenblatt for helpful suggestions and discussion, and Akash Wasil for some edits. In discussions of AI safety, people often propose the assumption that something goes as badly as possible. Eliezer Yudkowsky in particular has argued f…
…
continue reading
Feedback is essential for learning. Whether you’re studying for a test, trying to improve in your work or want to master a difficult skill, you need feedback. The challenge is that feedback can often be hard to get. Worse, if you get bad feedback, you may end up worse than before. Original text: https://www.scotthyoung.com/blog/2019/01/24/how-to-ge…
…
continue reading
I’ve been obsessed with managing information, and communications in a remote team since Get on Board started growing. Reducing the bus factor is a primary motivation — but another just as important is diminishing reliance on synchronicity. When what I know is documented and accessible to others, I’m less likely to be a bottleneck for anyone else in…
…
continue reading
(In the process of answering an email, I accidentally wrote a tiny essay about writing. I usually spend weeks on an essay. This one took 67 minutes—23 of writing, and 44 of rewriting.) Original text: https://paulgraham.com/writing44.html Author: Paul Graham A podcast by BlueDot Impact. Learn more on the AI Safety Fundamentals website.…
…
continue reading
This introduces the concept of Pareto frontiers. The top comment by Rob Miles also ties it to comparative advantage. While reading, consider what Pareto frontiers your project could place you on. Original text: https://www.lesswrong.com/posts/XvN2QQpKTuEzgkZHY/being-the-pareto-best-in-the-world Author: John Wentworth A podcast by BlueDot Impact. Le…
…
continue reading
In this discussion from SOSV's 2024 EarthDay+ sessions (Apr 22-26, 2024), Bret Kugelmass, the CEO and co-founder of the nuclear fission startup Last Energy, which focuses on developing small nuclear reactors, talks about: The potential of nuclear power to provide abundant, inexpensive energy with minimal environmental impact. Kugelmass emphasizes n…
…
continue reading
![Artwork](/static/images/128pixel.png)
1
s04e06 - CVCs Doubling Down on Climate, with Microsoft Climate Innovation Fund, GS Ventures and Honda Innovations
1:00:12
1:00:12
Відтворити пізніше
Відтворити пізніше
Списки
Подобається
Подобається
1:00:12
In this panel discussion from SOSV's 2024 EarthDay+ sessions (Apr 22-26, 2024) moderated by Dr. Pae Wu of IndieBio and SOSV, corporate venture capitalists Brandon Middaugh from Microsoft's Climate Innovation Fund, Taehong Huh from GS Futures, and Aditya Sharma from Honda Innovations discussed their strategies for investing in climate tech startups.…
…
continue reading
![Artwork](/static/images/128pixel.png)
1
s04e05 - Should VCs Measure and Attribute Climate Impact, with Norrsken VC, Planet A, AENU, World Fund and Pale Blue Dot
1:04:17
1:04:17
Відтворити пізніше
Відтворити пізніше
Списки
Подобається
Подобається
1:04:17
In this panel discussion from SOSV's 2024 EarthDay+ sessions (Apr 22-26, 2024) focused on measuring and attributing climate impact and moderated by Hampus Jakobsson of Pale Blue Dot, investors from various climate tech-focused funds discussed the importance of impact measurement in startups and investment decisions. The challenges and methodologies…
…
continue reading
![Artwork](/static/images/128pixel.png)
1
s04e04 - After the Tumble: Climate VC In 2024 With Breakthrough Energy, Lowercarbon and SOSV
53:44
In this panel discussion from SOSV's 2024 EarthDay+ sessions (Apr 22-26, 2024) moderated by Tim De Chant, Senior Climate Reporter at TechCrunch, panelists Christina Karapataki from Breakthrough Energy Ventures, Shuo Yang from Lowercarbon Capital, and Duncan Turner from HAX and SOSV discussed the current state and future of climate tech venture capi…
…
continue reading
![Artwork](/static/images/128pixel.png)
1
s04e03 - Keeping the Heat On: Antora Energy’s Thermal Batteries Plan to Solve For Industrial Heat
57:39
In this discussion from SOSV's 2024 EarthDay+ sessions (Apr 22-26, 2024), Andrew Ponec, Co-founder and CEO of Antora Energy, discusses with Casey Crownhart, Climate Reporter at MIT Technology Review, the challenges and innovations in industrial heating. Antora Energy is electrifying heavy industry with thermal energy storage and raised over 200 mil…
…
continue reading
![Artwork](/static/images/128pixel.png)
1
s04e02 - Dressing for Climate Success: Three Startups Decarbonizing Fashion, with unspun, Smartex and Gozen
1:00:26
1:00:26
Відтворити пізніше
Відтворити пізніше
Списки
Подобається
Подобається
1:00:26
In this panel discussion from SOSV's 2024 EarthDay+ sessions (Apr 22-26, 2024) moderated by Dr. Sabriya Stukes, Chief Scientific Officer at SOSV's IndieBio New York program, founders Beth Esponnette of unspun, Gilberto Loureiro of SMARTEX, and Onur Eren of GOZEN discussed innovative approaches to sustainable fashion. The conversation focused on lev…
…
continue reading
![Artwork](/static/images/128pixel.png)
1
s04e01 - Can Form Energy meet the rapidly growing demand of grid-level, long-duration energy storage?
1:01:13
1:01:13
Відтворити пізніше
Відтворити пізніше
Списки
Подобається
Подобається
1:01:13
In this discussion from SOSV's 2024 EarthDay+ sessions (Apr 22-26, 2024), the CEO and co-founder of Form Energy, Mateo Jaramillo, talks about: Form Energy's advancements in energy storage, with their unique iron-air battery technology designed for multi-day energy storage, which is more cost-effective and durable than traditional lithium-ion batter…
…
continue reading
I am approaching the end of my AI governance PhD, and I’ve spent about 2.5 years as a researcher at FHI. During that time, I’ve learnt a lot about the formula for successful early-career research. This post summarises my advice for people in the first couple of years. Research is really hard, and I want people to avoid the mistakes I’ve made. Origi…
…
continue reading
This week James chats with Tom Herman, CEO of Meta-Carbon, a company focused on empowering brands to commit authentically to sustainability while also maximizing their revenue to help make sustainability more appealing to companies.James Atfield
…
continue reading
The next four weeks of the course are an opportunity for you to actually build a thing that moves you closer to contributing to AI Alignment, and we're really excited to see what you do! A common failure mode is to think "Oh, I can't actually do X" or to say "Someone else is probably doing Y." You probably can do X, and it's unlikely anyone is doin…
…
continue reading
We took 10 years of research and what we’ve learned from advising 1,000+ people on how to build high-impact careers, compressed that into an eight-week course to create your career plan, and then compressed that into this three-page summary of the main points. (It’s especially aimed at people who want a career that’s both satisfying and has a signi…
…
continue reading
![Artwork](/static/images/128pixel.png)
1
Working in AI Alignment
1:08:44
1:08:44
Відтворити пізніше
Відтворити пізніше
Списки
Подобається
Подобається
1:08:44
This guide is written for people who are considering direct work on technical AI alignment. I expect it to be most useful for people who are not yet working on alignment, and for people who are already familiar with the arguments for working on AI alignment. If you aren’t familiar with the arguments for the importance of AI alignment, you can get a…
…
continue reading
This post summarises a new report, “Computing Power and the Governance of Artificial Intelligence.” The full report is a collaboration between nineteen researchers from academia, civil society, and industry. It can be read here. GovAI research blog posts represent the views of their authors, rather than the views of the organisation. Source: https:…
…
continue reading
Most conversations around the societal impacts of artificial intelligence (AI) come down to discussing some quality of an AI system, such as its truthfulness, fairness, potential for misuse, and so on. We are able to talk about these characteristics because we can technically evaluate models for their performance in these areas. But what many peopl…
…
continue reading
The UK recognises the enormous opportunities that AI can unlock across our economy and our society. However, without appropriate guardrails, such technologies can pose significant risks. The AI Safety Summit will focus on how best to manage the risks from frontier AI such as misuse, loss of control and societal harms. Frontier AI organisations play…
…
continue reading
Generative AI allows people to produce piles upon piles of images and words very quickly. It would be nice if there were some way to reliably distinguish AI-generated content from human-generated content. It would help people avoid endlessly arguing with bots online, or believing what a fake image purports to show. One common proposal is that big c…
…
continue reading
We’ve released a paper, AI Control: Improving Safety Despite Intentional Subversion. This paper explores techniques that prevent AI catastrophes even if AI instances are colluding to subvert the safety techniques. In this post: We summarize the paper; We compare our methodology to the methodology of other safety papers. Source: https://www.alignmen…
…
continue reading
Research in mechanistic interpretability seeks to explain behaviors of machine learning (ML) models in terms of their internal components. However, most previous work either focuses on simple behaviors in small models or describes complicated behaviors in larger models with broad strokes. In this work, we bridge this gap by presenting an explanatio…
…
continue reading
Using a sparse autoencoder, we extract a large number of interpretable features from a one-layer transformer. Mechanistic interpretability seeks to understand neural networks by breaking them into components that are more easily understood than the whole. By understanding the function of each component, and how they interact, we hope to be able to …
…
continue reading
By studying the connections between neurons, we can find meaningful algorithms in the weights of neural networks. Many important transition points in the history of science have been moments when science “zoomed in.” At these points, we develop a visualization or tool that allows us to see the world in a new level of detail, and a new field of scie…
…
continue reading
Widely used alignment techniques, such as reinforcement learning from human feedback (RLHF), rely on the ability of humans to supervise model behavior—for example, to evaluate whether a model faithfully followed instructions or generated safe outputs. However, future superhuman models will behave in complex ways too difficult for humans to reliably…
…
continue reading
Reinforcement learning from human feedback (RLHF) has emerged as a powerful technique for steering large language models (LLMs) toward desired behaviours. However, relying on simple human feedback doesn’t work for tasks that are too complex for humans to accurately judge at the scale needed to train AI models. Scalable oversight techniques attempt …
…
continue reading
Join James as he discusses the burgeoning relationship between venture capital and climate change investment as well as the importance of investing in sustainable companies.James Atfield
…
continue reading
Did you know you can have your gas-fueled car converted into a plug-in hybrid in as fast as an eight-hour turnaround? Believe it or not, it is possible! Join James as he discusses where EV and Hybrid technology and production are heading, from a state level to a global one, with Blue Dot Motors CEO and Founder, Tom Gurski.…
…
continue reading
James chats with Wendy Hapgood, Professor of Sustainability Management at Columbia University, to learn about her wildlife conservation nonprofit organization Wild Tomorrow.James Atfield
…
continue reading
If you loved episodes one and two of my new audio adventure A Race Around the World: Based on the True Story of Nellie Bly and Elizabeth Bisland, then head over to these feeds linked below. New episodes are out and you can listen to the full series at these links. Listen here -----> Apple Podcasts Listen here -----> Spotify Thank you so much for li…
…
continue reading