In a unique experiment blending generative AI tools, former OpenAI co-founder Andrej Karpathy used the Wall Street Journal’s front page to produce a music video on August 1, 2024. The process involved several AI applications to achieve the final product.
First, Karpathy copied the entire front page of the WSJ into the AI language model Claude. Claude generated multiple scenes and provided visual descriptions for each. These descriptions were then fed into Ideogram AI, an image generation tool, to create corresponding visuals.
Next, the generated images were uploaded into RunwayML’s Gen 3 Alpha to convert each image into a 10-second video segment. Concurrently, Claude was tasked with generating lyrics that captured the essence of the day’s news. These lyrics were inputted into Suno AI to produce the accompanying music.
The final step involved stitching the video segments and music together using iMovie. The project showcases the potential of generative AI tools to create multimedia content in a seamless and innovative manner.
Andrej Karpathy has been exploring various generative AI tools to create videos. Recently, he combined these tools to produce visual stories. As a demonstration, he took the opening sentences of Pride and Prejudice and turned them into a video.
Andrej Karpathy recently announced the launch of his new venture, Eureka Labs, an AI+Education company dedicated to creating an AI-native learning environment.
Eureka Labs aims to improve education by integrating generative AI with traditional teaching methods. Karpathy, who previously held key positions at OpenAI and Tesla, described Eureka Labs as “a new kind of school that is AI native.”
Karpathy’s recent experimentation with generative AI tools may be part of his effort to develop new learning tools. These tools could enable students to learn through AI-generated videos and music, enhancing their learning experiences.
