Crystalrohr

Crystalrohr's platform provides automatic captions that make it easier for people with visual impairments to know what's happening in video content. It is now possible for anyone with a visual impairment to ask about what is happening in a video scene without needing a human guide, additionally, the in-video search feature (not yet in production) should allow users to find specific moments within a video, making it easier to navigate through long videos. With the help of Crystalrohr, people with vision disabilities can now enjoy the diverse content available on YouTube and experience the power of connection and creativity that the platform offers. So we know that at this moment AI inferences do have some degree of hallucinations, so we have a StackOverflow kind of feature to help clarify moments in videos for the blind that adds the human touch to our process of making videos more accessible.

The Canvas API gets an image off a playing video, which gets sub-divided into a 4 by 8 block segment, next we get the dominant color of each block and store them as an array in memory, we then repeat the process for the next incoming frame and sub-divide just like before, and get the dominant colors, then we make a comparison for the next step. Once we get an average of the color change using the CIEDE2000 formula read more on color difference, we ascertain above a certain threshold whether a scene has changed and then request an inference from the Blip 2 API, which is an advanced vision-language model hosted in Lavis Repository to get details about a video scene which is then interpreted back to the user by the best TTS on the market from Eleven Labs.