Sha∫hank's website

[PROJ] Ramukaka

Aug 06, 2023

Really Advanced Most Useful Kollaborative Assistant Kollaboratively Assisting


There was an idea…

Imagine an assistant on your device. Which you don’t have to request to do tasks, but it just does.

Without prompting. Without a wake word.

Always working for you, always in the the background. Like the best of butlers. And when you need something, you just have to “glance” ;) at it. And it is there.

Vision (V.I.S.I.O.N?)

An immersive holographic UI with information floating around you in a dynamic graph of importance, with the most relevant and useful information bubbling up and the lower priority ones sinking and vanishing away.

We are getting there with the Apple Vision Pro, but not in a 1 day hack. We are limited to primitive screen for now. So, we have a dedicated screen, let’s say.

The actual idea, for sure this time

An always active Spotlight Search essentially. Imagine you have a second display always displaying a bunch of widgets and active context. This secondary display is connected to your primary device, say kept just beside the primary display. So the widgets are literally half a head turn away. They have the standard weather, calendar, time, etc widgets to start with.

Now, Ramukaka is always active, always listening. The first version will be “listening” to all your key presses. A key-logger, yes. And sending all the click stream data to the internal processor of Ramukaka.

Which is on-device. Or near-device at-least. Not going away into the internet.

The internal processor is where the magic happens. An on-device LLM decodes the key presses to build a context of what you are working on right now. And it is able to power some integrations like:

The idea boils down to removing the hops of a wake-word or a query input because if you are working or operating a device for atleast a while, it should be fairly easy to build context on what you want to search for looking at your prior interactions

Implementation Bullets


Architecture Overview

Getting to a Minimum Viable Product

We (the royal version) will use Streamlit to build the UI. Because we do not have the necessary frontend experience to build a slick moving-graph UI. What we do have experience in is backend systems, so we will stick to Python. And concentrate on a crucial part of the working - a dynamic scoring logic for content pieces.

Later, we will work on integrating with an LLM for parsing the input keyboard stream. Need to figure out the appropriate prompts and such.

The Decay Score

The scoring of the items needs to be dynamic, in the sense we don’t want to go back to the recommendation engine every second to fetch a new set of scores for all the items. The score defines how “big” an item will be in the moving-graph UI, essentially the weight of the node. And hence needs to have smooth increments.

Hence, the recommendation engine is not only going to predict a score for all the items but also a set of decay parameters for each item. These decay parameters are used to build a decay curve on the time axis.

The decay curves might look like so for 4 content pieces:

Decay Score Graphs

When you overlay the curves, at any given point we get a rank order of the items. As the time progresses onwards, the final ranking will changes, as the curves move relative to each other.

Some items can have high periodicity, some may have high priority but only for a limited time, some are lower priority background items best suited for fallback. All of these can be encoded with a score and a set of decay parameters to construct the decay curve.

A screen-shot of Ramukaka

Above the horizontal line is the actual display. The font size is mimicking the importance of information. The scene here is, we have Ramukaka running and are chatting about going to Turkey. So Ramukaka has surfaced what it thinks is relevant - the capital of Turkey.

Ramukaka v0.1

A screen-capture of Ramukaka

Ramukaka v0.1 GIF

Like, Share and Follow for updates on Ramukaka.