OmniQuery: Helping You Find Answers in Your Memories

📖 Introduction

These days, we capture lots of moments in our lives through pictures, screenshots, and videos. This could be to remember important info, save special times, or just catch funny moments. These "memory captures" show parts of your long-term memories from the past. They include specific experiences and details from that time. These memory stories are super important for answering questions about your own life, like "What social activities did I do at CHI 2024?" But on their own, single memory captures often miss key details to directly answer complex personal questions.

For example, a capture from a CHI 2024 party might not clearly say it was part of that event. To answer questions like these, we need to pull out and tie together context clues usually hidden across many captures. By linking captures mentioning "CHI 2024" and looking at their details, we can figure out when you were at the conference and connect social events from that time to better answer the question.

🔍 Related Work

Lots of research has looked at ways to enhance personal memories, answer questions across different types of information, and use context clues. OmniQuery learns from this work, especially around boosting what users remember and making question answering systems better. Current AI tools often only work with one type of info lookup. But OmniQuery aims to solve this by bringing together scattered context clues.

📅 Diary Study

To better grasp real-world info needs, we did a month-long diary study. Participants wrote down actual memory questions and background details, giving us 299 user queries total. Analyzing these queries, we found three main types - direct content searches, context filters, and mixed queries. Our research showed 74.4% of queries needed more than just direct content matching, integrating context too.

📊 Context Categories

Based on feedback, we made a system to classify context types into three big groups: atomic context, combined context, and semantic knowledge. Atomic context comes from a single capture, like time, place, people. Combined context is formed by linking multiple captures, like "ski trip with lab mates." Semantic knowledge is broader info built up over time, not tied to specific memories.

🛠️ OmniQuery: Enhancing Captured Memories

OmniQuery was designed based on our diary study categories to boost the context in your captured memories. This supports more complex personal questioning. The system works in these steps:

1. Structure captured memories: First process contents and tag atomic context types.
2. Identify combined context: Use sliding window analysis to spot potential combined context links.
3. Infer semantic knowledge: Draw out higher-level semantic info by looking across many captures.

❓ OmniQuery: Question Answering System

OmniQuery uses a Retrieval-Augmented Generation (RAG) setup to effectively handle tons of captured memories. Given a user's question, it first enhances the query by rewriting it to include context clues. Then it searches related captures from the enhanced data and uses a large language model (LLM) to generate a full answer.

📊 Experiment Results

We tested OmniQuery with users. Results showed it answered user questions with 71.5% accuracy. In direct comparisons, OmniQuery outperformed or tied baseline systems 74.5% of the time. User feedback said OmniQuery excelled at handling complex personal memory queries, especially those needing context filtering or mixing.

🌟 Discussion and Future Work

OmniQuery demonstrated big potential for personal memory question answering but still faces challenges. Better integrating different data sources could deepen context understanding. Ensuring user privacy is also important. Future research could explore multimodal interactions, enriching memory databases, and strengthening emotional reasoning abilities.

📚 References

1. Li, J., Zhang, Z., & Ma, J. (2024). OmniQuery: Contextually Augmenting Captured Multimodal Memory to Enable Personal Question Answering. arXiv:2409.08250.
2. Tulving, E. (2002). Episodic Memory: From Mind to Brain. Annual Review of Psychology, 53, 1-25.
3. Lewis, P., Perez, E., et al. (2021). Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. arXiv:2005.11401.
4. Zulfikar, W., Chan, S., & Maes, P. (2024). Memoro: Using Large Language Models to Realize a Concise Interface for Real-Time Memory Augmentation. In Proceedings of the CHI Conference on Human Factors in Computing Systems.
5. Yang, Z., Qi, P., et al. (2018). HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering. In Conference on Empirical Methods in Natural Language Processing.