Szczegóły publikacji
Opis bibliograficzny
Enhancing embedding representations for large language models: a comparative study of the hashing trick and one-tot encoding / Agata Kozina, Michał PIKUS // W: Advances in Computational Collective Intelligence : 17th International Conference, ICCCI 2025 : Ho Chi Minh City, Vietnam, November 12–15, 2025 : proceedings , Pt. 1 / eds. Ngoc Thanh Nguyen, [et al.]. — Switzerland : Springer, cop. 2026. — ( Communications in Computer and Information Science ; ISSN 1865-0929 ; CCIS 2747 ). — ISBN: 978-3-032-10201-0; e-ISBN: 978-3-032-10202-7. — S. 415–426. — Bibliogr., Abstr. — Publikacja dostępna online od: 2025-11-08
Autorzy (2)
- Kozina Agata
- AGHPikus Michał
Słowa kluczowe
Dane bibliometryczne
| ID BaDAP | 165497 |
|---|---|
| Data dodania do BaDAP | 2026-02-18 |
| DOI | 10.1007/978-3-032-10202-7_28 |
| Rok publikacji | 2026 |
| Typ publikacji | materiały konferencyjne (aut.) |
| Otwarty dostęp |  |
| Wydawca | Springer |
| Konferencja | International Conference on Computational Collective Intelligence: Semantic Web, Social Networks and Multiagent Systems 2025 |
| Czasopismo/seria | Communications in Computer and Information Science |
Abstract
This paper investigates modifications to traditional embedding representations in large language models (LLMs) through two approaches: the hashing trick and one-hot encoding. We conducted experiments on three families of models GPT-2, facebook/opt-1.3b and Microsoft/phi-4-mini evaluating three configurations: original embeddings, hash-based embeddings, and one-hot embeddings. Our evaluation considers metrics such as evaluation loss, perplexity, and training time, using the Wikitext/wikitext-2-raw-v1 dataset. The results indicate that, while the original embeddings yield the best performance in terms of predictive accuracy and training efficiency, the modified embeddings offer potential scalability benefits. Notably, the hash-based approach outperforms one-hot encoding by a small margin, albeit at a higher computational cost. We conclude by discussing the trade-offs inherent in these methods and propose directions for future work to optimize the balance between efficiency and accuracy.
