Audio-Based Assistive Device for Boxed Product Detection for The Visually Impaired
Article Sidebar
Published:
Apr 30, 2025
Keywords:
Object Detection, Visually Impaired, SSD Mobilenetv2, Audio, Raspberry Pi, Ultrasonik Sensor
Main Article Content
Abstract
Visually impaired individuals face challenges in recognizing and distinguishing boxed
products, which are typically identified visually. This study aims to develop an audio
based assistive system for detecting boxed product packaging to enhance the
independence of visually impaired individuals. The system utilizes the Single Shot
Multibox Detector (SSD) MobileNetV2 algorithm for object detection and classification,
with the Raspberry Pi 4 Model B as the main microcontroller and the Raspberry Pi
Camera OV5647 for image capture. The HC-SR04 ultrasonic sensor is used to measure
the distance of objects from the user, while the output information is provided through
wireless earphones in the form of audio. Experimental results show that the system can
detect and classify boxed products with an accuracy of 95%, precision of 96.8%, recall of
98%, and an F1-Score of 97.4%. This system provides real-time audio feedback, assisting
visually impaired individuals in identifying products without the need for tactile
exploration.
Article Details
How to Cite
[1]
M. D. Yoni and D. Yendri, “Audio-Based Assistive Device for Boxed Product Detection for The Visually Impaired”, chipset, vol. 6, no. 01, pp. 68-75, Apr. 2025.
Section
Articles
This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.
References
Kementerian Kesehatan Republik Indonesia. (2018). Infodatin: Gangguan penglihatan. Jakarta: Pusat Data dan Informasi Kementerian Kesehatan Republik Indonesia.
Yogi, M., Sari, M. I., & Handayani, R. (2023). Sistem Deteksi Teks Pada Kemasan Produk Makanan Menggunakan Metode Deep Learning Berbasis Raspberry Pi. eProceedings of Applied Science, 10(4).
Rahma, L., Syaputra, H., Mirza, A. H., & Purnamasari, S. D. (2021). Objek Deteksi Makanan Khas Palembang Menggunakan Algoritma YOLO (You Only Look Once). Jurnal Nasional Ilmu Komputer, 2(3), 213–232.
Saputra, R. D., & Fudholi, D. H. (2023). Model Mobile untuk Deteksi Objek pada On-Shelf Availability Produk Retail. AUTOMATA, 4(2).
Hidayat, Asep AS, Suwandi, Ate. (2013). Pendidikan anak berkebutuhan khusus tunanetra: peserta didik berkebutuhan khusus dengan hambatan penglihatan dalam perspektif pendidikan dan layanan (Cet. Ke-1). Jakarta: Luxima.
Pramana, E. (2023, Februari 11). Bantu Tunanetra, Ilmuwan Ciptakan Tongkat dengan Teknologi AI. Diambil kembali dari JawaPos.com: https://www.jawapos.com/teknologi/01435889/bantu-tunanetra-ilmuwan-ciptakan-tongkat-dengan-teknologi-ai
Klimchuk, M. R., & Krasovec, S. A. (2007). Desain kemasan: Perencanaan merek produk yang berhasil mulai dari konsep sampai penjualan. Penerbit Erlangga.
Nilsson, N. J. (1998). Artificial intelligence: a new synthesis. Morgan Kaufmann.
Yansens, C. Apa Itu Artificial Intelligence (AI)? Cara Kerja, Komponen, Jenis, Tujuan. Diambil kembali dari Tanya Digital: https://tanyadigital.com/artificial-intelligence-adalah/
Coelho, L. P., & Richert, W. (2015). Building machine learning systems with Python. Packt Publishing Ltd.
Ahmad, A. (2017). Mengenal Artificial Intelligence, Machine Learning, Neural Network, dan Deep Learning. Yayasan Cahaya Islam, Jurnal Teknologi Indonesia.
Batubara, N. A., Awangga, R. M., & Pane, S. F. (2020). Perbandingan Faster R-CNN dengan SSD Mobilenet Untuk Mendeteksi Plat Nomor. Kreatif Industri Nusantara, 9–10.
Le, H. H. (2015). Division Intro. Retrieved from Academia: https://www.academia.edu/27143145/Division_intro
Primartha, R. (2018). Belajar Machine Learning Teori dan Praktik. Bandung: Informatika Bandung, 10, 20–30.
Wahyu, R. (2019, Juli 6). Deep Learning Object Detection Menggunakan Tensorflow Dan Metode Convolutional Neural Network (CNN). Diambil kembali dari medium.com: https://medium.com/@rismitawahyu/deep-learning-object-detection-menggunakan-tensorflow-dan-metode-convolutional-neural-network-cnn-9c09267474e3
Saha, S. (2018, Desember 15). A Comprehensive Guide to Convolutional Neural Networks — the ELI5 way. Diambil kembali dari SaturnCloud: https://saturncloud.io/blog/a-comprehensive-guide-to-convolutional-neural-networks-the-eli5-way/
Zufar, M., & Setiyono, B. (2016). Convolutional neural networks untuk pengenalan wajah secara real-time. Jurnal Sains Dan Seni ITS, 5(2), A72–A77.
Liu, W., Anguelov, D., Erhan, D., Szegedy, C., Reed, S., Fu, C. Y., & Berg, A. C. (2016). SSD: Single Shot Multibox Detector. In Computer Vision–ECCV 2016, 21–37. Springer.
Liu, C., Anguelov, D., Erhan, D., Szegedy, C., Reed, S., Fu, C. Y., & Berg, A. C. (2016). SSD: Single Shot MultiBox Detector. Lecture Notes in Computer Science, 9905, 1–17. doi: 10.1007/978-3-319-46448-0_2
Ramalingam, B., Elara Mohan, R., Balakrishnan, S., Elangovan, K., Félix Gómez, B., Pathmakumar, T., ... & Baskar, C. (2021). Stetro-deep learning powered staircase cleaning and maintenance reconfigurable robot. Sensors, 21(18), 6279.
Badgujar, W. B. (2021, Juni 17). Creating MobileNets with TensorFlow from scratch. Diambil kembali dari medium.com: https://medium.com/@sumeetbadgujar/creating-mobilenets-with-tensorflow-from-scratch-c34ec79a59d2
Howard, A. G., Zhu, M., Chen, B., Kalenichenko, D., Wang, W., Weyand, T., ... & Adam, H. (2017). Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:1704.04861.
Team, P. (t.thn.). MOBILENET V2. Diambil kembali dari pytorch.org: https://pytorch.org/hub/pytorch_vision_mobilenet_v2/
Sandler, M., Howard, A., Zhu, M., Zhmoginov, A., & Chen, L. C. (2018). Mobilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE conference on computer vision and pattern recognition, 4510–4520.
Tryatmojo, B., & Maryati, R. I. S. (2019). Akurasi Sistem Face Recognition OpenCV Menggunakan Raspberry Pi dengan Metode Haar Cascade. Jurnal Ilmiah Informatika, 7(2), 92–98.
File:OpenCV Logo with text.png. (2022, Februari). Diambil kembali dari Wikimedia Commons: https://commons.wikimedia.org/wiki/File:OpenCV_Logo_with_text.png
Pattanayak, S., Pattanayak, J. S., & John, S. (2017). Pro deep learning with tensorflow (pp. 153–278). New York, NY, USA: Apress.
Santra, B. K. (2012). Genetic Algorithm and Confusion Matrix for Document Clustering. International Journal of Computer Science Issues.
Downey, A. (2012). Think Python. O'Reilly Media, Inc.
Wikimedia Commons. (2023, Februari). Diambil kembali dari File:Python-logo-notext.svg: https://commons.wikimedia.org/wiki/File:Python-logo-notext.svg
Soen, G. I. E., Marlina, M., & Renny, R. (2022). Implementasi Cloud Computing dengan Google Colaboratory pada Aplikasi Pengolah Data Zoom Participants. JITU: Journal Informatic Technology And Communication, 6(1), 24–30.
Johnston, B. (2020, Oktober 1). Google Colab. Diambil kembali dari Princeton.edu: https://mcgrawect.princeton.edu/tool/google-colab/
Fahri, U., Pd, S., & TI, M. (2016). Raspberry Pi. Akademi Manajemen Komputer Dan Informatika, Halaman 24.
Australia, I. (2019, Agustus 22). Raspberry Pi 4 Model B. Diambil kembali dari indomedia: https://indomedia.com.au/raspberry-pi-4-model-b/
Pi, R. Raspberry Pi 4 Tech Specs. Diambil kembali dari Raspberry Pi: https://www.raspberrypi.com/products/raspberry-pi-4-model-b/specifications/
Lazada. Raspberry Pi 3/4 Generation OV5647 Camera 130 Degree Fisheye Wide-angle Camera Module 3B+ 5 Million Pixels. Diambil kembali dari Lazada: https://www.lazada.co.id/products/raspberry-pi-34-generation-ov5647-camera-130-degree-fisheye-wide-angle-camera-module-3b-5-million-pixels-i6370208039
Uctronics. (t.thn.). 5mp Ov5647: Download Full Datasheet (Specs, Pinouts, Registers & Diagrams). Diambil kembali dari uctronics.com: https://www.uctronics.com/5mp-ov5647-download-full-datasheet-pdf
Sakti, E. (n.d.). Cara Kerja Sensor Ultrasonik, Rangkaian, & Aplikasinya. Diambil kembali dari ElangSakti.com: https://www.elangsakti.com/2015/05/sensor-ultrasonik.html
Widharma, I. G. S., Saputra, K. D. W., Purnamayana, I. M. S., Anam, S., & Prawiradnyana, I. G. P. B. (2020). Sensor Ultrasonik dalam Water Level Controller. No. December, 1–11.
Carlson, D. (2017). Wireless Charging for Bluetooth Earphones.
Yogi, M., Sari, M. I., & Handayani, R. (2023). Sistem Deteksi Teks Pada Kemasan Produk Makanan Menggunakan Metode Deep Learning Berbasis Raspberry Pi. eProceedings of Applied Science, 10(4).
Rahma, L., Syaputra, H., Mirza, A. H., & Purnamasari, S. D. (2021). Objek Deteksi Makanan Khas Palembang Menggunakan Algoritma YOLO (You Only Look Once). Jurnal Nasional Ilmu Komputer, 2(3), 213–232.
Saputra, R. D., & Fudholi, D. H. (2023). Model Mobile untuk Deteksi Objek pada On-Shelf Availability Produk Retail. AUTOMATA, 4(2).
Hidayat, Asep AS, Suwandi, Ate. (2013). Pendidikan anak berkebutuhan khusus tunanetra: peserta didik berkebutuhan khusus dengan hambatan penglihatan dalam perspektif pendidikan dan layanan (Cet. Ke-1). Jakarta: Luxima.
Pramana, E. (2023, Februari 11). Bantu Tunanetra, Ilmuwan Ciptakan Tongkat dengan Teknologi AI. Diambil kembali dari JawaPos.com: https://www.jawapos.com/teknologi/01435889/bantu-tunanetra-ilmuwan-ciptakan-tongkat-dengan-teknologi-ai
Klimchuk, M. R., & Krasovec, S. A. (2007). Desain kemasan: Perencanaan merek produk yang berhasil mulai dari konsep sampai penjualan. Penerbit Erlangga.
Nilsson, N. J. (1998). Artificial intelligence: a new synthesis. Morgan Kaufmann.
Yansens, C. Apa Itu Artificial Intelligence (AI)? Cara Kerja, Komponen, Jenis, Tujuan. Diambil kembali dari Tanya Digital: https://tanyadigital.com/artificial-intelligence-adalah/
Coelho, L. P., & Richert, W. (2015). Building machine learning systems with Python. Packt Publishing Ltd.
Ahmad, A. (2017). Mengenal Artificial Intelligence, Machine Learning, Neural Network, dan Deep Learning. Yayasan Cahaya Islam, Jurnal Teknologi Indonesia.
Batubara, N. A., Awangga, R. M., & Pane, S. F. (2020). Perbandingan Faster R-CNN dengan SSD Mobilenet Untuk Mendeteksi Plat Nomor. Kreatif Industri Nusantara, 9–10.
Le, H. H. (2015). Division Intro. Retrieved from Academia: https://www.academia.edu/27143145/Division_intro
Primartha, R. (2018). Belajar Machine Learning Teori dan Praktik. Bandung: Informatika Bandung, 10, 20–30.
Wahyu, R. (2019, Juli 6). Deep Learning Object Detection Menggunakan Tensorflow Dan Metode Convolutional Neural Network (CNN). Diambil kembali dari medium.com: https://medium.com/@rismitawahyu/deep-learning-object-detection-menggunakan-tensorflow-dan-metode-convolutional-neural-network-cnn-9c09267474e3
Saha, S. (2018, Desember 15). A Comprehensive Guide to Convolutional Neural Networks — the ELI5 way. Diambil kembali dari SaturnCloud: https://saturncloud.io/blog/a-comprehensive-guide-to-convolutional-neural-networks-the-eli5-way/
Zufar, M., & Setiyono, B. (2016). Convolutional neural networks untuk pengenalan wajah secara real-time. Jurnal Sains Dan Seni ITS, 5(2), A72–A77.
Liu, W., Anguelov, D., Erhan, D., Szegedy, C., Reed, S., Fu, C. Y., & Berg, A. C. (2016). SSD: Single Shot Multibox Detector. In Computer Vision–ECCV 2016, 21–37. Springer.
Liu, C., Anguelov, D., Erhan, D., Szegedy, C., Reed, S., Fu, C. Y., & Berg, A. C. (2016). SSD: Single Shot MultiBox Detector. Lecture Notes in Computer Science, 9905, 1–17. doi: 10.1007/978-3-319-46448-0_2
Ramalingam, B., Elara Mohan, R., Balakrishnan, S., Elangovan, K., Félix Gómez, B., Pathmakumar, T., ... & Baskar, C. (2021). Stetro-deep learning powered staircase cleaning and maintenance reconfigurable robot. Sensors, 21(18), 6279.
Badgujar, W. B. (2021, Juni 17). Creating MobileNets with TensorFlow from scratch. Diambil kembali dari medium.com: https://medium.com/@sumeetbadgujar/creating-mobilenets-with-tensorflow-from-scratch-c34ec79a59d2
Howard, A. G., Zhu, M., Chen, B., Kalenichenko, D., Wang, W., Weyand, T., ... & Adam, H. (2017). Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:1704.04861.
Team, P. (t.thn.). MOBILENET V2. Diambil kembali dari pytorch.org: https://pytorch.org/hub/pytorch_vision_mobilenet_v2/
Sandler, M., Howard, A., Zhu, M., Zhmoginov, A., & Chen, L. C. (2018). Mobilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE conference on computer vision and pattern recognition, 4510–4520.
Tryatmojo, B., & Maryati, R. I. S. (2019). Akurasi Sistem Face Recognition OpenCV Menggunakan Raspberry Pi dengan Metode Haar Cascade. Jurnal Ilmiah Informatika, 7(2), 92–98.
File:OpenCV Logo with text.png. (2022, Februari). Diambil kembali dari Wikimedia Commons: https://commons.wikimedia.org/wiki/File:OpenCV_Logo_with_text.png
Pattanayak, S., Pattanayak, J. S., & John, S. (2017). Pro deep learning with tensorflow (pp. 153–278). New York, NY, USA: Apress.
Santra, B. K. (2012). Genetic Algorithm and Confusion Matrix for Document Clustering. International Journal of Computer Science Issues.
Downey, A. (2012). Think Python. O'Reilly Media, Inc.
Wikimedia Commons. (2023, Februari). Diambil kembali dari File:Python-logo-notext.svg: https://commons.wikimedia.org/wiki/File:Python-logo-notext.svg
Soen, G. I. E., Marlina, M., & Renny, R. (2022). Implementasi Cloud Computing dengan Google Colaboratory pada Aplikasi Pengolah Data Zoom Participants. JITU: Journal Informatic Technology And Communication, 6(1), 24–30.
Johnston, B. (2020, Oktober 1). Google Colab. Diambil kembali dari Princeton.edu: https://mcgrawect.princeton.edu/tool/google-colab/
Fahri, U., Pd, S., & TI, M. (2016). Raspberry Pi. Akademi Manajemen Komputer Dan Informatika, Halaman 24.
Australia, I. (2019, Agustus 22). Raspberry Pi 4 Model B. Diambil kembali dari indomedia: https://indomedia.com.au/raspberry-pi-4-model-b/
Pi, R. Raspberry Pi 4 Tech Specs. Diambil kembali dari Raspberry Pi: https://www.raspberrypi.com/products/raspberry-pi-4-model-b/specifications/
Lazada. Raspberry Pi 3/4 Generation OV5647 Camera 130 Degree Fisheye Wide-angle Camera Module 3B+ 5 Million Pixels. Diambil kembali dari Lazada: https://www.lazada.co.id/products/raspberry-pi-34-generation-ov5647-camera-130-degree-fisheye-wide-angle-camera-module-3b-5-million-pixels-i6370208039
Uctronics. (t.thn.). 5mp Ov5647: Download Full Datasheet (Specs, Pinouts, Registers & Diagrams). Diambil kembali dari uctronics.com: https://www.uctronics.com/5mp-ov5647-download-full-datasheet-pdf
Sakti, E. (n.d.). Cara Kerja Sensor Ultrasonik, Rangkaian, & Aplikasinya. Diambil kembali dari ElangSakti.com: https://www.elangsakti.com/2015/05/sensor-ultrasonik.html
Widharma, I. G. S., Saputra, K. D. W., Purnamayana, I. M. S., Anam, S., & Prawiradnyana, I. G. P. B. (2020). Sensor Ultrasonik dalam Water Level Controller. No. December, 1–11.
Carlson, D. (2017). Wireless Charging for Bluetooth Earphones.