Skip to main content
Log in

Electroencephalography as implicit communication channel for proximal interaction between humans and robot swarms

  • Published:
Swarm Intelligence Aims and scope Submit manuscript

Abstract

Search and rescue, autonomous construction, and many other semi-autonomous multirobot applications can benefit from proximal interactions between an operator and a swarm of robots. Most research on proximal interaction is based on explicit communication techniques such as gesture and speech. This study proposes a new implicit proximal communication technique to approach the problem of robot selection. We use electroencephalography (EEG) signals to select the robot at which the operator is looking. This is achieved using steady-state visually evoked potential (SSVEP), a repeatable neural response to a regularly blinking visual stimulus that varies predictively based on the blinking frequency. In our experiments, each robot was equipped with LEDs blinking at a different frequency, and the operator’s SSVEP neural response was extracted from the EEG signal to detect and select the robot without requiring any conscious action by the user. This study systematically investigates several parameters affecting the SSVEP neural response: blinking frequency of the LED, distance between the robot and the operator, and color of the LED. Based on these parameters, we study two signal processing approaches and critically analyze their performance on 10 subjects controlling a set of physical robots. Our results show that despite numerous artifacts, it is possible to achieve a recognition rate higher than 85 % on some subjects, while the average over the ten subjects was 75 %.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

References

  • Akhtar, A., Norton, J. J., Kasraie, M., & Bretl, T. (2014). Playing checkers with your mind: An interactive multiplayer hardware game platform for brain–computer interfaces. In 36th Annual international conference of the IEEE engineering in medicine and biology society (EMBC), IEEE (pp. 1650–1653).

  • Aljshamee, M., Mohammed, M. Q., Malekpour, A., Luksch, P., et al. (2014). Beyond pure frequency and phases exploiting: Color influence in SSVEP based on BCI. Computer Technology and Application, 5(2), 111–118.

    Google Scholar 

  • Aljshamee, M., Nadir, S., Malekpour, A., & Luksch, P. (2016). Discriminate the brain responses of multiple colors based on regular/irregular SSVEP paradigms. Journal of Medical and Bioengineering, 5(2), 89–92. doi:10.18178/jomb.5.2.89-92.

    Google Scholar 

  • Beverina, F., Palmas, G., & Silvoni, S. (2003). User adaptive BCIs: SSVEP and P300 based interfaces. PsychNology, 1(4), 331–354.

    Google Scholar 

  • Bi, L., Fan, X. A., & Liu, Y. (2013). EEG-based brain-controlled mobile robots: A survey. IEEE Transactions on Human-Machine Systems, 43(2), 161–176. doi:10.1109/TSMCC.2012.2219046.

    Article  Google Scholar 

  • Bonani, M., Rétornaz, P., Magnenat, S., Bleuler, H., & Mondada, F. (2012). Physical interactions in swarm robotics: The hand-bot case study. In A. Martinoli, F. Mondada, N. Correll, G. Mermoud, M. Egerstedt, M. A. Hsieh, L. E. Parker, & K. Støy (Eds.), Distributed autonomous robotic systems, Springer tracts in advanced robotics (pp. 585–595). Berlin: Springer.

    Google Scholar 

  • Brambilla, M., Ferrante, E., Birattari, M., & Dorigo, M. (2013). Swarm robotics: A review from the swarm engineering perspective. Swarm Intelligence, 7(1), 1–41. doi:10.1007/s11721-012-0075-2.

    Article  Google Scholar 

  • Cao, T., Wan, F., Mak, P. U., Mak, P. I., Vai, M. I., & Hu, Y. (2012). Flashing color on the performance of SSVEP-based brain–computer interfaces. In Annual international conference of the IEEE engineering in medicine and biology society (EMBC), IEEE (pp. 1819–1822).

  • Chua, F., Daftari, A., Alvarez, T., DeMarco, R., Bergen, M., Beck, K., & Servatius, R. (2004). Effects of a single green flash versus a white flash of light on saccadic oculomotor metrics. In Proceedings of the IEEE 30th annual northeast bioengineering conference, IEEE (pp. 9–10).

  • Couture-Beil, A., Vaughan, R., & Mori, G. (2010). Selecting and commanding individual robots in a vision-based multi-robot system. In 5th ACM/IEEE international conference on human–robot interaction (HRI) (pp. 355–356). doi:10.1109/HRI.2010.5453167.

  • Durak, L., & Arikan, O. (2003). Short-time fourier transform: Two fundamental properties and an optimal implementation. IEEE Transactions on Signal Processing, 51(5), 1231–1242. doi:10.1109/TSP.2003.810293.

    Article  MathSciNet  Google Scholar 

  • Duvinage, M., Castermans, T., Petieau, M., Hoellinger, T., Cheron, G., & Dutoit, T. (2013). Performance of the emotiv EPOC headset for P300-based applications. Biomedical Engineering Online, 12(1), 56.

    Article  Google Scholar 

  • Faller, J., Müller-Putz, G., Schmalstieg, D., & Pfurtscheller, G. (2010). An application framework for controlling an avatar in a desktop-based virtual environment via a software SSVEP brain computer interface. Presence: Teleoperators and Virtual Environments, 19(1), 25–34. doi:10.1162/pres.19.1.25.

    Article  Google Scholar 

  • Fan, X., Bi, L., Teng, T., Ding, H., & Liu, Y. (2015). A brain–computer interface-based vehicle destination selection system using P300 and SSVEP signals. IEEE Transactions on Intelligent Transportation Systems, 16(1), 274–283.

    Article  Google Scholar 

  • Fong, T., Thorpe, C., & Baur, C. (2003). Multi-robot remote driving with collaborative control. IEEE Transactions on Industrial Electronics, 50(4), 699–704. doi:10.1109/TIE.2003.814768.

    Article  Google Scholar 

  • Gao, X., Xu, D., Cheng, M., & Gao, S. (2003). A BCI-based environmental controller for the motion-disabled. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 11(2), 137–140. doi:10.1109/TNSRE.2003.814449.

    Article  Google Scholar 

  • Goodrich, M. A., Pendleton, B., Kerman, S., & Sujit, P. (2013). What types of interactions do bio-inspired robot swarms and flocks afford a human? In Proceedings of robotics science and systems VIII (pp. 105–112).

  • Goodrich, M. A., & Schultz, A. C. (2007). Human–robot interaction: A survey. Foundations and Trends in Human Computer Interaction, 1(3), 203–275. doi:10.1561/1100000005.

    Article  MATH  Google Scholar 

  • Guger, C., Allison, B., Grosswindhager, B., Prückl, R., Hintermüller, C., Kapeller, C., et al. (2012). How many people could use an SSVEP BCI? Frontiers in Neuroscience, 6, 169. doi:10.3389/fnins.2012.00169.

    Google Scholar 

  • Güneysu, A., & Akin, H. L. (2013). An SSVEP based BCI to control a humanoid robot by using portable EEG device. In International conference of the IEEE engineering in medicine and biology society, (Vol. 2013, pp. 6905–6908). doi:10.1109/EMBC.2013.6611145.

  • Herrmann, C. S. (2001). Human EEG responses to 1–100 Hz flicker: Resonance phenomena in visual cortex and their potential correlation to cognitive phenomena. Experimental Brain Research, 137, 346–353. doi:10.1007/s002210100682.

    Article  Google Scholar 

  • Hvaring, F. T., & Ulltveit-Moe, A. H. (2014). A comparison of visual evoked potential (VEP)-based methods for the low-cost emotiv EPOC neuroheadset. Technical report, Norwegian University of Science and Technology, NTNU—Trondheim, Trondheim, Norway. http://www.diva-portal.org/smash/get/diva2:751718/FULLTEXT01.pdf.

  • Jacobs, G. (2013). SSVEP-based BCI control for navigating a robot. Technical Report. December, Faculteit der Sociale Wetenschappen, Radboud University, Nijmegen, The Netherlands. http://www.ru.nl/publish/pages/641151/jacobs_gj_ba-th-2013.pdf.

  • Jian, H. L., & Tang, K. T. (2014). Improving classification accuracy of SSVEP based BCI using RBF SVM with signal quality evaluation. In 2014 International symposium on intelligent signal processing and communication systems (ISPACS), IEEE (pp. 302–306).

  • Jones, G., Berthouze, N., Bielski, R., & Julier, S. (2010). Towards a situated, multimodal interface for multiple UAV control. In 2010 IEEE international conference on robotics and automation (ICRA) (pp. 1739–1744). doi:10.1109/ROBOT.2010.5509960.

  • Kernbach, S. (2013). Handbook of collective robotics: Fundamentals and challenges. Boca Raton: CRC Press. doi:10.4032/9789814364119.

    Book  Google Scholar 

  • Kirchner, E. A., de Gea Fernandez, J., Kampmann, P., Schröer, M., Metzen, J. H., & Kirchner, F. (2015). Intuitive interaction with robots—technical approaches and challenges. In R. Drechsler & U. Khne (Eds.), Formal modeling and verification of cyber-physical systems: 1st International summer school on methods and tools for the design of digital systems, Bremen, Germany, September 2015, Springer Fachmedien Wiesbaden, Wiesbaden (pp. 224–248). doi:10.1007/978-3-658-09994-78.

  • Kishore, S., González-Franco, M., Hintemüller, C., Kapeller, C., Guger, C., Slater, M., et al. (2014). Comparison of SSVEP BCI and eye tracking for controlling a humanoid robot in a social environment. Presence: Teleoperators and Virtual Environments, 23(3), 242–252.

    Article  Google Scholar 

  • Kolling, A., Walker, P., Chakraborty, N., Sycara, K., & Lewis, M. (2016). Human interaction with robot swarms: A survey. IEEE Transactions on Human-Machine Systems, 46(1), 9–26. doi:10.1109/THMS.2015.2480801.

    Article  Google Scholar 

  • Li, Y., Pan, J., Wang, F., & Yu, Z. (2013). A hybrid bci system combining P300 and SSVEP and its application to wheelchair control. IEEE Transactions on Biomedical Engineering, 60(11), 3156–3166.

    Article  Google Scholar 

  • Lin, Y. P., Wang, Y., & Jung, T. P. (2014). Assessing the feasibility of online SSVEP decoding in human walking using a consumer EEG headset. Journal of NeuroEngineering and Rehabilitation, 11(1), 119. doi:10.1186/1743-0003-11-119.

    Article  Google Scholar 

  • Liu, Y., Jiang, X., Cao, T., Wan, F., Mak, P. U., Mak, P. I., & Vai, M. I. (2012). Implementation of SSVEP based BCI with emotiv EPOC. In IEEE international conference on virtual environments human–computer interfaces and measurement systems (VECIMS), IEEE (pp. 34–37).

  • Martinoli, A., Mondada, F., Correll N., Mermoud, G., Egerstedt, M., Hsieh, M. A, Parker, L. E., & Støy, K. (2012). 10th International symposium on distributed autonomous robotic systems (DARS) (Vol. 83). Springer.

  • Mathews, N., Valentini, G., Christensen, A. L., O’Grady, R., Brutschy, A., & Dorigo, M. (2015). Spatially targeted communication in decentralized multirobot systems. Autonomous Robots, 38(4), 439–457.

    Article  Google Scholar 

  • Millan, J. R., Renkens, F., Mouriño, J., & Gerstner, W. (2004). Noninvasive brain-actuated control of a mobile robot by human EEG. IEEE Transactions on biomedical Engineering, 51(6), 1026–1033.

    Article  Google Scholar 

  • Monajjemi, V., Wawerla, J., Vaughan, R., & Mori, G. (2013). HRI in the sky: Creating and commanding teams of UAVs with a vision–mediated gestural interface. In IEEE/RSJ international conference on intelligent robots and systems (IROS) (pp. 617–623). doi:10.1109/IROS.2013.6696415.

  • Mouli, S., Palaniappan, R., Sillitoe, I. P., & Gan, J. Q. (2013). Performance analysis of multi-frequency SSVEP-BCI using clear and frosted colour led stimuli. In Bioinformatics and bioengineering (BIBE), 2013 IEEE 13th international conference on, IEEE (pp. 1–4).

  • Nagi, J., Giusti, A., Gambardella, L. M., & Di Caro, G. (2014). Human–swarm interaction using spatial gestures. In IEEE/RSJ international conference on intelligent robots and systems (IROS 2014) (pp. 3834–3841). doi:10.1109/IROS.2014.6943101.

  • Ortner, R., Guger, C., Prueckl, R., Grünbacher, E., & Edlinger, G. (2010). SSVEP based brain–computer interface for robot control. In International conference on computers for handicapped persons, Springer, Vol. 6180 LNCS (pp. 85–90).

  • Pourmehr, S., Monajjemi, V., Vaughan, R., & Mori, G. (2013). “You two! Take off!”: Creating, modifying and commanding groups of robots using face engagement and indirect speech in voice commands. In IEEE/RSJ international conference on intelligent robots and systems (IROS) (pp. 137–142). doi:10.1109/IROS.2013.6696344.

  • Renard, Y., Lotte, F., Gibert, G., Congedo, M., Maby, E., Delannoy, V., et al. (2010). OpenViBE: An open-source software platform to design, test, and use brain–computer interfaces in real and virtual environments. Presence: Teleoperators and Virtual Environments, 19(1), 35–53. doi:10.1162/pres.19.1.35.

    Article  Google Scholar 

  • Rencher, A. C. (2003). Methods of multivariate analysis (Vol. 492). New York: Wiley. doi:10.1002/0471271357.

    MATH  Google Scholar 

  • Riedo, F., Chevalier, M., Magnenat, S., Mondada, F. (2013). Thymio II, a robot that grows wiser with children. In IEEE workshop on advanced robotics and its social impacts (ARSO), IEEE (pp. 187–193). doi:10.1109/ARSO.2013.6705527.

  • Rubenstein, M., Ahler, C., & Nagpal, R. (2012). Kilobot: A low cost scalable robot system for collective behaviors. In IEEE international conference on robotics and automation (ICRA), IEEE (pp. 3293–3298).

  • Rubenstein, M., Cornejo, A., & Nagpal, R. (2014). Programmable self-assembly in a thousand-robot swarm. Science, 345(6198), 795–799.

    Article  Google Scholar 

  • Rzepecki, J., Delcourt, J., Da Silva, M. P., & Le Callet, P. (2012). Virtual interactions: Can EEG help make the difference with real interaction? In Proceedings of the 2012 IEEE international conference on multimedia and expo workshops, ICMEW 2012 (pp. 151–156). doi:10.1109/ICMEW.2012.33.

  • Salehuddin, M., Suprijanto, S., & Muchtadi, F. I. (2011). Prototype design of low cost four channels digital electroencephalograph for sleep monitoring. In 2nd International conference on instrumentation control and automation (ICA) (pp. 188–193). doi:10.1109/ICA.2011.6130154.

  • Stawicki, P., Gembler, F., & Volosyak, I. (2016). Driving a semiautonomous mobile robotic car controlled by an SSVEP-based BCI. Computational Intelligence and Neuroscience. doi:10.1155/2016/4909685.

  • Stytsenko, K., Jablonskis, E., & Prahm, C. (2011). Evaluation of consumer EEG device emotiv EPOC. In MEi: CogSci conference 2011, Ljubljana.

  • Trovato, G., Zecca, M., Sessa, S., Jamone, L., Ham, J., Hashimoto, K., et al. (2013). Cross-cultural study on human–robot greeting interaction: Acceptance and discomfort by egyptians and japanese. Paladyn, Journal of Behavioral Robotics, 4(2), 83–93. doi:10.2478/pjbr-2013-0006.

    Article  Google Scholar 

  • Van Vliet, M., Robben, A., Chumerin, N., Manyakov, N. V., Combaz, A., & Van Hulle, M. M. (2012). Designing a brain–computer interface controlled video-game using consumer grade EEG hardware. In Biosignals and biorobotics conference (BRC), 2012, IEEE (pp. 1–6).

  • Volosyak, I., Cecotti, H., & Graser, A. (2009). Optimal visual stimuli on LCD screens for SSVEP based brain–computer interfaces. In Neural Engineering (pp. 447–450).

  • Winfield, A. F., & Nembrini, J. (2006). Safety in numbers: Fault-tolerance in robot swarms. International Journal of Modelling, Identification and Control, 1(1), 30–37.

    Article  Google Scholar 

  • Wu, C. H., & Lakany, H. (2013). The effect of the viewing distance of stimulus on SSVEP response for use in brain–computer interfaces. In IEEE international conference on systems, man, and cybernetics (SMC), IEEE (pp. 1840–1845).

  • Yanco, H. A., & Drury, J. L. (2004). Classifying human–robot interaction: An updated taxonomy. In IEEE international conference on systems, man and cybernetics, SMC (pp. 2841–2846). doi:10.1109/ICSMC.2004.1400763.

  • Yin, E., Zeyl, T., Saab, R., Chau, T., Hu, D., & Zhou, Z. (2015). A hybrid brain–computer interface based on the fusion of P300 and SSVEP scores. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 23(4), 693–701.

    Article  Google Scholar 

  • Yin, E., Zhou, Z., Jiang, J., Chen, F., Liu, Y., & Hu, D. (2013). A novel hybrid BCI speller based on the incorporation of SSVEP into the P300 paradigm. Journal of Neural Engineering, 10(2), 026,012.

    Article  Google Scholar 

  • Zheng, Y., & Zhang, Y. (2010). An improved segmented match filters with FFT approach for GNSS signal acquisition. In ICCTD 2010—Proceedings of the 2nd international conference on computer technology and development, ICCTD (pp. 425–428). doi:10.1109/ICCTD.2010.5645833.

  • Zhu, D., Bieger, J., Molina, G. G., & Aarts, R. M. (2010). A survey of stimulation methods used in SSVEP-based BCIs. Computational Intelligence and Neuroscience, 702, 357. doi:10.1155/2010/702357.

    Google Scholar 

Download references

Acknowledgments

Many thanks to Dr. Ricardo Chavarriaga, Dr. Claire Braboszcz, and Dr. Serafeim Perdikis for the constructive discussions about experiments involving EEG; to Dr. Jérôme Scherer and Prof. Marco Picasso for their help on mathematical issues in the signal processing; to the reviewers who contributed with detailed and constructive comments during the submission process; and to all subjects who were available for the experiments. This work was partially supported by the Swiss National Center of Competence in Research “Robotics.”

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Francesco Mondada.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Mondada, L., Karim, M.E. & Mondada, F. Electroencephalography as implicit communication channel for proximal interaction between humans and robot swarms. Swarm Intell 10, 247–265 (2016). https://doi.org/10.1007/s11721-016-0127-0

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11721-016-0127-0

Keywords

Navigation