Improving Reinforcement Learning Algorithm Based on Non-Negative Matrix Factorization Method for Controlling an Arm Model
Abstract
Purpose: Reinforcement Learning (RL) is attracting great interest because it enables systems to learn by interacting with the environment. This study aims to enhance the RL algorithm to become more similar to human motor control by combining it with the Non-negative matrix factorization (NMF) method.
Materials and Methods: In the study, the signals recorded from six muscles involved in arm-reaching movement without carryinga certain weight.were pre-processed, and the optimal number of synergy patterns was extracted using NMF and the Variance Account For (VAF) methods. This, in turn, contributes to reducing the calculations. Subsequently, the robustness of the two-link arm model with six muscles was evaluated under various noise levels applied to the action coefficient matrix. Finally, the average synergy pattern was done on the mentioned arm model, and the RL algorithm controlled it by producing the action coefficient matrix.
Results: The average VAF% was 97.25±2.0%, and the number of synergies was four. The tip-of-the-arm model was able to reach the target after an average of 100 episodes.
Conclusion: The results indicated that the similarity in the extracted synergy patterns helps to model a system that is more similar to motor control. Additionally, the results of the synergistic patterns revealed that the two-link arm model with six muscles was suitable for the model. While controlling the model with the RL algorithm, the desired end-point position and path were achieved.
1- Padmaja Kulkarni, Jens Kober, Robert Babuška, and Cosimo Della Santina, "Learning assembly tasks in a few minutes by combining impedance control and residual recurrent reinforcement learning." Advanced Intelligent Systems, Vol. 4 (No. 1), p. 2100095, (2022).
2- Claire Glanois et al., "A survey on interpretable reinforcement learning." Machine Learning, pp. 1-44, (2024).
3- Frensi Zejnullahu, Maurice Moser, and Joerg Osterrieder, "Applications of Reinforcement Learning in Finance--Trading with a Double Deep Q-Network." arXiv preprint arXiv:2206.14267, (2022).
4- Dong Han, Beni Mulyana, Vladimir Stankovic, and Samuel Cheng, "A survey on deep reinforcement learning algorithms for robotic manipulation." Sensors, Vol. 23 (No. 7), p. 3762, (2023).
5- Nat Wannawas and A Aldo Faisal, "Towards AI-controlled FES-restoration of arm movements: neuromechanics-based reinforcement learning for 3-d reaching." In 2023 11th International IEEE/EMBS Conference on Neural Engineering (NER), (2023): IEEE, pp. 1-4.
6- Jonas Tebbe, Lukas Krauch, Yapeng Gao, and Andreas Zell, "Sample-efficient reinforcement learning in robotic table tennis." In 2021 IEEE International Conference on Robotics and Automation (ICRA), (2021): IEEE, pp. 4171-78.
7- Kathleen M Jagodnik, Philip S Thomas, Antonie J van den Bogert, Michael S Branicky, and Robert F Kirsch, "Training an actor-critic reinforcement learning controller for arm movement using human-generated rewards." IEEE Transactions on Neural Systems and Rehabilitation Engineering, Vol. 25 (No. 10), pp. 1892-905, (2017).
8- Tobias Johannink et al., "Residual reinforcement learning for robot control." In 2019 International Conference on Robotics and Automation (ICRA), (2019): IEEE, pp. 6023-29.
9- Meng Fan-Cheng and Dai Ya-Ping, "Reinforcement learning adaptive control for upper limb rehabilitation robot based on fuzzy neural network." In Proceedings of the 31st Chinese Control Conference, (2012): IEEE, pp. 5157-61.
10- Don Liang et al., "Synergistic activation patterns of hand muscles in left-and right-hand dominant individuals." Journal of Human Kinetics, Vol. 76 (No. 1), pp. 89-100, (2021).
11- Ν Bernstein, "The co-ordination and regulation of movements, Oxford Pergamon." Search in (1967).
12- Cristiano Alessandro, Ioannis Delis, Francesco Nori, Stefano Panzeri, and Bastien Berret, "Muscle synergies in neuroscience and robotics: from input-space to task-space perspectives." Frontiers in computational neuroscience, Vol. 7p. 43, (2013).
13- Emilio Bizzi and Vincent CK Cheung, "The neural origin of muscle synergies." Frontiers in computational neuroscience, Vol. 7p. 51, (2013).
14- Vincent CK Cheung et al., "Muscle synergy patterns as physiological markers of motor cortical damage." Proceedings of the National Academy of sciences, Vol. 109 (No. 36), pp. 14652-56, (2012).
15- José Antonio Martin and H De Lope, "A distributed reinforcement learning architecture for multi-link robots." in 4th International Conference on Informatics in Control, Automation and Robotics, (2007), Vol. 192, p. 197.
16- Fereidoun Nowshiravan Rahatabad and Parisa Rangraz, "Combination of reinforcement learning and bee algorithm for controlling two-link arm with six muscle: simplified human arm model in the horizontal plane." Physical and Engineering Sciences in Medicine, Vol. 43 (No. 1), pp. 135-42, (2020).
17- Jun Izawa, Toshiyuki Kondo, and Koji Ito, "Biological arm motion through reinforcement learning." Biological Cybernetics, Vol. 91 (No. 1), pp. 10-22, (2004).
18- Albert Albers, Wenjie Yan, and Markus Frietsch, Application of reinforcement learning to a two DOF robot arm control. Na, (2009).
19- Xiaoling Chen et al., "Muscle activation patterns and muscle synergies reflect different modes of coordination during upper extremity movement." Frontiers in Human Neuroscience, Vol. 16, p. 912440, (2023).
20- Vincent CK Cheung et al., "Plasticity of muscle synergies through fractionation and merging during development and training of human runners." Nature Communications, Vol. 11 (No. 1), p. 4356, (2020).
21- Yushin Kim, Thomas C Bulea, and Diane L Damiano, "Novel methods to enhance precision and reliability in muscle synergy identification during walking." Frontiers in Human Neuroscience, Vol. 10, p. 455, (2016).
22- Akira Saito, Aya Tomita, Ryosuke Ando, Kohei Watanabe, and Hiroshi Akima, "Similarity of muscle synergies extracted from the lower limb including the deep muscles between level and uphill treadmill walking." Gait & posture, Vol. 59, pp. 134-39, (2018).
23- Alessandro Scano, Robert Mihai Mira, and Andrea d’Avella, "Mixed matrix factorization: A novel algorithm for the extraction of kinematic-muscular synergies." Journal of Neurophysiology, Vol. 127 (No. 2), pp. 529-47, (2022).
24- Daniel Soukup and Ivan Bajla, "Robust object recognition under partial occlusions using NMF." Computational intelligence and neuroscience, Vol. 2008 (No. 1), p. 857453, (2008).
25- Jingcheng Chen, Yining Sun, and Shaoming Sun, "Muscle synergy of lower limb motion in subjects with and without knee pathology." Diagnostics, Vol. 11 (No. 8), p. 1318, (2021).
26- Vincent CK Cheung, Andrea d'Avella, Matthew C Tresch, and Emilio Bizzi, "Central and sensory contributions to the activation and organization of muscle synergies during natural motor behaviors." Journal of Neuroscience, Vol. 25 (No. 27), pp. 6419-34, (2005).
27- Sensor Locations. [Online]. Available: http://seniam.org/sensor_location.html.
28- Hyun K Kim, Jose M Carmena, S James Biggs, Timothy L Hanson, Miguel AL Nicolelis, and Mandayam A Srinivasan, "The muscle activation method: an approach to impedance control of brain-machine interfaces through a musculoskeletal model of the arm." IEEE Transactions on Biomedical Engineering, Vol. 54 (No. 8), pp. 1520-29, (2007).
29- Fereidoun Nowshiravan Rahatabad and Elham Farzaneh Bahalgerdy, "The Most Effective VAF Threshold for Extracting the Optimum Number of Synergies for Reaching Movement in a Two-Link Arm Model with Two DoF." Frontiers in Biomedical Technologies, (2024).
30- Kenji Tahara, Zhi-Wei Luo, Suguru Arimoto, and Hitoshi Kino, "Task-space feedback control for a two-link arm driven by six muscles with variable damping and elastic properties." in Proceedings of the 2005 IEEE International Conference on Robotics and Automation, (2005): IEEE, pp. 223-28.
31- Tools for Reinforcement Learning, Neural Networks, and Robotics (Matlab and Python). [Online]. Available: http://jamh-web.appspot.com/download.htm#Reinforcement_Learning.
32- Ilge Akkaya et al., "Solving Rubik'sRubik's cube with a robot hand." arXiv preprint arXiv:1910.07113, (2019).
2- Claire Glanois et al., "A survey on interpretable reinforcement learning." Machine Learning, pp. 1-44, (2024).
3- Frensi Zejnullahu, Maurice Moser, and Joerg Osterrieder, "Applications of Reinforcement Learning in Finance--Trading with a Double Deep Q-Network." arXiv preprint arXiv:2206.14267, (2022).
4- Dong Han, Beni Mulyana, Vladimir Stankovic, and Samuel Cheng, "A survey on deep reinforcement learning algorithms for robotic manipulation." Sensors, Vol. 23 (No. 7), p. 3762, (2023).
5- Nat Wannawas and A Aldo Faisal, "Towards AI-controlled FES-restoration of arm movements: neuromechanics-based reinforcement learning for 3-d reaching." In 2023 11th International IEEE/EMBS Conference on Neural Engineering (NER), (2023): IEEE, pp. 1-4.
6- Jonas Tebbe, Lukas Krauch, Yapeng Gao, and Andreas Zell, "Sample-efficient reinforcement learning in robotic table tennis." In 2021 IEEE International Conference on Robotics and Automation (ICRA), (2021): IEEE, pp. 4171-78.
7- Kathleen M Jagodnik, Philip S Thomas, Antonie J van den Bogert, Michael S Branicky, and Robert F Kirsch, "Training an actor-critic reinforcement learning controller for arm movement using human-generated rewards." IEEE Transactions on Neural Systems and Rehabilitation Engineering, Vol. 25 (No. 10), pp. 1892-905, (2017).
8- Tobias Johannink et al., "Residual reinforcement learning for robot control." In 2019 International Conference on Robotics and Automation (ICRA), (2019): IEEE, pp. 6023-29.
9- Meng Fan-Cheng and Dai Ya-Ping, "Reinforcement learning adaptive control for upper limb rehabilitation robot based on fuzzy neural network." In Proceedings of the 31st Chinese Control Conference, (2012): IEEE, pp. 5157-61.
10- Don Liang et al., "Synergistic activation patterns of hand muscles in left-and right-hand dominant individuals." Journal of Human Kinetics, Vol. 76 (No. 1), pp. 89-100, (2021).
11- Ν Bernstein, "The co-ordination and regulation of movements, Oxford Pergamon." Search in (1967).
12- Cristiano Alessandro, Ioannis Delis, Francesco Nori, Stefano Panzeri, and Bastien Berret, "Muscle synergies in neuroscience and robotics: from input-space to task-space perspectives." Frontiers in computational neuroscience, Vol. 7p. 43, (2013).
13- Emilio Bizzi and Vincent CK Cheung, "The neural origin of muscle synergies." Frontiers in computational neuroscience, Vol. 7p. 51, (2013).
14- Vincent CK Cheung et al., "Muscle synergy patterns as physiological markers of motor cortical damage." Proceedings of the National Academy of sciences, Vol. 109 (No. 36), pp. 14652-56, (2012).
15- José Antonio Martin and H De Lope, "A distributed reinforcement learning architecture for multi-link robots." in 4th International Conference on Informatics in Control, Automation and Robotics, (2007), Vol. 192, p. 197.
16- Fereidoun Nowshiravan Rahatabad and Parisa Rangraz, "Combination of reinforcement learning and bee algorithm for controlling two-link arm with six muscle: simplified human arm model in the horizontal plane." Physical and Engineering Sciences in Medicine, Vol. 43 (No. 1), pp. 135-42, (2020).
17- Jun Izawa, Toshiyuki Kondo, and Koji Ito, "Biological arm motion through reinforcement learning." Biological Cybernetics, Vol. 91 (No. 1), pp. 10-22, (2004).
18- Albert Albers, Wenjie Yan, and Markus Frietsch, Application of reinforcement learning to a two DOF robot arm control. Na, (2009).
19- Xiaoling Chen et al., "Muscle activation patterns and muscle synergies reflect different modes of coordination during upper extremity movement." Frontiers in Human Neuroscience, Vol. 16, p. 912440, (2023).
20- Vincent CK Cheung et al., "Plasticity of muscle synergies through fractionation and merging during development and training of human runners." Nature Communications, Vol. 11 (No. 1), p. 4356, (2020).
21- Yushin Kim, Thomas C Bulea, and Diane L Damiano, "Novel methods to enhance precision and reliability in muscle synergy identification during walking." Frontiers in Human Neuroscience, Vol. 10, p. 455, (2016).
22- Akira Saito, Aya Tomita, Ryosuke Ando, Kohei Watanabe, and Hiroshi Akima, "Similarity of muscle synergies extracted from the lower limb including the deep muscles between level and uphill treadmill walking." Gait & posture, Vol. 59, pp. 134-39, (2018).
23- Alessandro Scano, Robert Mihai Mira, and Andrea d’Avella, "Mixed matrix factorization: A novel algorithm for the extraction of kinematic-muscular synergies." Journal of Neurophysiology, Vol. 127 (No. 2), pp. 529-47, (2022).
24- Daniel Soukup and Ivan Bajla, "Robust object recognition under partial occlusions using NMF." Computational intelligence and neuroscience, Vol. 2008 (No. 1), p. 857453, (2008).
25- Jingcheng Chen, Yining Sun, and Shaoming Sun, "Muscle synergy of lower limb motion in subjects with and without knee pathology." Diagnostics, Vol. 11 (No. 8), p. 1318, (2021).
26- Vincent CK Cheung, Andrea d'Avella, Matthew C Tresch, and Emilio Bizzi, "Central and sensory contributions to the activation and organization of muscle synergies during natural motor behaviors." Journal of Neuroscience, Vol. 25 (No. 27), pp. 6419-34, (2005).
27- Sensor Locations. [Online]. Available: http://seniam.org/sensor_location.html.
28- Hyun K Kim, Jose M Carmena, S James Biggs, Timothy L Hanson, Miguel AL Nicolelis, and Mandayam A Srinivasan, "The muscle activation method: an approach to impedance control of brain-machine interfaces through a musculoskeletal model of the arm." IEEE Transactions on Biomedical Engineering, Vol. 54 (No. 8), pp. 1520-29, (2007).
29- Fereidoun Nowshiravan Rahatabad and Elham Farzaneh Bahalgerdy, "The Most Effective VAF Threshold for Extracting the Optimum Number of Synergies for Reaching Movement in a Two-Link Arm Model with Two DoF." Frontiers in Biomedical Technologies, (2024).
30- Kenji Tahara, Zhi-Wei Luo, Suguru Arimoto, and Hitoshi Kino, "Task-space feedback control for a two-link arm driven by six muscles with variable damping and elastic properties." in Proceedings of the 2005 IEEE International Conference on Robotics and Automation, (2005): IEEE, pp. 223-28.
31- Tools for Reinforcement Learning, Neural Networks, and Robotics (Matlab and Python). [Online]. Available: http://jamh-web.appspot.com/download.htm#Reinforcement_Learning.
32- Ilge Akkaya et al., "Solving Rubik'sRubik's cube with a robot hand." arXiv preprint arXiv:1910.07113, (2019).
| Files | ||
| Issue | Articles in Press | |
| Section | Original Article(s) | |
| Keywords | ||
| Reinforcement Learning Algorithm Non-Negative Matrix Factorization Muscle Synergy Action Coefficient Matrix Optimization Two-Link Arm Model | ||
| Rights and permissions | |
|
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License. |
How to Cite
1.
Farzaneh Bahalgerdy E, Nowshiravan Rahatabad F. Improving Reinforcement Learning Algorithm Based on Non-Negative Matrix Factorization Method for Controlling an Arm Model. Frontiers Biomed Technol. 2025;.
