都立大「ユビキタスロボティクス特論」5月12日
- 1. About OpenRTM-aist and Outline of RT-Component Programming National Institute of Advanced Industrial Science and Technology (AIST), Japan Industrial CPS Research Center Software Platform Research Team, Team Leader Noriaki Ando, Ph.D. 1 May 13, 2021 Ubiquitous Robotics
- 2. Outline • Basic concept and overview of RT- Middleware • Comparison with ROS • Activities of RT-Middleware community • RTC development overview • Conclusion 2
- 4. • RT = Robot Technology cf. IT – not only standalone robots, but also robotic elements (sensors, actuators, etc….) • RT-Middleware (RTM) – middleware and platform for RT-element integration • RT-Component (RTC) – basic software unit in RT-Middleware RT-Middleware + + + + + What is RT? RT-Middleware developed by AIST OpenRTM-aist
- 5. About Robot Middleware • Platform software that provides common functions to streamline robot system construction – Sometimes called "robot OS" – Commonization and standardization of interface and protocols – Examples • Providing modular or componentized frameworks • Supports communication among modules • Provides parameter setting, deployment, startup, and module composition functions • Realize inter-OS and inter-language cooperation / interoperability by abstraction • Development became active from around 2000 – Various middleware is being developed and released all over the world 5
- 6. 6 Conventional systems Controller Controller software Compatible interfaces are connectable Robot Arm Control software Robot Arm1 Robot Controller Program Target Robot
- 7. 7 Each robot has each control interfaces. If no compatibility, cannot connect each other. Robot Arm2 Robot Arm2 Controller software Humanoidʼs Arm Control software Robot Arm Control software Conventional systems Controller Robot Controller Program Target Robot Target Robot
- 8. 8 compatible arm interfaces RTM provides a common I/F for connecting separately made software modules Improved software reusability Easy to build RT system Controller software Arm A Control software Arm B Control software By using RT-Middleware Robot Arm2 Robot Arm1 Controller Robot Controller Program Target Robot Target Robot
- 9. Trend of Robot Software Development Component Oriented Development Conventional Style Camera Mic Actuator Speech Stereo vision Interaction Speech Recognition Image Recognition Middleware Stereo vision Interaction Speech Recog. Image recognition Camera Mic Actuator Speech ü Integrated design of various functions ü High run-time efficiency, but inflexible ü Development becomes difficult as the system becomes more complex ü Division/integration of large-scale complex functions ü Improvement of development and maintenance efficiency (reuse of functions, etc.) ü Increased system flexibility 9
- 10. The benefits of modularization • Reusability – A component can be reused in various systems. • Diversification – Various type of same functional modules can be tried in systems. • Flexibility – System structure can be changed easily. • Reliability – Easy to test a module and well tested modules are reliable. • Durability – Well divided and independent module error does not affect too much to whole systems.
- 11. 11 The benefits of RT-Component model • Provides rich component lifecycle to enforce state coherency among components • Defines data structures for describing components and other elements • Supports fundamental design patterns – Collaboration of fine-grained components tightly coupled in time (e.g. Simulink) – Stimulus response with finite state machines – Dynamic composition of components collaborating synchronously or asynchronously
- 12. Inactive Active Error Life cycle management, core logic execution Common state machine Sensor RTC Composite execution Real-time execution Controller RTC Actuator RTC Encoder component Actuator Component Controller Component 1 TI s TDs Kp + - Referencepos pos torque • Data centric communication • Continuous data transfer • Dynamic connection/disconnection Data-centric communication • User defined interface • Access to detailed functionality of RTC – Getting/setting parameters – Changing modes – etc… Image data 3D depth data Stereo vision interface ・set_mode() ・set_coordination() ・do_calib() ・etc… Service port Stereo Vision Component Data port Service oriented interaction Name Value Set name Name Value Set name • Function for internal parameter • Multiple parameter sets • They can be changed from remote in run-time Activity, Execution context Data Port Service Port Configuration Ex. Servo control Ex. Stereo vision RTC can have several configuration sets. Runtime reconfiguration and dynamic switching are supported Main features of RT-Component
- 13. Advantages of RTM based development 13 ℓ1 ℓ2 θ2 ( x , y ) θ1 Reusable RTCs Develop from scratch Self development using existing libraries 3D camera Point Cloud Grasping strategy Planning Inverse kinematics Arm control Gripper control By using RTM, existing module can be reused Reduce development from scratch Bin-picking robot example * This diagram is simplified version
- 14. RTM based Distributed Systems RTC RTC RTM Windows RTC RTC RTM uITRON RTC RTC RTM Linux RTC RTC RTM VxWorks RTC RTC RTM QNX RTC RTC RTM RT Linux RTC Application Input device Sensor Robot A Robot B Robot C Network RTM can manage distributed RTCs implemented by various languages or executed on various OSs on the network Connections Between RTCs Can be established dynamically
- 15. 15 The aim of RT-Middleware Problem Solving by Modularization • 仕様の明確化 • 最新技術を容易に利用可能 • 誰でもロボットが作れる Realize low-cost robots Satisfy various needs Cost Technical Issue Needs ！ ！ ！ ！ The state of the art Mobile base Manipulator Sensors・・・ Various users System developers Easy to customize RT-Component Utilize the state of the art Robot System Integration Innovation Reusable modules Spec.
- 16. Practical/commercialization examples Robot operation simulator: NEDO S-ONE︓SCHAFT DAQ-Middleware: KEK/J-PARC KEK: High Energy Accelerator Research Organization J-PARC: Japan Proton Accelerator Research Complex HIRO, NEXTAGE open: Kawada Robotics HRP series: KAWADA and AIST TOYOTA L&F︓Air-T VSTONEʼs education robots OROCHI（RT corp.） 16 THK: SIGNAS system
- 17. RTM as a International Standard Implementation Vendor Features Compatibility OpenRTM-aist AIST Reference implementation by AIST --- HRTM Honda R&D ASIMO is now moving to HRTM ◎ OpenRTM.NET SEC .NET(C#,VB,C++/CLI, F#, etc..) ◎ RTM on Android SEC RTM implementation for Android ◎ RTC-Lite AIST Tiny implementation on PIC and dsPIC 〇 Mini/MicorRTC SEC RTM/RTC for CAN and Zigbee 〇 RTMSafety SEC/AIST Functional safety standard capable RTM implementation 〇 RTC CANOpen SIT, CiA RTM for CANOpen standard 〇 PALRO Fujisoft Yet another C++ PSM implementation × OPRoS ETRI Implementation of Korean national project × GostaiRTC GOSTAI, THALES C++ PSM implementation on a robot language × Standardized by OMG process → It can not be modified by just one company → Various compatible implementation → It promoted competition and interoperability OMG Standard Users can chose and continue to use on of the RTM implementations Ten or more RT-Middleware implementation exist History • September, 2005 Request for Proposal issued (starting standardization) • September, 2006 Specification approved by OMG • April, 2008 OMG RTC ver.1.0 released • September, 2012 Updated to ver. 1.1 • September, 2015 FSM4RTC (FSM based RTC standard) adopted
- 19. Project web pages • Users can upload their own RTCs on the openrtm.org • Users can search and download other users RTCs 19 Project type Number RT-Components 405 RT-Middleware 14 Tools 19 Documents 4 Hardware related RTCs 28
- 20. RT-Middleware Summer Camp • 1 week camp every summer • This year: August 24-28 – The first online camp • Number of participants: 11 • Venue: AIST Tsukuba center (Tsukuba city, Ibaraki pref.) – Onlie (Zoom) • Lectures, practical work and presentation by five teams. • Staying in the AIST’s accommodation and coding endlessly every night :-P. 20
- 21. RT-Middleware Contest • Held as an organized session in SICE SI conference – Various prizes – Entry deadline: Sep. 23rd – Software registration：Oct. – Paper submission due: Oct. 26th – Online examination：from end of Nov. – Presentation and award ceremony: Dec. • Record of year 2019 – Number of applications ：11 – SICE RT-Middleware award x1 – Product supporting award x2 – Company supporting award x9 – Personal supporting award x10 • See more details: openrtm.org – Menu: community -> events 21
- 22. Recommendation • Letʼs stop reinventing the wheel!! – Code that has been executed thousands of times by different people works better than your code from scratch!! – Let's write the code you really need to write and borrow the other non-essential part for you. – A program released by someone is a program that has worked once!! – Other persons code is hard to read, but you shouldn't throw it away for that reason!! • Commit to open source projects!! – Don't hesitate to ask questions on ML and forums!! – No matter how rudimentary a question is, it is valuable information for others. – Letʼs complain to the project!! (good feedback grow the project) – Debug and send patches if you can!! 22
- 23. Conclusion • Basic concept and overview of RT- Middleware • Comparison with ROS • Activities of RT-Middleware community • RTC development overview 23
- 25. Framework and core-logic ステレオビジョン ルゴリズム ア コアロジック 右目画像 左目画像 デプスマップ RT ンポーネント レームワーク コ フ RT ンポーネント 準インターフェース コ 標 ステレオビジョン RT ンポーネント コ RT ンポーネント 準インターフェース コ 標 右目画像 左目画像 デプスマップ ＋ ＝ 中身は空 RTC framework ＋ Core logic ＝ RT-Component RT-Component framework Core-logic Stereo vision algorithm RTC standard interfaces RTC standard interfaces Stereo vision RT-Component empty Left image Right image Left image Right image Depth map Depth map
- 26. Code generation by model 26 RTC’s specification MyComp temp.sensor device temp. sensor RTC STATIC PERIODIC mode:TimedBool temp: TimedDouble name: category: description: comp_type: act_type: InPorts: OutPorts: RTCBuilder (Template code generator) C++ backend Java backend Python backend RTC source for C++ RTC source for Java RTC source for Python class MyComp : public DataflowComponent { public: virutal onExecute(ec_id); : private: TimedBool m_mode; TimedDouble m_temp; }; import RTC.DataFlowComponent; public class MyCompImpl extends DataFlowComponent { public ConsoleInImpl(mgr) { } : }; #/usr/bin/env python import RTC class MyComp( DataFlowComponent): def __init__(self, manager): : def onExecute(self, ec_id): : Same (model) component will be generated by same specification, even if implementation languages is different Core-logic is embedded to the template code Implement the procedure in a specific function of the generated class
- 27. RTC development flow 27 RTBUilder CMake Visual C++ Input spec of RTC Generating VC project or Makefile Implement logic and compile RTBUilder CMake make + gcc (g++) Windows Linux Almost all steps are the same, except compiler
- 28. CMake • Open source software for compiler- independent build automation • Can generate build files for different development environments on different operating systems • Generate Makefile on Linux • Generate VC (Visual C++) project file on Windows • Most of the recent open source software is built with CMake. 28
- 29. State machine and lifecycle of RTC ActiveDo/RTC::onExecute() callback function is here You don’t need to aware here Be aware state here and its callback functions.
- 30. 30 30 Activity (Callback functions) Callback functions Meanings onInitialize Initialization onActivated Called once when RTC is activated onExecute Called periodically when RTC is in the active state onDeactivated Called once when RTC is deactivated onAborting Called once when entering ERROR state onReset Called once when resetting onError Called periodically when RTC is in the error state onFinalize Called once when finalizing RTC onStateUpdate Called after onExecute everytime onRateChanged Called when ExecutionContext’s rate is changed onStartup Called once when ExecutionContext starting onShutdown Called once when ExecutionContext stopping
- 31. InPort • InPort – Input port for data flow type communication • Methods of InPort class – isNew(): check if new data arriving – read(): retrieve data from InPort buffer to the variable bound to the InPort – >> : same as above リングバッファ バインドされた変数 read() ! " # $ % & ! $ ' ' 最新値 InPort Robot Component Sensor Data Example Basically paired with OutPort Data ports (InPort/OutPort) must have the same type
- 32. OutPort リングバッファ バインドされた変数 ! " # $ % () & ' % " ( $ & " ) ) * + $ Port 最新値 Sensor Component Sensor Data 例 Basically paired with InPort Data ports (InPort/OutPort) must have the same type • OutPort – Output port for data flow type communication • Methods of OutPort class – write(): push data from OutPort’s variable into OutPort’s buffer to be published to the remote InPort – << : same as above
- 33. Recommendation • Letʼs stop reinventing the wheel!! – Code that has been executed thousands of times by different people works better than your code from scratch!! – Let's write the code you really need to write and borrow the other non-essential part for you. – A program released by someone is a program that has worked once!! – Other persons code is hard to read, but you shouldn't throw it away for that reason!! • Commit to open source projects!! – Don't hesitate to ask questions on ML and forums!! – No matter how rudimentary a question is, it is valuable information for others. – Letʼs complain to the project!! (good feedback grow the project) – Debug and send patches if you can!! 33
- 34. Conclusion • Basic concept and overview of RT- Middleware • Comparison with ROS • Activities of RT-Middleware community • RTC development overview 34