Wise things news on November 3,Yesterday, Zuckerberg stated in his Facebook account that Meta (Facebook’s new name) released a touch-sensitive “skin” ReSkin, which was jointly researched by the company’s researchers and Carnegie Mellon University in the United States.
Compilation | Cheng Qian
Editor | Li Shuiqing
ReSkin uses machine learning and magnetic induction, and uses unsupervised learning algorithms to help automatically calibrate the touch sensor, providing a cheap, versatile, durable and replaceable long-term solution.
Zuckerberg said that ReSkin has been tested in a variety of application scenarios. For example, researchers put ReSkin and circuit boards on the soles of dog shoes to track the amount of force the dog exerts during rest, walking, and running. And direction. “It can also unlock the possibilities of AR and VR, and lead the innovation of industrial, medical and agricultural robot technology.” Meta research scientist said.
▲ReSkin worn on the dog’s feet (picture source is Meta)
In addition to ReSkin, Meta also outlined progress in the development of hardware, simulators, data sets, and benchmarks for touch sensing testing in the field of tactile perception. The company said that it has formed the basis for AI systems to understand and interact through touch.
In addition, Meta open-sourced ReSkin’s design, data documents, codes and basic models to help other researchers use the sensor faster, further promote the development of tactile sensors, and promote the application of the AI industry.
1. From 18mm to 2mm, Meta digital skin has evolved
“We usually think of touch as a way to convey warmth and caring, but it is also a key way for us to perceive the world around us.” Calandra and Lambeta said, “Touch provides us with information that cannot be discerned by any other senses, such as about matter. The temperature, texture, weight, and sometimes even its state.”
“Tactile sense helps us’navigate’ in the world around us. With it, we can collect information about objects, such as whether they are light or heavy, soft or hard, stable or unstable, and we can also use touch to complete the From shoes to the daily tasks of preparing meals.” Meta researcher manager Abhinav Gupta and postdoctoral researcher Tess Hellebrekers said.
Tactile sensing is an emerging field in robotics, which aims to understand and replicate human-level touch in the physical world. In environments such as homes to factory floors, robots can learn and use their own “touch”, thereby making robots more sophisticated. Efficient, safer and gentler.
In the past few years, Meta has been developing tactile sensors, mainly focusing on robotic grasping tasks. In 2020, Meta has launched a high-resolution, low-cost small tactile sensor Digit, which can be installed on the hand of a multi-finger robot.
▲Digit’s decomposition diagram (picture source is Meta)
Digit’s plastic body shell can be made by 3D printing and injection molding. At the same time, Digit is also equipped with three RGB light-emitting diodes, as shown in the figure below, which can provide illumination for custom-designed elastomer gel surfaces using silicon and acrylic manufacturing processes, which can balance robustness and sensitivity. During the Digit production process, a “press fit” connection process was used to mount the camera and gel on the body so that the components can be replaced, and the housing can also be replaced to accommodate different lens focal lengths.
▲Digit installed on the manipulator (picture source is Meta)
When experimenting with Digit, it was equipped with a manipulator, and the research team used the thumb and middle finger of the manipulator to hold and manipulate glass marbles. In the course of 50 trials, the marbles in the manipulator will drop approximately 25% of the time. The researchers attribute this to the inaccuracy and variability of the data, rather than a flaw in the Digit design.
Digit’s plastic case, gel, and electronics manufacturing files, as well as the firmware binary files used for programming, were open sourced on GitHub in June last year. At the same time, Meta recently announced that it will cooperate with MIT’s spin-off company GelSight to produce Digit.
Compared to Digit, the now released ReSkin is a deformable elastomer embedded with magnetic particles.
▲Demonstration model of deformable elastomer ReSkin
Secondly, ReSkin may be cheaper than Digit. The cost of producing 100 pieces of ReSkin is already less than US$6, while the cost of producing 1,000 pieces of Digit is still US$15. According to Gupta and Hellebrekers, ReSkin has a thickness of 2 mm to 3 mm, while Digit has a thickness of 18 mm and can perform more than 50,000 interactions. This makes ReSkin a variety of shapes from robotic hands, tactile gloves to arm sleeves and even dog shoes. Ideal choice.
▲ReSkin installed on the manipulator (picture source is Meta)
“ReSkin can also provide high-frequency three-axis tactile signals for quick operation tasks, such as sliding, throwing, picking, and clapping. When it wears out, it can be easily peeled off and replaced with a new one.Accessories. “Gupta and Hellebrekers said.
According to Gupta and Hellebrekers, the actual goal of ReSkin is to establish a contact data source, which may help integrate data resources and further advance artificial intelligence when performing a series of touch-based tasks such as object classification. The AI model with tactile perception skills developed by ReSkin may also be used to work in a healthcare environment or grasp soft objects.
ReSkin can be integrated with other sensors to collect visual, sound and touch data to create a multi-modal data set, thus helping to build a more realistic world model than before.
▲ReSkin sensor is used to measure tactile force (picture source is Meta)
“Today’s artificial intelligence effectively integrates the senses such as vision and hearing, but touch is still an ongoing challenge due to the limited access to tactile sensor data outside the human body. Therefore, AI researchers hope to take advantage of the richness and redundancy of people’s tactile perception. To better integrate the sense of touch into their model.”
2. ReSkin is expected to solve mass production problems
It is worth noting that neither Digit nor ReSkin are the first tactile sensors in this field. Other tactile sensors include OmniTact developed by a research team at the University of California, Berkeley and robotic grippers from the MIT Computer Science and Artificial Intelligence Laboratory. GelFlex, also used by the National University of SingaporeIntelThe prototype chip developed a touch-sensing robot “skin”.
▲The robot gripper GelFlex is grabbing objects (picture source is Massachusetts Institute of Technology)
However, these previous practices have shown that “soft skin” is difficult to manufacture on a large scale because they will change during the manufacturing process. First, the material properties of the device itself will change over time, and secondly, The use of different materials will also make changes, which all add challenges to the large-scale manufacturing of “skins”. At the same time, each sensor must determine its own response results through a calibration procedure, which also means that the calibration procedure must adapt itself to the above-mentioned changes.
ReSkin uses machine learning and magnetic induction, and adopts an unsupervised learning model to reduce contact during equipment installation and use, and reduce its damage rate, which can alleviate the difficulty of large-scale manufacturing to a certain extent.
ReSkin eliminates the electrical connection between the soft material and the measuring electronic equipment, eliminating the need for close contact with the connection, ensuring that the material is not interfered by the outside world, and thusgramTo overcome its difficulties in large-scale manufacturing. The magnetic signal of the tactile sensor depends on the distance close to it, so the electronic device only needs to be nearby and does not need to be connected to receive the magnetic signal.
In addition, ReSkin has also developed a mapping function that trains data from multiple data sources, making it more versatile and robust than traditional mapping functions. And ReSkin’s sensor uses an unsupervised model, which can be automatically and continuously fine-tuned using a small amount of unlabeled data.
▲ReSkin’s unsupervised model demonstration
In unsupervised learning, the algorithm is affected by “unknown” data, which does not have previously defined categories or labels. This is the opposite of “supervised” learning. In “supervised” learning, algorithms are trained on input data annotated with specific outputs until they can detect the underlying relationship. Those unsupervised machine learning systems running on ReSkin must learn from inherent data and be able to classify and process unlabeled data, rather than learning from annotations.
“We can use the relative position of the unlabeled data to help fine-tune the calibration procedure of the sensor instead of providing a priori mandatory labeling. For example, we know that of the three contact points, the two that are physically closer to each other will have more similar Tactile signals.” Gupta and Hellebrekers explained.
“All in all, ReSkin opens up a multi-functional, scalable, and low-cost tactile module that cannot be achieved with existing systems. Existing camera-based tactile sensors require the distance between the device surface and the camera to be minimized, resulting in a more cumbersome design. In contrast, ReSkin can be used as a surface layer to cover the hands and arms of humans and robots.”
“Our research on universal tactile sensors gave birth to today’s ReSkin, which has the advantages of low cost, portability and long battery life. Secondly, its skin is as easy to replace as peeling and putting on a new bandage, and it can be used immediately. We The learned model performs well on new devices out of the box. This is a powerful tool that will help researchers build AI models to power a wide variety of applications.” Gupta and Hellebrekers wrote.
3. Open source simulator, learning framework, basic model…
To support hardware like Digit and ReSkin, Meta this summer open sourced Tacto and PyTouch, which are libraries for the PyTorch machine learning framework. Tacto is a vision-based tactile sensor simulator, while PyTouch is a collection of machine learning models and functions for touch sensing.
▲PyTouch interface
The Tacto simulator can present touch readings at hundreds of frames per second, and it can be configured as different sensors, including Meta’s own Digit. As Calandra and Lambeta pointed out, simulators play an important role in prototyping, debugging, and robot benchmarking because they can pass the test and avoid expensive experiments. They said: “Simulation experiments can make the device run faster. In addition, the correct hardware can also be obtained through simulation experiments, and the wear and tear of the hardware surface in tactile sensing can be reduced. This makes the simulation change for touch sensing. More important.”
As for PyTouch, it provides basic functions for the sensor, such as detecting touches and sliding, and estimating the posture of objects. PyTouch can integrate real-world sensors and Tacto to achieve model verification and transfer the concept of simulation training to the “Sim2Real” function in real-world applications. Meta also envisions that PyTouch will enable the robotics community to use models dedicated to “as a service” tactile sensing, where researchers can connect sensors, download pre-trained models, and use them as components in applications part.
“We are currently studying the Sim2Real transfer for training PyTouch models in simulations and deploying them on real sensors, and as a way to quickly collect data sets and train models.” Calandra and Lambeta said, “In simulations, Collecting large-scale data sets containing large amounts of data can be completed in a few minutes, and collecting data using real sensors requires time and manpower to physically detect objects. Therefore, we plan to explore the Real2Sim method to better adjust the simulator from real data. “
There are a number of obstacles to overcome in tactile perception, including hardware limitations, a lack of understanding of which touch functions are used for specific tasks, and a lack of widely used benchmarks.
To overcome the above obstacles, Meta took a small step and released ReSkin’s design, data documents, code, and basic model to help researchers use the sensor without collecting or training their own data set.
Conclusion: Multiple players work together to unlock more possibilities with touch sensors
Meta released ReSkin, open source a large amount of software and data, use machine learning and magnetic induction, and use unsupervised learning models to further explore the field of touch sensors and is expected to make significant progress in the field of machine learning.
Generations of general-purpose touch sensors have given birth to ReSkin, which has taken a step forward in the mass production problem, which is inseparable from the efforts of many companies and researchers on the stuck neck problem. Meta said that regardless of the incremental development of touch sensors, it can help advance AI technology and help researchers build robots with enhanced features.