The demand for sensing in robotics and automation has increased because of a decrease in the labor force. Recent advances in computational performance have bolstered the widespread use of image processing technology across various applications. This special issue aims to provide researchers with the opportunity to access the latest research and practical case studies on advanced image processing, computer vision, and sensing techniques for robotics and automation. The topics of interest in this special issue are as follows:

1) Theory and algorithms: Image Processing, Computer Vision, Pattern Recognition, Object Detection, Image Understanding, Media Understanding, Machine Learning, Deep Learning, 3D Measurement, Simultaneous Localization and Mapping (SLAM), Multispectral Image Processing, Visualization, Virtual Reality (VR) / Augmented Reality (AR) / Mixed Reality (MR) , Datasets for Image Processing;

2) Industrial applications: Factory automation, machine vision, visual inspection, monitoring, surveying, logistics;

3) Sensing techniques for robotics and automation: Robot vision, advanced driver-assistance systems (ADAS), autonomous driving, robotic picking, assembly, and palletizing;

4) Image processing hardware and software: Image acquisition devices, image sensors, image processing systems, sensor information processing;

5) Man machine interface: Visualization, human interface devices.

We extend our heartfelt gratitude to all the contributors, reviewers, and editorial staff for their dedication and support in realizing this special issue.

In this study, we propose a system to detect changes in three-dimensional (3D) space for autonomous plant visual inspection by a mobile robot. The videos captured by a mobile robot during past inspections are compared with the videos obtained during the current inspection using both pose information and the acquired images. To ensure robustness against changes in shooting conditions, change detection is executed employing deep learning techniques. Subsequently, the detected information is projected onto a 3D space to localize the changes. To verify the effectiveness of the proposed method, experiments were conducted both in a real plant environment and a simulated indoor plant environment. The results of the outdoor experiments showed that the proposed system achieved image pair determination, change detection, and integration into a 3D space. The results of the indoor experiments and evaluations confirmed that the proposed method for image pair determination was suitable based on considerations of detection accuracy and computation time.

In this paper, we propose a novel approach to detection and localization of abnormal sound sources for robotic inspection in oil refineries. Such environments are difficult environments with high noise from multiple machines and where swift detection of anomalies is critical. The rarity of anomalies hinders the gathering of a balanced training dataset for the common supervised learning approach. Our previous work, based on autoencoders, bypassed this issue but lacked the ability to locate the abnormal sound source. Our proposed method first learns a spatial map of the normal sounds, allowing to predict what sound should be present at each robot position. This enables a detection based on a comparison between the predicted and observed sound. Localization can then be conducted based on this comparison using optimization. Experiments conducted in laboratory conditions showed the effectiveness of the proposed method. Additionally, experiments in field conditions in an actual oil refinery further showed the potential of the proposed method.

Accurate calibration of robot joint compliance poses a significant challenge with limited existing research. For camera-based calibration, concurrently identifying both joint offsets and compliance errors becomes intricate due to measurement inaccuracies. To overcome this problem, this paper proposes an innovative approach that leverages measurement pose optimization. By leveraging the local product of exponentials (POE) model, our method enables the simultaneous identification of the geometric parameters (joint offsets) and non-geometric parameters (joint compliance). The introduction of a modified visual observability index minimizes sensitivity to camera errors during joint compliance calibration. Experimental results conducted on a 6R serial robot show superior accuracy compared to existing indices, validating the effectiveness of our approach.

In this paper, we propose a flexible device control method using personalized command spaces that function as buttons on a virtual remote control that follows the user. By performing two different gestures in each space, the users can control various devices in a room. This system is implemented through multiple cameras and 3D human keypoint tracking. We experimentally evaluated the influence of command spaces arrangement on gesture recognition and determined the recognition accuracies for different gestures in each command space. The system demonstrated high usability, with even inexperienced users achieving high gesture recognition accuracy.

This study proposes a method to efficiently reduce distortion effects in equirectangular images. Spherical cameras provide a wide field of view, advantageous for localization tasks. When applying standard image processing techniques to spherical images, they are commonly converted into equirectangular images by equirectangular projection, which introduces geometric distortions that can impair localization accuracy. Existing approaches for distortion mitigation frequently encounter a trade-off between accuracy and processing speed. We propose a method that mitigates distortion effects while reducing computational costs to overcome these limitations. Our method incorporates an innovative strategy for image rotation and region selection, improving computational efficiency in feature detection and description. Experimental results for two-view pose estimation, an essential component of localization, showed that our method achieves the fastest processing speed while maintaining accuracy comparable to that of distortion mitigation techniques.

Object detection is a fundamental problem in computer vision that has been extensively investigated over the past decades. Although deep neural networks (DNNs) improve object detection, they cannot effectively recognize small objects. Detecting small objects remains challenging owing to several factors, such as low resolution and scale variance. These challenges are particularly evident in object detection using surveillance cameras, where small objects typically appear in cluttered environments and at varying distances. In object detection using surveillance cameras, the static nature of the background has not been fully exploited in DNN-based object detection methods, although it can be an important cue for detection. In this study, we propose a simple yet effective method to enhance object detection in areas of small objects not detectable by state-of-the-art DNN-based instance segmentation methods. The proposed method extracts foreground regions using a background subtraction model and classifies them, thereby enabling the identification of small objects. In our experiments, we evaluate two real-world scenarios: detecting a person walking on campus and identifying vehicles in road-surveillance footage. The results show that our method improves the detection of small objects and performs better than baseline methods.

The efficient investigation of fishery resources is critical for rapidly understanding the effects of abrupt environmental changes. Seabed imagery has been used extensively for resource assessment in scallop fisheries in the Sea of Okhotsk, Hokkaido, Japan. However, the potential of these images for the broader investigation of epibenthos remains unclear. In this paper, we propose an automatic detection method for epibenthos from seabed images using deep learning, specifically Mask R-CNN and Mask2Former models. We focus on four species: Asterias amurensis, Distolasterias nipon, Halocynthia aurantium, and Patiria pectinifera. The Mask R-CNN X101-FPN 3x model show the highest overall accuracy, with a mask mAP of 77.8%, whereas Mask2Former excelled in specific species detection. The trained models are successfully used to generate epibenthic distribution maps, demonstrating the effectiveness of the proposed method for monitoring large-scale marine ecosystems. This approach significantly enhances our ability to conduct comprehensive assessments of benthic communities, thereby providing an effective tool for marine-biodiversity assessment and fishery resource management.

The number of parakeratotic corneocytes is an important parameter for diagnosing stratum corneum conditions. However, parakeratotic corneocytes are often visually diagnosed by an expert, which involves human error and is time-consuming. In this study, we proposed a method for classifying corneocytes, parakeratotic corneocytes, and ghost nucleus corneocytes. Our proposed system extracts each corneocyte region from a BG-stained image using a trained cell-specific deep learning model. We evaluated a method to classify corneocytes, parakeratotic corneocytes, and ghost nucleus corneocytes using different deep learning models: VGG16, VGG19, EfficientNet, EfficientNetV2, and Vision Transformer. The results showed that Vision Transformer achieved a 99.08% accuracy rate, which was sufficient for the diagnosis of stratum corneum conditions via imaging.

In laser drilling, precise parameter optimization is essential to achieving the desired hole characteristics. This study investigates the influence of the pulse width and pulse spacing on the machined hole geometry and proposes an artificial intelligence-based framework to predict hole shapes in multilayer composite substrates. The distribution of hole diameters resulting from CO₂ laser machining was evaluated via response surface methodology, considering variations in the pulse width and irradiation time. The results demonstrated a strong dependency of the hole diameters on the laser conditions and revealed significant autocorrelation among the machined-hole parameters.

In this paper, we propose a method for generating reaching motions for the insertion of flat cables. Despite the need to insert flat cables into sockets in the circuit assembly of various electronic devices, there has been little research on automating the insertion flat cables that are already fixed on one side to the circuit board. In this regard, we focus on the generation of reaching motions in the posture for grasping such flat cables. Our method uses deep reinforcement learning in a simulation environment, and the features extracted from the image and the pose of the manipulator are used as states. For the transfer from the simulation environment to the real-world environment, we use a CycleGAN-based domain adaptation method. We conducted experiments under several different conditions in a real-world environment to verify operation of the trained agent. The results demonstrated that the success rate of the generated reaching motions exceeded 70% under all conditions.

High-precision displacement control for water-hydraulic artificial muscles is challenging because of their strong hysteresis characteristics, which are difficult to be modeled precisely. Recently, data-driven control methods have attracted considerable attention because they do not explicitly use mathematical models, making the design much easier. In our previous work, we proposed a fictitious reference iterative tuning (FRIT)-based model predictive control (FMPC), which combines data-driven and model-based methods for the muscle, and showed its effectiveness because it can also consider input constraints. However, the problem in which control performance strongly depends on prior input-output data remains unsolved. Adaptive FRIT (A-FRIT) based on directional forgetting has also been proposed; however, achieving the desired transient performance is difficult because it cannot consider the input constraints, and there are no design parameters that directly determine the control performance. This paper proposes a novel data-driven adaptive model matching-based controller that combines MPC with the A-FRIT. The experimental results show that the proposed method can significantly improve the control performance and achieve high robustness against inappropriate initial experimental data while considering the input constraints in the design phase.

Injection molding is the most widely used process for manufacturing plastic products. Because the quality of plastic products depends significantly on the process conditions, an accurate understanding of the state inside the mold is crucial. This approach leads to front-loading, reducing losses and stabilizing the quality of molded products. In recent years, the importance of computer simulations for minimizing the number of experimental iterations required for manufacturing and for reducing the overall development costs has increased. However, concerns regarding the accuracy of computer simulations are increasing as well, thus highlighting the necessity to address uncertainties in simulation conditions. Herein, we propose a high-precision state-estimation method using computational fluid dynamics (CFD) and data assimilation to accurately understand the state inside a mold. Computer simulations are initially performed using OpenFOAM. Subsequently, pressure sensors are installed inside the mold to obtain observational data. The simulation results are compared with the observed data. Next, data assimilation is performed to improve the simulation accuracy and investigate the internal state of the mold more accurately. The data-assimilation method employed in this study is the ensemble Kalman filter. We successfully demonstrated the effectiveness of high-precision state estimation using CFD and data assimilation.

A trajectory, which determines the change of the displacement and posture of a robot with time, influences dynamic torque or energy during the operation. Although various methods for optimizing the trajectories have been proposed, most of them require the exact model that is difficult to construct. Previously, the authors proposed the method for optimizing the trajectories of an industrial robot without the dynamic model by using a heuristic algorithm. The proposed method can reduce the energy or peak current value consumed in the motor. However, the proposed method cannot be available for the operation where the required path of a robot often varies, because those cases need to explore the optimal trajectory with each path, and then a quite few numbers of driving the actual robot for exploring will be required. Therefore, this paper proposed a method for instantaneously optimizing the trajectories by using a neural network (NN) that is constructed by using the optimal trajectories obtained with the heuristic algorithm. In this paper, first, the optimal start position and the optimal operation times of an industrial robot is respectively explored by the heuristic algorithm using the average power consumed in the motor as the evaluation value. Next, a NN learned the parameters of the optimal trajectories explored by the heuristic algorithm for various paths. The constructs NN will generate optimal trajectories that reduce the evaluation values to the same extent as the trajectories explored by the heuristic algorithm. Namely, the proposed method using the NN can estimate the optimal trajectories in a shorter time than the method using the heuristic algorithm.

In mechanical systems such as cranes and stratospheric balloon systems, the suspended load is affected by translational acceleration and wind disturbances, resulting in sway and rotation, which can pose safety risks, and reduce operational efficiency. Previous research has focused on controlling either sway or rotation individually; however, this study proposes a new method for simultaneously controlling both. In bias moment control, a high-speed rotating momentum wheel is attached to the suspended load, inducing a gyroscopic effect that alters the plane of oscillation in a manner similar to the behavior of a Foucault pendulum. This method requires minimal control input, making it possible to control both vibration and rotation simultaneously. As a result, it offers a practical solution that enables a smaller and lighter control system. The proposed method builds on previous research involving angular momentum exchange devices, such as reaction wheels and control moment gyroscopes, but is considered to offer a more compact and efficient control mechanism. In this study, the motion characteristics of the control system were analyzed in detail, and its control performance was evaluated through a combination of mathematical modeling and numerical simulation. Additionally, the relationships among parameters such as control gain, bias angular momentum, and damping performance were investigated. The results demonstrate that the proposed method is an effective approach for improving the control of suspended loads and is expected to provide practical benefits for a wide range industrial applications.

Grinding is used to finish thrust metal attachment parts, such as crankshafts, which have both journal and thrust surfaces. In side plunge grinding, a thrust surface and a cylindrical surface of a shaft workpiece with collars are finished in a single plunge grinding process. However, the surface quality near the ground internal corner, where grinding fluid may not penetrate, can deteriorate, causing high residual stress and cracks owing to grinding heat. While it has been reported that quality issues at the inner corners of the ground surface can be mitigated by reducing the grinding point temperature through efficient cooling fluid supply, the mechanisms of grinding phenomena and heat generation in side plunge grinding are not yet fully understood. In this study, the variations in the grinding temperature at the thrust surface of a workpiece with a collar were experimentally investigated using a wire/workpiece thermocouple to clarify these phenomena. The results revealed a significant increase in the grinding temperature at the corners of the grinding zone. However, it slightly decreases as the thermocouple output approaches the center of the workpiece, indicating a slight effect of the grinding speed. The surface temperature of the workpiece in side plunge grinding is primarily influenced by the wheel depth-of-cut in the thrust direction. Additionally, the effect of workpiece rotational speed and grinding infeed speed on temperature distribution has been demonstrated.