Image processing based package volume measurement system using kinect sensor

Abstract

Keywords: Camera calibration; Hough transform; Image processing; Kinect; Package volume detection

Introduction

Volume measurement systems are of interest to logistics companies and postal service providers to save time and costs. The trend in this area has concentrated on non-destructive measurement systems, which are mainly based on image processing techniques [1]. Various hardwares are currently being used, such as laser-based sensors, infrared sensors, stereo systems and acoustic systems. Laser sensors can be basically a combination of point lasers or high-cost 2D laser scanners

[2]. In these systems misalignment between the reference and the package can lead to incorrect calculations. Deformation of the package surface is another major problem associated with such measuring approaches. More sensors should be used to increase the accuracy. It is also possible to measure volume on a moving platform using point laser source and camera together. Such an approach is also used in 3D laser scanners and studies have been carried out using these sensors for volume

*Corresponding author. *E-mail address: gurkan.kucukyildiz@usak.edu.tr This paper was recommended for publication in revised form by Regional Editor N. Özlem Ünverdi Published by Yıldız Technical University Press, İstanbul, Turkey Copyright 2021, Yıldız Technical University. This is an open access article under the CC BY-NC license (http://creativecommons.org/licenses/by-nc/4.0/).

measurement in various studies [3]. The study given in [4] has been able to provide accurate measurements up to 1 mm. Camera based methods are also used in volume measurement applications. Image-based volume measurement approaches can be classified as stereo camera and structured light systems [5]. Stereo systems use depth data provided by a pair of calibrated cameras [6]. An example study of such approach is provided in [7]. The authors used two calibrated cameras to measure the volume and position of a box. They calculated the vanishing points of the box border to obtain the 3D model. The measurement accuracy of these systems depends on various factors, such as proper stereo calibration, image resolution, camera and lens quality. Moreover, such systems may be adversely affected by varying visible light conditions. Therefore, camera-based systems typically require a natural light-free environment where the amount of light is controlled by artificial light sources. Unfortunately, this may not always be appropriate to implement in many work places. It also has a relatively higher computational burden in real-time applications. Structured light systems are a current approach to the problem of volume measurement [8]. Another approach consists of a single camera and a rotating platform, which usually measures the volume of small volume objects. The main problem in this approach is that the objects to be measured are smaller than the common-used commercial products and that no satisfying results are obtained in real-time industrial applications [9]. Volume measurement methods are also available by processing single images obtained from different angles. The surface volume of the human body was estimated by applying the Elliptical Tube Volume method given in [10]. It is known that it is far from being an industrial method. Another hardware used for the volume measurement is the depth sensors. Kinect, which operates on the principle of structured infrared light, offers a measurement range of 0.7 cm to 6 m for (mostly) indoor environments. The measurement error increases up to a maximum of 4 cm proportional to the increase in distance. Depth images that can be captured with high-cost infrared cameras before the development of Kinect are now easily available. Depth data can be used in many areas such as medicine [11], gaming technology [12], mapping [13] and indoor layout [14]. Academic interest in Kinect continues to grow. Kinect has been used in a variety of areas such as hand gesture recognition [15], human tracking, and avoidance of obstacles [16]. Kinect depth data processing on transparent surfaces is a compelling problem. Improvement on the performance of the accuracy level of the depth data on transparent surfaces is presented in [17]. Kinect calibration studies were performed. In a study [18], calibration is performed by a quadratic transformation. Kinect data was used as 2D laser distance data in certain conditions [19]. Kinect has disadvantages with 2D laser scanners, such as narrower viewing

angles and larger output data. On the other hand, Kinect is more cost-effective than 2D laser scanners. Alternative depth cameras were calibrated and compared on the same platform [20]. The authors presented that SLR stereo was the most accurate while the SR-4000 was the least accurate. In another study, which presented the volume-intersection approach using Kinect, finding the volumes of objects of various geometry, the volumes of medium-sized household objects could be calculated with an average error of 5.2% [21]. In our study, we have developed a measurement platform supported with Kinect. The sensor was mounted on a custom made measuring cage shown in Figure 1. Although the proposed system has been developed for volume measurement of box-type packages, it also provides an approximate solution for randomly shaped objects. The proposed system is not only a cost-effective alternative to existing systems, but also provides accurate measurements of up to 1 cm at a distance of 1.5 m. Unlike stereo camerabased systems, measurement accuracy is not affected by changing light conditions unless placed directly under sunlight or infrared light source. The method presented in this study is not affected by visible light, unlike stereo camera based systems. It significantly increases the measurement capacity by providing continuous operation in both dark

and illuminated environments. It can operate with the plugand-play principle without the need for additional camera calibration and light control despite changing operating conditions. The proposed method is robust against disturbances in the sensor field of view. The rest of the article is organized as follows: In section 2, the calibration routine, background subtraction and Hough transform are presented. In section 3, the results are interpreted. Finally, the conclusions are discussed in section 4.

Technical Background

Kinect calibration for measuring on planar surface The calibration process is basically performed to compensate for the misalignment caused by the fact that the surface of the measuring plate on which we put the object is not parallel to the kinect camera plane. There are frequently used parameters and analytical solutions in the calibration stages. Calibration process between Kinect and measuring plate is given in steps by using parameters and equations which can be examined in the literature [22-24]. A problem-specific solution was proposed during the calibration stage. Roll and pitch angular misalignments were calculated with the assumption that the height of the sensor from the ground is known without the use of tools such as calibration board in standard calibration procedures. Intrinsic matrix depends on camera internal parameters such as focal length (f), principal point (cx, cy), pixel width (sx) and pixel height (sy). The intrinsic matrix, M, of a camera is given in equation (1).  f − s  x  M = 0   0 

The internal parameters of Kinect were obtained from Kinect Software Development Kit (SDK). The parameters are given in Table 1. Real world coordinates of a pixel can be calculated using equation (2).

Figure 2. Misalignement of the sensor and tranformed coordinates.

 xk  z  i   k −1    yk  = M  j  z  1   k 1  

Extrinsic matrix depends on the camera placement. Because of the imperfections in the manufacturing process, the measurement and the Kinect mounting plates may not be perfectly aligned. Consequently, it results in the depth value of each point on the measuring plate being different. Therefore, it is necessary to make a coordinate transformation so that all points on the measurement plate should have the approximately equal depth value. To make this transformation, the rotation matrices of the Kinect coordinate system relative to the x and y axes (roll and pitch angles) must be defined. Rotation matrices (Rx, Ry) around the x and y axes are given in equation (3). A single point in Kinect coordinate system (pk=[xk yk zk]) can be transformed to a corresponding point (pm = [xm ym zm]) in the new coordinate system parallel to x-y plane of the measurement plate by equation (5). 1 0 0    R x = 0 cos( θ ) − sin( θ ) , 0 sin( θ ) cos( θ )     cos( φ ) 0 sin( φ )    Ry =  0 1 0   − sin( φ ) 0 cos( φ )  

Combined rotation matrix (R = RxRy) is given in equation (4). Rz is neglected since rotation around z-axis does not affect the depth measurement. The elimination of the roll-pitch angular misalignment is shown visually in Figure 2 for one dimension.  c(φ ) 0 s(φ )    R =  s(θ )s(φ ) c(θ ) − s(θ )c(φ )  − c(θ )s(φ ) s(θ ) c(θ )c(φ )   

Equation (5) can be rewritten as, xm = r11 x k + r12 y k + r13 z k ym = r21 x k + r22 y k + r23 z k

where rij is the i-th row and the j-th column element of R. The depth value dm must be constant for any point on the measurement plate. The third row of equation (6) can be rewritten as equation 7, (7)

where r3 is equal to the third row of the rotation matrix R and dm is equal to the depth value of the measurement plate. The superscript T means the transpose of the matrix. Every

discrete point on the measurement plate must satisfy equation (8). N points were selected on the measuring plate to find solutions for r3 and dm. The set of N equations is reexpressed in matrix format as following equation.

This equation can be solved by using singular value decomposition (SVD). The solution is the null space of A. As a result of the solution, R matrix is defined using θ and ϕ angles. The equivalents of a point in the Kinect space in a new coordinate system that is concentric but with a

distortion equal to θ and ϕ angles are calculated in the last step by applying equation (5). Figure 3 depicts the depth measurements taken at various points on the measurement plate before and after a sample calibration procedure. The blue circles and the red crosses represent the original and the corrected depth values, respectively. Table 2 provides the mean, standard deviation and the range of the original and corrected measurements. It can be seen that the standard deviation of the original depth values on the plate is very high compared to the one for the corrected depth values. After the calibration procedure, depth values on the plate deviates from the mean by approximately ±2 cm, which is roughly equivalent to Kinect’s measurement accuracy at 150 cm. In the image on Figure 4.a, the depth map shows an inhomogeneous distribution on the ground and the measuring plate. This is due to the fact that the sensor is not fully parallel to the ground and this is converted to the format on the right by applying the recommended calibration process. In this way, the depth information of the upper

surface of the box to be measured is evenly distributed (Figure 4.b). Background Subtraction Since a height measurement will be made with reference to the depth of the measuring plate, the average of the first 20 depth images is calculated before the initialization. Images read at 30fps for a period of 3.3 ms are subjected to averaging to minimize noise. The averaging depth image is assumed to be the background (reference) image. In the next step, the volume measurement algorithm is triggered when pixels above a certain threshold are detected on the measuring plate. Step by step definition of the background substraction is given below. • Estimate the background depth matrix by averaging the 20 frames captured at the initial state, • Subtract the background depth matrix from the input frame, • Apply a treshold to the absolute difference to obtain the foreground image (height of the box on the measurement plate), (equation 10). I(x , y , t ) , n = 20 n t =1 20

Table 2. Original and calibrated image statistics Original Depth Corrected Depth Image Image Standard Deviation

I f (x , y , t ) = I(x , y , t ) − Ib > threshold where, Ib is the background depth matrix, I(x, y, t) is input frame at time t and If(x, y, t) is the foreground image at time t. At this stage, pixels with a height higher than a certain threshold in depth value are considered as a possible

Figure 4. Original (a) and corrected (b) depth map after calibration.

Figure 5. Sample (a), background (b) and morphological operations applied (c) images.

Hough Transform Theory of the Hough transform was presented to literature in 1972 [23]. Another major study given in [25] proved that geometric forms defined in parametric expressions, can be detected by using Hough transform. Hough transform can be described in equation (11).

Hough transform has robustness against gaps and noise, but its computational cost can be challenging in real time applications [24]. Pseudo code of Hough transform is given in the following statement: For each pixel in the binary image{ If the pixel is white { For each θ { Compute ρ = xcos(θ)+ysin(θ)

box and used to obtain a black and white (BW) image. Morphological opening was applied to the image for noise removal. Morphological closing and region filling were then applied to fill the gaps. After morphological operations, connected components algorithm was used to segment the image. The object which has the maximum pixel area overlapping with the measurement plate was assumed as package of interest and others were eliminated. A sample image (a), background image (b) and morphological operations applied image (c) are given in Figure 5. The procedures applied for detecting the minimum bounding box on the resulting BW image are shown in the following topic.

The final BW image was divided into four zones with major and minor axes to reduce computational cost. Orientation angle (β), approximate width (d1) and approximate height (d2) of the object were obtained by connected components labeling algorithm. The line passing through the middle of the detected object and having the orientation angle β is accepted as the major axis. Similarly, the line which passes through the center point of the detected object and has an orientation angle β + 90 was assumed to be the minor axis. The parameter ρ given in equation (10) is calculated for both axes (ρ1, ρ2). Hough lines and major-minor axis representation for a sample object are given in Figure 6. Hough lines, which have θ values β ± δθ and ρ values ρ1 − 0.5d1 ± δρ and ρ1 + 0.5d1 ± δρ, were searched in the Hough space. Maximum scored lines above and below the major

axis (Line1, Line2) were assumed as the first and the second edge lines of the package surface. Similarly, Hough lines, which have θ values β + 90 ± δθ and ρ values ρ2 − 0.5d1 ± δρ and ρ2 + 0.5d1 ± δρ, were searched in the Hough space. Maximum scored lines above and below minor axis (Line3, Line4) were assumed as the third and the fourth edge lines of the package surface. Searching dominant lines without regard to edge information will likely cause false detections. Dividing the image into four regions and searching an independent Hough line along each edge guarantees that the selected four Hough lines cover the entire package surface. Figure 6 shows the four detected Hough lines for a sample image. Package corners were determined by finding the intersection points of the Hough lines. Real world coordinates of the corner points were calculated by equation (6). The flow chart of developed image processing algorithm is given in Figure 7. Because of the non-perfect package geometries and deformations on the package corners, the detected corner points can possibly fall on the measurement plate. Therefore,

the depth values of a corner point was calculated by averaging the depth values of the points on a circle centered on the corner point with a certain radius which overlap with the package. The average of the Euclidian distances between 1st and 2nd corner points and 3rd and 4th corner points in real world coordinates were calculated and assumed to be the width of the package. Similarly, the average of the Euclidian distances between 1st and 4th corner points and 2th and 3rd corner points in real world coordinates were calculated and assumed to be the length of the package. The package height was determined as the difference between the plate depth (which was found after the calibration step) and the average depth values of points on the package surface.

Results

The proposed algorithm was executed on a Windows based PC with Matlab software. To test the robustness of the proposed volume measurement algorithm, possible disturbances must be considered. Due to operating conditions,

the operator might partly obstruct Kinect’s field of view while handling the packages. In addition, there might be other packages around the measuring plate either handled or waiting to be handled. These packages might completely or partly fall in Kinect’s field of view. Given the fact that Kinect’s minimum range for depth measurement is 80 cm and the horizontal and vertical viewing angles are 43o and 57o, respectively, the sensor was placed at a height for measuring packages up to 60cm x 60cm x 60cm in size. Another scenario is when part of the package is not on the measuring plate. Various tests were performed under these circumstances. The results prove that the algorithm could determine the package dimensions (width, length, height) with maximum 1 cm error in all dimensions regardless of the disturbances.

Figure 8 illustrates a sample measurement performed while other packages are present on the floor around the measurement plate. The legs of the operator are also visible in the depth image as another disturbance effect. The measurement results for the given scenario are provided in Table 2 under package number 1. The measurement errors are 0.1 cm, 0.1 cm and 0.3 cm for the width, length and the height, respectively. If a non-rectangular object is placed on the measurement platform, then the image processing algorithm cannot detect four Hough lines. In such a case as illustrated in Figure 9, object dimensions are determined based on the minimum bounding rectangle. In a sample measurement depicted in Figure 10, the operator’s arm is partly obstructing Kinect’s field of view while handling the packages. The system successfully

Figure 13. Left to right (a), (b), (c). Table 3. Measurement results Package number

measured the package volume even though part of the package was missing in the depth image. The measurement results for this scenario are provided in Table 3 under package number 2. The measurement errors are 0.8 cm, 0.8 cm and 0.4 cm for the width, length and the height, respectively. Figure 11 presents a sample measurement taken while part of the package was outside the measuring plate. Package dimensions were also effectively measured for the given case with measurement errors of 0.1 cm, 0.3 cm and

Conclusions And Future WORK

In this study an image processing based volume measurement system is presented. The system consists of the Kinect sensor and a computer. Captured depth images were procesed by a developed image processing algorithm. Algorithms are designed on a suitable basis for software and hardware transition for embedded operation on a single board that reduces cost. The developed code can also be employed on a embeded computer such as Raspberry Pi. On the other hand, the proposed system could integrate with enterprise resource planning (ERP) systems via web service. The proposed system could determine only one box’s dimensions per mesurement. The code can also easily be adapted to measure multiple packages simultaneously.

The manufacturing of the Kinect sensors(both Kinect One, Kinect has expired. Therefore, an commercial depth cameras (Intel, Logitech etc) could be used to gather the depth images.

Acknowledgement

The prototype of this study is currently evaluated as a commercial product by Romeda Inc. and thus limited visual material is shared.

Data Availability Statement

The authors confirm that the data that supports the findings of this study are available within the article. Raw data that support the finding of this study are available from the corresponding author, upon reasonable request.

Conflict Of Interest

The author declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Ethics

There are no ethical issues with the publication of this manuscript.

References

1. Balogh Z, Magdin M, Monlar G. Motion detection reduction in segment-based classification of dam- and face recognition using raspberry pi, as a part aged building roofs in airborne Laser scanning data. of, the internet of things. Acta Polytech Hungarica Proceedings of the 4th GEOBIA. Rio de Janeiro, 2019;16:167–185. [CrossRef] Brazil, 2012.
2. Yamada T, Ito T, Ohya A. Detection of road surface damage using mobile robot equipped with 2D laser mobile robot localization and navigation. Int J Rob scanner. Proceedings of IEEE/SICE International Res 2013;2:1679–1694. Symposium on System Integration (SII). Kobe, [15] Yi L. Hand gesture recognition using Kinect. IEEE Japan 2013. [CrossRef] International Conference on Computer Sciences
3. Wei G, Shalei S, Bo Z, Shuo S, Faquan L, Xuewu C. and Applications. Beijing, China, 2012. [CrossRef] Multiwavelength canopy Lidar for remote sensing of [16] DeSouza GN, Kak AC. Vision for mobile robot vegetation, design and system performance. ISPRS J. navigation: A survey. IEEE Trans Pattern Anal Mach Photogramm. Remote Sens 2012;69:1–9. [CrossRef] Intell 2002;24:237–267. [CrossRef]
4. Zhang X, Morris J, Klett R. Volume measure- ment using a laser scanner. Communication and depth map refinement for transparent objects. ieee Information Technology Research (CITR) Technical international conference on pattern recognition. Report. Computer Science Department, The Stockholm, Sweden, 2014. [CrossRef] University of Auckland, 2005. [18] Essmaeel K, Gallo L, Damiani E, Pietro, G, Dipanda
5. Li J, Liu G, Liu Y. A dynamic volume measurement A. Temporal denoising of Kinect depth data. system with structured light vision. Proceedings IEEE International Conference on Signal Image 38 Sigma J Eng Nat Sci, Vol. 40, No. 1, pp. 27-44, March, 2022 Technology and Internet Based Systems. Naples, [22] Pagliari D, Pinto L. Calibration of kinect for Xbox Italy, 2012. [CrossRef] one and comparison between the two generations
6. Zug S, Penzlin F, Dietrich A, Tran T, Albert S. Are of microsoft sensors. Sensors 2015;15:27569–27589. laser scanners replaceable by Kinect sensors in [CrossRef] robotic applications. IEEE International Symposium [23] Richard OD, Peter EH. Use of the Hough trans- on Robotic and Sensors Environments (ROSE). formation to detect lines and curves in pictures. J Magdeburg, Germany, 2012. [CrossRef] Commun ACM 1972;15:11–15. [CrossRef]
7. Smisek J, Jancosek M, Pajdla T. 3D with Kinect. IEEE International Conference on Computer Vision mization of a DSP-based real-time lane detection Workshops (ICCV Workshops). Barcelona, Spain, algorithm on a mobile platform. Turk J Elect Eng 2011. [CrossRef] Comput Sci 2014;22:1484–1500. [CrossRef]

Share and Cite