Friday Presentation – Irumva

Using Convolutional Neural Networks and Pose Estimation to Monitor and Classify Agricultural Machinery Operators’ Safety Behaviors in Real-Time

Terence Irumva, Graduate Student, University of Nebraska-Lincoln; Herve Mwunguzi, Dr. Santosh K. Pitla, Dr. Bethany Lowndes, Dr. Aaron Yoder, and Dr. Ka-Chun Siu

I am from Rwanda, 22 years old, and a graduate student and a research assistant at UNL in the department of Biological Systems Engineering. I have a two years of experience working on agricultural robotics projects and agricultural systems and machinery automation projects.

Learning objective:
Participants will be able to sufficiently describe how machine learning can be used in monitoring agricultural machinery operators’ safety behaviors.

Discuss this presentation with the authors on Thursday, November 19 from noon – 12:30 on the Zoom Live-stream.


Nineteen percent of injuries to agricultural producers are related to tractors or large machinery (CS-CASH Injury Surveillance Surveys 2011-2016) and yet only limited studies are found that address tools and methods for monitoring safety behaviors of agricultural machinery operators in real-time. Our research aims at developing a monitoring system called Ag-OMS, which uses cameras and machine learning techniques – Convolutional Neural Networks (CNNs), to enable real-time assessment of agricultural machinery operators’ safety practices while entering and exiting a tractor cab. By using pose (position) estimation through a human skeleton detecting model called openpose and CNNs, we developed a machine learning model that deduces and labels a machinery operator’s safety practices by analyzing a livestreaming video from a webcam. The model can be trained to detect and differentiate numerous safety behaviors in real-time. By accurately identifying a person’s posture using openpose, the model analyzes the current frame in the livestreaming video, and after labeling the detected safety behaviors, it classifies the operator actions as safe or unsafe. Preliminary data collected by the Ag-OMS corresponding to operator ingress and egress behavior in and out of a tractor cab will be presented. A methodology to categorize behaviors into high, medium, and low-risk behaviors based on ergonomic safety conditions will be discussed.

Click on Thumbnail to view: Resources: