AI-Powered Excavator Detection and Analysis Using YOLO and Google Gemini

Project Overview:

This project implements an AI-powered video processing system to detect, track, and analyze excavators using YOLO object detection and Google Gemini AI. The system processes a video file, detects excavators in real-time, tracks them, extracts relevant details (such as status, color, and company name), and saves the analyzed data for further use.

The project is designed to automate monitoring of excavation activities in construction or mining sites, providing structured reports on machine usage and ownership.

Key Features:

1. Excavator Detection & Tracking

• YOLO-based detection: The system uses a pre-trained YOLO model (best.pt) to detect excavators in a video file.
• Object tracking: Assigns unique track IDs to each detected excavator to prevent duplicate processing.

2. Image Processing & Analysis

• Image cropping: Extracts individual excavator images from detected frames.

3. Google Gemini AI Integration

• Sends the excavator image to Google Gemini AI along with specific queries:
  • Working Status: Identifies if the excavator is actively working or in standby mode.
  • Color Recognition: Determines the primary color of the excavator.
  • Company Name Recognition: Extracts company branding or logo details.
• The AI returns the analysis in a structured table format.

4. Data Logging & Report Generation

• Saves analysis results in a structured text report (gemini_response_<date>_report.txt).
• Stores cropped excavator images for future reference and verification.

5. Multithreading for Performance Optimization

• Uses background threads to process images asynchronously, improving real-time performance.

6. Interactive Visualization

• OpenCV GUI: Displays real-time video with bounding boxes, track IDs, and labels for detected excavators.

Project Workflow:

1. Video Loading

   • The system reads a video file (vid.mp4) using OpenCV.

2. YOLO-based Detection & Tracking

   • Each frame is passed through the YOLO model to detect excavators.
   • The model assigns bounding boxes and track IDs.

3. Cropped Image Processing

   • If a track ID is new, the excavator image is cropped and saved.

4. Google Gemini Analysis

   • The cropped image is sent to Google Gemini AI for further analysis.
   • AI response is formatted in a structured report.

5. Data Storage & Logging

   • Saves AI analysis and processed images in a structured format.

6. Real-Time Visualization

   • The processed video frame is displayed using OpenCV.
   • The system runs until all frames are processed or ‘q’ is pressed.

Project Dependencies:

• YOLO (ultralytics) – For object detection and tracking.
• OpenCV (cv2) – For image processing and visualization.
• cvzone – For improved text rendering on video frames.
• Google Gemini API (langchain_google_genai) – For AI-based analysis of excavator images.
• Threading – For efficient background processing.
• Base64 Encoding – For sending images to Google Gemini AI.

Use Cases & Applications:

🔹 Construction Site Monitoring – Track excavator activity and generate usage reports.
🔹 Automated Fleet Management – Identify company-owned machines from videos.
🔹 Mining & Heavy Equipment Analysis – Monitor machine operations and optimize resource usage.
🔹 Surveillance & Compliance – Verify if excavators are active or idle in restricted zones.

Next Steps & Improvements:

• Dashboard Integration – Display results in a web-based analytics dashboard.
• Better Image Preprocessing – Improve image quality before sending to AI.
• Fine-Tune YOLO Model – Enhance model accuracy by training on custom excavator datasets.

Conclusion:

This project showcases an AI-powered excavator detection system that integrates YOLO object detection, image processing, and Google Gemini AI to provide structured analysis on excavator activity. The implementation enables automated monitoring, real-time tracking, and data-driven decision-making in industrial applications.