How to detect hands, eyes, or other things by the live camera in the flutter

We were developing one app that we need to implement hand detection and eye detection features from a live camera feed. So we have done a lot of R&D on object (hand and eye) detection.

We found some solutions for it but all are different for hand and eye detection. we need only one solution which can acquire both hand and eye detection in a flutter. So after some more R&D, we came up with one solution which can fulfil our requirements.

We have used the image labelling functionality of the goole_ml_vision pub package to detect hands and eyes.

Let me tell you how we used it for Hand and Eyes detection.

Add pub dependency

google_ml_vision: ^0.0.7
camera: 0.9.5

Init Image labeller and camera

final ImageLabeler _imageLabeler = GoogleVision.instance.imageLabeler();
CameraController? camera;
bool palmFound = false;
int palmFoundCount = 0;

Detect Hand or Eye from a live camera feed

Future<void> _initializeCamera() async {
  final CameraDescription description =
      await getCamera(_direction);

  camera = CameraController(
    description,
    ResolutionPreset.high,
    enableAudio: false,
  );
  await camera!.initialize();
  update();
  await camera!.startImageStream((CameraImage image) {
    if (!palmFound.value) {
     detect(
        image: image,
        detectInImage: _getDetectionMethod(),
        imageRotation: description.sensorOrientation,
      ).then(
        (dynamic results) {
          for (var element in results) {
            //// you can use 'eyelash' for eye detection.
            if (element.text!.toLowerCase() == 'hand' &&
                element.confidence! > 0.75) {
              palmFoundCount = palmFoundCount + 1;
              if (palmFoundCount == 50) {
                palmFound = true;
                if (palmFound) {
                  // do your code if hand detect.
               }
              }
            }
          }
        },
      );
    }
  });
}
static Future<dynamic> detect({
  required CameraImage image,
  required Future<dynamic> Function(GoogleVisionImage image) detectInImage,
  required int imageRotation,
}) async {
  return detectInImage(
    GoogleVisionImage.fromBytes(
      _concatenatePlanes(image.planes),
      _buildMetaData(image, _rotationIntToImageRotation(imageRotation)),
    ),
  );
}
static Uint8List _concatenatePlanes(List<Plane> planes) {
  final WriteBuffer allBytes = WriteBuffer();
  for (var plane in planes) {
    allBytes.putUint8List(plane.bytes);
  }
  return allBytes.done().buffer.asUint8List();
}

static GoogleVisionImageMetadata _buildMetaData(
  CameraImage image,
  ImageRotation rotation,
) {
  return GoogleVisionImageMetadata(
    rawFormat: image.format.raw,
    size: Size(image.width.toDouble(), image.height.toDouble()),
    rotation: rotation,
    planeData: image.planes.map(
      (Plane plane) {
        return GoogleVisionImagePlaneMetadata(
          bytesPerRow: plane.bytesPerRow,
          height: plane.height,
          width: plane.width,
        );
      },
    ).toList(),
  );
}

static ImageRotation _rotationIntToImageRotation(int rotation) {
  switch (rotation) {
    case 0:
      return ImageRotation.rotation0;
    case 90:
      return ImageRotation.rotation90;
    case 180:
      return ImageRotation.rotation180;
    default:
      assert(rotation == 270);
      return ImageRotation.rotation270;
  }
}
static Future<CameraDescription> getCamera(CameraLensDirection dir) async {
  return availableCameras().then(
    (List<CameraDescription> cameras) => cameras.firstWhere(
      (CameraDescription camera) => camera.lensDirection == dir,
    ),
  );
}
Future<dynamic> Function(GoogleVisionImage image) _getDetectionMethod() {
  return _imageLabeler.processImage;
}

So what we have done here
First, init camera and image labeller from goole_ml_vision package.
then start startImageStream on the camera it will give you continuously live images from the camera feed. Then process that detect method It will provide you with different types of labels with the confidence of things which are available on that image.
In this way, we can simply detect the hand and eye from the live camera feed.

You can also find or detect labels from picked or capture images as explained below.

Detect Hand or Eye from the image file

File image;
checkEyelashFoundOrNotInImage() async {
  final GoogleVisionImage visionImage = GoogleVisionImage.fromFile(image!);
  List<ImageLabel> results;
  results = await _imageLabeler.processImage(visionImage);
  for (var element in results) {
    if (kDebugMode) {
      print('${element.text} - ${element.confidence}');
    }
    if (element.text!.toLowerCase() == 'hand' &&
        element.confidence! > 0.75) {
       palmFound = true;
      break;
    }
  }
}

You also can detect a face, barcode and text recognised by this package.

final BarcodeDetector barcodeDetector =
    GoogleVision.instance.barcodeDetector();
final FaceDetector faceDetector = GoogleVision.instance.faceDetector();
final ImageLabeler labeler = GoogleVision.instance.imageLabeler();
final TextRecognizer textRecognizer = GoogleVision.instance.textRecognizer();

For more information check out this awesome package

google_ml_vision | Flutter Package

You can find the demo project on this Github repo.

GitHub - Inficial-Infotech/object-detection-demo: A object detection demo

To detect face with landmarks, the hand with landmarks check out this Github repo which uses tensor flow models.

GitHub - JaeHeee/FlutterWithMediaPipe: Flutter with MediaPipe ML models

Detect face on real-time

Detect face landmark on real-time

Detect hand on real-time

Estimate body pose on real-time

Visualize inference results

Happy Hunting!

Follow InFicial To get such articles: www.linkedin.com/company/inficial-infotech