ml5.js: Webcam Image Classification

 ml5.js: Webcam Image Classification - YouTube

https://www.youtube.com/watch?v=D9BoBSkLvFo

Transcript:

(00:00) [BELL RINGS] Hello. I'm back. I'm back. I'm here to do another video using the ml5 JavaScript library, a library-- a beginner-friendly library about machine learning. I'm going to use, again, the same pre-trained model called MobileNet that I talked probably way too much about in the previous video.

(00:18) But instead of attempting to classify this image as a puffin, which the model failed to it. It thinks it's an oyster catcher, which is not an unreasonable guess. I am going to attempt to classify images coming straight in in real time from the web camera. So this would actually not take us very long, frankly, because I've mostly written all the code for it.

(00:38) So the first thing I want to change in the code to turn it into video is I want to go and look at where I created the image. So I said, puffin = createImg. I had just this static image. Now what I want to do is I want to use the P5 function called Create Capture, and instead of pointing to-- I can't spell today.

(00:58) Instead of-- I can't spell every day-- instead of pointing to an image file, the argument that I give the Create Capture function is just the constant video. Video means I want to create a video capture. It's going to default to the default camera that's connected to the computer. So just to be sure about this, I'm going to comment out puffin.hide.

(01:18) I'm going to change this. I'm going to get rid of this, which has to do with drawing the image. I'm going to get rid of the predict, comment that out, and I'm going to change this variable to video. So the only thing that I should see right now is I should see the video appearing on the page.

(01:39) So let's run this. Oh, first, it's asking me permission. It wants to use my camera. I'm going to say Allow, and then there it is. There's me. Hi, everybody! Look. I'm there and there. I'm here. I'm here. I'm double double double [? band. ?] OK. So there it is. Now here's the thing.

(01:57) I could just use the video element there, and it's fine. But I just have this thing that I obsess over and I feel is important. What I'm going to do is I'm going to hide the video. This is-- this has nothing to do with the machine learning image classification aspect, but I feel like you might have a chance to express yourself to be more creative, instead of just using the default dom element.

(02:18) If you actually draw the video to the canvas itself, this might allow you just to do things like draw on top of it and other type of stuff. So there we go. Now I have the video in the actual canvas. Now I need to figure out now when to call the predict function, so in theory, I should just call the predict function right here with the video.

(02:36) But here's one of the nice things about ml5. Since ml5 knows about things like, hey, you might want to do this with a continuously running video, I can actually, when I create the image class, I'm going to-- I'm going to take this out of here, so I'm going to take video out of the .predict function.

(02:55) And actually, the way to do this is when I create this image classifier, I can actually say, I want the MobileNet model to act continuously on the video. So I want to hook those things up together, so I can actually insert the video, the P5 video element right there into the image classifier function. And then I can just call predict without any particular-- I mean I could always say, predict on this specific image, but now the video is just linked to that image classifier, and I could just call predict.

(03:27) So now if I run this, I'm going to refresh it.  There it goes. And we should see, there I am. I am a poncho or a cardigan, or I'm often considered chain mail. Once again, we're seeing the failings of the image-- the MobileNet model in this particular [? way. ?] Now the thing is it just did it once.

(03:52) It just called predict the moment the model was ready. But guess what. What if-- how do I call it continuously? Well, you know what? Once it calls predict, and it got the results, why not just ask it to do it again? So if there was a mistake, maybe I want to not move on. But if it works, I'm going to rid of the createP stuff here and get rid of the probability just for right now.

(04:13) I can just do this. So what I'm doing here is this is like-- this is essentially like a loop, another loop that's happening. When the model is ready, that's when I can start predicting. Then once I've got the results, I've got the results, draw them, and then show me the next one. Oh, but guess what.

(04:32) Hold on. That's weird. It didn't actually show up. So this is actually a big problem, so I'm going to run this again. Well, hopefully, this will work. It's going to-- so weirdly, this is working. I don't like that this is working. Let's see. This is something it generally tends to get correctly-- kneepad.

(04:51) Really? Water bottle. OK. So-- but this worked by accident, the fact that I'm seeing the text there. I probably shouldn't draw in multiple places. What would make more sense for me to do is make a global variable called Label, and then just-- oh, and then have this and have that set initially to an empty string.

(05:15) And then this drawing of the label should actually be right here in draw itself. So let's try this again. Let's make it a little smaller, and let's try one more time. And here we go. Let's see what types of things it can classify.  And you know what? I should-- hm, this is-- you know, so I'd have to be more thoughtful about the design here.

(05:43) So let's do this. Let's make the canvas a little bit taller, and, of course, I could just write it into an HTML element, but just for simplicity, let's see here. Let's see if this does the trick.  I just drew-- you know, that's a little bit better. I meant to-- I should make the canvas a bit taller.

(06:17) I don't know why I was so stingy there.  So what I'm trying to do is just draw a little area. There we go. OK, so now-- and I probably should take out this console.log. These console.logs can really slow things down. And I'm going to do this again. Last time-- here we go. All right, poncho-- oh, let's see.

(06:45) Spatula-- that's pretty good. Spatula. OK, does it recognize a bell? Magnetic compass. What about a train whistle? A matchstick. That's pretty good. Does it recognize a book? Broccoli. What about a marker? I can do this-- I could do this all day long. This video is going to be like 17 hours long. Pill bottle.

(07:18) Kneepad. Water tower. I think that green background is kind of likely confusing it. What else do I got in this room?  Accordion. Piano accordion. So you can see, as long as it's something that's in there, if it's been trained on that particular-- if I just happen to have a chihuahua with me, I probably could get it to recognize that chihuahua.

(07:43) So this is a fun thing that you can now play with. You can make your own project about. Sort of ske-- there's a-- Google made a project like this called an emoji scavenger hunt, where you had to find certain things, get the image classifier to recognize it. So there's all sorts-- of course, it used some custom training there to make that work for particular objects.

(08:04) But so I think there's a lot of possibilities of things you could explore here, but now you can really have your hands in there, play with the code, visualize things, draw stuff, and really experiment with what is the MobileNet model? What works well? What doesn't work well? And the net-- one of the things that I really want to show you how to do, which I will do in a future video in this series, is use the MobileNet model only as a starting point.

(08:29) So the MobileNet model is a starting point, but I can then retrain it with some custom images and some new labels. So by golly, I could get that MobileNet model to recognize puffins if I wanted it to. So thanks for tuning in. Is that a thing you say? Do you tune into videos? I don't know. It's a live stream.

(08:48) If you're watching this now, thanks for watching. And stay tuned. I'm just going to keep going through it and show you more features of the ml5 library. What should I do next?  I don't know. I think I might do--  I'm not sure yet. Stay tuned. Come back. Goodbye. Thanks for watching. Just look for the description.

(09:14) It'll say Next Video. [WHISTLES] [MUSIC PLAYING]