Privatized/Deleted Videos now automatically removed!

Hi Boys and Girls and welcome back to another map update! Since the release of the Video Finder, my focus has been to make the available data of the map as clean and usable as possible. I have now figured out something very important: How to automatically detect videos that have been privatized/deleted.

There is no point in having a video stick around on the map if you can’t even watch it. But this wasn’t as straightforward as you might think as the YouTube API didn’t quite let me do what I wanted. There are two things to consider here:

This video has either been privatized or deleted, it thus provides no information for the map and should be removed from the video finder.

I didn’t want to remove unlisted videos because they were still accessible. The YouTube API however didn’t differentiate between Unlisted/Privatized/Deleted.
In Database Science, it is much easier to detect if something is there rather than to detect if something is not there. Computationally, checking for deleted items is really expensive.

As such, a new solution needed to be found, and thankfully, I did and it’s hilarious!

No, it truly is hilarious, because when checking for videos that have been deleted or privatized, I noticed that the thumbnails for those videos returned a default image along with a 404 not found error. Basically, the same error that appears when you access a website that doesn’t exist. See this example. Your browser should still display the default image, but behind the scenes the browser is also informed of that 404 not found status code.

So now, what I do is to periodically check my database for deleted videos by using the following Python script to check if the thumbnail still exists. If the thumbnail exists, I know that the video is either still public or unlisted. An easy solution! If you can read code you may also see that I try to verify if the video has been deleted by waiting another 30 seconds and doing the request again. This is of course to try to avoid a situation, where the server was just briefly down or couldn’t serve the request. It’s rare, but it happens. Always double-check your work!

At this current time, a total of 905 videos have already been privatized/deleted! This just shows how important this feature is to keep the data clean.

Anyways, Cheerio!

#Python Code to check for deleted/privatized videos
yt_video_id="ab3CBS3AayE"
thumbnail_url = f"https://img.youtube.com/vi/{yt_video_id}/hqdefault.jpg"

response = requests.get(url=thumbnail_url)
is_disabled = False
if response.status_code == 404:
    time.sleep(30)
    response2 = requests.get(url=thumbnail_url)
    if response2.status_code == 404:
        is_disabled = True
elif response.status_code == 200:
    is_disabled = False

if is_disabled:
    set_video_disabled(yt_video_id)

You must be logged in to post a comment.