7 – YouTube Data API – Paging & MaxResults And How to Cross the 500 Results Limit

This is article 7 of the YouTube API With PHP series.

THE PROBLEM WITH MAXRESULTS

Everyone who starts to work with the YouTube Data API runs into these problems very soon and then spends a few hours or even a few days trying to figure out  how to deal with these issues.

Problem 1: 99% of the API calls which return multiple results , will not allow you to retrieve more than 500 items even if the API says there are more.  For eg.if you want to get all the videos in a channel, the API will report that there are 4,000 videos but it will not let you retrieve the details of more than the first 500 (depending on the sorting order). There are ways around this, which we will look into, in further sections, but by and large , you have to keep this limit in mind.

Problem 2: Most API calls which return multiple results, have a field called maxResults which tells you how many results are there in total for this API call. This value keeps changing between multiple calls and cannot be used for any reliable calculations. Eg. if you execute a search API , the first call will say 200 results, the call for the next page will say 4500 results and the third page may show a different value again.

It is good if you know about these two issues early on, so that it saves you a lot of frustration later. Both these problems can be overcome , as you will find in later sections.

 

HANDLING PAGING

One more basic concept to deal with before we look into the actual API entities is how to page through the results. Some API calls like fetching a video only return a single data entity. Others like search or getting all the videos in a channel or all Activities of a user get multiple items of data.

When an API call returns multiple results, then due to practical reasons, the API won’t let you retrieve more than 50 items at a time. The maxResults parameter lets you set the max results to be returned for each call. It can be anything from 0 to 50. If it is not set, then the default maxResults are 5.

So if a call has 200 results and you have not set maxResults, you will have to page through it (200 / 5 = 40) times. If you set maxResults to 50, you will have (200/50=4) pages of data to deal with.

As you will see in the later sections, all API results will return two fields nextPageToken and prevPageToken, if the number of total results exceed the number of fetched results. The tokens are in the form of unique strings like CDIQAA , CKYEEAA etc.

So in order to iterate through all the results, you get the value of nextPageToken in the first API call and pass it as a parameter (pageToken) in the subsequent calls until the nextPageToken returns a null. When it becomes null, it means no more data is available.

Similar is the case with prevPageToken. This value is only available when you get the second page of results. If you need to fetch the previous page of results , pass the prevPageToken within the pageToken parameter in the API call. When you reach the first page, prevPageToken will become null.

Here is some sample code which shows how the iteration is done. This logic is more or less the same for whatever API entity is being called. Ignore the actual call details and the JSON handling. Just focus on how the nextPageToken is handled.

<?php
    error_reporting(E_ALL ^ E_NOTICE ^ E_WARNING ^ E_DEPRECATED);
    set_time_limit(60 * 3);
    $g_YouTubeDataAPIKey = "****";
    $channelId = "UCddiUEpeqJcYeBxX1IVBKvQ";
    
   	 // make api request
   	 $url = "https://www.googleapis.com/YouTube/v3/activities?part=snippet,contentDetails,id&channelId=" .
   	 $channelId. "&maxResults=50&key=" . $g_YouTubeDataAPIKey;
   	 
   	 $curl = curl_init();
   	 curl_setopt_array($curl, array(
   				 CURLOPT_RETURNTRANSFER => 1,
   				 CURLOPT_URL => $url,
   				 CURLOPT_USERAGENT => 'YouTube API Tester',
   				 CURLOPT_SSL_VERIFYPEER => 1,
   				 CURLOPT_SSL_VERIFYHOST=> 0,
   				 CURLOPT_CAINFO => "../cert/cacert.pem",
   				 CURLOPT_CAPATH => "../cert/cacert.pem",
   				 CURLOPT_FOLLOWLOCATION => TRUE
   				 ));
   	 $resp = curl_exec($curl);

   	 curl_close($curl);

   	 
   	 if ($resp) {
   		 echo("Found data. <br>");
   		 $pages = 1;
   		 $json = json_decode($resp);
   		 if ($json) {
   			 echo("JSON decoded<br>");
   			 $nextPageToken = $json->nextPageToken;
   			 $total = $json->pageInfo->totalResults;

   			 $items = $json->items;
   		 }
   		 else
   			 exit("Error. could not parse JSON." . json_last_error_msg());

   	 
   		 foreach($items as $item) {
   			 // process each item in the list   						 
   		 }


   		 // second level search using nextpage token
   		 while ($nextPageToken != null) {
   			 echo("Fetching page " . (++$page) . " using pagetoken=" . $nextPageToken . "<br>");
   			 $url = "https://www.googleapis.com/YouTube/v3/search?part=snippet&channelId=" .
   				 $channelId. "&maxResults=50&order=date&type=video&" .
   				 "&pageToken=" . $nextPageToken . "&key=" . $g_YouTubeDataAPIKey;
   			 
   			 $nextPageToken = null; // clear current value;
   			 $curl = curl_init();
   			 curl_setopt_array($curl, array(
   						 CURLOPT_RETURNTRANSFER => 1,
   						 CURLOPT_URL => $url,
   						 CURLOPT_USERAGENT => 'Codular Sample cURL Request',
   						 CURLOPT_SSL_VERIFYPEER => 1,
   						 CURLOPT_SSL_VERIFYHOST=> 0,
   						 CURLOPT_CAINFO => $g_docRoot . "../cert/cacert.pem",
   						 CURLOPT_CAPATH => $g_docRoot . "../cert/cacert.pem",
   						 CURLOPT_FOLLOWLOCATION => TRUE
   						 ));
   			 $resp = curl_exec($curl);
   			 if ($resp) {
   				 $json = json_decode($resp);
   				 if ($json) {
   					 $nextPageToken = $json->nextPageToken;
   					 $total = $json->pageInfo->totalResults;

   					 $items = $json->items;
   					 
   				 }

   		 
   				 foreach($items as $item) {
   						 // process each item
   				 } // foreach
   				 
   			 } // if $resp
   			 
   			 // sometimes pagetoken is filled but no items are fetched , in such a case stop the code
   			 if (count($items) == 0)
   				 $nextPageToken = null;
   				 
   		 } // while $nextPageToken

   		 echo("Finished");
   		 

   	 } // if $resp
   	 else {
   	 }
    
?>

 

Here is the output:

Found data.
JSON decoded
Fetching page 1 using pagetoken=CDIQAA
Fetching page 2 using pagetoken=CGQQAA
Fetching page 3 using pagetoken=CJYBEAA
Fetching page 4 using pagetoken=CMgBEAA
Fetching page 5 using pagetoken=CPoBEAA
Fetching page 6 using pagetoken=CKwCEAA
Fetching page 7 using pagetoken=CN4CEAA
Fetching page 8 using pagetoken=CJADEAA
Fetching page 9 using pagetoken=CMIDEAA
Fetching page 10 using pagetoken=CPQDEAA
Fetching page 11 using pagetoken=CKYEEAA
Finished

 

CROSSING THE 500 RESULTS LIMIT

As shown above, one of the biggest practical limitations while retrieving data is that you hit the 500 results limit of the YouTube API, if you working with a lot of data. Officially the limit still holds, but there is a way to go beyond that limit.

The key to this is the publishedBefore  and the order parameter passed in the API call. When you are doing a search or perhaps retrieving a list of videos from a channel, specify that the results be sorted as per date. Then we use the publishedBefore or publishedAfter parameter to get more data once we cross 500 results.

Here is how the logic flows:

1. Make the first call by specifying order=date and maxResults=50
2. As long as there is a nextPageToken available, keep making successive calls
3. Get the date of the last result (this will have the earliest date) and make a new call with order=date and publishedBefore=the date of the last result obtained
4. Ignore any nextPagetokens obtained.
5. Out of the results obtained, get the date of the earliest result and then repeat the call with order=date and publishedBefore=earliest date
6. Repeat step 5 till you reach either 1 or zero results.

The above logic is not foolproof and has not been properly tested for edge cases and missing data. But the code based on this has been used to fetch thousands of videos from channels.

Sample code which uses this algorithm is given below:

 

<?php
    error_reporting(E_ALL ^ E_NOTICE ^ E_WARNING ^ E_DEPRECATED);
    set_time_limit(60 * 3);
    
    $g_youtubeDataAPIKey = "**";
        
   
    $channelId = "***";
    
        // make api request
        $url = "https://www.googleapis.com/youtube/v3/search?part=snippet&channelId=" .
            $channelId. "&maxResults=50&order=date&type=video&key=" . $g_youtubeDataAPIKey;

        $curl = curl_init();
        curl_setopt_array($curl, array(
                    CURLOPT_RETURNTRANSFER => 1,
                    CURLOPT_URL => $url,
                    CURLOPT_USERAGENT => 'Codular Sample cURL Request',
                    CURLOPT_SSL_VERIFYPEER => 1,
                    CURLOPT_SSL_VERIFYHOST=> 0,
                    CURLOPT_CAINFO => $g_docRoot . "../cert/cacert.pem",
                    CURLOPT_CAPATH => $g_docRoot . "../cert/cacert.pem",
                    CURLOPT_FOLLOWLOCATION => TRUE
                    ));
        $resp = curl_exec($curl);

        curl_close($curl);

    
    if ($resp) {
    
        // first level search
            $json = json_decode($resp);
            if ($json) {
                $nextPageToken = $json->nextPageToken;
                $total = $json->pageInfo->totalResults;

                $items = $json->items;
                
            }

            echo("nextpage=" . $nextPageToken . "<br>total=" . $total . "<br>items=" . count($items));
            
            foreach($items as $item) {
                $videoId = $item->id->videoId;
                $videoTitle = $item->snippet->title;
                $videoDesc = $item->snippet->description;
                $thumbnail= $item->snippet->thumbnails->high->url;
                $rawDate = $item->snippet->publishedAt;

                echo("<hr>" . $videoId . ", " . $videoTitle . " date :" . $rawDate);
                    
            }

            // second level search using nextpage token
            echo("<br><h2>Second phase</h2><br>");
            while ($nextPageToken != null) {
                $url = "https://www.googleapis.com/youtube/v3/search?part=snippet&channelId=" .
                    $channelId. "&maxResults=50&order=date&type=video&" .
                    "&pageToken=" . $nextPageToken . "&key=" . $g_youtubeDataAPIKey;
                
                $curl = curl_init();
                curl_setopt_array($curl, array(
                            CURLOPT_RETURNTRANSFER => 1,
                            CURLOPT_URL => $url,
                            CURLOPT_USERAGENT => 'Codular Sample cURL Request',
                            CURLOPT_SSL_VERIFYPEER => 1,
                            CURLOPT_SSL_VERIFYHOST=> 0,
                            CURLOPT_CAINFO => $g_docRoot . "../cert/cacert.pem",
                            CURLOPT_CAPATH => $g_docRoot . "../cert/cacert.pem",
                            CURLOPT_FOLLOWLOCATION => TRUE
                            ));
                $resp = curl_exec($curl);

                if ($resp) {
                    $json = json_decode($resp);
                    if ($json) {
                        $nextPageToken = $json->nextPageToken;
                        $total = $json->pageInfo->totalResults;

                        $items = $json->items;
                    }

                    echo("nextpage=" . $nextPageToken . "<br>total=" . $total .
                            "<br>items=" . count($items));
            
                    foreach($items as $item) {
                        $videoId = $item->id->videoId;
                        $videoTitle = $item->snippet->title;
                        $videoDesc = $item->snippet->description;
                        $thumbnail= $item->snippet->thumbnails->high->url;
                        $rawDate = $item->snippet->publishedAt;
                        
                        echo("<hr>" . $videoId . ", " . $videoTitle . " date :" . $rawDate);


                    } // foreach
                    
                } // if $resp
                
                // sometimes pagetoken is filled but no items are fetched , in such a case stop the code
                if (count($items) == 0)
                    $nextPageToken = null;
                    
            } // while $nextPageToken


            // rawData contains the last added video , so we for further back using it
                
            
            // third level search using date if there were any search results
            echo("<br><h2>Third phase</h2><br>");
            echo("<br>RawDate of latest added video=" . $rawDate . "<br>");
            if ($rawDate != null) {
            
              $doSearch = true;    
              $nextPageToken = null;
              
              $firstRawDate = $rawDate;  // remember where this loop starts from

            
              
              while ($doSearch) {
               
                if ($nextPageToken != null)
                    $url = "https://www.googleapis.com/youtube/v3/search?part=snippet&channelId=" .
                        $channelId. "&maxResults=50&order=date&type=video&" .
                        "&pageToken=" . $nextPageToken . "&key=" . $g_youtubeDataAPIKey;
                else
                    $url = "https://www.googleapis.com/youtube/v3/search?part=snippet&channelId=" .
                        $channelId. "&maxResults=50&order=date&type=video&" .
                        "&publishedBefore=" . $rawDate . "&key=" . $g_youtubeDataAPIKey;
                echo($url . "<br>");
                $curl = curl_init();
                curl_setopt_array($curl, array(
                            CURLOPT_RETURNTRANSFER => 1,
                            CURLOPT_URL => $url,
                            CURLOPT_USERAGENT => 'Codular Sample cURL Request',
                            CURLOPT_SSL_VERIFYPEER => 1,
                            CURLOPT_SSL_VERIFYHOST=> 0,
                            CURLOPT_CAINFO => $g_docRoot . "../cert/cacert.pem",
                            CURLOPT_CAPATH => $g_docRoot . "../cert/cacert.pem",
                            CURLOPT_FOLLOWLOCATION => TRUE
                            ));
                $resp = curl_exec($curl);
                if ($resp) {
                    $json = json_decode($resp);
                    if ($json) {
                        $nextPageToken = $json->nextPageToken;
                        $total = $json->pageInfo->totalResults;

                        $items = $json->items;
                    }


                    echo("nextpage=" . $nextPageToken . "<br>total=" . $total .
                            "<br>items=" . count($items) . "<br>");
                    
                    // we use an array to store the dates because it is not necessary that the API will
                    // observe date sorting within the results.

                    $arrDates = array();

                    
                    foreach($items as $item) {
                        $videoId = $item->id->videoId;
                        $videoTitle = $item->snippet->title;
                        $videoDesc = $item->snippet->description;
                        $thumbnail= $item->snippet->thumbnails->high->url;
                        $rawDate = $item->snippet->publishedAt;

                        echo("<hr>" . $videoId . ", " . $videoTitle . " date:" . $rawDate . "<br>");

                
                        $arrDates [] = $rawDate; // store date into our array
                    } // foreach
                    
                } // if $resp

                $nextPageToken = null; // do not go to the next page
                
                // sort array to get earliest raw data
                if ($arrDates && count($arrDates) > 0) {
                    sort($arrDates);
                    $rawDate = $arrDates[0];
                }

                
                if (($nextPageToken == null || $nextPageToken == "") &&
                    (count($items) == 0) && $rawDate == null)
                        $doSearch = false;
                        
            
                if ($nextPageToken == null && $rawDate == $firstRawDate)
                    $doSearch = false;
                else if ($nextPageToken == null && $rawDate != $firstRawDate) {

                    echo("<br>Raw Date=" . $rawDate . "<br>");
                }

                // special end case - if there is only one video in the search results
                // then we have reached the end of search
                if (count($items) <= 1)
                    $doSearch = false;

                    

              } // while ($doSearch)
            } // if (!rawDate != null)
            
    } // if $resp || incremental

    
?>

From the next section we are going to look into each Resource exposed in the YouTube Data API.

2023 DEC 25 ADDITION

It has been my experience and others have also pointed out the fact that the code to fetch videos from a channel can sometimes skip videos for no logical reason. If you manually compare videos visible in Youtube for that channel and compare the output from the API, sometimes videos will be missing. After more testing, I found that if videos are created in different playlists then they do not show up in the video listing of the channel. Given that circumstance, the above code will not fetch all the videos, even if we cross the 500 video limit.

The only alternative solution is to

• Fetch all the Playlists in the channel
• For each Playlist, get all the playlistItems which will contain the videos

To use the above logic, you can refer the sample code in the following blog posts:

32 – YouTube Data API – Get All Playlists For A Channel

33 – YouTube Data API – Get All PlaylistItems For A Playlist