BigQuery: RowIterator is not an iterator, it is an Iterable

[evanj]

RowIterator is not actually an Iterator, it is an Iterable. Either it should be an Iterator, or it should be renamed? At this point, but maybe api_core.Iterator could implement both Iterable and Iterator? Or maybe the documentation could specify this? This is very very minor, since most use will be in a for loop, so maybe this should be closed as "won't fix", but it did confuse me when trying to figure out how to get the next_page_token as early as possible. I need to call iter(row_it) because `RowIterator is not actually an iterator ... it is an iterable!

Example code

from google.cloud import bigquery
bq_client = bigquery.Client()
table = bq_client.dataset('tmp').table('supply_house_ramp9_NTM_split2')
row_iterator = bq_client.list_rows(table, max_results=10)
print('row_iterator:', row_iterator)
# fix: next(iter(row_iterator))
next(row_iterator)

Output:

row_iterator: <google.cloud.bigquery.table.RowIterator object at 0x10a44d090>
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: 'RowIterator' object is not an iterator

Expected output:

Return the first row of the table, since it is called an iterator, you should be able to pass it to next()

Environment details

OS: Any
Python version: Python 3.7.4
google-cloud-bigquery version: 1.20.0

[HemangChothani]

@evanj Utilize iterator properties returned with row data:

iterator = client.list_rows(table)
page = next(iterator.pages)
rows = list(page)
total = iterator.total_rows
token = iterator.next_page_token

hope this information is helpful for you to get next_page_token.

[evanj]

Yes thanks. That is basically equivalent to what I had done: next(iter(iterator)). The confusing part here is that Python has a precise definition of an "iterator": It is an object that supports next. RowIterator does not support that, hence Python produces the confusing error message: RowIterator object is not an iterator. I think this could be fixed, since RowIterator only supports iteration once, but this almost certainly doesn't matter and is probably not worth bothering. This also probably affects most of the google.cloud APIs, since this behaviour comes from google.api_core.Iterator.

Python's definition of iterator: https://docs.python.org/3/library/stdtypes.html#typeiter

[HemangChothani]

@evanj Yes thanks for the quick response, know about an "iterator", The code sample is to get the next_page_token as early as possible, class name RowIterator is a bit confusing, not sure about change or rename the class name.

[tseaver]

@evanj There is no chance that we will rename the google.api_core.iterator.Iterator class (or the google.cloud.bigquery.table.RowIterator subclass) at this point: such a rename would be a breaking change.

[Senhaji-Rhazi-Hamza]

Here is a work around to transform the behaviour as a normal iterator :

`

client = bigquery.Client()
class BqRowIterator:
  client = client
  def __init__(self, project_id, dataset_id, table_id, batch_size=10000):
    self.project_id = project_id
    self.dataset_id = dataset_id
    self.table_id = table_id
    self.batch_size = batch_size
    self.initialized = False
    self.nth_batch = 0

def __iter__(self):
    return  self

def __next__(self):
    if not self.initialized:
        self.initialize()
        return next(self.row_iterator)
    if (el := next(self.row_iterator, None)):
        return el
    elif self.row_iterable.next_page_token is None:
        raise StopIteration 
    else:
        self.call_next_batch()
        return next(self.row_iterator)

def call_next_batch(self):
    page_token = self.row_iterable.next_page_token if hasattr(self, 'row_iterable') else None
    self.row_iterable = self.client.list_rows(
        f"{self.project_id}.{self.dataset_id}.{self.table_id}",  
        max_results=self.batch_size,
        page_token=page_token,
    )
    self.row_iterator = iter(self.row_iterable)

def initialize(self):
    self.call_next_batch()
    self.initialized = True


  bq_iterator = BqRowIterator(
  project_id, # to be defined
  dataset_id, # to be defined
  table_id, # to be defined
  batch_size=30
 )

rows = [next(bq_iterator, None) for i in range(50)]

`