Using Iterators for list_topics() in Pub/Sub. by dhermes · Pull Request #2602 · googleapis/google-cloud-python

dhermes · 2016-10-25T06:45:59Z

In the process, had to add custom support for the GAX page iterator in our core Iterator implementation.

@tseaver @jonparrott I am sending this out as-is hoping for some feedback (list_subscriptions and topic_list_subscriptions also need to be done). I don't feel that great about the way I did custom paging in Iterator, maybe it's worth defining a GAX subclass of Iterator?

Also, in order to avoid too many radical changes to the code, I punted on getting a client into the Iterator returned by the _gax / connection implementations. As can be seen by the unit tests, this effectively makes them useless (more-so in the HTTP case than in the GAX case). I'd be happy to send a follow-up PR to make client accessible in the _PublisherAPI implementation, but again, wanted to make this commit digestible.

NOTE: Has #2594 as diffbase.

core/google/cloud/_testing.py

+            raise StopIteration
+        else:
+            items = self._items
+            self._items = None


core/google/cloud/iterator.py

+                      :class:`Iterator` that started the page. By default uses
+                      the HTTP pages iterator. Meant to provide a custom
+                      way to create pages (potentially with a custom
+                      transport such as gRPC).


pubsub/google/cloud/pubsub/_gax.py

+        #       iterator instance returned.
+        return Iterator(client=None, path=path,
+                        item_to_value=_item_to_topic,
+                        page_iter=page_iter)


pubsub/google/cloud/pubsub/connection.py


-    def list_topics(self, project, page_size=None, page_token=None):
+    @staticmethod
+    def list_topics(project, page_size=None, page_token=None):


system_tests/pubsub.py


    def test_list_topics(self):
-        before, _ = Config.CLIENT.list_topics()
+        before = list(Config.CLIENT.list_topics())


tseaver · 2016-10-25T14:44:24Z

Can we disable the Circle CI bit until we fix its config? Issues with it:

The bit which skips system-tests for PRs is Travis-specific.
It is barfing trying to find the correct Python 3.4 / 3.5 interpreter.

In the process, had to add custom support for the GAX page iterator in our core Iterator implementation.

Also tweaking an exhausted _GAXPageIterator in google.cloud._testing.

tseaver · 2016-10-25T18:43:12Z

LGTM with the _client attr on the API objects.

dhermes · 2016-10-25T18:46:25Z

@tseaver So I am good to merge? I am currently in the process of breaking Iterator up into Iterator (a base), HTTPIterator and GAXIterator. I think that'd be better as a standalone PR.

As for the Travis failure, that is due to a merge of RuntimeConfig conflicting with #2594 (which is totally fine, I'll send a PR to fix ASAP).

tseaver

@dhermes Yes, the "LGTM" was sincere.

dhermes · 2016-10-25T18:57:19Z

@tseaver Haha thanks. (I wasn't sure if you were LGTM-ing the PR or just the single commit, pending work to be done.)

I'll make sure to get the refactor out soon.

Using Iterators for list_topics() in Pub/Sub.

dhermes added the api: pubsub Issues related to the Pub/Sub API. label Oct 25, 2016

dhermes assigned daspecster, tseaver and theacodes Oct 25, 2016

googlebot added the cla: yes This human has signed the Contributor License Agreement. label Oct 25, 2016

tseaver reviewed Oct 25, 2016

View reviewed changes

Using Iterators for list_topics() in Pub/Sub.

83beb2c

In the process, had to add custom support for the GAX page iterator in our core Iterator implementation.

dhermes force-pushed the pubsub-iterators branch from f757733 to 83beb2c Compare October 25, 2016 16:46

dhermes mentioned this pull request Oct 25, 2016

Circle CI discussion #2605

Closed

Passing client to Pub/Sub API classes.

95bdd83

Also tweaking an exhausted _GAXPageIterator in google.cloud._testing.

tseaver approved these changes Oct 25, 2016

View reviewed changes

dhermes merged commit 0e4a660 into googleapis:master Oct 25, 2016

dhermes deleted the pubsub-iterators branch October 25, 2016 18:57

dhermes mentioned this pull request Nov 14, 2016

Upgrading core to version to 0.21.0. #2733

Merged

richkadel pushed a commit to richkadel/google-cloud-python that referenced this pull request May 6, 2017

Merge pull request googleapis#2602 from dhermes/pubsub-iterators

cbd5c20

Using Iterators for list_topics() in Pub/Sub.

theacodes unassigned daspecster Sep 28, 2018

parthea pushed a commit that referenced this pull request Nov 24, 2025

Merge pull request #2602 from dhermes/pubsub-iterators

46d00d7

Using Iterators for list_topics() in Pub/Sub.

Conversation

dhermes commented Oct 25, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

tseaver commented Oct 25, 2016

Uh oh!

tseaver commented Oct 25, 2016

Uh oh!

dhermes commented Oct 25, 2016

Uh oh!

tseaver left a comment

Choose a reason for hiding this comment

Uh oh!

dhermes commented Oct 25, 2016

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

dhermes commented Oct 25, 2016 •

edited

Loading