Can't import google.protobuf.timestamp_pb2 from App Engine

[barakcoh]

I'm getting ImportError: No module named protobuf from the line from google.protobuf import timestamp_pb2 because the google module refers to the AppEngine SDK.

This is a well-known issue that's been around for years. Maybe you can bundle protobuf directly inside gcloud-python instead of an external dependency?

[dhermes]

Thanks for reporting and for upgrading!

I am curious about a few things:

1. How do you vendor in gcloud-python for App Engine?
2. Do you vendor in protobuf? (AFAIK it isn't part of the GAE runtime)
3. Which services / gcloud-python subpackages do you use? (I'm curious which imports / import paths your code has been using.)
4. Are you modify the gcloud module object so that it acts like a namespace package? If no, this is the solution to your problem. I made an app that shows how to do this. The app is now out-of-date but the file that handles protobuf imports is still valid.

[theacodes]

Are you modify the gcloud module object so that it acts like a namespace package? If no, this is the solution to your problem. I made an app that shows how to do this. The app is now out-of-date but the file that handles protobuf imports is still valid.

I don't think this shouldn't be necessary anymore, I'm interested to see if that's somehow changed.

[dhermes]

Me too. Thanks for chiming in @jonparrott I was going to ping you.

[theacodes]

I should really write an App Engine test case.

[dhermes]

@SaMnCo also reported this import error, it may be

• we don't understand our requirements like we think we do.
• pip doesn't upgrade deps in setup.py as I expect
• a PYTHONPATH issue

[dhermes]

@SaMnCo can you do a pip show protobuf, pip show gcloud in the environment where this is failing and then make sure the same versions are imported:

$ python -c 'import google.protobuf; print(google.protobuf.__version__)'
$ python -c 'import gcloud; print(gcloud.__version__)'

[kvdb]

@dhermes I think I've got a similar problem as the reporter of the issue. Perhaps I can help by providing my answers to your questions:

How do you vendor in gcloud-python for App Engine?

In appengine_config.py:

from google.appengine.ext import vendor                                         
vendor.add('libs')                                                              
import google.protobuf; print(google.protobuf.__version__)                      
import gcloud; print(gcloud.__version__) 

Results in:

3.0.0b2
0.10.0

Do you vendor in protobuf? (AFAIK it isn't part of the GAE runtime)

It's in libs, so yes.

Which services / gcloud-python subpackages do you use? (I'm curious which imports / import paths your code has been using.)

  from gcloud import storage
File "/x/libs/gcloud/storage/__init__.py", line 43, in <module>
  from gcloud.storage.blob import Blob
File "/x/libs/gcloud/storage/blob.py", line 27, in <module>
  from gcloud._helpers import _rfc3339_to_datetime
File "/x/libs/gcloud/_helpers.py", line 26, in <module>
  from google.protobuf import timestamp_pb2
ImportError: No module named protobuf

[dhermes]

@kvdb Thanks a lot. I'm wondering now if it may be related to Darth-Vendor and namespace packages not being the best of friends. I need to dig in and reproduce. Can you confirm that's what you used to vendor in your package?

[theacodes]

Darth is part of the SDK (from google.appengine.ext import vendor) so that's definitely what's being used. It explicitly has support for namespaces. So I'm also unsure what's going on.

[dhermes]

Maybe it's an issue with old versions of pip? The module certainly shows up:

$ mkdir lib
$ pip install -t lib gcloud
$ cd lib/google/protobuf/
$ ls -1 *pb2.py | egrep -v unittest
any_pb2.py
api_pb2.py
descriptor_pb2.py
duration_pb2.py
empty_pb2.py
field_mask_pb2.py
source_context_pb2.py
struct_pb2.py
timestamp_pb2.py
type_pb2.py
wrappers_pb2.py

Going to run a test app in dev_appserver to verify import works.

[dhermes]

OK I just confirmed this works in dev_appserver as well.

@kvdb Have you tried clearing out lib and starting fresh? What version of pip are you using?