| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-06-25 | |||
| 17:38:29 | sean-k-mooney | but its been at least 6 years since i manually did an openstack in stall so i can more or less tell you that the install guides are not hevilly tested so its not uncommon for them to be outdated and have bugs like this | |
| 17:40:01 | harsha24 | okay | |
| 17:57:04 | mnasiadka | Hi there | |
| 17:57:33 | mnasiadka | Is there any reason or known bug that nova-compute logs are stuck on "Creating image" while spawning instance? (Train) | |
| 18:19:11 | openstackgerrit | Harshavardhan Metla proposed openstack/nova master: moved the quoted section https://review.opendev.org/738081 | |
| 18:29:27 | openstackgerrit | Harshavardhan Metla proposed openstack/nova master: Raising a failure in case of 403 https://review.opendev.org/735068 | |
| 18:43:51 | sean-k-mooney | mnasiadka: kindof | |
| 18:44:26 | openstack | Launchpad bug 1874032 in OpenStack Compute (nova) "nova-compute become stuck when doing IO on busy file system" [Wishlist,In progress] - Assigned to Alexandre arents (aarents) | |
| 18:44:26 | sean-k-mooney | mnasiadka: https://bugs.launchpad.net/nova/+bug/1874032 https://review.opendev.org/#/c/734776/6 | |
| 18:45:14 | mnasiadka | sean-k-mooney: rather not the case, the disk is not i/o busy | |
| 18:45:14 | sean-k-mooney | mnasiadka: so if you are hitting ^ aarents almost has the fix ready | |
| 18:45:43 | sean-k-mooney | in that case no im not aware of a bug where nova hangs on taking a snapshot | |
| 18:46:30 | mnasiadka | It’s spawning a new vm, not a snapshot... then let me try to do some tracing what is it stuck on... | |
| 18:47:09 | sean-k-mooney | oh "openstack server create image" is how you create a snapshot sorry | |
| 18:47:52 | sean-k-mooney | mnasiadka: so its hangin on actully creating the image on disk are you sueing the default qcow/flat image backend | |
| 18:49:22 | openstack | Launchpad bug 1880828 in OpenStack Bundles "New instance is always in "spawning" status" [Undecided,New] | |
| 18:49:22 | sean-k-mooney | mnasiadka: i assume its not this https://bugs.launchpad.net/nova/+bug/1880828 | |
| 18:53:11 | mnasiadka | sean-k-mooney: in my case it looks like it’s stuck on downloading image, because there’s only disk.info file and no image fragments anywhere | |
| 18:58:45 | openstack | Launchpad bug 1884068 in OpenStack Compute (nova) "Instance stuck in build state when some/one compute node is unreachable" [Undecided,New] | |
| 18:58:45 | sean-k-mooney | gibi: does my last comment #3 on https://bugs.launchpad.net/nova/+bug/1884068 make sense to you? i think we shoudl close that as nto a bug | |
| 18:59:04 | sean-k-mooney | mnasiadka: it would be good to check the glance log then | |
| 18:59:25 | sean-k-mooney | and see if it handeled the requiest properly? | |
| 20:00:40 | mnasiadka | sean-k-mooney: glance says it did, but I don’t trust it ;) | |
| 21:07:51 | mnaser | hmm | |
| 21:08:43 | mnaser | https://github.com/openstack/nova/blob/master/nova/scheduler/client/report.py#L1893 and https://github.com/openstack/nova/blob/master/nova/scheduler/client/report.py#L1941 and https://github.com/openstack/nova/blob/master/nova/scheduler/client/report.py#L1612 and https://github.com/openstack/nova/blob/master/nova/scheduler/client/report.py#L1744 all bomb out prettty hard if placement isn't available | |
| 21:08:55 | mnaser | because json() fails to decode json content and it blows up in very bad ways | |
| 21:10:53 | mnaser | does anyone know if there's a way of checking if the content is json first? | |
| 21:11:06 | mnaser | i could try/except but eh | |
| 21:49:38 | dansmith | mnaser: content-type? | |
| 21:50:11 | dansmith | but try..except around the .json is the python way | |
| 22:04:58 | mriedem | seems like that would be a lot of moles to whack over time, i'm assuming you're getting html back? also, isn't there a retries wrapper on that? | |
| 22:05:18 | mriedem | twould be way down in ksa if there is | |
| 22:09:17 | dansmith | I was just assuming that was coming straight from requests | |
| 22:09:30 | mriedem | else yeah you can check the response headers https://requests.readthedocs.io/en/master/user/quickstart/#response-headers | |
| 22:09:36 | dansmith | but yeah, looks like lots of assuming success there | |
| 22:09:40 | mriedem | `r.headers.get('content-type')` | |
| 22:09:57 | mriedem | i think our stuff uses ksa which wraps requests way down in the bowels | |
| 22:10:07 | dansmith | ack | |
| 22:10:12 | mriedem | and i'm assuming there is a urllib3 Retry wrapper in the mix | |
| 22:10:59 | mriedem | if retries timeout you get a different error, but i've also seen the weirdness where you get back like a 302 with html content assuming `if r:` (truthy) and then blow on on `r.json()` | |
| 23:23:28 | openstackgerrit | Merged openstack/nova master: Plumb image import functionality through our glance module https://review.opendev.org/731550 | |
| #openstack-nova - 2020-06-26 | |||
| 00:49:13 | openstackgerrit | Ghanshyam Mann proposed openstack/nova master: DNM: testing nova gate on ubuntu focal https://review.opendev.org/738126 | |
| 01:45:41 | openstackgerrit | Ghanshyam Mann proposed openstack/os-vif master: DNM: testing os-vif gate on ubuntu focal https://review.opendev.org/738130 | |
| 01:48:33 | mnaser | dansmith, mriedem: in my case, i'm getting non-json content (because there's a load balancer infront of placement) | |
| 01:48:36 | mnaser | content-type seems sane though | |
| 02:32:51 | openstackgerrit | Jinsheng Zhang proposed openstack/nova master: Add API to support multiple boot-volumes with boot order selection https://review.opendev.org/737913 | |
| 02:40:09 | gmann | dansmith: melwitt sean-k-mooney elod gibi pep8 job cherry pick script failure was just rebase issue (somehow zuul way of merging the code at runtime did not work for this script). after rebase, script can find all hashes correctly and job pass - https://review.opendev.org/#/c/728057/ | |
| 02:43:37 | sean-k-mooney | gmann: it might be down to the branch name | |
| 02:44:03 | sean-k-mooney | e.g. how zuul create the reiview branch and merges things | |
| 02:44:40 | sean-k-mooney | but glad it worked. | |
| 02:45:36 | gmann | yeah and commit msg is somethings not being picked correctly that is why hashes were not picked by script | |
| 02:48:29 | sean-k-mooney | anyway im going to go to sleep now o/ i was just getting my phone which i left on my desk :) | |
| 03:32:16 | openstackgerrit | Merged openstack/nova master: Raise InstanceMappingNotFound if StaleDataError is encountered https://review.opendev.org/734400 | |
| 03:35:01 | openstackgerrit | Wenping Song proposed openstack/nova master: WIP: Cyborg shelve/unshelve support https://review.opendev.org/729563 | |
| 08:07:08 | openstackgerrit | Jinsheng Zhang proposed openstack/nova master: Add API to support multiple boot-volumes with boot order selection https://review.opendev.org/737913 | |
| 08:38:12 | openstackgerrit | Wenping Song proposed openstack/nova master: cyborg evacuate support https://review.opendev.org/715326 | |
| 08:53:37 | openstackgerrit | Merged openstack/nova master: Remove deprecated nova.image.download hook https://review.opendev.org/725338 | |
| 08:53:44 | openstackgerrit | Merged openstack/nova master: libvirt: Remove workaround for unsupported libvirt version https://review.opendev.org/729656 | |
| 08:54:58 | openstack | Launchpad bug 1884068 in OpenStack Compute (nova) "Instance stuck in build state when some/one compute node is unreachable" [Undecided,New] | |
| 08:54:58 | gibi | sean-k-mooney: responeded in https://bugs.launchpad.net/nova/+bug/1884068 | |
| 09:05:19 | openstackgerrit | Takashi Natsume proposed openstack/nova master: Remove six.PY2 and six.PY3 https://review.opendev.org/726512 | |
| 09:05:28 | openstackgerrit | Takashi Natsume proposed openstack/nova master: Remove six.add_metaclass https://review.opendev.org/726565 | |
| 09:05:38 | openstackgerrit | Takashi Natsume proposed openstack/nova master: Remove six.reraise https://review.opendev.org/726898 | |
| 09:05:56 | openstackgerrit | Takashi Natsume proposed openstack/nova master: Remove six.moves https://review.opendev.org/727224 | |
| 09:06:07 | openstackgerrit | Takashi Natsume proposed openstack/nova master: Remove six.iteritems/itervalues/iterkeys https://review.opendev.org/727757 | |
| 09:06:17 | openstackgerrit | Takashi Natsume proposed openstack/nova master: Remove six.byte2int/int2byte https://review.opendev.org/727777 | |
| 09:06:27 | openstackgerrit | Takashi Natsume proposed openstack/nova master: Remove six.binary_type/integer_types/string_types https://review.opendev.org/728094 | |
| 09:15:32 | openstack | Debian bug 963339 in src:nova "nova: FTBFS: ValueError: Sentinels must not start with _" [Serious,Open] | |
| 09:15:32 | zigo | Does anyone know what's going on with this bug ? https://bugs.debian.org/963339 | |
| 09:15:38 | zigo | Is there any solution already commited to master? | |
| 09:17:26 | openstackgerrit | Takashi Natsume proposed openstack/nova master: Remove six.text_type (1/2) https://review.opendev.org/728109 | |
| 09:17:32 | gibi | zigo: I haven't seen this before. let me try to reproduce the test failure locally | |
| 09:17:55 | zigo | gibi: I don't think you will unless you use the same set of dependency as in Debian Sid. | |
| 09:18:13 | zigo | It's probably one dependency that was upgraded or something similar. | |
| 09:18:13 | openstackgerrit | Takashi Natsume proposed openstack/nova master: Remove six.text_type (2/2) https://review.opendev.org/728117 | |
| 09:19:30 | zigo | Maybe this is py3.8 related? | |
| 09:19:38 | gibi | zigo: what version of oslo_utils that build uses? | |
| 09:20:03 | zigo | gibi: Probably 4.1.1 | |
| 09:20:34 | zigo | checking ... | |
| 09:21:03 | zigo | Yeah, that's correct. | |
| 09:21:09 | openstackgerrit | Elod Illes proposed openstack/nova stable/ussuri: libvirt: Mark e1000e VIF as supported https://review.opendev.org/736957 | |
| 09:21:58 | gibi | hm the ValueError is coming from here https://github.com/openstack/oslo.utils/blame/master/oslo_utils/fixture.py#L82 but that piece of code is 2 years old | |
| 09:22:15 | zigo | Oh ... | |
| 09:23:40 | zigo | Indeed, I was missreading the logs. | |
| 09:24:26 | openstackgerrit | Stephen Finucane proposed openstack/nova stable/train: libvirt: Mark e1000e VIF as supported https://review.opendev.org/736962 | |
| 09:24:39 | openstackgerrit | Stephen Finucane proposed openstack/nova stable/train: libvirt: Mark e1000e VIF as supported https://review.opendev.org/736962 | |
| 09:24:41 | zigo | gibi: Do you think it'd have any consequence if we were to remove the underscore check? | |
| 09:26:27 | gibi | zigo: as far as I understand the logs this fails in the test code part of the test case so I think it is save to gut that check temporarly | |
| 09:26:39 | gibi | s/save/safe/ | |
| 09:28:57 | gibi | looking more to the stacktrace I suspect this is some kind of mock autospecing issue. So far the same test passes for me locally in py38 | |
| 09:29:50 | zigo | gibi: The full log with what version of each component is here: qa-logs.debian.net/2020/06/20/nova_21.0.0-2_unstable.log | |
| 09:29:56 | gibi | zigo: thanks | |
| 09:36:57 | zigo | gibi: The easiest way is probably to run Debian Sid, and do "apt-get build-dep nova" then pip freeze, then compare the output with your venv. | |
| 09:37:10 | zigo | gibi: I can do the sid part send send it somewhere if you want. | |
| 09:38:00 | gibi | zigo: yeah please provide that info for me | |
| 09:39:16 | sean-k-mooney | gibi: oh your right it was --availabity-zone <az> <server name> | |
| 09:40:01 | zigo | I'm installing a fresh sid machine. | |
| 09:40:27 | gibi | thanks | |
| 09:50:07 | masterpe | Hi, I have a question about a upgrade from Rocky to Stein. | |
| 09:50:28 | openstackgerrit | Alexandre Arents proposed openstack/nova master: Limit the number of concurrent snapshots https://review.opendev.org/736169 | |