| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-01-14 | |||
| 16:49:18 | openstackgerrit | Lee Yarwood proposed openstack/nova master: WIP zuul: Add nova-live-migration-ceph job https://review.opendev.org/c/openstack/nova/+/768466 | |
| 16:50:08 | sean-k-mooney | bauzas: i tought gibis main concern in relation to code duplciation was the rest of the code for tracking and assigning mdevs | |
| 16:50:19 | sean-k-mooney | although i may have misread that | |
| 16:52:20 | sean-k-mooney | andrewbogott: by the way did you do the normal db migration before running the online ones? | |
| 16:52:32 | sean-k-mooney | andrewbogott: e.g. nova-manage db sync | |
| 16:52:35 | andrewbogott | I did | |
| 16:53:13 | andrewbogott | sean-k-mooney: I've now determined that the keypair errors were a red herring. It might be that the whole thing was a red herring and stdout was just overly verbose but I need to circle back. | |
| 16:53:26 | andrewbogott | the keypair migration throws a log warning about missing keypairs but moves on as it should | |
| 16:53:48 | sean-k-mooney | andrewbogott: so it might be related to this | |
| 16:53:50 | sean-k-mooney | https://review.opendev.org/c/openstack/nova/+/633351/31/nova/objects/instance_mapping.py#250 | |
| 16:54:02 | sean-k-mooney | we skipi instance that dont have instance cell mappings | |
| 16:54:29 | andrewbogott | yeah, I think that's likely the issue. I need to get out of my deployment window and then I'll circle back and debug :) | |
| 16:54:29 | sean-k-mooney | so before you run that you need to ensure all instance have the cell mappings set | |
| 16:55:01 | andrewbogott | hm, I thought that the cell mapping thing was handled by an upgrade script many versions ago | |
| 16:55:09 | andrewbogott | but since I only have one cell it should be easy to fix by hand in the db | |
| 16:56:14 | sean-k-mooney | it could be deleted but unachived/purged isntances | |
| 16:56:28 | sean-k-mooney | which is why it suggest nova-manage db archive_deleted_rows --purge | |
| 16:57:19 | andrewbogott | yeah, I ran that but it didn't do anything (didn't report anything on stdout and also didn't affect behavior of the migration) | |
| 17:01:17 | sean-k-mooney | ya so i would guess you jsut need to add the mappings and hopefully it will correct the isseu you have | |
| 17:19:16 | openstackgerrit | Lee Yarwood proposed openstack/nova master: libvirt: Remove MIN_LIBVIRT_VIR_ERR_DEVICE_MISSING https://review.opendev.org/c/openstack/nova/+/754702 | |
| 17:19:18 | lyarwood | gibi: ^ fixed, sorry about that. | |
| 17:24:24 | openstackgerrit | Lee Yarwood proposed openstack/nova master: libvirt: Remove MIN_LIBVIRT_VIR_ERR_DEVICE_MISSING https://review.opendev.org/c/openstack/nova/+/754702 | |
| 17:36:49 | lyarwood | does anyone recall where we list the supported distros for a release? | |
| 17:37:06 | openstackgerrit | Stephen Finucane proposed openstack/nova-specs master: Add spec for configurable-instance-hostnames https://review.opendev.org/c/openstack/nova-specs/+/770842 | |
| 17:37:08 | stephenfin | lyarwood: that was in the wiki | |
| 17:37:24 | stephenfin | actually, no - I'm thinking of the libvirt version matrix | |
| 17:38:04 | lyarwood | https://github.com/openstack/governance/blob/2bdd9cff00fb40b2f95b66cad47ae1cfd14a2f1b/reference/project-testing-interface.rst#linux-distributions is all I can find | |
| 17:38:16 | stephenfin | artom: Looking at your spec now, but just FYI I've a super low priority proposal for a '--hostname' parameter here https://review.opendev.org/c/openstack/nova-specs/+/770842 Had a look earlier and it's pretty easy | |
| 17:38:19 | lyarwood | and yeah our distro matrix in the wiki just lists versions | |
| 17:38:27 | stephenfin | lyarwood: fungi would probably know | |
| 17:38:33 | stephenfin | lyarwood++ good call | |
| 17:38:57 | fungi | it's in governance, getting it now | |
| 17:39:48 | fungi | https://governance.openstack.org/tc/reference/runtimes/wallaby.html | |
| 17:39:49 | lyarwood | oh sorry I see the runtime link at the bottom now, helps if I view the hosted page instead of the source | |
| 17:39:57 | lyarwood | fungi: many thanks | |
| 17:40:00 | fungi | yw | |
| 17:42:41 | sean-k-mooney | fungi: that does not cover libvirt verions | |
| 17:42:46 | sean-k-mooney | or qemu | |
| 17:43:01 | sean-k-mooney | we manage that seperatly | |
| 17:43:28 | lyarwood | I didn't need that, just the distros | |
| 17:43:48 | sean-k-mooney | ah then ya latest lts of ubuntu centos and suse | |
| 17:43:48 | lyarwood | or openSUSE sorry | |
| 17:44:39 | sean-k-mooney | fungi: has there been any moemente on updating centos 8 sot centos 8 stream | |
| 17:44:54 | sean-k-mooney | fungi: i assume we will change that in X | |
| 17:45:27 | fungi | sean-k-mooney: i'm not aware of any tc level decision around that yet. discussions were that we only needed to solve it for the cycle in which 8 becomes eol | |
| 17:45:49 | sean-k-mooney | ok so i guess technialy Y then | |
| 17:46:02 | sean-k-mooney | EOL is decemebr | |
| 17:46:06 | fungi | though i'm not entirely sure what it means for stable branches which supported 8 and how we choose to test them | |
| 17:46:25 | sean-k-mooney | ya | |
| 17:46:38 | fungi | since opendev doesn't usually continue to provide images for eol distro versions for very long | |
| 17:46:46 | sean-k-mooney | one option might be to use rocky linux or one of the repelacement for centos | |
| 17:47:41 | sean-k-mooney | i guess we will see how things play out | |
| 17:48:59 | sean-k-mooney | centos 8 stream should sablise once centos 9 stream is a thing but no idea when that will be a thing or how long centos 8 stream will be supported when its released | |
| 17:58:02 | openstackgerrit | Lee Yarwood proposed openstack/nova master: libvirt: Remove MIN_{LIBVIRT,QEMU}_BLOCKDEV https://review.opendev.org/c/openstack/nova/+/754701 | |
| 17:58:02 | openstackgerrit | Lee Yarwood proposed openstack/nova master: libvirt: Bump MIN_{LIBVIRT,QEMU}_VERSION and NEXT_MIN_{LIBVIRT,QEMU}_VERSION https://review.opendev.org/c/openstack/nova/+/754703 | |
| 17:58:03 | openstackgerrit | Lee Yarwood proposed openstack/nova master: libvirt: Remove MIN_LIBVIRT_S390X_CPU_COMPARE https://review.opendev.org/c/openstack/nova/+/754704 | |
| 18:05:34 | openstackgerrit | sean mooney proposed openstack/nova stable/train: only wait for plugtime events in pre-live-migration https://review.opendev.org/c/openstack/nova/+/770844 | |
| 18:11:53 | stephenfin | lyarwood: Seeing as you're looking at DB stuff at the moment, think you could take a look at these, my two oldest outstanding patches? :) https://review.opendev.org/c/openstack/nova/+/530905/ https://review.opendev.org/c/openstack/nova/+/530487/ | |
| 18:12:44 | openstackgerrit | sean mooney proposed openstack/nova master: use psycopg2 binary instead of source package https://review.opendev.org/c/openstack/nova/+/770473 | |
| 18:16:30 | sean-k-mooney | stephenfin: why do we need uuid in the pci devices table? | |
| 18:18:04 | sean-k-mooney | oh its for tracking them in placment | |
| 18:18:09 | sean-k-mooney | so its not used currently | |
| 18:21:22 | sean-k-mooney | +1 on both but its also strictly speaking not needed currently unless i missed where its beign used | |
| 18:21:36 | sean-k-mooney | in anycase time for dinner | |
| 19:35:17 | artom | melwitt, so, for the record, I am looking at https://review.opendev.org/c/openstack/nova/+/769614 :) | |
| 19:35:35 | artom | But it's in an area of hardware.py that I'm not really familiar with, so there's a learning curve | |
| 19:38:33 | dansmith | lyarwood: since you've brought it up a couple times, I think this is going to fix the multistore test hang/fail: https://review.opendev.org/c/openstack/tempest/+/770850 | |
| 19:38:37 | dansmith | also gibi ^ | |
| 19:56:08 | qqmber | hi there | |
| 22:41:37 | openstackgerrit | Ghanshyam proposed openstack/nova master: Reuse code from oslo lib for JSON policy migration https://review.opendev.org/c/openstack/nova/+/763487 | |
| 22:46:46 | openstackgerrit | Ghanshyam proposed openstack/placement master: [goal] Deprecate the JSON formatted policy file https://review.opendev.org/c/openstack/placement/+/764394 | |
| 22:54:24 | dansmith | melwitt: replied | |
| 22:55:03 | melwitt | dansmith: danke | |
| 22:55:59 | melwitt | I was coincidentally looking at some different gate failures involving presumable stress on node resources and wanted to see if there are similarities | |
| 22:56:43 | dansmith | well, this was a pretty standard OOM once I realized that, | |
| 22:57:09 | dansmith | but it's unusual that the OOM is caused by the test runner, so I hadn't seen an OOM manifest itself as a killing of one worker before | |
| 22:58:52 | melwitt | I see now, in syslog of the example you linked. literal oom-killer messages. what I've been looking at today is different | |
| 23:00:06 | dansmith | this one blows up so fast that it doesn't even register in dstat, | |
| 23:00:15 | dansmith | which is what took me a bit to realize it was an oom | |
| 23:00:28 | dansmith | it goes from like 45M to 1.5g in a couple seconds locally | |
| 23:00:33 | melwitt | wow | |
| 23:01:09 | melwitt | how were you able to tell it's the test runner? something in the logs? | |
| 23:01:45 | dansmith | well, the oom was a python process, and one worker (which is python) just never reports its results, | |
| 23:02:21 | dansmith | then reproing locally (with plenty of memory) I see one test runner blip to 1.5g, before exiting normally, which is the same size as the victim from the killer | |
| 23:02:42 | dansmith | I don't oom because I have lots of memory locally, but I could see that balloon, and if it happens at the wrong time in the gate, it'll trigger and die | |
| 23:03:06 | melwitt | I see, thanks | |
| 23:05:30 | melwitt | nice find. I've been looking at a lot of gate failures lately that seem to be outside of openstack and trying to find hints about what could be causing them | |
| 23:21:49 | openstackgerrit | melanie witt proposed openstack/nova stable/stein: Change default num_retries for glance to 3 https://review.opendev.org/c/openstack/nova/+/764963 | |
| 23:24:43 | openstackgerrit | melanie witt proposed openstack/nova stable/stein: Use fair locks in resource tracker https://review.opendev.org/c/openstack/nova/+/770657 | |
| 23:49:38 | sean-k-mooney | melwitt: actully regarding ^ | |
| 23:50:00 | sean-k-mooney | i realised that while the upper-constatis allows it im not sure lower constartis on stien does | |
| 23:50:50 | sean-k-mooney | https://github.com/openstack/nova/blob/stable/stein/lower-constraints.txt#L79 oslo.concurrency==3.26.0 | |
| 23:51:19 | sean-k-mooney | it need 3.29.0 | |
| 23:51:42 | sean-k-mooney | are we allowed to bump it like that in a backport? | |
| 23:54:02 | sean-k-mooney | for what its worth i think its a good thing to backport just dont know what stable policy allows | |
| 23:54:06 | melwitt | oh yeah, we aren't. so I guess I was right originally, that it couldn't be backported | |
| 23:54:34 | melwitt | sigh | |
| 23:55:15 | sean-k-mooney | well with the patch on gerrit if peopel want to use it it should be eaiser for them | |
| 23:56:13 | sean-k-mooney | to do the backport you would have to add the fair lock form oslo to noava | |
| 23:56:32 | sean-k-mooney | and just use that instead | |