| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-01-14 | |||
| 14:16:57 | stephenfin | sean-k-mooney: Sorry, missed the ping. Yeah, the functional tests for OSC aren't great. They're not fully isolated which means running them in parallel can produce all kinds of weird results, depending on the speed on the CI node | |
| 14:17:19 | sean-k-mooney | ah i see | |
| 14:17:20 | stephenfin | sean-k-mooney: so rechecking is the only thing to do until I've time to rework them all | |
| 14:17:53 | sean-k-mooney | well that or run the serially but that proably will be too long | |
| 14:18:03 | stephenfin | yeah, and that just masks the problem | |
| 14:18:13 | sean-k-mooney | yep | |
| 14:18:25 | openstackgerrit | Merged openstack/nova master: tests: Clean up 'test_hypervisors' https://review.opendev.org/c/openstack/nova/+/769903 | |
| 14:18:44 | sean-k-mooney | it need to do what tempest does and create seperate tenant per senario when its doing things like testing list instnaces | |
| 14:18:53 | stephenfin | yes, exactly | |
| 14:19:05 | stephenfin | but I just haven't had time to write a fixture or whatever to do that | |
| 14:20:34 | lyarwood | just dumping this here while I work on something downstream | |
| 14:20:45 | openstackgerrit | Lee Yarwood proposed openstack/nova master: WIP libvirt: Bump MIN_{LIBVIRT,QEMU}_VERSION and NEXT_MIN_{LIBVIRT,QEMU}_VERSION https://review.opendev.org/c/openstack/nova/+/754700 | |
| 14:20:46 | openstackgerrit | Lee Yarwood proposed openstack/nova master: WIP libvirt: Remove MIN_LIBVIRT_VIR_ERR_DEVICE_MISSING https://review.opendev.org/c/openstack/nova/+/754702 | |
| 14:20:46 | openstackgerrit | Lee Yarwood proposed openstack/nova master: WIP libvirt: Remove MIN_{LIBVIRT,QEMU}_BLOCKDEV https://review.opendev.org/c/openstack/nova/+/754701 | |
| 14:20:47 | openstackgerrit | Lee Yarwood proposed openstack/nova master: WIP libvirt: Remove MIN_LIBVIRT_S390X_CPU_COMPARE https://review.opendev.org/c/openstack/nova/+/754704 | |
| 14:20:47 | openstackgerrit | Lee Yarwood proposed openstack/nova master: WIP libvirt: Remove MIN_LIBVIRT_VTPM https://review.opendev.org/c/openstack/nova/+/754703 | |
| 14:20:51 | lyarwood | .. git-review really is slow since the update to gerrit .. | |
| 14:21:03 | stephenfin | I find it varies | |
| 14:21:11 | stephenfin | It's usually fast after a restart | |
| 14:21:28 | stephenfin | *feels fast | |
| 14:21:31 | stephenfin | I've no data to back up that assertion :-\ | |
| 14:23:03 | stephenfin | We're getting close to API microversion 2.100. It could be another cycle or two, but that's coming | |
| 14:30:10 | sean-k-mooney | at some point we might go to nova v4 | |
| 14:41:40 | dansmith | lyarwood: has seemed painfully slow to me as well | |
| 14:41:46 | dansmith | longer for multi-patch uploads | |
| 14:47:35 | gmann | brinzhang0: no, your patch is all ok as it is and fixed the 500 ->403, mine is to improve it as general when other API use this exception further with unit tests | |
| 14:48:34 | gmann | brinzhang0: to avoid conflict I made mine depends on top of your otherwise they both can go independent also | |
| 14:54:26 | sean-k-mooney | dansmith: i havent really noticed much of a change but i havent done many multi patch updates recently | |
| 14:59:58 | gibi | lyarwood: thanks for https://review.opendev.org/c/openstack/nova/+/754702 I will review that and rebase the libvirt event patch top of that | |
| 15:00:31 | lyarwood | gibi: I'll clean that series up later once I'm done with something downstream | |
| 15:00:38 | lyarwood | and np :) | |
| 15:01:33 | gibi | lyarwood: ack | |
| 15:29:15 | artom | stephenfin, when you have time, mind taking another look at https://review.opendev.org/c/openstack/nova-specs/+/765551 ? It feels awefully short to me, but I think I've covered everything...? | |
| 15:29:36 | gibi | lyarwood: as far as I see https://review.opendev.org/c/openstack/nova/+/754702 does not need to depend on the libvirt bump as our current minimum libvirt (5.0.0) already bigger than what is needed (4.1.0) for this patch | |
| 15:44:01 | andrewbogott | melwitt: on the offhand chance that you are around… I'm in the midst of a db migration from rocky to stein and the migration is stuck in a loop. the stuck phase seems to be from your patch https://review.opendev.org/c/openstack/nova/+/633351 (which I can see you refactored quite a lot in T so I'm wondering if this is a known issue) | |
| 15:44:10 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: DNM try to replace retry with libvirt event in detach https://review.opendev.org/c/openstack/nova/+/770246 | |
| 15:44:13 | andrewbogott | obviously you're under no obligation to provide me with tech support :) | |
| 15:47:20 | sean-k-mooney | andrewbogott: do you have an error | |
| 15:47:46 | andrewbogott | sean-k-mooney: stdout says this, forever: "50 rows matched query populate_user_id, 18 migrated" | |
| 15:47:49 | andrewbogott | error log says... | |
| 15:48:02 | andrewbogott | https://www.irccloud.com/pastebin/pNsHlrU0/ | |
| 15:48:09 | andrewbogott | ^ also forever | |
| 15:48:19 | andrewbogott | hm, which doesn't seem to be from the same migration phase as stdout is reporting | |
| 15:48:57 | sean-k-mooney | ya that for keypairs | |
| 15:49:49 | andrewbogott | seems so, although I don't know what gives with stdout logging a different phase | |
| 15:49:51 | sean-k-mooney | so it kind of sounds like it was not in a consitent stae when you start the online migrations | |
| 15:50:15 | gibi | nova meeting starts in 10 minutes in #openstack-meeting-3 | |
| 15:50:15 | andrewbogott | It does | |
| 15:51:09 | andrewbogott | Although I've no idea how it got here, this is a cloud that's been running since Mitaka | |
| 15:52:40 | andrewbogott | Is the 'keypair' referred to there the user-defined keypairs for ssh access? | |
| 15:52:51 | andrewbogott | Because our setup doesn't use those at all, I'd expect it to be unset in all cases. | |
| 15:52:58 | andrewbogott | (we have a PAM setup for ssh) | |
| 15:53:25 | andrewbogott | sean-k-mooney: any chance that we're just a use case that wasn't anticipated by that code? | |
| 15:59:27 | lyarwood | gibi: ah sorry, it's likely left over from the previous bump cleanup and I rebased it into this series by mistake | |
| 15:59:38 | lyarwood | gibi: feel free to pull that out and rebase it under your changes | |
| 16:00:11 | gibi | lyarwood: I think I will do it but not today | |
| 16:00:11 | gibi | lyarwood: ack | |
| 16:00:30 | lyarwood | gibi: np | |
| 16:06:18 | sean-k-mooney | andrewbogott: its the user keypair yes | |
| 16:06:42 | sean-k-mooney | not an unique constratint or forien key at the db level | |
| 16:06:55 | andrewbogott | and yet the migration fails if it's unset :/ | |
| 16:07:10 | andrewbogott | I don't see keypairs in my db though, I see mostly NULLS and some json elsewhere | |
| 16:07:14 | andrewbogott | https://www.irccloud.com/pastebin/CIh6rCTi/ | |
| 16:07:15 | sean-k-mooney | yep so it looke like you missed a previous migration to populate tehm | |
| 16:09:54 | andrewbogott | maybe. My test cloud has the same weird state. Lots of nulls, a few json objects. | |
| 16:10:23 | andrewbogott | anyway I can insert the empty json object into my null records | |
| 16:13:31 | andrewbogott | ok, I've confirmed that the keypair issue is not the thing that's breaking my migration. It fails gracefully if there are no keypairs, as it should. | |
| 16:29:23 | lyarwood | melwitt: https://review.opendev.org/c/openstack/devstack-plugin-ceph/+/711625 - so this works if I enable c-bak and c-vol on the subnode, fun. | |
| 16:43:09 | sean-k-mooney | bauzas: so lookign that the cyborg vgpu spec the last version i reviewd was version 9 | |
| 16:43:16 | sean-k-mooney | bauzas: what was your outstanding issue | |
| 16:43:51 | bauzas | sean-k-mooney: the 'other deployer impact' section, starting PS12 AFAICR | |
| 16:44:11 | sean-k-mooney | it looks like many of my comments were adress but i need to review it porperly | |
| 16:44:18 | sean-k-mooney | ah the advataghes stuff | |
| 16:44:32 | bauzas | and the fact that they weren't using the same methods for modifying the XML, but eventually the last revision fixes it | |
| 16:44:53 | sean-k-mooney | ya | |
| 16:45:03 | bauzas | anyway, /me needs to stop working early | |
| 16:45:03 | sean-k-mooney | that is one of the the thigns i rememebred | |
| 16:45:17 | sean-k-mooney | i porably would just drop the comparison form the spec | |
| 16:45:29 | sean-k-mooney | in the other deployer impact section | |
| 16:45:42 | sean-k-mooney | and jsut leave the note about not configuring the same device in both services | |
| 16:45:43 | bauzas | sean-k-mooney: that's my point indeed | |
| 16:45:48 | bauzas | yeah | |
| 16:46:16 | sean-k-mooney | ok ill leave the same comment after i do a full review | |
| 16:46:32 | bauzas | my other concern would be to maybe use the same module for creating both inventories | |
| 16:46:46 | bauzas | but we could do it as a follow-up | |
| 16:46:53 | bauzas | like, creating a common package | |
| 16:47:13 | bauzas | for both the cyborg agent and the nova-libvirt code | |
| 16:47:29 | bauzas | this way, gibi's concerns about maintenance would be fixed | |
| 16:47:47 | bauzas | if both services would eventually share the same code | |
| 16:47:59 | bauzas | anyway, /me needs to run | |
| 16:48:20 | sean-k-mooney | as in a shared lib | |
| 16:48:37 | sean-k-mooney | we could but im not sure we want to create a new repo at this point in the cycle | |
| 16:49:00 | sean-k-mooney | given the pacakging work that would entail | |
| 16:49:18 | openstackgerrit | Lee Yarwood proposed openstack/nova master: WIP zuul: Add nova-live-migration-ceph job https://review.opendev.org/c/openstack/nova/+/768466 | |
| 16:50:08 | sean-k-mooney | bauzas: i tought gibis main concern in relation to code duplciation was the rest of the code for tracking and assigning mdevs | |
| 16:50:19 | sean-k-mooney | although i may have misread that | |
| 16:52:20 | sean-k-mooney | andrewbogott: by the way did you do the normal db migration before running the online ones? | |
| 16:52:32 | sean-k-mooney | andrewbogott: e.g. nova-manage db sync | |
| 16:52:35 | andrewbogott | I did | |
| 16:53:13 | andrewbogott | sean-k-mooney: I've now determined that the keypair errors were a red herring. It might be that the whole thing was a red herring and stdout was just overly verbose but I need to circle back. | |