| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-08-28 | |||
| 14:47:37 | mriedem | we're losing the tracing too | |
| 14:47:38 | dansmith | I still wanna know why this instance didn't build in case it is related to tweaking the claim stuff, | |
| 14:47:41 | mriedem | from one of the sourced scripts | |
| 14:47:45 | dansmith | but it seems like it's waiting for the neutron event | |
| 14:48:13 | dansmith | it's still in building state at the end of the run | |
| 14:48:55 | dansmith | ah, artom said something about that early this morning.. | |
| 14:49:17 | mriedem | instance c2b876e6-b616-4259-b03b-f0b2bd3d8908 right? | |
| 14:49:25 | dansmith | yup | |
| 14:50:46 | mriedem | yup start waiting for the event here https://zuul.opendev.org/t/openstack/build/d3d4357c13be452fa8e64a6b612c256d/log/logs/subnode-2/screen-n-cpu.txt.gz#1905 | |
| 14:50:55 | mriedem | Aug 28 04:38:17.325822 ubuntu-bionic-vexxhost-vexxhost-sjc1-0010477321 nova-compute[24765]: DEBUG nova.compute.manager [None req-4d7771be-f2e2-429d-87c7-1cd4469cede5 demo admin] [instance: c2b876e6-b616-4259-b03b-f0b2bd3d8908] Preparing to wait for external event network-vif-plugged-afd502d7-a21d-48f2-8607-881a86f0781f {{(pid=24765) prepare_for_instance_event /opt/stack/new/nova/nova/compute/manager.py:329}} | |
| 14:51:52 | mriedem | got the event here | |
| 14:51:53 | mriedem | https://zuul.opendev.org/t/openstack/build/d3d4357c13be452fa8e64a6b612c256d/log/logs/subnode-2/screen-n-cpu.txt.gz#1941 | |
| 14:51:58 | mriedem | Aug 28 04:38:21.643105 ubuntu-bionic-vexxhost-vexxhost-sjc1-0010477321 nova-compute[24765]: DEBUG nova.compute.manager [req-61851260-be42-4748-9d79-4b0ec5c08ed6 req-3c63a866-a638-4ec7-9f56-3a7aeb2e9958 service nova] [instance: c2b876e6-b616-4259-b03b-f0b2bd3d8908] Received event network-vif-plugged-afd502d7-a21d-48f2-8607-881a86f0781f {{(pid=24765) external_instance_event /opt/stack/new/nova/nova/compute/manager.py:8819}} | |
| 14:52:03 | mriedem | 4 seconds later | |
| 14:52:34 | dansmith | presumably unrelated to his change though if we've made it to the neutron event stage | |
| 14:54:22 | mriedem | i think during artom's revert resize ovs hybrid plug race thing that evacuate could have the same kind of race b/c of the time that we trigger the event by updating the port binding and when the driver's spawn method start waiting for it | |
| 14:55:30 | mriedem | although, if we didn't have a handler registered for the event callback we'd see "Received unexpected event" in the logs | |
| 14:55:31 | mriedem | which we don't | |
| 14:55:37 | mriedem | so idk what's going on there | |
| 14:56:46 | mriedem | the guest power state goes from paused > running > paused agin | |
| 14:56:47 | mriedem | *agian | |
| 14:56:49 | mriedem | gah | |
| 14:57:10 | mriedem | but still, i'm wondering why the run_tests script didn't actually run any live migration tests | |
| 14:58:12 | kashyap | aspiers: When you get a sec, summarized our discussion from #virt (OFTC) here, for other Stackers: https://review.opendev.org/#/c/673790/ | |
| 15:01:29 | openstackgerrit | Matt Riedemann proposed openstack/nova master: DNM: Run LM integration tests with NUMA flavor https://review.opendev.org/678887 | |
| 15:02:24 | mriedem | dansmith: artom: let's see if ^ helps | |
| 15:02:52 | dansmith | cool | |
| 15:05:03 | melwitt | mriedem: follow up patch is here https://review.opendev.org/678988 | |
| 15:07:35 | melwitt | dansmith: the multi-cell archive_deleted_rows set (this patch and the one on top of it) have a +2 from mriedem, if you would fancy taking a look please | |
| 15:07:40 | melwitt | https://review.opendev.org/507486 | |
| 15:08:02 | mriedem | got it | |
| 15:09:23 | melwitt | danke | |
| 15:14:33 | dansmith | melwitt: I know, I've had it open just have to get to it | |
| 15:15:21 | melwitt | ok, thanks. sorry didn't know you already knew | |
| 15:15:43 | dansmith | yeah, I opened the tab when mriedem pinged us yesterday | |
| 15:18:00 | melwitt | k | |
| 15:22:27 | mriedem | artom: did you ever follow up with getting something on the starlingx weekly meeting about testing the numa LM series? | |
| 15:30:33 | mriedem | just realized that we have 41 approved blueprints for train with only 12 complete and 2 weeks to FF | |
| 15:31:57 | mriedem | probably time to start deferring things that we know aren't going to land | |
| 15:34:25 | donnyd | so yesterdays test of mounting _base on a shared location didn't go well | |
| 15:35:46 | donnyd | for some unknown reason qemu-img info reports back that the cached image could not be found | |
| 16:00:21 | mriedem | stephenfin: can we effectively mark this complete? https://blueprints.launchpad.net/nova/+spec/remove-cells-v1 | |
| 16:00:29 | mriedem | there are only 2 open changes https://review.opendev.org/#/q/topic:bp/remove-cells-v1+(status:open) | |
| 16:00:54 | openstackgerrit | Merged openstack/nova master: Specify availability_zone to unshelve https://review.opendev.org/663851 | |
| 16:00:58 | stephenfin | Yup, it's complete. Those two are very tricky post-removal cleanup patches | |
| 16:01:05 | stephenfin | I'll come back to them when I can | |
| 16:03:41 | cdent | cells v1 is gone? wooooooo! | |
| 16:03:42 | mriedem | melwitt: would be sweet if we can get https://review.opendev.org/#/c/640898/ approved by the end of the week - it's 2 weeks to FF and it'd be nice to land that series in time for train | |
| 16:04:46 | melwitt | mriedem: indeed, I'm gonna update it after this last call I'm on. thanks for reminding | |
| 16:09:01 | dansmith | mriedem: are you pretty confident on this archive multi cells thing? I've skimmed the logic and tests and just can't really get myself to go deep on it | |
| 16:09:40 | dansmith | seems pretty low risk even if there is something wrong in here, so I'm tempted to just +W based on my skimming | |
| 16:09:52 | dansmith | s/seems/should be/ | |
| 16:12:32 | openstackgerrit | Merged openstack/nova master: scheduler: Flatten 'ResourceRequest.from_extra_specs', 'from_image_props' https://review.opendev.org/674894 | |
| 16:12:41 | openstackgerrit | Merged openstack/nova master: Remove 'hw:cpu_policy', 'hw:mem_page_size' extra specs from API samples https://review.opendev.org/675338 | |
| 16:12:57 | mriedem | dansmith: bombs away | |
| 16:13:05 | mriedem | and yeah i've gone over it several times in detail now | |
| 16:13:09 | mriedem | probably annoying detail for melwitt | |
| 16:13:21 | mriedem | the nova-next change on top also shows it working as expected | |
| 16:15:11 | dansmith | oh sweet I didn't even notice | |
| 16:16:21 | dansmith | got those as well | |
| 16:19:05 | openstackgerrit | Stephen Finucane proposed openstack/nova master: objects: Add online migration for legacy NUMA objects https://review.opendev.org/537414 | |
| 16:19:06 | openstackgerrit | Stephen Finucane proposed openstack/nova master: trivial: Rename 'nova.tests.unit.test_nova_manage' https://review.opendev.org/679095 | |
| 16:19:06 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Ensure online migrations have a unique name https://review.opendev.org/679096 | |
| 16:19:37 | stephenfin | mriedem: You'll probably be interested in the last of those ^ | |
| 16:21:29 | dansmith | mriedem: 2019-08-28 16:12:53.222764 | primary | 2019-08-28 16:12:53.222 | Missing value auth-url required for auth plugin password | |
| 16:21:37 | dansmith | mriedem: on the grenade lm job | |
| 16:21:49 | dansmith | mriedem: the regular one failed with node_failure | |
| 16:29:48 | melwitt | thanks dansmith++ | |
| 16:33:40 | artom | mriedem, dansmith, was on calls in the car all morning, catching up on scrollback | |
| 16:33:49 | artom | mriedem, crap, forgot about that starlingx meeting thing :( | |
| 16:33:57 | artom | Thanks for reminding me | |
| 16:35:38 | artom | (And right now I need to drive daughter to school, so I need to run again) | |
| 16:35:56 | artom | Though I'll try to respond to dansmith on that initial claims patch before I do so | |
| 16:36:09 | dansmith | artom: we should try to get that test patch running again first | |
| 16:36:13 | dansmith | I'm guessing we | |
| 16:36:19 | dansmith | just need to source openrc first somewhere | |
| 16:38:41 | artom | dansmith, yeah, I need to look at that too - sean-k-mooney seemed to also have ideas | |
| 16:38:55 | dansmith | gI think I got it | |
| 16:39:10 | openstackgerrit | Dan Smith proposed openstack/nova master: DNM: Run LM integration tests with NUMA flavor https://review.opendev.org/678887 | |
| 16:39:28 | dansmith | pulled a couple lines from the ceph hook which use osc | |
| 16:39:52 | dansmith | artom: I think his ideas were about making it suitable for all time, and that's not the point of this :) | |
| 16:40:24 | artom | dansmith, heh, indeed, quick and dirty all the way :D | |
| 16:40:26 | sean-k-mooney | well i was just suggestin using the way i know works | |
| 16:40:39 | sean-k-mooney | but also yes they are the less hacky way of doing it | |
| 16:40:46 | sean-k-mooney | if you way works thats good enough for this | |
| 16:41:51 | dansmith | I wish we could control which job runs on a patch.. like if the commit message starts with "DNM" then zuul can -2 it and only run tests the commit message asks for or something | |
| 16:46:23 | artom | dansmith, https://review.opendev.org/#/c/635669/45 there, hope that answers is | |
| 16:46:25 | artom | *it | |
| 16:49:14 | dansmith | artom: yeah, although I'm not sure why we need the call out if the current ordering will work | |
| 16:49:24 | dansmith | especially if we plan to remove it from the sriov path and have it in here in the long term | |
| 16:49:26 | mriedem | dansmith: we could also make that DNM change run faster but commenting out all other jobs except the ones we want to run | |
| 16:49:40 | dansmith | mriedem said he would go through the hackery in there and see if he could stomach it | |
| 16:49:43 | dansmith | post-haircut Iguess | |
| 16:50:05 | mriedem_away | i got distracted with the DNM debugging earlier | |
| 16:50:13 | dansmith | that's fine, | |
| 16:50:16 | dansmith | I want to see that test run anyway | |
| 16:51:43 | artom | dansmith, I don't like depending on an ordering that's so far away, but maybe we could call it out in a comment in the compute manager, and lest _test_pci() do its thing | |
| 16:51:48 | artom | *let | |
| 16:52:11 | dansmith | artom: can you just remove the other accounting and rely on this one? | |
| 16:52:30 | dansmith | I mean, I don't want to go all into refactoring it, just curious | |