| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-08-28 | |||
| 14:22:28 | mriedem | stephenfin: replied on https://review.opendev.org/#/c/637605/37 | |
| 14:22:57 | mriedem | and up, thanks for hitting those | |
| 14:23:29 | dansmith | mriedem: you wanna look at artom's latest changes on this per our discussion yesterday and opine on whether or not you can tolerate the hackery? https://review.opendev.org/#/c/635669/45 | |
| 14:24:14 | stephenfin | Ah, I meant would it make sense to add a 'should_raise' parameter to the 'delete' function (i.e. make it optional). It was a nit though | |
| 14:24:26 | stephenfin | should_raise or something along those lines | |
| 14:26:00 | mriedem | dansmith: sure | |
| 14:29:05 | dansmith | artom: left some clarifying questions in there, btw | |
| 14:32:53 | dansmith | artom: have you looked at why that DNM tempest test failed the live migration job? | |
| 14:33:02 | dansmith | looks like during setup, so maybe just something in the script | |
| 14:33:59 | dansmith | er, no | |
| 14:40:17 | mriedem | https://storage.bhs1.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/logs_87/678887/6/check/nova-live-migration/d3d4357/ara-report/result/092050bb-201e-4657-ac3f-16aebd4578ff/ failed but not sure why yet | |
| 14:40:44 | mriedem | oh the job timed out | |
| 14:40:45 | mriedem | /home/zuul/workspace/devstack-gate/functions.sh: line 1018: 5091 Killed timeout -s 9 ${REMAINING_TIME}m bash -c "source $WORKSPACE/devstack-gate/functions.sh && $cmd" | |
| 14:41:14 | dansmith | yeah, but I figure it might be because it was polling for that to go active or something | |
| 14:43:36 | dansmith | looks like it started building an instance right around the on the subnode | |
| 14:44:27 | mriedem | the funny thing is, | |
| 14:44:34 | mriedem | it doesn't even seem to run the live migration stuff | |
| 14:44:41 | mriedem | it goes right to the test_evacuate.sh script | |
| 14:45:07 | dansmith | yeah | |
| 14:45:43 | dansmith | I didn't look to see the ordering | |
| 14:46:10 | mriedem | 2019-08-28 04:37:56.230 | 1. test with all local storage (use default for volumes) 2019-08-28 04:37:56.232 | NOTE: test_volume_backed_live_migration is skipped due to https://bugs.launchpad.net/nova/+bug/1524898 2019-08-28 04:37:57.817 | Missing value auth-url required for auth plugin password 2019-08-28 04:37:57.909 | + environment:post_test_hook:L1: /opt/stack/new/nova/gate/test_evacuate.sh | |
| 14:46:11 | openstack | Launchpad bug 1524898 in OpenStack Compute (nova) "Volume based live migration aborted unexpectedly" [High,In progress] - Assigned to Lee Yarwood (lyarwood) | |
| 14:46:16 | mriedem | maybe the flavor creation is blowing up? | |
| 14:46:52 | dansmith | ah | |
| 14:47:37 | mriedem | we're losing the tracing too | |
| 14:47:38 | dansmith | I still wanna know why this instance didn't build in case it is related to tweaking the claim stuff, | |
| 14:47:41 | mriedem | from one of the sourced scripts | |
| 14:47:45 | dansmith | but it seems like it's waiting for the neutron event | |
| 14:48:13 | dansmith | it's still in building state at the end of the run | |
| 14:48:55 | dansmith | ah, artom said something about that early this morning.. | |
| 14:49:17 | mriedem | instance c2b876e6-b616-4259-b03b-f0b2bd3d8908 right? | |
| 14:49:25 | dansmith | yup | |
| 14:50:46 | mriedem | yup start waiting for the event here https://zuul.opendev.org/t/openstack/build/d3d4357c13be452fa8e64a6b612c256d/log/logs/subnode-2/screen-n-cpu.txt.gz#1905 | |
| 14:50:55 | mriedem | Aug 28 04:38:17.325822 ubuntu-bionic-vexxhost-vexxhost-sjc1-0010477321 nova-compute[24765]: DEBUG nova.compute.manager [None req-4d7771be-f2e2-429d-87c7-1cd4469cede5 demo admin] [instance: c2b876e6-b616-4259-b03b-f0b2bd3d8908] Preparing to wait for external event network-vif-plugged-afd502d7-a21d-48f2-8607-881a86f0781f {{(pid=24765) prepare_for_instance_event /opt/stack/new/nova/nova/compute/manager.py:329}} | |
| 14:51:52 | mriedem | got the event here | |
| 14:51:53 | mriedem | https://zuul.opendev.org/t/openstack/build/d3d4357c13be452fa8e64a6b612c256d/log/logs/subnode-2/screen-n-cpu.txt.gz#1941 | |
| 14:51:58 | mriedem | Aug 28 04:38:21.643105 ubuntu-bionic-vexxhost-vexxhost-sjc1-0010477321 nova-compute[24765]: DEBUG nova.compute.manager [req-61851260-be42-4748-9d79-4b0ec5c08ed6 req-3c63a866-a638-4ec7-9f56-3a7aeb2e9958 service nova] [instance: c2b876e6-b616-4259-b03b-f0b2bd3d8908] Received event network-vif-plugged-afd502d7-a21d-48f2-8607-881a86f0781f {{(pid=24765) external_instance_event /opt/stack/new/nova/nova/compute/manager.py:8819}} | |
| 14:52:03 | mriedem | 4 seconds later | |
| 14:52:34 | dansmith | presumably unrelated to his change though if we've made it to the neutron event stage | |
| 14:54:22 | mriedem | i think during artom's revert resize ovs hybrid plug race thing that evacuate could have the same kind of race b/c of the time that we trigger the event by updating the port binding and when the driver's spawn method start waiting for it | |
| 14:55:30 | mriedem | although, if we didn't have a handler registered for the event callback we'd see "Received unexpected event" in the logs | |
| 14:55:31 | mriedem | which we don't | |
| 14:55:37 | mriedem | so idk what's going on there | |
| 14:56:46 | mriedem | the guest power state goes from paused > running > paused agin | |
| 14:56:47 | mriedem | *agian | |
| 14:56:49 | mriedem | gah | |
| 14:57:10 | mriedem | but still, i'm wondering why the run_tests script didn't actually run any live migration tests | |
| 14:58:12 | kashyap | aspiers: When you get a sec, summarized our discussion from #virt (OFTC) here, for other Stackers: https://review.opendev.org/#/c/673790/ | |
| 15:01:29 | openstackgerrit | Matt Riedemann proposed openstack/nova master: DNM: Run LM integration tests with NUMA flavor https://review.opendev.org/678887 | |
| 15:02:24 | mriedem | dansmith: artom: let's see if ^ helps | |
| 15:02:52 | dansmith | cool | |
| 15:05:03 | melwitt | mriedem: follow up patch is here https://review.opendev.org/678988 | |
| 15:07:35 | melwitt | dansmith: the multi-cell archive_deleted_rows set (this patch and the one on top of it) have a +2 from mriedem, if you would fancy taking a look please | |
| 15:07:40 | melwitt | https://review.opendev.org/507486 | |
| 15:08:02 | mriedem | got it | |
| 15:09:23 | melwitt | danke | |
| 15:14:33 | dansmith | melwitt: I know, I've had it open just have to get to it | |
| 15:15:21 | melwitt | ok, thanks. sorry didn't know you already knew | |
| 15:15:43 | dansmith | yeah, I opened the tab when mriedem pinged us yesterday | |
| 15:18:00 | melwitt | k | |
| 15:22:27 | mriedem | artom: did you ever follow up with getting something on the starlingx weekly meeting about testing the numa LM series? | |
| 15:30:33 | mriedem | just realized that we have 41 approved blueprints for train with only 12 complete and 2 weeks to FF | |
| 15:31:57 | mriedem | probably time to start deferring things that we know aren't going to land | |
| 15:34:25 | donnyd | so yesterdays test of mounting _base on a shared location didn't go well | |
| 15:35:46 | donnyd | for some unknown reason qemu-img info reports back that the cached image could not be found | |
| 16:00:21 | mriedem | stephenfin: can we effectively mark this complete? https://blueprints.launchpad.net/nova/+spec/remove-cells-v1 | |
| 16:00:29 | mriedem | there are only 2 open changes https://review.opendev.org/#/q/topic:bp/remove-cells-v1+(status:open) | |
| 16:00:54 | openstackgerrit | Merged openstack/nova master: Specify availability_zone to unshelve https://review.opendev.org/663851 | |
| 16:00:58 | stephenfin | Yup, it's complete. Those two are very tricky post-removal cleanup patches | |
| 16:01:05 | stephenfin | I'll come back to them when I can | |
| 16:03:41 | cdent | cells v1 is gone? wooooooo! | |
| 16:03:42 | mriedem | melwitt: would be sweet if we can get https://review.opendev.org/#/c/640898/ approved by the end of the week - it's 2 weeks to FF and it'd be nice to land that series in time for train | |
| 16:04:46 | melwitt | mriedem: indeed, I'm gonna update it after this last call I'm on. thanks for reminding | |
| 16:09:01 | dansmith | mriedem: are you pretty confident on this archive multi cells thing? I've skimmed the logic and tests and just can't really get myself to go deep on it | |
| 16:09:40 | dansmith | seems pretty low risk even if there is something wrong in here, so I'm tempted to just +W based on my skimming | |
| 16:09:52 | dansmith | s/seems/should be/ | |
| 16:12:32 | openstackgerrit | Merged openstack/nova master: scheduler: Flatten 'ResourceRequest.from_extra_specs', 'from_image_props' https://review.opendev.org/674894 | |
| 16:12:41 | openstackgerrit | Merged openstack/nova master: Remove 'hw:cpu_policy', 'hw:mem_page_size' extra specs from API samples https://review.opendev.org/675338 | |
| 16:12:57 | mriedem | dansmith: bombs away | |
| 16:13:05 | mriedem | and yeah i've gone over it several times in detail now | |
| 16:13:09 | mriedem | probably annoying detail for melwitt | |
| 16:13:21 | mriedem | the nova-next change on top also shows it working as expected | |
| 16:15:11 | dansmith | oh sweet I didn't even notice | |
| 16:16:21 | dansmith | got those as well | |
| 16:19:05 | openstackgerrit | Stephen Finucane proposed openstack/nova master: objects: Add online migration for legacy NUMA objects https://review.opendev.org/537414 | |
| 16:19:06 | openstackgerrit | Stephen Finucane proposed openstack/nova master: trivial: Rename 'nova.tests.unit.test_nova_manage' https://review.opendev.org/679095 | |
| 16:19:06 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Ensure online migrations have a unique name https://review.opendev.org/679096 | |
| 16:19:37 | stephenfin | mriedem: You'll probably be interested in the last of those ^ | |
| 16:21:29 | dansmith | mriedem: 2019-08-28 16:12:53.222764 | primary | 2019-08-28 16:12:53.222 | Missing value auth-url required for auth plugin password | |
| 16:21:37 | dansmith | mriedem: on the grenade lm job | |
| 16:21:49 | dansmith | mriedem: the regular one failed with node_failure | |
| 16:29:48 | melwitt | thanks dansmith++ | |
| 16:33:40 | artom | mriedem, dansmith, was on calls in the car all morning, catching up on scrollback | |
| 16:33:49 | artom | mriedem, crap, forgot about that starlingx meeting thing :( | |
| 16:33:57 | artom | Thanks for reminding me | |
| 16:35:38 | artom | (And right now I need to drive daughter to school, so I need to run again) | |
| 16:35:56 | artom | Though I'll try to respond to dansmith on that initial claims patch before I do so | |
| 16:36:09 | dansmith | artom: we should try to get that test patch running again first | |
| 16:36:13 | dansmith | I'm guessing we | |
| 16:36:19 | dansmith | just need to source openrc first somewhere | |