| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-02-06 | |||
| 17:35:48 | bauzas | since we set the right value in serial if that's not already in there | |
| 17:35:52 | mriedem | to try and normalize a lot of that stuff | |
| 17:36:08 | mriedem | but, i didn't pursue it | |
| 17:36:15 | bauzas | your fault, then | |
| 17:36:17 | bauzas | boooo | |
| 17:36:23 | mriedem | sort of like the MigrateData thing that dansmith did for live migration | |
| 17:36:33 | bauzas | yup | |
| 17:36:39 | mriedem | yes it's my fault, sadly enough, | |
| 17:36:48 | mriedem | it would have also meant converting like 80 volume drivers over to that new model... | |
| 17:37:48 | bauzas | :) | |
| 17:38:43 | ildikov | yeah, this comes up every half year and we note that harmonize the drivers is just not likely to happen | |
| 17:39:11 | ildikov | or you end up with a free-form part where most of the info sits anyway... | |
| 17:43:53 | bauzas | mriedem: I haven't seen https://etherpad.openstack.org/p/nova-aggregate-filter-allocation-ratio-snafu somehow planned for the PTG in the PTG etherpad | |
| 17:44:02 | bauzas | mriedem: ack if I'm adding it? | |
| 17:45:55 | mriedem | jaypipes had a thing in the ptg etherpad for this issue, | |
| 17:45:57 | mriedem | i'll link it in | |
| 17:46:25 | mriedem | starts at L34 in https://etherpad.openstack.org/p/nova-ptg-rocky | |
| 17:48:44 | bauzas | cool | |
| 17:50:00 | mriedem | it's enchilada time! | |
| 17:53:38 | dansmith | mriedem: remind me what the legacy-tempest job is? | |
| 17:53:53 | dansmith | it's failing my service version patches for seemingly related reasons, but the regular job is fine | |
| 17:54:51 | mriedem | hmm, legacy-tempest shouldn't be running on our stuff | |
| 17:54:54 | mriedem | i don't think | |
| 17:55:17 | mriedem | that was replaced by tempest-full | |
| 17:55:30 | dansmith | legacy-tempest-dsvm-py35FAILURE in 1h 14m 39s | |
| 17:55:33 | dansmith | on one of my patches | |
| 17:55:39 | dansmith | https://review.openstack.org/#/c/539590/ | |
| 17:55:43 | mriedem | oh that's different | |
| 17:55:56 | mriedem | it's just tempest with devstack on py35 | |
| 17:56:15 | dansmith | tbachman: no | |
| 17:56:19 | dansmith | mriedem: hmm, wonder why it failed | |
| 17:56:27 | tbachman | dansmith: thx | |
| 17:56:29 | mriedem | dansmith: http://logs.openstack.org/90/539590/2/check/legacy-tempest-dsvm-py35/79137ee/job-output.txt.gz#_2018-01-31_19_33_29_779861 | |
| 17:56:40 | mriedem | likely something in the reserve_block_device_name rpc call? | |
| 17:56:49 | dansmith | mriedem: right, hence my "seemingly related reason" | |
| 17:57:00 | dansmith | no, it's because we check for service version before allowing tags | |
| 17:57:06 | dansmith | which is what that patch is doing | |
| 17:57:17 | dansmith | s/doing/changing/ | |
| 17:57:21 | dansmith | s/changing/touching/ | |
| 17:57:43 | bauzas | mriedem: mgagne: just added a thought on a possible workaround for https://review.openstack.org/#/c/541018/ | |
| 17:58:04 | bauzas | mriedem: mgagne: tl;dr just set allocation ratios on every compute to be 99999.0 | |
| 17:58:17 | sean-k-mooney | cfriesen: stephenfin yes numa support is off by default in dpdk so that libnuma can remain an optional dependecy not a required depency of dpdk | |
| 17:58:26 | mriedem | dansmith: maybe "if result is" is different in py35 | |
| 17:58:33 | mriedem | i don't see much "foo is bar" | |
| 17:58:50 | bauzas | mriedem: mgagne: that will allow Placement to accept all the hosts, and just keep the filtering logic in the appropriate AggregateRAMFilter and so on | |
| 18:00:25 | dansmith | mriedem: um, which thing are you talking about? | |
| 18:00:26 | kashyap | I'm Neutron-challenged, can anyone tell at _which_ of these points is Nova waiting (by pausing) on Neutron to let it complete things? http://git.openstack.org/cgit/openstack/neutron/commit/?h=master&id=b7c303e ("Notify nova with network-vif-plugged in case of live migration") | |
| 18:00:29 | dansmith | the sentinel check has to be "is" | |
| 18:01:21 | kashyap | "during live migration on pre migration" is terribly unclear | |
| 18:02:52 | dansmith | kashyap: nova waits when it uses the virtapi wait_for_events context manager | |
| 18:02:53 | dansmith | it waits to exit the context until timeout or the event arrives | |
| 18:03:22 | kashyap | I see. | |
| 18:03:44 | kashyap | Is this what the Neutron events telling Nova: | |
| 18:04:00 | kashyap | "Hey, I've done with doing all the stuff I need to do, so I'm signalling that by sending out this event (from the above Neutron commit). Over to you now." | |
| 18:04:22 | kashyap | I.e. the destination Neutron has completed virtual interface plugging. | |
| 18:04:40 | dansmith | yes | |
| 18:04:54 | dansmith | I mean, that's what vif-plugged means | |
| 18:05:24 | kashyap | Noted. | |
| 18:05:31 | stvnoyes | mriedem: matt, I added a change to enable multiattach on the zfssa array, but when tempest runs during the zfssa ci, it's not running with multiattach enabled (in tempest.conf). Where would I make a change to have this enabled when running the Oracle ZFSSA CI? thanks... | |
| 18:05:43 | sean-k-mooney | dansmith: actully vif-plugged has different menanings | |
| 18:06:13 | sean-k-mooney | vif-plugged in the nova-compute agent log means os-vif finisshed plugging the interface | |
| 18:06:33 | sean-k-mooney | vif-plugged event form neutron means neutron finished wireing it up | |
| 18:06:54 | dansmith | sean-k-mooney: we're talking about when neutron does a POST of the event to nova's external event interface | |
| 18:07:27 | dansmith | and it better (still) mean that the wiring is done, else we have a problem :) | |
| 18:07:32 | mriedem | stvnoyes: do you have these set in devstack? https://github.com/openstack/nova/blob/master/playbooks/legacy/nova-multiattach/run.yaml#L31 | |
| 18:07:46 | sean-k-mooney | dansmith: ah then yes that is sent when neutron finishes wiring up the port that was plugged by nova/os-vif/libvirt into the network backend so your correct in what you said | |
| 18:08:17 | mriedem | stvnoyes: also, is tempest running against master nova so the 2.60 microversion is available? | |
| 18:08:37 | dansmith | sean-k-mooney: thanks :) | |
| 18:09:16 | kashyap | sean-k-mooney: Hey, I'm about to write some questions on this change: https://review.openstack.org/#/c/497457/18 | |
| 18:09:26 | kashyap | sean-k-mooney: Post some design discussion w/ QEMU & libvirt folks | |
| 18:09:39 | kashyap | Would appreciate if you also look at it | |
| 18:10:24 | sean-k-mooney | kashyap: am i am behind in some internal paper work but ill try and take a look tomorow | |
| 18:10:37 | kashyap | sean-k-mooney: Sure, tomm is fine; I'm already late for dinner by an hour | |
| 18:10:55 | kashyap | Just want to 'commit this to paper' before my "cache gets flushed" | |
| 18:11:12 | kashyap | As I've spent the last 2 hours in this rabbit hole :P | |
| 18:11:33 | mriedem | efried: on that service user / CONF.glance.api_servers bug https://bugs.launchpad.net/nova/+bug/1747511 - it's definitely related to using a service user; i removed that code and now the nova-next job passes | |
| 18:11:35 | openstack | Launchpad bug 1747511 in OpenStack Compute (nova) "legacy-tempest-dsvm-neutron-nova-next-full fails with "Can not find requested image" since 1/31" [Undecided,New] | |
| 18:12:04 | stvnoyes | mriedem: i didn't make any changes to the CI devstack (although I can run locally ok). I believe the CI is against nova master, but will check. The zfssa CI test is running for this review - https://review.openstack.org/#/c/541368/ | |
| 18:12:54 | mriedem | stvnoyes: so this job result right? https://openstackci.oracle.com/zfssa_ci/refs-changes-68-541368-1-iscsi/ | |
| 18:13:13 | stvnoyes | mriedem: yes. I'm seeing tempest.api.compute.volumes.test_attach_volume.AttachVolumeMultiAttachTest) ... SKIPPED: Volume multi-attach is not available. | |
| 18:13:43 | mriedem | 2018-02-06 09:16:16.276 12703 DEBUG tempest [-] compute-feature-enabled.volume_multiattach = False log_opt_values /usr/local/lib/python2.7/dist-packages/oslo_config/cfg.py:2898 | |
| 18:13:43 | mriedem | stvnoyes: that's b/c it's not set in tempest.conf | |
| 18:13:47 | mriedem | https://openstackci.oracle.com/zfssa_ci/refs-changes-68-541368-1-iscsi/tempest.log | |
| 18:13:56 | openstackgerrit | Dan Smith proposed openstack/nova master: Make host_manager use scatter-gather and ignore down cells https://review.openstack.org/539617 | |
| 18:13:56 | openstackgerrit | Dan Smith proposed openstack/nova master: Make service all-cells min version helper use scatter-gather https://review.openstack.org/539590 | |
| 18:13:57 | openstackgerrit | Patricia Domingues proposed openstack/nova master: load up the volume drivers by checking architecture https://review.openstack.org/541393 | |
| 18:14:07 | mriedem | stvnoyes: where is the code that controls the local.conf for the devstack setup in this CI? | |
| 18:14:13 | mriedem | is it internal to oracle? or on github? | |
| 18:15:40 | stvnoyes | mriedem: good question. idk. I haven't played with CI stuff so all this is pretty new to me. I will see if there's someone around on the zfssa team who would know. | |
| 18:16:25 | mriedem | stvnoyes: https://wiki.openstack.org/wiki/ThirdPartySystems/Oracle_ZFSSA_CI | |
| 18:16:36 | odyssey4me | howdy all - we're having some trouble in the OSA deployment test for upgrades from newton (without cells v2) to ocata (with cells v2), it looks like there's some weird state which has been hit where the compute registration isn't working right any more. I'm seeing this: https://pastebin.com/C9ji6vdS | |
| 18:17:04 | odyssey4me | Our last working test was the last patch we did which was in early December. | |
| 18:17:25 | odyssey4me | Any idea what may have changed to cause the previously working implementation to no longer work? | |
| 18:19:22 | mriedem | odyssey4me: i don't see anything related to that in https://github.com/openstack/nova/commits/stable/ocata going back to november | |
| 18:19:29 | mriedem | are you sure it wasn't something that changed on the OSA side? | |
| 18:19:47 | odyssey4me | mriedem yep, very sure - see https://review.openstack.org/#/q/branch:stable/ocata+project:openstack/openstack-ansible-os_nova | |
| 18:21:27 | mriedem | https://github.com/openstack/nova/commits/stable/ocata/nova/cmd/manage.py | |
| 18:21:37 | mriedem | oldest thing in stable there is from october | |
| 18:22:14 | odyssey4me | yeah, I'm wondering if something has perhaps changed in the database tables either in newton (the source), or in ocata (the destination) | |
| 18:22:40 | mriedem | have you verified if the instances you're verifying actually exist? | |