Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-04-12
18:29:36 mriedem oh yeah sec
18:33:19 mriedem eandersson: btw did you figure out where you had migration failures leaking allocations?
18:34:04 eandersson Yea - we are doing a lot of migration testing, and those are causing it.
18:34:19 eandersson Another person on our team are performing them.
18:34:22 mriedem what's actually failing in confirm resize?
18:34:29 eandersson Yes, we believe so.
18:34:48 eandersson He thought it was someone else, as the uuid logged didn't match his VMs.
18:34:59 eandersson But the uuids are actually migration uuids.
18:35:03 mriedem ok, good to know - confirm resize shouldn't really be failing like that, so you've got some other issue
18:35:20 mriedem but yeah fixing that leak is something we need (on rocky for you)
18:58:02 openstackgerrit Matt Riedemann proposed openstack/nova stable/rocky: Error out migration when confirm_resize fails https://review.openstack.org/652127
18:58:08 mriedem eandersson: ^ is the first half of that backport, i have to run to a school thing for my kid but with get the other patch when i'm back
18:58:14 mriedem *will
19:04:24 lajoskatona mriedem: Hi, regarding https://review.openstack.org/640600 & https://review.openstack.org/640601 do you have special reason to keep the workflow -1 on them?
19:22:21 openstackgerrit Merged openstack/nova master: Add --dry-run option to heal_allocations CLI https://review.openstack.org/651932
19:54:32 eandersson Thanks a lot mriedem
20:16:10 fried_rice melwitt: ack, looking. Generally speaking, the heavy part is building the ksa adapter, which can be heavy or not so heavy depending how the admin set up the conf options.
20:18:31 fried_rice openstack: cdent: I don't know anything about bug 1724633 off the top, would have to try to repro. lmk if you need that.
20:18:32 openstack bug 1724633 in OpenStack Compute (nova) "AllocationCandidates.get_by_filters hits incorrectly when traits are split across the main RP and aggregates" [Medium,Confirmed] https://launchpad.net/bugs/1724633
20:20:36 sean-k-mooney o/
20:20:46 sean-k-mooney i have some good news and bad news
20:21:03 sean-k-mooney which would people like to hear frist :P
20:24:24 sean-k-mooney ill send an email to the list but we nolonger need nested virt to test hugepages,numa,cpu pinning or dpdk in the gate
20:24:56 sean-k-mooney i also found we apparently have a deadlock in the nova compute agent if we run it on python 3.7
20:25:15 sean-k-mooney mriedem: ^
20:26:29 sean-k-mooney im going to try and port my thidparty jobs upstream before teh summit but we should be able to replace the intel nfv ci with upstream gate jobs
20:26:47 mriedem we aren't testing on py37 so...
20:26:50 mriedem meh?
20:29:11 sean-k-mooney for now i guess
20:29:38 sean-k-mooney we seam to lockup the compute agent when we go to spawn an instance under py3.7
20:29:46 sean-k-mooney no errors or warnings
20:30:37 sean-k-mooney but anyway if we use fedora 29 + the virt preveiw repo we can get libvirt 5.2 and a new enough qemu that allows cpu pinnign to work without kvm
20:30:58 fried_rice melwitt: responded
20:31:20 sean-k-mooney so i shoudl be able to pull together a gate job that will fully test cpu pinning and hugepages
20:31:23 mriedem fedora 29 + virt preview with bleeding edge libvirt/qemu sounds super stable
20:31:40 sean-k-mooney ya i hate useing fedroa for ci or dev
20:31:46 openstackgerrit Julia Kreger proposed openstack/nova master: [ironic] ignore conflict upon cleaning up node https://review.openstack.org/643484
20:31:47 mriedem i guess if it's just a smaller set of tests in a non-voting job, sure
20:31:51 mriedem better than 0 coverage
20:32:00 sean-k-mooney we jsut need the new libvirt really
20:32:11 sean-k-mooney i have a devstack plugin that can compile it form source
20:32:14 sean-k-mooney and qemu
20:32:23 fried_rice tjgresha: this conversation may interest you ^
20:32:33 sean-k-mooney so i caould also use that do it with ubuntu or centos
20:32:58 fried_rice sean-k-mooney: I think there's work in progress to resurrect nfv CI, so if you think we don't need it at all, that'd be a good thing to know.
20:33:26 sean-k-mooney https://github.com/openstack/devstack-plugin-libvirt-qemu
20:34:04 sean-k-mooney yes i heard
20:34:29 sean-k-mooney we currently dont have vm with multiple numa nodes in the gate
20:34:41 sean-k-mooney which is onething the nfv ci had
20:35:10 sean-k-mooney we coudl maybe ask operators to donate such a nodeset
20:35:38 mriedem that's what the openlab request from artom was fore
20:35:40 mriedem *for
20:35:56 mriedem but that would definitely be a periodic queue kind of thing
20:36:06 mriedem or on-demand at best
20:36:20 artom And also, if we get nested virt, do we need multiple physical NUMA nodes?
20:36:42 artom mriedem, yeah, but better than nothing
20:37:09 openstackgerrit Matt Riedemann proposed openstack/nova stable/rocky: Error out migration when confirm_resize fails https://review.openstack.org/652127
20:37:10 openstackgerrit Matt Riedemann proposed openstack/nova stable/rocky: Delete allocations even if _confirm_resize raises https://review.openstack.org/652146
20:37:19 mriedem eandersson: ^ that should do it
20:37:54 sean-k-mooney artom: about 10
20:38:01 sean-k-mooney we started wtih 3
20:38:07 sean-k-mooney then grew to 5
20:38:11 artom sean-k-mooney, ah, that makes more sense
20:38:23 mriedem speaking of ci, the ibm zkvm ci should be shutdown
20:38:25 artom So clearly the 1 or 2 machines we might get from openlab won't handle it ;)
20:38:29 mriedem fails immediately
20:38:49 sean-k-mooney artom: actully it coudl depending on how you set it up
20:38:55 artom sean-k-mooney, remind me why need multiple physical NUMA nodes? I recall having this conversation with you before...
20:39:08 artom Can't the all be nested virt multinode?
20:39:11 artom *they all
20:39:52 sean-k-mooney we coudl if i was allowed to change the libvirt dirver code but everytime i suggest it people complain
20:40:05 artom Eh?
20:40:33 sean-k-mooney in the libvirt driver we implemented it in such a way as to always map virtual numa nodes to different phyical numa nodes
20:40:52 sean-k-mooney but we never ever said that was guarenteed
20:41:08 sean-k-mooney but it was someing people observed and started to depend on
20:41:14 artom Oh right, that was the thing. If this was pure libvirt/kvm we could
20:41:26 artom But because our dsvm will be servers in an openstack cloud
20:41:26 sean-k-mooney yes and we could in nova
20:41:34 artom We're limited by that thing
20:41:44 sean-k-mooney this is purly a nova limitation
20:42:19 sean-k-mooney we coudl cange the code however
20:42:30 artom Seems like a lot of work :)
20:42:44 sean-k-mooney if you use multipl virtual numa nodes on hyperv i dont think they give the same guarentee
20:42:45 melwitt fried_rice: thanks
20:42:46 artom No reason we can't push the openlab thing as far as it'll go first
20:43:00 mriedem cfriesen: how much does this matter to you guys? https://bugs.launchpad.net/nova/+bug/1824420
20:43:01 openstack Launchpad bug 1824420 in OpenStack Compute (nova) "Live migration succeeds but instance-action-list still has unexpected Error status" [Undecided,Triaged]
20:43:36 sean-k-mooney artom: you know after i get a gate job working i might actully put up a patch to make the numa maping configurable
20:44:23 sean-k-mooney i at least want to see how invaisge it woudl be to allow it. i thnk it would be a very minor change but that code is ...
20:44:46 artom Doesn't feel very minor, but you probably know that code better than I do
20:45:26 sean-k-mooney its a slight tweek to how we iterate over the posible permutaion in the hardware.py module when we fit an instance to the host
20:46:10 sean-k-mooney currently wehn we alocate a instance numa node to a host cell we dont consider that cell again for subsequent guest numa nodes
20:54:16 mriedem melwitt: dansmith: please to be reviewing these backports https://review.openstack.org/#/q/topic:bug/1669054+branch:stable/queens
20:56:17 openstackgerrit Matt Riedemann proposed openstack/nova stable/queens: Error out migration when confirm_resize fails https://review.openstack.org/652150
20:56:36 dansmith feel like I've reviewed these so many times
20:57:14 openstackgerrit François Palin proposed openstack/nova master: nova diagnostics command is not working with all interfaces https://review.openstack.org/648123
20:59:28 dansmith mriedem: shall I slam these?
21:03:32 melwitt I can review too
21:03:46 dansmith they're ready for +W
21:04:00 melwitt coolio

Earlier   Later