| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-04-12 | |||
| 16:37:09 | openstackgerrit | Eric Fried proposed openstack/nova-specs master: Updates to image-metadata-prefiltering spec https://review.openstack.org/652101 | |
| 16:53:23 | spatel | sean-k-mooney: hey!!! | |
| 16:53:31 | spatel | How are you doing sean!! | |
| 16:55:33 | spatel | Does Google cloud provide SR-IOV function if anyone aware of it? | |
| 17:11:26 | sean-k-mooney | spatel: am im not sure. i have not looked at gce much | |
| 17:12:00 | spatel | I don't think they provide SR-IOV because i am not seeing anywhere mentioned | |
| 17:12:00 | sean-k-mooney | spatel: i konw aws whas some offerenings that have gpus and fpgas so they might but not sure where google stands in that regard | |
| 17:12:03 | artom | It can be googled in 2 seconds, FWIW... https://stackoverflow.com/questions/33114144/does-google-compute-engine-offer-sr-iov-single-root-i-o-virtualization | |
| 17:12:10 | artom | ... and has nothing to do with OpenStack? | |
| 17:12:44 | spatel | Google using DPDK style enhance networking... if i am not wrong | |
| 17:13:14 | sean-k-mooney | artom: spatel has recently (last 6 months) moved form aws to runign an sriov based openstack cloud | |
| 17:13:35 | sean-k-mooney | artom: im guessing spatel you are interested in comparing it to a few other options | |
| 17:16:08 | melwitt | fried_rice: we've been talking about SchedulerReportClient whether it's heavy to construct, if you might have anything to add https://review.openstack.org/#/c/638073/24/nova/quota.py@1154 | |
| 17:19:49 | spatel | sean-k-mooney: artom we are using aws in many place and costing is killing us so looking for other cloud provide like Google or tencent etc.. | |
| 17:20:20 | spatel | We have requirement of high performance networking so exploring options related SR-IOV | |
| 17:21:05 | spatel | artom: i did read that post already but not sure if Google change something recently and someone aware here so i post question here | |
| 17:21:32 | spatel | That post is 4 year old also :( | |
| 17:25:39 | cdent | fried_rice: what you know of the state of https://bugs.launchpad.net/nova/+bug/1724633 . Still alive? | |
| 17:25:40 | openstack | Launchpad bug 1724633 in OpenStack Compute (nova) "AllocationCandidates.get_by_filters hits incorrectly when traits are split across the main RP and aggregates" [Medium,Confirmed] | |
| 18:13:04 | mriedem | who would have know that our actions api is not completely solid https://bugs.launchpad.net/nova/+bug/1824420 | |
| 18:13:06 | openstack | Launchpad bug 1824420 in OpenStack Compute (nova) "Live migration succeeds but instance-action-list still has unexpected Error status" [Undecided,Triaged] | |
| 18:13:07 | mriedem | *known | |
| 18:28:40 | eandersson | mriedem, if you have some time can you take a stab at backporting that patch? If not I can probably do it. | |
| 18:29:01 | eandersson | The allocation leak from resize/cold migration | |
| 18:29:36 | mriedem | oh yeah sec | |
| 18:33:19 | mriedem | eandersson: btw did you figure out where you had migration failures leaking allocations? | |
| 18:34:04 | eandersson | Yea - we are doing a lot of migration testing, and those are causing it. | |
| 18:34:19 | eandersson | Another person on our team are performing them. | |
| 18:34:22 | mriedem | what's actually failing in confirm resize? | |
| 18:34:29 | eandersson | Yes, we believe so. | |
| 18:34:48 | eandersson | He thought it was someone else, as the uuid logged didn't match his VMs. | |
| 18:34:59 | eandersson | But the uuids are actually migration uuids. | |
| 18:35:03 | mriedem | ok, good to know - confirm resize shouldn't really be failing like that, so you've got some other issue | |
| 18:35:20 | mriedem | but yeah fixing that leak is something we need (on rocky for you) | |
| 18:58:02 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/rocky: Error out migration when confirm_resize fails https://review.openstack.org/652127 | |
| 18:58:08 | mriedem | eandersson: ^ is the first half of that backport, i have to run to a school thing for my kid but with get the other patch when i'm back | |
| 18:58:14 | mriedem | *will | |
| 19:04:24 | lajoskatona | mriedem: Hi, regarding https://review.openstack.org/640600 & https://review.openstack.org/640601 do you have special reason to keep the workflow -1 on them? | |
| 19:22:21 | openstackgerrit | Merged openstack/nova master: Add --dry-run option to heal_allocations CLI https://review.openstack.org/651932 | |
| 19:54:32 | eandersson | Thanks a lot mriedem | |
| 20:16:10 | fried_rice | melwitt: ack, looking. Generally speaking, the heavy part is building the ksa adapter, which can be heavy or not so heavy depending how the admin set up the conf options. | |
| 20:18:31 | fried_rice | openstack: cdent: I don't know anything about bug 1724633 off the top, would have to try to repro. lmk if you need that. | |
| 20:18:32 | openstack | bug 1724633 in OpenStack Compute (nova) "AllocationCandidates.get_by_filters hits incorrectly when traits are split across the main RP and aggregates" [Medium,Confirmed] https://launchpad.net/bugs/1724633 | |
| 20:20:36 | sean-k-mooney | o/ | |
| 20:20:46 | sean-k-mooney | i have some good news and bad news | |
| 20:21:03 | sean-k-mooney | which would people like to hear frist :P | |
| 20:24:24 | sean-k-mooney | ill send an email to the list but we nolonger need nested virt to test hugepages,numa,cpu pinning or dpdk in the gate | |
| 20:24:56 | sean-k-mooney | i also found we apparently have a deadlock in the nova compute agent if we run it on python 3.7 | |
| 20:25:15 | sean-k-mooney | mriedem: ^ | |
| 20:26:29 | sean-k-mooney | im going to try and port my thidparty jobs upstream before teh summit but we should be able to replace the intel nfv ci with upstream gate jobs | |
| 20:26:47 | mriedem | we aren't testing on py37 so... | |
| 20:26:50 | mriedem | meh? | |
| 20:29:11 | sean-k-mooney | for now i guess | |
| 20:29:38 | sean-k-mooney | we seam to lockup the compute agent when we go to spawn an instance under py3.7 | |
| 20:29:46 | sean-k-mooney | no errors or warnings | |
| 20:30:37 | sean-k-mooney | but anyway if we use fedora 29 + the virt preveiw repo we can get libvirt 5.2 and a new enough qemu that allows cpu pinnign to work without kvm | |
| 20:30:58 | fried_rice | melwitt: responded | |
| 20:31:20 | sean-k-mooney | so i shoudl be able to pull together a gate job that will fully test cpu pinning and hugepages | |
| 20:31:23 | mriedem | fedora 29 + virt preview with bleeding edge libvirt/qemu sounds super stable | |
| 20:31:40 | sean-k-mooney | ya i hate useing fedroa for ci or dev | |
| 20:31:46 | openstackgerrit | Julia Kreger proposed openstack/nova master: [ironic] ignore conflict upon cleaning up node https://review.openstack.org/643484 | |
| 20:31:47 | mriedem | i guess if it's just a smaller set of tests in a non-voting job, sure | |
| 20:31:51 | mriedem | better than 0 coverage | |
| 20:32:00 | sean-k-mooney | we jsut need the new libvirt really | |
| 20:32:11 | sean-k-mooney | i have a devstack plugin that can compile it form source | |
| 20:32:14 | sean-k-mooney | and qemu | |
| 20:32:23 | fried_rice | tjgresha: this conversation may interest you ^ | |
| 20:32:33 | sean-k-mooney | so i caould also use that do it with ubuntu or centos | |
| 20:32:58 | fried_rice | sean-k-mooney: I think there's work in progress to resurrect nfv CI, so if you think we don't need it at all, that'd be a good thing to know. | |
| 20:33:26 | sean-k-mooney | https://github.com/openstack/devstack-plugin-libvirt-qemu | |
| 20:34:04 | sean-k-mooney | yes i heard | |
| 20:34:29 | sean-k-mooney | we currently dont have vm with multiple numa nodes in the gate | |
| 20:34:41 | sean-k-mooney | which is onething the nfv ci had | |
| 20:35:10 | sean-k-mooney | we coudl maybe ask operators to donate such a nodeset | |
| 20:35:38 | mriedem | that's what the openlab request from artom was fore | |
| 20:35:40 | mriedem | *for | |
| 20:35:56 | mriedem | but that would definitely be a periodic queue kind of thing | |
| 20:36:06 | mriedem | or on-demand at best | |
| 20:36:20 | artom | And also, if we get nested virt, do we need multiple physical NUMA nodes? | |
| 20:36:42 | artom | mriedem, yeah, but better than nothing | |
| 20:37:09 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/rocky: Error out migration when confirm_resize fails https://review.openstack.org/652127 | |
| 20:37:10 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/rocky: Delete allocations even if _confirm_resize raises https://review.openstack.org/652146 | |
| 20:37:19 | mriedem | eandersson: ^ that should do it | |
| 20:37:54 | sean-k-mooney | artom: about 10 | |
| 20:38:01 | sean-k-mooney | we started wtih 3 | |
| 20:38:07 | sean-k-mooney | then grew to 5 | |
| 20:38:11 | artom | sean-k-mooney, ah, that makes more sense | |
| 20:38:23 | mriedem | speaking of ci, the ibm zkvm ci should be shutdown | |
| 20:38:25 | artom | So clearly the 1 or 2 machines we might get from openlab won't handle it ;) | |
| 20:38:29 | mriedem | fails immediately | |
| 20:38:49 | sean-k-mooney | artom: actully it coudl depending on how you set it up | |
| 20:38:55 | artom | sean-k-mooney, remind me why need multiple physical NUMA nodes? I recall having this conversation with you before... | |
| 20:39:08 | artom | Can't the all be nested virt multinode? | |
| 20:39:11 | artom | *they all | |
| 20:39:52 | sean-k-mooney | we coudl if i was allowed to change the libvirt dirver code but everytime i suggest it people complain | |
| 20:40:05 | artom | Eh? | |
| 20:40:33 | sean-k-mooney | in the libvirt driver we implemented it in such a way as to always map virtual numa nodes to different phyical numa nodes | |
| 20:40:52 | sean-k-mooney | but we never ever said that was guarenteed | |
| 20:41:08 | sean-k-mooney | but it was someing people observed and started to depend on | |
| 20:41:14 | artom | Oh right, that was the thing. If this was pure libvirt/kvm we could | |