| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-05-24 | |||
| 17:59:31 | artom | aspiers, ¯\_(ツ)_/¯ | |
| 17:59:37 | aspiers | :) | |
| 17:59:43 | aspiers | mriedem: thanks | |
| 17:59:47 | mriedem | the out of tree drivers are maintained by vendors that push more features into them than what's in the in-tree versions | |
| 17:59:51 | aspiers | gotcha | |
| 17:59:54 | mriedem | some aren't in tree at all, like lxd | |
| 17:59:58 | aspiers | thought it might be something like that | |
| 18:00:21 | aspiers | OK, so just randomly removing parameters from the ComputeDriver interface isn't cool | |
| 18:00:22 | mriedem | some we could probably deprecate from in-tree like xen, zvm and powervm | |
| 18:01:06 | mriedem | there are no guarantees on the virt driver interface since it's internal and not versoined | |
| 18:01:08 | mriedem | *versioned | |
| 18:01:24 | mriedem | we generally try to be nice and at least email to the list if we're changing an interface | |
| 18:01:28 | aspiers | right | |
| 18:02:03 | mriedem | also, yup, enabling n-novnc in the multi-cell job on 5/20 is when the test started failing https://github.com/openstack/nova/commit/c5b83c3fbca83726f4a956009e1788d26bcedde0#diff-7415f5ff7beee2cdf9ffe31e12e4c086 | |
| 18:02:07 | mriedem | melwitt: finucannot: ^ | |
| 18:02:12 | mriedem | i'm going to report a bug since i don't have time to dig in, | |
| 18:02:24 | mriedem | but that could put a damper on the remove-consoleauth shindig | |
| 18:02:38 | aspiers | well I suppose I could at least add a NOTE to the base class referencing https://opendev.org/openstack/nova/commit/8d97118be776fcaad3053d1f93f61d339685a4ae and https://opendev.org/openstack/nova/commit/f51526b596f3d89cda2ec4501e1 | |
| 18:03:07 | aspiers | to save anyone else from needing ancient archaelogy to understand why the parameter is there | |
| 18:03:12 | mriedem | i've walked backward into this conversation so i don't know what the context is | |
| 18:03:28 | aspiers | mriedem: I just noticed that the host parameter to ComputeDriver.init_host() is never used | |
| 18:03:33 | mriedem | i'd say we want the host arg in init_host | |
| 18:03:34 | aspiers | at least, not by anything in-tree | |
| 18:03:50 | mriedem | because we can use that for our fake virt driver to avoid global config | |
| 18:04:00 | mriedem | in fact i have a patch doing that i think | |
| 18:04:10 | aspiers | oh, I didn't think to check the fakevirt driver | |
| 18:04:25 | mriedem | CONF.host doesn't work in tests when you have multiple computes | |
| 18:04:36 | mriedem | https://review.opendev.org/#/c/656709/2/nova/virt/fake.py@149 | |
| 18:05:28 | artom | Wait, so we officially have a precedent for writing code *just* for CI to work? | |
| 18:05:51 | finucannot | mriedem: I'll stick that on my queue to investigate early next week | |
| 18:06:33 | aspiers | mriedem: OK, that's a bit over my head at this stage so I'll just defer to your judgement and not do anything | |
| 18:07:03 | mriedem | melwitt: stephenfin: https://bugs.launchpad.net/nova/+bug/1830417 | |
| 18:07:04 | openstack | Launchpad bug 1830417 in OpenStack Compute (nova) "NoVNCConsoleTestJSON.test_novnc fails in nova-multi-cell job since 5/20" [Medium,Confirmed] | |
| 18:07:07 | aspiers | at least the history is now in eavesdrop if we ever need it again | |
| 18:08:39 | artom | aspiers, the premier searchable, accessible, information repository ;) | |
| 18:08:49 | aspiers | artom ;-) | |
| 18:16:52 | mriedem | melwitt: in this case i don't see anything going on in the novnc service for cell2 http://logs.openstack.org/73/638073/31/check/nova-multi-cell/6ea3306/controller/logs/screen-n-novnc-cell2.txt.gz | |
| 18:16:55 | mriedem | and that's where the instance is | |
| 18:24:44 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Skip novnc tests in multi-cell job until bug 1830417 is fixed https://review.opendev.org/661371 | |
| 18:24:45 | openstack | bug 1830417 in OpenStack Compute (nova) "NoVNCConsoleTestJSON.test_novnc fails in nova-multi-cell job since 5/20" [Medium,Confirmed] https://launchpad.net/bugs/1830417 | |
| 18:24:53 | mriedem | efried: you said in the nova meeting yesterday that the multi-cell job was wonky, well, that's why ^ | |
| 18:26:51 | mriedem | oooo this is great http://logs.openstack.org/04/656304/1/check/nova-grenade-live-migration/083d5ec/logs/subnode-2/screen-n-cpu.txt.gz?level=TRACE#_May_24_16_55_08_219083 | |
| 18:26:56 | mriedem | May 24 16:55:08.219083 ubuntu-bionic-rax-ord-0006536941 nova-compute[867]: ERROR nova.virt.libvirt.driver [-] [instance: da0c957c-7a7d-4673-bd2d-0336d22f6fff] Live Migration failure: internal error: process exited while connecting to monitor: Failed to initialize module: /usr/lib/x86_64-linux-gnu/qemu/block-rbd.so May 24 16:55:08.219083 ubuntu-bionic-rax-ord-0006536941 nova-compute[867]: Note: only modules from the same build | |
| 18:26:57 | mriedem | be loaded. | |
| 18:36:22 | mriedem | i now realize there was no release note with the eventlet monkeypatch change in stable/stein https://review.opendev.org/#/c/647310/ | |
| 18:36:30 | mriedem | i wonder if we should have had a release note on that | |
| 18:36:44 | mriedem | to at least mention overriding that with OS_NOVA_DISABLE_EVENTLET_PATCHING | |
| 18:37:03 | melwitt | mriedem: gah, looking | |
| 18:37:43 | sean-k-mooney | mriedem: well OS_NOVA_DISABLE_EVENTLET_PATCHING is not ment to be a public thing for people to use | |
| 18:37:46 | mriedem | iow, because of bug 1829062 on stable/stein, should we have a release note for ^ | |
| 18:37:47 | openstack | bug 1829062 in StarlingX "nova placement api non-responsive due to eventlet error" [Critical,In progress] https://launchpad.net/bugs/1829062 - Assigned to Gerry Kopec (gerry-kopec) | |
| 18:37:53 | mriedem | sean-k-mooney: starlingx is using it | |
| 18:37:57 | sean-k-mooney | it was intended for use in tox | |
| 18:38:28 | sean-k-mooney | mriedem: well i know mdbooth never intended it to be used by anything in production | |
| 18:39:21 | mriedem | ok no reno it is | |
| 18:40:27 | sean-k-mooney | have we told starling x that they can work around the issue by seting the wsgi server threads=1 instead | |
| 18:40:44 | mriedem | i haven't | |
| 18:40:47 | mriedem | and there is nothing in that bug | |
| 18:40:59 | mriedem | as far as i know when you guys all talk about this every other week none of the results of the conversation are written down publicly | |
| 18:42:41 | sean-k-mooney | ok... ill try and write something up as a post to the mailing list on monday | |
| 18:47:56 | melwitt | mriedem: I think what's wrong is that the vnc proxy urls in the nova-cpu.conf files are both using port 6080 when one of them (the compute in cell2) should be using 6084 | |
| 18:48:05 | melwitt | http://logs.openstack.org/73/638073/31/check/nova-multi-cell/6ea3306/controller/logs/etc/nova/nova-cpu_conf.txt.gz vs http://logs.openstack.org/73/638073/31/check/nova-multi-cell/6ea3306/compute1/logs/etc/nova/nova-cpu_conf.txt.gz | |
| 18:48:59 | mriedem | ah yup | |
| 18:49:18 | melwitt | and from what I recall in devstack, I don't know how we can configure separate ones and use them in an easy way | |
| 18:49:53 | mriedem | we can pass that information into each node from the job config if we want to hard-code them | |
| 18:50:47 | melwitt | oh, I see, so if we know the multi-cell job only has two cells and we know what cell2 port is going to be, hard-code it into the job config | |
| 18:51:34 | mriedem | right we have NOVA_SERVICE_LISTEN_ADDRESS in devstack | |
| 18:51:42 | mriedem | for the host | |
| 18:51:53 | mriedem | which points back at the controller | |
| 18:51:56 | melwitt | because having devstack properly set up a separate nova-cpu.conf per cell and then getting the right ones passed in job configs would be a pretty big change | |
| 18:52:44 | mriedem | i think we'd add a devstack variable alongside NOVA_SERVICE_LISTEN_ADDRESS but for NOVNC_PROXY_PORT or something and default to '' but allow the job to set that via zuul/ansible on the subnode | |
| 18:53:30 | melwitt | I think anyway. I wondered about this when I did the patch for the different ports to avoid collisions but then it was working in your devstack env so I thought I was missing something. I briefly looked through thinking how different nova-cpu.conf could be made and it didn't look straightforward | |
| 18:55:05 | mriedem | it might just be a matter of taking the NOVA_CPU_CELL variable into account on the subnode | |
| 18:55:22 | mriedem | https://github.com/openstack/nova/blob/master/.zuul.yaml#L275 | |
| 18:56:20 | melwitt | oh, and letting devstack configure nova-cpu.conf accordingly? that would be a better way, if we can. because I'm thinking, in theory, there's more than just the novnc port, there's all the other possible console proxy ports too | |
| 18:56:43 | sean-k-mooney | mriedem: wait that bug was for the placemnet api | |
| 18:56:54 | mriedem | so rather than this https://review.opendev.org/#/c/649473/3/lib/nova@613 | |
| 18:57:01 | sean-k-mooney | we dont monkeypatch the placmenet api? | |
| 18:57:09 | mriedem | maybe configure_console_proxies should just be using NOVA_CPU_CELL as the offset | |
| 18:57:30 | mriedem | but... | |
| 18:57:34 | melwitt | oh, hmm | |
| 18:57:46 | mriedem | then we won't have separate n-novnc services for cell1 and cell2 on the controller | |
| 18:58:07 | mriedem | sean-k-mooney: in stein, placement is in nova | |
| 18:58:09 | mriedem | for some | |
| 18:58:55 | sean-k-mooney | starlingx is not using extracted plamcent? | |
| 18:59:39 | sean-k-mooney | eventls and wsgi shoudl normally be ok if you dont have long lived threads like the heartbeat | |
| 18:59:55 | sean-k-mooney | so even using the intree placment they should have been fine | |
| 19:00:32 | sean-k-mooney | oh they are hitting the other issue | |
| 19:00:45 | sean-k-mooney | with the cannot switch to a different trhead | |
| 19:01:20 | mriedem | most people are probably not going to be using extracted placement at stein ga | |
| 19:01:21 | sean-k-mooney | my comment still applies then | |
| 19:01:44 | mriedem | your comment about threads=1? | |
| 19:01:55 | sean-k-mooney | ya https://bugs.launchpad.net/nova/+bug/1829062/comments/7 | |
| 19:01:56 | openstack | Launchpad bug 1829062 in StarlingX "nova placement api non-responsive due to eventlet error" [Critical,In progress] - Assigned to Gerry Kopec (gerry-kopec) | |
| 19:02:04 | mriedem | if so, then my comment about a known issue release note applies :) | |
| 19:02:27 | sean-k-mooney | if they set wsgi to use 1 thread per process then they wont get the context swtich error | |
| 19:02:35 | sean-k-mooney | *thread switch | |
| 19:02:57 | melwitt | mriedem: I'm actually thinking that in devstack lib/nova, we could use NOVA_CPU_CELL to mimic the other offset code to set the ports properly in the one nova-cpu.conf it writes. I'll give it a try and see if it could work | |
| 19:03:22 | sean-k-mooney | also i spend the last 5 minutes trying to figure out how the starling x repos work and i am more confused then when i started | |
| 19:03:37 | openstackgerrit | Adam Spiers proposed openstack/nova master: Provide HW_CPU_AMD_SEV trait when SEV is supported https://review.opendev.org/638680 | |