| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-05-24 | |||
| 18:48:05 | melwitt | http://logs.openstack.org/73/638073/31/check/nova-multi-cell/6ea3306/controller/logs/etc/nova/nova-cpu_conf.txt.gz vs http://logs.openstack.org/73/638073/31/check/nova-multi-cell/6ea3306/compute1/logs/etc/nova/nova-cpu_conf.txt.gz | |
| 18:48:59 | mriedem | ah yup | |
| 18:49:18 | melwitt | and from what I recall in devstack, I don't know how we can configure separate ones and use them in an easy way | |
| 18:49:53 | mriedem | we can pass that information into each node from the job config if we want to hard-code them | |
| 18:50:47 | melwitt | oh, I see, so if we know the multi-cell job only has two cells and we know what cell2 port is going to be, hard-code it into the job config | |
| 18:51:34 | mriedem | right we have NOVA_SERVICE_LISTEN_ADDRESS in devstack | |
| 18:51:42 | mriedem | for the host | |
| 18:51:53 | mriedem | which points back at the controller | |
| 18:51:56 | melwitt | because having devstack properly set up a separate nova-cpu.conf per cell and then getting the right ones passed in job configs would be a pretty big change | |
| 18:52:44 | mriedem | i think we'd add a devstack variable alongside NOVA_SERVICE_LISTEN_ADDRESS but for NOVNC_PROXY_PORT or something and default to '' but allow the job to set that via zuul/ansible on the subnode | |
| 18:53:30 | melwitt | I think anyway. I wondered about this when I did the patch for the different ports to avoid collisions but then it was working in your devstack env so I thought I was missing something. I briefly looked through thinking how different nova-cpu.conf could be made and it didn't look straightforward | |
| 18:55:05 | mriedem | it might just be a matter of taking the NOVA_CPU_CELL variable into account on the subnode | |
| 18:55:22 | mriedem | https://github.com/openstack/nova/blob/master/.zuul.yaml#L275 | |
| 18:56:20 | melwitt | oh, and letting devstack configure nova-cpu.conf accordingly? that would be a better way, if we can. because I'm thinking, in theory, there's more than just the novnc port, there's all the other possible console proxy ports too | |
| 18:56:43 | sean-k-mooney | mriedem: wait that bug was for the placemnet api | |
| 18:56:54 | mriedem | so rather than this https://review.opendev.org/#/c/649473/3/lib/nova@613 | |
| 18:57:01 | sean-k-mooney | we dont monkeypatch the placmenet api? | |
| 18:57:09 | mriedem | maybe configure_console_proxies should just be using NOVA_CPU_CELL as the offset | |
| 18:57:30 | mriedem | but... | |
| 18:57:34 | melwitt | oh, hmm | |
| 18:57:46 | mriedem | then we won't have separate n-novnc services for cell1 and cell2 on the controller | |
| 18:58:07 | mriedem | sean-k-mooney: in stein, placement is in nova | |
| 18:58:09 | mriedem | for some | |
| 18:58:55 | sean-k-mooney | starlingx is not using extracted plamcent? | |
| 18:59:39 | sean-k-mooney | eventls and wsgi shoudl normally be ok if you dont have long lived threads like the heartbeat | |
| 18:59:55 | sean-k-mooney | so even using the intree placment they should have been fine | |
| 19:00:32 | sean-k-mooney | oh they are hitting the other issue | |
| 19:00:45 | sean-k-mooney | with the cannot switch to a different trhead | |
| 19:01:20 | mriedem | most people are probably not going to be using extracted placement at stein ga | |
| 19:01:21 | sean-k-mooney | my comment still applies then | |
| 19:01:44 | mriedem | your comment about threads=1? | |
| 19:01:55 | sean-k-mooney | ya https://bugs.launchpad.net/nova/+bug/1829062/comments/7 | |
| 19:01:56 | openstack | Launchpad bug 1829062 in StarlingX "nova placement api non-responsive due to eventlet error" [Critical,In progress] - Assigned to Gerry Kopec (gerry-kopec) | |
| 19:02:04 | mriedem | if so, then my comment about a known issue release note applies :) | |
| 19:02:27 | sean-k-mooney | if they set wsgi to use 1 thread per process then they wont get the context swtich error | |
| 19:02:35 | sean-k-mooney | *thread switch | |
| 19:02:57 | melwitt | mriedem: I'm actually thinking that in devstack lib/nova, we could use NOVA_CPU_CELL to mimic the other offset code to set the ports properly in the one nova-cpu.conf it writes. I'll give it a try and see if it could work | |
| 19:03:22 | sean-k-mooney | also i spend the last 5 minutes trying to figure out how the starling x repos work and i am more confused then when i started | |
| 19:03:37 | openstackgerrit | Adam Spiers proposed openstack/nova master: Provide HW_CPU_AMD_SEV trait when SEV is supported https://review.opendev.org/638680 | |
| 19:03:56 | melwitt | I didn't realize there was a variable that would tell us what cell we (nova-compute) we are | |
| 19:04:04 | melwitt | -we | |
| 19:04:08 | aspiers | efried: this might be ready to go now ^^^ | |
| 19:04:24 | sean-k-mooney | melwitt: we do? | |
| 19:04:30 | mriedem | melwitt: ok, yeah that's probably easy to test with a nova patch depending on it since nova-multi-cell doesn't run on devstack changes (we could add it to the devstack experimental queue for testing stuff like this on-demand) | |
| 19:04:50 | mriedem | i'd just add nova-multi-cell to devstack experimental, i can push a change for that | |
| 19:04:51 | melwitt | sean-k-mooney: yeah NOVA_CPU_CELL, zuul job configs can set it | |
| 19:04:54 | sean-k-mooney | melwitt: oh a devstack variable | |
| 19:04:57 | melwitt | yes | |
| 19:05:14 | melwitt | mriedem: ok, sounds good | |
| 19:05:15 | sean-k-mooney | ah ok i taught you ment in the nova code | |
| 19:05:30 | melwitt | heh, yeah no | |
| 19:05:42 | openstackgerrit | Adam Spiers proposed openstack/nova master: Reduce logging of host hypervisor capabilities to DEBUG level https://review.opendev.org/661379 | |
| 19:09:07 | efried | aspiers: I was partway through PS12, was going to suggest moving your trait setting into _get_cpu_traits itself. And also, where's the bit where the SEV trait was going to be moved under the X86 namespace? | |
| 19:10:23 | aspiers | efried: I did consider putting it in _get_cpu_traits but that already seemed pretty over-grown | |
| 19:11:04 | openstackgerrit | Merged openstack/nova stable/rocky: [stable-only] Delete allocations even if _confirm_resize raises (part 2) https://review.opendev.org/659338 | |
| 19:11:12 | openstackgerrit | Merged openstack/nova stable/rocky: Add functional confirm_migration_error test https://review.opendev.org/658834 | |
| 19:11:18 | aspiers | efried: maybe it can be split up | |
| 19:11:46 | aspiers | efried: oh, I remember now - that wasn't the only reason | |
| 19:13:08 | aspiers | efried: I'm still behind with the latest news on the great CPU trait taxonomy debate | |
| 19:14:00 | aspiers | efried: this None check gave me pause https://opendev.org/openstack/nova/src/branch/master/nova/virt/libvirt/driver.py#L6777 | |
| 19:14:37 | aspiers | efried: *theoretically* all the traits could vanish from a host, and then this code would fail to remove them | |
| 19:15:04 | aspiers | but if that actually happened, maybe it would be a sure sign things have gone very badly wrong, in which case maybe that's a desirable accident anyway | |
| 19:16:11 | dansmith | efried: mriedem I'll admit I had my head on backwards with what I was expecting, but: http://logs.openstack.org/02/661002/4/check/nova-next/39819ea/job-output.txt.gz#_2019-05-24_18_14_23_954469 | |
| 19:16:20 | dansmith | seems to work to make sure we don't archive recent stuff | |
| 19:16:35 | mriedem | i just -1ed that :) | |
| 19:16:38 | mriedem | we are sympatico | |
| 19:16:45 | dansmith | my intent was to make the job actually call this with "tomorrow" or something | |
| 19:16:50 | mriedem | heh yeah | |
| 19:18:22 | efried | okay, that was clear to me, thought it was intentional. But I guess if this is the only place we test the script, it makes sense to invoke it in a way that does something. | |
| 19:18:52 | dansmith | efried: I was expecting to actually land this, so it needs to not not run | |
| 19:19:07 | dansmith | I was expecting to make it do what we do today, but with a date, which would be tomorrow not yesterday | |
| 19:19:30 | dansmith | but in reality, making sure it doesn't eff up today's records when called for yesterday is what we wanted to sanity check | |
| 19:19:31 | mriedem | plus note that purge failed | |
| 19:19:33 | mriedem | as a result | |
| 19:19:38 | mriedem | but b/c no set -e the job doesn't fail | |
| 19:19:46 | efried | dansmith: What about running it twice, once like this to make sure it outputs "Nothing was archived" and then the second time without --before (or with --before $(date -d tomorrow)) to make sure it works? | |
| 19:19:49 | mriedem | dansmith: you could run it both ways | |
| 19:19:53 | mriedem | jinx | |
| 19:19:57 | dansmith | yep, doing that now | |
| 19:20:22 | efried | what went wrong with purge? | |
| 19:20:30 | efried | And is there a fup change to turn on -e? | |
| 19:20:39 | mriedem | purge fails with rc=3 if nothing was purged | |
| 19:20:43 | mriedem | "fails" | |
| 19:20:59 | mriedem | i'm all fup'ed out | |
| 19:22:02 | efried | yeah, so 3 shouldn't be a failure condition, from the pov of the calling script, right? | |
| 19:24:17 | openstackgerrit | Dan Smith proposed openstack/nova master: Make nova-next archive using --before https://review.opendev.org/661002 | |
| 19:24:34 | dansmith | I'm just about done for the holiday, we can finish this up next week | |
| 19:25:15 | efried | aspiers: If the traits all vanished, that dict would still not be None. It would just have all False values. | |
| 19:28:23 | aspiers | efried: OK, because CPU_TRAITS_MAPPING is hardcoded. Makes sense | |
| 19:28:32 | efried | yah | |
| 19:28:54 | aspiers | _get_cpu_traits() is still too big though | |
| 19:28:56 | efried | aspiers: When mriedem was doing the trait reporting in the RT, I made sure we were covering that corner case | |
| 19:29:02 | aspiers | nice | |
| 19:29:11 | efried | aspiers: Sure, I would be fine splitting that up. | |
| 19:29:36 | aspiers | efried: well, shouldn't be too hard to do that and then move the SEV bit in there | |
| 19:30:06 | aspiers | efried: other than that and maybe some trait taxonomy thing which I don't yet know about, hopefully everything else is in order with this patch | |
| 19:30:09 | efried | aspiers: https://review.opendev.org/#/c/538498/6/nova/compute/resource_tracker.py@904 (patch look familiar?) | |
| 19:31:14 | aspiers | efried: yeah, I do remember reading that and vaguely understanding | |
| 19:31:23 | aspiers | I'm sure a re-read will make a lot more sense now | |
| 19:47:20 | mriedem | ganso: ok so i think things in queens are ok except the functional test is failing because of some missing test handling stuff on teardown in queens, working on that now | |