| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-12-11 | |||
| 14:37:33 | efried | dansmith: because of all the *other* nova-net removal churn | |
| 14:37:38 | stephenfin | mocks, for one | |
| 14:38:00 | stephenfin | we no longer need to care about mocking stuff that nova-net was doing. If you backport a test, that goes back to not being true | |
| 14:38:05 | dansmith | efried: okay but that's hard like code conflict, not hard like "this file doesn't exist in the old branch" hard | |
| 14:38:41 | efried | I get it. | |
| 14:39:27 | openstackgerrit | sean mooney proposed openstack/nova master: support pci numa affinity policies in flavor and image https://review.opendev.org/674072 | |
| 14:39:40 | mriedem | anywho, i didn't -1 it and it's later in the series, just mentioning it | |
| 14:39:54 | mriedem | someone has to take on the current big ass bottom change that i +2ed already | |
| 14:39:56 | efried | stephenfin: Given that we have nova/image/glance.py, I don't see a problem with having nova/networking/neutron.py | |
| 14:40:02 | sean-k-mooney | efried: i have not added the spy function to check the pci assignment but i think i have made all the other chagnes you suggested | |
| 14:40:08 | stephenfin | dansmith: but it won't - git is smart | |
| 14:40:12 | mriedem | efried: that's a good point | |
| 14:40:16 | mriedem | we also have a cinder.py | |
| 14:40:24 | efried | nova/volume/cinder.py yeah. | |
| 14:40:45 | stephenfin | dansmith: Go make a trivial modification to nova/tests/unit/test_nova_manage.py, commit on master, then backport to stable/stein | |
| 14:40:47 | efried | we should make sure Sundar names it nova/accelerator/cyborg.py and we're golden. | |
| 14:41:00 | stephenfin | it'll happen cleanly, even though I renamed that file in commit nova/tests/unit/test_nova_manage.py | |
| 14:41:08 | stephenfin | whoops, commit 463017b51b8cde48582b2f55ad7a1f2321d03d02 | |
| 14:41:26 | mriedem | actually i did just -1 | |
| 14:42:23 | mriedem | i think having the module having neutron in the name would be less confusing to someone coming to hack on nova and knowing that there was at least once a nova-network thing, and finding nova/network/api.py and wondering if it's nova-net or neutron | |
| 14:42:34 | sean-k-mooney | efried: that would imply that that code is virt driver indepentet. if they cyborg code in that folder was then sure | |
| 14:42:36 | dansmith | stephenfin: aight, well, when we moved tests/ into tests/unit things were not "smart", but whatevs | |
| 14:42:38 | mriedem | since we have explicit glance.py and cinder.py modules those are pretty clear what they are for | |
| 14:42:52 | mriedem | dansmith: oh yeah the tests -> tests/unit was another big one | |
| 14:43:19 | efried | mriedem: oh, I hope we're getting rid of the API shim | |
| 14:43:27 | efried | stephenfin: ? ^ | |
| 14:43:34 | mriedem | efried: i'm fine with removing base_api.py and api.py (nova-net) | |
| 14:43:42 | mriedem | but i don't think we need to rename neutronv2.api to api.py | |
| 14:43:55 | efried | yeah, I'm fine leaving it named neutron | |
| 14:44:04 | sean-k-mooney | well on that how would people feel about eventurally moving the neutron code to os-vif? | |
| 14:44:34 | mriedem | why? | |
| 14:44:36 | efried | that sounds like a conversation for a far future release | |
| 14:44:57 | sean-k-mooney | well partly to not need to have any netowrking code in nova | |
| 14:44:59 | mriedem | "hey let's move some already really complicated and not very well understood code out to a library with a different core team" | |
| 14:45:10 | canori01 | Is it safe to rebuild the placement database? I have an issue where all my hypervisors are running into conflicts (conflicting resource provider name) | |
| 14:45:16 | sean-k-mooney | well all nova cores are os-vif cores | |
| 14:45:19 | canori01 | Can I just empty the db and bounce nova-compute? | |
| 14:45:27 | sean-k-mooney | but anyway its just an idea | |
| 14:45:33 | mriedem | sean-k-mooney: not worth the effort | |
| 14:45:38 | mriedem | pick your battles | |
| 14:45:56 | sean-k-mooney | its currently plugable so i was thinking of porting it then we could swap after | |
| 14:45:58 | sean-k-mooney | but ok | |
| 14:46:01 | mriedem | canori01: the inventory will rebuild itself automatically, the consumers/allocations will not | |
| 14:46:08 | sean-k-mooney | ill drop it for now | |
| 14:46:59 | mriedem | canori01: nova-manage placement heal_allocations should heal those up those if you do have to rebuild the placement db | |
| 14:47:13 | mriedem | *heal those up though | |
| 14:47:55 | sean-k-mooney | mriedem: that was added in rocky right. in queens we still had the periodic heal task form the move to placment | |
| 14:48:29 | mriedem | wrongish | |
| 14:48:57 | canori01 | mriedem: What happened is I used to have a third-party backup service that had service entries under nova. After removing them, all the hypervisors complain that an entry for them already exists and are unschedulable as a result | |
| 14:49:13 | canori01 | So my thought was to rebuild the placement db. I don't know if there's a better option | |
| 14:49:42 | sean-k-mooney | canori01: did you remvoe all the nova services | |
| 14:50:08 | canori01 | no, the nova services are still there | |
| 14:51:42 | canori01 | sean-k-mooney: would removing the service and bouncing nova-compute put things back in order? | |
| 14:53:07 | mriedem | kvm right? | |
| 14:53:23 | mriedem | the problem is placement has a unique constraint on the hostname, but the uuid on your computes has changed | |
| 14:53:41 | mriedem | the uuid on the compute_nodes table record that nova creates and uses to report the resource_providers to placement | |
| 14:55:19 | mriedem | so i think you're hitting some version of this https://bugs.launchpad.net/nova/+bug/1817833 | |
| 14:55:19 | openstack | Launchpad bug 1817833 in OpenStack Compute (nova) "Check compute_id existence when nova-compute reports info to placement" [Medium,In progress] - Assigned to Matt Riedemann (mriedem) | |
| 14:56:20 | sean-k-mooney | canori01: no. i asked because if you remvoed the service the uuid would change when the agent restarts but the hostname would be the same and would cause a resouce provider conflcit | |
| 14:57:06 | mriedem | if you really need to rebuild the placement db, then i think your steps would be: | |
| 14:57:12 | mriedem | 1. backup your current placement db | |
| 14:57:20 | mriedem | 2. drop it and rebuild the schema so it's empty | |
| 14:57:34 | mriedem | 3. let the computes report their inventory in which will create resource providers on the first run | |
| 14:57:40 | mriedem | 4. run: nova-manage placement heal_allocations | |
| 14:57:42 | stephenfin | mriedem: replied on https://review.opendev.org/#/c/696509/ | |
| 14:57:48 | mriedem | 5. run: nova-manage placement sync_aggregates | |
| 14:58:09 | stephenfin | tl;dr: if I don't do that req stuff, s*** breaks, so I did enough to make it work ¯\_(ツ)_/¯ | |
| 14:58:13 | mriedem | this is also assuming you aren't using some of the more advanced features like QoS ports in neutron | |
| 14:58:43 | canori01 | mriedem: I am not yet using QoS for neutron | |
| 14:59:12 | stephenfin | mriedem: also, it's way up the stack but you should definitely look at https://review.opendev.org/#/c/696746/ since it affects your security group caching changes. I think what I did is correct | |
| 14:59:31 | sean-k-mooney | canori01: it specficlly woudl only be an issue if you were using minium bandwidth qos policy | |
| 14:59:32 | mriedem | jesus that is a big change | |
| 14:59:44 | sean-k-mooney | canori01: the other qos polices do not interact with placment | |
| 15:00:12 | sean-k-mooney | canori01: are you useing routed networks out of interest. e.g. calico | |
| 15:00:15 | canori01 | sean-k-mooney: So if I sync the uuid on the database to match the uuid of my existing service entries, that would also solve the issue? | |
| 15:00:28 | canori01 | For example, nova service-list has: | |
| 15:00:38 | canori01 | 2c1037b3-4977-4a13-aea8-700a805cc11c | nova-compute | bctlz7nova36 | |
| 15:01:05 | sean-k-mooney | canori01: that is easier said then done as you would have to also consider exitsting allocation too but in principal yes | |
| 15:01:06 | canori01 | placement has: | 2019-10-22 19:12:06 | 2019-11-25 19:27:36 | 157 | c53e4b12-0b0b-4eaa-9fb1-373da8538cea | bctlz7nova36 | |
| 15:01:21 | mriedem | no those aren't the same | |
| 15:01:32 | mriedem | the nova services table uuid and nova compute_nodes uuid are not the same | |
| 15:01:41 | canori01 | ah ok | |
| 15:01:49 | sean-k-mooney | right the plcamment uuid is the compute node node uuid | |
| 15:02:13 | canori01 | sean-k-mooney: I'm not using calico. Just overlay vxlan networks advertised out with the neutron bgp agent | |
| 15:02:39 | sean-k-mooney | canori01: ok neutron report the network segment for routed networks to placmenet too | |
| 15:02:50 | sean-k-mooney | canori01: if you are using vxlan then you are fine | |
| 15:05:31 | mriedem | stephenfin: ok +2 on the 2nd from bottom change | |
| 15:05:40 | stephenfin | ta | |
| 15:05:58 | efried | stephenfin: and the bottom one is +A | |
| 15:06:06 | canori01 | mriedem: so would the safest course of action be to rebuild the placement db and heal the allocations? | |
| 15:06:09 | mriedem | i'm assuming canori01 didn't understand the question about what routed networks as a feature in neutron is | |
| 15:06:24 | mriedem | https://docs.openstack.org/neutron/latest/admin/config-routed-networks.html | |
| 15:06:30 | mriedem | tl;dr it relies on aggregates | |
| 15:06:42 | mriedem | and they aren't supported in nova anyway so it's a red herring here | |
| 15:07:36 | mriedem | canori01: i think doing that (what i laid about above) is likely more fool proof than trying to hack the uuids to get all synced up | |
| 15:08:14 | mriedem | disclaimer: if you run into problems with that this isn't a support channel nor am i your paid vendor so i'm not going to be walking you through every issue you run into :) | |
| 15:08:16 | efried | It would be a good real world test of that procedure, anyway. | |
| 15:08:29 | efried | A canori in a coal mine, so to speak. | |
| 15:08:33 | canori01 | mriedem: ok, thanks. Also, I'm definitely not using the routed networks. My provider network is just one segment | |
| 15:08:36 | mriedem | efried: indeed - test it in (someone else's) production | |