| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-07-25 | |||
| 16:03:55 | sean-k-mooney | stephenfin: or are you suggesting adding a src dir? | |
| 16:04:17 | stephenfin | sean-k-mooney: That I'm not sure about. No one else puts stuff inside a src dir so we can't really do that either | |
| 16:04:27 | stephenfin | sean-k-mooney: Assume it's a once off move on? | |
| 16:04:31 | stephenfin | *and move on | |
| 16:05:09 | sean-k-mooney | stephenfin: i guess we can add a test to one of the ci jobs. | |
| 16:06:07 | sean-k-mooney | ill add it to my list but let go with melwitt fix for now then. | |
| 16:06:25 | melwitt | we could make a dumb test that walks the entries in setup.cfg and raises if something is missing | |
| 16:06:56 | melwitt | I don't know of a smarter way to do it off the top of my head | |
| 16:07:46 | sean-k-mooney | melwitt: that is what i was thinking ya. specifically parsting the os_vif entrypoint and checking the plugin class path are rooted in a dir listed in packages | |
| 16:08:41 | stephenfin | You could do fancy things with pkg_resources too. I really don't think it's worth the effort though. If it bites us again, maybe then | |
| 16:10:03 | sean-k-mooney | i dont like leaveing gotchas like that lieing around. its not super high on my list but its now on my list | |
| 16:13:25 | jangutter | wow. that is doubleplusspam. | |
| 16:13:34 | sean-k-mooney | yeah spam | |
| 16:21:04 | openstackgerrit | Eric Fried proposed openstack/nova master: [placement] Add /reshaper handler for POST https://review.openstack.org/576927 | |
| 16:21:05 | openstackgerrit | Eric Fried proposed openstack/nova master: reshaper: Look up provider if not in inventories https://review.openstack.org/585033 | |
| 16:21:06 | openstackgerrit | Eric Fried proposed openstack/nova master: Make get_allocations_for_resource_provider sane https://review.openstack.org/584598 | |
| 16:21:07 | openstackgerrit | Eric Fried proposed openstack/nova master: Report client: Real get_allocs_for_consumer https://review.openstack.org/584599 | |
| 16:21:08 | openstackgerrit | Eric Fried proposed openstack/nova master: Report client: get_allocations_for_provider_tree https://review.openstack.org/584648 | |
| 16:21:09 | openstackgerrit | Eric Fried proposed openstack/nova master: Report client: _reshape helper, placement min bump https://review.openstack.org/585034 | |
| 16:21:10 | openstackgerrit | Eric Fried proposed openstack/nova master: WIP: Report client: update_from_provider_tree w/reshape https://review.openstack.org/585049 | |
| 16:21:11 | openstackgerrit | Eric Fried proposed openstack/nova master: WIP: Compute: Handle reshaped provider trees https://review.openstack.org/576236 | |
| 16:36:16 | sean-k-mooney | stephenfin: its passing the schduler because the network_metadata only contains info for the first instance on rebuild | |
| 16:36:43 | stephenfin | what do you mean? | |
| 16:37:25 | sean-k-mooney | stephenfin: on rebuild the network_metadata in the numa topology filter only list 1 of the 2 physnets requested | |
| 16:37:37 | sean-k-mooney | stephenfin: the one form the attached interface is missing | |
| 16:39:57 | stephenfin | sean-k-mooney: Yeah, I'm seeing something possibly related here. In my case, the info_cache.network_info is empty when I attempt to populate the request spec | |
| 16:40:11 | stephenfin | I'm trying to figure out what populates that in he first place in case we have that mocked out too | |
| 16:41:49 | sean-k-mooney | stephenfin: this https://review.openstack.org/#/c/585339/5/nova/objects/request_spec.py | |
| 16:42:59 | sean-k-mooney | i wonder if i need to call that again on attach. | |
| 16:43:56 | sean-k-mooney | there getting quicker at killing the spam | |
| 16:44:05 | stephenfin | sean-k-mooney: It should be called here https://review.openstack.org/#/c/585339/5/nova/conductor/manager.py | |
| 16:45:07 | openstackgerrit | Stephen Finucane proposed openstack/nova master: WIP: Add functional tests for cold migrate with NUMA networks https://review.openstack.org/585819 | |
| 16:45:38 | stephenfin | sean-k-mooney: But in the test above, it seems instance.info_cache.network_info is unpopulated. I don't know why | |
| 16:45:41 | sean-k-mooney | ok ill see if that does the right thing. that code is being called | |
| 16:48:58 | sean-k-mooney | so ya the instance.info_cache.network_info only has 1 vif so it was not updated when we did the attach | |
| 16:49:37 | stephenfin | sean-k-mooney: Yup, likely the same thing I'm seeing so. I wonder what causes that to be refreshed? | |
| 16:50:07 | stephenfin | I guess we need to call ComputeManager._heal_instance_info_cache | |
| 16:51:10 | sean-k-mooney | its proably missing in the fake driver | |
| 16:55:41 | sean-k-mooney | stephenfin: looks like that is normally called by neutron when it updates the port | |
| 17:01:25 | stephenfin | sean-k-mooney: I can only see it when we attach a fixed IP, no? | |
| 17:02:10 | sean-k-mooney | i dont see any calls to it directly in the codebase. its spawned as a periodic task so not sure how to call it | |
| 17:02:31 | sean-k-mooney | i just ran wait a breakpoint and it was never called during the test run | |
| 17:06:55 | stephenfin | sean-k-mooney: Yeah, and it is getting set in a production system. Dodgy mocks again | |
| 17:07:52 | sean-k-mooney | ya any idea of how to run them? | |
| 17:08:14 | sean-k-mooney | if i create an instance of the compute manager can i just call it? | |
| 17:09:13 | stephenfin | Not a clue. I guess I need to figure out how to set a watch on an attribute of an o.v.o object so I can see the callers? | |
| 17:09:46 | openstackgerrit | melanie witt proposed openstack/nova stable/ocata: [stable only] Always commit quota decrement if build request found https://review.openstack.org/582413 | |
| 17:11:59 | xgerman_ | hi, I am from Octavia and have a question: I had a compute host shutdown. Issued a nova delete and was checking in neutron if a port got detached… but that didn’t happen for 5 minutes. Wonder if there is a nova setting or if I need to increase that timeout… | |
| 17:12:09 | stephenfin | sean-k-mooney: fwiw, I know it works in "production". By default, DevStack deploys with an flat (l2) "public" network and vxlan "private" network. If I configure the two of those to use different NUMA nodes, boot the instance with a port from public, add a port from private and rebuild, it fails with NoValidHost | |
| 17:12:28 | stephenfin | So this has to be a mock not doing something but I just don't know what :) | |
| 17:13:30 | sean-k-mooney | stephenfin: my guess is that the comptue service we start is not running periodic tasks or the neutron mock is not sending the correct event | |
| 17:14:08 | sean-k-mooney | on the plus side i think i figured out how to call the heal method. | |
| 17:14:38 | stephenfin | I don't think it's the periodic task. If I put a breakpoint there (well, LOG.warning), the info_cache.network_info attribute looks correct beforehand | |
| 17:15:34 | stephenfin | sean-k-mooney: This looks promising https://github.com/openstack/nova/blob/master/nova/network/base_api.py#L33-L59 | |
| 17:24:05 | sean-k-mooney | stephenfin: we could just add a floating ip to the instance i guess. | |
| 17:24:20 | stephenfin | sean-k-mooney: Exactly what I'm trying to do at the moment | |
| 17:24:30 | stephenfin | well, fixed | |
| 17:24:58 | sean-k-mooney | do you know what context object we need | |
| 17:25:13 | stephenfin | In what context? | |
| 17:25:58 | sean-k-mooney | well we have a reference to the compute instance and call the heal fuction but i dont have a context to pass it | |
| 17:26:08 | sean-k-mooney | same with those api methods | |
| 17:26:26 | stephenfin | Ah, for those, I've no idea :( | |
| 17:26:47 | stephenfin | Unless you do it from the NeutronFixture, in which case I think self.context is a thing? | |
| 17:26:59 | stephenfin | We obviously shouldn't be doing that in the functional test itself | |
| 17:27:32 | sean-k-mooney | looks like we can call get_instance_nw_info and it will update the cache | |
| 17:29:49 | openstackgerrit | melanie witt proposed openstack/nova stable/ocata: [stable only] Always commit quota decrement if build request found https://review.openstack.org/582413 | |
| 17:42:57 | jangutter | Wow, looks like the killbot is getting quicker! | |
| 17:52:06 | sean-k-mooney | stephenfin: the periodic tasks are disable on the compute manager that is created by self._test_create_server_with_networks | |
| 17:54:17 | openstackgerrit | Merged openstack/os-vif master: Add vif_plug_noop to setup.cfg packages https://review.openstack.org/585530 | |
| 18:02:02 | openstackgerrit | Stephen Finucane proposed openstack/nova master: WIP: Add additional functional tests for NUMA networks https://review.openstack.org/585385 | |
| 18:17:47 | stephenfin | sean-k-mooney, dansmith, mriedem_away: A'ight, I've got a bus to Dublin in 6 hours and still need to pack. Gotta wrap this up :( | |
| 18:19:03 | dansmith | stephenfin: does that test work? | |
| 18:19:44 | stephenfin | dansmith: Nope. I think sean-k-mooney and I have the tests mostly right. Remaining blocker is that instance.info_cache.network_info isn't being updated. That happens when running IRL so I suspect a mocking issue but I've at a loss to figure out what I'm missing | |
| 18:20:11 | dansmith | okay that's the part I'm most suspicious about | |
| 18:20:13 | dansmith | you're saying if you're running devstack or something it works? | |
| 18:20:25 | stephenfin | Yeah | |
| 18:20:38 | stephenfin | (a) boot an instance with a physnet network attached | |
| 18:21:02 | stephenfin | (b) attach an interface from a tunneled network | |
| 18:21:09 | stephenfin | (c) rebuild and see NoValidHost | |
| 18:21:33 | dansmith | that's your IRL test? | |
| 18:22:02 | stephenfin | For lack of anything better, yes | |
| 18:22:12 | sean-k-mooney | dansmith: the periodic_tasks dont run in the functional tests so the instance.info_cache.network_info never gets updated. | |
| 18:22:19 | stephenfin | Given that's where I was going with the functional test until this afternoon | |
| 18:22:38 | dansmith | sean-k-mooney: that's how it gets _refreshed_ not how it ever gets set ever | |
| 18:23:15 | stephenfin | Naturally if I skip (b) or attach an interface from a network associated with the same NUMA node, it rebuilds just fine | |
| 18:23:29 | dansmith | stephenfin: you must be rebuilding with a new image right? | |
| 18:23:50 | dansmith | er, wait, I forget where we left the asking-scheduler-during-rebuild thing | |
| 18:23:53 | stephenfin | yes, to make sure the scheduler/claims code gets called | |
| 18:23:58 | dansmith | right okay | |
| 18:25:29 | stephenfin | The only things I can find that result in the network_info attribute getting re-built are attaching fixed/floating IPs, but I haven't been able to simulate that | |
| 18:25:41 | artom | dansmith, we left it in a working state, normally. But only some filters are run, as specified in the creatively named RUN_ON_REBUILD attribute ;) | |
| 18:26:04 | dansmith | artom: right and numa does run on rebuild right? | |
| 18:26:22 | stephenfin | dansmith: Yup, mriedem_away and I checked that yesterday | |
| 18:26:22 | dansmith | but what I'm not remembering is if we asking the scheduler _at_all_ if we aren't changing images | |
| 18:26:24 | artom | dansmith, ya | |
| 18:26:28 | sean-k-mooney | dansmith: ya im localy chaging the image and hitting the schduoler but the filter sees the instacne with only one interface because the info_cash only has one vif instead of 2 | |
| 18:26:37 | stephenfin | dansmith: That ^ | |
| 18:26:44 | artom | dansmith, I believe it's only if there's a new image | |
| 18:26:56 | dansmith | sean-k-mooney: yeah, doing this with an attach seems more complicated than the simplest case | |
| 18:26:58 | dansmith | artom: right okay | |